CC BY
13
LABELLING OF DATA ON FUNDUS COLOR PICTURES USED TO TRAIN A DEEP LEARNING MODEL ENHANCES ITS MACULAR PATHOLOGY RECOGNITION CAPABILITIES
Takhchidi HP1, Gliznitsa PV2 123, Svetozarskiy SN3, Bursov AI4, Shusterzon KA5
1 Pirogov Russian National Research Medical University, Moscow, Russia
2 OOO Innovatsioonniye Tekhnologii (Innovative Technologies, LLC), Nizhny Novgorod, Russia
3 Volga District Medical Center under the Federal Medical-Biological Agency, Nizhny Novgorod, Russia
4 Ivannikov Institute for System Programming of RAS, Moscow, Russia
5 L.A. Melentiev Energy Systems Institute, Irkutsk, Russia
Retinal diseases remain one of the leading causes of visual impairments in the world. The development of automated diagnostic methods can improve the efficiency and availability of the macular pathology mass screening programs. The objective of this work was to develop and validate deep learning algorithms detecting macular pathology (age-related macular degeneration, AMD) based on the analysis of color fundus photographs with and without data labeling. We used 1200 color fundus photographs from local databases, including 575 retinal images of AMD patients and 625 pictures of the retina of healthy people. The deep learning algorithm was deployed in the Faster RCNN neural network with ResNet50 for convolution. The process employed the transfer learning method. As a result, in the absence of labeling, the accuracy of the model was unsatisfactory (79%) because the neural network selected the areas of attention incorrectly. Data labeling improved the efficacy of the developed method: with the test dataset, the model determined the areas with informative features adequately, and the classification accuracy reached 96.6%. Thus, image data labeling significantly improves the accuracy of retinal color images recognition by a neural network and enables development and training of effective models with limited datasets.
Keywords: retinal diseases, fundus camera, machine learning, screening, biomedical visualization, data labeling
Funding: this work was financially supported by the Foundation for Assistance to Small Innovative Enterprises in Science and Technology (contract №150ГС1ЦТНТИС5/64226 dated December 22, 2020)
Author contribution: Takhchidi HP — manuscript editing; Gliznitsa PV — study concept and design, data collection and processing, results analysis, manuscript writing; Svetozarskiy SN — participation in data collection, literature and results analysis, manuscript writing; Bursov AI — literature analysis, algorithms development, manuscript editing; Shusterzon KA — algorithms development and validation, illustrations preparation, text writing.
[><1 Correspondence should be addressed: Pavel V. Gliznitsa
Belinskogo, 58/60, et. 5, 603000, Nizhny Novgorod; [email protected]
Received: 27.07.2021 Accepted: 15.08.2021 Published online: 28.08.2021
DOI: 10.24075/brsmu.2021.040
РАЗМЕТКА ЦВЕТНЫХ ФОТОГРАФИЙ ГЛАЗНОГО ДНА УЛУЧШАЕТ РАСПОЗНАВАНИЕ МАКУЛЯРНОЙ ПАТОЛОГИИ С ПОМОЩЬЮ ГЛУБОКОГО ОБУЧЕНИЯ
Х. П. Тахчиди1, П. В. Глизница2 С. Н. Светозарский3, А. И. Бурсов4, К. А. Шустерзон5
1 Российский национальный исследовательский медицинский университет имени Н. И. Пирогова, Москва, Россия
2 ООО «Инновационные технологии», Нижний Новгород, Россия
3 Приволжский окружной медицинский центр Федерального медико-биологического агентства, Нижний Новгород, Россия
4 Институт системного программирования имени В. П. Иванникова РАН, Москва, Россия
5 Институт систем энергетики имени Л. А. Мелентьева, Иркутск, Россия
Заболевания сетчатки остаются одной из ведущих причин слабовидения в мире. Разработка методов автоматизированной диагностики может повысить эффективность и доступность программ массового скрининга патологии макулярной области. Целью работы было разработать и провалидировать алгоритмы машинного обучения для диагностики макулярной патологии на основе анализа цветных фотографий глазного дна с предварительной разметкой данных и без нее на примере возрастной макулярной дегенерации (ВМД). В исследовании использовали 1200 цветных фотографий глазного дна из локальных баз данных, включая 575 изображений сетчатки пациентов с ВМД и 625 ретинальных фотографий здоровых пациентов. Алгоритм глубокого обучения был реализован на основе нейронной сети Faster RCNN c ResNet50 в качестве сверточной основы с использованием трансферного обучения. В результате, при отсутствии разметки валидация показала неудовлетворительную точность модели (79%), что было связано с неправильным выбором нейросетью областей внимания. Выполнение разметки повысило эффективность разработанной методики, на тестовом наборе данных модель продемонстрировала адекватное определение информативных участков, точность классификации достигла 96,6%. Таким образом, применение разметки изображений значительно повышает точность распознавания цветных изображений сетчатки с помощью нейросетевых технологий и позволяет создавать эффективные модели при использовании ограниченных по объему наборов данных.
Ключевые слова: болезни сетчатки, фундус-камера, машинное обучение, скрининг, биомедицинская визуализация, разметка данных
Финансирование: работа выполнена при финансовой поддержке Фонда содействия инновациям (договор №150ГС1ЦТНТИС5/64226 от 22.12.2020).
Вклад авторов: Х. П. Тахчиди — редактирование рукописи. П. В. Глизница — концепция и дизайн исследования, сбор и обработка данных, анализ результатов, написание текста рукописи; С. Н. Светозарский — участие в сборе данных, анализ результатов, работа с литературой, написание текста рукописи; А. И. Бурсов — работа с литературой, разработка алгоритмов, редактирование рукописи; К. А. Шустерзон — разработка и валидация алгоритмов, подготовка иллюстраций, участие в написании текста.
gg Для корреспонденции: Павел Викторович Глизница
ул. Белинского, д. 58/60, эт. 5, 603000, г. Нижний Новгород; [email protected]
Статья получена: 27.07.2021 Статья принята к печати: 15.08.2021 Опубликована онлайн: 28.08.2021
DOI: 10.24075/vrgmu.2021.040
In the Russian Federation, retinal diseases rank second and cause 28.9% of the visual impairment cases [1]. An effective retinal pathology early detection system that would be part of the mass preventive examination campaigns is yet to be deployed. Such systems require special logistics and dedicated staff, which, in addition to the one-time deployment expenses, translates into the need for regular funding to support the system and pay the people powering it. Computers can analyze big data faster, and machine learning algorithms automate the time-consuming and labor-intensive screening of patients to nominate those who need extensive examination. Thus, artificial intelligence capable of screening for eye diseases can mitigate the primary health care personnel shortage and reduce the clinical examination costs while increasing the number of patients reasonably referred to an ophthalmologist because of the suspected ophthalmic pathology [2].
Age-related macular degeneration (AMD), a retinal disease common among people aged 50 and over, remains one of the main causes of poor eyesight. The disease manifests in soft drusen measuring 63 pm or above in the macular zone, hyperpigmentation and/or hypopigmentation of the pigment epithelium, detachment of pigment and neuroepithelium, pigment epithelium geographic atrophy, retinal hemorrhages and cicatricial changes in the retina [3].
AMD is of great clinical and social importance. The prevalence of AMD among people aged 50 to 85 years is 8.69%, with 8.01% being early AMD and 0.37% late stage AMD [4]. Mathematical model forecasts growth of the absolute number of AMD patients from 196 million in 2020 to 288 million in 2040. [4]. Late stage AMD translates into a pronounced degradation of central vision, which worsens quality of life, limits daily living activities and impairs working capacity. Timely detection of the disease and adequate monitoring of the patients are instrumental to successful treatment of neovascular AMD because the efficacy of antiangiogenic therapy directly depends on the time elapsed from the moment of manifestation to administration of the first dose of the drug [5]. Fundus photography is a widely adopted and highly sensitive method of macular pathology visualization; it has been used in a number of countries for mass screening and yielded a significant increase of the early stage AMD detection rates [6].
The objective of this work was to develop and validate machine learning algorithms diagnosing macular pathology (AMD) based on the analysis of color pictures of the fundus with data labeled and unlabeled, and to assess sensitivity and specificity of the developed method with the help of a test dataset.
METHODS
The sets of color images of the fundus used in this study were collected at the Tsentr Zreniya clinic (Chelyabinsk) and the ophthalmological department of the Volga District Medical Center under FMBA of Russia (Nizhny Novgorod). All the pictures were taken with Visucam 500 fundus camera (Carl Zeiss; USA). The inclusion criteria applied to the images were: diagnosed AMD in one eye, registered in the patient's digital
medical record; presence of specific signs of AMD on the image; absence of signs of other retinal diseases (diabetic retinopathy, etc). Image quality was assessed in points on a scale from 1 to 4, the assessment relied on the method by Klais C et al., with 1 point given to high quality pictures, 2 points to average quality images, 3 points to those of low quality and 4 points to indiscernible pictures [7]. The images that scored 3-4 points were rejected. We used the widely adopted clinical classification of AMD that distinguishes early, intermediate and late stages of the disease (Table 1) [8]. The initial set of images was anonymized and blind classified independently by two ophthalmologists with over 5 years of experience.
The resulting set included 1200 color fundus photographs, including 575 retinal images of AMD patients and 625 pictures of the retina of healthy people. Under the AMD classification, 127 images were classified into the early AMD group, 341 were marked as intermediate stage and 107 as late stage AMD pictures.
The data were distributed into training and test sets randomly, with 994 images used in the neural network training (475 eyes with AMD, 519 eyes of healthy people) and 206 photographs used for testing (100 from patients with AMD, 106 from healthy people).
To accomplish the task set, we practiced two approaches to training:
1) training a convolutional neural network (CNN) on a dataset consisting of binary classified images without specified regions of interest;
2) training a CNN on a dataset consisting of binary classified images with the regions of interest specified in bounding boxes;
We relied on the ResNet-50 deep learning architecture and transfer learning for both approaches [9]. Transfer learning involves use of CNNs that are pretrained on a large set of third-party data. Following pretraining, the network, which already has its weighing system set up, goes through training on a small set of data of immediate interest. The large set of third-party data used for pretraining in this work was the ImageNET dataset, which includes millions of images divided into 1000 different classes [10].
Fundus pictures from the local databases were preprocessed (converted to 512 x 512 pixel images) and then processed by a pretrained Faster RCNN neural network with ResNet50 enabling convolution. Each output window was linked with a category tag and a softmax score at [0, 1]. A score threshold of 0.7 was used to display these images. The execution time needed to obtain these results was 120 ms per image, all steps included. All in all, the image analysis sequence can be outlined as follows: preprocessing, processing by the CNN with a feature map as output, highlighting regional suggestions thereon, determining regions of interest and classifying the image as either an AMD picture or a normal eye photograph based on the features found within the regions of interest (Fig. 1).
All algorithms were developed in Python 3.7 using libraries PyTorch 1.5.0, TorchVision 0.6.0, Tensorflow 1.14.0, Keras 2.0.8, Pillow 7.2, OpenCV 4.5.2, Cuda 10.1, cudnn 7.6.5. The hardware configuration of the computer used to do the
Table 1. Clinical classification of AMD [8]
Stage Symptoms
Normal age-related changes Small druses up to 63 pm, no pigmentation defects
Early Druses with a diameter of 63-125 microns, no pigmentation defects
Intermediate Druses with a diameter over 125 microns, pigmentation defective
Late Neovascular form or geographic atrophy
Classifier Classifier
Map of attributes
Fig. 1. Stages of image analysis by Faster RCNN
calculations was as follows: Intel Core i7 9750H (Intel; USA), RTX 2070 Max-Q 8GB GDDR6, 16 GB RAM 2666 MHz.
RESULTS
Image classification by a CNN without specified regions of interest
All color images of the fundus belonging to the training set were reduced to a resolution of 512 x 512 pixels and normalized to the average pixel. Then the dataset was submitted to the neural network for training. The training lasted 193 min and took 50 iterations. A batch (combined load) included 10 images. Nesterov accelerated gradient was used as an optimizer; the learning rate parameter was 0.0005, the moment was 0.9. Loss function — categorical cross-entropy, metric — accuracy.
Validation of the resulting model on the test dataset revealed that its specificity reached 77.4%, sensitivity — 80.9%, accuracy — 79% (Table 2). To learn what regions of the images the model used for classification we imported the class activation heatmaps (Fig. 2).
Fundus image
As a result, it was found that the network selected the areas of attention incorrectly: one of them was the area of the optic nerve head, which is not involved in AMD's pathological process, another — paramacular area. Thus, the neural network used incorrect features in training, which nevertheless correlate with the classification result.
Image classification by a CNN with regions of interest pre-specified
The training dataset was the same as for the first approach, but for this case, we marked the macular region as the region of interest with the help of bounding boxes. All the images were reduced to a resolution of 512 x 512 pixels and normalized to the average pixel. Faster RCNN + FPN network combination enabled object detection [11]. The training lasted 158 min and took 10 iterations. A batch included 10 images. Nesterov accelerated gradient was used as an optimizer; the learning rate parameter was 0.0001, the moment was 0.05, weight decay — 0.0005. Classification categorical cross-entropy was the loss function, mean average accuracy was the classification accuracy metric,
Table 2. Developed models' performance indicators reflecting the quality of detection of AMD in color fundus photographs
Indicator Machine learning without labeling Machine learning with pre-labeling
Sensitivity 80,9% 99,0%
Specificity 77,4% 94,3%
Accuracy 79% 96,6%
Positive result predictability 74% 94,3%
Negative result predictability 82% 99,0%
Fig. 2. Class activation heatmap visualization example, fundus photograph of an AMD patient
intersection over union — detection accuracy metric. The training was stopped after 10 iterations because of the emerging overtraining effect [12].
On the test dataset, the model demonstrated the classification accuracy of 96.6% at sensitivity of 99.0% and specificity of 94.3% (Table 2). Visualization of the areas of interest showed that the model identified informative areas of the images adequately (Fig. 3).
DISCUSSION
This study showed that Faster RCNN neural network with ResNet50 enabling convolution can effectively differentiate between AMD patient fundus pictures and those of healthy
retina. We have also established that even with a small sample (1200 images) the resulting classification accuracy can be high if the data are pre-labeled.
Researchers investigating application of neural networks to diagnose AMD through analysis of color pictures of the retina reported sensitivity of 84.5-89.0%, specificity of 83.1-89.0% and accuracy of 88.4-91.6% [13, 14]. One study aimed to detect AMD at the early stage using images of the fundus; its authors claimed to have achieved sensitivity and specificity of 96.7%, 96.4% [15]. The datasets used in these works were not pre-labeled, but each of them relied on the sample comprised of over 50000 images, which is an order of magnitude greater than the sample used for this study [13-15]. In this connection, it is interesting to note that, considering the
Fig. 3. Results of detection of regions of interest and classification of images from the test dataset by Faster RCNN with ResNet50 for convolution. Images correctly identified by the model as healthy retina photographs have green boxes, those with AMD detected have red boxes
relatively small dataset employed, by some parameters we received comparable results with the help of a simple and fast labeling procedure.
A meta-analysis of 13 studies averaged the neural networks' sensitivity and specificity in AMD detection at 0.92 and 0.89, respectively [16]. However, this analysis included studies that made use of fundus camera images exclusively and works that relied on the pictures obtained with optical coherence tomography. Another meta-analysis considered papers reporting on the automated AMD diagnosing models that processed only color photographs of the retina; this analysis averaged the models' sensitivity and specificity at 0.88 and 0.90, respectively [17]. Thus, the level of accuracy we have achieved is comparable to the results of studies based on much larger datasets.
It should be noted that instant AMD diagnostics using color images of the fundus traditionally underpins the relevant mass screening programs, but has limited application in specialized care. What shows promise in this field is the determination of AMD stages from the available dataset [18-20] and the identification of individual pathological elements in the images [21], which can serve the purposes of monitoring in the context of clinical observation and during clinical trials.
On the one hand, small size of the training dataset and the decision to not differentiate between stages of AMD (we used one class for all of them) can be considered a limitation of this work. On the other hand, with these prerequisites, we managed
to answer the questions posed. The small dataset confirmed that, with a limited sample available at a local database, it is possible to successfully develop models capable of automated retinal disease diagnosing provided the training dataset is pre-labeled. The clinical heterogeneity of pathological changes allows simulation of a real life screening situation, where it is necessary to detect various pathologies with high sensitivity in order to refer the patients for further examination.
CONCLUSIONS
Automated diagnostics of retinal diseases, which are among the top causes of blindness and poor eyesight, opens new opportunities for mass screening for AMD. The fast and easy-to-use method of image markup with bounding boxes significantly increases accuracy of the developed methods of recognition of medical images relying on neural networks. As a result, it is possible to achieve high classification accuracy even when there are only small local databases available. At the same time, it underscores the importance of the role played by medical specialists in the development of new diagnostic methods based on machine learning, which requires consolidation of efforts of ophthalmologists and IT engineers in order to create large annotated databases of retinal images collected with various models of fundus cameras, which, when labeling the data thereon, would ensure high accuracy and reproducibility of the results in real clinical practice.
References
1. Nazarjan MG, Vertash OYu. Analiz pokazatelej pervichnoj i povtornoj invalidnosti vsledstvie boleznej glaza u lic pensionnogo vozrasta v Rossijskoj Federacii i Moskve. Uspehi gerontologii. 2019; 32 (1-2): 215-17. Russian.
2. Armstrong GW, Lorch AC. A(eye): A Review of Current Applications of Artificial Intelligence and Machine Learning in Ophthalmology. Int Ophthalmol Clin. 2020; 60 (1): 57-71. DOI: 10.1097/II0.0000000000000298. PMID: 31855896.
3. Bird AC, Bressler NM, Bressler SB, Chisholm IH, Coscas G, Davis MD, et al. An international classification and grading system for age-related maculopathy and age-related macular degeneration. The International ARM Epidemiological Study Group. Surv Ophthalmol. 1995; 39 (5): 367-74.
4. Wong WL, Su X, Li X, et al. Global prevalence of age-related macular degeneration and disease burden projection for 2020 and 2040: a systematic review and meta-analysis. Lancet Glob Health. 2014; 2 (2): e106-e116. DOI: 10.1016/S2214-109X(13)70145-1.
5. Schwartz R, Warwick A, Olvera-Barrios A, Pikoula M, Lee AY, Denaxas S, et al. Evolving treatment patterns and outcomes of neovascular age-related macular degeneration over a decade. Ophthalmol Retina. 2021; 5 (8): e11-e22. DOI: 10.1016/j. oret.2021.04.001. Epub ahead of print. PMID: 33866023.
6. Danis RP, Domalpally A, Chew EY, et al. Methods and reproducibility of grading optimized digital color fundus photographs in the Age-Related Eye Disease Study 2 (AREDS2 Report Number 2). Invest Ophthalmol Vis Sci. 2013; 54 (7): 4548-54.
7. Klais C, et al. Photoscreening for diabetic retinopathy: a comparison of image quality between film photography and digital imaging. Clin Experiment Ophthalmol. 2004; 32: 393-6
8. Ferris 3rd FL, Wilkinson CP, Bird A, et al. Clinical classification of age-related macular degeneration. Ophthalmology. 2013; 120 (4): 844-51.
9. He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016, pp. 770-778. DOI: 10.1109/ CVPR.2016.90.
10. Deng J, Dong W, Socher R, Li L, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009; pp. 248-255. DOI: 10.1109/CVPR.2009.5206848.
11. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards RealTime Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017; 39 (6): 1137-49. DOI: 10.1109/TPAMI.2016.2577031.
12. Lee CS, Baughman DM, Lee AY. Deep learning is effective for the classification of OCT images of normal versus Age-related Macular Degeneration. Ophthalmol Retina. 2017; 1 (4): 322-7. DOI: 10.1016/j.oret.2016.12.009.
13. Burlina P, Joshi N, Pacheco KD, Freund DE, Kong J, Bressler NM. Utility of Deep Learning Methods for Referability Classification of Age-Related Macular Degeneration. JAMA Ophthalmol. 2018; 136 (11): 1305-7. DOI: 10.1001/jamaophthalmol.2018.3799.
14. Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated Grading of Age-Related Macular Degeneration From Color Fundus Images Using Deep Convolutional Neural Networks. JAMA Ophthalmol. 2017 Nov 1; 135 (11): 1170-6. DOI: 10.1001/jamaophthalmol.2017.3782. PMID: 28973096; PMCID: PMC5710387.
15. Keel S, Li Z, Scheetz J, Robman L, Phung J, Makeyeva G, et al. Development and validation of a deep-learning algorithm for the detection of neovascular age-related macular degeneration from colour fundus photographs. Clin Exp Ophthalmol. 2019 Nov; 47 (8): 1009-18. DOI: 10.1111/ceo.13575. Epub 2019 Jul 25. PMID: 31215760.
16. Cheung R, Chun J, Sheidow T, Motolko M, Malvankar-Mehta MS. Diagnostic accuracy of current machine learning classifiers for age-related macular degeneration: a systematic review and metaanalysis. Eye (Lond). 2021 May 6. DOI: 10.1038/s41433-021-01540-y. Epub ahead of print. PMID: 33958739.
17. Dong L, Yang Q, Zhang RH, Wei WB. Artificial intelligence for the detection of age-related macular degeneration in color fundus photographs: A systematic review and meta-analysis.
E Clinical Medicine. 2021 May 8; 35: 100875. DOI: 10.1016/j. eclinm.2021.100875. PMID: 34027334; PMCID: PMC8129891.
18. Grassmann F, Mengelkamp J, Brandl C, Harsch S, Zimmermann ME, Linkohr B, et al. A Deep Learning Algorithm for Prediction of Age-Related Eye Disease Study Severity Scale for Age-Related Macular Degeneration from Color Fundus Photography. Ophthalmology. 2018 Sep; 125 (9): 1410-20. DOI: 10.1016/j. ophtha.2018.02.037. Epub 2018 Apr 10. PMID: 29653860.
19. Peng Y, Dharssi S, Chen Q, Keenan TD, Agrön E, Wong WT, et al. DeepSeeNet: A Deep Learning Model for Automated Classification of Patient-based Age-related Macular Degeneration Severity from Color Fundus Photographs. Ophthalmology. 2019 Apr; 126 (4): 565-75. DOI: 10.1016/j.ophtha.2018.11.015. Epub
2018 Nov 22. PMID: 30471319; PMCID: PMC6435402.
20. Keenan TD, Dharssi S, Peng Y, Chen Q, Agron E, Wong WT, et al. A Deep Learning Approach for Automated Detection of Geographic Atrophy from Color Fundus Photographs. Ophthalmology. 2019 Nov; 126 (11): 1533-40. DOI: 10.1016/j. ophtha.2019.06.005. Epub 2019 Jun 11. PMID: 31358385; PMCID: PMC6810830.
21. Keenan TDL, Chen Q, Peng Y, Domalpally A, Agron E, Hwang CK, et al. Deep Learning Automated Detection of Reticular Pseudodrusen from Fundus Autofluorescence Images or Color Fundus Photographs in AREDS2. Ophthalmology. 2020 Dec; 127 (12): 1674-87. DOI: 10.1016/j.ophtha.2020.05.036. Epub 2020 May 21. PMID: 32447042.
Литература
1. Назарян М. Г., Верташ О. Ю. Анализ показателей первичной и повторной инвалидности вследствие болезней глаза у лиц пенсионного возраста в Российской Федерации и Москве. Успехи геронтологии. 2019; 32 (1-2): 215-17.
2. Armstrong GW, Lorch AC. A(eye): A Review of Current Applications of Artificial Intelligence and Machine Learning in Ophthalmology. Int Ophthalmol Clin. 2020; 60 (1): 57-71. DOI: 10.1097/Il0.0000000000000298. PMID: 31855896.
3. Bird AC, Bressler NM, Bressler SB, Chisholm IH, Coscas G, Davis MD, et al. An international classification and grading system for age-related maculopathy and age-related macular degeneration. The International ARM Epidemiological Study Group. Surv Ophthalmol. 1995; 39 (5): 367-74.
4. Wong WL, Su X, Li X, et al. Global prevalence of age-related macular degeneration and disease burden projection for 2020 and 2040: a systematic review and meta-analysis. Lancet Glob Health. 2014; 2 (2): e106-e116. DOI: 10.1016/S2214-109X(13)70145-1.
5. Schwartz R, Warwick A, Olvera-Barrios A, Pikoula M, Lee AY, Denaxas S, et al. Evolving treatment patterns and outcomes of neovascular age-related macular degeneration over a decade. Ophthalmol Retina. 2021; 5 (8): e11-e22. DOI: 10.1016/j. oret.2021.04.001. Epub ahead of print. PMID: 33866023.
6. Danis RP, Domalpally A, Chew EY, et al. Methods and reproducibility of grading optimized digital color fundus photographs in the Age-Related Eye Disease Study 2 (AREDS2 Report Number 2). Invest Ophthalmol Vis Sci. 2013; 54 (7): 4548-54.
7. Klais C, et al. Photoscreening for diabetic retinopathy: a comparison of image quality between film photography and digital imaging. Clin Experiment Ophthalmol. 2004; 32: 393-6
8. Ferris 3rd FL, Wilkinson CP, Bird A, et al. Clinical classification of age-related macular degeneration. Ophthalmology. 2013; 120 (4): 844-51.
9. He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016, pp. 770-778. DOI: 10.1109/ CVPR.2016.90.
10. Deng J, Dong W, Socher R, Li L, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009; pp. 248255. DOI: 10.1109/CVPR.2009.5206848.
11. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards RealTime Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017; 39 (6): 1137-49. DOI: 10.1109/TPAMI.2016.2577031.
12. Lee CS, Baughman DM, Lee AY. Deep learning is effective for the classification of OCT images of normal versus Age-related Macular Degeneration. Ophthalmol Retina. 2017; 1 (4): 322-7. DOI: 10.1016/j.oret.2016.12.009.
13. Burlina P, Joshi N, Pacheco KD, Freund DE, Kong J, Bressler NM.
Utility of Deep Learning Methods for Referability Classification of Age-Related Macular Degeneration. JAMA Ophthalmol. 2018; 136 (11): 1305-7. DOI: 10.1001/jamaophthalmol.2018.3799.
14. Burlina PM, Joshi N, Pekala M, Pacheco KD, Freund DE, Bressler NM. Automated Grading of Age-Related Macular Degeneration From Color Fundus Images Using Deep Convolutional Neural Networks. JAMA Ophthalmol. 2017 Nov 1; 135 (11): 1170-6. DOI: 10.1001/jamaophthalmol.2017.3782. PMID: 28973096; PMCID: PMC5710387.
15. Keel S, Li Z, Scheetz J, Robman L, Phung J, Makeyeva G, et al. Development and validation of a deep-learning algorithm for the detection of neovascular age-related macular degeneration from colour fundus photographs. Clin Exp Ophthalmol. 2019 Nov; 47 (8): 1009-18. DOI: 10.1111/ceo.13575. Epub 2019 Jul 25. PMID: 31215760.
16. Cheung R, Chun J, Sheidow T, Motolko M, Malvankar-Mehta MS. Diagnostic accuracy of current machine learning classifiers for age-related macular degeneration: a systematic review and metaanalysis. Eye (Lond). 2021 May 6. DOI: 10.1038/s41433-021-01540-y. Epub ahead of print. PMID: 33958739.
17. Dong L, Yang Q, Zhang RH, Wei WB. Artificial intelligence for the detection of age-related macular degeneration in color fundus photographs: A systematic review and meta-analysis. E Clinical Medicine. 2021 May 8; 35: 100875. DOI: 10.1016/j. eclinm.2021.100875. PMID: 34027334; PMCID: PMC8129891.
18. Grassmann F, Mengelkamp J, Brandl C, Harsch S, Zimmermann ME, Linkohr B, et al. A Deep Learning Algorithm for Prediction of Age-Related Eye Disease Study Severity Scale for Age-Related Macular Degeneration from Color Fundus Photography. Ophthalmology. 2018 Sep; 125 (9): 1410-20. DOI: 10.1016/j. ophtha.2018.02.037. Epub 2018 Apr 10. PMID: 29653860.
19. Peng Y, Dharssi S, Chen Q, Keenan TD, Agron E, Wong WT, et al. DeepSeeNet: A Deep Learning Model for Automated Classification of Patient-based Age-related Macular Degeneration Severity from Color Fundus Photographs. Ophthalmology. 2019 Apr; 126 (4): 565-75. DOI: 10.1016/j.ophtha.2018.11.015. Epub 2018 Nov 22. PMID: 30471319; PMCID: PMC6435402.
20. Keenan TD, Dharssi S, Peng Y, Chen Q, Agron E, Wong WT, et al. A Deep Learning Approach for Automated Detection of Geographic Atrophy from Color Fundus Photographs. Ophthalmology. 2019 Nov; 126 (11): 1533-40. DOI: 10.1016/j. ophtha.2019.06.005. Epub 2019 Jun 11. PMID: 31358385; PMCID: PMC6810830.
21. Keenan TDL, Chen Q, Peng Y, Domalpally A, Agron E, Hwang CK, et al. Deep Learning Automated Detection of Reticular Pseudodrusen from Fundus Autofluorescence Images or Color Fundus Photographs in AREDS2. Ophthalmology. 2020 Dec; 127 (12): 1674-87. DOI: 10.1016/j.ophtha.2020.05.036. Epub 2020 May 21. PMID: 32447042.
