Evaluating the Influence of Clinical Data on Inter-Observer Variability in Optic Disc Analysis for AI-Assisted Glaucoma Screening

Introduction

Glaucoma is the leading cause of irreversible blindness worldwide, affecting 3.5% of individuals aged 40 to 80 years.¹ Over half of cases remain undiagnosed due to the asymptomatic nature of early disease,² often leading to advanced stages with significant visual impairment, impacting the quality of the patient’s life. Additionally, the economic burden associated with late-stage disease is substantial; the direct care costs for advanced glaucoma (stages 4–5, Bascom Palmer system) are 1.7 times higher than for early-stage disease.³ These challenges highlight an urgent need for improved strategies for early detection to prevent blindness.

However, mass screening faces logistical and clinical challenges, including reliance on skilled personnel, limited diagnostic tools, and high patient volumes in underserved regions. Inter-observer variability in optic disc assessment further complicates glaucoma diagnosis, emphasizing the need for innovative tools to enhance accuracy.

Recent advancements in artificial intelligence (AI), particularly deep learning, have demonstrated diagnostic performance exceeding 95% AUC in internal datasets.^4–6 AI has the potential to reduce variability and provide consistent evaluations, making it invaluable in busy clinics and resource-limited settings.

This study originated from a glaucoma screening campaign conducted on World Sight Day. Seventy individuals were screened, and fundus photographs were rapidly classified as normal or abnormal, reflecting the time constraints typical of a busy outpatient clinic condition. The observed unexpected variability during this rapid decision-making process among trained professionals motivated further investigation with masked evaluations by experienced colleagues, transforming this campaign into a research opportunity.

Purpose

To evaluate inter-observer variability in optic disc classification using colour fundus photographs, with and without intraocular pressure (IOP) data, and to assess the reliability of ground truth labels based solely on fundus images for the external validation of AI algorithms in glaucoma detection.

Methods

This cross-sectional observational study was conducted at our hospital in accordance with the principles of the Declaration of Helsinki. The Institutional Ethical Commission of the Cliniques Universitaires Saint-Luc in Brussels approved the study protocol (OVO-AI Study: Observer Variability in Optic disc and AI Implications). This research adheres to the principles outlined in the Declaration of Helsinki for studies involving human subjects. It complies with all relevant national ethical guidelines, including the General Data Protection Regulation (GDPR) for data privacy and confidentiality within Europe. While HIPAA is specific to the United States, the GDPR ensures similar standards in the European context. Written informed consent was obtained from each participant before inclusion in the study.

Fundus images from both eyes were obtained from a cohort of 70 individuals presenting randomly during a glaucoma screening campaign in the hospital. The only two exclusion criteria were age and a confirmed diagnosis of glaucoma. The minimum age for screening was set at 30 years due to the racial heterogeneity (Caucasians, Asians, Latinos, and African descendants) in Brussels. The clinical parameters were age, bilateral fundus pictures, taken with a Crystalvue NFC600 camera and intraocular pressure measured with air tonometry, NIDEK Tonoref III.

The Crystalvue NFC600 camera is equipped with MONA AI-driven software capable of providing a probability score for glaucoma based on fundus images. MONA, a Belgian healthcare AI startup, is a spin-off of KULeuven and VITO (the Flemish technology agency). Initially developed for detecting diabetic retinopathy (DR) and diabetic macular edema (DME), MONA AI was trained on an extensive dataset of over 273,000 retinal images. The system achieved a sensitivity of 90% and specificity of 95% for referable DR and similarly high accuracy for DME detection. Its convolutional neural network (CNN) architecture leverages these vast datasets to identify subtle pathological changes with high consistency. Although specific details about MONA’s glaucoma training dataset are unavailable, it likely employs a comparable AI framework, assigning optic disc scores from 0.0 to 1.0, with higher scores indicating a greater likelihood of glaucoma.

While the MONA AI system is not yet commercially available, its use in this study was dictated by camera availability. AI-generated probability scores were accessed post hoc for supplementary analysis and were not part of the initial evaluations. The inclusion of MONA was not intended as validation or a comparative assessment of the software.

Fundus photographs were visually evaluated by two masked glaucoma specialists (SP and GB) who categorized the images as normal (NL) or abnormal (GS) including glaucoma and glaucoma suspects, in real-time, simulating the time constraints of a busy outpatient clinic. The classification adhered to the European Glaucoma Society (EGS) guidelines,⁷ using parameters such as optic disc size and rim, vessel bearing, peripapillary haemorrhages, and cup-to-disc ratio.

Intraocular pressure (IOP) was subsequently measured using an air tonometer (NIDEK Tonoref III), which underwent routine calibration per manufacturer recommendations to ensure accuracy. Elevated IOP values were defined as >21 mmHg. Patients with abnormal fundus classifications (GS) and/or elevated IOP were referred for further ophthalmological evaluation. During the screening, one asymptomatic patient showed an important branch occlusion (Figure 1) and was directly sent to the Ophthalmology emergency for further examination and was diagnosed with a toxoplasmosis-caused occlusion. He was discarded from the study.

Figure 1 Asymptomatic patient during screening with branch occlusion and vasculitis caused by recent toxoplasmosis infection.

In the second phase, the fundus photographs of 138 optic discs were reanalyzed by four ophthalmologists (three glaucoma specialists and one resident). The images were categorized into three groups: normal (N), glaucoma-suspect (S), and glaucomatous (G) per EGS guidelines. Vertical Cup-to-Disc Ratio (VCDR) values were recorded for each optic nerve. The classifications were conducted in a masked manner without knowledge of IOP or the condition of the contralateral disc. Following unblinding, IOP and contralateral disc appearance were revealed sequentially, allowing observers to update their classifications.

Statistical Analysis

The Kappa coefficients were used to assess the agreement among different specialists for categorical variables and the Weighted Kappa coefficients for categorical ordinal variables. Kappas with their standard error are reported.

Intraclass correlation coefficients (ICC) were used to evaluate agreement in VCDR scoring on the image level for continuous variables.

Results

138 images of 69 subjects were taken. The statistical descriptive data of the continuous variables are shown in Table 1.

Table 1 Descriptive Analysis of the Participants

Results from the Hospital Screening

During the screening in the hospital, the flow of incoming patients limited the decision and scoring time. It was nearly instantaneous, less than one minute per eye mimicking a normal clinical setting.

Of the total 138 right and left eye pictures, 120 were labelled NL (61/ 69 for the RE, 61 and 59/ 69 for the LE). Eighteen subjects were referred for further examination based solely on their fundus photographs when either observer classified one or both eyes as GS. The number of referred subjects increased to 27, based on their initial allocation to the GS group and, or high IOP, after revealing the eye pressure (Table 2). It indicates that IOP, for clinicians, is considered a crucial and independent risk factor for further investigation.

Table 2 Number of Referred Patients with or Without Knowledge of the IOP

The concordance between the 2 masked observers (Table 3) during the screening, where the observers took the pictures and scored them only as NL or GS without knowing the IOP, is quantified by the kappa’s values for both eyes (kappa ± SE = 0.55 ± 0 0.11, p < 0.001). It means an acceptable reproducibility between the glaucoma specialist (SP) and her junior resident (GB) score.⁸

Table 3 Reproducibility Between SP & GB Based on Fundus Pictures

The Kappa values for the initial assessment (without IOP) indicate “moderate” agreement (Table 4).

Table 4 Reproducibility Between SP & GB Based on Fundus Pictures with or Without IOP

The values of the weighted Kappa’s decline when the IOP is revealed. The lower Kappa values after including IOP fall into the “fair” agreement category, suggesting less consistency when IOP data is factored in. This reduction in Kappa values suggests that the inclusion of IOP data may have introduced some variability in the assessment, leading to a lower level of agreement between the observers. This variability could arise from differing interpretations of the significance of IOP values in the context of fundus images.

Results from the Masked Picture Analysis

In the second phase of the study, three glaucoma specialists (SP, LO, CM) and one glaucoma resident (GB) independently reviewed the fundus photographs under various conditions to assess classification consistency. The evaluations were performed in the following scenarios:

1. Separate Classification: Each eye was assessed individually without additional information or time constraints.
2. Incorporating Contralateral Eye Information: Observers compared the images of both eyes from the same subject.
3. With and Without IOP Consideration: Observers classified the images without IOP data initially, followed by a second round where IOP values were revealed, allowing for potential reclassification.

The images were categorized into three groups: Normal (N), Suspect (S), and Glaucoma (G). Additionally, the vertical cup-to-disc ratio (VCDR) was recorded for each optic disc. Table 5 and Table 6 summarize the descriptive analysis of classifications and inter-observer agreement, respectively.

Table 5 Descriptive Analysis of Patient Classifications Across Different Conditions and Comparisons, with Data Provided by Multiple Observers. N= Normal, S= Suspect, G= Glaucoma

Table 6 Comparison of Classification Performance Across Different Conditions and Among Different Specialists. The Bolded Values Reflect the Highest Agreement for Each Specialist

The comparison of classification performance across the different conditions is summarized below:

• SP (Senior Specialist) achieved perfect agreement (Kappa = 1.00) when using contralateral eye information (Condition 2). However, performance declined when relying solely on IOP or without any additional information.
• GB (Resident) demonstrated consistently high performance across all conditions, indicating robust diagnostic capability irrespective of data inputs.
• LO (Specialist) showed significant improvement when contralateral eye information was available (Conditions 2 and 3) but lower agreement when relying only on IOP.
• CM (Specialist) consistently performed well in all conditions, with the highest agreement observed when both contralateral eye information and IOP were included (Condition 3).

Table 6 shows that SP performs perfectly when the other eye’s information is available (Condition 2), but its performance is lower when relying solely on IOP or without any additional information. GB performs consistently well across all conditions, indicating its robustness to various data inputs. LO performance benefits most when the other eye’s information is available (Conditions 2 and 3) but shows lower performance when relying solely on IOP. CM performs strongly in all conditions, with the highest agreement when other eye information is available (Conditions 2 and 3).

In Condition 3, where classifications were based on contralateral eye information but excluded IOP, emerged as the optimal scenario, showing high inter-observer agreement. This condition provides a balance where all raters perform well, suggesting that using the other eye’s information without IOP gives the most reliable classification across the board and makes it the most robust scenario for ground truth classification. In conclusion, including contralateral eye information significantly improved classification reliability across all observers. In contrast, including IOP often introduced variability, leading to lower agreement in several conditions. Condition 3—leveraging contralateral eye data without IOP—proved to be the most reliable for establishing ground truth. The consistently high performance of GB and CM further highlights the potential for enhanced diagnostic accuracy when incorporating bilateral information.

Results from VCDR Scoring

Estimation of the optic disc excavation by all the raters is shown in Table 7.

Table 7 Estimation of the Optic Disc Excavation by All the Raters

This table shows that the more “experimented” ophthalmologists rate the excavations smaller than the “younger” colleagues.

Figure 2 shows the Vertical Cup to Disc scoring (VCDR) for each rater.

Figure 2 Shows the Vertical Cup to Disc scoring (VCDR) for each rater.

Result from AI-MONA

The MONA algorithm analyzes fundus images to assess the likelihood of glaucoma, as shown in Figure 3. While the system does not provide a numeric image-quality score, it automatically rejects poor-quality images where the optic nerve head is not visible or is improperly centred. For acceptable images, MONA generates a probability score between 0 and 1, where higher scores indicate a greater likelihood of glaucoma.

• The algorithm operates with a defined threshold: Scores below 0.73: Classified as healthy and Scores of 0.73 or higher: Indicate a high probability of glaucoma.

Figure 3 The original picture and the processed picture by MONA.

Notably, the algorithm does not classify images as “Suspect”, focusing only on binary outcomes (healthy or glaucomatous). Table 8 provides a descriptive analysis of the risk scores generated by MONA, illustrating the distribution of scores and their correlation with the likelihood of glaucoma (Table 8). Figure 4 shows the Scatter Plot of MONA- AI risk factors for both eyes separately.

Table 8 The Descriptive Analysis by MONA, Based on Risk Scores. The Higher the Risk Scores, the Higher the Possibility of Having Glaucoma

Figure 4 The Scatter Plot of MONA- AI risk factors for both eyes separately.

The Pearson correlation coefficient between the RE and LE is 0.23 indicating a weak positive correlation.

Figure 5 shows the different histograms and kernel density estimates (KDEs) for both eyes to represent the distribution differences.

Figure 5 Histograms and kernel density estimates (KDEs) for both eyes to represent the.

Discussion

Initially designed as an awareness campaign, this study uncovered significant inter-observer variability in clinical glaucoma diagnostics. Despite shared training and adherence to standardized methodologies, the findings highlighted the inherently subjective nature of optic disc evaluation. Notably, each specialist’s diagnostic reasoning, shaped by individual experience and cognitive processes, operated like a distinct “black box”, even within a structured framework. For example, while intraocular pressure (IOP) often introduced complexity, assessing both optic discs simultaneously improved diagnostic accuracy in some cases, further illustrating the variability in clinical approaches.

The small sample size of 70 subjects reflects the practical limitations of data collection during the campaign. However, the results provide valuable insights into diagnostic variability and decision-making processes under real-world conditions. The increase in referrals from 18 to 27 when intraocular pressure (IOP) data was included underscores IOP’s role as an independent glaucoma risk factor, though its inclusion introduced more significant variability and lowered agreement among observers. Notably, Condition 3, where bilateral fundus information was available, but IOP was excluded, showed the highest inter-observer agreement, indicating its potential as a reliable framework for establishing ground truth.

Clinical and AI Implications

The moderate agreement observed in this study highlights limitations in using fundus photographs as the sole basis for training and validating AI models. Despite employing the European Glaucoma Society (EGS) Guidelines, subjective interpretations of these criteria led to inconsistencies, complicating comparisons between AI and human evaluations. These findings align with prior research showing that AI models trained on datasets with comprehensive clinical parameters demonstrate superior generalizability compared to those reliant solely on fundus photo-based labels.

While not a validation focus in this study, the MONA AI system generated probability scores post hoc, revealing a weak positive correlation between scores for both eyes. This contrasts with the higher agreement among clinicians when bilateral images were evaluated together. MONA’s inability to classify “Suspects” or identify non-glaucomatous pathologies, such as visible vasculitis, underscores the limitations of narrow task-specific training. This further emphasizes the need for broader and more diverse clinical datasets to enhance AI robustness and generalizability.

AI employment in clinical decision-making as a support system faces several challenges. Several studies indicate that high image quality is essential for accurately classifying fundus images.^9,10 Standard fundus cameras lack integrated software to assess image quality. In contrast, optical coherent tomography (OCT) and angio-OCT devices provide an image quality score and recommend discarding images that fall below a certain threshold.¹¹ The determination of image quality in fundus photography is often subjective. An observer may choose to classify the optic disc even if the image exhibits mild blurriness. Additionally, many patients participating in screening programs are elderly and may have cataracts or corneal ageing, leading to inherently lower image quality. The data’s quality should represent the expected data level that the model will encounter in practice,¹² meaning elderly with mild to pronounced blurriness of the media. Therefore, the ground truth is fundamentally subjective by default.

Consequently, comparing the performance of AI algorithms to such variability becomes complex. These findings are in keeping with our previous work,^13,14 where the performance of a robust glaucoma-AI model trained on a dataset with ground truth labels based on deep phenotyping including all available clinical parameters had excellent generalizability in various external databases, but with the least good performance in datasets which had ground truth labels bases on only fundus photo evaluation.

In this study, other important clinical parameters, such as visual field, optic nerve head measurements with optical coherence tomography, and corneal thickness, were not performed to confirm the presence of glaucoma at the end. In a similar study by Al Aswad. et al,¹⁵ the deep learning AI (Pegasus) could be compared to a gold standard of glaucoma versus healthy diagnosis because the testing was performed on a known database of patients: fundus images randomly selected from the Singapore Malay Eye Study (SiMES).¹⁶

This study utilized a small cohort of 69 subjects obtained during a glaucoma screening campaign. While small sample sizes can be useful for initial exploratory studies, they pose significant generalizability and statistical power limitations. In glaucoma screening, the disease prevalence may vary across different populations, making it essential to have a representative sample size to draw accurate conclusions. A larger patient sample would improve the robustness and external validity of the results, allowing for better assessments of AI performance and generalizability to real-world clinical settings. In comparison, in a large systematic review of 2021 including 6 original studies comparing the performance of ophthalmologists and AI, the rate of included patients with glaucomatous optic neuropathy ranged from 41.4% to 50%. In our study, the subjects were in hospital for different reasons with a much lower rate of glaucoma, closer to the actual prevalence in the population. This difference in prevalence in our cohort was also a source of lower inter-observer reproducibility.

VCDR scoring, we observed considerable variation in how clinicians rated excavation. In this study, the longer a clinician had been practicing, the smaller their evaluation of the excavation tended to be, highlighting both inter- and possible intra-clinician variability in C/D ratings.^17,18

Conclusion

The limitations of inter-observer variability and small cohort highlighted in this discussion, emphasize the need for cautious interpretation of studies evaluating AI systems for glaucoma screening in clinical practice solely based on fundus photos. These observations should remind AI researchers that the performance of AI algorithms relies on the robustness of the ground truth labels. We should consider a hybrid approach, for example, supplementing “Condition 3” with other clinically relevant information in this study. Future research should address these limitations by incorporating larger and more diverse patient cohorts considering multiple diagnostic parameters: specifically for glaucoma, it is recommended to determine the ground truth based on not only fundus pictures but also additional metadata such as visual field examinations or Optical Coherence Tomography scans.¹⁹ Our findings resonate with experiences in other medical domains, where AI has been shown to complement rather than replace human expertise. Studies demonstrate that while AI and specialists independently achieve similar diagnostic accuracy for recognizing a disease, their combined efforts significantly outperform either alone. This underscores AI’s strength in analyzing vast and diverse datasets, particularly for rare conditions, while clinicians bring a holistic perspective grounded in clinical experience. Similarly, in glaucoma diagnostics, AI has the potential to enhance the accuracy of human assessments by mitigating inter-observer variability and identifying subtle features that may be overlooked in busy clinical settings”. By addressing these challenges, the potential of AI for glaucoma screening in clinical settings should be more effectively evaluated.