Test-Retest Reliability of Epworth Sleepiness Scale Score in Patients with Untreated Obstructive Sleep Apnea

Introduction

Obstructive sleep apnea (OSA) is a common, complex, and heterogeneous disorder caused by repeated complete or partial collapse of the upper airway during sleep, resulting in intermittent hypoxia, sleep fragmentation, and autonomic fluctuations during sleep.¹ Globally, approximately 900 million adults aged 30–69 years are affected by mild, moderate, or severe OSA, with China alone reporting a population of 176 million.² Excessive daytime sleepiness (EDS), a common symptom of OSA, negatively affects public safety, social interactions, work, mood, cognitive functioning, and quality of life. Recent findings show that OSA with EDS is a more severe subtype of the disorder, resulting in more adverse events and a worse prognosis.^3–5

Epworth Sleepiness Scale (ESS), a subjective questionnaire, is the most commonly used tool for assessing EDS in clinical practice. Patients are required to answer eight questions, each of which can be given any of four grades (0 for never dozing, 1 for possible mild dozing, 2 for probable dozing, and 3 for very likely dozing or even falling asleep), with total scores ranging from 0 to 24. Generally, ESS scores ≥ 11 are considered to reflect subjective EDS.⁶ It has been widely used in OSA screening, assessment, diagnosis, and treatment.^7–9 The degree of change in ESS score is also frequently used as an important endpoint in determining treatment effects in clinical studies.^7,10,11 For example, a 2- or 3-point decrease in ESS from baseline after continuous positive airway pressure (CPAP) treatment for OSA is considered the minimal clinically important difference (MCID).^10,12

The ESS is appreciated for its straightforward nature and ease of use in gauging EDS. Nevertheless, concerns have been raised regarding the dependability and consistency of ESS results across multiple evaluations, notably within clinical cohorts where reliability may deviate from findings in the initial validation.^13–15 Several studies faced constraints due to restricted sample sizes,^13,16,17 while others veered off course from examining ESS stability as they were not designed for that purpose.^18,19 Furthermore, certain investigations faced limitations stemming from statistical methodologies.^16,17,20,21 Although the ESS is designed to reflect long-term average sleep propensity rather than immediate sleepiness, it is recommended that the interval between two measurements should not exceed four weeks to avoid capturing real changes in average sleep propensity.²² However, only two small-sample studies set the interval between repeated measurements within four weeks.^13,14 Therefore, evaluating the ESS over short intervals could provide a clearer assessment of its reliability and consistency. Moreover, there is a glaring gap in research concerning the test-retest reliability of ESS in individuals with untreated OSA.²³ Given concerns about inconsistent ESS scores in these patients, it is important to consider whether repeated measurements, followed by averaging, similar to blood pressure monitoring, might be implemented. Additionally, it would be useful to investigate whether convenient methods, such as telephone follow-ups, can achieve comparable effectiveness to in-person assessments. Notably, no viable resolutions have been put forth to address this issue.

The main objective of this study was to assess how reliably the ESS can measure daytime sleepiness over short intervals (less than one week) among patients with untreated OSA. Additionally, this study aimed to investigate the factors that affect this consistency. To address potential inconsistencies in ESS measurements, we examined the stability of ESS using different follow-up approaches (eg, “face-to-face” or “telephone”), providing researchers with an easy-to-use and effective means to enhance the reliability of the ESS assessments.

Methods

Participants

This sub-investigation was affiliated with an ongoing prospective study of the pathogenesis and intervention techniques for sleep-breathing disorders approved by the Institutional Review Board of West China Hospital, Sichuan University (No.2022797) and performed in accordance with the Declaration of Helsinki. All participants provided informed consent. This study was performed at the Sleep Center of West China Hospital of Sichuan University.

The research team consecutively screened and enrolled patients who visited the Sleep Center of West China Hospital of Sichuan University between October 2022 and May 2023 and underwent all-night polysomnography (PSG). The inclusion criteria were: (1) Chinese individuals aged 18–75 years. (2) Diagnosis of OSA (ICD-10-CM code: G47.33) fulfilling either of the following two points:²⁴ a. Apnea-hypopnea index (AHI) ≥ 5/h and accompanied by symptoms of sleepiness, snoring, and apnea, etc. b. AHI ≥ 15/h. The exclusion criteria were: (1) central sleep apnea (CSA; ICD-10-CM code: R06.3; CSA was defined as > 5 episodes of central apnea + central hypoventilation per hour, or the ratio of central apneas + hypoventilation to all apneas + hypoventilation > 50%, or Cheyne–Stokes breathing); (2) disorders that cause chronic sleep disruption (eg, insomnia, pain); (3) current concomitant mental illness (eg, depression) that could affect the accuracy of the questionnaire; (4) comorbidity with other sleep disorders such as narcolepsy, restless legs syndrome, obesity hypoventilation syndrome; (5) combination of uncontrolled acute heart failure, coronary heart disease, chronic obstructive pulmonary disease, and other serious underlying diseases; (6) previous diagnosis of OSA or history of CPAP, or any intervention (such as mandibular orthodontic appliances, hypoglossal nerve stimulation, or sleep position change) during the interval between the two follow-up periods; and (7) inability to independently complete the ESS questionnaire.

Measurements

All patients completed a one-night PSG using the Philips Respironics Alice 6 polysomnographic recorder and Sleepware G3 data analysis software, and all sleep parameters recorded on the PSG were analyzed by senior technicians who were blinded to the design of the study, according to the international standards of American Academy of Sleep Medicine.²⁴ The following information about the patients were also collected before sleep monitoring: sex, age, body mass index (BMI), smoking (yes/no), alcohol consumption (yes/no), underlying diseases such as hypertension (yes/no) and diabetes mellitus (yes/no), and a first assessment of ESS. We used the validated Chinese version of the ESS.²⁵ Patients who met the inclusion criteria were categorized into face-to-face or telephone follow-up groups based on random sampling. The initial ESS assessment occurred in the evening at the Sleep Medicine Center prior to PSG. Subsequently, the ESS score was reassessed via face-to-face or telephone follow-up by experienced sleep medicine physicians (X Feng, Y Shi) within one week (1–6 days) after patients underwent their initial comprehensive sleep evaluation. None of the participants were informed of the purpose of the study.

Sample Size Calculation

A total of 355 participants were required to complete the study protocol, for a power of 90%, α = 0.05, and an estimated mean difference in change scores of 0.5 on the ESS, with a projected standard deviation (SD) of 2.9 points. This estimate is based on conservative projections derived from our previous pre-experiment.

Statistical Analysis

Continuous variables are expressed as mean ± SD and categorical variables as numbers (percentages). Comparisons of variables between groups were made using the independent samples t-test for continuous variables that conformed to or approximated a normal distribution, the Mann–Whitney U-test for those that did not conform to a normal distribution, and the χ² test for comparisons between categorical variables.

Intraclass correlation coefficients (ICCs) were used to assess the correlation between the first and second ESS scores.²⁶ The consistency between the two ESS scores was assessed using Bland-Altman analysis, where the mean difference and the 95% limit of consistency (mean difference ± 1.96 × SD) were calculated as estimates of reproducibility²⁷ and visualized.²⁸ Univariate linear regression was used to assess the correlation among the baseline ESS score, sex, age, BMI, smoking, alcohol consumption, hypertension, diabetes mellitus, AHI, nocturnal minimum oxygen saturation, and ESS score difference. In addition, the proportion of participants with ESS score changes ≥ 2, 3, 5, and 7 points between the test and retest was calculated.

A paired t-test was performed to assess intragroup differences in ESS comparing follow-up scores to baseline scores. Intergroup differences in ESS changes across follow-ups were analyzed using linear mixed-effects models. The ESS difference served as the dependent variable, with fixed effects for the follow-up group, visit (baseline or within one week), and their interaction (group × visit), which was interpreted as the follow-up effect.

We used SPSS 26.0 for data analysis and GraphPad Prism 9.0 for plotting. P < 0.05 was considered to indicate statistical significance.

Results

Participants

Overall, 587 people visited the sleep center for PSG between October 2022 and May 2023, and 401 met the inclusion criteria (Figure 1). Eventually, 382 (mean age, 43.52 ± 12.41, 80.6% men) completed the study, and the mean test-retest interval was 3.60 ± 1.56 days. The number of patients who were followed up face-to-face was 191 (Group A), and those who were followed up by telephone was 191 (Group B). The baseline characteristics and ESS scores of the two groups are shown in Table 1.

Table 1 Demographic, Clinical, and Sleep Characteristics of the Study participants

Figure 1 Patient flowchart.

Groups A and B did not differ significantly in age, sex, BMI, AHI, lowest SaO₂, or test-retest intervals. Additionally, there were no differences in smoking, alcohol consumption, or the number of patients with comorbid hypertension or diabetes between the two groups. There were no significant differences in baseline, follow-up, mean, or the difference in ESS scores.

Test-Retest Reliability in ESS Scores

The difference in the ESS scores before and after the test was slightly higher in the face-to-face follow-up group (1.70 ± 2.70) compared to the telephone follow-up group (1.66 ± 3.17), but this difference was not significant (P = 0.967) (Table S1). Across all patients, the percentages with ESS score absolute differences of ≥ 2, 3, 5, and 7 were 66.5%, 49.2%, 17.5%, and 3.9%, respectively. The face-to-face group had a slightly lower percentage of individuals with an ESS score absolute difference of ≥ 3 points (42.3% vs 55.5%, P = 0.014).

Figure 2 illustrates the mean ICC values with 95% confidence intervals (CI), both overall and between the groups. The overall ICC for all patients was 0.820 (95% CI: 0.653–0.894). There was no significant difference between the face-to-face and telephone groups. Subgroup analyses were conducted to compare ICC values. Patients with severe OSA (AHI≥30) had a lower ICC (ICC=0.784) than those without it (ICC=0.889). Older patients (aged ≥60 years) had a higher ICC than younger patients, though the lower limit of the CI (0.593) was reduced. The higher ICC for older than younger patients was minimal. Patients with obesity (BMI >28) had a relative lower ICC (ICC=0.765), while the female patients demonstrated excellent stability with an ICC of 0.900 (95% CI: 0.832–0.939). However, intragroup comparisons within each subgroup revealed no statistically significant differences, as the 95% CIs overlapped.

Figure 2 ICCs with the corresponding 95% confidence intervals of the whole cohort and each subgroup.

A Bland-Altman analysis was performed on all untreated OSA patients (Figure 3a). The y-axis represents the difference between second ESS minus first ESS. The variation in ESS was 1.68 (−3.35 to 8.25). The face-to-face group (1.70 [−3.58 to 6.99]) showed similar results to the telephone group (1.66 [−4.51 to 7.84]) (Figure 3b and c). Subgroup analysis was performed based on OSA severity (Figure S1), BMI (Figure S2), age (Figure S3), and sex (Figure S4). Patients with severe OSA and obesity showed a variation of 1.77 (−4.13 to 7.67) and 2.39 (−4.39 to 9.18), respectively. The variation observed in female OSA patients was 0.82 (−3.69 to 5.34).

Figure 3 Bland-Altman plots for (a) whole cohort, (b) face-to-face group, and (c) the telephone group. The difference (second minus first) between 2 consecutive ESS scores is plotted against their mean for analyzing levels of accordance between 2 measurements. The Orange dots represent individual test-retest mean ESS scores, while the dashed lines indicate the mean difference and the 95% limit of consistency.

Influence of Baseline Variables on the Variation of ESS

Table 2 illustrates the relationship between various baseline characteristics and the signed difference in sequential ESS scores (second minus first), as analyzed through univariate linear regression. Factors that influenced the stability of repeated ESS measurements included BMI, sex, AHI, and baseline ESS.

Table 2 Univariate Linear Regression of Influence of Baseline Parameters on ESS Differences

Discussion

Our study assessed the short-term reliability of the ESS in individuals with untreated OSA. We meticulously evaluated the test-retest reliability of the ESS in 382 patients diagnosed with OSA at the Sleep Medicine Center of West China Hospital. Statistically, the ESS, a common measure of EDS, demonstrated an ICC of 0.820, which is greater than the cut-off value of 0.8, indicating substantial reliability.²⁹ The reliability was consistent for both face-to-face (ICC = 0.829) and telephone follow-ups (ICC = 0.814), suggesting that the administration method does not significantly impact this stability. Given its consistent performance across follow-up methods, overall, the ESS proves to be a reliable tool for assessing daytime sleepiness in untreated OSA patients. This reliability increases its value in routine clinical practice, helping monitor the severity of sleepiness and guiding treatment decisions in OSA management. However, from the perspective of clinical practice, the mean difference between repeated ESS measurements was 1.68 ± 2.93. Although relatively small, this difference is significant when considering the MCID for the ESS.^10,12 The study also found that patients with severe OSA and those with obesity had moderate stability in ESS scores, with ICC of 0.784 and 0.765, respectively. This reduced reliability in specific subgroups is crucial, as it underscores potential variability in EDS assessments among these patients. Further analysis revealed that BMI, male sex, and the AHI were positively correlated with variations in ESS scores. Conversely, baseline ESS scores showed a negative correlation with these variations. These findings indicate that specific demographic and clinical factors can affect the reliability of ESS scores, requiring more nuanced interpretation in clinical practice.

Johns,^6,15 the developer of the ESS, identified strong reliability coefficients in healthy populations, confirming the scale’s robustness. The reliability of the ESS found in this study is consistent with previous research. For example, studies by Grewe et al¹³ and Veqar et al³⁰ reported ICCs ranging from 0.73 to 0.86, indicating moderate reliability of the ESS in clinical settings. The short interval might contribute to higher ICCs observed, as longer periods typically introduce more variability in measurements. However, our study introduces a new perspective by highlighting the variability among patients with untreated OSA. While this study confirmed the statistically good stability of the ESS, the mean difference between the two measurements was approximately 1.7 points. Notably, among patients with obesity, this mean difference increased to 2.39 points. Such a high value is not a good sign for the reliability despite the other methods, particularly in the context of OSA, where a large proportion of patients are obese. To our knowledge, a change of 2 points on the ESS is clinically significant, as numerous studies on OSA treatment consider a 2–3-point change to be an indicator of significant effectiveness.^10,12 Additionally, a recent meta-analysis revealed that the placebo effect of sham-CPAP on the ESS was equivalent to a change of −1.55 points.³¹

Previous studies, constrained by limited sample sizes and study designs,^13,14,32 have not adequately addressed this issue from both a statistical and clinical practice perspective. This gap underscores the significance of our findings. In our subgroup analysis, we discovered that, particularly for patients with severe OSA and obesity, a single ESS measurement may be unreliable. While the exact mechanism remains unclear, we speculated that this reduced reliability might be due to heightened symptom variability and psychological factors affecting the perception of sleepiness, and to the subjective nature of the ESS. For instance, individuals with obesity might encounter more significant fluctuations in daily activities and mood,³³ which can impact their ESS scores. Additionally, more severe OSA exacerbates sleep fragmentation, leading to fluctuations in sleep quality and daytime performance. Simultaneously, the psychological impact of severe OSA, such as anxiety and depression, influences how patients perceive and communicate their symptoms.

Our findings may have meaningful clinical implications. While the ESS is generally reliable, the identified variability in patients with severe OSA and obesity has crucial implications for clinical practice. Clinicians should exercise caution when interpreting ESS scores in these groups, as the scores may fluctuate more than in other patients. This variability also highlights the importance of considering individual patient factors when using the ESS to assess sleepiness and developing tailored assessment strategies for these specific populations.

Averaging the ESS scores twice in a week may provide more reliable results in this population. This approach is akin to the method used in clinical practice for measuring blood pressure, where the average of two or three readings is typically taken.^34,35 Our study’s randomization into face-to-face and telephone follow-ups offers unique insights into how the mode of administration affects reliability, an aspect not extensively covered in previous research. The similar ICC and mean difference of ESS scores for both follow-up methods indicate that ESS can be reliably administered through different modes, enhancing its usefulness in various clinical and research settings. Developing modified versions of the ESS for patients with OSA could further improve reliability. This could include additional questions addressing mood or a broader range of daily activities, providing a more comprehensive understanding of these patients’ experiences. Training clinicians on the proper administration and interpretation of the ESS, with a focus on individual patient factors, could further enhance the scale’s utility.

This study has several limitations. First, the study involved 382 patients from a single sleep center, which may not be representative of all patients with OSA, especially those from different geographic, cultural, or socioeconomic backgrounds. This might limit the generalizability of the findings. Future studies with larger, more diverse populations are needed to confirm these findings. Second, this study repeated ESS measurements within a one-week interval. However, until now, there is no consensus on the optimal follow-up interval. Third, measurement errors could arise from patients’ recall bias or misunderstanding of the ESS questions. Despite efforts to standardize the administration of the ESS, individual differences in comprehension and recall could influence the reported scores, thereby affecting the observed reliability. Fourth, although ICC may be more appropriate than Pearson’s correlation test in assessing the stability of repeated measures of subjective scales,³⁶ we must acknowledge the limitations of this statistical method itself.²⁶ ICC assumes that the variance components (between-subject and within-subject) are homogeneous across all subjects. If the variability differs significantly between subjects, the ICC value might be misleading, overestimating or underestimating the true reliability.³⁷ Moreover, there is no universally accepted threshold for interpreting ICC values. While some studies consider an ICC greater than 0.75 to indicate good reliability, others set the threshold at 0.8 or even 0.9, highlighting the lack of consensus in this regard.^38,39 Fifth, the absence of a control group of healthy individuals without OSA is an important limitation. This omission prevented a comparison of ESS reliability between OSA patients and healthy individuals, which could provide insight into whether ESS reliability is primarily influenced by OSA or other factors, such as individual response biases. Sixth, during one week, the reliability can vary due to a concomitant event that may considerably change the ESS both a baseline and for the second measurement. Furthermore, although a one-week interval was selected to minimize external influences and ensure comparability, this relatively short timeframe may not capture long-term variations in sleepiness. Extended intervals could better reflect fluctuations due to lifestyle changes, adaptation to OSA symptoms, or other dynamic factors, thus providing a more comprehensive evaluation of ESS reliability. Future research should address these limitations by incorporating a healthy control group, evaluating reliability over extended intervals, and investigating the impact of specific confounders such as comorbidities (eg, depression, anxiety), lifestyle factors, medication use, and variations in OSA severity.

Conclusions

In conclusion, this study revealed that the ESS is a moderately reliable tool to assess sleepiness in patients with untreated OSA. Reassessing the ESS score within one week—either through face-to-face or telephone follow-up—and averaging the results may offer a more precise evaluation of sleepiness, particularly in patients with severe OSA and obesity. Future research should aim to validate these findings across broader populations and develop strategies to enhance the reliability of sleepiness assessments in diverse OSA groups.