Constructing a Resilience Assessment Index System for Tuberculosis Healthcare Services Under Public Health Emergencies: A Modified Delphi Study

Introduction

Resilience, originating in physics, materials science, and engineering, emphasizes the ability of physical materials to return to their original state after disturbances.¹ The concept was introduced to the health system following the 2014 Ebola outbreak.² The COVID-19 pandemic extensively disrupted essential health services in numerous countries and regions, causing substantial collateral damage and loss of life and livelihoods.³ Consequently, health system resilience (HSR) has garnered significant attention in the international community since 2020, particularly with regard to improving HSR and mitigating the damage and disruptions to essential health services caused by the COVID-19 pandemic and similar public health emergencies (PHEs).^4,5 In October 2021, the World Health Organization (WHO) developed a position paper, “Building health systems resilience for UHC and health security during the COVID-19 pandemic and beyond”,⁶ which systematically presents the rationale and seven policy recommendations for building resilient health system based on primary health care. In March 2023, China’s document “Opinions on Further Improving the Medical and Health Service System” clearly put forward the goal of “forming an integrated medical and health service system compatible with the basic realization of socialist modernization by 2035”.⁷ Notably, resilience was first proposed by the Chinese government in designing the health system, highlighting its importance.

However, building a resilient health system faces three challenges: defining a resilient health system, evaluating health system resilience, and improving it.⁸ Although no international consensus exists on the definition of HSR,^9,10 a widely recognized characteristic of resilient health system is the ability to maintain service continuity when facing external shocks.² Therefore, the focus lies on how to improve the resilience of the health system, with comprehending its resilience being the prerequisite.^11,12 However, consensus on how to assess the HSR is absent, and no tailored theoretical framework exists for the evaluating the HSR.⁸ Some studies have suggested that a resilient health system should be a capability rather than an outcome,¹³ and pointed out that positive effects are achieved by sufficient capability and effective strategies.^13,14 Other scholars considered the assessment of health system performance during PHEs as an important aspect in measuring its resilience.¹⁵ In February 2024, WHO released a package of indicators for measuring and monitoring the resilience of a national health system,¹⁶ including 64 core indicators divided into three dimensions: input/structure, process, and output, covering seven domains, such as service delivery, health financing, workforce, for evaluators to choose reasonably. This indicator system measures health system resilience but requires tailored, clear setlist indicators for sub-health fields.

Tuberculosis (TB) is the leading killer among communicable diseases.¹⁷ The respiratory infectious diseases TB and COVID-19, as well as other similar PHEs, usually share health resources.^18,19 Therefore, TB healthcare services (TB-HSs) are among the domains most susceptible to the shock of PHEs.^20,21 Hence, it is significant and pressing to establish resilient TB-HSs. Nevertheless, owing to the scarcity of evaluation tools, it is difficult to assess the resilience of TB-HSs and direct their construction. China ranked third in the number of new patients with TB worldwide,¹⁷ and its TB-HSs have been affected by the COVID-19 pandemic with extensive, long-duration, and multiple waves during 2020–2022.^19,22 Abundant experiences and lessons have been accumulated in carrying out TB-HSs during the COVID-19 pandemic.

In light of the above context, this study aims to construct a resilience assessment index system (RAIS) for TB-HSs in China, addressing a critical gap in the current literature and practice.

Methods

A modified Delphi study was conducted by consensus among experts and developed through a two-round Delphi survey. We used an analytic hierarchy process (AHP) to weigh the indices. The study lasted seven months in 2024. The selection of indices and initial RAIS designing began in April, the Delphi surveys were completed between July and September, and the final RAIS was completed in October. The RAIS constructing flowchart is shown in Figure 1.

Figure 1 The flowchart of constructing a resilience assessment index system for tuberculosis healthcare services. Abbreviations: TB-HSs, tuberculosis healthcare services; RAIS, resilience assessment index system.

Designing the Initial RAIS

We established a research team that included five experts specializing in TB control and responsible for defining resilient TB-HSs and formulating an index pool of the RAIS, which was completed by systematically reviewing the content of TB-HSs and analyzing the relevant literature on the resilience of health system and TB-HSs in the context of PHEs. We then introduced an expert meeting method to design the initial RAIS by referencing the index pool but not limited to this pool. The services of seven TB control experts, four of whom had experience handling the COVID-19 pandemic, were engaged. After thorough discussions at two expert meetings, the initial RAIS was established following Donabedian’s “structure-process-outcome” (SPO) framework.²³

Expert Selection

Delphi experts were purposefully selected based on recommendations from expert meetings. The inclusion criteria required experts to be professionals in tuberculosis (TB)-related fields, with practice settings covering all seven geographic regions of China to ensure national representation. Their affiliations spanned four administrative levels: national, provincial, prefectural, and county. Additionally, no less than 50% of the participants were required to be from cities that implemented lockdowns lasting at least one month during 2020–2022 due to the COVID-19 pandemic. Furthermore, the experts had to meet the following criteria: 1) a minimum of five years of TB-related work experience, 2) a bachelor’s degree or higher, 3) an intermediate or higher technical title or a position at or above the section level, and 4) voluntary participation.

Delphi Process

The Delphi survey was conducted following the guidelines of the Conducting and Reporting Delphi Studies,²⁴ in which experts were requested to fill in and return electronic questionnaires within 14 days of receipt. If they failed to do so within ten days, reminder phone calls were made. The returned questionnaires were summarized, and the indicator rating results and expert comments were analyzed. In the second round, a revised Delphi questionnaire and feedback on the results of the first round were simultaneously sent to experts, who were requested to re-rate the indicators with reference to the feedback.

The experts were asked to rate the four dimensions (necessity, feasibility, stability, and sensitivity) of each indicator using a 9-point Likert scale, with the level of agreement ranging from “1” (strongly disagree) to “9” (strongly agree). The experts were allowed to provide revision comments on the indicators. The experts’ judgment (Ca) and degree of familiarity (Cs) with each indicator were simultaneously rated based on self-evaluation. The basis for making the judgment comes from four aspects: practical experiences, theoretical analysis, counterparts, and intuitive feeling; each aspect was classified into three levels (large, medium, and mild) according to the experts’ degree of influence,²⁵ (Table 1). The degree of familiarity also used the 9-point Likert scale, where “1” referred to “very unfamiliar”, and “9” referred to “very familiar”. In Round 1, experts were required to weigh the necessity, feasibility, stability, and sensitivity of the indicators, with the sum of these weights equal to 100.00%.

Table 1 Expert Judgment Basis and Corresponding Scores

Criteria for Indicator Revision

Based on the literature,²⁶ the following criteria were applied for indicator revision:

1. An indicator with an importance score () of ≥6.5 and a coefficient of variation () of <0.25 was retained.

2. If the importance score () was between 3.5 and 6.5, or if was ≥6.5 but ≥0.25, the indicator was required to enter the next round of Delphi survey.

3. An indicator with importance score () of <3.5 was removed.

The importance score () of an indicator was calculated as the arithmetic mean of the importance scores assigned by all participating experts. For each expert, the importance score () of an indicator was derived by summing the products of the weights and ratings for each of its four dimensions, as follows:

where: =1−4 represents the four dimensions of the indicator (necessity, feasibility, stability, and sensitivity), denotes the rating result of indicator in dimension (assigned by the expert), and represents the weight of the -th dimension, obtained by calculating the arithmetic mean of the expert assessments. Additionally, the is calculated as the standard deviation of all values divided by, reflecting the degree of dispersion among the importance scores assigned by the experts.

Regarding indicators suggested by two or more experts for amendment, addition, or removal, the research team decided through a thorough discussion.

Determining the Weight of Indicators

The differences between the importance scores of indicators within the same classes, when subtracted pairwise, were compared with a set value based on the Saaty 1–9 scale,²⁷ and importance values were assigned accordingly (Table 2). Subsequently, judgment matrices were established, and the weights of each indicator were determined using AHP, provided that the judgment matrices passed the consistency test,²⁸ which assessed the consistency of the judgment matrices. A consistency ratio (CR) of ≤0.10 indicated that the judgment matrix had acceptable consistency.

Table 2 The Saaty1-9 Scale Assignment for Pairwise Comparison of Indicator Importance

Statistical Analysis

Data were analyzed using SPSS software (version 22.0). Categorical data were described as frequencies and percentages, and the ratings of indicators were expressed as means and CVs. Independent Samples t-tests (Student’s t-test or Welch’s t-test, selected based on the results of Levene’s test for equality of variances) were performed to compare the mean importance scores of indicators between male and female experts. Differences were considered statistically significant at P<0.05. The online Scientific Platform Serving for Statistics Professional (www.spsspro.com) was used to calculate the weights of the indicators.

The following three parameters were calculated to evaluate the reliability of the Delphi survey results: 1) the expert positive coefficient, 2) the expert authority coefficient (Cr), and 3) expert coordination. The expert positive coefficient was reflected by the questionnaire return rate before the deadline, with a rate of >70% indicating high expert motivation.²⁹ The formula for Cr calculation is Cr= (Ca + Cs)/2. The Ca score was computed based on judgment and degree (Table 1). The Cs value was derived by dividing the 9-point familiarity rating result by 9; Cr of >0.7 indicated high expert authority.³⁰ Expert coordination was judged by Kendall’s coefficient of concordance (W). W ranges from 0 to 1, with higher values denoting greater coordination.³¹ The significance of W was determined using a chi-squared test.

Results

Characteristics of Delphi Experts

Among the 26 experts consulted, 12 were males (46.2%), mostly within the 40–49 age group. Twenty-one experts (81.8%) held doctoral or master’s degrees, and 20 experts (76.9%) held senior technical titles. Furthermore, 23 experts (88.5%) played administrative roles. Thirteen (53.8%) experts had provincial affiliations. All experts had over five years of professional experience in the TB field. Half of the participants were engaged in TB control (Table 3). Additionally, 23 participants (88.5%) were from cities that experienced COVID-19 lockdowns lasting over one month.

Table 3 Characteristics of Experts Took Part in Delphi Survey

Reliability of the Delphi Results

Twenty-six questionnaires were distributed in both rounds of the Delphi survey, and the number (rate) of returned questionnaires was 26 (100.0%) and 25 (96.2%), respectively. This demonstrates a strong willingness to participate in the study. In Round 1, Cr was 0.908 (Ca = 0.886, Cs = 0.930), and in Round 2, Cr was 0.906 (Ca =0.879, Cs = 0.933), indicating high expert authority. In the first round, W was 0.130 (χ²=151.538, P<0.001), and in the second round, it was 0.315 (χ²=393.815, P<0.001), revealing good coordination among expert opinions in both rounds, with the second round showing even better coordination.

Establishment of the RAIS

Based on a literature review, we defined resilient TB-HSs in China as the continuous provision and utilization of TB case finding, diagnosis, treatment, and management services during PHEs. Accordingly, a pool of index systems comprising of 55 items was formulated (Table S1). Subsequently, the initial RAIS, including three first-grade indicators, nine second-grade indicators, and 34 third-grade indicators, was developed using the expert meeting method (Table S2). The weights for the four indicator dimensions were as follows: necessity (30.22%), feasibility (28.73%), stability (18.82%), and sensitivity (22.23%). Based on this, importance scores for the indicators were calculated using the following formula: necessity × 0.3022 + feasibility × 0.2873 + stability × 0.1882 + sensitivity × 0.2223. During the first round of the Delphi process, importance scores for all indicators ranged from 6.88 to 8.51, with CVs falling between 0.07 and 0.23. Statistical analysis revealed no significant differences in the mean importance scores of all indicators between male and female experts (P>0.05) (Table S3). Nineteen experts provided 59 suggestions for indicator revision. After synthesizing their opinions, the research team removed one third-grade indicator, amended six third-grade indicators, and added six third-grade indicators according to predefined criteria. During the second round of the Delphi survey, the mean importance scores for all indicators ranged from 6.92 to 8.49, with CVs falling between 0.04 and 0.13, and no statistically significant differences were observed in the mean importance scores of all indicators between male and female experts (P>0.05), with the exception of one tertiary indicator, which exhibited a borderline difference (P=0.049) (Table S4). Only one expert recommended deleting one indicator and modifying another. Ultimately, a consensus was reached, leading to the establishment of an RAIS for TB-HSs comprising three first-grade, nine second-grade, and 39 third-grade indicators (Table 4).

Table 4 Final Resilience Assessment Index System for Tuberculosis Healthcare Services in China

Weight Determination of Indicators

The importance scores for each indicator were calculated based on the results of the second round of the Delphi survey. Using the Saaty 1–9 scale method and AHP, the weights of all indicators included in the final RAIS were determined. Notably, all indicators had weights greater than 0.5% except T2, T4, T5, and T6. (Figure 2). During the calculation process, all the judgment matrices passed the consistency test (CRs<0.1).

Figure 2 Diagram of weight distribution indicators in the resilience assessment index system for tuberculosis healthcare services. The four indicators with weights below 0.5%, namely T2 (0.46%), T4 (0.29%), T5 (0.20%), and T6 (0.34%), were not displayed.

Discussion

Accelerated climate change, expanding human activities, and cross-species pathogen transmission fuel the persistent and intensifying threat of emerging/re-emerging infectious diseases.³² The emergence of “X diseases” such as COVID-19 is inevitable in the future.³³ In China, a considerable proportion of TB institutions also treat emerging infectious diseases, which will strain TB healthcare resources during outbreaks.¹⁹ Thus, assessing and bolstering TB-HS resilience is imperative.

The current study, rooted in the TB prevention and control landscape of China, developed a comprehensive index system to assess the resilience of TB-HSs. This index system comprises three primary, nine secondary, and 39 tertiary indicators, which are rigorously constructed to provide practical insights into enhancing the resilience of TB-HSs. RAIS is based on robust methodologies. The Delphi method facilitates consensus across domains,³⁴ while the AHP provides a logical and reliable means of quantifying indicator weights through mathematical processing.³⁵ The indicators are structured around the classic SPO framework, which logically posits that “Service guarantee” underpins “Service provision and utilization”, which subsequently leads to “Service outcomes and effects”.³⁶ Our panel of experts represented all seven major regions of China, ensuring broad geographical representation. With over five years of experience in TB-related work and recent experience during the COVID-19 pandemic, they possessed a deeper understanding of TB-HSs’ resilience. Finally, we employed an innovative approach to measure the indicators’ importance by considering four dimensions: necessity, feasibility, stability, and sensitivity. Expert judgment determined the weights assigned to these dimensions, ensuring a scientific basis for assessing the indicator weights.

In the final RAIS, F2, “Services provision and utilization” had the highest weight (49.05%) among the three primary indicators. This differs from some Chinese medical quality assessment frameworks based on the Donabedian model,^37,38 potentially because of the clearer definition and easier accessibility of “Structure” and “Outcome” indicators in such systems. Another possible explanation is the emphasis on maintaining service processes during PHEs, a key attribute of HSR: maintaining service continuity during crises.² Regarding structural aspects, while some developed nations have ample, high-quality healthcare facilities under normal circumstances, they demonstrated low resilience during the COVID-19 pandemic,¹² highlighting the importance of rapid response and efficient organization and coordination during PHEs.

Among the nine secondary indicators, S8 accounted for 20.79%, highlighting the significance of service outcomes in assessing the resilience of TB-HSs. These outcomes reflect short-term service provision and utilization and directly gauge resilience levels. This underscores the need to focus on both TB-HSs’ abilities and processes as well as results-oriented resilience building. Additionally, S5 and S4 shared 20.67% and 14.68% of the weight, respectively, emphasizing the maintenance of patient diagnosis and treatment services as core tasks and resilience manifestations during PHEs.

Among the 39 tertiary indicators, T34, the proportion of patients with TB who discontinued treatment stood alone with a weight exceeding 10%. Typically, such treatment interruptions are linked to various factors.³⁹ However, during PHEs, they are closely tied to the operational status of TB-designated facilities and anti-TB drug supply. Similarly, T22 weighed 8.90%, highlighting the importance of ensuring that diagnosed patients receive treatment during crises. Access to treatment hinges on the availability of services and drugs. During the COVID-19 pandemic, the facilities and human resources responsible for TB-HSs were redirected, potentially leaving patients diagnosed at non-designated facilities without treatment. These two indicators also correlate with upstream preparedness, downstream treatment outcomes, and patient satisfaction and can sensitively reflect the resilience of TB-HSs. For future crises, strengthening patient treatment safeguards is crucial to enhance TB-HSs’ resilience.

Unlike other evaluation index systems used under normal conditions,⁴⁰ the RAIS comprehensively captures the dynamic capabilities of the resilient TB-HSs. For instance, indicator T13, which quantifies the ratio of TB doctors on duty during and before PHEs, exemplifies the dynamic resilience of TB-HSs by illustrating how TB doctors are deployed during such crises. A higher proportion of TB doctors on duty during emergency situation suggests that the region can effectively mobilize and maintain TB diagnosis and treatment services. The RAIS, which evaluates specific time points within the same region, facilitates an understanding of the evolution of TB-HSs over time. Furthermore, the RAIS encompasses various domains, allowing for the identification of vulnerabilities when applied regionally and enabling targeted enhancements and improvements.

In contrast to the traditional Delphi method, which relies on open-ended questionnaires to gather research questions in the first round,⁴¹ the initial RAIS for Round 1 was designed based on a literature review and expert meetings. This alternative approach was necessitated by the nascent application of the resilience theory to healthcare services, particularly in the realm of TB prevention and control. Given that experts have limited familiarity with the resilience theory, it would have been difficult to achieve consensus among freely generated opinions. Furthermore, the experts involved in the research team and panel discussions had extensive experience in TB control and frontline management during the COVID-19 pandemic, ensuring that the initial RAIS was precisely targeted and problem-oriented.

However, this study has several limitations. First, the predetermined approach, deviating from the classic Delphi method, may introduce bias in the selection of indicators. Second, as all Delphi experts were TB healthcare service providers, the perspectives of TB service utilizers, particularly regarding the resilience of TB healthcare services during the COVID-19 pandemic, were not captured, which could affect the generalizability of the findings. Finally, the application of resilience theory in healthcare services, especially in TB-HSs, is still in its early stages, with limited research literature available for reference. Despite these limitations, this study provides a foundational framework for future research to validate and refine the index system.

Conclusion

A novel hierarchical indexing system was established to assess the resilience of TB-HSs in China during PHEs, using a modified Delphi process with AHP. We expect this tool to aid in the construction of resilient TB-HSs. However, further refinement and enhancement are required for practical applications.