Developing and Validating a Novel Generic Patient-Reported Outcome Measure – Postoperative Recovery Scale for Adult (PRSA): A Prospective Observational Study

Introduction

The global annual surgical volume is estimated to exceed 313 million. Despite the mortality rate of 4.2 million patients within 30 days after surgery, over 300 million patients will experience the early and long-term postoperative recovery.^1,2 Although surgery is effective in treating surgical diseases, the process of anesthesia and surgical procedure can lead to complications and a significant decline in the patient’s health status for a certain period.³ Therefore, the quality and speed of postoperative recovery continues to be significant concerns for patients, families, and clinicians.

Evaluating postoperative recovery relied on objective indicators in early clinical practice and studies, such as the incidence of complications, mortality rates, and length of hospital stay. However, improving these objective indicators fails to comprehensively capture the essence of postoperative recovery or accurately depict its trajectory from the patients’ perspective.⁴ Patient-reported outcomes (PROs) refer to the direct utilization of patient-derived information to depict health status, thereby circumventing intermediaries and minimizing the risk of crucial data loss.⁵ In recent years, numerous scholars have advocated the incorporation of PROs into clinical research to establish a theoretical foundation for patient-centered medical decision-making and enhance medical quality.^6,7

According to a review, fourteen PRO scales are currently available for estimating postoperative recovery in adult patients, including the 15-item quality of recovery scale (QoR-15), surgical recovery index, postoperative quality of recovery scale and so on.⁸ However, these scales have several limitations as follows: Firstly, some scales only evaluate patients’ postoperative recovery during hospitalization or within the first week after surgery. This may result in the items of these scales not being available for the evaluation of preoperative baseline status and long-term recovery. Secondly, some scales lack comprehensive content and often ignore social function as an important domain. Thirdly, terminology and statistical methods employed for scale validation exhibit inconsistency. Finally, all scales were developed based on native English speakers. Therefore, some definitions of items may not align with the Chinese patients’ perception of postoperative recovery rooted in their cultural background.

This study aimed to develop and validate a comprehensive postoperative recovery scale based on patient-reported outcomes that could be employed to evaluate early and long-term postoperative recovery in adult patients.

Patients and Methods

Study Design

This study included adult patients after surgery in the development phase between October 2021 and December 2021 and adult patients who planned to undergo elective abdominal surgery in the validation phase between June 2022 and August 2022. This study was registered in the Chinese Clinical Trial Registry (www.chictr.org.cn) with the registration number of ChiCTR2100051503. This study adhered to the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) consensus to ensure comprehensive and standard reporting.⁹ Figure 1 showed the study process.

Figure 1 Development and validation process of Postoperative Recovery Scale for Adult (PRSA).

Development of the Scale

Building a Conceptual Framework

The research team developed the conceptual framework of the scale through group discussions on previous literature reports, given the absence of a universally accepted definition of postoperative recovery. Accordingly, we identified the focal point of the systematic literature review and devised an interview questionnaire that aligned with this conceptual framework.

Establishing Item Pool

PROs related to postoperative recovery from prior clinical studies were gathered by searching literature from the following databases: PubMed, Embase, Cochrane Library (via the Ovid), China National Knowledge Infrastructure (CNKI) and Wanfang database. Additionally, face-to-face interviews were conducted among postoperative adult patients across all surgical departments of West China Hospital. Each patient independently completed a semi-open questionnaire that captured their actual experiences after surgery and understanding of postoperative recovery. Based on the International Classification of Functioning, Disability, and Health, the results of the systematic literature review and patient interviews were converted into items to join the item pool. And the reporting frequency of each item was calculated.

Formulating the Final Version

Based on item reporting frequency and topic relevance, the research team developed an initial draft of the scale and item definitions via group discussions. To improve the content of the scale, a Delphi consultation was held.¹⁰ An expert panel of doctors and nurses with extensive clinical experience in perioperative management was invited from various surgery, anesthesiology, and rehabilitation departments at West China Hospital. The importance of each item was independently assessed by experts using a Likert scale ranging from 0 to 9 points, with a score of ≥7 points indicating a high level of importance. The experts could propose revisions to each item or add new items via email. The support rate was defined as the proportion of experts considering an item highly significant. If the support rate was less than 70%, the item needed to be modified or deleted. Experts reached a consensus regarding the significance of individual items when the coefficient of variation (CV) was less than 0.25. Kendall’s coordination coefficient was computed to assess the consistency of the importance scores across all experts, with a P-value less than 0.05, indicating agreement among all experts. The termination criteria for the Delphi method involved satisfying the conditions of the CV and Kendall’s coefficient, as mentioned above. The final version of the PRSA was derived by consolidating the outcomes from the final round of consultation.

Validation of the Scale

The inclusion criteria were as follows: (1) patients aged over 18 years who planned to undergo elective abdominal surgeries under general anesthesia, local anesthesia, or regional nerve block; (2) patients who provided signed informed consent. Patients who could not communicate, understand the scale for various reasons, or complete the scale were excluded.

PRSA scores were collected on preoperative day 1 and postoperative days 1, 3, 5, 7, 30, and 60. Similarly, QoR-15 scores were collected on preoperative day 1 and postoperative days 1, 7, and 30. Repeated measurements of PRSA were performed on postoperative days 3 and 30, with an interval of 6 hours. The time taken to complete both scales and the ease of completion were recorded. The patients evaluated the complexity of completing the scales using a five-grade Likert scale (very difficult, difficult, moderate, easy, and very easy). The postoperative morbidity survey (POMS) was used to determine the incidence of complications within 30 days following surgery.¹¹

The measurement properties of the PRSA were validated based on the following five aspects, as outlined in the COSMIN consensus:⁹

Validity

Content validity was assessed based on the development process. Qualitatively evaluating whether the content of PRSA scale effectively captured the intended concepts. Criterion validity was assessed by the strength of correlation between PRSA and QoR-15 scores. Furthermore, a hypothesis posited that patients without postoperative complications would demonstrate higher PRSA scores than those with complications on postoperative day 30.

Reliability

Internal consistency was assessed by Cronbach’s α value, which greater than 0.7 indicated good internal consistency. The intraclass correlation coefficient (ICC) was calculated by repeated measurements of PRSA, and ICC greater than 0.7 indicated good test–retest reliability. The ceiling or floor effect was present when the proportion of patients with the highest or lowest scores exceeded 15%. Measurement error can be obtained by calculating the smallest detectable change (SDC) according to the formula provided by previous study.¹²

Reactivity

The responsiveness of the scale was judged by comparing the PRSA scores at different time points, indicated by the observed changes in scale scores over time. The effect size was measured by calculating Cohen’s value for PRSA and QoR-15. A Cohen’s value ≥0.8 indicated a substantial response.

Interpretability

The QoR-15 score less than 90 indicated poor, and more than 121 indicated good postoperative recovery.¹³ Receiver operating characteristic (ROC) curve was used to identify the optimal threshold of PRSA score. The area under curve (AUC) represented the discriminative ability of the threshold of PRSA score, and the AUC over 0.7 suggested strong discrimination. The minimal clinically important difference (MCID) of the QoR-15 was 6.0.¹⁴ The ROC curve was also constructed using changes in PRSA scores from postoperative day 1 to 7, and the point on this curve with the maximum sensitivity and specificity denoted the MCID value of the PRSA.

Feasibility

The feasibility of PRSA was evaluated by comparing the time and complexity associated with completing the PRSA and QoR-15 scales based on data collected on days 1 and 30 following surgery.

Statistics Analysis

Previous research suggests that the recommended sample size should be ten times the number of items in the scale.¹⁵ The QoR-15 has 15 items, and the PRSA has 10 items. An appropriate sample size is 180, accounting for a loss to follow-up rate of 20%.

Continuous data were presented as means and standard deviations if normally distributed, or as medians and quartiles if not. Categorical data were presented as frequencies and percentages. Spearman correlation coefficient was used to assess the correlation between the PRSA and QoR-15 scores. The coefficient above 0.7 indicated a great criterion validity. The Mann–Whitney U-test was used to compare the PRSA scores of patients with or without complications. The Friedman test was used to compare the PRSA scores at different time points, and significance levels were adjusted using the Bonferroni method. The completion complexity of the PRSA and QoR-15 was compared by the Wilcoxon test. The Hodges–Lehmann method was performed to estimate the 95% confidence interval (CI) for the difference in completion time between the PRSA and QoR-15. Statistical analyses were performed using SPSS version 26.0 (SPSS Inc., Chicago, IL, USA), and statistical significance was set at 2-sided P < 0.05.

Results

Development Phase

According to the trajectory of postoperative recovery proposed by previous literature,^8,16 the patient’s daily global health state was the baseline, and disease caused the health state to decline. Until the day of surgery, the health state was at its worst. Therefore, this study defined postoperative recovery as the gradual restoration of the patient’s symptoms and physical, psychological, and social functions to their premorbid state or the average health level of peer group immediately following surgery. The early recovery was defined as the period from the end of surgery to the preoperative state. Long-term recovery was defined as the period from the preoperative state to the premorbid state or the average health level of peer group.

A total of 1602 articles were included in the systematic literature review, and 50 items were extracted and categorized into five domains: symptom, physiological function, psychological function, role, and overall assessment (Supplementary Table 1). The top 10 most frequently reported items comprised pain, daily life, social life, self-care, mood, family life, fatigue, health status, sleep quality, and nausea/vomiting. A total of 138 patients were interviewed, and 22 items were identified and categorized into three domains: symptom, physiological function, and psychological function (Supplementary Table 2). The top 10 most frequently reported items included pain, self-care, sleep quality, vitality, nausea/vomiting, appetite, digestive function, daily life, defecation function, and coughing/expectoration.

The research team selected 11 items from the item pool to compose the initial version of the scale: pain, nausea and vomiting, sleep quality, vitality, diet and drinking, defecation and urination, emotion and mood, self-care abilities, communication and learning capabilities, reintegration with family members, and reintegration into society. Subsequently, 32 experts were invited to participate in a Delphi consultation. The response rate of the experts in each round of consultations was 100%.

Results of the first round of consultation revealed discrepancies regarding the significance of five items: vitality, diet and drinking, urination and defecation, communication and learning, and nausea and vomiting (CV > 0.25), where less than 70% supported “diet and drinking” along with “communication and learning” (Table 1). After incorporating feedback from the first round of consultation, the CV values for importance scores decreased below 0.25 during the second round, while support rates exceeded 70%. The panel reached a consensus on the significance of all items (P < 0.001), and no further amendments were proposed.

Table 1 Results of Delphi Experts’ Consultation

The final PRSA content is presented in Table 2. Each item was scored by the numerical rating scale ranging from 0 to 100. During hospitalization, “family life” and “social life” were assigned a default value of 0. The total score of the scale was calculated as the sum of all items’ scores, with higher score indicating better recovery.

Table 2 Postoperative Recovery Scale for Adult (PRSA)

Validation Phase

A total of 180 patients were enrolled, with exclusion criteria applied to seven patients whose operations were temporarily canceled, three were lost to follow-up, and four could not complete the scale due to serious complications. Finally, the analysis was conducted on a cohort of 166 patients. The baseline characteristics of these patients were listed in Table 3.

Table 3 Baseline Characteristics of the Patients

Validity

The scale-construction process demonstrated content validity. A systematic literature review and patient interviews were conducted to establish the PRSA scale based on the conceptual framework of postoperative recovery. An item pool was formed using the results from both sources. Following the group discussion and Delphi consultation, it was ensured that the content of PRSA was related to postoperative recovery and could accurately reflect its aspects. The Spearman correlation coefficient between the PRSA and QoR-15 scores ranged from 0.780 to 0.904 at each time point, which was statistically significant (Supplementary Table 3). Patients without postoperative complications had a higher PRSA score than those with complications on postoperative day 30 [867.5 (816.3, 930.0) vs 740.0 (646.3, 812.5) (z = 5.61, P < 0.001)].

Reliability

The Cronbach’s α coefficients of PRSA on postoperative days 1 and 30 were 0.785 and 0.819, respectively, indicating good internal consistency reliability. Repeated measurements of the PRSA were performed in 166 patients. The ICC values on postoperative days 3 and 30 demonstrated high test–retest reliability, with scores of 0.993 and 0.994, respectively. A small proportion of patients (4.2% before and 0.6% after surgery) achieved the maximum PRSA score, suggesting a significant improvement in their condition over time. Notably, none of the patients exhibited extreme PRSA scores at any other assessed time point during the study period. The SDC value for the PRSA was approximately 18.3.

Reactivity

The PRSA scores demonstrated temporal variations (Supplementary Table 4): the scores showed statistically significant differences between two adjacent follow-up days (P < 0.005). While the PRSA score on postoperative day 60 did not show a significant difference from the preoperative baseline, other follow-up time points displayed significant differences from the preoperative baseline (P < 0.001). The Cohen values for PRSA and QoR-15 were 6.88 and 4.20, respectively.

Interpretability

The optimal threshold for PRSA was 337.5, and the AUC was 0.91 (95% CI: 0.86–0.96), using a QoR-15 score of 90 as the reference point. Similarly, when a QoR-15 score of 121 was set as a reference, the optimal threshold for PRSA was 657.5, and the AUC was 0.93 (95% CI: 0.90–0.97). The MCID value for PRSA was 217.5 with an AUC of 0.88 (95% CI: 0.81–0.96).

Feasibility

On postoperative days 1 and 30, the median time to complete the PRSA was 102.0 s and 118.0 s, respectively. While QoR-15 took 156.5 s and 146.5 s for the respective time points. The completion time of PRSA was significantly reduced by 44.5 s (95% CI: 41.0–48.0 s) and 27.5 s (95% CI: 24.5–30.0 s), respectively. There was also a difference in the complexity of completing the PRSA and QoR-15. A total of 65.7% of patients found completing the PRSA to be very easy, while the corresponding proportion for QoR-15 was 57.2% (z = 3.77, P < 0.001).

Discussion

In the development phase, the PRSA scale was developed by creating a conceptual framework, establishing an item pool, and Delphi consultation. PRSA is a comprehensive patient-reported outcome scale designed to assess early and long-term postoperative recovery in adult patients. In the validation phase, the PRSA scale demonstrated favorable reliability, validity, responsiveness, and feasibility in adult patients undergoing abdominal surgery. And the score of PRSA could be interpreted by the threshold and MCID values.

A well-constructed conceptual framework serves as the basis for establishing a scale and represents the structured embodiment of the concepts.⁵ Within the conceptual framework, our study offered a precise description of early and long-term recovery, indicating that the health status of patient would continue to change over time. The first eight items of the PRSA incorporate the domains of symptoms, physiological function, and psychological function, rendering them appropriate for a comprehensive evaluation of recovery at any stage. The other two items, “family life” and “social life”, allow patients to report how well they fit into group life. Both items are also relevant for evaluating the preoperative baseline and long-term recovery. Such item composition enables PRSA to effectively meet the needs throughout various stages of postoperative recovery, while previous scales often ignore the influence of the time span of postoperative recovery on the content of scale.⁸

Another reality is that the preoperative interaction between patients and doctors significantly shortened as the proportion of day surgery increases.¹⁷ Moreover, the long-term recovery of patients will gradually improve without medical support after discharge from the hospital. Therefore, some items, such as “getting support from hospital doctors and nurses” in QoR-15, are unsuitable for the evaluation of preoperative baseline and long-term recovery.

The item pool in this study contained numerous items comprising various domains. However, not all were incorporated into the scale. Prior studies have established that a prudent decrease in the number of items of a scale does not undermine its reliability and validity while enhancing its feasibility.^18,19 Currently, there is no standardized method available for screening appropriate items. In this study, we used the reporting frequency of item and topic relevance to conduct the screening process. The identification of items with high reporting frequency reveals prevalent problems that significantly affect the postoperative recovery of patients, thereby establishing the universality of such items. Furthermore, stronger correlations between the items and postoperative recovery could accurately reflect patients’ recovery progress and enhance the content validity of scale. Even though QoR-15 considers “getting support from hospital doctors and nurses” as a crucial aspect of patient experience, its low correlation coefficient (0.29) with total QoR-15 score and poor response (Cohen value 0.04) suggested that it may be unrelated to the postoperative recovery status.¹⁸ Therefore, similar items were excluded from the PRSA.

In the validation phase, this study was designed with the COSMIN consensus. The COSMIN consensus has been published in 2010, which provides terminology and definitions of measurement properties, the design requirements, and preferred statistical methods.⁹ Since there was no gold standard in the measurement of postoperative recovery, the QoR-15 was selected because it had good measurement properties compared with other similar scales and was validated in Chinese patients.^15,18 Some overlapping items exist between the PRSA and QoR-15, such as pain, sleep quality, nausea, and vomiting. However, the two scales describe these items differently: QoR-15 evaluates the duration of items, whereas PRSA evaluates the extent of improvement. The results showed that PRSA had great reliability and validity, establishing a robust foundation for its implementation.

In terms of responsiveness, the Cohen value of PRSA surpassed that of QoR-15 measured during the same period, suggesting that PRSA can more accurately detect changes in the postoperative recovery status among patients. Regarding feasibility, the completion time for QoR-15 in this study was consistent with previous studies.^15,18 But the completion time for the PRSA was shorter than that for the QoR-15. In addition, more patients thought that completing the PRSA scale was very easy. This could be attributed to the fact that the PRSA scale was developed based on the language and cultural background of Chinese patients. And item descriptions of the PRSA are tailored to each patient’s premorbid state (Table 2), thereby facilitating their comprehension of items. In short, the strong responsiveness of the PRSA enables the identification of subtle changes resulting from interventions in clinical research.²⁰ The accessible content and shorter completion time of the PRSA scale emphasize its clinical feasibility, promoting the wider adoption of PRSA in clinical practice and research.

The PRSA scores are continuous numerical values, and a lack of reference values can result in ambiguous information. Consequently, this study computed thresholds and MCID to interpret the scores and change values. The variable of interest is the change value of score. It should be noted that a statistically significant change in score after treatment intervention does not necessarily imply a clinical benefit.²¹ The minimal difference in scores associated with perceived benefit and no side effects by patients was defined as MCID.²² Considering the MCID of QoR-15 as a reference, the MCID for PRSA was 217.5 which exceeded its SDC value. The fact indicates that meaningful clinical changes identified by the PRSA cannot be solely attributed to measurement error.²³

In the current perioperative care, enhanced recovery after surgery (ERAS) has received more and more attention, and its core goal is to improve the quality of postoperative recovery by optimizing the perioperative care process.²⁴ Accurate measurement of postoperative recovery is a prerequisite for improving clinical interventions. In this regard, the PRSA scale is closely related to the principles of ERAS and is designed to evaluate patient-centered postoperative recovery outcomes. The PRSA can effectively capture the key indicators that are consistent with the goals of ERAS, such as symptom relief, improvement of physical and psychological function, and recovery of family life and social life after discharge. These patient-reported outcomes not only help doctors to understand the recovery progress but also provide the basis for the development of personalized recovery plans.

The present study had several limitations. First, this was an observational single-center study, and results were susceptible to selection bias, recall bias, and other confounding factors. Second, although this study has finished the validation of the PRSA scale in patients undergoing abdominal surgery with a sample size meeting precalculated requirements, further studies are required to establish the universality of the scale by including patients undergoing different types of surgeries. Third, the score of PRSA is the sum of all item scores, but it is likely that patients do not view all items as equally valued. In addition to monitor the changes in the total score of PRSA, doctors should also pay attention to specific item score based on patient characteristics in clinical practice. Finally, the native language and cultural background of patients may affect the use of scale. Considering that the initial language version of the PRSA scale is Chinese, we recommend that different language versions of the PRSA scale be validated according to the guideline prior to use.⁵

Conclusion

The PRSA scale consists of 10 items and the content is comprehensive and universal, which can be used to evaluate the postoperative recovery in adult patients. Among patients undergoing abdominal surgery, the PRSA scale has great reliability and validity. Compared with the QoR-15, the PRSA is more suitable for evaluating preoperative baseline and long-term recovery, is more sensitive to postoperative changes, takes less time to complete, and is easier for patients to understand.