Validating Patient Perspectives: A Study on the Reliability of Satisfaction Survey Tools

Introduction

Patient satisfaction is crucial for assessing the quality of healthcare services. Since it reflects patients’ perceptions of their care, it serves as a key indicator for healthcare providers seeking to improve service quality.^1–4 Ensuring positive patient experiences is essential for promoting trust, increasing treatment adherence, and achieving better health outcomes.^2,3,5 In fact, patients’ opinions and satisfaction levels can also influence their future behaviors, particularly in relation to treatment outcomes. Analyzing subjective patient feedback provides a thorough understanding of areas that require improvement, elevating the overall quality of medical care.⁶ Reliable and valid tools are necessary to accurately measure patient satisfaction. The validation process ensures their reliability and applicability in different healthcare settings. This validation not only supports the development of robust measurement tools but also contributes to the broader effort of advancing patient-centered care. In this way, healthcare institutions are better equipped to pursue excellence by implementing improvements directly informed by patients’ voices and experiences.^7,8

As the importance of patient satisfaction continues to grow, various measurement tools have been developed and evaluated, including suggestion boxes, formal complaints, qualitative methods, audits, and satisfaction questionnaires. Among these, satisfaction questionnaires have emerged as the most effective and widely used method of assessing patient satisfaction.⁶

In the past decades, numerous questionnaires have been designed to address the needs of different patient populations and various aspects of medical care, especially in well-developed countries.^{1,2,4,6,7,9–12} However, some of these instruments have been criticized for issues related to their validity and reliability,⁶ while Dunsch and colleagues have pointed out that measurements of patient satisfaction can be biased because they are deeply sensitive to the framing of questions, with patients tending to report extremely high levels of satisfaction with positively framed statements (eg, “This health care facility is clean. Do you agree or disagree?”).¹³

Therefore, ensuring the development of high-quality questionnaires remains essential to accurately capture patients’ experiences and support healthcare providers in their efforts to provide higher-quality care while also taking into account the peculiarities and specificities of the health service organization of the country in which the assessment tools are developed.¹²

The Azienda Ospedaliera (AO) Ordine Mauriziano Hospital of Torino, northwest Italy, has developed a comprehensive system for surveying patients’ satisfaction that employs facility-level questionnaires. These questionnaires are designed to assess cross-cutting aspects of satisfaction rather than facility-specific elements and monitor their trend over time.

A review of the literature and practices among primary inpatient and nursing facilities at the regional and national levels revealed very few validated instruments available^14,15 and many not validated questionnaires in use in most hospitals that better align with the characteristics of the Italian healthcare system (ie universal coverage funded through direct taxation of citizens). This study aims to quantitatively validate two patient satisfaction questionnaires, one for outpatient care and one for inpatient care, developed by the working group established within the Ordine Mauriziano Hospital. Following the face and content validity phase, which involved a collaboration between the mentioned working group and the University of Torino, the reliability and validity of the two questionnaires were assessed. Through this validation process, we seek to affirm that the questionnaires are a solid tool that can be used with confidence by healthcare providers to measure patient satisfaction and identify areas for quality of care improvement. In addition, the widespread use of these questionnaires, at least at the regional level, could facilitate the comparison of results over time and among different health facilities and ease the consequences to be drawn from the data collected, obviating the problem of the difficulty of comparison related to the use of different questionnaires also highlighted in the literature.¹²

Methods

Setting and Questionnaires

The Ordine Mauriziano Hospital in Torino has 400 beds dedicated to inpatients and an annual activity of about 11,500 admissions. Medical and surgical departments serve a diverse population across various specialized wards, including cardiology, oncology, abdominal surgery, orthopedics, cardiothoracic surgery, and vascular surgery.

In recent years, two questionnaires have been used to survey the satisfaction of patients admitted to the hospital or accessing it for outpatient visits. These instruments were developed through a collaborative effort involving medical management, the Clinical Trial Unit, the Health Professions Direction (Di.P.Sa), the Quality Management Office, the Customer Relations Office (URP), and the medical and nursing staff representatives. Recognizing the importance of testing these tools’ validity and reliability, a validation process was started for both questionnaires. An interdisciplinary team was established, composed of the Ordine Mauriziano Hospital working group and biostatisticians from the Department of Public Health and Pediatrics at the University of Torino. The team conducted a series of meetings to review both questionnaires, focusing on face and content validity.^1,7,8,16 The review process led to removing items deemed inadequate or irrelevant, reformulating those unclear, and including previously ignored aspects.

The resulting questionnaires are provided in Italian and English as Supplementary material (Figures S1-S4). Both questionnaires were designed to assess multiple domains of perceived quality of care. The outpatient care questionnaire was a 16-item questionnaire grouped into five domains: front desk staff, administrative staff, Healthcare Workers (HCWs) care and visit, privacy, and comfort; the inpatient questionnaire was a 29-item questionnaire grouped into nine domains: administrative staff, doctor care, nurse care, other HCWs’ care (physiotherapist, logopedist, dietician), aide care, privacy, comfort, food, and after-discharge information.

Items were formulated to evaluate specific aspects of care using a five-point Likert scale (very poor, poor, sufficient, good, and excellent), allowing patients to express their satisfaction. Each option was assigned a score, from 1=very poor to 5=excellent, and the scores of each item were summed, providing the overall score within each domain of the questionnaire.

Moreover, both questionnaires investigated overall satisfaction with two questions: the first was about the judgment of the received care with respect to the patient’s expectation: “Compared to your expectations, the assistance you received was: worse than expected/ as expected/ better than expected.” The second question asked for an overall evaluation of the received care using the same 5-point Likert scale as the other satisfaction questions along the questionnaire (from very poor to excellent): “Overall, how do you rate your experience at our facility?”

Finally, sex, age, nationality, employment status, education level, residence, and length of stay (only in the inpatient questionnaire) were investigated through closed-response questions. A free-response space was provided at the end of both questionnaires to allow patients to highlight aspects of their satisfaction or dissatisfaction and suggest ways to improve their experience in our hospital.

The questionnaires were available online and accessible via a QR code. Patients could use their mobile phone or a tablet provided by the Hospital to fill in the forms.

Population

The two questionnaires were administered to patients admitted to the Ordine Mauriziano Hospital or who made outpatient visits to the same facility. Patients who could answer questions independently, without support from caregivers, were considered eligible; exceptions were minor patients whose parents/guardians had completed the questionnaire.

There is no precise criterion in the literature to define sample size in this type of validation: Nunnally suggested a numerosity of 300, while other authors indicated 4–10 subjects per item and, in any case, a minimum of 100 respondents, as appropriate for validation.^17–19

Based on the latter criterion, the required number of patients was 310 for the inpatient and 180 for the outpatient care questionnaires, respectively. Since the factor analysis could only be conducted on fully completed questionnaires, an oversampling of 10% was provided to ensure sufficient numbers for validation: 340 and 200 questionnaires, respectively.

Based on the annual volume of inpatients and outpatient visits in the Ordine Mauriziano Hospital, it was assumed that the required numerosity would be reached within one quarter. All patients afferent to the facility who met the inclusion criteria and were willing to complete the online version of the questionnaire in the hospital using their mobile device or a company tablet were then enrolled over four months until the required numerosity was reached.

Ethics

Since the questionnaires are anonymous and the data collected do not allow any traceability to the person who provided them, the Italian law does not require the signature of the informed consent of the subjects participating in the study (Italian Data Protection Authority resolutions No. 146 of 05/06/2019 and 417 of 15/12/2022). Before proceeding to fill out the questionnaire, the QR code directed the patient to an information page in which the purposes of data collection, the regulatory references for this type of survey, the absolutely anonymous form of filling out, and the impossibility for those who would have access to the data to trace the identity of the respondent were made explicit. Furthermore, demographic data (gender, age group, level of education, occupation, nationality) were not compulsory and did not block the completion of the remaining part of the questionnaire, and in case of non-completion, no warning was given to the patient with a request to fill in those fields. Moreover, patients were informed about the possible use of the results (in an aggregate form) obtained from the analysis of the questionnaires for planning, improving the quality of care, and possibly scientific publications.

The data were collected through a form set up on the Ordine Mauriziano Hospital’s REDCap platform, which is accessible via the web with a username and password only to users authorized for data analysis.^20,21

The Comitato Etico Territoriale Interaziendale AOU Città della Salute e della Scienza of Torino approved this study (IEC n.499/23). The study was performed in accordance with the principles stated in the Declaration of Helsinki.

Statistical Analysis

The demographic characteristics of patients are shown as absolute frequencies and percentages of the modalities of the categorical variables. The Chi-square test was performed to assess the association between demographic characteristics and the completeness of the questionnaires; the length of stay was also compared between patients admitted to medical wards and those in surgical wards.

Questionnaires with missing responses in the domains related to the perceived quality of care were excluded from the subsequent analyses. Mean, standard deviation (SD), absolute frequencies, and percentages of responses were calculated for all subscales in both questionnaires.

Cronbach’s Alpha (α) was used to assess the internal reliability of scales within each questionnaire domain. It shows how related the items in a dimension are, that is, the degree of homogeneity of the items in a subscale.²² A threshold of 0.7 or higher was used to assess the reliability of constructs.^1,7,17,22,23 Pearson correlation coefficients between each subscale sum score and the patient’s overall satisfaction were calculated, expecting positive and significant correlations.

The adequacy of the sample size for the validation analyses of the collected data was assessed for each questionnaire using the Kaiser-Meyer-Olkin indicator (KMO) and Bartlett’s test of sphericity.^23–26 The KMO statistic, which can vary from 0 to 1, is conducted to examine the strength of the partial correlation, ie, how the factors explain each other between the variables. A KMO>0.8 is considered excellent. Bartlett’s test of sphericity tests whether a matrix of correlations is significantly different from an identity matrix. A significant statistical test leads to the rejection of the null hypothesis of the identity matrix.

A Confirmatory Factor Analysis (CFA) was performed using the unweighted robust estimation method to account for the ordinal nature of the data. We evaluated measures of model fit using SEM techniques. The validity of the questionnaires was assessed by calculating the following indices: the Comparative Fit Index (CFI), which compares the fit of the target model to an independent (null) model, with a threshold of CFI ≥0.90 considered as excellent; the Tucker-Lewis index (TLI), which measures the goodness of fit considering the size of correlations in the data and the number of parameters in the model, with a threshold of TLI ≥0.95; the Root Mean Square Error of Approximation (RMSEA), which is a parsimony-adjusted index with threshold RMSEA <0.08 and a 90% confidence interval; and the Standardized Root Mean Residual (SRMR), which is the square root of the difference between the residuals of the sample covariance matrix and the hypothesized model, with a threshold of SRMR <0.08.^27,28

Finally, for both questionnaires, univariate associations between patient demographic characteristics (sex, age, employment status, and education level) and inpatient/outpatient satisfaction for every subscale domain were analyzed using analysis of variance and t-test, or Kruskal-Wallis and Wilcoxon test, as appropriate. The ratings of care received compared to expectations (worse than expected, as expected, better than expected) were tested with the analysis of variance or the Kruskal-Wallis test.

All the tests performed were two-tailed, and the significance level was set at 5%.

Analyses were performed using Stata18²⁹ and R software (version 4.2.6) with psych and lavaan packages.^30–32

Results

Between February 1st and May 8th, 2024, 481 and 475 inpatient and outpatient care questionnaires were filled out, respectively. 374 and 371 (78%) fully filled questionnaires were considered for the validation analysis.

When sociodemographic data were compared, no differences were found between the completed and incomplete questionnaires, except for age among inpatients, where 91% of patients under 50, 74% of those between 51 and 70, and 79% of those over 70 years completed the questionnaire (p<0.01).

Table 1 shows the characteristics of the patients who fully completed the questionnaires. A higher proportion of patients older than 70 filled out the inpatient questionnaire compared to those who completed the outpatient one (41.4% vs 18.0%, p<0.001). More than 80% of the patients who completed the outpatient questionnaire had at least high school education, compared to 58% of those who completed the inpatient one (p<0.001).

Table 1 Demographic Characteristics of the Patients. Data are Absolute Frequencies and Percentages (%)

Among the 374 patients hospitalized, 170 (45.5%) were in medical wards and 204 (54.5%) in surgical wards. 60.6% of patients in medical wards and 24% in surgical wards had a length of stay longer than seven days (p<0.05).

Internal Reliability

For both questionnaires, all subscale domains were internally reliable, with Cronbach’s α coefficients equal to or higher than 0.80 (Table 2). Specifically, the inpatient questionnaire’s doctor, nurse, other HC workers’ care, and food subscale domains reached α coefficients of 0.9 or higher.

Table 2 Reliability Coefficients for Each Subscale Domain and Correlations Between Each Subscale Domain Score and Patient’s Overall Satisfaction With the Care Received

In both questionnaires, the score of each subscale domain correlated positively and significantly with the overall satisfaction score, suggesting that all subscales measured independent aspects of patient satisfaction.

Sampling Adequacy

The KMO Statistics registered scores of 0.92 and 0.89 for the outpatient and inpatient care questionnaire, respectively. The significance values (p<0.001) of Bartlett’s test of sphericity indicated that factor analysis could be applied to both questionnaires.

Confirmatory Factor Analysis (CFA)

The CFA was performed to test how well the data of the 16 and 29 items of the two questionnaires fit the 5- and 9-factor models, ie, to what extent the items loaded on the appropriate factors (Tables S1 and S2). All items showed loadings above 0.4, except for that concerning the ease of access to the ward (0.3) of the inpatient care questionnaire (Table S2).

Table 3 shows the CFA results for both questionnaires in light of the thresholds proposed by the literature.

Table 3 Fit Indices From Confirmatory Factor Analysis (CFA)

The factor models were tested against a null model, which assumes zero covariance between items (no factors). The model fit was tested using the TLI, close to the 0.95 threshold, while the CFI was higher than the 0.90 threshold for both questionnaires.

Regarding adequacy tests, both the RMSEA and the SRMR were found to be lower than the 0.08 cut-off, representing a good fit.

The subscale domain scores of the outpatient questionnaire showed no significant differences when compared among the study sample’s subgroups (sex, age, education, and occupation). However, significantly lower mean scores were found in older patients (over 70 years old) compared to patients under 50 for the front desk staff and administrative staff domains (both p-value=0.02) of the outpatient care questionnaire (data not shown).

Despite the very small number of patients who rated the care they received as worse than they expected, statistically significant differences were found when subscale domain scores were stratified according to patient expectations (all p-values<0.001), with higher mean scores in patients who reported better care than they expected (Table 4). Specifically, in the outpatient questionnaire, the domain with the lowest mean score among those who judged the care to be worse than expected was the HCWs care and visit, which included items related to specific aspects of the visit and relationship with health personnel, post-visit information, and respect for schedules. In the inpatient care questionnaire, the domains relating to respect for privacy (during visits and in the communication of confidential information), assistance provided by medical staff (doctor care), and that relating to information provided for post-discharge scored significantly lower among patients who were overall disappointed with the care they received than among those who judged it to be in line with or better than their expectations.

Table 4 Mean (SD) Score of Each Subscale Domain Stratified by Patient Expectations

Discussion

A review of the practices among major Italian hospitals could not identify the use of validated patient satisfaction questionnaires in the current practice. However, there has been widespread use nationwide of questionnaires, often very similar in question-wording and item grouping into domains, to investigate patients’ perceptions of the quality of care they received. The Ordine Mauriziano Hospital designed questionnaires to survey patient satisfaction through a dedicated working group. It started a process of reviewing and validating them in collaboration with researchers from the University of Torino. In this framework, the main aim of the present study was to test the reliability and validity of two perceived patient satisfaction questionnaires (one for inpatients and one for outpatient care).

Initial face and content validity was planned to revise already available tools and agree on the items to be included, the measurement scale, and the domains. Then, the quantitative validation was performed. For both questionnaires, the internal reliability of all subscale domains was confirmed by Cronbach’s α coefficients (>0.8). In addition, the scores of all subscales correlated positively and significantly with overall patient satisfaction, suggesting that they measure different aspects of patient satisfaction. Finally, the TLI, CFI, RMSEA, and SRMR results from the CFA showed a very good fit, confirming the construct validity of the questionnaires.

The literature suggests that satisfaction with care may be associated with patient demographic characteristics such as sex, age, and education. Among them, the most consistent determinant seems to be patient age, with older people who tend to be more satisfied with health care than younger people.^1,2,7,33 However, a systematic review of studies published between 1980 and 2014 aimed at identifying evidence related to determinants of patient satisfaction found inconsistent results with respect to sociodemographic characteristics.³⁴ We found no significant differences in the mean subscale domain scores or overall satisfaction when compared by sex, occupation, or education level. Regarding age, the data analysis from the outpatient care questionnaire showed that older patients were significantly less satisfied with the service provided by both the front desk and administrative staff. Furthermore, although it is well known that “expectation” is a problematic concept to examine, the literature recognizes that patients’ expectations play an essential role in determining their satisfaction with the care they receive, and some studies suggest that patients with lower expectations tend to be more satisfied.^33,35 Our questionnaires didn’t specifically explore patient expectations, but we found statistically significant differences when the mean scores of the domains of patient satisfaction were stratified by the overall satisfaction of the patients compared to their expectations. Despite the small number of comprehensively dissatisfied patients, the mean scores of each subscale domain showed progressively and significantly increasing values as they moved from a rating of care “worse than expected” to “as expected” to “better than expected”. These results enabled us to identify the areas that most impacted negative patient judgments. In both questionnaires, it was the domain relating to the care received from healthcare personnel, particularly the doctor, for inpatients. In addition, the privacy and after-discharge domains seemed to be responsible for overall inpatient dissatisfaction.

Despite the patients’ very high satisfaction, the questionnaires’ structure into subscale domains assessing different aspects of the patient’s experience during the hospitalization or outpatient visit seems to be able to identify areas where improvements can be made. Further analyses on larger samples of patients could confirm these results and provide other facets of the validity of both tools. Another aspect we would like to evaluate in the medium term is the possibility of using these questionnaires in other regional hospitals to measure their constructs when applied to different patient populations. Moreover, using the same validated questionnaires on a larger scale would allow a uniform assessment of patient satisfaction, at least on a regional level. More importantly, it would facilitate the identification of organizational aspects of excellence that could then be used as a reference to enhance the quality of services throughout the region.

Conclusion

The present results support the reliability and validity of the questionnaires under study for assessing patient satisfaction with admissions or outpatient visits. In our opinion, the main limitation of the study lies in the limited number of patients who have used the questionnaires so far, aimed at validating the instruments. Although the sample size was sufficient to perform the analyses to validate them, only performing time-series analyses on sample sizes commensurate with the hospital’s volume of activity will enable health care personnel and decision makers to have sufficient data to confirm the suggestions emerging from these analyses, focusing efforts and investments on dimensions that can be improved, and enhancing those that already excel. In addition, in the near future, we will promote the adoption of the questionnaires at the regional level to overcome the problem of having data that are often difficult to compare because they are derived from different measurement tools.