Comparing the Self-Reported Acceptability of Discrete Choice Experiment and Best-Worst Scaling: An Empirical Study in Patients with Type 2 Diabetes Mellitus

Introduction

Incorporating patients’ values, preferences and unmet medical needs to inform decision making may ultimately result in clinical care, health technology lifecycle and reimbursement decisions that better reflect the broader public preferences on disease-specific management.^1–3 This has led to a growing interest in measuring and quantifying patient and public preferences in healthcare through various stated-preference methods, particularly discrete choice experiment (DCE) and best-worst scaling (BWS).^4–6 DCE and BWS were both grounded in random utility theory, which posit that the utility of service attributes could be decomposed into an explainable or systematic component and a random component, and that the random unobservable component exists because of the inherent variability within and between individuals.^7,8 Within a DCE, respondents are asked to select between a set of alternative profiles, each characterized by several attributes and their levels; respondents’ choices subsequently determine how preferences are influenced by each attribute level, as well as their relative importance.^9,10 BWS could elicit additional information on the least preferred option, with three variants allowing respondents to identify the best and the worst attributes (object case, case 1), attribute levels (profile case, case 2), or combinations of multiple attribute levels (multi-profile, case 3).^8,11 To understand the relative ranking or prioritization of the content levels of attributes, case 2 BWS (BWS-2) appears to be the preferred presentation, as respondents do not require to consider the value of the profile as a whole.^12,13

According to the random utility theory, patients’ healthcare-related preferences are elicited by capturing their intentions expressed in hypothetical situations. When faced with multiple trade-offs between two or more alternatives, patients often require sufficient time and knowledge to comprehend complex information and carefully consider their personal preferences. This becomes particularly crucial when contemplating uncertain future scenarios and outcome states they have never experienced before.¹⁴ The excessive cognitive burden imposed by decision fatigue undermines the assumption of utility maximization and leads to inconsistent and poorly considered choices, particularly among individuals with lower health literacy.^15–17 The concept of acceptability is closely related to participants’ rational responses within the stated-preference methods, which requires a clear understanding of both the choice context and choice task. This entails the participants being willing to engage in and capable of completing the task. Failure to meet this critical requirement for eliciting preferences may adversely affect the validity of the responses and impede the reflection of true healthcare preferences.¹⁸

Limited empirical research has been conducted to determine the acceptability of both methods for eliciting healthcare preferences, with existing studies being inconclusive. Flynn et al conducted the first comparison of BWS-2 with DCE and found that whilst the vast majority of (older) respondents provided usable data from the BWS-2 task, only around one half did so for the DCE.¹² Rogers et al demonstrated a better understanding of and ability to complete BWS-2 tasks for valuing health states compared to DCE.¹⁹ The cognitive burden of BWS-2 tasks may be lower as individuals only need to focus on one set of attribute levels in each choice task, as opposed to multiple in DCE.²⁰ However, a systematic review suggested that respondents may favor DCE, as it was perceived to have lower self-reported task difficulty and was preferred over BWS-2 in a priority setting.¹³ Studies by Himmler et al (valuing quality of life measures among older people) and Soekhai et al (treatment preference for patients with neuromuscular diseases) also confirmed that DCE is less cognitively burdensome than BWS-2.^21,22 Furthermore, another study suggested that there may not be a significant difference in completion rates between DCE and BWS-2.⁴ The aforementioned studies serve as a valuable reminder that the acceptability of stated preference methods may vary considerably depending on the specific decision context, and selecting the ideal method to capture a patient perspective is essential for optimal therapeutic decision making and resource allocation in health or community-based care services.

Type 2 diabetes mellitus (T2DM) is a common chronic disease and one of the greatest public health challenges to an aging society. T2DM has a high global prevalence and is rising across all regions.²³ As of 2021, China has the largest number of patients with T2DM; this number is anticipated to exceed 257 million by 2045, which will pose unprecedented challenges to the country’s healthcare and social care system.²⁴ Medications if required could control blood glucose levels in patients with T2DM, thereby delaying disease progression or complications.²³ The American Diabetes Association recommends that a patient-centered approach should be used to guide the choice of pharmacologic agents as well, considering multidimensional factors associated with patients’ preferences.²⁵ The present study was designed within the context of preference elicitation for second-line anti-hyperglycemic medicine among patients with T2DM to specifically document stated-preference methods for assessing acceptability from the perspectives of patients with T2DM. These treatments were chosen because the diverse properties of glucose-lowering agents can guide individualized treatment choices for patients with T2DM during long-term disease self-management, further influencing their confidence and adherence to treatment.²⁶ Additionally, patients with T2DM may be accompanied by a measure of age-related cognitive decline,²⁷ and it is important for them to reduce the cognitive burden and ensure validity of medication preference elicitation. Hence, we aim to achieve two objectives: (1) to describe the difference between perceived acceptability of DCE compared to BWS-2 in patients with T2DM; and (2) to identify factors related to sociodemographic characteristics, health status, and disease treatment that may influence the acceptability of these preference elicitation methods.

Material and Methods

Survey Design and Participants

We used a multistage stratified cluster-sampling procedure that considered the geographical region and economic development status for data collection between September 2021 and January 2022. In stage one, we selected six provinces and municipalities that represented the socioeconomic status and lifestyles of five major geographical regions in China. In stage two, we selected a provincial capital city (developed city) and one or two non-provincial capital cities (underdeveloped cities) from each geographical region in China. In stage three, we randomly selected one or two hospitals and two or three primary-care institutions from each city. The inclusion criteria were as follows: (1) age ≥18 years, (2) confirmed T2DM, and (3) voluntary participation. Following the rule of thumb proposed by Johnson and Orme, we settled on a minimum sample size of 250.^28,29 The questionnaire for patients with T2DM comprised three parts: (1) patient sociodemographic characteristics and information related to health status and disease treatment; (2) six DCE choice tasks, and six BWS-2 choice tasks; and (3) self-reported acceptability of the two methods.

We followed recommended guidelines proposed by the International Society for Pharmacoeconomics and Outcomes Research for DCE and BWS study design and analysis.⁵ Regarding attributes development, an initial list of attributes was obtained through a literature review, and the seven attributes included in the final survey were identified through multistage expert focus-group discussions and a case 1 BWS experiment, including treatment efficacy (reduction in HbA1c), hypoglycemia events risk, gastrointestinal events risk, weight change in 6 months, cardiovascular benefits, route of administration, and monthly out-of-pocket costs; the levels were further developed based on the clinical practice of T2DM in China. The final set of DCE and BWS-2 questions included in the survey was generated using D-optimal algorithm by SAS to construct a fractional factorial experimental design. The resulting design included 48 unique DCE and BWS-2 questions, assembled into eight blocks of six tasks each. The final experimental design was evaluated for level balance, minimal overlap, and orthogonality, as shown in Figure S1 (examples of the DCE and BWS-2 choice tasks). Each DCE task offered respondents a choice between two hypothetical medication profiles; patients were asked which medication they would prefer, and each BWS task asked respondents to choose the attribute that was most or least important to them in one hypothetical medication profile. In the DCE, a strictly dominant option (alternative A is preferred over alternative B) is included to evaluate the internal validity of the data. In the BWS, the validity task included only three attribute levels: 0, 100, and 600 Chinese Yuan. Respondents were expected to select the first option as the best feature and the last option as the worst. Details on the attributes and levels of development, as well as the experimental design of medications, are available elsewhere.³⁰

After completing the choice tasks for DCE and BWS-2, the respondents self-reported the acceptability of the methods. The conceptual framework of acceptability outlines three key variables: completion difficulty, comprehension complexity and response preference. A 5-point Likert scale (1 = very simple, and 5 = very difficult) was used to rate the difficulty of completing a series of choice tasks associated with DCE and BWS-2. For comprehension complexity and response preference, the respondents were asked to make a trade-off between the two methods and consider which was easier to comprehend and which they preferred.

Data Collection

We collected data face-to-face to ensure the quality of data collection. The DCE and BWS-2 instructions were explained in detail to the investigators, who received training from the research team. Cognitive pre-tests were administered from June to July 2021 to ensure the validity and comprehensibility of the survey instrument. Participants (n=12) were provided with pre-test questions and given the option to verbally share comments for revision. During the formal survey (September 2021 to January 2022), participants completed the questionnaire anonymously after providing written informed consent. Each participant was randomly assigned six DCE and six BWS-2 choice tasks in one of the eight blocks. Participants who had difficulty filling in the questionnaire, such as those with impaired eyesight or were illiteracy, were given the option to complete the questionnaire verbally. The completion of the questionnaire took approximately 10–20 minutes, and each participant was reimbursed for their time. All completed questionnaires were returned directly to the investigators, and patients’ identities and information were strictly protected by the researchers.

Statistical Analysis

Patient-specific characteristics and the tasks’ completion difficulties were assessed and reported as the mean ± the standard deviation. The difficulties were compared using the Wilcoxon rank sum test. All continuous variables were then transformed into categorical variables and presented as frequencies and percentages. Cohen’s kappa was employed to assess intra-rater reliability in consideration of potential variations in completion difficulty among individual respondents, as well as variations between understanding complexity and response preference. To compare categorical variables between the DCE and BWS-2 groups, univariate analysis was conducted using the chi-square test or the Cochran-Mantel-Haenszel tests.

To further examine the potential factors associated with the acceptability of the eliciting preference methods, multivariate logistic regression analyses were used based on statistically significant variables in the chi-square tests, and odds ratios (ORs) and corresponding 95% confidence intervals (CIs) were reported. According to the variance expansion factor (VIF) and tolerance results, there was no evidence of multicollinearity in the logistic regression model (VIF > 2, tolerance < 0.5). The statistical analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).

To assess the robustness of the multivariate analysis and explore the variable importance more accurately, we used random forest (RF) as a nonparametric machine-learning algorithm to construct classification models for predicting the choice of method and determining variables’ importance. We analyzed the full dataset to develop these RF models, and significant variables from the chi-square tests were entered into the models as potential classifiers. The parameters’ default settings were used for classification in the algorithm. We used the out-of-bag (OOB) proportion of the data left outside of the algorithm as validation data to compute the classification error, which was ultimately averaged over all trees. We also reported the OOB estimation error. The variable importance analysis considers the mean decrease in accuracy as an evaluation indicator to predict the acceptability of the methods by testing the OOB sample. The RF was implemented using R software version 4.2.2 (R Development Core Team, Vienna, Austria) “random Forest” package.

To validate the robustness of the base-case analysis, we also performed sensitivity analyses to include inconsistent responses in the preference analysis and analyzed the order of method to investigate whether the sequence of completing the DCE or BWS-2 would have any impact on acceptability. The level of statistical significance was set to 0.05 for all statistical analyses.

Results

Respondents’ Sociodemographic and Disease-Related Characteristics

In total, 3487 eligible patients with T2DM completed the survey. A total of 201 respondents (5.8%) failed the validity test owing to their choice of an inferior option, leaving 3286 respondents for the final analysis. The self-reported sociodemographic and disease-related characteristics of the respondents are summarized in Table S1. The mean age of the respondents was 61.4 ± 11.9 years, and the median duration of disease was 9.5 ± 7.1 years. Of the respondents, 50.2% were female, 49.5% had a body mass index (BMI) ≥24, and the majority were urban residence (64.2%) and outpatients (69.5%), making the sample nationally representative.³¹

Description of the Self-Reported Acceptability in DCE and BWS-2

Table 1 shows the responses to the questions regarding the acceptability of the two methods. Respondents indicated there was no statistically significant difference in completion difficulty between DCE and BWS-2, although the DCE scores were slightly higher (3.07 ± 0.68 vs 3.03 ± 0.67, P = 0.06). This indicates that more respondents thought the DCE might be more difficult to complete than the BWS-2. However, regarding comprehension complexity, approximately 60.2% of respondents tended to choose DCE as being easier to comprehend, and there was almost no difference in response preference for the methods. The weighted Cohen’s kappa value for intra-rater agreement was 0.62 (95% CI: 0.60–0.65) for the difficulty of tasks, and 0.67 (95% CI: 0.64–0.69) for agreement between comprehension complexity and response preference, which showed the possible disagreement among individual responders regarding their choices (Figure 1).

Table 1 Acceptability in Completing DCE and BWS-2 Tasks (n=3286)

Figure 1 Agreement between the self-reported acceptability of DCE and BWS-2 (N=3286). (A) agreement between the difficulty of the two methods, (B) agreement between comprehension complexity and response preference of the two methods. Abbreviations: DCE, discrete choice experiment; BWS-2, case 2 best-worst scaling.

Factors Associated with Comprehension Complexity

The variable assignments are provided in Table S2. In the univariate analysis, seven variables (residence, annual income, illness duration, BMI, route of administration, monthly out-of-pocket costs, and gastrointestinal events) were statistically significant (P < 0.05) and included in the multivariate model (Table S3). Results displayed in Table 2 indicate that the identifying factors of comprehension complexity were significantly associated with residence (OR=1.46, 95% CI=1.25–1.71), annual income (OR=1.27, 95% CI=1.16–1.39), illness duration (OR=1.30, 95% CI=1.11–1.53), BMI (OR=0.55, 95% CI=0.38–0.81), preferred route of administration (OR=0.79, 95% CI=0.63–0.98), monthly out-of-pocket costs (OR=0.90, 95% CI=0.86–0.95), and gastrointestinal events (OR=0.77, 95% CI=0.66–0.91). Specifically, respondents who were rural residents, had a higher level of annual income, had a longer disease duration, were overweight or obese, preferred oral administration, had lower monthly out-of-pocket costs, and experienced gastrointestinal events were more likely to consider BWS-2 easier to comprehend; respondents who demonstrated the opposite were more likely to consider DCE as being easier to comprehend.

Table 2 Association of Sociodemographic and Disease-Related Factors with Comprehension Complexity Between DCE and BWS-2a

The RF analysis was used to examine the seven variables associated with comprehension complexity in the univariate analysis. The overall OOB estimate of the error rate for the full dataset was 39.2%. The importance scores are shown in Figure 2(A), and following predictors with positive scores were arranged in descending order of importance: residence, monthly out-of-pocket costs, gastrointestinal events, annual income, and preferred route of administration.

Figure 2 Random forest to classify comprehension complexity and response preference. (A) permutation variable importance measures for comprehension complexity among DCE and BWS-2, (B) permutation variable importance measures for response preference among DCE and BWS-2. Abbreviation: BMI, body mass index. Notes: Variables with positive importance values increased the accuracy of the random forest algorithm, whereas negative values decreased the accuracy.

Factors Associated with Response Preference

Four variables (residence, education level, illness duration, and monthly out-of-pocket costs) in the univariate analysis were included in multivariable modeling (Table S4), which found that respondents with a rural residence (OR=1.27, 95% CI=1.09–1.48), longer disease duration (OR=1.30, 95% CI=1.12–1.52), and less monthly out-of-pocket costs (OR=0.94, 95% CI=0.90–0.98) preferred the BWS-2 over the DCE. No significant association was found between the educational level and response preference (Table 3).

Table 3 Association of Sociodemographic and Disease-Related Factors with Response Preference Between DCE and BWS-2a

The overall OOB estimate of the error rate for the full dataset in the RF analysis was attenuated to 46.8% using only four important covariates. Figure 2(B) shows that all variables had positive scores and were useful for classifying the response preferences of the methods: residence, illness duration, education level, and monthly out-of-pocket costs.

Sensitivity Analyses

Table S1 displays the sociodemographic and disease-related characteristics of all respondents, including those excluded based on the validity test. These characteristics were not significantly different from the base-case characteristics, and their inclusion in the sensitivity analyses yielded results that were generally consistent with those of the base-case analysis (Tables S5–S9 and Figure S2). Furthermore, this analysis revealed a notable association between fasting blood glucose levels and acceptability of the methods. There was no statistical difference between the DCE and BWS-2 completion difficulty scores (3.06 ± 0.69 vs 3.03 ± 0.68, P = 0.09). Additionally, we discovered that the order of the methods affected completion difficulty, although there was no difference in comprehension complexity or response preference (Table S10).

Discussion

Comparative studies examining the merits of DCE and BWS, have gained significant attention, emphasizing the need for further exploration and understanding of their comparative advantages. This study’s objective was to investigate the acceptability of DCE and BWS. Additionally, we aimed to identify the key sociodemographic characteristics and disease-related factors that are most influential in determining the acceptability of these methods by performing a preference study using a nationally representative T2DM sample in China. To our knowledge, there has been limited research examining factors related to the acceptability of stated-preference methods using a large patient sample. In this study, we discovered that DCE and BWS-2 had similar levels of difficulty. Interestingly, the DCE was found to be easier to comprehend, but there was no substantial difference in the response preference between the two methods.

Several studies have addressed acceptability in various ways; our findings were somewhat different from those in previous literature.^4,13,32,33 Specifically, we observed that there was no statistically significant difference in completion difficulty between the DCE and BWS-2, despite the DCE scores being slightly higher. Interestingly, DCE is associated with a relatively simpler cognitive process in terms of comprehension complexity. These seemingly contradictory findings can be explained by examining the qualitative information provided by the respondents.³⁴ The DCE task allows for a realistic decision-making scenario involving multiple attributes and levels that requires thorough analysis and decision-making based on a comprehensive framework. However, the BWS task involves selecting the most/least important attribute/level combination that is furthest apart on the latent utility scale, which is a more abstract process and can sometimes be less time-consuming. For researchers, BWS can offer more comprehensive insights into potential utility functions and enhance the statistical efficiency of preference elicitation.³⁵ However, this may not hold true for respondents involved in decision-making processes. Individuals without prior experience in a specific application area may find it challenging to identify extreme preferences from a set of choices, thereby hindering the accurate reflection of realistic preference weights. The notion of “comprehension” is frequently recognized as a vital aspect in establishing the validity of DCE and making informed decisions. Decreased cognitive burden associated with DCE may lead to a reduction in decision uncertainty. However, it is worth noting that research has demonstrated that “failing” consistency tests does not necessarily imply irrational or uninformed responses, nor does this indicate a lack of understanding of a task. This perspective is further supported by the results of our sensitivity analyses.^18,32

Our study findings align with Janssen et al’s study on patients with T2DM, which indicated no significant difference in response preference between DCE and BWS.³⁶ In Janssen’s study, pre-test interviews revealed that participants did not exhibit a preference for either elicitation method. However, it is important to note that participants in other studies showed distinct preferences for one task over another. For instance, in a preference survey on funding for new health technologies, the Australian public favored the DCE, while children in the United Kingdom preferred the BWS to assess dental caries.^19,32 To explore potential discrepancies and determine whether these preferences are unique to healthcare priority-setting or more widely applicable, further disease-specific studies or replication of our study in different settings are crucial.

Regarding the sociodemographic and T2DM-related factors for the acceptability of DCE and BWS-2, we found that residence, illness duration and monthly out-of-pocket costs were significantly associated with acceptability, while additional factors such as annual income and gastrointestinal events were specifically related to comprehension complexity. Rogers et al considered using BWS as a more suitable method for valuing health states in children, prompting its selection in another study focused on generating value sets for adolescent caries-specific oral health-related quality of life.^19,37 Similarly, another study examining social care outcomes opted for BWS owing to its perceived lower cognitive burden and greater suitability for collecting preference data from service users.²⁰ Based on our model analysis results, we assigned the DCE to urban residents in the preference survey among patients with T2DM to mitigate their cognitive burden. The logistic regression and RF models complemented each other, enhancing our ability to identify the factors associated with acceptability from various perspectives. Key drivers, such as residence and monthly out-of-pocket costs identified by the RF model can be targeted for further surveys. These results may inform future research on the most suitable method of stated preferences for target T2DM populations with specific sociodemographic characteristics in the context of limited healthcare resources. No significant association was found between education level and acceptability of the methods. However, the RF model highlighted the importance of considering this variable for classification purposes. One plausible explanation is that certain vulnerable populations received assistance from the investigators in completing the DCE and BWS choice tasks. The investigators clarified the meaning of the tasks and provided guidance for eliciting preferences.

This study has some limitations that must be acknowledged. First, because our study involved a face-to-face survey, we did not collect the exact completion times of the DCE and BWS choice tasks, which could be a metric of the acceptability of the two methods. However, completion time alone may not fully capture the acceptability of DCE and BWS, as individuals who spend more time on tasks might have been engaging in more careful trade-offs during their decision-making processes. Additionally, the face-to-face survey method may have introduced some bias, particularly for participants who received assistance in completing tasks. Second, because of the inherent design limitations of cross-sectional research, causal inferences could not be derived. While our study employed a self-report form to investigate patient acceptance of stated-preference methods, which included a large and representative sample obtained through stratified cluster random sampling, it only partially reflected patients’ perspectives and did not reveal the underlying reasons behind their choices. To gain a deeper understanding, additional qualitative research, such as interviews, is necessary to explore patients’ perspectives. Finally, caution should be exercised when interpreting results based on self-reporting measures, as they are subject to inherent limitations and potential biases. It is also important for future research to explore the comparative merits of DCE and BWS from additional perspectives, such as by evaluating the concordance of different methods using the generalized multinomial logit model. This would provide more empirical evidence for choosing the most suitable preference elicitation method.

Nevertheless, our study provides valuable insights for identifying optimal healthcare preference methods that adapt to the cognitive abilities of patients with T2DM, informing future studies that focus on patient acceptability in healthcare decision-making. This, in part, allows patients to make a thorough trade-off with limited cognitive capacity, thus plausibly eliciting evidence of healthcare preference, improving the healthcare decision-making process, and promoting patient-centered care.

Conclusion

We conducted an empirical study based on a sample of Chinese patients with T2DM to examine the relative self-reported acceptability of two stated-preference methods, DCE and BWS-2. Specifically, respondents perceived the difficulty of completing the DCE and BWS-2 to be similar, but the DCE was easier for respondents to comprehend than the BWS-2, and no significant differences were found in response preference between the two methods. We also observed that respondents’ sociodemographic and disease characteristics partially influenced the acceptability of the methods. This study promotes a focus on patient acceptability in quantifying individual healthcare preferences to inform tailored optimal stated-preference methods for a target population within the context of limited healthcare resources.