An Empirical Comparison of Discrete Choice Experiment and Best-Worst Scaling to Estimate Patient Preferences in Infertility Treatment in China

Introduction

Preference can be defined as a specific expression of the maximum value of utility.¹ In this context, treatment willingness and choice preference can be understood as rational choices made by patients after weighing different treatment situations. These choices reflect the individual’s assessment of the value of health services and prioritization.² An understanding of patient preferences is meaningful in the development of a patient-centered healthcare service model,³ which is a crucial element in the provision of high-quality healthcare services and an effective means of enhancing the public’s healthcare experience and contributing to the advancement of the healthcare industry. Among the stated preference research methods, discrete choice experiments (DCE) and profile case best-worst scaling (BWS-2) are two typical measurement tools. Both National institute for Health and Care Excellence and Food and Drug Administration recommend the application of such methods to obtain patients’ preference information.⁴ However, there are notable differences in experimental design, analytical models, and participants’ behavioral decisions.^5–7 In details, they differ in the approach to choice set design. BWS requires patients choose the best and worst options at the intrinsic attributes of the object, frequently employing counting and modelling methods for data analysis. DCE requires patients to make choices in a virtual situation consisting of levels of attributes (trade-offs), often using modelling methods for data analysis. In summary, BWS-2 captures the advantages of patient prioritization through extreme value selection. In contrast, DCE is effective in modelling complex clinical decision-making scenarios. In the field of medicine and healthcare, the ISPOR organization has presented best practices and guidelines for DCE preference studies.^8–10 While BWS-2 was introduced to healthcare research relatively recently and has relevant guidelines for its use,¹¹ it has unique advantages as an emerging research method.¹² The advantages of BWS are demonstrated by the decreasing sample size and number of attributes in the study, as well as the relative simplicity of the analysis method. BWS is an attractive and relatively easy method. It is therefore important to compare the preferred outcomes of the two methods and to assess patients in order to promote the application of both methods.

In recent years, China’s fertility and population growth rates have continued to decline, and the incidence of infertility has continued to climb, which has become a serious problem affecting social development.¹³ The prevalence of infertility in China has increased from 11.9% in 2007 to 17.6% in 2020. The infertility rate among Chinese women is estimated to be as high as 15%-20%.¹⁴ A quarter of couples of reproductive age suffer from infertility, compared to the global average of 15%.¹⁵ This contributes significantly to an ageing population and a shrinking labor force. For patients, it is a heavy financial burden and a social stigma.¹⁶ Infertility patients tend to choose assisted reproductive technologies (ART) to achieve their fertility wishes.¹⁷ Techniques such as embryo transfer and preimplantation genetic screening have increased the probability of patients having children to varying degrees.¹⁸ However, there are significant differences in specific procedures, treatment side effects, and maternal and infant safety, all of which influence patients’ choice preferences.¹⁹ The diverse applicability of multiple techniques makes treatment decisions highly preference sensitive.²⁰ Relevant studies emphasize the significance of patient preferences for decision-making or policy implementation in the field of assisted reproduction. For example, Harrison et al²¹ found that patients’ willingness correlates with stronger perceived control and proactive decision-making. This emphasizes the need for clinics to adopt a multi-cycle framework during consultations that aligns with patient preferences. In addition, patients value cost during treatment, particularly embryo donation in frozen embryo transfer,²² which complicates decision-making. In DCE study, patients in the Netherlands realized two patterns of preference, valuing success or valuing low-burden.²³ Therefore, it is important to accurately and efficiently measure patient preferences and incorporate them into multi-cycle plans using a shared decision-making model that balances efficacy, burden and other factors. In China fertility treatments are gradually being included in health insurance payments and patient out-of-pocket costs are decreasing. In addition, the adoption of patient-centered approaches, such as providing more information about treatment and helping patients to participate in decision-making, may also alter patient choice.²⁴ ARTs are rapidly evolving and new treatment options are complicating decision-making.²⁵ The selection of safe, cost-effective, and appropriate treatment options that incorporate patient preferences has become key to improving the quality of care and facilitating scientific decision-making,²⁶ which can help to optimize the level of healthcare services, improve patient compliance as well as improve pregnancy outcomes.

A number of international studies have been conducted on the applicability of preference measurement tools, in areas such as disease treatment and prioritization. However, there is an absence of research related to the preference measurement tools in infertile patients. The objective of this study was to identify the factors that influence the treatment preferences of infertility patients and to examine the differences in the preferred outcomes of patients when surveyed using the BWS-2 and DCE questionnaire formats. Furthermore, patients’ assessments of the two methods were compared in order to provide lessons and references for empirical research and practical use of disease treatment preferences.

Method

Study Population

Currently, ART has been implemented on a significant scale, with 622 medical institutions in China having obtained approval to conduct human assisted reproduction technology. Of these, 38 are located in Jiangsu Province, which is the second largest region in China. The inclusion criteria for patients were: women aged 20–45 years, suffering from infertility, seeking medical assistance for infertility, having good communication skills, and voluntary participation in the questionnaire study. The study was conducted in three hospitals, of which Nanjing Drum Tower Hospital and Jiangsu Province Hospital both have hospital districts exclusively dedicated to carrying out assisted human reproduction technology, while the research in Jiangsu Women and Children Health Hospital was carried out in the reproductive medicine center. The study was conducted in the female-specific clinic rooms of the three sample hospitals. Participants were recruited by simple random sampling from patients who met the criteria. At present, there is a paucity of uniform international standards for the calculation of sample sizes in BWS-2 studies. The most common method employed is to draw upon DCE to determine the requisite sample size. A minimum of 84 respondents (up to three levels, nine choice sets and two alternatives) were required, as determined according to the Orme’s formula (N > 500*c/(t*a)).²⁷ Further details are available in the eAppendix 1. In consideration of the response rate and tolerable error, and given that the objective of the study was not to account for heterogeneity of preferences, a sample size of 300 patients was planned to ensure reliable statistical analyses. The anonymous research was conducted between April and June 2024. Prior to the administration of the questionnaire, respondents were provided with comprehensive information regarding the purpose and content of the survey by a trained researcher. The study was approved by the Ethics Committee of Nanjing Medical University, with ethics approval number 2020103. All methods were performed in accordance with the relevant guidelines and regulations outlined in the Declaration of Helsinki. Informed consent was obtained from all participants.

Attributes and Levels

This study was based on an extensive literature review, which identified three main categories of attributes:²⁸ outcome attributes, process attributes, and cost attributes. In total, 13 attributes were identified (details in eTable 1). An object case BWS-1 was employed to prioritize the attributes from the patient’s perspective. The study was conducted with 150 patients. The results of this study (see in eTable 2) led to the finalization of six attributes, as detailed in Table 1. To ensure that the study design was practical and feasible within the current healthcare and policy frameworks, an expert consultation was conducted with an associate professor of medicine and a nurse from a reproduction center. This consultation was used to further refine and modify the formulation of the attributes. Following this, the formulation of the identified attributes was further refined and modified. Refinements and modifications were then made to the questionnaires designed for the identified attributes, and a pilot study was conducted. A sample of 40 patients was selected from Nanjing Drum Tower Hospital to assess the practicality, acceptability, and reliability of the study. The final questionnaire comprised three sections. The first section collected basic information about the patients, while the second section consisted of a series of choice sets. The order of the questions in the DCE and the profile case BWS-2 was randomized to avoid any potential bias due to differences in the order of completion.²⁹ Patients were asked to choose their preferred option after a detailed and concise explanation of the questionnaire’s content, structure, and requirements for completion. Ultimately, it is essential to ascertain the patients’ assessment of the two preference measurement tools.

Table 1 Attributes and Associated Levels Identified for This Stated Preference Study

BWS-2 Questionnaire Design

In this study, 18 distinct profile case BWS-2 choice sets, each comprising six attributes and one of their corresponding levels as previously described, were constructed using orthogonal main effect design (OMED) using R 4.3.3 software. To minimize the burden on patients, the questionnaires were distributed equally between the two versions. Furthermore, each version was supplemented with a single set of consistency test questions. In order to facilitate comprehension of the questions posed in this study, attribute and level explanation cards and corresponding sample questions were prepared. See Figure 1 for details.

Figure 1 Example of BWS-2 and DCE choice tasks.

DCE Questionnaire Design

Similarly, 18 choice sets were generated and randomized into two versions using bayesian efficient design using Ngene 1.4 software. The cond command was used in the design to avoid the implausible situation where the live birth rate is higher than the pregnancy rate. In addition, patients were given the option to decline healthcare services regardless of their attributes and level of care. The unforced choice provided an opt-out option that required respondents to complete the forced choice and choose between opt-out options.^30,31 Each version was supplemented with an additional set of consistency testing questions. The questionnaire design for this study was conducted in accordance with the ten criteria for stated preference research set forth by the ISPOR.

Statistical Analysis

Both BWS-2 and DCE are founded on random utility theory, which is employed for the purpose of eliciting patients’ stated preferences. The BWS-2 data was transformed with the marginal sequential model using R 4.3.3 software. The counting method was calculated based on the frequency with which the attribute levels were selected as the best and worst. The attribute level with the lowest utility in the count model was employed as a reference term for further conditional logit model analysis. Significant positive and negative coefficients indicate a preference for certain attribute levels. The same conditional logit model analysis was carried out for the DCE data using Stata 17 software. In order to facilitate comparison between DCE and BWS-2, the relative importance of attributes was calculated separately. A comparative analysis of the two preference measurement tools was conducted by plotting the scatterplot of the main effects model and measuring the correlation of the preference results under the different preference tools. Additionally, the results of the evaluation questions of the two methods were analyzed. The level of statistical significance was determined using a threshold of P<0.05 (two-sided), and 95% confidence intervals (CI) were calculated using the delta method.

Results

A total of 40 patients diagnosed with infertility completed the questionnaire utilized for the prior survey. The model estimates demonstrated positive and negative signs and weights (see eTable 3). This indicates that respondents are able to comprehend and complete the DCE choice sets. The prior values and patient feedback will be applied to the DCE choice set design, with modifications to the final version of the formal questionnaire.

Basic Information

A total of 400 respondents were invited to complete the questionnaire. A total of 330 valid questionnaires were included in the study, with a high-quality control pass rate, following the exclusion of sample that did not complete all questions and did not pass the consistency test. The results are demonstrated in Table 2. The mean age of the respondents was 32.9 years old. Only 23.6% of them had their usual place of residence in the city. They were well-educated, with 64.6% having obtained a university or college degree or above. Additionally, half of them had a monthly household income within the range of $743 and $2,226. With regard to the patients’ experiences of treatment, 44.8% had been preparing for pregnancy for 3–5 years, over half of the patients had received previous fertility treatment, and 65.8% were currently engaged in the treatment cycle.

Table 2 Characteristics of Respondents Included for Analysis

Comparison of Preferences

The results of the model in the BWS-2 (Table 3) show that using neonatal complications as the reference attribute, the magnitude of the regression coefficient value indicates the magnitude of the utility value obtained at that level in comparison to the reference level. The conditional logit model results showed the highest positive impact was the live birth rate attribute at the 55% level (1.91, p<0.001), and the negative impact was the $50,000 project cost (−0.83, p<0.001). Patients had no significant preference for the intermediate levels of each attribute (p>0.1). These findings were also reflected in the priority of Std.BW in the counting method. To directly estimate attributes weights and level preferences, the BWS-2 utilized effect coding. In order to ensure consistency in the coding process, both effect variable coding and dummy variable coding were applied to the data collected for the DCE in this study. Table 4 presents the regression model results for DCE. It can be seen that the preference weights are similar to those of BWS-2, with a monotonically increasing trend in the preference weights within the attributes, with the exception of the two complications attributes, which have low values. An increase in the live birth rate from 15% to 55% would result in the patient gaining a marginal utility of 0.77 (p < 0.001), which is 4.5 times the lowest utility value. Furthermore, the coefficient of the exit option in the DCE study was 0.31, indicating that opting out yielded a positive value utility for the respondents and that infertility patients were more inclined to abandon the treatment. The results of the present study demonstrated statistically significant findings for both the DCE and BWS-2 preference studies, indicating that both preference measurement tools exhibited validity in this study. The BWS-2 was observed to be more responsive to alterations in level (from level 1 to level 2). For instance, the estimated relative utility change associated with a shift in the live birth rate from a medium (35%) to a high (55%) level was considerably higher in the BWS-2 than in the DCE. A Spearman correlation analysis was conducted on the data from the two models, and the results demonstrated a strong positive correlation between the preference results of the two methods, with a correlation coefficient of 0.811 (P < 0.001). Furthermore, the coefficient of determination of the linear fit in the scatter plot (Figure 2) is 0.76, which also indicates a high level of agreement between the preference results of the two measurement tools. Furthermore, the slope of the fitted line is 3.56, indicating that the difference in coefficient sizes across attribute levels in BWS-2 is considerably larger than in DCE for equal unit changes.

Table 3 Results of the Profile Case best-Worst Scaling (BWS-2) Analysis

Table 4 Results of the Discrete Choice Experiment (DCE) Analysis

Figure 2 Plotting scatterplots of preference weight comparisons for BWS-2 and DCE.

Further analysis reveals difference between the relative importance results of the two methods. In order to facilitate comparison between studies, this study sets the live birth rate attribute with the highest utility value as the common denominator for scaling. As illustrated in Figure 3, the difference between the two method preference outcomes is specifically seen in the importance of program costs, which ranked fourth in BWS-2, much higher than neonatal complications, and fifth in DCE. Maternal complications were also elevated in importance in DCE to almost the same status as participation in TDM.

Figure 3 Relative importance of attributes for BWS-2 and DCE.

Comparison of Burden

In order to facilitate a comparative analysis of the two methods, this study required patients to complete both the DCE and BWS-2 questionnaires, as well as the assessment questions. To eliminate any potential order bias, the order in which the choice sets appeared was reversed in both versions. Table 5 presents the descriptive analyses, which indicate that patients were more likely to understand the DCE questions and preferred to complete the DCE questionnaire. A paired samples Wilcoxon test was conducted to compare respondents’ scores on method difficulty. When the BWS-2 questions appeared first, patients demonstrated a higher mean difficulty in understanding the BWS-2 relative to the DCE. Additionally, a greater proportion of patients selected the DCE as their preferred measurement tool. Conversely, when the DCE appeared first, no significant difference in difficulty was observed. No significant difference was observed in the difficulty ratings of the BWS-2 by education level (p = 0.107). However, for the DCE choice set, there were statistically significant differences between the education groups (difficulty scores for respondents with different levels of education were 5.33,4.12,3.85 and 3.44 respectively; p=0.007). The same results are shown on the eTables 4 and 5. Respondents’ choice of measurement tools was related to their educational qualifications.

Table 5 Evaluation Questions for BWS-2 and DCE

Discussions

This study focuses on the differences in the estimation of preference weighting outcomes for infertility patients by BWS-2 and DCE in different assisted reproduction scenarios, along with patient problem assessment. It was found that the attribute levels of BWS-2 and DCE were similarly weighted, with patients preferring treatment options that were highly effective, had low side effects, were patient-centered, and were low-cost. There were only slight differences in the relative importance priority ranking of attributes in terms of individual attribute rankings. For patients, DCE questions were perceived to be less difficult to understand and were more likely to be filled out, whereas the difference in utility scale for DCE was smaller.

The strength of this study is that, although this is not the first study to focus on outcome differences between DCE and BWS-2, to our knowledge, no similar studies have been conducted within a specific infertility patient. Furthermore, this study is significant in light of the rising prevalence of infertility and the increasing trend towards younger age groups, a demographic that often exhibits positive attitudes towards triggering preferences and can act throughout the course of treatment. One of the primary findings of this study was the quantification of infertility patients’ preference for medical treatment. The statistically significant preference results obtained from both the BWS-2 and DCE preference measurement tools, as well as the concordance, indicate that both preference measurement tools possess good validity in this study. This suggests that the DCE and BWS-2 research methodology are effective in revealing the preferences of infertility patients. The findings indicate that female infertility patients who are prepared to undergo or are undergoing ART derive greater utility from programs with higher success rates and lower risks. Furthermore, attributes related to effectiveness and safety, such as healthy live birth rate, pregnancy rate, and maternal complications, have the greatest impact on patient preference. A previous domestic study employing choice-based conjoint analysis to examine infertile patients’ preferences for fertility care yielded comparable outcomes, indicating that patients placed the greatest value on their physician’s attitude and treatment success.³² Domestic patients are demonstrating a growing desire for enhanced quality of care, encompassing not only advances in health technology and health outcomes, but also the treatment process itself.²⁶ Patients anticipate more comprehensive doctor-patient interactions, more transparent treatment protocols, and so forth. In contrast to the Skedgel et al study,²⁴ the Chinese respondents all preferred treatments with a higher degree of shared decision making, but with a different emphasis on cost. In the past, the effect of preference on cost has not been significant. However, this study found that, similar to other countries, patients are starting to value cost attributes and prefer lower-cost treatment options. This may be due to the fact that China’s fertility support policy is gradually improving and patients’ out-of-pocket costs are becoming lower. Despite the fact that the technology employed in In Vitro Fertilization (IVF) in China has reached the world’s advanced level, the invasive nature of this treatment means that it is not without certain side effects. In particular, the risk of neonatal complications was relatively less valued by older couples than by younger couples.²³ Differences in preferences may be due to the different respondents included in the study. Both measurement tools in this study indicated that maternal complications were slightly more important than neonatal complications. However, in the last two studies,^33,34 the importance of preventing neonatal complications was higher when respondents included men or when couples were involved in the research at the same time. It is therefore imperative to gain an understanding of how patients weigh up the benefits and risks of treatment from their perspective if improvements are to be made to the patient treatment process and the quality of healthcare services enhanced. The emphasis should be placed on the patient experience and value-based healthcare, with the objective of enabling ART to transition from a consumer healthcare role to a core healthcare technology, with a particular focus on the patient experience in their services.

Despite adherence to international guidelines and recommendations, particularly those pertaining to DEC studies, in the construction of the experimental design. However, the specific choices and assumptions varied depending on the purpose of the study, which may affect the generalizability and results of this study.³⁵ These findings align with those of previous stated preference studies,^29,36,37 which demonstrated that different instruments exhibit comparable preference patterns. In this study, the importance of cost was found to vary in terms of its relative importance to other attributes. This may be attributed to the fact that relative importance is calculated as incremental differences rather than absolute numbers.³⁶ Secondly, this may be related to the context of health decision-making. Previously, the cost of ART was borne by the patient. However, since the introduction of the Active Fertility Policy by the Provincial Health Insurance Bureau in 2023, more than a dozen provinces and regions have included ART in their health insurance to implement reimbursement. It is possible that the implementation of the policy in Shanghai during the research process may have had an impact on the patients’ preferences. Shanghai has included some therapeutic ART programs in the scope of payment under medical insurance and industrial injury insurance. For example, in the case of embryo culture programs, patients can receive up to three medical insurance reimbursements at the same medical institution. Finally, BWS-2 and DCE are compared, despite sharing the same random utility theory. However, the question format differs. BWS-2 is more inclined towards the sequential ordering of attribute levels, whereas DCE is in the form of combinations with trade-offs.³⁸ It can thus be supposed that patients may not arrive at the same decisions when making their choices. Additionally, studies have identified inconsistencies in the results obtained from the two tools. For instance, Whitty’s review revealed a lack of concordance between DCE and BWS-2 findings.³⁵ The authors proposed that the BWS-2 may yield comparable results to the DCE only when assessing individual health preferences, rather than public preferences. Accordingly, this is closely related to the purpose of the study. The study design and the type of decision being made (eg, treatment choices, prioritization) may influence the decision-making process and the resulting choices, which may be affected by a variety of factors such as the study context and the disease population. This may result in different results. Furthermore, a significant benefit of BWS-2 over DCE is that by separating attribute weights and level preferences, researchers can obtain direct insights into preferences for attributes, eliminating the need for additional mathematical calculations. Additionally, policies that alter attributes may prove more effective.³⁹

The second major finding of this study is the high degree of consistency between the BWS-2 and DCE preference results. However, contrary to expectations, an investigation into the willingness to complete the questionnaire and the difficulty of completing the questionnaire revealed that the sample population preferred to complete the DCE questions in this study. Previous studies have indicated that there may be some discrepancies in the acceptability of completing the two preference measurement tools, BWS-2 and DCE, among respondents in different sample populations.^6,40 In the present study, a positive correlation was observed between the level of education and the ratings of difficulty for the DCE questions. Patients with higher levels of education were more inclined to complete the DCE questionnaire. It is hypothesized that the difficulty in completing the questionnaire may be attributed to the cognitive burden resulting from the detachment of the choices from reality in more experimental designs and the overabundance of questions.^36,41 This may be partially related to the characteristics of interviewees. Furthermore, this study revealed that the sequence in which the choice sets were presented influenced respondents’ comprehension. Despite the identical number of choice sets, the clear explanation of different choice tasks, along with patients’ preconceived notions, may have contributed to the observed differences in comprehension. In conclusion, the results must be interpreted with caution. Ultimately, the change in utility associated with obtaining higher levels of the attributes differed between the two methods. In this study, it was found that the utility scale of DCE is relatively limited and that patients’ decision-making is not as stable Consequently, a larger sample size is required for DCE studies than for the other.⁴²

There are some limitations to this study. Firstly, it was not possible to draw causal inferences due to the inherent limitations of cross-sectional studies. Secondly, the study employed a simple sampling method that only partially reflected the patients’ views, and did not reveal the underlying reasons for their preferences. Further qualitative research, such as interviews, is necessary to gain a deeper understanding. Finally, it is imperative to exercise caution when interpreting the results of patient assessment questions, as they are subject to inherent limitations and potential biases.

Conclusions

In this study, the BWS-2 and DCE methodologies were employed to estimate attribute weights and level preferences for infertility patients’ treatment decisions. The results demonstrated that patients’ preferred decisions were stable and that the two measures yielded highly consistent results, with only slight differences in the calculated relative importance. However, patients indicated a preference for answering DCE questions and found the DCE approach to be straightforward and comprehensible. Further empirical research is required to gain a deeper understanding of the relative merits of these methods and to inform subsequent developments. The selection of an appropriate method, or the joint use of both DCE and BWS-2 methods, is essential for the purpose of informing health technology decisions, including the determination of utility values and risk-benefit ratios.