Challenges in Results Robustness of Trials with Missing Data for the Primary Endpoint: Insights from Coronary Balloon/Stent Trials

Introduction

Patients withdrawing before study completion is an explicitly anticipated potential problem for trial sponsors and investigators despite a well-designed study protocol.¹ The reasons for the patient’s withdrawal include side effects, intercurrent health problems, unpleasant treatment, lack of improvement, recovery, and external factors unrelated to the study.²

Drug-coated balloon (DCB) or drug-eluted stent (DES) have been shown to reduce the incidence of target vessel revascularization and become commonly used treatments for coronary artery disease.^3–5 For randomized controlled trials (RCTs) involving these devices, clinical endpoints require a large sample size; therefore, quantitative coronary angiography (QCA) results were usually used as surrogate endpoints for its feasibility.⁶ However, due to its invasiveness, these studies always have a high dropout rate.^7,8 Patients’ dropout causes missing data which can produce bias for the study results and may even change the conclusions.⁹

Common imputation methods such as maximum likelihood, multiple imputation, and Bayesian methods are applied to data missing at random and rely on parametric assumption. Since the mechanism of missing data is always unknown and unverifiable, sensitivity analyses for handling missing data are required by regulatory agencies to test the robustness of the study result.¹⁰ Tipping-point analysis was proposed by the US Food and Drug Administration (FDA)¹⁰ and improved by Liublinska and Rubin¹¹ which did not need to postulate any missing data mechanism or model parameters. This method enumerates all possible results caused by missing data and then conducts hypothesis tests one by one to find the value that changes the conclusion and therefore obtains the most comprehensive results in the presence of missing data.

In the current study, we hypothesize that the results of RCTs with high dropout rates might be unreliable. We will systematically sort out the RCT focusing on DCB or DES with angiographic outcomes as the primary endpoints, and use the tipping-point analysis to deal with the missing data, to test the robustness of RCT results and their influence factors.

Methods

Study Subjects and Inclusion/Exclusion Criteria

The systematic literature review was carried out under the guidance of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.¹² Four databases, PubMed, EMBASE, Cochrane Library and ClinicalTrials.gov were searched up to Nov 30, 2023, with the search terms (“drug-coated balloon” or “drug-eluting stent”) and (“Randomized controlled trial”). The citations were imported into Endnote (Endnote X9.3.1; Thomson Reuters, San Francisco, CA) for screening. Two reviewers (L.X. and Z.Y.) independently reviewed the abstracts and confirmed the eligibility through full-text assessment. A third reviewer (W.Y.) made the final decision if disagreement existed (A detailed search strategy was shown in the Supplement method).

The following inclusion criteria were used to assess the eligibility: (1) patients with coronary artery lesions (native coronary vessel lesion or in-stent restenosis); (2) RCT with DCB arm or DES arm; (3) setting the angiographic outcomes (LLL, MLD, %DS, or restenosis) as the primary endpoint; (4) with a clear study hypothesis (superiority or non-inferiority) and sample size calculation. Exclusion criteria were: (1) conference abstract, protocol or review; (2) the same clinical trial reporting long-term follow-up results and (3) H0 not rejected trials with an inferior result.

Data Collection and Definition

The following study information was extracted from the included trials, including first author, trial name, study design (noninferiority or superiority), interventions, sample size calculation parameter (power, alpha level, target standardized effect size and drop-out rate), number of patients/lesions at baseline, number of patients/lesions at follow-up, the primary endpoint definition and the observed standardized effect size.

The target/observed standardized effect size (SES) was created for the comparability of effect size among the different studies. The SES was calculated using formula (1) for continuous outcomes and (2) for binary outcomes,^13,14 where and were the mean value of the endpoints for the treatment and control, respectively. and were the sample size, and represented the standard deviation (SD) of each group, and represented the event rate of each group.

(1)

The primary endpoints in the included trials were mainly late lumen loss, percentage diameter stenosis, or binary angiographic restenosis, and a smaller value corresponds to better effectiveness. A smaller observed SES or target SES indicated that the treatment group had a better effect than the control. So, we took the negative number of SES on minimal lumen diameter (MLD) to maintain the same direction of benefit with other endpoints.

Statistical Analysis

Tipping-point analysis was used to deal with the missing data of the primary endpoint.

We used three indexes to test the result robustness of the included trials, inconsistency rate, tipping-point SES and tipping-point ratio.

The inconsistency was considered if the outcome was different from the reported (For example, a study reported a superior/non-inferior result of the treatment group, but after the imputation, the result became inconclusive). The definition of the study result is plotted in Figure 1A.

Figure 1 The definition of the study result and tipping-point analysis. (A) Forest plot with differences and 2-sided CIs showing the hypothetical outcomes from noninferiority-designed and superiority-designed studies. The blue one indicates a superior result, the grey one indicates an inconclusive result, and the Orange one indicates an inferior result. (B) An example of the result (LONG-DES V) after tipping point analysis. In the study of Lee 2014, The blue area means that the treatment group is noninferior or superior to the control. The white area indicates that the treatment group was inconclusive with the control. The orange area indicates that the treatment group is inferior to the control. Abbreviations: CI, confidence interval; RR, relative risk.

For continuous outcomes, we imputed 150 times evenly for missing values in the treatment and control arm with a range of . A total of 22,500 possible outcomes were obtained. Welch’s t-test method was then used to test the group difference. For binary outcomes, we enumerated all possible outcomes for the missing data, (from all failures to all successes) and compared the differences between groups using the chi-square test (See supplementary marital for the R code).

The inconsistency rate was calculated as follows:

The tipping point (tipping-point SES) was defined as the standardized difference of means or event rates between the treatment and control group in the missing cohort where the result changed when imputing.¹⁰ For trials with superior results, the tipping point was considered as the result became “inconclusive”. For trials with inconclusive results, two tipping points were considered where the result became “superior” or “inferior”. We chose the one which was closer to the observed SES (See supplementary marital for the detailed description). The derived indicator (tipping-point ratio) was calculated as follows:

Continuous variables were described using means (standard deviations) and median (Interquartile range) and compared by Student’s t-test or Mann–Whitney U-test for two groups, Analysis of variance or Kruskal–Wallis H-test for three groups. Categorical variables were described using frequencies (percentages) and compared by the chi-squared test or Fisher’s exact test. Multivariable generalized linear regression models were used to assess the influence factors of the inconsistency rate, tipping-point SES and tipping-point ratio. Covariates were selected in the models according to the clinical experience, including study design (superiority or noninferiority), primary endpoint (LLL or other), time to primary endpoint analysis (≤6m, 8m–9m or ≥12m), sample size, power, 1-side alpha (0.050 or 0.025), missing rate (<10%, 10–19% or ≥20%) and observed-target SES. Pearson correlation analysis was used to test the correlation among inconsistency rate, tipping-point SES and tipping-point ratio. All analyses were conducted using R version 4.0.3 and SAS version 9.4. A 2-sided p < 0.05 was considered significant.

Results

Study Selection

Of 1057 articles, 808 papers were excluded after the titles and abstracts screening. Through full texts reading, 147 studies were further deleted for the following reasons: result from the same study (n = 53), without angiographic outcomes (n = 42), without dropout (n = 8), without DCB or DES as the study arm (n = 18), without study hypothesis (n = 26), with inferior result (n = 1). Finally, 101 RCTs with 31862 patients were enrolled (Figure S1). Specific information on the included publications is available in Table S1.

Summary of the Included Trials

Among the included 101 trials, 93 trials had both a single control and treatment arm. The ISAR-DESIRE 3,¹⁵ TAX-001¹⁶ and STRESSED¹⁷ trials had 1 active treatment arm and 2 control arms, the ISAR-DESIRE,¹⁸ PAINT,¹⁹ ISAR-TEST 3,²⁰ and EVOLVE²¹ had 2 active treatment arms and 1 control arm, and Song²² conducted the trial under two conditions with two comparisons. Finally, we got 109 powered trial comparisons.

Of the 109 trial comparisons, 67 (61.5%) had a noninferiority hypothesis, and 42 (38.5%) had a superiority hypothesis. 86 (78.9%) trials used LLL as the primary endpoint.74 (66.7%) studies used 8–9 months as the time for the primary endpoint comparison (range, 6–36 months). The median sample size was 230 (IQR 148, 397) and the projected power was 80% (IQR 80%, 90%). One-sided alpha of 0.025 or 0.05 was used in 78 (71.6%) trials, and 31 (28.4%) trials, separately. The median missing rate for the primary endpoint was 16.1% (IQR 10.6%, 20.2%). And the missing rate of 24 (22.0%) trials was <10%, of 55 (50.5%) trials was 10–19%, and of 30 (27.5%) trials was >20%. The median target SES and observed SES were −0.4 (IQR −0.6, −0.4) and −0.6 (IQR −0.8, −0.3), respectively. Forty-four (40.4%) trials did not reach the target SES (target SES minus margin for noninferiority trials). The median observed-target SES was −0.1 (IQR −0.3, 0.1). 89 (81.7%) trial comparisons rejected the H₀ hypothesis, and 20 (18.3%) failed and gave inconclusive results. Except for a larger power, a larger observed SES and a larger observed-target SES, H₀ not rejected trials had no difference from H₀ rejected trials (Table 1).

Table 1 Studies Characteristics of Included Studies by Study Original Result

Tipping-Point Analysis for the Primary Endpoints

Figure 1B plots the imputation result of a typical trial (the LONG-DES V trial) after tipping-point analysis. A 51.3% inconsistency rate was shown (the white area plus the yellow area). The tipping-point SES and tipping-point ratio were −0.10 and 0.48, respectively. The LONG-DES V trial assumed that the 9m in-segment LLL of the biolimus A9-eluting stent were not inferior to those of the durable PtCr-EES and observed their noninferiority. However, this trial had a high drop-out rate of the angiographic outcomes, 33.1% for the treatment arm, and 35.7% for the control arm. And its observed effect size did not meet the target effect size.

After conducting a tipping-point analysis, for 20 H₀ not rejected trial comparisons, the median of the inconsistency rate and tipping-point SES was 56.9% (IQR 37.8%, 68.9%) and −0.5 (IQR −0.7, 0.6), separately. The tipping-point ratio is not available because H₀ not rejected trial comparisons always have a small observed SES, which causes the tipping-point ratio extremely large. For 89 H₀ rejected trial comparisons, the median of the inconsistency rate, tipping-point SES, and tipping-point ratio was 23.2% (IQR 1.8%, 38.4%), 1.3 (IQR 0.5, 3.7) and −2.1 (IQR −4.3, −0.8), separately. H0 rejected comparisons were significantly more robust than H0 not rejected comparisons. For 89 H₀ rejected trial comparisons, the study result became less robust with increasing missing rate (Table 2).

Table 2 Tipping-Point Analysis for the Primary Endpoints

Figure 2A–C plots the imputation results after tipping-point analysis in the H₀ rejected trial comparisons with a missing rate of ≥10%. In 68 trial comparations, the median of the inconsistency rate, tipping-point SES, and tipping-point ratio was 32.2% (IQR 19.7%, 45.4%), 0.90 (IQR 0.17, 1.79) and −1.53 (IQR −2.43, −0.39), separately.

Figure 2 Tipping-point analysis for the primary endpoints of the H0 rejected comparisons with a missing rate of no less than 10%. (A) The proportion of the H0 rejected comparisons with a missing rate of no less than 10% after tipping point analysis (the inconsistency rate was the grey area plus the yellow area). (B) The tipping-point SES. (C) The tipping-point ratio. Abbreviation: SES, standardized effect size.

Association Between Study Characteristics and Inconsistency Rate, Tipping-Point SES and Tipping-Point Ratio

The Pearson correlation analysis showed the association of the missing rate with the inconsistency rate, tipping-point SES and tipping-point ratio (Figure S2). The inconsistency rate, tipping-point SES and tipping-point ratio were correlated with each other (Figure S3).

The multivariable generalized linear regression model demonstrated that missing rate (<10%, 10–19%, ≥20%) and observed-target SES were associated with inconsistency rate, tipping-point SES and tipping-point ratio for the H₀ rejected studies after adjusting other study characteristics.

Compared with missing rate <10%, the missing rate at 10–19% increased 16.5% of the inconsistency rate (95% CI, 0.194 to 0.309 p < 0.001), decreased −5.236 of the tipping-point SES (95% CI, −6.202 to −4.270, p < 0.001), increased 6.291 of the tipping-point ratio (95% CI, 4.954 to 7.627, p < 0.001), and missing rate ≥20% increased 25.1% of the inconsistency rate (95% CI, 0.194 to 0.309, p < 0.001), decreased −6.187 of the tipping-point SES (95% CI, −7.245 to −5.128, p < 0.001), increased 7.747 of the tipping-point ratio (95% CI, 6.284 to 9.211, p < 0.001).

As for observed-target SES, 32.7% increased in inconsistency rate per unit increased in observed-target SES (95% CI, 0.264 to 0.390, p < 0.001), 4.559 decreased in tipping-point SES per unit increased in observed-target SES (95% CI, −5.699 to −3.419, p < 0.001), 2.007 increased in tipping-point ratio per unit increased in observed-target SES (95% CI, 0.430 to 3.584, p < 0.001) (Table 3). The results were stable in the sensitivity analysis of all 109 trial comparations (Table S2).

Table 3 Multivariable Generalized Linear Regression Analysis of Study Characteristics in the H0 Rejected Comparisons

Discussion

Main Findings

This is the first study, to our knowledge evaluating the robustness of the coronary drug-eluted stent/balloon study results using tipping-point analysis. We used three indicators to assess the robustness, inconsistency rate, tipping-point SES and tipping-point ratio. We found that (1) for trials with an angiographic primary endpoint, only 22.0% of trials with a missing rate of less than 10%; (2) the conclusions of some clinical trials may change after tipping-point analysis. (3) a higher missing rate and a larger observed-target SES were associated with a more unreliable result (a higher inconsistency rate, a lower tipping-point SES and a higher tipping-point ratio).

Except for the first proposed tipping-point SES,¹⁰ we posed another two indexes (inconsistency rate and tipping-point ratio) to test the robustness of the results. They all had a good correlation with each other except for trials with high robustness. Since mean±3SD was considered when filling, for studies with an inconsistency rate of 0, tipping-point SES and tipping-point ratio could distinguish these studies by filling in values beyond this range. However, there is little need to distinguish these situations. The inconsistency rate could more intuitively display all results after filling than the tipping-point SES. As for the tipping-point ratio, which compares the tipping-point SES with the observed SES, the value would help to make a judgment of whether such a difference was implausibly unfavourable. For example, if a tipping-point ratio was less than −1, it would indicate a robust result (because the tipping-point SES was less likely to be the opposite number of the observed SES, or even farther). These indexes all have their advantages, Evaluation together allows for a more comprehensive presentation of the results.

Relation to Other Literature

Our findings were primarily based on the H₀ rejected trial comparisons. H₀ not rejected comparisons had a significantly not robust result compared with H₀ rejected comparisons and were considered in the sensitivity analysis. For H₀ rejected trial comparisons with missing rate ≥ 10%, the median of inconsistency rate, tipping-point SES and tipping-point ratio was 32.2% (IQR 19.7%, 45.4%), 0.90 (IQR 0.17, 1.79) and −1.53 (IQR −2.43, −0.39), separately. The results emphasize a sensitivity analysis for trials with missing data. For the clinical trials, some patients might not be missing at random, and the reason included poor curative effect, adverse reactions, etc. In addition, it is recommended by National Research Council (US) Panel on Handling Missing Data in Clinical Trials that not only a primary analysis that assumes data are missing at random, but also sensitivity analyses which allow for data not missing at random are needed.²³ However, a review of 77 RCTs at the top medical journals showed that only 27 (35%) trials performed a sensitivity analysis, among them 10 (37%) trials weakened the assumptions regarding missing data from their primary analysis.²⁴

In our study, we demonstrated that the more missing, the less robust the findings. For H₀ rejected trials with a missing rate of ≥20%, the median inconsistency rate was even 37.3% with an interquartile range from 28.5% to 47.5%. As described by Dong et al,²⁵ they used a listwise deletion method, multiple imputations, full information maximum likelihood, and expectation-maximization algorithm to deal with a complete real-world data set, and indicated that any level of missing data was a potential risk to the validity of a trial, and missing over 20% would give a significant risk. A systematic review of RCTs testing palliative interventions also demonstrated that missing data reduced the power and potentially became a source of bias for trials.²⁶

We also found that the observed-target SES was the stable influence factor of the inconsistency rate, tipping-point SES and tipping-point ratio instead of the study power, alpha level or sample size. The larger the observed-target SES (the observed SES not reached and is worse than the target SES), the more unstable the study result. A large observed-target SES indicates a poor observed SES, or an inflated target SES. An excessive target SES corresponds to arbitrarily large margins or overestimated mean values/event rates for non-inferiority designed trials and an inflated expected difference for superiority designed trials. Previous studies have demonstrated that the target SES always determined the sample size and the study costs, hence, it was tempting for researchers to inflate the target SES to reduce the sample size, which would cause an inappropriate trial design and introduce bias.^27–30

Take ISAR-DESIRE 4 trial³¹ for example, it reported that neointimal modification with scoring balloon pre-dilation before a drug-coated balloon showed superior results of the primary endpoint (in-segment percentage diameter stenosis) than a drug-coated balloon. Its observed SES was −0.281 (treatment vs control: 35.0±16.8 vs 40.4±21.4), which did not reach the target SES (−0.398, treatment vs control: 26.25±22 vs 35±22). Although the authors used a Student’s t-test to calculate the difference and found the superiority, the 95% confidence intervals of the primary endpoint of the treatment and the control were overlapped (treatment, 35.0, 95% CI 31.8 to 38.2; control, 40.4, 95% CI 36.2 to 44.6). Thus, the result was unstable and it was not easy to pass the sensitivity analysis.

For clinicians, it is important to use clinical endpoints, such as target vessel failure, to assess the safety and effectiveness of new devices.^32,33 When surrogate endpoints are used due to feasibility, missing data for the primary endpoint should be considered. Clinicians could use our R codes provided in the supplementary material to calculate the inconsistency rate of their research, thereby providing more rigorous evidence for their results.

Strengths and Limitations

This study has some limitations. Firstly, our study was limited to coronary drug-eluting stent or drug-coated balloon trials using angiographic outcomes as the primary endpoint. This kind of trial always has a higher dropout rate due to the invasive examination. However, drop-out is a common problem for RCTs, and the tipping-point analysis is applicable for other studies, not only RCT but also cohort, case-control and cross-sectional studies.^34–39 Additionally, we did not have the original datasets of the eligible trials, so we filled them based on the normal distribution according to the reported mean and SD. Third, although we conducted a comprehensive search, some unpublished articles were not included due to negative results. To avoid publication bias, we also included H₀ not rejected trials. According to the analysis of the characteristics of included studies, H₀ rejected and H₀ not rejected had similar study characteristics. We consider that the publication bias may have little impact on the results. Last but not least, we only used tipping-point analysis to fill in the missing data, and the indexes we proposed still need further sensitivity and specificity tests.

Conclusions

Rigorous sensitivity analysis (such as tipping-point analysis) adds more difficulty to the trials, which may restrict some new technologies entering the market. Having said this, the changes in results after filling in the missing data presented from the stent/balloon trials should raise concerns about the high dropout rate and inflated target effect size and their consequences. We emphasize a reliability evaluation of the results for further clinical trials with missing data for the primary endpoint.