Validation of Mortality Data Sources Compared to the National Death Index in the Healthcare Integrated Research Database

Introduction

Generating evidence using real-world evidence depends heavily on the underlying quality and information diversity data sources. Thus, using such data, eg, administrative claims or electronic medical records, which was not explicitly designed for research is challenging. Therefore, important information, eg, mortality data, is often only available with additional efforts, such as linkage. Several mortality data sources are utilized for research purposes, such as the United States (US) National Death Index (NDI) contains over 100 million death records in the US since 1979.¹ NDI is the gold standard of death ascertainment with high sensitivity and specificity available for research with the ability to match other databases.^1–6

NDI is a robust resource due to its completeness; however, NDI has a data lag of up to two years, which limits its utility for research projects. In addition, obtaining NDI linkage is impossible for data sources unable to share personal identifiers and can be operationally intensive, particularly for large studies.^1–5,7 Other sources of death date and cause are available, eg, the Social Security Administration’s Death Master File (DMF) from the National Technical Information Service Social Security Administration (NTIS).⁸ However, a policy change resulted in rigorous and limited disclosure of the state death record data with limited exceptions. Historically, DMF contained 89 million records, and the policy change resulted in the removal of approximately 4.2 million records, lowered the annual records, and decreased the addition of new records by 1 million annually.⁷ Similarly, the Center for Medicare and Medicaid Services Vital Status files have also been used. However, the limitation of only including Medicare members makes it suitable as a stand-alone source for only a portion of the population.

Hospital discharge claims are another source of mortality data that captures in-hospital deaths precisely and does not capture other death cases;^9–12 however, the decline in the number of deaths occurring during hospitalization¹³ makes this data source unable to capture an increasing proportion of deaths. Obituaries have been used for several decades as a mortality data source, for example, the field of occupational epidemiology has utilized this source, and it has been reported to have a high sensitivity.^14,15 With obituaries becoming available online, their use increased as one of the sources for ascertaining mortality data and was shown to be a valid and reliable source,^16,17 though it may be limited in sensitivity. We hypothesized that combining each of these six mortality data sources in a composite algorithm could provide a sensitive and specific measure for mortality that could be a reliable approach when NDI linkage is not possible. The main objective of our study is to assess the validity and completeness of six mortality data sources and a composite algorithm within the Healthcare Integrated Research Database (HIRD) compared to NDI data, ie, the gold standard, in the advanced-stage cancer patient population from 2010 to 2018.¹⁸

Methods

This study was performed using data from the HIRD, which contains medical and pharmacy claims data from health plan members across the US. The HIRD is approximately 88 million unique individuals with comprehensive medical and pharmacy claims since 2006, representing the commercial and Medicare advantage-insured US population. This is a secondary analysis of de-identified data from a cohort of individuals in the HIRD diagnosed with advanced cancer from 2010 to 2018. More recent data was not available from the NDI at the time of the analysis.

Several sources of death status (Yes/No) and death date (if available) were available, including:

• DMF: National Technical Information Service, Social Security Administration (NTIS) known as the Social Security Death Master File¹⁹
• Discharge: Hospital discharge status on medical claims (Discharge status codes, 20, 40, 41, 42) available through medical claims data from Elevance Health insurance plans
• Disenrollment: Health plan reason-for-disenrollment data is available through administrative data from Elevance Health insurance plans
• CMS: Center for Medicare and Medicaid Services
• U.M.: Utilization of management data
• Obituary: Online obituary data obtained through natural language processing^17,20,21

Dates obtained from a subset of online sources included only the month and year of death and were assumed to occur on the 15th of the month. Death data from these six available sources in the HIRD are updated monthly. NDI data were obtained for this study. Patients identified with death in 2018 were matched to NDI early-release data, which may need to be completed. The study protocol was reviewed and approved by the WCG Institutional Review Board (IRB) and the National Center for Health Statistics (NCHS) overseeing the NDI. The time trends in the completeness and validity of the mortality sources compared to the NDI are described in a separate manuscript.²²

Data Analysis

For each of the six death sources, the difference in days between the death dates from a death source and the NDI was calculated among members with a death date.

In addition, the mean and standard deviation (SD) of the difference in dates and the proportion of members in different categories of difference between the dates from death source and NDI was calculated.

The mean, standard deviation, and proportion of participants within categories of sex, age (<18, 18–44, 45–64, ≥65 years), geographic region (Northeast, Midwest, South, West), and plan type (commercial, Medicaid advantage) for all and by death status (death source only, NDI death only, death source, and NDI death) were calculated.

Validation metrics were calculated to assess the performance of internally available death sources.^23,24

• Se = proportion of NDI death with an identifiable death in a death source (True positives/(True positives + false negatives))
• Sp = proportion of patients without an NDI death and no identifiable death in a death source (True negatives/(True negatives + false positives))
• PPV = proportion of patients with an identifiable death in a death source and NDI death (True positives/(True positives + false positives))
• NPV = proportion of patients without death in the death source and no NDI death (True negatives/(True negatives + false negatives))

The 95% confidence intervals (95% CI) were calculated using the binomial method. Death dates from death sources were a match with an NDI if they occurred 60 days before or 30 days after the NDI date.

We created a composite death data algorithm that assigned a death if a death was recorded in any of the six death sources. The death date of the composite algorithm was assigned using the following hierarchy of death sources, based on a) the highest proportion of internally sourced dates with NDI dates for members who had dates from death source and NDI, b) the proportion of members with internally sourced dates lacking an NDI date, and c) the proportion of members with an NDI date where the internally sourced date occurred within ±30 days. The same methods were followed for each death source. The hierarchy was defined as the DMF > Discharge > Disenrollment > CMS > U.M. > Online Obituary. If death was recorded in one data source (eg discharge) and not the others, the individual was recorded as a death and the death date was defined as the date from the source. When a death was recorded in multiple data sources, the death date was defined as the date recorded in the source highest in the hierarchy.

To assess the impact of the gap between the internally sourced data and NDI, different allowable gaps were used for sensitivity analysis: ±60, ±30, ±15, ±7 days of the NDI date. Analyses were performed using SAS^® Enterprise Guide^® 8.3; Cary, NC: SAS Institute Inc.

Results

Among a cohort of advanced cancer patients from 2010 to 2018, 27,396 (67.3%) had at least one death date. The mean age was 67.2 years with an SD of 13.5 years; 54.3% were women, and 56.1% were commercially insured (Table 1). Compared to patients without a death, those who had a death date in at least one death source were older (mean age 69.1 vs 63.1 years) and more likely to be male (49.7% vs 37.4%) and have a Medicare plan (49.8% vs 32.5%). Among all patients with death in NDI and death source, the mean difference in days was 1.83 (95% CI = 1.25–2.40). Death dates were available in 53.4% of obituaries, compared to 33.2% and 35.2% of discharge and DMF data sources (data is not shown).

Table 1 Baseline Characteristics for the Study Population

Table 2 shows the distribution of the various death gaps between the composite mortality, six internal death sources, and NDI death date. For composite mortality, 84.7% of captured death dates were precisely matched to the NDI death date. In contrast, among all six sources, CMS and U.M. had the highest exact match death date, 99.3% and 91.5%, respectively, followed by DMF and obituary, 86.1% and 70.1%, respectively. For discharge, 54.3% were an exact match, and 27.8% were one to seven days after the NDI date. For deaths obtained from disenrollment data, only 38.8% were an exact match, and 31.7% were reported one to fifteen days earlier than the NDI death date.

Table 2 The Difference in Days Between Each Death Date Source and the NDI Death Date

Table 3 shows the performance of the composite mortality database and the individual death sources. The composite mortality had sensitivity at 88.9% (95% CI = 88.5%, 89.3%), specificity at 89.1% (95% CI = 88.6%, 89.6%), PPV with 93.4% (95% CI = 93.1%, 93.7%), and NPV at 82.3% (95% CI = 81.7%, 82.9%). When comparing sources, their sensitivity and NPV varied dramatically; obituary data had the highest sensitivity and NPV, 53.4% (95% CI = 52.8%, 54.0%) and 54.0% (95% CI = 53.4%, 54.6%), respectively. While U.M. and disenrollment had the lowest sensitivity, 7.4% (95% CI = 7.1%, 7.8%) and 14.9% (95% CI = 14.4%, 15.3%), respectively, and the lowest NPV, 38.4% (95% CI = 37.9%, 38.9%), and 40.2% (95% CI = 39.7%, 40.7%), respectively. Regarding specificity, the U.M. and disenrollment death databases had the highest specificity at 99.3% (95% CI = 99.2%, 99.4%) and 98.8% (95% CI = 98.6%, 99.0%), respectively. In addition, DMF and discharge had the highest PPV 96.4% (95% CI = 96.0%, 96.7%) and 95.4% (95% CI = 95.4%, 96.2%), respectively, while the obituary data had lower specificity and PPV 94.3% (95% CI = 93.9%, 94.7%) and 94.2% (95% CI = 93.8%, 94.6%), respectively.

Table 3 Validation Performance Metrics of Death Data Sources Compared to the NDI

Supplemental Table 1 shows the performance of the death date gap segments between the composite mortality database and NDI. The highest sensitivity and NPV were observed for ±60 days gap 89.0% (95% CI = 88.6%, 89.4%) and 82.4% (95% CI = 81.8%, 83.0%), respectively; and the lowest sensitivity and NPV were observed among the ±7 days gap, 82.4% (95% CI = 81.4%, 82.8%) and 74.5% (95% CI = 73.9%, 75.2%), respectively. While specificity and PPV remained consistent across all death day gaps, 89.1% (95% CI = 88.6%, 89.6%) and 93.4% (95% CI = 93.1%, 93.7%), respectively.

Discussion

In our secondary analysis of cancer patients with an established NDI linkage between January 2010 and December 2018, the composite mortality database was highly sensitive and accurate in identifying deaths, 88.8% (95% CI = 88.4%, 89.2%) and 93.3% (95% CI = 93.0%, 93.6%), respectively, compared to NDI death dates. When we compared the composite mortality database to the individual death data sources, all had greater than PPV at or greater than 94.0%; however, each of their sensitivities was less than 54%. Thus, a composite of death data sources of the DMF, obituary, discharge, disenrollment, CMS, and U.M. could provide an accurate and more sensitive death assessment than a single source.

Mortality ascertainment is challenging because it is only partially ascertained directly in administrative claims or EMR. The two most populated death data sources were online obituary and DMF, but both still had limited sensitivities on their own. Our study suggests that using several mortality sources, including linkages to other sources, is more robust and accurate for the death assessment events than relying on a single death source alone. Historically, the DMF death database was previously considered a reliable and complete source of death information. However, with the 2011 policy change surrounding sharing state death registries, its sensitivity declined over time. Many reports noted that DMF has become unreliable as a single source of death data for research and that there is a need to find additional sources.^3,8

As previously reported, NDI is considered the gold standard for mortality ascertainment in the US^1–6 However, this source is time-lagged, and privacy restrictions of some populations prevent NDI linkage. Historically, occupational epidemiology utilized obituaries from industry-specific journals as a source of mortality.^14,15 With advanced technology, such as natural language processing, the use of online obituaries for research is surging.^16,18,20 However, other studies suggested that the under-capture of women and racial and ethnic minorities is a limitation in addition to its limited availability before 2015.²⁰ Even though NDI is the gold standard for death ascertainment, it has several limitations, such as using probabilistic matching,¹⁸ which could involve errors in matching the right individuals.

In addition, the time lag of NDI could create hurdles in utilizing the NDI data. For our study, the ascertainment of death dates in 2018 was impacted by the availability of NDI data as the early-release NDI death data was the only available data, which likely impacted the sensitivity of the composite death database. Other limitations of the NDI data include its privacy restriction on some populations. In addition, the performance of the NDI varies in different populations, including women and minorities, which has been noted for the death sources.¹⁸

Though our study included a large population of patients with linked NDI data, one limitation is its population type: commercially and Medicare advantage-insured, advanced cancer older adults in the US. Thus, the performance of the composite mortality database and the various death databases might differ in other populations. This study is also limited in that it does not provide death data after 2018, so further research in assessing the performance of the composite mortality algorithm, especially in the COVID-19 era, is warranted.

Conclusion

Our study demonstrates that creating an algorithm combining DMF with additional death sources, such as discharge, disenrollment, and online obituary, provides an accurate and more sensitive assessment of death outcomes among advanced cancer subjects. The algorithm offers high sensitivity and accuracy compared to NDI and suggests that using a composite algorithm is particularly important when estimating the absolute risk of death in real-world database studies. Further research is needed to understand the performance of this algorithm in other populations and the COVID-19 era.