Validity of the Updated Rx-Risk Index as a Disease Identification and Risk-Adjustment Tool for Use in Observational Health Studies

Introduction

Health claims data are increasingly available and have contributed to studies of healthcare safety and quality as well as health surveillance, policy planning and evaluation.^1,2 However, reliable clinical records on diagnosis information are not always available in these data sources. When available, it is often derived from hospital records, which are inherently biased toward sicker populations and only available for those that have a hospital visit. Therefore, medicines use data can serve as a valuable proxy for identifying individuals with certain diseases as many medicines are used for specific diseases. Generating comorbidity scores based on prescription medicine use can help to estimate the overall burden of health conditions, to support strategies for confounding risk adjustment.¹

The Rx-Risk Index is an example of a tool that identifies specific conditions for which medicines can be used to treat. The premise is that patients, if using specific medicines to treat a condition, can be considered to have that condition. The Rx-Risk Index was first developed in 2003, called the Rx-Risk-V Index,³ which determined an individual’s current health condition based on their prescription medicine dispensing over a discrete period.³ The index only includes health conditions for which a medicine could be prescribed based on conditions identified from the Chronic Disease Score (CDS).⁴

Older adults often experience multiple health conditions, which can greatly impact overall health. Improving tools that can help to measure overall health can help to more appropriately understand health outcomes in research and policy. The validity of the original Rx-Risk-V Index in identifying specific diseases was evaluated in an older Australian population, where the Index demonstrated high specificity but low to moderate sensitivity compared to self-reported conditions.⁵ Additionally, when the Index was used to calculate a weighted sum of identified health conditions as an overall score, it showed a significant relationship with self-rated health (p < 0.001). The study also found that the Rx-Risk-V Index was strong predictor of mortality, with hazard ratios of 1.079 (95% CI 1.045–1.114).⁵

In 2018, to improve the usability of the index across different health systems, the index was mapped to the World Health Organisation’s (WHO) Anatomical Therapeutic Chemical (ATC) classification system.⁶ This first ATC-coded Rx-Risk Index version consisted of 46 health conditions mapped to the ATC standardised international medicines classification system by consensus between two pharmacists. While the study found that the index was highly predictive of one-year mortality in older population (c-statistic = 0.833, 95% confidence interval (CI): 0.829–0.837), it was not validated for disease identification.⁷

Due to continual advances in disease management and changes in prescribing practice, the Rx-Risk Index requires regular update and re-validation. In this study, we built upon the original ATC-coded version of the Rx-Risk Index by updating it to reflect current medicine management practices and re-validating its performance. Specifically, we aimed to 1) evaluate the accuracy of the Rx-Risk Index in identifying health conditions by determining the agreement between conditions identified using the Rx-Risk Index compared to patient self-reported conditions and hospital diagnoses and 2) assess the effectiveness of the Rx-Risk Index as a measure of overall health by determining the validity of the Rx-Risk Index Score in predicting one-year mortality.

Materials and Methods

Data Sources

Two Australian national linked datasets were used: Person-Level Integrated Data Asset (PLIDA) and National Integrated Health Services Information (NIHSI), which has transitioned to the National Health Data Hub (NHDH) datasets.^8,9 Both the PLIDA and NHDH datasets provide comprehensive and nationally representative insights into health service utilisation and health outcomes across the Australian population.

PLIDA integrates datasets from health (excluding hospitalisation), education, social services, income and taxation, employment, and population surveys including the National Health Survey (https://www.abs.gov.au/about/data-services/data-integration/integrated-data/multi-agency-data-integration-project-PLIDA). The PLIDA datasets used in this analysis were the National Health Survey (NHS) 2017–18 and Pharmaceutical Benefits Scheme (PBS) prescription claims data from the 2017–18 financial year (Australian financial year is between 1 July and 30 June the following year). NHDH is a national linked health administrative dataset that includes data on hospitalisation encounters, primary healthcare visits funded under the Medicare Benefits Schedule (MBS), medicines dispensed on the national PBS, Residential Aged Care services data, and National Death Index (NDI)(https://www.aihw.gov.au/reports-data/nhdh/about). The NHDH dataset used in this analysis was the NHDH version 0.5 which was the available version at the time of analysis. NHDH 0.5 includes data from 2010–11 to 2016–17 financial years.

Study Population

Individuals in the PLIDA dataset were eligible for inclusion if they were aged 65–100 in the 2017–18 NHS record and had a unique PLIDA linkage identifier, which enabled linkage between NHS and PBS claims data. Linkage rates were consistently high across older age groups, with 93.9% for those aged 65–74, 95.5% for those aged 75–84, and 96.5% for those aged 85 and over, ensuring comprehensive representation of the older population.⁸ The Rx-Risk Index was applied to PBS claims data from the 2017–18 financial year.

For the NHDH dataset, we included individuals aged 65–100 who were alive as of 30 June 2016. To ensure comprehensive healthcare utilisation data, these individuals were required to have at least one MBS or PBS claim between 1 July 2012 and 30 June 2015 and at least one hospitalisation record from 1 July 2010 to 30 June 2016. The inclusion of at least one hospitalisation record allowed the use of diagnosis codes from admitted patient care data to validate disease diagnoses identified through PBS claims. The Rx-Risk Index was applied to PBS claims data from the 2015–16 financial year.

Rx-Risk Index Update and Application

The updated Rx-Risk Index now includes a total of 52 conditions and incorporates new medicines for existing conditions as applicable. Nine new conditions have been added: constipation, cystic fibrosis, inflammatory autoimmune diseases, macular degeneration, multiple sclerosis, myasthenia gravis, opioid dependence, pain (paracetamol), and pain (gabapentinoid). Additionally, two conditions have been removed: malnutrition and ischemic heart disease (hypertension-specific), with ischemic heart disease (hypertension-specific) merged into the hypertension category (Table 1). This update was conducted through consensus among three pharmacists.

Table 1 Updated Rx-Risk Index

Rx-Risk conditions were identified if there was at least one dispensing of a medicine mapped to the Rx-Risk condition category within a 12-month period (2017–18 for PLIDA dataset and 2015–16 for NHDH dataset). Additionally, a sensitivity analysis was conducted requiring two or more dispensings of a medicine mapped to the Rx-Risk condition category within the two different time periods (3 and 12 months) to indicate the presence of that condition.

Reference Standard

To determine the validity of the Rx-Risk Index disease classification, we compared them to self-reported conditions from the National Health Survey (NHS) in PLIDA and hospital diagnoses (ICD-10-AM codes) in NHDH. Out of the 52 Rx-Risk conditions, we were able to map 19 conditions to the self-reported conditions in PLIDA and to ICD-10 diagnosis codes in NHDH (Table 2). Hospitalisations with relevant ICD-10 codes in either primary or secondary diagnoses were used to indicate the presence of the condition.

Table 2 Rx-Risk Conditions and Corresponding Self-Reported Conditions as Recorded in the Person-Level Integrated Data Asset (PLIDA) Dataset and in the National Health Data Hub (NHDH) Hospital Diagnosis Code (ICD-10-AM)

The NHDH dataset was used for the predictive validity analysis, with death in the 2016–17 financial year as the outcome. The Rx-Risk Index, applied to PBS claims data from the 2015–16 financial year, was used as the predictor variable with the index date set within this period.

Statistical Methods for Agreement

Agreement between conditions identified by the Rx-Risk Index and the self-reported conditions or hospital diagnosis codes was assessed using Cohen’s kappa statistic and F1 score. Cohen’s kappa measures the level of agreement between two raters (classifiers) in classifying binary categories. The F1 score is a harmonic mean of positive predictive value (precision) and sensitivity (recall); it describes the proportion of correct positive predictions. Both kappa and F1 scores range from 0 (worse) to 1 (best). The kappa value can be interpreted as follows: no agreement (≤0), none to slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect agreement (0.81–1.00).

In addition to these measures of agreement, we evaluated sensitivity and specificity as measures of the validity of the Rx-Risk Index in identifying people with and without the conditions. Sensitivity and specificity assess the ability of the Rx-Risk Index to correctly classify individuals as having or not having a condition based on medicines dispensed. We also calculated positive and negative predictive values (PPV and NPV) to provide practical information about the likelihood of a condition being present given a positive or negative Rx-Risk result. However, it is important to note that PPV and NPV are influenced by the prevalence of the condition in the study population and are not measures of the validity of the Rx-Risk Index. The values for sensitivity, specificity, and predictive values were interpreted as follows:

• Sensitivity (recall): proportion of people with the condition as defined by hospital diagnosis or self-reported condition who had medicines dispensed in the corresponding Rx-Risk condition
• Specificity: proportion of people without the condition as defined by hospital diagnosis or self-reported condition who did not have medicines dispensed in the corresponding Rx-Risk condition
• Positive predictive value (PPV/precision): proportion of people with medicines dispensed in a Rx-Risk condition who also have the condition as defined by hospital diagnosis or self-reported condition
• Negative predictive value (NPV): proportion of people without medicines dispensed in a Rx-Risk condition who also did not have the condition as defined by hospital diagnosis or self-reported condition

Statistical Methods for Predictive Validity

A baseline logistic regression model was performed with one-year mortality as the outcome and age and sex as the predictors. The Rx-Risk condition scores and individual Rx-Risk conditions were added to the baseline model separately. In this analysis, three forms of the Rx-Risk index were generated: an unweighted score, a weighted score, and an individual condition with an indicator variable indicating the presence or absence of each of the 52 Rx-Risk categories. Cystic fibrosis and Hepatitis B were removed as categories in the calculation of the RxRisk score for predictive validity analysis as the number of individuals with these conditions was less than 10. Hence, the Rx-Risk score was calculated as the count of the number of different Rx-Risk conditions a person had been treated for with a possible score ranging from 0 to 50.

The overall goodness of fit for each model was compared with the baseline model using the Akaike information criterion (AIC).¹⁰ The difference between the AIC values of two models must be greater than 10 for one model to be considered superior to the other. Model discrimination was compared based on c-statistic which can range from 0 to 1, with 1 indicating perfect prediction and 0.5 indicating chance predictions. A c-statistic between 0.8 and 0.9 was considered as excellent and between 0.7 and 0.8 acceptable.¹¹ Using 250 bootstrap samples, 95% CIs were generated for the c-statistic.¹²

The weighted Rx-Risk score was calculated by adding the Rx-Risk conditions to a binary logistic regression model with 1-year mortality as the outcome, adjusting for age and gender. From this model, each Rx-Risk condition was weighted according to its statistical significance and the magnitude of its odds ratio.¹³ The weighted Rx-Risk score for an individual was then derived as the sum of the weighted Rx-Risk conditions.

To validate the weights calculated for the Rx-Risk conditions, 10-fold cross validation was used. This subset the NHDH cohort using random sampling without replacement into 10 equal folds, where each fold is 10% of the data. For cross-validation, one-fold was chosen as the testing set and the remaining 9 folds were used as the training set. The training set was used to calculate the condition weights and then used to calculate a weighted Rx-Risk score for patients in training and testing sets. A binary logistic regression model with one-year mortality including age, sex and weighted Rx-Risk score was built separately for the training and testing sets and the c-statistic recorded. This process was repeated 10 times until each fold was used as a testing set once. This process calculated 20 c-statistics, 10 each on the training and testing set. The average c-statistic was recorded for each set. Statistical analyses were undertaken using SAS version 9.4 (SAS Institute, Cary, NC, US) and R version 4.1.0.

Results

The PLIDA analytic cohort included 3,959 NHS participants, with an average age of 74.2 years (Standard Deviation (SD): 6.6) and 56% females. The NHDH analytic cohort consisted of 157,709 patients, with an average age of 74.6 years (SD: 7.6) and 53% females. Overall, the prevalence of each of the 19 selected conditions was higher when identified as self-reported conditions compared to hospital diagnosis codes, except for heart failure, diabetes, ischemic heart disease (angina), and dementia (Table 3). The largest discrepancy was observed in the prevalence of hyperlipidaemia, with the Rx-Risk Index identifying approximately 50%, self-reported data showing 26.8%, and hospital diagnosis codes identifying only 1.2% (Table 3).

Table 3 Prevalence and Kappa Statistics of Rx-Risk Conditions Between Self-Report and Hospital Diagnosis Data

We found moderate to almost perfect agreement between self-reported conditions and Rx-Risk Index conditions for depression, hypertension, chronic airways disease, diabetes, osteoporosis/Paget’s, gout, and glaucoma (kappa 0.51–0.81). Substantial agreement between hospital diagnosis records and Rx-Risk was evident only for diabetes (kappa 0.73) (Table 3). The sensitivity of the Rx-Risk Index for identifying heart failure was 72% when assessed against hospital diagnosis but only 52% when compared to self-report.

When using self-reported conditions as reference standard, sensitivity was greater than 75% for hyperlipidemia, hypertension, epilepsy, diabetes, glaucoma, and chronic airways disease, suggesting that of those who self-reported these conditions, a high proportion were dispensed medicines identified by the Rx-Risk Index for that condition. Renal failure, bipolar disorder, psoriasis, urinary incontinence, malignancies and migraine, however, all had low sensitivity, suggesting that of those who self-reported these conditions only a small proportion were dispensed medicines identified by the Rx-Risk Index for that condition.

Specificity was generally high, above 90%, for most conditions identified using Rx-Risk compared to self-reported conditions, suggesting that of those who did not self-report these conditions a high proportion were not dispensed medicines identified by the Rx-Risk index for that condition. The Rx-Risk Index had low specificity for identifying hyperlipidemia and heart failure (65% and 54% respectively) despite their high sensitivity. This suggests that a high proportion of patients who do not self-report having these conditions are taking medicines for hyperlipidemia and heart failure.

The Rx-Risk Index conditions for diabetes, gout, and renal failure had high PPV (>80%) (Figure 1A), indicating that most patients receiving medications for these conditions also self-reported having them. Migraine and macular degeneration had reasonable PPV (>70%). NPV was high across all conditions (>80%) (Figure 1B), suggesting that over 80% of patients without Rx-Risk identified condition-specific medicines did not self-report having the condition. Conversely, approximately 20% of patients who reported having a condition did not have any record of condition-specific medicines dispensed. This discrepancy could be due to several factors, including non-pharmacological management, the use of over-the-counter medicines or medicines not subsidised under the PBS, alternatively this could be due to potential errors in self-reporting.

Figure 1 Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of Rx-Risk Index against self-reported conditions and ICD10-AM. (A) Sensitivity and PPV of the Rx-Risk Index compared to self-reported conditions (B). Specificity and NPV of the Rx-Risk Index compared to self-reported conditions (C). Sensitivity and PPV of the Rx-Risk Index compared to ICD10-AM codes (D). Specificity and NPV of the Rx-Risk Index compared to ICD10-AM codes.

When using hospital diagnoses as reference standard, the Rx-Risk identified conditions had on average low PPV, with only diabetes and renal failure showing high positive predictive values of around 80% (Figure 1C). Negative predictive value of the Rx-Risk Index was high for all conditions (Figure 1D).

The sensitivity analysis for disease identification agreement found that overall, the prevalence of a specific condition was inversely related to the number of dispensings required and the time window duration to define a condition (Supplementary Tables 1 and 2).

The baseline model, with only age and sex only, predicted one-year mortality moderately well (c-statistic = 0.780). The addition of the unweighted Rx-Risk score to the model increased the performance of the model (c-statistic = 0.820). The addition of indicator variables for each conditions performed best (c-statistic 0.856) (Table 4). Results of the 10-fold cross validation were consistent with the primary analysis. The average c-statistic on the 10 training datasets was 0.857 and for testing datasets was 0.856.

Table 4 Comparison of the Different Rx-Risk Scoring and Modelling Methods to Predict 1-year Mortality

In the sensitivity analysis, where two or more prescriptions dispensed in a 12-month period were required to indicate the presence of a condition, the c-statistics were similar to the main analysis. The unweighted Rx-Risk c-statistic was 0.817, the weighted Rx-Risk c-statistic was 0.855, and individual conditions c-statistic was 0.860. Despite differences in the weights assigned to some Rx-Risk conditions compared to the primary analysis, which required only one or more prescriptions, the overall performance remained consistent (Supplementary Table 3).

Discussion

In this study, we found that conditions identified by the updated Rx-Risk Index had reasonable agreement with self-reported conditions but poor agreement with hospitalisation records for nearly all conditions except diabetes and renal failure. This variation likely reflects the nature of certain conditions, such as those primarily managed in outpatient settings or associated with preventive care (eg, hyperlipidaemia), which are less likely to be documented in hospital records. Additionally, with self-reported conditions, some individuals may be taking medications for conditions they do not report, either due to low health literacy or lack of awareness of the condition for which the medication is prescribed.

A previous study assessed the sensitivity and specificity of the Rx-Risk-V Index (non-ATC coded) conditions and self-reported conditions using the Australian Longitudinal Study on Ageing (ALSA) survey and found that the Rx-Risk category for thyroid disease was the only condition with high sensitivity (82%).⁵ The differences in sensitivities could be attributed to the use of self-reported medicines in the ALSA study, which is susceptible to recall bias or underreporting, compared to the pharmacy claims data used in the present study, which provides a more comprehensive and objective record of dispensed medications.

In the Irish LongituDinal study on Ageing (TILDA), which used a modified first ATC-coded Rx-Risk Index,⁷ the Rx-Risk Index demonstrated moderate-to-high agreement with self-reported conditions for asthma, hyperlipidaemia, glaucoma, and diabetes (kappa 0.41–0.81).¹⁴ In our study, the Rx-Risk Index showed moderate-to-high agreement with self-reported conditions for seven conditions – depression, hypertension, chronic airways disease, diabetes, osteoporosis/Paget’s disease, gout, and glaucoma demonstrated moderate-to-high agreement with self-reported conditions (kappa 0.41–0.81).

The PPV of the Rx-Risk Index was high for conditions like renal failure, diabetes, gout, migraine, and macular degeneration, indicating Rx-Risk Index effectively identified these diseases, as many people with condition-specific medicines self-reported having the condition. However, for conditions with overlapping treatment options, the Rx-Risk Index may have limited utility for disease identification, as it may be unable to distinguish between conditions treated with similar medications.

Our sensitivity analysis showed that for some conditions increasing the number of dispensing required to identify a Rx-Risk Index condition or reducing the time window over which dispensings were considered increased the PPV, however there was a trade off with decreased NPV (Supplementary Table 2 and 3). This means that increased ability to correctly identify conditions comes at the cost of increased false negative results. High PPVs were harder to achieve when hospital diagnosis was used as the reference standard as only a fraction of people will have a record of hospitalisation and not all conditions are recorded in the hospital admission records.

Epidemiological studies often aim to understand the effects of an exposure—such as a medication, environmental factor, or behavioral risk factor— on a defined outcome. However, these associations can be biased due to selective prescribing of exposures to those with existing comorbidities.¹⁵ In this context, validated tools like the Rx-Risk Index, which identify individuals with diseases and comorbidities, can play a critical role in controlling for bias by providing a proxy for underlying baseline risk. By providing a standardized approach to disease identification based on medicines use data, the Rx-Risk Index can help as a measure of overall health, which can be used to reduce systematic error, due to confounding.^15–17 In this study, we found that the Rx-Risk score is a valid tool for adjusting for confounding. The Rx-Risk score is a strong predictor of one-year mortality, with a c-statistic of 0.8, consistent with previous study.⁷ These results also align with the validation analysis of the first ATC-coded Rx-Risk Index in predicting one-year mortality (c-statistic of 0.8) in the Norwegian Epidemiologic Osteoporosis Studies (NOREPOS), which used pharmacy claims data.¹⁸ Additionally, our current findings are consistent with those of another medicines-based comorbidity index, the Modified Chronic Disease Score (M-CDS), developed and validated in the Italian population.¹⁹ The M-CDS was found to be predictive of one-year mortality and outperformed the Charlson Comorbidity Index (CCI), which is based on hospital diagnoses information (c-statistic: 0.761 for M-CDS vs 0.696 for CCI).¹⁹

Limitations

The findings are primarily applicable to older Australians aged 65 years and older, as the study was based on Australian datasets. These data sources may not fully capture all health conditions or represent populations outside this demographic. Further validation would be required to ensure the accuracy and applicability of the Rx-Risk Index if applied to other populations, such as younger individuals or those in different healthcare systems.

The validity analysis of the updated Rx-Risk Index is constrained by the quality and scope of the reference standard. Self-reported conditions suffer from underreporting and missing information due to recall bias and health literacy, while hospital records have limited population coverage and varied disease coding rules. Consequently, despite low kappa statistics due to reference standard limitations, the Rx-Risk Index may remain valid for disease classification, especially when the medicines identified by the index are indeed used for treating a specific disease.

The Rx-Risk Index was developed as a risk adjustment measure; therefore, a specific medicine should only be mapped to one condition. This is done to avoid double counting conditions in the overall comorbidity score; however, in practice, one medicine may be indicated for use in several different conditions. In this situation, using Rx-Risk Index for disease classification can result in high sensitivity but low PPV.

Conclusion

The use of the Rx-Risk Index as a tool for disease classification when no diagnosis information is available may be valid, particularly when specific medicines are used in disease management. The Rx-Risk Index category for conditions, such as diabetes and renal failure, were found to have high PPVs, meaning that individuals flagged based on prescription data are highly likely to have the condition. However, the variability in treatment patterns and the overlap in medications used for different conditions could limit the Rx-Risk Index’s use as a standalone tool for comprehensive comorbidity measurement. To address this limitation, incorporating additional data, such as diagnostic information from primary care, could help to improve disease identification performance. Solely relying on hospital diagnoses may bias the tool to individuals with more severe conditions, potentially under estimating those with less severe cases. The updated Rx-Risk Index when used as an overall number of conditions present remains a valid comorbidity risk-adjustment measure that is strongly predictive of one-year mortality and can be an important proxy measure of overall health.