Adherence Definitions, Measurement Modalities, and Psychometric Properties in HIV, Diabetes, and Nutritional Supplementation Studies: A Scoping Review

Background

Suboptimal adherence to medications often hinders effective care for patients who regularly consume medications over prolonged periods.¹ High levels of suboptimal adherence and nonadherence can lead to increased morbidity and mortality across illnesses.² Diabetes, HIV, and micronutrient deficiencies are three chronic public health conditions that can be managed or ameliorated by adhering to medication treatments. We explored HIV and diabetes due to the large body of literature on adherence within the health topics. We included nutrition because this paper is part of a larger project on adherence to nutritional supplements.

Current estimates indicate that adherence to treatment remains low for all three conditions. While 38.4 million people worldwide lived with HIV in 2021, on average, only about 60% adhered to antiretroviral therapy guidelines.^3–7 Adherence can prevent viral drug resistance, slow HIV progression, and reduce the risk of HIV transmission.⁸ Worldwide, 537 million adults lived with diabetes in 2021, and treatment adherence ranged from 38.5 to 93.1%.^9,10 Adherence to diabetes treatments such as insulin or oral hypoglycemic medication can help control hyperglycemia and prevent vision loss, limb amputations, and myocardial infarction.¹¹ Finally, one in three people around the world is estimated to have a micronutrient deficiency.¹² Data on adherence to micronutrient supplements largely focuses on adherence to iron supplements in women of reproductive age, especially pregnant women. A study exploring iron folic acid adherence for pregnant women in 22 countries with high burdens of undernutrition found that only 8% of pregnant women adhered to the ideal iron folic acid supplementation schedule.¹³ While the effects of micronutrient deficiencies depend on the micronutrient, they can cause weakness, brain damage, and increase the risk of severe infections.^14–17

The World Health Organization defines adherence as “the extent to which a person’s behavior (including medication-taking) corresponds with agreed recommendations from a health care provider”.² Most researchers agree on the conceptual definition of adherence, but there is no consensus on its operational definition, mostly because a widely accepted “gold standard” adherence measure or operational definition does not exist. Studies measure adherence using different modalities (including self-reports and blood samples), which may lead to different conclusions about adherence patterns.

We define modalities of adherence measurement as channels used to assess adherence or the way that adherence information is collected. We define measures as the means of data collection that can fall within a modality. For example, questionnaires, pill counts (self-reported), in-person interviews, and telephone interviews are measures under the self-report modality. Adherence measures also fall under two categories: direct and indirect measures. Direct measures assess the concentration of a medicine in the body. Indirect measures do not assess the amount of the medication in the body but measure something approximating the amount of the medication ingested. For example, self-reported questionnaires may ask about medication-taking habits to approximate the concentration of medications in participants.

Multiple operational definitions and measurement modalities of adherence can pose research challenges such as inconsistent results and conclusions. When different adherence measurement methods are used, comparing results across studies and identifying erroneous results can be more difficult. However, because there is not a commonly agreed upon gold standard to assess adherence, triangulation may be the most viable strategy for assessing and reporting adherence. Triangulation (ie, relying on multiple measures or modalities) can increase the rigor of research findings by limiting the impact of bias or error associated with any one method and demonstrating similar findings across different adherence measurement methods.¹⁸

The purpose of this scoping review is to identify methods and modalities used to measure adherence. We focus specifically on HIV medication, diabetes medication, and nutritional supplementation and explore the variation in adherence definitions, measurement modalities, and psychometric property reporting across studies.

Methods

We conducted this scoping review largely following the methodology published by Peters et al.¹⁹ The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping review (PRISMA-ScR) reporting guideline was used (Supplementary Material 1 PRISMA-ScR Checklist).²⁰

We searched three databases for studies on psychometric properties of adherence measurements: PubMed, PsycINFO, and Scopus. The search strategy included three sections: treatment adherence/compliance terms, study area terms (ie, HIV, diabetes, or nutritional supplementation terms), and psychometric terms (Supplementary Material 2 Search Strategy). The PubMed search included both MeSH and text word search field tabs. The PsycINFO and Scopus searches included title, abstract, and keyword search field tabs. We included both “adherence” and “compliance” as search terms in our strategy because the terms are often used interchangeably.^21–23 The search ran in May 2022, and we uploaded all papers resulting from our search strategy onto Covidence.

Inclusion and Exclusion Criteria

We included only peer-reviewed, primary research studies written in English. We limited our review to studies published between January 1, 2012, to January 1, 2022, those that reported psychometric properties of adherence/compliance (ie, validity, reliability) and had adherence or compliance to HIV or diabetes medications or nutritional supplements as a behavioral outcome. However, we included glucose monitor articles if they met all other inclusion criteria except having an adherence/compliance behavioral outcome. Many anti-diabetic drugs influence glucose levels, so we included glucose monitors as they have been used as proxies for adherence.

We excluded studies that explored adherence or compliance but did not present data on the psychometric properties of adherence or compliance measures. For example, some studies reported the level of nutritional supplementation adherence in a community and identified individual- and community-level factors that were associated with nutritional supplementation adherence such as education or income. However, we excluded these studies because they did not report any psychometric data on their adherence data.

Selection Process

All studies from the database searches were uploaded into Covidence, which automatically deleted duplicate studies. One reviewer then manually screened the studies in Covidence for duplicates. Two reviewers then performed initial screenings on ten articles based on an article’s title, abstract, and keywords. The reviewers had 90% agreement. After resolving the disagreement through consensus, one reviewer completed the title, abstract, keyword, and full-text screening for the rest of the articles. Occasionally, the second reviewer assisted in article screening when the first reviewer was unsure if the article should be included.

Data Charting Process

We developed an extraction form in Covidence to facilitate the extraction process. Two members of the research team piloted the form on five articles to ensure information was captured consistently and completely. We did not revise the extraction table. The rest of the data were single-extracted, and two research team members reviewed the completed extraction table.

Data Items

The extracted variables included: general article information (title, authors, year of publication, funding, possible conflicts of interest), study characteristics (aims, study topic area, eligibility criteria, country, country income level), intervention characteristics (adherence measure type, adherence measure name, adherence definitions, validity calculations, reliability calculations), population characteristics (sample size, age, sex), study findings and conclusions.

Synthesis of Results

We explored the frequency and type of modality and measure used by health topic (ie, HIV, diabetes, and nutrition). In each study, the researchers only looked at the psychometric properties of adherence for one health topic. We examined differences in adherence definition categories by measures and modalities. We analyzed triangulation of adherence modalities and measures by country income group. Finally, we looked at the distribution of validity and reliability data by health topic.

Results

The review identified 591 articles across PubMed, PsycINFO, and Scopus that met the inclusion criteria. After removing duplicates, we screened titles, abstracts, and keywords for 494 articles and excluded 354 articles, mainly because they did not focus on adherence or compliance to HIV or diabetes medications or nutritional supplements. After a full-text screening of 140 articles, we excluded 62 more articles. In total, we included 78 articles in the review, which corresponded to 88 studies as some articles contained multiple studies (Table 1). The PRISMA-ScR flowchart (Figure 1) shows the screening process.

Table 1 Overview of Study Characteristics

Figure 1 PRISMA ScR Flowchart.. Notes: PRISMA figure adapted from Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Journal of clinical epidemiology. 2009;62(10). Creative Commons.100

Overall, 36 studies addressed adherence to HIV medications, 51 studies examined adherence to diabetes medications, and one study explored adherence to nutritional supplements. The highest percentage of studies focused on North America (32%) followed by East Asia and Pacific (20%), and Sub-Saharan Africa (16%). Most studies (73%) explored the psychometric properties of adherence as a primary aim, while 27% investigated the psychometric properties of adherence as a secondary aim.

Measurement Modalities and Measures

We identified 9 modalities, 221 total measures, and 143 unique measures across all studies. The nine modalities included blood samples, hair samples, urine samples, electronic records, pharmacy records, medical records, pill counts (non-self-reported), self-reports (including questionnaires and pill counts), and other. Overall and within the HIV and diabetes studies, the three most common modalities were self-reports, blood samples, and electronic records, in that order (Table 2). Within the self-report modality, 97% of measures were questionnaires. The 8-item Morisky Medication Adherence Scale (MMAS-8) was the most frequently used self-reported measure, with 14 instances of use. Among the blood sample measures, 40% tested HbA1c levels and 25% examined viral loads.

Table 2 Distribution of Measures and Modalities by Health Area

By country income groups, 49 studies were conducted in high-income countries (HICs), 27 studies in upper-middle-income countries (UMICs), 14 studies in lower-middle-income countries (LMICs), and 5 studies in low-income countries (LICs). (Seven studies in multiple country income groups were double counted). The use of triangulation was similar across country income groups. In each group, more than half the studies included at least two types of data collection methods and two different measures (Figure 2). The mean number of modalities in each study was 1.84 (SD = 0.73) overall, 1.90 (SD = 1.17) for HICs, 1.85 (SD = 1.18) for UMICs, 1.86 (SD = 0.86) for LMICs, and 1.8 (SD = 1.14) for LICs. In HICs, UMICs, and LICs, studies with more than two modalities and more than two measures were most common, whereas in LMICs, studies with exactly two modalities and two measures were most common. Almost no relationship exists between country income level and triangulation.

Figure 2 Modalities And Measures, By Country Income Group.

Adherence Definitions

The operational definition of adherence fell into four categories: numerical, dichotomous, ranked ordinal, and undefined. Numerical definitions define adherence discretely (eg, scale scores) or continuously (eg, percentages). Examples of numerically defined adherence include percentages of pills taken in a certain time frame or adherence levels on visual analog scales in self-reported questionnaires. For example, Zhang et al asked participants how many days they took their medications as prescribed in the last month.⁵⁸ From the responses, the authors calculated the percentage of days the participants were adherent in the past 30 days.⁵⁸

Dichotomous definitions describe adherence in two states, such as adherent/non-adherent, good glycemic control/poor glycemic control, or undetectable viral load/detectable viral load. Cutoff points to dichotomize adherence across studies were the same for some measures such as HbA1c where good glycemic control was defined as HbA1c <7%. The threshold for adherence varied across studies for other measures such as viral load, where participants could be classified as “adherent” if their viral load was ≤20 copies/mL or ≤400 copies/mL depending on the study.

Finally, ranked ordinal definitions describe adherence as having multiple levels. Most measures (65%) defining adherence with a ranked ordinal scale used MMAS, which categorized adherence into high/medium/low.^{62,70,71,77–81,93–96,98,99} Based on the total score of the scale, high adherence was defined as a score of eight, medium as a score of six or seven, and low as a score less than six.^{62,70,71,77–81,93–96,98,99}

Adherence was not defined for 40 (18%) measures. Approximately 14% of HIV measures, 21% of diabetes measures, and 25% of nutritional supplement measures did not define adherence. Many of the studies that did not define adherence aimed to create a new measure or to translate an existing measure. Some reasons why studies did not define adherence included having study aims testing the correlation between adherence measures,^{24,28,41,45,46,67,81–83,90,101} testing the internal reliability of a new measure,^53,59,61 testing the content validity of a new measure,^53,61,88 and testing the construct validity of a new measure.^53,59

Overall, dichotomously defined measures were the most popular (89 of 221 measures or 40%) followed by numerical (27%) and ranked ordinal (14%). Dichotomous measures were also the most popular way to define adherence across health topics, country income levels, and measure types (direct/indirect). However, the most common type of adherence definition fluctuated across modalities because of the measures within the modality. For example, the dichotomous adherence definition type was most common within the blood sample modality because viral load and HbA1c measures define adherence dichotomously and comprise 64% of the blood samples in the review. Similarly, the majority of adherence definitions in the pharmacy modality were numerical because adherence was often expressed as a rate, such as the proportion of days covered or medication possession ratio.^{27,31,34,63,102}

The amount of variation in an adherence definition category within a modality depended on 1) whether the measures within the modality could be assessed numerically and 2) whether widely accepted cutoffs existed for the measure. There was more variation in adherence definition types within modalities if measures could initially be measured numerically. This is because the researchers decided how they wanted to categorize adherence after collecting numeric data. For example, pill count measures were initially measured numerically. Some studies reported pill count as a discrete number or a percentage of remaining pills given the original number of pills dispensed.^{24,26,27,29,32,35,37,38,43,47,50,94} Other studies reported pill count in a dichotomous or ranked ordinal manner. For example, Teshome et al defined “high adherence” as the healthcare worker not seeing ≥80% of pills dispensed for the past 30 days.²⁴

For self-reported measures, the wide variety of scales used across studies and the originally continuous nature of most scales resulted in a wide variation of adherence definitions within the modality. For example, Ayoub et al and Mallah et al were the only studies in the review to measure adherence using the Lebanese Medication Adherence Scale (LMAS-14), which initially measured adherence numerically on a scale from 0 to 42.^64,82 Ayoub et al defined adherence on LMAS-14 dichotomously by classifying patients as adherent or non-adherent using a cut-off point of 38.⁶⁴ Meanwhile, Mallah et al did not define adherence for LMAS-14 as the scale was a reference measure a new scale.⁸² Finally, there was less variation in adherence definition types within modalities if measures had widely accepted cutoffs, even if the measures could initially be defined numerically. For example, all measures for HbA1c in the review were originally measured numerically. However, adherence was reported dichotomously across studies, with good glycemic control as HbA1c <7%.^{62,66,67,70,71,73,75–77,81,82,85–87,91,92,96,98}

Reporting Validity/ Reliability

Among the studies in the review, 46 (52%) reported both validity and reliability, 28 studies (31%) only reported validity, and 14 studies (16%) only reported reliability. Fourteen types of validity and six types of reliability appeared across the studies (Table 3). Almost half of all studies measured internal consistency reliability (through Cronbach’s alpha), and more than a quarter of studies tested sensitivity and specificity. Scores for the five most common validity types and three most common reliability types had overall distributions of at least 0.5 across all studies. Overall, Cronbach’s alpha had the lowest score range of 0.5 (0.97 to 0.47) with a mean score of 0.76 (Figure 3). The next most common type of reliability measured in the review (test-retest reliability) had a mean score of 0.55 with a range from 0.376 to 0.975. Finally, the Kappa coefficient for inter-instrument reliability had a mean score of 0.49 with a range from 0.107 to 0.995. Of the five most common validity measures, the negative predictive value had the highest mean score (0.76, range: 0.311–1), followed by sensitivity (0.66, range: 0.049–1), specificity (0.60, range: 0.177–0.98), positive predictive value (0.55, range: 0.237–0.948), and convergent validity (0.53, range: 0.36–0.88).

Table 3 Types of Validity and Reliability

Figure 3 Distribution Of Select Validity and Reliability Scores, By Health Topic.

Within each type of validity and reliability, the distribution of scores varied greatly between HIV and diabetes studies, except for internal consistency (Figure 3). (Nutritional supplementation was excluded from Figure 3 because there was only one study included in the review). The range of validity/reliability scores for diabetes studies was the same or similar to the overall score ranges, except for sensitivity and inter-instrument reliability. The sensitivity range for diabetes studies is about half (0.4) of the overall range because the overall sensitivity includes outliers from HIV studies. The range of inter-instrument reliability scores was also smaller (0.105) for diabetes than for the overall range. Diabetes studies reported a greater number of psychometric scores across the most common validity/reliability categories in Figure 3, except for inter-instrument reliability. Only two inter-instrument reliability scores were reported across diabetes studies.

In HIV studies, the median scores for specificity, positive predictive values, and negative predictive values were higher compared to diabetes studies, whereas in diabetes studies, the median scores for sensitivity and convergent validity were higher compared to HIV studies. Additionally, across the three most common types of reliability, median scores were higher for diabetes studies compared to HIV studies, except for internal consistency.

Among HIV studies, about 30% explored only validity or reliability, and 39% explored both validity and reliability. Most (61%) diabetes studies examined both validity and reliability, and the nutritional supplement study examined both psychometric properties as well. For UMICs, LMICs, and LICs, studies reporting both validity and reliability were most common. Studies only investigating validity were most common for HICs. Additionally, most studies explored both validity and reliability in all regions except Europe (where studies exploring only validity were most common).

True Effect Sizes

Of the 46 studies that reported validity and reliability, 26 shared a Pearson’s correlation coefficient, indicating the strength of the linear relationship between the test and reference measures used to measure adherence in each study. Studies that reported these results were nearly evenly divided between diabetes and HIV studies (57% and 43%, respectively). Pearson r values, which in these studies were calculated to show the correlation between test and reference measures, ranged from 0.09–0.93. From those values, we grouped studies into “low” (0.09–0.36), “medium” (0.37–0.65), or “high” (0.66–0.03) effect size. Some studies reported multiple effect sizes, using different reference and test measures, so those effect sizes have been included to reach a total of 28 reported effect sizes. Overall, we found that seven studies fell into the “low” range, 11 in the “medium” range, and 13 in the “high” effect size range. There was no significant difference found in whether these studies were reporting on adherence to diabetes or HIV medication; whether they were conducted in high-, middle-, or low-income settings; or based on population size.

Discussion

In this review, we summarized the methods and modalities used to measure adherence to HIV medication, diabetes medication, and nutritional supplements. We also analyzed variations in adherence definitions and how the psychometric properties are measured and reported.

Across 88 studies in the review, there were 9 modalities, 221 total measures, and 143 unique measures. All modalities and measures have strengths and limitations related to their acceptability, feasibility, reliability, and validity. Through triangulation, researchers can compare modalities and measures to choose those that best suit their study needs. In practice, triangulating modalities or measures lead to more flexibility in the field. Instead of relying on one measure, such as a medication event monitoring system, which requires specific equipment and training, having multiple acceptable adherence measures or modalities grants more feasibility for adherence research in a variety of contexts, especially in low-resource settings. Some measures may also overestimate or underestimate adherence systematically, while others may do so randomly. If the same measure is used in all studies, researchers would replicate the same limitation associated with the measure or modality across studies. Researchers can better balance measurement errors across adherence measures by using some that are more likely to overestimate and underestimate adherence to more accurately measure adherence behavior. Therefore, since no gold standard adherence measure exists, we recommend researchers include multiple measures and modalities in their studies and use triangulation to balance individual measurement errors and leveraging the diverse strengths of various measures and modalities.

In the review, we also identified three types of operational definitions for adherence: numerical, dichotomous, and ranked ordinal. While categorizing patients into levels of adherence can help clinicians or researchers divide patients for treatment interventions, follow-up, or other actions, the scientific significance of a cutoff is often marginal. Someone who is 94% adherent or 96% adherence on the scale may not have different health outcomes even if they are categorized as having poor adherence and good adherence respectively. Similarly, many studies measuring viral load defined dichotomous adherence cutoffs based on the test’s sensitivity or limit of detection rather than differences in clinical outcomes.^{29,33,35,42,44,46,47,49,50,54} This is why in some studies using sensitive instruments to measure viral load, undetectable viral load/adherence is classified as ≤20 copies/mL whereas in other studies using less sensitive instruments, undetectable viral load/adherence is classified as ≤400 copies/mL. One solution to these cutoffs is to conceptualize adherence as a spectrum and report adherence numerically.

Finally, the limited number of statistics for one type of validity or reliability in the review restricted our results to be largely qualitative and hindered our ability to compare statistical values. To allow for future quantitative reviews of the psychometric properties of validity and reliability, researchers could report a core group of statistics across studies. Future research and discussions are needed to determine which measures would be most meaningful and feasible to include in the core statistics group.

The distribution of scores within one type of validity or reliability measure differed greatly between HIV and diabetes, except for internal consistency. One reason for this difference among validity measures could be the nature of the diseases. HIV is an infectious disease, while diabetes is a non-communicable disease. The median scores and ranges for negative predictive value and specificity were higher than positive predictive value and sensitivity. This indicates that when choosing adherence measures for HIV, researchers prioritize minimizing false positives and maximizing true negatives over minimizing false negatives and maximizing true positives. In other words, it is more important to correctly identify people who are non-adherent than people who are adherent for HIV. This could be because people who are not virally suppressed may transmit the disease to others and are more susceptible to other diseases.

Finally, when calculating true effect sizes between the reference and test adherence measures, we found that 7 studies fell into the “low” range, 11 in the “medium” range, and 13 in the “high” effect size range. The higher effect size suggests more agreement between the reference adherence measure and the measure being tested. The variability in effect size among studies in our review shows a lack of consistency in the strength of the relationships between reference and test adherence measurements. Therefore, we are likely still far from finding a “gold standard” approach to adherence measurement.

Limitations

One limitation is the exclusion of non-English language studies, which would have provided more insights into the psychometric properties of adherence measures. Additionally, restricting the search to peer-reviewed primary research studies published within a ten-year period might have overlooked relevant literature published before or after this timeframe, potentially leading to gaps in the understanding of adherence. Expanding the timeframe searched would have improved our ability to present an unbiased summary of the psychometric properties of adherence within the HIV, diabetes, and nutritional supplementation literature. Finally, only one reviewer looked at all potential studies and analyzed the data, which limited the robustness of the review by constraining the diversity of perspectives in the analysis.

Conclusion

Measuring adherence accurately and reliably continues to be a challenge for research in HIV, diabetes, and nutritional supplementation. The ability to accurately measure adherence is imperative to assessing and monitoring the health of people with chronic diseases and reducing their morbidity and mortality. Currently, there is no standard operational definition for adherence or a widely accepted adherence measure for HIV, diabetes, or nutritional supplementation. Instead of searching for a standard measure, a rigorous way to measure adherence could be through multiple modalities and measures that all triangulate to a common conclusion.

Recommendations for Reporting

Based on the findings in our review, we offer three key recommendations for reporting adherence measurements. First, researchers should report adherence results from multiple measures and modalities. Each measurement and modality type has different strengths and limitations, which triangulation can help balance. In the absence of a gold standard measure, using multiple measures and cross-checking results can help enhance the validity of research findings and mitigate bias and provide a practical and nuanced solution to measuring adherence. Second, adherence results should be reported numerically. While categorizing patients into different adherence categories may be helpful in clinical settings, the cutoffs for categorization can differ based on a test’s precision and may not reflect differences in clinical outcomes. If researchers feel that categorization is appropriate, we still recommend reporting numeric results in case cutoffs change in the future. Finally, researchers investigating the validity and reliability of adherence measures should report multiple types of validity and reliability in their studies (including Cronbach’s alpha) to improve statistical comparisons across adherence measures.