Predictive validity of a brief antiretroviral adherence index: Retrospective cohort analysis under conditions of repetitive administration
© Mathews et al; licensee BioMed Central Ltd. 2008
Received: 20 May 2008
Accepted: 29 August 2008
Published: 29 August 2008
Newer antiretroviral (ARV) agents have improved pharmacokinetics, potency, and tolerability and have enabled the design of regimens with improved virologic outcomes. Successful antiretroviral therapy is dependent on patient adherence. In previous research, we validated a subset of items from the ACTG adherence battery as prognostic of virologic suppression at 6 months and correlated with adherence estimates from the Medication Event Monitoring System (MEMS). The objective of the current study was to validate the longitudinal use of the Owen Clinic adherence index in analyses of time to initial virologic suppression and maintenance of suppression.
278 patients (naïve n = 168, experienced n = 110) met inclusion criteria. Median [range] time on the first regimen during the study period was 286 (30 – 1221) days. 217 patients (78%) achieved an undetectable plasma viral load (pVL) at median 63 days. 8.3% (18/217) of patients experienced viral rebound (pVL > 400) after initial suppression. Adherence scores varied from 0 – 25 (mean 1.06, median 0). The lowest detectable adherence score cut point using this instrument was ≥ 5 for both initial suppression and maintenance of suppression. In the final Cox model of time to first undetectable pVL, controlling for prior treatment experience and baseline viral load, the adjusted hazard ratio for time updated adherence score was 0.36score ≥ 5 (95% CI: 0.19–0.69) [reference: <5]. In the final generalized estimating equations (GEE) logistic regression model the adjusted odds ratio for time-updated adherence score was 0.17score ≥ 5 (0.05–0.66) [reference: <5].
A brief, longitudinally administered self report adherence instrument predicted both initial virologic suppression and maintenance of suppression in patients using contemporary ARV regimens. The survey can be used for identification of sub-optimal adherence with subsequent appropriate intervention.
In previous research, we validated a subset of items from the ACTG adherence battery as prognostic of virologic suppression at 6 months and moderately correlated with adherence estimates from the Medication Event Monitoring System (MEMS) . The objective of the current study was to validate the longitudinal use of the Owen Clinic adherence index in analyses of time to initial virologic suppression and maintenance of suppression.
Patient Characteristics at Study Entry (n = 278)
Sex [n (%)]
HIV Transmission Risk Factor [n (%)]
MSM1, not IDU2
Race/Ethnicity [n (%)]
ART3 Treatment Experience [n (%)]
Baseline absolute CD4
Baseline log10 HIV-1 Plasma Viral Load
Days on new regimen
Year of study entry [n (%)]
New Regimen Type4 [n(%)]
NNRTI & ≥ 2 NRTIs
PIb & ≥ 2 NRTIs
NNRTI & PIb & ≥ 1 NRTI
≥ 2 NRTI
# Adherence Scores per patient
Of the 1155 records in the final analysis dataset representing the longitudinal histories of 278 patients, HIV viral load and adherence were measured on the same date in 556 (48%) records. Of the 1155 records, 599 (52%) represented missing adherence scores at dates of viral load measurement. Of the 599 missing adherence scores, 426 were imputed using the last observation carried forward approach (LOCF) and 173 were imputed by backfilling values. Even though these missing adherence scores technically represent missing values at the time the viral load measures were taken, they conceptually represent values that were obtained at a different time point than the viral load measures. These instances typically represent patients for whom blood is drawn either before of after a clinic visit at which adherence assessment was conducted. The median (IQR) time between the regimen start date and date of the first recorded adherence score was 21 (13–60) days.
Time to First Viral Suppression Analysis
Unadjusted and adjusted effects of time-updated adherence scores on time to first HIV viral load ≤ 400 copies/ml in Cox regression models (n = 278 patients)
Baseline log10 HIV viral load
Maintenance of Viral Suppression Analysis
Unadjusted and adjusted effects of time-updated adherence scores on maintenance of HIV viral load ≤ 400 copies/ml in generalized estimating equation logit regression models (n = 217 patients achieving initial viral suppression)
Baseline log10 HIV viral load
In the developmental phase of adherence measurement in our clinic, we constructed a 5-item instrument whose individual items were selected from the 51-item ACTG adherence battery  on the basis of factor structure and internal consistency reliability. In the manuscript presenting this developmental work, we showed that responses on the 5-item adherence index, administered on one occasion 30 days after initiating a new antiretroviral regimen, were moderately correlated (Spearman rho 0.40 – 0.48) with measures of electronic drug monitoring (EDM) and were predictive of HIV viral load responses at 3 and 6 months after start of treatment in models controlling for baseline viral load and prior antiretroviral experience. We also showed that a cut point of 5 or more on the index distinguished those with viral load suppression (≤ 400 copies/ml) at 3 and 6 months from those failing to suppress at the same time points . The currently reported analyses were conducted to evaluate whether the same 5-item index, when administered repetitively under longitudinal follow up, predicted initial viral suppression and maintenance of suppression while patients continued the index regimen. We found, conditional upon the study eligibility criteria and analytic methods, that the self-report adherence index scores were predictive of both outcomes in models controlling for prior antiretroviral treatment experience and baseline plasma viral load. For the time to initial viral suppression outcome, adherence scores ≥ 5 were associated with an approximately 60% reduced hazard of achieving a plasma viral load ≤ 400 copies/ml. For the maintenance of viral suppression outcome, adherence scores ≥ 5 predicted an approximately 80% lower chance of maintaining viral suppression relative to scores less than 5.
These findings are not directly comparable to the effects demonstrated in our earlier study for several reasons including: (1) period effects (1998 – 1999 vs. 2003 – 2006) associated with changes in potency and simplicity of antiretroviral regimens; (2) differences in prior treatment experience (22% vs. 60% antiretroviral naïve comparing the earlier to the current study); (3) conditions of adherence measurement (written completion [earlier study] vs. computer assisted [current study]); and (4) differences in analytic approach (outcomes analyzed cross sectionally at fixed time points [earlier study] vs. longitudinally in continuous time [current study]). Nonetheless, the current results contribute to the predictive validation of the instrument as it has been used in routine clinical care of patients on antiretroviral therapy.
In a recent review of the status of HIV adherence measurement, Chesney presented a conceptual model of adherence assessment and intervention, distinguishing research from clinical applications, and resource-rich from resource-poor settings. In discussing the "elusive gold standard" of adherence measurement, she emphasized that "efforts should continue to develop a portfolio of different valid and reliable self-report measures with varying strengths and weaknesses that can be optimally applied, depending on the situation ." In that spirit, we discuss a number of challenges that emerged in exploring the relationship between routine longitudinal adherence measurement using the Owen Clinic instrument and viral suppression.
Second, the modeling of adherence score is not straightforward. As constructed its scale of measurement is discrete numerical with a possible range of 0 – 25 with skewness not amenable to a normalizing transformation. Although cut point selection for an underlying numerical measure may introduce bias in effect measurement  and may reduce power to detect effects in comparison with use of the numerical measure , cut point models are often preferred because of simplicity of data summarization and interpretation. Post hoc cut point selection, as pointed out by the authors of the STARD initiative  (Item 9), may not be replicable with other datasets. In our modeling of the effect of adherence score, we employed an approach adapted from Williams et al , first exploring the functional form of the relationship between adherence score as a numerical measure using smoothing regression splines as implemented by Royston and Sauerbrei in STATA followed by cut point examination adjusted for multiple comparisons . Cut points alternative to what we have described as the lowest detectable cut points could be recommended if alternate methodologies of correction for multiple comparisons were employed (e.g. cross validation or split sample approaches, or examination in independent data sets). It is of interest that in our earlier study, a similar cut point on the same instrument (≥ 5/< 5) was felt to be the most discriminating cut point . After examining the regression spline plots for both outcome metrics (Figures 3 and 4) in the current study, we felt that a cut point around 5 identified a region above which a monotonic relationship between adherence score and functions of the outcome metrics was suggested. In clinical care settings, we believe, based on these data, that our clinicians should be alert to clinically significant problems with adherence for scores at or above 5.
Third, because of the observational nature of the data, measurements of adherence and HIV plasma viral load were not scheduled to occur simultaneously. Typically clinicians order viral loads every 3 – 6 months depending on clinical factors. Adherence in contrast is measured in our clinic at all routine visits. Conceptually, adherence is a construct representing a daily health behavior for which various self-report indicators have been developed and mapped to estimates of percentage adherence over a defined period or, as in the case of the Owen Clinic instrument, given interpretability primarily through demonstrated association with viral suppression. Because of the staggered nature of data accrual in the clinic, decisions must be made regarding how to line up sequential viral load and adherence measures. At a conceptual level, it is a non-trivial question to decide over how long a period an adherence measure based on a limited recall period (4 days in the case of our instrument) can be extrapolated with regard to preceding and future adherence behaviors for which the self-report data represents an imperfect indicator. In our primary analysis, we made the assumption that a given adherence assessment carried forward no longer than 90 days from the antecedent adherence measurement. Whether the observations that are not temporally matched represent truly missing observations is debatable since the very nature of the data accrual process in clinical care did not require temporal matching of adherence and viral load measurement. Because the LOCF principle has been criticized in recent years , we explored alternate analyses to evaluate the robustness of our findings. First, to determine if the frequency of adherence measurement was related to adherence scores such that longer intervals between measurements were associated with better or poorer adherence, we calculated rates of adherence measurement per 100 days of follow up. We then divided the adherence measurement rate distribution into quartiles and used analysis of variance to test for equality of mean adherence scores across the quartiles, finding no significant difference (p = 0.89). This provided limited evidence that, in our data set, adherence scores were not systematically related to frequency of measurement, although others have found that missing adherence values were associated with nonadherence . Second, we restructured the data set by grouping follow up time in 6 month intervals, taking the median adherence score for the interval as representative, the last viral load in the interval as the outcome, and repeating the panel regression for longitudinal viral suppression. In a model comparable to that shown in Table 3 controlling for prior treatment experience and baseline log10-HIV viral load, the adjusted odds ratio for viral suppression was 0.14 (95% CI: 0.06 – 0.33, p < 0.0001) for a 6-month median adherence score greater than 5. Finally, in a third analysis of maintenance of longitudinal viral suppression, mean adherence scores were calculated for the period immediately prior to each viral load measurement, creating a score for each interval between viral load measurements. This operationalization of adherence was then fit in a GEE logit model for maintenance of viral suppression, again controlling for prior treatment experience and baseline log10-HIV viral load. The adjusted adherence odds ratio for maintaining viral suppression for a mean interval adherence score greater than 5 was 0.28 (95% CI: 0.14 – 0.57, p < 0.0001) Therefore, although the adherence effect estimates were model dependent, the direction of effect was consistent and significant across models.
Despite the limitations of self-report adherence measures, they are likely to remain the most frequent modality of adherence assessment in clinical settings. The brief self-report instrument examined in this study and in an earlier developmental study has been demonstrated to correlate with electronic drug monitoring and to be predictive of viral load responses both when administered at baseline and also when administered in longitudinal follow up of unselected patients in clinical care for HIV infection.
A retrospective observational cohort study was conducted including all HIV-infected adults under care at the UCSD Owen Clinic between January 2003 and June 2006. Patients were included in the analyses reported here if they: (1) had at least one self report medication adherence score recorded; (2) either initiated antiretroviral therapy for the first time or began a new regimen during the study period; (3) had a plasma viral load ≥ 400 copies/ml prior to initiation of the index regimen; (4) had at least one post baseline plasma viral load; and (5) remained on the index regimen for at least 30 days. Only time on the first regimen during the study period (index regimen) is included in reported analyses. During the study period, patients on antiretroviral therapy were asked to complete, prior to meeting with their medical provider, a computer-assisted four item antiretroviral medication adherence survey  (Figure 1) at every primary care visit. The adherence assessment takes 2–3 minutes to complete and is overseen by the medical assistant who is also recording vital signs. Clinicians review adherence scores and are expected to document adherence counseling in the clinic electronic medical record if scores indicate adherence problems. The adherence items are a subset of the AIDS Clinical Trials Group (ACTG) adherence battery . Items 1 and 2 query the number of missed doses of each antiretroviral medication over each of the preceding four days. The number of missed doses for each drug is summed across all four antecedent days. The sum scores of the two drugs with the highest number of missed doses (designated items 1 and 2) are included in the index score. Item 3 asks "During the past 4 days, on how many days have you missed all your pills?" (response options (numeric code): no days (0), one day (1), two days (2), three days (3), all four days (4)). Item 4 inquires "How closely did you follow your specific schedule over the last 4 days?" (response options (numeric code): never (4), some of the time (3), about half the time (2), most of the time (1), all of the time (0)). Item 5 deals with weekend adherence behavior asking "Did you skip any of the HIV medications last weekend – last Saturday or Sunday?" (response options (numeric code): no (0), yes (1)). The index score is the sum of responses to the four items with a possible range of 0 (best adherence) to 25 (poorest adherence) if each component of the regimen was dosed twice daily.
Two outcome measures were operationally defined as: (1) time to first virologic suppression defined as HIV plasma viral load (pVL) ≤ 400 copies/ml after regimen initiation; and (2) maintenance of virologic suppression (pVL ≤ 400 copies/ml). Follow up time for each patient began with the date of initiation of the index antiretroviral regimen and ended with the earliest of the following events: (1) change or discontinuation of the index regimen; (2) last clinic visit date; or (3) end of the study period. Time to first virologic suppression on the index regimen was examined using extended Cox models incorporating time-updated adherence scores. It was confirmed that the proportional hazards assumption was met for all covariates included in the Cox models using log(t) by covariates interactions . Maintenance of virologic suppression was evaluated in logit models using population-averaged generalized estimating equations (GEE) with time varying covariates [22, 23]. GEE are a family of methods suitable for the analysis of the longitudinal relationship between a continuous or dichotomous outcome variable and both time-dependent and time independent covariates. The within subject dependency of observations is handled by assuming a working correlation structure for the repeated measurements of the outcome variable . The analysis for the maintenance of virologic suppression analysis included only those patients who achieved an initial pVL ≤ 400 copies/ml and their follow up began on the date of initial virologic suppression. The primary independent variable was time-updated adherence score. Because adherence scores were highly skewed toward higher scores (reflecting poorer adherence), adherence scores were first fit using univariate regression splines to examine the functional relationship between adherence score and the outcome measures [16, 17]. Spline techniques are a family of methods for determining the functional form of the relationship between a continuous predictor variable (e.g. adherence score) and an outcome variable . After determining that the functional relationships were approximately monotonic, eight binary cut points on adherence score were examined in ascending order (e.g. ≥ 1/< 1, ≥ 2/< 2, ≥ 3/< 3) until a threshold demonstrating statistical significance in adjusted models was found (lowest detectable cut point). Because multiple ascending potential cut points were examined, tests of significance for adherence score were adjusted using the Bonferroni method to maintain a overall type I error rate of 0.05 [16, 26]. Thus the critical p-value for each cut point was 0.05/8 = 0.00625. Examined covariates included: age, sex, race/ethnicity, HIV transmission risk factor, treatment experience (naïve or experienced at time of index regimen initiation), regimen type (number and type of antiretroviral drug classes in the regimen), and both CD4 and pVL measured at the closest time prior to initiation of the index regimen.
Because HIV plasma viral load and adherence score were not always measured on the same dates, records with missing values for adherence score after the first adherence measurement date were imputed using the last observation carried forward (LOCF) principle. Because the first adherence measurement date usually occurred after the regimen start date, records with missing early adherence scores were backfilled to the regimen start date using the score of the first adherence measurement. Adherence scores were carried forward and backfilled no more than 90 days from the temporally closest adherence measurement date. Viral load data were not carried forward.
Statistical analyses were performed using Stata 10.0 (Stata Corporation, College Station, TX). This research was approved by the University of California San Diego Human Subjects Committee (Project No. 040394)
This work was supported in part by the UCSD Center for AIDS Research (AI 36214) and by the CFAR-Network of Integrated Clinical Systems (AI067039). The funding agencies had no role in the study design; collection, analysis, or interpretation of the data; manuscript preparation; or decision to submit the work for publication.
- Mathews WC, Mar-Tang M, Ballard C, Colwell B, Abulhosn K, Noonan C, Barber RE, Wall TL: Prevalence, predictors, and outcomes of early adherence after starting or changing antiretroviral therapy. AIDS Patient Care STDS. 2002, 16 (4): 157-172. 10.1089/10872910252930867View ArticlePubMed
- Chesney MA, Ickovics JR, Chambers DB, Gifford AL, Neidig J, Zwickl B, Wu AW: Self-reported adherence to antiretroviral medications among participants in HIV clinical trials: the AACTG adherence instruments. Patient Care Committee & Adherence Working Group of the Outcomes Committee of the Adult AIDS Clinical Trials Group (AACTG). AIDS Care. 2000, 12 (3): 255-266. 10.1080/09540120050042891View ArticlePubMed
- Chesney MA: The elusive gold standard. Future perspectives for HIV adherence assessment and intervention. J Acquir Immune Defic Syndr. 2006, 43 (Suppl 1): S149-155.View ArticlePubMed
- Hessling RM, Traxel NM, Schmidt TJ: Ceiling Effect. The SAGE Encyclopedia of Social Science Research Methods. Edited by: Lewis-Beck MS, Bryman A, Futing Liao T. 2004, 1: Thousand Oaks, CA: Sage Publications,
- Lu M, Safren SA, Skolnik PR, Rogers WH, Coady W, Hardy H, Wilson IB: Optimal Recall Period and Response Task for Self-Reported HIV Medication Adherence. AIDS Behav. 2007,
- Pearson CR, Simoni JM, Hoff P, Kurth AE, Martin DP: Assessing antiretroviral adherence via electronic drug monitoring and self-report: an examination of key methodological issues. AIDS Behav. 2007, 11 (2): 161-173. 10.1007/s10461-006-9133-3View ArticlePubMed
- Berg KM, Arnsten JH: Practical and conceptual challenges in measuring antiretroviral adherence. J Acquir Immune Defic Syndr. 2006, 43 (Suppl 1): S79-87.PubMed CentralView ArticlePubMed
- Bangsberg DR: Monitoring adherence to HIV antiretroviral therapy in routine clinical practice: The past, the present, and the future. AIDS Behav. 2006, 10 (3): 249-251. 10.1007/s10461-006-9121-7View ArticlePubMed
- King MF, Bruner GC: Social Desirability Bias: A Neglected Aspect of Validity Testing. Psychology and Marketing. 2000, 17 (2): 79-103. 10.1002/(SICI)1520-6793(200002)17:2<79::AID-MAR2>3.0.CO;2-0.View Article
- Simoni JM, Kurth AE, Pearson CR, Pantalone DW, Merrill JO, Frick PA: Self-report measures of antiretroviral therapy adherence: A review with recommendations for HIV research and clinical management. AIDS Behav. 2006, 10 (3): 227-245. 10.1007/s10461-006-9078-6PubMed CentralView ArticlePubMed
- Crowne DP, Marlowe D: A new scale of social desirability independent of psychopathology. J Consult Psychol. 1960, 24: 349-354. 10.1037/h0047358View ArticlePubMed
- Morisky DE, Ang A, Sneed CD: Validating the effects of social desirability on self-reported condom use behavior among commercial sex workers. AIDS Educ Prev. 2002, 14 (5): 351-360. 10.1521/aeap.14.6.351.24078View ArticlePubMed
- Ewald B: Post hoc choice of cut points introduced bias to diagnostic research. J Clin Epidemiol. 2006, 59 (8): 798-801. 10.1016/j.jclinepi.2005.11.025View ArticlePubMed
- Royston P, Altman DG, Sauerbrei W: Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006, 25 (1): 127-141. 10.1002/sim.2331View ArticlePubMed
- Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Moher D, Rennie D, de Vet HC, Lijmer JG: The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med. 2003, 138 (1): W1-12.View ArticlePubMed
- Williams BA, Mandrekar JN, Mandrekar SJ, Cha SS, Furth AF: Finding Optimal Cutpoints for Continuous Covariates with Binary and Time-to-Event Outcomes. Technical Report Series. 2006, 79: Rochester, Minnesota: Mayo Foundation,
- Royston P, Sauerbrei W: Multivariable modeling with cubic regression splines: A principled approach. The Stata Journal. 2007, 7 (1): 45-70.
- Lane P: Handling drop-out in longitudinal clinical trials: a comparison of the LOCF and MMRM approaches. Pharm Stat. 2007,
- Liu H, Golin CE, Miller LG, Hays RD, Beck CK, Sanandaji S, Christian J, Maldonado T, Duran D, Kaplan AH, Wenger NS: A comparison study of multiple measures of adherence to HIV protease inhibitors. Ann Intern Med. 2001, 134 (10): 968-977.View ArticlePubMed
- Owen Clinic Antiretroviral Medication Adherence Survey.http://health.ucsd.edu/owenclinic/PatientSurveyForms.html
- Hosmer DW, Lemeshow S: Assessment of Model Adequacy. Applied Survival Analysis: Regression Modeling of Time to Event Data. 1999, 196-240. New York: John Wiley and Sons,
- Fitzmaurice GM, Laird NM, Ware JH: Applied Longitudinal Analysis. 2004, New York: John Wiley and Sons,
- Hardin J, Hilbe J: Generalized Linear Models and Extensions. 2001, College Station, TX: Stata Press,
- Twisk JW: Longitudinal data analysis. A comparison between generalized estimating equations and random coefficient analysis. Eur J Epidemiol. 2004, 19 (8): 769-776. 10.1023/B:EJEP.0000036572.00663.f2View ArticlePubMed
- Sauerbrei W, Royston P, Binder H: Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat Med. 2007, 26 (30): 5512-5528. 10.1002/sim.3148View ArticlePubMed
- Mazumdar M, Glassman JR: Categorizing a prognostic variable: review of methods, code for easy implementation and applications to decision-making about cancer treatments. Stat Med. 2000, 19 (1): 113-132. 10.1002/(SICI)1097-0258(20000115)19:1<113::AID-SIM245>3.0.CO;2-OView ArticlePubMed
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.