Developing quality indicators for the care of HIV-infected pregnant women in the Dutch Caribbean

Background Effective interventions to prevent mother-to-child HIV transmission (PMTCT) exist and when properly applied reduce the risk of vertical HIV transmission. As part of optimizing PMTCT in the Dutch Caribbean we developed a set of valid and applicable indicators in order to assess the quality of care in HIV-infected (pregnant) women and their newborns. Methods A multidisciplinary expert panel of 19 experts reviewed and prioritized recommendations extracted from locally used international PMTCT guidelines according to a 3-step-modified-Delphi procedure. Subsequently, the feasibility, sample size, inter-observer reliability, sensitivity to change and case mixed stability of the potential indicators were tested for a data set of 153 HIV-infected women, 108 pregnancies of HIV-infected women and 79 newborns of HIV-infected women in Aruba, Curaçao and St Maarten from 2000 to 2010. Results The panel selected and prioritized 13 potential indicators. Applicability could not be tested for 4 indicators regarding HIV-screening in pregnant women because of lack of data. Four indicators performed satisfactorily for Curaçao ('monitoring CD4-cell count', 'monitoring HIV-RNA levels', 'intrapartum antiretroviral therapy and infant prophylaxis if antepartum antiretroviral therapy was not received', 'scheduled caesarean delivery') and 3 for St Maarten ('monitoring CD4-cell count', 'monitoring HIV-RNA levels', 'discuss and provide combined antiretroviral therapy to all HIV-infected pregnant women') whilst none for Aruba. Conclusions A systemic evidence-and consensus-based approach was used to develop quality indicators in 3 Dutch Caribbean settings. The varying results of the applicability testing accentuate the necessity of applicability testing even in, at first, comparable settings.


Background
Acquired immunodeficiency syndrome (AIDS) is a leading cause of illness and death among women and children in countries with high rates of human immunodeficiency virus (HIV) infection [1]. Mother-To-Child HIV Transmission (MTCT) is by far the most significant route of HIV-infection in children. Several interventions have proven to be effective in reducing MTCT, including elective caesarean delivery [2,3], substitution of breastfeeding [4][5][6] and access to antiretroviral therapy during pregnancy, labour and post-partum [7]. If properly applied, these interventions reduce the MTCT rates to 2% [8,9].
In the Netherlands Antilles, 1812 HIV-1-cases were reported in 2008, with 83 new cases in 2007. The Dutch Caribbean consists of Aruba and the Netherlands Antilles (Saba, St Eustatia, Bonaire, St Maarten and Curaçao) and has an estimated prevalence of HIV-1-infection of 0.61%-1.05% in the adult population [10]. Forty percent of the registered patients are female and there have been approximately 5 to 10 pregnancies in HIV-infected women annually.
Since 1996 guidelines regarding the prevention of mother-to-child HIV transmission (PMTCT) have been implemented in regular health care systems in the Dutch Caribbean and the annual number of paediatric HIV-cases has dropped dramatically since [10]. However, new paediatric HIV-cases have been reported in recent years. Limited data on the quality of care provided after implementation of the guidelines are available and the question rises as to whether opportunities for the prevention of HIV transmission were missed. Monitoring and evaluating the quality of care in HIVinfected women to achieve PMTCT is important as it can identify strategies to improve the quality of care provided and thereby lead to a better outcome in the prevention of HIV transmission [11]. As part of optimizing the quality of prenatal and delivery care in HIVinfected (pregnant) women in the Dutch Caribbean, this study aims to develop a validated and applicable set of quality indicators to measure the quality of care in HIVinfected (pregnant) women and their newborns in 3 Dutch Caribbean settings; Aruba, Curaçao and St Maarten.

Phase 1: Consensus procedure
Locally used PMTCT guidelines, including guidelines for care of HIV-infected pregnant women, were collected from which a hundred key recommendations were preselected by three independent researchers. An extensive literature search was performed using PubMed to identify already existing quality of care indicators for the care of HIV-infected pregnant women. On the basis of the available literature, the level of evidence was graded [12] for each recommendation to determine its scientific soundness or the likelihood that improvement of the quality indicator reflects improvements in quality of care [13]. (Table 1) According to a 3-step-Delphiapproach the group judgement of experts was used to assess the validity of the preselected recommendations [14]. During 3 rating rounds an expert panel rated the preselected recommendations by judging their relevance with regard to effectiveness of the intervention related to PMTCT, the applicability of the recommendation for the current setting, and health care costs [15][16][17]. The multidisciplinary expert team consisted of 19 experts: 3 paediatricians, 3 gynaecologists, 3 midwifes, 2 general practitioners, 2 epidemiologists, 3 internal medicine specialists, 2 HIV/AIDS programme managers and 1 microbiologist. After the selection and prioritization the recommendations were further developed as potential indicator by defining its numerator and denominator.

Phase 2: Applicability test of potential quality indicators
Before the indicator set is used in a specific setting, its applicability in the chosen practice setting has to be tested. The next step is therefore to provide empirical evidence of the feasibility, sample size, reliability, sensitivity to change and case mix stability of each indicator. (Figure 1) Since national registries of pregnancies are not available in the Dutch Caribbean, the applicability testing of the set of potential indicators was limited to the outpatient clinical setting of the HIV specialists and the clinical setting of the general hospitals in Aruba, Curaçao and St Maarten. Eligible patients included HIV-infected women of childbearing age, HIV-infected pregnant women, and exposed children between January 2000 and January 2010. Data were selected by using clinical data systems of the general hospitals, the outpatient clinic of the gynaecologists, paediatricians, HIV specialists and national registries available at the Public Health Department of each island. In Curaçao, a national electronic registration system (Stichting HIV Monitoring, SHM) [18] was consulted and in Aruba, the national registration database of the Services of Contagious Diseases, Public Health Department was used to select HIV-infected women of childbearing age. In St Maarten, no electronic database was available, therefore no patient selection could be made for indicators regarding HIVinfected women of childbearing age. Non-electronic Systematic review of randomized controlled trials.

A2
One or more rigorous studies designed to answer the question but not formally combined.
Randomized controlled trial.

B
One or more prospective clinical studies that illuminate but do not rigorously answer the question.
Prospective cohort study; unpowered or poor quality randomized controlled trial; or nonrandomized controlled trial. registrations conducted by health care workers were also consulted in the 3 settings. Excluded from analysis were pregnancies ending before the second trimester, pregnancies ending in abortion with unknown gestation duration, or deliveries abroad.

Feasibility
of the indicator was defined as the availability of administrative data required to evaluate the indicator. An indicator was considered to be feasible if the data necessary to score the indicator could be abstracted from the available data for > 70% of the cases [19].

Sample size
of the indicator was related to the number of patients to whom the indicator could be applied. Considering the existing literature, the period and the estimated number of patients or events eligible for this study, the research team considered an indicator to be applicable if it could be applied to at least 15 patients or events based on consensus rather than statistical analysis.
Inter-rater reliability refers to the extent in which a measurement of an indicator is reproducible, between observers and between cases. A second investigator rated 10% of all the records in the 3 different medical centres to assess the interrater reliability. To assess the agreement between 2 investigators corrected for chance, a Cohen kappa coefficient was calculated. Indicators with a value of < 0.60 were considered unreliable [20].

Sensitivity to change
was defined as the need to detect changes in the quality of care in order to discriminate between and within subjects hence showing the possibilities for improvement in the present care. Potential indicators with an overall performance score of > 85% were defined as having little room for improvement and were not selected [21].
Case mix stability referred to the need for the correction of certain patient characteristics. The relationship between patient parameters and the indicator result can identify whether there is need for correction for case mix. Indicators that are not case mix stable require comparable patient populations when comparing the quality of care. Patient characteristics possibly influencing the quality of care were defined as: type of health care insurance, age, not born in the Dutch Caribbean and number of previous deliveries. Outcome of the indicator was supposed to be influenced by the patient characteristic if the p < 0.05. Correction of these patient characteristics was performed and analysed if the characteristics were of influence to the outcome of the indicator.

Phase 1: Consensus procedure
Of the in total 19 panel members, 15 panellists (79%) completed the questionnaire in the first round, 15 panellists (79%) completed the second round and 10 panellists (53%) were present during the panel meeting. After the first rating round 57 recommendations were rated as potential indicators. (Figure 1) Nine recommendations were considered not-suitable as potential indicators. Thirty-three recommendations were discussed and reformulated during the panel meeting. Two recommendations were added. More than 200 comments were added, encoded and grouped by the research team for discussion during the panel meeting. After the second rating round 28 recommendations were selected as potential indicators and 7 recommendations were rejected. A final set of 13 recommendations was prioritized for which numerators and denominators were defined. (Table 2) Phase 2: Applicability test of potential quality indicators

Feasibility
Indicator 5 ('preconception counselling for all HIVinfected women') had a low feasibility for Curaçao (18% of patients had available data) and moderate feasibility for Aruba (59%). Indicator 6 ('maximally suppress viral load in HIV-infected women who wish to get pregnant') scored low feasibility in Curaçao and Aruba (15% and 17% respectively). In St Maarten feasibility for indicator 5 and indicator 6 could not be assessed, as there was no data set of HIV-infected women of childbearing potential. Indicators 7 to 11 were feasible in all 3 settings, and indicator 12 and 13 were exclusively feasible in Aruba. (Table 2) Sample size In Curaçao, indicator 6 ('maximally suppress viral load in HIV-infected women who wish to get pregnant') had a sample size of < 15 patients and was therefore rejected. All other indicators had large enough sample sizes for Curaçao. In Aruba only indicator 5 ('preconception counselling for all HIV-infected women') met the required sample size. In St Maarten, indicator 10 ('HIV-infected pregnant women who do not receive antiretroviral therapy antepartum') and indicator 11 ('scheduled caesarean section') could only be applied to 11 patients.

Sensitivity to change
None of the potential indicators showed an overall high performance score. The performance of indicator 12 and 13 scored higher than 85% in St Maarten and indicator 12 scored higher than 85% in Aruba. The range between the highest and the lowest score of each indicator between the different settings was high for the indicators 5, 11, 12, and 13 (48%, 33%, 43% and 60% respectively).

Case mix stability
In St Maarten correction for multiparous women was necessary for indicator 12 ('counselling breastfeeding'). This indicator was more often measured in HIVinfected pregnant women with 2 or more pregnancies in the past than women with none or 1 pregnancy. No correction for type of health care insurance, age, or not born in the Dutch Caribbean was necessary for the other potential indicators. No consensus exists on how to best monitor the quality of care in HIV-infected pregnant women [22]. Most international studies report effectiveness of PMTCT services in a country or region by outcome or access to care (indicating the percentage of children infected or the percentage of HIV-infected pregnant women accessing PMTCT services) [11,[22][23][24][25][26][27]. However, in order to reach the global goal of eliminating MTCT, monitoring the quality of the process of care seems to be as equally important as ensuring access especially in countries or regions that have already achieved high access to PMTCT services.

Discussion
Several organizations and study groups have developed indicators regarding the care of HIV-infected pregnant women, mostly as part of a set of key indicators to measure the effectiveness of the implementation of a regional PMTCT program [28][29][30][31][32][33][34][35]. Five of such indicators, are process indicators, and show similarity to the quality indicators in our study namely indicator 1 ('HIV screening in all pregnant women'), indicator 5 ('preconception counselling'), indicator 9 ('antiretroviral therapy in all HIV-infected pregnant women'), indicator 12 ('counseling breastfeeding') and indicator 13 ('antiretroviral therapy in newborn'). Remarkably however, most of these well-known and internationally used indicators are currently not applicable in a Dutch Caribbean setting because they currently show lack of feasibility, interrater reliability or small sample sizes.
This study shows the importance of testing potential indicators for their applicability which has also been reported by others [21]. After assessing the applicability of each indicator in the 3 Dutch Caribbean settings, only 4 indicators could be satisfactorily tested in practice in Curaçao, 3 in St Maarten whilst none in Aruba. Firstly, applicability can only be tested if data are available to give information about the quality of care. In this study the indicators concerning HIV-infected women (indicator 5 and 6) and the indicator concerning newborns (indicator 13) showed low feasibility. For indicators with low feasibility it cannot be concluded that the limitation of data are due to improper data registration or incorrect implementation of the used guidelines. Proper surveillance, tracking systems or registration tools for collecting the necessary data should therefore be developed and made available before these quality indicators can be applied in the Dutch Caribbean setting. Secondly, sizes of the samples on which the indicator operates have to be large enough. In small settings or in settings with low prevalence of HIV infection or with highly specific quality indicators accounting for only a specific proportion of the population, quality indicators cannot be used because of insufficient number of patients. This was evident in our study of the Aruban setting where only one indicator had a large enough sample size over a period of 10 years. Lowering the number of a sample size limits the statistical analyses necessary to develop the indicator and the statistical power when using the indicator in practice. The practical implication of limited statistical power is that patients and policymakers may not be able to properly identify quality problems in the clinical setting [36]. Given the limited usefulness of quality indicators in small populations it is worth considering additional approaches for judging quality of care in HIV infected (pregnant) women and their infants. The first approach would seem increment of sample size number by lengthening the time of the measurement, however this is not desirable since indicators should be dynamic over time. A second approach could be to provide more detailed information of the processes of care like review of complications [36] or case reporting. As the Caribbean region consists of multiple islands with relatively small populations like the Dutch Caribbean, the practical value of (specific) quality indicators for the region has to be questioned and a combination of methods of monitoring quality of care should be considered.
This study gives an overview of prenatal, delivery and child care in regard to PMTCT in 3 Dutch Caribbean islands. The study has led to identification of previously non-registered HIV-infected pregnancies and HIVexposed children. It also created awareness of the quality of care regarding PMTCT and enhanced the possibilities for further discussion among health care professionals who are involved in planning and coordinating care. Although the applicability of some potential indicators was limited by overall small sample sizes and lack of feasibility one should note that the set of potential indicators had an overall low performance score. Only 2 indicators scored higher than 85%. Future initiatives aimed at improving the quality of care and eliminating the vertical transmission of HIV-infection in Curaçao, Aruba and, St Maarten should therefore be based on these study results.
Since access to HIV treatment has increased worldwide, a trend towards reporting on the quality of HIV treatment should be encouraged. To our knowledge this is one of the first reports on the quality of HIV treatment in the Caribbean.
Because pregnancies are currently not officially registered in the Dutch Caribbean, no dataset was available to test the 'screenings indicators' (indicator 1 to 4). This is a limitation of the study since the timely identification of HIV-infection by means of screening is essential in care and treatment of HIV-infected pregnant women. Also, no HIV rapid tests were available in the 3 settings, which may result in underreporting especially for those women presenting in labour with unknown HIV sero-status. Lack of proper screening may have influenced the applicability as well as the outcome of the quality of care provided, since reports show that patients who do not (timely) access proper care have worse outcomes [37][38][39]. Future initiatives to monitor the quality of care in HIVinfected pregnant women in the Dutch Caribbean should include the implementation of an official registration system for pregnancies or a prospective study in which screening patterns in pregnant women will be assessed. Another limitation of the study was that different clinical monitoring systems for HIV-infected patients were available in the 3 settings, none of them aimed at collecting data regarding the quality of care of HIV-infected pregnant women. Although we developed a unique Clinical Report Form for this study specific data may have been missed because data were collected retrospectively.

Conclusion
In conclusion this is one of the first studies describing the systematic development of quality indicators for HIV-infected (pregnant) women. Our study shows the importance of applicability testing before implementing potential indicators even when the settings initially seem to be similar. In relatively small settings or settings with low prevalence, one should consider alternative approaches to monitor the quality of care; for example the reviewing of complications or case reporting. Furthermore, this study identifies areas for improvement in the collection of data and registration as well as areas for improvement in the quality of prenatal and delivery care in HIV-infected (pregnant) women and their newborns in the Dutch Caribbean. all health care workers who performed the consensus procedure. We would like to thank the medical specialists and staff of the participating hospitals for their contribution to the data collection: the gynaecology department, internal medicine department and the paediatric department of the St Elisabeth Hospital of Curaçao, the gynaecology department, the internal medicine department and the paediatric department of the Dr. Horacio E. Oduber Hospital of Aruba and the gynaecology department, the internal medicine department and the paediatric department of the St Maarten Medical Centre of St Maarten. Also, we thank the staff of the Services of Contagious Diseases of Aruba and the Public Health Departments of Curaçao, Aruba and St Maarten. We are exceedingly grateful to M. Hellemonds and M. Jansen for the preparation phase of this study and E. van 6 Stichting HIV Monitoring (SHM), Amsterdam, The Netherlands.