james presentation - holbrook et al

CATCH-IT Journal ClubHolbrook et al. Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial. CMAJ. 2009 Jul 7;181(1-2):37-44

James MullenOctober 19th, 2009

HAD 5726

Holbrook et al, CMAJ, 2009 2

Outline

• Background and context of the paper• Methodological issues

– The good, the bad, and the ugly!

• Questions for the authors


The Paper

• Purpose:“There have been few randomized trials to confirm that computerized decision support systems can reliably improve patient outcomes”

• Interesting to:– eHealth professionals– Healthcare decision makers– Policy makers– Clinicians– Consumers (ie. patients with chronic diseases)


Study investigators

• Anne Holbrook (principal investigator), Division of ClinicalPharmacology and Therapeutics, McMaster University, Hamilton, Ont.• Lisa Dolovich, Department of Family Medicine, McMaster University,Hamilton, Ont.• Hui Lee (deceased), Group Health Centre, Sault Ste. Marie, Ont.• Robert Bernstein, Department of Family Medicine, University ofToronto, Toronto, Ont.• David Chan, Department of Family Medicine, McMaster University,Hamilton, Ont.• Hertzel Gerstein, Department of Medicine, McMaster University,Hamilton, Ont.• Dereck Hunt, Department of Medicine, McMaster University, Hamilton,Ont.• Rolf Sebaldt, Department of Medicine, McMaster University, Hamilton,Ont.• Karim Keshavjee, InfoClin, Toronto, Ont.• Lehana Thabane, Department of Clinical Epidemiology


Principle author

• Dr. Anne Holbrook MD, PharmD, B.Sc, M.Sc, FRCPC, FISPE

Holbrook AM, Thabane L, Shcherbatykh IY, O'Reilly D. E-health interventions as complex interventions: improving the quality of methods of assessment. AMIA.Annu.Symp.Proc. 2006:952.

Mollon B, Chong J,Jr, Holbrook AM, Sung M, Thabane L, Foster G. Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med.Inform.Decis.Mak. 2009 Feb 11;9:11.

Mollon B, Holbrook AM, Keshavjee K, Troyan S, Gaebel K, Thabane L, et al. Automated telephone reminder messages can assist electronic diabetes care. J.Telemed.Telecare 2008;14(1):32-36.

Shcherbatykh I, Holbrook A, Thabane L, Dolovich L, COMPETE III investigators. Methodologic issues in health informatics trials: the complexities of complex interventions. J.Am.Med.Inform.Assoc. 2008 Sep-Oct;15(5):575-580.


Hypothesis

“The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care”


Patients

• Diagnosed with Diabetes Mellitus• >18 years of age• Fluent in English• Rostered with a community-based primary care

providersn = 1610

n = 511

n = 258 n = 253

Total patients identified

Total patients eligible

ControlIntervention


Intervention

Web-based diabetes tracker of the ‘Computerization of Medical Practices for the Enhancement of Therapeutic Effectiveness study II (COMPETE II).

– Interfaces with the providers EMR– Interfaces with an automated telephone reminder system


Intervention

COMPETE II

EMR

Telephone

5 different types of EMRs


Patient Screen

*http://www.cmaj.ca/cgi/data/cmaj.081272/DC2/2


Physician Screen



Intervention

• 13 variables selected by a clinical subcommittee– Based on practices by CDA, ADA, and internal literature

review

• Mailed out tracker page 4x/year (twice per patient)

• Phone reminders once a month• Tracker updated nightly

– Patients had the option of personal data entry– Trend data was available


Context

COMPETE I: focused on successful implementation of EMRs in small, community-based primary care offices

COMPETE II: this study

COMPETE III: broadened to vascular risk: diabetes, hypertension, cholesterol, previous heart attacks or stroke.


Context

•COMPETE II considered a Pilot project

*“A deliberate goal of the project was to have patients regularly visit their family physician”

*http://www.hc-sc.gc.ca/hcs-sss/pubs/chipp-ppics/2003-compete/final-eng.php


Methods

1. Recruit primary care providers who were already using EMRs in their practices

2. Have providers identify patients3. Selection of patient participants

• Allocation concealment and central computer generation of group assignment

4. Participants and providers complete baseline questionnaires

5. Patients in intervention sent for set of relevant blood tests (pre-determined) then make appt with FP. (NOT CONTROL GROUP)


Timelines

• Study conducted from late 2002 to the end of 2003

• Study follow up was 6 months (mean follow up was 5.9 months)


Scores

• Process composite score– Sum of the quality monitoring of the 8 variables

• Glycated hemoglobin; BP; LDL cholesterol; Albuminuria; BMI; Foot surveillance; Exercise; Smoking; ABC composite

• Clinical composite score (8 variables)– Clinical targets for the composite scores

• E.g., BP < 130/80 mm Hg


Analysis

• Alpha = 0.05• 2-tailed test (1:1 allocation)• Used t tests to assess the difference between

groups in terms of change in the process composite score


Analysis

• They analyzed clinical outcomes in 2 ways:– Change in variable (+1 if positive, 0 if no change, -1 if

negative change)– Whether outcome met predefined targets


Process Outcomes

• Improvement for intervention group for 7 out of the 9 variables (no improvement in exercise of smoking)

“Number of visits to primary care provider increased significantly more in the intervention group than in the control group (diff of 0.66, 95% CI 0.37 to 1.02. P< 0.001).”


Clinical Outcomes

• Intervention had an improvement on a number of clinical composite variables that were on target (though not significantly significant!)

• Statistically significant improvements in BP and glycated hemoglobin


Other Outcomes

• Patients were more ‘optimistic’• 75.9% of intervention patients were ‘more

satisfied’ with their care• No statistically significant changes in quality-of-

life measures (SF-12 and Diabetes-39)


Interpretations

“Care of complex chronic disease can be improved with electronic tracking and decision support shared by patients and providers”


Hypothesis validated?

“The specific hypothesis was that patients in the intervention group, who had electronic and paper access to an individual diabetes tracker (with data related to recent monitoring and results and targets for 13 risk factors) and whose information was shared with their primary care providers, would have improved quality of diabetes care”

Is the result valid?


What can:

• Health professionals learn from this?• Consumers learn from this?• Policy makers learn from this?• Researchers learn from this?


Thoughts…

• Overall… this was a useful in that it adds to the literature base. But I want to know if the benefits outweigh the investment in time, money, and resources.


What others are saying

*Pros:•Successful implementation of an RCT without major flaws such as:

– temporal trends– participatory bias (Hawthorne effect)

•Environment of the study– Not based in a research hospital

*Grant RW, Middleton B. Improving primary care for patients with complex chronic disease: can health information technology play a role? CMAJ. 2009 Jul 7;181(1-2):37-44.


What others are saying

*Cons:•“The authors ascribe the positive impact of their intervention to the influence of individualized decision support and to the role (again) of reminders”•Generalizability

*Grant RW, Middleton B. Improving primary care for patients with complex chronic disease: can health information technology play a role? CMAJ. 2009 Jul 7;181(1-2):37-44.


Class comments

• More than 50% of participants never used computers or Internet.

• What about the patients’ view?• Usability of the tracker• Did they educate the participants in using

computers and the tracker?• Is there any data on the usage amount on the

tracker by the participants? (interaction)• How was composite score validated?


Questions

• If the control group had as many physicians visits as the Intervention group, would there be a difference in care?

• How would this system improve care with at least a neutral effect on time. (follow up time was only 6 months, what about 2 years?)

• Were the results (improved score) from the decision support system or from the increase in physician visits - better coordination of care? In other words, what was the causal agent? Decision support? Reminders to patient?


Questions

• What about the difficulties involved with integrating with COMPETE II with the PCP’s EMRs. What about the technical difficulties.

• 51% of the Intervention group NEVER used the internet. Did this have an effect on the results?

• Scoring was +1 if there was improvement, 0 with no change, and -1 without change. – How much improvement was measured? Was this

improvement a result of the decision support (COMPETE II), automated reminders, or coincidence?


Questions

• Why didn’t the researchers do a repeated measures design t test.

• Why didn’t the researchers do baseline lab tests on the control group

• Qualitative data was not explained? Was it anecdotal or statistically calculated?

• How did they measure if patients were actually using the system?– Attrition, adoption, acceptance, and user not really

addressed


The end


Questions

• What about confounders:– Age– Education– Socio-economic status– Region (three regions used in the study)– Morbidity

• What about negative outcomes?• Why did patients dropout

james presentation - holbrook et al

Health & Medicine