evaluating health information technology: a primer
Post on 14-Jan-2016
42 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Evaluating Health Information Evaluating Health Information Technology: A PrimerTechnology: A Primer
Eric Poon, MD MPHEric Poon, MD MPHClinical Informatics Research and Development,Clinical Informatics Research and Development,
Partners Information SystemsPartners Information Systems
Davis Bu, MD MADavis Bu, MD MACenter for Information Technology Leadership,Center for Information Technology Leadership,
Partners Information SystemsPartners Information Systems
AHRQ National Resource Center for Health AHRQ National Resource Center for Health Information TechnologyInformation Technology
2
Pre-Conference Logistics Pre-Conference Logistics
To Access Slides:To Access Slides: Go to http://extranet.ahrq.gov/rcGo to http://extranet.ahrq.gov/rc
Login with username and passwordLogin with username and password
Follow the links to download slidesFollow the links to download slides
Problems? Email ResourceCenter@norc.org Problems? Email ResourceCenter@norc.org
Q&A Session at the End Q&A Session at the End Dial *1 to ask a questionDial *1 to ask a question
Please pick up handset (not speakerphone)Please pick up handset (not speakerphone)
Note that this teleconference is being recordedNote that this teleconference is being recorded
3
OutlineOutline Why evaluate?Why evaluate?
General Approach to EvaluationGeneral Approach to Evaluation
Deciding what to MeasureDeciding what to Measure
Study Design TypesStudy Design Types
Analytical issues in HIT evaluationsAnalytical issues in HIT evaluations
Some practical advice on specific Some practical advice on specific evaluation techniquesevaluation techniques
4
Why Measure Impact of HIT?Why Measure Impact of HIT?
Impact of HIT often hard to predict Impact of HIT often hard to predict Many “slam dunks” go awryMany “slam dunks” go awry
Understand how to clear barriers to effective Understand how to clear barriers to effective implementationimplementation Understand what works and what doesn’tUnderstand what works and what doesn’t
Justify enormous investmentsJustify enormous investments Return on investmentReturn on investment Allow other institutions to make tradeoffs intelligentlyAllow other institutions to make tradeoffs intelligently
Use results to win over late adoptersUse results to win over late adopters You can’t manage/improve what isn’t measuredYou can’t manage/improve what isn’t measured Good publicity for organizationGood publicity for organization
5
General Approach to Evaluating HITGeneral Approach to Evaluating HIT
Understand your interventionUnderstand your intervention
Select meaningful measuresSelect meaningful measures
Pick the study designPick the study design
Validate data collection methodsValidate data collection methods
Data analysisData analysis
6
Getting Started: Get to know your Getting Started: Get to know your interventionintervention
Clarify the question: What problem does it Clarify the question: What problem does it address?address?
Think about intermediate processesThink about intermediate processes
Identify potential barriers to successful Identify potential barriers to successful implementationimplementation
Identify potential managerial and Identify potential managerial and behavioral process to overcome behavioral process to overcome implementation barriersimplementation barriers
7
Array of MeasuresArray of Measures
Quality and SafetyQuality and Safety Clinical OutcomesClinical Outcomes Clinical ProcessesClinical Processes
KnowledgeKnowledge Patient knowledgePatient knowledge Provider knowledgeProvider knowledge
SatisfactionSatisfaction Patient satisfactionPatient satisfaction Provider satisfactionProvider satisfaction
Resource utilizationResource utilization Costs and chargesCosts and charges LOSLOS Employee time/workflowEmployee time/workflow
8
Introducing the Evaluation ToolkitIntroducing the Evaluation Toolkit
Rough guides on general approach, costs and Rough guides on general approach, costs and potential pitfallspotential pitfalls
Major domains:Major domains: Clinical OutcomesClinical Outcomes Clinical ProcessClinical Process Provider Adoption & Attitudes Provider Adoption & Attitudes
Measure Characteristics:Measure Characteristics: IOM DomainIOM Domain Data SourceData Source Relative CostRelative Cost
Would love to hear your feedbackWould love to hear your feedback
Patient Knowledge & AttitudesPatient Knowledge & Attitudes Workflow ImpactWorkflow Impact Financial ImpactFinancial Impact
Potential PitfallsPotential Pitfalls General NotesGeneral Notes
9
Selecting Evaluation Measures Selecting Evaluation Measures for HIT: for HIT:
Three ExamplesThree Examples
10
Computerized Provider Order Entry Computerized Provider Order Entry (CPOE) Example(CPOE) Example
Clarify the primary question:Clarify the primary question: Does CPOE improve quality of care?Does CPOE improve quality of care?
Competing questions:Competing questions: Does CPOE save money?Does CPOE save money? What are the barriers to physician What are the barriers to physician
acceptance?acceptance? Does CPOE introduce new errors?Does CPOE introduce new errors?
11
CPOE: How can it affect quality?CPOE: How can it affect quality?
Think about intermediate processesThink about intermediate processes Patient data is presented to ordering physicianPatient data is presented to ordering physician ADE alerts may be triggered and presented at the ADE alerts may be triggered and presented at the
point of care (which alerts?)point of care (which alerts?) Guideline reminders may be triggered an presented at Guideline reminders may be triggered an presented at
the point of care (which guidelines?)the point of care (which guidelines?) Medication order is enteredMedication order is entered Medication order is executed by pharmacyMedication order is executed by pharmacy Medication order is executed by nursing staffMedication order is executed by nursing staff
12
Does CPOE Improve Quality of Care?Does CPOE Improve Quality of Care?
Identify measuresIdentify measures
Process MeasureData Presentation (Redundant test ordering)
ADE Alert Alert frequency, ADE frequency
Guideline Reminder
Guideline compliance, clinical outcome
Order Entry Ordering errors
Pharmacy (Time to process order)
Administration Time to administration
13
Evaluating CPOE’s Impact on QualityEvaluating CPOE’s Impact on Quality
Select Appropriate MethodologySelect Appropriate Methodology Does existing data exist that can be Does existing data exist that can be
leveraged? (e.g. ongoing QA activities)leveraged? (e.g. ongoing QA activities) Does concurrent control exist?Does concurrent control exist? How will the data be analyzed?How will the data be analyzed?
14
Electronic Medical Records (EMR) Electronic Medical Records (EMR) ExampleExample
Clarify the primary question:Clarify the primary question: What are the barriers and facilitators to What are the barriers and facilitators to
effective EMR implementation?effective EMR implementation?
Competing questions:Competing questions: Do EMRs save money?Do EMRs save money? Do EMRs improve quality of care?Do EMRs improve quality of care? Do EMRs introduce new errors?Do EMRs introduce new errors?
15
EMR: Dissecting the EMR EMR: Dissecting the EMR Implementation ProcessImplementation Process
Identify stakeholdersIdentify stakeholders Providers, et al.Providers, et al.
Catalogue stakeholder interests and valuesCatalogue stakeholder interests and values Workflow efficiencyWorkflow efficiency
Clarify stakeholder role in implementationClarify stakeholder role in implementation Users of system, clinical leaders, administrative leadersUsers of system, clinical leaders, administrative leaders
Clarify impact of Implementation on clinical Clarify impact of Implementation on clinical processesprocesses User interface optimization, workflow re-engineeringUser interface optimization, workflow re-engineering
Define implementation success criteriaDefine implementation success criteria Provider buy-in, provider use and acceptanceProvider buy-in, provider use and acceptance
16
EMR: Understanding the Barriers EMR: Understanding the Barriers and Facilitators to Implementationand Facilitators to Implementation
Identify measuresIdentify measuresProcess MeasureStakeholder attitudes
Attitude/Satisfaction surveys Readiness Assessment Staff Turnover
Workflow Efficiency metrics
Process improvements
Staffing levels Patient flow Practice productivity Implementation participation from staff
Success Criteria Usage data Training attendance Measures listed above
17
EMR: Understanding the Barriers EMR: Understanding the Barriers and Facilitator to Implementationand Facilitator to Implementation
Select Appropriate MethodologySelect Appropriate Methodology Combination of quantitative and qualitative Combination of quantitative and qualitative
studiesstudies Example: efficiency measures:Example: efficiency measures:
Time motion studies: how did the system affect provider Time motion studies: how did the system affect provider efficiency?efficiency?
Attitude Surveys: How did the system affect provider Attitude Surveys: How did the system affect provider perception of efficiency?perception of efficiency?
Semi-structured interviews: How did the implementation Semi-structured interviews: How did the implementation affect stakeholder workflow? Did that effect change over time affect stakeholder workflow? Did that effect change over time and why?and why?
18
Local Health Information Local Health Information Infrastructure (Laboratory)Infrastructure (Laboratory)
Clarify the primary questionClarify the primary question Can LHIIs for labs generate a positive ROI?Can LHIIs for labs generate a positive ROI?
Competing questions:Competing questions: Can LHIIs for labs improve quality of care?Can LHIIs for labs improve quality of care? Which architecture is best suited for LHIIs for Which architecture is best suited for LHIIs for
labs?labs? How do LHIIs for labs affect provider and How do LHIIs for labs affect provider and
patient perception of the health care system?patient perception of the health care system?
19
LHII (Laboratory): Defining the ROILHII (Laboratory): Defining the ROI
Specify intermediate processesSpecify intermediate processes Data is pulled from local laboratoriesData is pulled from local laboratories
Previous labs pulledPrevious labs pulled Lab order enteredLab order entered Lab order transmittedLab order transmitted Administrative handlingAdministrative handling Lab results reportedLab results reported Lab results recordedLab results recorded
Data is pulled from primary providerData is pulled from primary provider Authorization and payment is coordinated with payerAuthorization and payment is coordinated with payer Implementation of LHIOImplementation of LHIO
20
LHII (Laboratory): Defining the ROILHII (Laboratory): Defining the ROI
Identify associated measuresIdentify associated measures
Process MeasureProvider requests data
Volume of requests
Data is pulled Chart pulls, time for chart pulls, administrative costs
Provider interprets data
Amount of missing information
Provider Orders test Volume of redundant tests
21
LHII (Laboratory): Evaluating the ROILHII (Laboratory): Evaluating the ROI
Select Appropriate MethodologySelect Appropriate Methodology Does concurrent control exist?Does concurrent control exist? Are there ongoing trends over time?Are there ongoing trends over time? How will the data be analyzed?How will the data be analyzed?
22
Selecting Outcome Measures: Selecting Outcome Measures: General CommentsGeneral Comments
Generally want to pick 1-3 outcomes of primary interestGenerally want to pick 1-3 outcomes of primary interest If choose more, need to make correction (e.g. Bonferroni)If choose more, need to make correction (e.g. Bonferroni)
Outcome must be sufficiently frequent to be detectableOutcome must be sufficiently frequent to be detectable Rare events such as adverse events due to errors particularly Rare events such as adverse events due to errors particularly
challengingchallenging
Important enough to provoke interestImportant enough to provoke interest Whether study is positive or negativeWhether study is positive or negative How would the results change policy (local or national)?How would the results change policy (local or national)?
Process vs. outcomeProcess vs. outcome Legitimate to measure processLegitimate to measure process
Outcome often takes too longOutcome often takes too long In many situations link between process, outcome clearIn many situations link between process, outcome clear
23
Study TypesStudy Types
Commonly used study types:Commonly used study types: Before-and-after time series TrialsBefore-and-after time series Trials Randomized Controlled TrialsRandomized Controlled Trials Factorial DesignFactorial Design
Study design often influenced by Study design often influenced by implementation planimplementation plan
24
Time Series vs. Randomized Time Series vs. Randomized Controlled TrialsControlled Trials
Before-and-after trial common in informaticsBefore-and-after trial common in informatics Concurrent randomization is hardConcurrent randomization is hard Don’t lose the opportunity to collect baseline data!Don’t lose the opportunity to collect baseline data!
Off-On-Off trial design possibleOff-On-Off trial design possible But may not be politically/ethically acceptable to turn But may not be politically/ethically acceptable to turn
off a highly used featureoff a highly used feature
RCT preferable if feasibleRCT preferable if feasible Eliminates the issue of secular trendEliminates the issue of secular trend Balance of baseline confoundingBalance of baseline confounding
25
Randomization ConsiderationsRandomization Considerations
Justifiable to have a control arm as long as Justifiable to have a control arm as long as benefit not already demonstrated (usual care)benefit not already demonstrated (usual care)
Want to choose a truly random variable Want to choose a truly random variable Not day of the weekNot day of the week Legitimate to stratify on baseline variables (e.g. Legitimate to stratify on baseline variables (e.g.
education for pt, computer experience for providers)education for pt, computer experience for providers) Minimal number of armsMinimal number of arms
More arms, less powerMore arms, less power Strongest possible interventionStrongest possible intervention
26
Unit of RandomizationUnit of Randomization
PatientsPatients PhysiciansPhysicians Practices/wardsPractices/wards
27
Randomization Unit:Randomization Unit:How to Decide?How to Decide?
Small units (patients) vs. Large units (practices Small units (patients) vs. Large units (practices wards)wards) Contamination across randomization unitsContamination across randomization units If risk of contamination is significant, consider If risk of contamination is significant, consider
larger unitslarger units Effect contamination-can underestimate impactEffect contamination-can underestimate impact
However, if you see a difference, impact is presentHowever, if you see a difference, impact is present
Randomization by patient generally undesirableRandomization by patient generally undesirable ContaminationContamination Ethical concernEthical concern
28
Randomization Schemes:Randomization Schemes:Simple RCTSimple RCT
Burn-in periodBurn-in period Give target population time to get used to new Give target population time to get used to new
intervention intervention Data not used in final analysisData not used in final analysis
XX Clinics
Baseline Period
Baseline Data Collection Data Collection for RCT
No Intervention
Intervention Period
3 month burn-in period
Intervention Deployed
Intervention
arm
Control
arm
Control arm gets intervention
Post- Intervention
Period
29
Randomization schemes: Randomization schemes: Factorial DesignFactorial Design
May be used to May be used to concurrently evaluate concurrently evaluate more than one more than one intervention:intervention: Assess interventions Assess interventions
independently and in independently and in combinationcombination
Loss of statistical powerLoss of statistical power
Usually not practical for Usually not practical for more than 2 interventionsmore than 2 interventions
Control (no interventions)
A
B
A+B
30
Randomization Schemes:Randomization Schemes:Staggered DeploymentStaggered Deployment
AdvantagesAdvantages Easier for user education and trainingEasier for user education and training Can fix IT problems up frontCan fix IT problems up front
Need to account for secular trendNeed to account for secular trend Time variable in regression analysisTime variable in regression analysis
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4Pilot Practice B x Practice A-Intervention B x Practice B-Control B Practice C-Intervention B x Practice D-Control B Practice E-Intervention B x Practice F-Control B Practice G-Intervention B x Practice H-Control B
2003 2004 2005
Deployment of InterventionB Baseline Data Collectionx Burn-in period
31
Randomization Schemes:Randomization Schemes:Multiple InterventionsMultiple Interventions
Time efficient designTime efficient design Every clinic gets something. (Keeps clinics and IRB Every clinic gets something. (Keeps clinics and IRB
happy)happy) Watch out for cross-arm intervention contaminationWatch out for cross-arm intervention contamination
Arm 2
Arm 1
12clinics
6 clinics
6 clinics
18 mo
Randomize
Medication Tracking; Diabetes Care
Prev Care Reminders;Family History
Control for Arm 1
Control for Arm 2
4 Interventions involving patient’s use of shared online medical records:
• Medication Tracking• Diabetes Care• Prev. Care Reminders• Family History
32
Inherent Limitations of RCTs in Inherent Limitations of RCTs in InformaticsInformatics
Blinding is seldom possibleBlinding is seldom possible
Effect on documentation vs. clinical actionEffect on documentation vs. clinical action
People always question generalizabilityPeople always question generalizability Success is highly implementation independentSuccess is highly implementation independent Efficacy-effectiveness gap: ‘Invented here’ Efficacy-effectiveness gap: ‘Invented here’
effecteffect
33
Data CollectionData Collection
Electronic data abstractionElectronic data abstraction Convenient and time-saving, but…Convenient and time-saving, but…
Some chart review (selected) to get Some chart review (selected) to get information not available electronicallyinformation not available electronically
Get ready for nasty surprisesGet ready for nasty surprises
Pilot your data collection protocol earlyPilot your data collection protocol early And then pilot some more…And then pilot some more…
34
Data Collection Issue: Data Collection Issue: Baseline DifferencesBaseline Differences
Randomization schemes Randomization schemes often lead to imbalance often lead to imbalance between intervention and between intervention and control arms:control arms: Need to collect baseline Need to collect baseline
data and adjust for data and adjust for baseline differences baseline differences
Interaction term ( Time * Interaction term ( Time * Allocation Arm) gives Allocation Arm) gives effect for intervention in effect for intervention in regression analysisregression analysis
35
Data Collection Issue: Data Collection Issue: Completeness of FollowupCompleteness of Followup
The higher the better:The higher the better: Over 90%Over 90% 80-90%80-90% Less than 80%Less than 80%
Intention to treat analysisIntention to treat analysis In an RCT, should analyze outcomes In an RCT, should analyze outcomes
according to the original randomization according to the original randomization assignmentassignment
36
A Common Analytical Issue A Common Analytical Issue The Clustering EffectThe Clustering Effect
Occurs when your observations are not Occurs when your observations are not independent:independent: Example: Each physician treats multiple patients:Example: Each physician treats multiple patients:
Intervention Group Control Group
Physicians
Patient -> Outcome assessed
37
Options for Dealing with the Options for Dealing with the Clustering EffectClustering Effect
Analyze at the level of clinicianAnalyze at the level of clinician Example: Analyze % of MD’s patients in compliance with Example: Analyze % of MD’s patients in compliance with
guideline, and make MD unit of analysisguideline, and make MD unit of analysis Huge drop in statistical power.Huge drop in statistical power. Not recommended.Not recommended.
Generalized Estimating Equations Generalized Estimating Equations PROC GENMOD in SAS, or PROC RLOGIST in SUDAAN PROC GENMOD in SAS, or PROC RLOGIST in SUDAAN Allows you to randomize at one level (e.g. physician) and Allows you to randomize at one level (e.g. physician) and
then do analysis at another (e.g. patient) then do analysis at another (e.g. patient) Accounts for correlation of behaviors within a single physician (i.e. Accounts for correlation of behaviors within a single physician (i.e.
adjusts for the fact that observations across patients are NOT adjusts for the fact that observations across patients are NOT independent)independent)
38
A Word About SurveysA Word About Surveys
Survey of user believes, attitude and Survey of user believes, attitude and behaviorsbehaviors Response rate – responder bias: Aim for Response rate – responder bias: Aim for
response rate > 50-60%response rate > 50-60% Keep the survey conciseKeep the survey concise Pilot survey for readability and clarityPilot survey for readability and clarity Need formal validation if you want plan to Need formal validation if you want plan to
develop a scaledevelop a scale
39
Looking at Usage DataLooking at Usage Data
Great way to tell how well the intervention Great way to tell how well the intervention is goingis going Target your trouble-shooting effortsTarget your trouble-shooting efforts
In terms of evaluating HIT:In terms of evaluating HIT: Correlate usage to implementation/training Correlate usage to implementation/training
strategystrategy Correlate usage to stakeholder characteristicsCorrelate usage to stakeholder characteristics Correlate usage to improved outcomeCorrelate usage to improved outcome
40
Studies on Workflow and UsabilityStudies on Workflow and Usability
How to make observations?How to make observations? Direct observationsDirect observations Stimulated observationsStimulated observations
Random paging methodRandom paging method Subjects must be motivated and cooperativeSubjects must be motivated and cooperative
Usability LabUsability Lab What to look for?What to look for?
Time to accomplish specific tasks:Time to accomplish specific tasks: Need to pre-classify activitiesNeed to pre-classify activities Handheld/Tablet PC tools may be very helpfulHandheld/Tablet PC tools may be very helpful
Workflow analysisWorkflow analysis Asking users to ‘think aloud’Asking users to ‘think aloud’
Unintended consequences of HITUnintended consequences of HIT
41
Qualitative MethodologiesQualitative Methodologies
Major techniquesMajor techniques Direct observationsDirect observations Semi-structured interviewsSemi-structured interviews Focus groupsFocus groups
Adds richness to the evaluationAdds richness to the evaluation Explains successes and failures. Generate Lessons learnedExplains successes and failures. Generate Lessons learned Captures the unexpectedCaptures the unexpected Great for forming hypothesesGreat for forming hypotheses People love to hear storiesPeople love to hear stories
Data analysisData analysis Goal is to make sense of your observationsGoal is to make sense of your observations Iterative & interactiveIterative & interactive
42
Cost Benefit AnalysisCost Benefit Analysis
Cost DataCost Data Generally availableGenerally available Caveat: allocation of indirect costsCaveat: allocation of indirect costs
Financial Benefit DataFinancial Benefit Data Revenue EnhancementRevenue Enhancement Cost AvoidanceCost Avoidance
Benefit AllocationBenefit Allocation Benefits may accrue to multiple partiesBenefits may accrue to multiple parties Are benefits realizable (e.g. labor savings)?Are benefits realizable (e.g. labor savings)? Calculation of benefits to external parties may be of Calculation of benefits to external parties may be of
interest, even if it does not impact on ROIinterest, even if it does not impact on ROI
43
Cost Benefit AnalysisCost Benefit Analysis
Activity Based Costing ExampleActivity Based Costing Example Simply put, a method for assigning costs to particular activitiesSimply put, a method for assigning costs to particular activities Alternate method of assigning indirect costs to the projectAlternate method of assigning indirect costs to the project Also, may serve as a framework for capturing cost savingsAlso, may serve as a framework for capturing cost savings
Step* Example
Identify activities Paper chart maintenance
Determine cost for each activity Cost data for medical records
Determine cost drivers Number of chart pulls
Obtain activity data How many charts were pulled
Calculate total cost Savings from decreased pulls
* http://www.pitt.edu/~roztocki/abc/abctutor/
44
Concluding RemarksConcluding Remarks
Don’t bite off more than what you can chewDon’t bite off more than what you can chew Pick a few study outcomes and study them well. Pick a few study outcomes and study them well.
It’s a practical worldIt’s a practical world Balancing operational and research needs is Balancing operational and research needs is
always a challenge.always a challenge. Life (data collection) is like a box of Life (data collection) is like a box of
chocolates…chocolates… You don’t know what you’re going to get until you You don’t know what you’re going to get until you
look, so look early!look, so look early!
45
Thank youThank you
Eric Poon, MD MPHEric Poon, MD MPH Email: Email: epoon@partners.orgepoon@partners.org
Davis Bu, MD MADavis Bu, MD MA Email: Email: dbu@partners.orgdbu@partners.org
46
Give Us Feedback! Give Us Feedback!
We are eager to hear your feedback!We are eager to hear your feedback!
Go to http://extranet.ahrq.gov/rcGo to http://extranet.ahrq.gov/rc
Login with username and passwordLogin with username and password
Follow the links to provide feedback-thanks!Follow the links to provide feedback-thanks!
Want to hear this teleconference again?Want to hear this teleconference again?
Dial 1-800-486-4195Dial 1-800-486-4195 to replay until 5/4/05 to replay until 5/4/05
top related