evidence farming and open architecture
DESCRIPTION
Presented at Mobile Health 2011, May 2011, Stanford.TRANSCRIPT
Evidence Farming1: Implications forOpen Architecture
Ida Sim, MD, PhDDirector, Center for Clinical and Translational Informatics
University of California San FranciscoMay 5, 2011
1With thanks to Rich Kravitz MD, UC Davis and Naihua Duan, Columbia
Rephrasing “Does it Work?”
(Complexes of)Exposures Outcome
strength of association?
individual
population
IncreasedbreastfeedingText4Baby
Current Approaches: RCT
• Tests prespecified interventions and outcomes• To confirm a hypothesis at the population level• Strong internal validity• Problems: slow to set-‐up, expensive, short-‐term, lack
relevance to the real world
ER visits at 1 year50 people population
100 people
ER visits at 1 year50 people
Asthma App
Usual Care
Exposures Outcomes?
population
Current Approaches: Data Mining
• Exposures and outcomes from care process systems• To generate hypotheses at the population level• Problems: limited to data collected, weak internal
validity (data not complete or systematic)
EHR
Apps
Current Approaches:N-‐of-‐1 Studies
• Within-‐subject multiple crossover• Only formal method for determining individual
treatment effectiveness• Problems: complicated to set up, analysis is
difficult, little known, not widely used
individual
peak flowpeak flowUsual Care
Asthma app
Asthma app
Usual Care
Asthma app
Usual Care
Evidence Extraction
• Evidence is something to be extractedfrom the care process– mining it from the data– directly manipulating the care process withrigid and pre-‐defined protocols
Evidence Strip Mining
Evidence Farming
Hay, et al. J Eval Clin Prac 14(2008):707-713.
Rooting for Evidence
Industrial Evidence Farming
ER visits at 1 year50 people population
100 people
ER visits at 1 year50 people
Asthma App
Usual Care
Personal Evidence Gardens
individual
peak flowpeak flowUsual Care
Asthma app
Asthma app
Usual Care
Asthma app
Usual Care
Personal Evidence Gardens
individual
dancingFlovent PRN
Flovent
Flovent
Flovent PRN
Flovent
Flovent PRN
dancing
Crowdsourcing What Matters
• (Complexes of) Exposures– does chocolate trigger (my) asthma?– testing common regimens (ACEI, statin, b-‐blocker),
complementary medicines
• (Complexes of) Outcomes– what outcomes do patients care about?
Evidence MacrosystemRooting forEvidence
Industrial EvidenceFarming
Personal EvidenceGardens
How can we scale evaluation?
StovepipedmHealth
• Health apps builtindependently– little data sharing and
interoperability
• Limits efficiency andimpact of qualitymHealth
Internet Hourglass Model
• Standardize andmake open the“narrow waist”
• Reduces duplication,spurs communityinnovation, supportscommercial and non-‐profit uses
OpenmHealth.org
Estrin DE, Sim I. Science; 330: 759-60. 2010.
• The waist should supportthe evidence macrosystem
OpenmHealth.org
Open Architecture for anEvidence Macrosystem
• Modules for usage analytics– # of text messages, # of sessions, etc.
• Rooting for (glocal) evidence– data sharing with shared syntax and semantics
• Industrial farming, e.g., with RCTs– modules for informed consent, randomization, adaptive
treatment strategy, mixed methods, etc.
• Personal evidence gardening, e.g., N-‐of-‐1– modules for scripting and analyzing individualized N-‐of-‐
1 protocols, etc.
Open Architecture for anEvidence Macrosystem
• Social media for discovery of exposures andoutcomes that matter
• Shared libraries of validated measures andinstruments (e.g., PROMIS)– measures that get at finer-‐grained mechanisms based
on theoretical models of change, etc.
Goal for mHealth Evidence
• A learning community coupled with anopen architecture for broad, rapid, anditerative dissemination of evaluationmethods and findings that matter
• Ida Sim [email protected]• Deborah Estrin [email protected]• http://openmhealth.org/