If We Want More Evidence-based Practice, If We Want More Evidence-based Practice, We Need More Practice-Based EvidenceWe Need More Practice-Based Evidence

UCSF Translation-2 Course, Oct 30, 2008UCSF Translation-2 Course, Oct 30, 2008

Lawrence W. GreenLawrence W. GreenUniversity of California at San FranciscoUniversity of California at San Francisco

The Shifting Context for Dissemination The Shifting Context for Dissemination & Evaluation in Translational Research& Evaluation in Translational Research

NIH Roadmap InitiativeNIH Roadmap Initiative--translating discoveries --translating discoveries

into health*into health*

*Zerhouni E. Science 2003, Oct 3;302(5642):63-72 .

The roadmap lesstraveled?**

“The Roadmap identifies the most compelling opportunities in three arenas: new pathways to discovery, research teams of the future, and reengineering the clinical research enterprise” (Zerhouni, p. 63).*

**Green LW. Am J Prev Med., 2007; 33(2):137-38, after K. Grumbach.

*Westfall, J. M. et al. JAMA 2007;297:403-406.

"Blue Highways" on the NIH Roadmap*

ProgramEvaluation,CQI, PolicyAnalysis, TA in EBPadaptation. Systems research.

Pu

blic

atio

n

Bib

liog

rap

hic

dat

abas

es

Su

bm

issi

on

Rev

iew

s, g

uid

elin

es, t

extb

oo

ks

Negative results

Unknown 0.3 year 6. 0 - 13.0 years

50%46%18% 35%

0.6 year0.5 year 9.3 years

Dickersin, 1987 Poynard, 1985

Kumar, 1992 Poyer, 1982 Antman, 1992

Lack of numbers,

Design issues

Inconsistentindexing

Ori

gin

al r

esea

rch

Acc

epta

nce

Imp

lem

enta

tio

n

““It takes 17 years to turn 14 per It takes 17 years to turn 14 per cent of original researchcent of original research

to the benefit of patient care” *to the benefit of patient care” *

Koren, 1989*Balas, 1995

Where Have All the Data Gone? Longtime Passing…

Lack of numbers,

Design issues

Kumar, 1992

17 yrs

R

The Pipeline Fallacy of Producing & VettingResearch to Get Evidence-Based Practice*

Peer ReviewOf Grants

PublicationPriorities &Peer Review

ResearchSynthesis

Guidelines forEvidence-BasedPractice

Academic appointments,promotion, & tenure criteria

Funding; patientneeds, demands; local practicecircumstances;professionaldiscretion; credibility & fit ofthe evidence.

-

Evidence-basedMedicine movement

Practice

Priorities for Research Funding

*Based on Green, L.W. From research to “best practices” in other settings and populations. Am J Health Behavior 25:165-178, April-May 2001. Full text: www.ajhb.org/25-3.htm

The 17-year odyssey

Blame the practitioneror blame dissemination

The research indicates that we really should do something with all this research.

DiffusionAdoptionQualityEBP

“Bridging the gap”

5 Ways of Making Research More 5 Ways of Making Research More Relevant for Practice Relevant for Practice

Making research more theory-basedMaking research more theory-based Setting research & evaluation prioritiesSetting research & evaluation priorities Making research findings actionable, Making research findings actionable,

usable, relevant (to whom?)usable, relevant (to whom?) Disseminating & translating (adapting) Disseminating & translating (adapting)

research to local circumstances, research to local circumstances, cultures, and personnelcultures, and personnel

Making evidence more practice-basedMaking evidence more practice-based

Filling the Chasm, as Filling the Chasm, as Conceived by the U.S. Conceived by the U.S. Translation Agency* Translation Agency*

Practice is Practice is We want it toWe want it toherehere be here be here

TRIPTRIP

ImplementationImplementation

DiffusionDiffusion

AdoptionAdoption

EducationEducation

InnovationInnovation

*Carolyn Clancy. Agency for Healthcare Research & Quality2003.

Reminiscent of the “Fallacy of the EmptyVessel” from earlyhealth education

R

The Pipeline Fallacy of Producing & VettingResearch to Get Evidence-Based Practice*

Peer ReviewOf Grants

PublicationPriorities &Peer Review

ResearchSynthesis

Guidelines forEvidence-BasedPractice

Academic appointments,promotion, & tenure criteria

Funding; patientneeds, demands; local practicecircumstances;professionaldiscretion; credibility & fit ofthe evidence.

-

Evidence-basedMedicine movement

Practice

Priorities for Research Funding

*Based on Green, L.W. From research to “best practices” in other settings and populations. Am J Health Behavior 25:165-178, April-May 2001. Full text: www.ajhb.org/25-3.htm

The 17-year odyssey

Blame the practitioneror blame dissemination

RFA (PAR) from NIH, RFA (PAR) from NIH, 20062006

Applications to “ identify, develop, and refine Applications to “ identify, develop, and refine effective and efficient methods, structures, and effective and efficient methods, structures, and strategies that test models to disseminate and strategies that test models to disseminate and implement implement research-testedresearch-tested health behavior change health behavior change interventions and evidence-based prevention, early interventions and evidence-based prevention, early detection, diagnostic, treatment, and quality of detection, diagnostic, treatment, and quality of life improvement services into public health and life improvement services into public health and clinical practice settings.”clinical practice settings.”

Two problems with these framings of the issue:Two problems with these framings of the issue: Are the “research-tested interventions” adequate?Are the “research-tested interventions” adequate? Are they appropriate to other settings, Are they appropriate to other settings, populations?populations?

To illustrate the first problem:To illustrate the first problem:

Canadian Cancer Society RFP for a Canadian Cancer Society RFP for a Review to Answer 4 QuestionsReview to Answer 4 Questions

Are group counseling programs for Are group counseling programs for smoking cessation effective?smoking cessation effective?

If so, what is the optimal content of the If so, what is the optimal content of the sessions?sessions?

What is the optimum number and What is the optimum number and frequency of sessions that should be frequency of sessions that should be offered?offered?

What are the characteristics of the most What are the characteristics of the most effective facilitators?effective facilitators?

University of Waterloo Results*University of Waterloo Results*

A comprehensive literature review of A comprehensive literature review of over 40 years of published and over 40 years of published and unpublished studiesunpublished studies

Deficiencies in purpose, design and Deficiencies in purpose, design and reportingreporting

Research could answer only the first of Research could answer only the first of 4 questions: that group programs for 4 questions: that group programs for smoking cessation are effectivesmoking cessation are effective..

*Manske SR, Miller S, Moyer C, Phaneuf MR, Cameron RC. Best practice in group-based smoking cessation: Results of a literature review. AJHP 18:409-23, 2004.

Evidence-Based Medicine and Evidence-Based Medicine and Patient-Centered Medicine*Patient-Centered Medicine*

Information of importance to patient

choice that is not even potentially of “evidence-based type.”

CA

B

Area where there is currently

good evidence-based information

of importance to patients

in making choices.

Information of importance to patient

choice that is potentially of evidence-

based type.

*In A.L. Cochrane, from T. Hope. Evidence-based patient choice and the doctor-patient

relationship. In But Will It Work, Doctor? Kings Fund, London, 1997, 20-24.

A “Good evidence”

B Potential for “good evidence”

C Information of potential importance to patients in making health care choices

Issues for Evidence-Based Practice Issues for Evidence-Based Practice and Translating Research to Practiceand Translating Research to Practice

Making practice more theory-basedMaking practice more theory-based Setting research prioritiesSetting research priorities Making research findings actionable, Making research findings actionable,

usable, relevant within settingsusable, relevant within settings Translating research Translating research from outsidefrom outside to local to local

circumstances, cultures, personnelcircumstances, cultures, personnel Making evidence more practice-basedMaking evidence more practice-based

Priority-Setting for Health Research*Priority-Setting for Health Research*

KnowledgeAcquisition

Knowledge Validation

KnowledgeTransfer

ClinicalInvestigations

ClinicalTrials

KnowledgeTranslation

CDC

NIH

Surveillance Community & StatewideEffectiveness Trials

Program Evaluation

AppliedResearch &Development

BasicResearch

Demonstration& Education

Research

Molecular Level

Population Level

*Green LW, Popovic T, et al. CDC Futures Workgroup on Research. Atlanta, 2004.

PBRNs, CQI

T1 T2

The Internal Validity Drift of Health Sciences The Internal Validity Drift of Health Sciences Evidence “Lost in Translation”Evidence “Lost in Translation”

Evidence-based medicine movement taken to Evidence-based medicine movement taken to scale in general practice & health promotionscale in general practice & health promotion

The peer review preferences for experimental The peer review preferences for experimental control and certainty of causationcontrol and certainty of causation

The publishing preferences for RCTs and The publishing preferences for RCTs and positive resultspositive results

The limitations of print space driving out The limitations of print space driving out richer description of interventions, protocols, richer description of interventions, protocols, procedural lessons, subgroup variationsprocedural lessons, subgroup variations

But a more “natural” type of practice-based But a more “natural” type of practice-based evidence has greater influence on multi-level evidence has greater influence on multi-level program planning, practice & policy…program planning, practice & policy…

Change in Per Capita Cigarette ConsumptionChange in Per Capita Cigarette ConsumptionCalifornia & Massachusetts vs Other 48 States, 1984-1996California & Massachusetts vs Other 48 States, 1984-1996

-25

-20

-15

-10

-5

0

5

Percent Reduction

Other 48 States California Massachusetts

1984-1988 1990-1992 1992-1996

Issues for Evidence-Based Practice Issues for Evidence-Based Practice and Translating Research to Practiceand Translating Research to Practice

Making practice more theory-basedMaking practice more theory-based Setting research prioritiesSetting research priorities Participatory research to make findings Participatory research to make findings

actionable, usable, relevant within settingsactionable, usable, relevant within settings Translating research Translating research from outsidefrom outside to local to local

circumstances, cultures, personnelcircumstances, cultures, personnel Making evidence more practice-basedMaking evidence more practice-based

Some Benefits of Participatory Some Benefits of Participatory Research in Practice-Based EvidenceResearch in Practice-Based Evidence

Results are relevant to interests, circumstances, Results are relevant to interests, circumstances, and needs of those who would apply themand needs of those who would apply them

Results are more immediately actionable in local Results are more immediately actionable in local situations for people and/or practitionerssituations for people and/or practitioners

Generalizable findings more credible to people, Generalizable findings more credible to people, practitioners and policy makers elsewhere practitioners and policy makers elsewhere because they were generated in partnership with because they were generated in partnership with people like themselvespeople like themselves

Helps to reframe issues from health behavior of Helps to reframe issues from health behavior of individuals to encompass system and structural individuals to encompass system and structural issues.issues.

Green LW, Mercer SL. Am J Public Health Dec. 2001.

Definition and Standards of Definition and Standards of Participatory Research for Health*Participatory Research for Health*

Systematic investigation…Systematic investigation…

Actively involving people in a co-learning Actively involving people in a co-learning process…process…

For the purpose of action conducive to health**For the purpose of action conducive to health**

--not just involving people more intensively as --not just involving people more intensively as subjectssubjects of research or evaluation of research or evaluation

*Green, George, Daniel, et al., Participatory Research…Ottawa: Royal Society of Canada, 1997. www.lgreen.net/guidelines.html

The Lenses of The Lenses of Scientists, Health Scientists, Health

Professionals and Lay Professionals and Lay PeoplePeople

ObjectiveIndicators of Health

SubjectiveIndicatorsof Health

Professional,Scientific

Layperson

Issues for Evidence-Based Practice Issues for Evidence-Based Practice and Translating Research to Practiceand Translating Research to Practice

Making practice more theory-basedMaking practice more theory-based Setting research prioritiesSetting research priorities Making research findings actionable, Making research findings actionable,

usable, relevant: participatory researchusable, relevant: participatory research Translating research to local cultures & Translating research to local cultures &

circumstances: External validity & “fidelity” circumstances: External validity & “fidelity” vs adaptationvs adaptation

Making evidence more practice-basedMaking evidence more practice-based

Building Policy and Practice from Building Policy and Practice from Evidence + TheoryEvidence + Theory

Not starting with theory and looking for problems Not starting with theory and looking for problems on which to test them, but starting with problems on which to test them, but starting with problems and looking for theories to help us solve them*and looking for theories to help us solve them*

Evidence on solutions generalizes to other Evidence on solutions generalizes to other circumstances, settings, & populations in the circumstances, settings, & populations in the form of either replication or theoryform of either replication or theory

Replication is limited by the infinite number of Replication is limited by the infinite number of context-population combinationscontext-population combinations

"In theory, theory and practice are the same "In theory, theory and practice are the same thing. In practice they're not..“ thing. In practice they're not..“ -Jan L.A. van de -Jan L.A. van de SnepscheutSnepscheut

““All models are wrong. Some are useful” --BoxAll models are wrong. Some are useful” --Box

*Green LW. Public health asks of systems science… Amer J Public Health 96, March 2006.

““Fidelity” vs Fidelity” vs Adaptation*Adaptation*

Researchers test an intervention for its efficacyResearchers test an intervention for its efficacy Rigorous test (efficacy) qualifies it for official lists Rigorous test (efficacy) qualifies it for official lists

of “evidence-based practices” and guidelinesof “evidence-based practices” and guidelines Practitioners try to incorporate it into their Practitioners try to incorporate it into their

programs in other populations, circumstancesprograms in other populations, circumstances Poor fit produces failure of programPoor fit produces failure of program Practitioners are blamed for not implementing Practitioners are blamed for not implementing

with “fidelity”with “fidelity” Now buy the producers’ training programNow buy the producers’ training program

* Green LW, Glasgow RE, …external validity…Evaluation & the Health Professions, Mar. 2006.

Efficacy vs. Efficacy vs. EffectivenessEffectiveness: :

EfficacyEfficacy. The tested impact of an intervention . The tested impact of an intervention under highly controlled circumstances.under highly controlled circumstances.

EffectivenessEffectiveness. The tested impact of an . The tested impact of an intervention under more normal circumstances intervention under more normal circumstances ((relativelyrelatively less controlled, real-time, “typical” less controlled, real-time, “typical” setting, population, and conditions).setting, population, and conditions).

Broad Program Evaluation.Broad Program Evaluation. The tested impact of The tested impact of a blended set of interventions on larger systems a blended set of interventions on larger systems and populations. “Natural Experiments” with and populations. “Natural Experiments” with minimal control, maximum variability.minimal control, maximum variability.

The Trade-offsThe Trade-offs

Efficacy. Efficacy. Maximizes Maximizes internal validityinternal validity, i.e., the , i.e., the degree to which one can conclude with degree to which one can conclude with confidence that the intervention caused the confidence that the intervention caused the result.result.

Effectiveness.Effectiveness. Maximizes Maximizes external validityexternal validity,* ,* i.e., the degree to which one can generalize i.e., the degree to which one can generalize from the test to other times, places, or from the test to other times, places, or populations. populations.

Program Evaluation. Program Evaluation. Maximizes reality Maximizes reality testing in particular settings, & with the testing in particular settings, & with the combination of interventions at multiple combination of interventions at multiple levels required for public health effectlevels required for public health effect..

* Green LW, Glasgow RE, …external validity…Evaluation & the Health Professions, Mar. 2006.

Issues for Evidence-Based Practice and Issues for Evidence-Based Practice and Translating Research to PracticeTranslating Research to Practice

Blending evidence-based practice with Blending evidence-based practice with theory-based practicetheory-based practice

Setting research prioritiesSetting research priorities Making research findings actionable, usable, Making research findings actionable, usable,

relevant: Participatory Researchrelevant: Participatory Research Translating research to local circumstancesTranslating research to local circumstances Making evidence more practice-based: the Making evidence more practice-based: the

centrality of evaluation and continuous centrality of evaluation and continuous quality improvement researchquality improvement research

Interventionor Program

Mediator

Mediator

Outcome Variable(s)

Moderators

Mediating and Moderating VariablesMediating and Moderating Variables

Moderators

Green & Kreuter, Health Program Planning: An Educational and Ecological

Approach. 4th ed. New York: McGraw-Hill, 2005. Green & Glasgow, E&HP, 2006.

Aligning Evidence with (and deriving it from) Aligning Evidence with (and deriving it from) Practice: Matching, Mapping, Pooling and PatchingPractice: Matching, Mapping, Pooling and Patching

MatchingMatching ecological levels of a system or community ecological levels of a system or community with evidence of with evidence of efficacyefficacy for interventions at those for interventions at those levelslevels

MappingMapping theory to the causal chain to fill gaps in the theory to the causal chain to fill gaps in the evidence for evidence for effectivenesseffectiveness of interventions of interventions

PoolingPooling experience to blend interventions to fill gaps experience to blend interventions to fill gaps in evidence for the effectiveness of programs in in evidence for the effectiveness of programs in similar situationssimilar situations

PatchingPatching pooled interventions with indigenous pooled interventions with indigenous wisdom and professional judgment about plausible wisdom and professional judgment about plausible interventions to fill gaps in the interventions to fill gaps in the programprogram for the for the specific populationspecific population

*Green & Kreuter, Health Program Planning: An Educational and EcologicalApproach. 4th ed. NY: McGraw-Hill, 2005, Chapter 5. Green & Glasgow, 2006.

3 Conceptualizations of the Gap 3 Conceptualizations of the Gap Between Research & PracticeBetween Research & Practice

Practitioners need to receive the lessons of Practitioners need to receive the lessons of research and put them into practice.research and put them into practice.

Research and practice are entirely separate Research and practice are entirely separate disciplines and each must develop their own disciplines and each must develop their own answers to their own problemsanswers to their own problems

Research and practice have complementary Research and practice have complementary perspectives and skills that need to be used perspectives and skills that need to be used together to address the real need, together to address the real need, collaborative knowledge production. collaborative knowledge production.

Add to this the need to include the patient’s Add to this the need to include the patient’s perspective. Whose perspective prevails?perspective. Whose perspective prevails?

Van De Ven A, Johnson P. Knowledge for theory and practice. Academy of Management Review. 2006;31(4).

The Bridge (not the Pipeline) from The Bridge (not the Pipeline) from Research to Practice and BackResearch to Practice and Back

If we want more evidence-based practice, we If we want more evidence-based practice, we need more practice-based evidence. need more practice-based evidence.

The importance of practitioners and policy-The importance of practitioners and policy-makers in shaping the research questions.makers in shaping the research questions.

Practitioners and their organizations Practitioners and their organizations represent the structural links (and barriers) represent the structural links (and barriers) to addressing the important determinants of to addressing the important determinants of health behavior at each level. Engage them, health behavior at each level. Engage them, not at passive recipients, but as partners…not at passive recipients, but as partners…

*Green, L.W. From research to “best practices” in other settings and populations. Am J Health Behavior 25:165-178, April-May 2001. Full text: www.ajhb.org/25-3.htm.

The Vision for The Vision for Translation 2Translation 2

A future in which we would not need to ask A future in which we would not need to ask how to get more evidence-based practice, how to get more evidence-based practice, rather rather

How to sustain the engagement of students, How to sustain the engagement of students, practitioners, patients and communities in practitioners, patients and communities in a participatory process of practice-based a participatory process of practice-based research and program evaluation? research and program evaluation?

How to adapt the “best practices” How to adapt the “best practices” guidelines through guidelines through best processesbest processes of of collecting data to diagnose the collecting data to diagnose the biopsychosocial needs of their patients biopsychosocial needs of their patients and communities…and communities…

Translation 2 Vision Translation 2 Vision (expanded)(expanded)

How to match the proposed evidence-based How to match the proposed evidence-based interventions to those needs, filling gaps interventions to those needs, filling gaps in the evidence-based interventions with in the evidence-based interventions with the use of theory and mutual consultation, the use of theory and mutual consultation, and prospective testing of complementary and prospective testing of complementary interventions interventions

The cumulative, building-block tradition The cumulative, building-block tradition of evidence-based medicine from RCTs would of evidence-based medicine from RCTs would be complemented by a parallel be complemented by a parallel strengthening and support of a tradition strengthening and support of a tradition of participatory research and evaluation of participatory research and evaluation conducted in practice settings.conducted in practice settings.

6 Conclusions 6 Conclusions (Remedies)(Remedies)

Adapt the research funding priorities Adapt the research funding priorities Adapt publication criteria Adapt publication criteria Adapt the criteria for inclusion and weighting Adapt the criteria for inclusion and weighting of studies into systematic reviews and research of studies into systematic reviews and research syntheses syntheses

Adapt the derivation and qualification of Adapt the derivation and qualification of practice guidelines from the systematic reviews practice guidelines from the systematic reviews

Adapt the academic promotion and tenure criteria Adapt the academic promotion and tenure criteria and weights given to community- & practice-based and weights given to community- & practice-based research research

Adapt the research training of students and Adapt the research training of students and fellows in methods of practice-based and fellows in methods of practice-based and participatory researchparticipatory research