evaluation of the effectiveness of ahrq’s grant supported research on healthcare costs,...
TRANSCRIPT
EVALUATION OF THE EFFECTIVENESS OF AHRQ’s GRANT SUPPORTED RESEARCH ON HEALTHCARE COSTS, PRODUCTIVITY,
ORGANIZATION, AND MARKET FORCES
OVERVIEW OF PROJECT FINDINGSby
Marsha Gold, Sc.D., Project DirectorTimothy Lake, Ph.D., Deputy Director
Kate Stewart, Ph.D., ResearcherTara Krissik, M.P.P., Researcher
December 2008This project was funded by the Agency for Healthcare Research and Quality
EVALUATION OF THE EFFECTIVENESS OF AHRQ’s GRANT SUPPORTED RESEARCH ON HEALTHCARE COSTS, PRODUCTIVITY,
ORGANIZATION, AND MARKET FORCES
OVERVIEW OF PROJECT FINDINGSby
Marsha Gold, Sc.D., Project DirectorTimothy Lake, Ph.D., Deputy Director
Kate Stewart, Ph.D., ResearcherTara Krissik, M.P.P., Researcher
December 2008This project was funded by the Agency for Healthcare Research and Quality
2
Rationale for ProjectRationale for Project
While AHRQ’s statutory mission explicitly includes support for grant funded research on healthcare costs, productivity, organization, and market forces, the agency does not systematically track the work it funds in this area and what has been accomplished through it.
Purpose of this project is to address this gap and identify how AHRQ may improve the visibility and impact of this research.
While AHRQ’s statutory mission explicitly includes support for grant funded research on healthcare costs, productivity, organization, and market forces, the agency does not systematically track the work it funds in this area and what has been accomplished through it.
Purpose of this project is to address this gap and identify how AHRQ may improve the visibility and impact of this research.
3
Project DetailsProject Details
MPR evaluation began in October 2006; the two phased project was completed in December 2008.
Work was guided by an Expert Panel: Sharon Arnold (AcademyHealth), John Christianson (University of Minnesota), Paul Ginsburg (Center for Studying Health Systems Change), Robert Helms (AEI), and Gail Wilensky (Project Hope)
Michael Hagan was the AHRQ project officer.
MPR evaluation began in October 2006; the two phased project was completed in December 2008.
Work was guided by an Expert Panel: Sharon Arnold (AcademyHealth), John Christianson (University of Minnesota), Paul Ginsburg (Center for Studying Health Systems Change), Robert Helms (AEI), and Gail Wilensky (Project Hope)
Michael Hagan was the AHRQ project officer.
4
Key Questions of InterestKey Questions of Interest1. Since the late 1990s, what grant research has AHRQ
funded that relates to healthcare costs, productivity, organization, and market forces?
2. How are the research findings disseminated to public and private decision makers and what factors contribute to their use?
3. What is AHRQ’s role in supporting research in this area and how does it compare with that of others, such as NIH and private funders?
4. What actions, if any, could enhance AHRQ’s efforts to track, disseminate and encourage use of these research findings?
1. Since the late 1990s, what grant research has AHRQ funded that relates to healthcare costs, productivity, organization, and market forces?
2. How are the research findings disseminated to public and private decision makers and what factors contribute to their use?
3. What is AHRQ’s role in supporting research in this area and how does it compare with that of others, such as NIH and private funders?
4. What actions, if any, could enhance AHRQ’s efforts to track, disseminate and encourage use of these research findings?
5
Components of the Study - IComponents of the Study - I
Descriptive Analysis: Reviewed AHRQ administrative data on all grants, and conducted indepth analysis of a stratified random sample of nine grants.
Conceptual Framework: Using relevant social science literature, identified pathways for translation and key variables that are likely to influence use of research.
Case Studies: In depth description and analysis of 7 grants that illustrate diverse ways in which study findings reach target audiences and influence policy debates; based on PI and user interviews.
Descriptive Analysis: Reviewed AHRQ administrative data on all grants, and conducted indepth analysis of a stratified random sample of nine grants.
Conceptual Framework: Using relevant social science literature, identified pathways for translation and key variables that are likely to influence use of research.
Case Studies: In depth description and analysis of 7 grants that illustrate diverse ways in which study findings reach target audiences and influence policy debates; based on PI and user interviews.
6
Components of the Study - IIComponents of the Study - II Comparative Funder Analysis: Comparison of AHRQ’s
infrastructure and funding for grants in this area to that of NIH (government) and HCFO (private sector).
Federal Research Translator/User Interviews: Identify how and when this type of research gets used, and awareness of AHRQ research. (Included CBO, GAO, MedPAC, FTC, DOJ, and selected others).
PI Survey: Web based survey in July-August 2008 to all 149 grantees (70% response). Detail on research focus and key findings, dissemination modes and preferences, uses made of results, and PIs interaction with policymakers. Also obtained feedback on AHRQ’s role and performance and how AHRQ compares to other funders.
Comparative Funder Analysis: Comparison of AHRQ’s infrastructure and funding for grants in this area to that of NIH (government) and HCFO (private sector).
Federal Research Translator/User Interviews: Identify how and when this type of research gets used, and awareness of AHRQ research. (Included CBO, GAO, MedPAC, FTC, DOJ, and selected others).
PI Survey: Web based survey in July-August 2008 to all 149 grantees (70% response). Detail on research focus and key findings, dissemination modes and preferences, uses made of results, and PIs interaction with policymakers. Also obtained feedback on AHRQ’s role and performance and how AHRQ compares to other funders.
7
Components of the Study - IIIComponents of the Study - III
Key Research Questions
Descriptive Analysis of
Grants
Framework on Research
UseCase Studies
of UseComparative
Funder Analysis
Federal Research Translator Interviews
Principal Investigator Survey
1. What has AHRQ funded in these areas and how is it
used?
X X X
2. How are the findings disseminated and what
contributes to their use?
X X X X
3. What is AHRQ’s role in this research area and how does
its performance compare with other funders?
X X
4. What actions, if any, could enhance AHRQ’s efforts to
track, disseminate, and encourage use of these research findings?
X X X X X X
8
KEY FINDINGS: WHAT HAS AHRQ FUNDED IN THIS AREA BETWEEN
1998 – 2006?
KEY FINDINGS: WHAT HAS AHRQ FUNDED IN THIS AREA BETWEEN
1998 – 2006?
9
Number of New AHRQ Grants for Research on Healthcare Costs, Productivity, Organization, and
Market Forces Funded, 1998-2006
Number of New AHRQ Grants for Research on Healthcare Costs, Productivity, Organization, and
Market Forces Funded, 1998-2006
Fiscal Year
Source: MPR analysis of AHRQ Administrative Data.
10
AHRQ Funding for Research on Healthcare Costs, Productivity, Organization, and
Market Forces, 1998-2006
AHRQ Funding for Research on Healthcare Costs, Productivity, Organization, and
Market Forces, 1998-2006
Fiscal Year
Source: MPR analysis of AHRQ Administrative data.
Note: Includes funding for new grants and continuing grants that were funded in 1998 or later. The combined spending over the period was $81.4 million, of which $12.6 million was for three PO1 grants. 47 RO3 grants under $100,00 accounted for about $4 million in spending.
11
Characteristics of Funded StudiesCharacteristics of Funded Studies
Of the 149 studies:
– 97 studies had organizations as the unit of analysis. Studies of hospitals and health plans were particularly common.
– 37 studies examined consumer behavior.
– 15 studies examined markets or purchaser behavior.
75 percent of the 149 were national in scope.
Multiple outcomes studied including quality (53%), use (44%), cost (38%) and access (19%), among others.
Of the 149 studies:
– 97 studies had organizations as the unit of analysis. Studies of hospitals and health plans were particularly common.
– 37 studies examined consumer behavior.
– 15 studies examined markets or purchaser behavior.
75 percent of the 149 were national in scope.
Multiple outcomes studied including quality (53%), use (44%), cost (38%) and access (19%), among others.
12
Nature of ResearchNature of Research
PI survey indicates that findings commonly examine how specific outcomes of care are influenced by:
Economic factors (e.g., provider payment, insurance coverage)
Organizational characteristics (e.g., nurse leadership, volume)
Systems and markets (e.g., HMO penetration, capacity constraints)
Patient characteristics and preferences (e.g., percent minority, DNR orders)
Findings address questions of substantial policy interest today about influences on the performance of health care system.
PI survey indicates that findings commonly examine how specific outcomes of care are influenced by:
Economic factors (e.g., provider payment, insurance coverage)
Organizational characteristics (e.g., nurse leadership, volume)
Systems and markets (e.g., HMO penetration, capacity constraints)
Patient characteristics and preferences (e.g., percent minority, DNR orders)
Findings address questions of substantial policy interest today about influences on the performance of health care system.
13
KEY FINDINGS: HOW ARE FINDINGS DISSEMINATED AND
WHAT FACTORS CONTRIBUTE TO USE
KEY FINDINGS: HOW ARE FINDINGS DISSEMINATED AND
WHAT FACTORS CONTRIBUTE TO USE
14
Dissemination May Occur Through Diverse Pathways - I
Dissemination May Occur Through Diverse Pathways - I
We identified 10 pathways that differ in means used to communicate findings and the role researchers, intermediaries and users play in the process. Pathways include:– Traditional: “Big Bang,” Gradual Accumulation and
Diffusion, Gradual Communication and Formal Syntheses
– Effective Use of Intermediaries: Researcher as Messenger/Expert, Formal Intermediary Broker, Press Publicizes
– User Guided: User Directed Syntheses, User Partner Grant Review, User Commissioned Studies, Researcher as User
We identified 10 pathways that differ in means used to communicate findings and the role researchers, intermediaries and users play in the process. Pathways include:– Traditional: “Big Bang,” Gradual Accumulation and
Diffusion, Gradual Communication and Formal Syntheses
– Effective Use of Intermediaries: Researcher as Messenger/Expert, Formal Intermediary Broker, Press Publicizes
– User Guided: User Directed Syntheses, User Partner Grant Review, User Commissioned Studies, Researcher as User
15
Dissemination May Occur Through Diverse Pathways - II
Dissemination May Occur Through Diverse Pathways - II
Some pathways involve researchers directly engaging with users and others involve intermediaries that help synthesize, apply or communicate the findings.
Diverse pathways will be effective in different circumstances; effectively reaching diverse audiences for the work enhances the use of well targeted high quality research.
Some pathways involve researchers directly engaging with users and others involve intermediaries that help synthesize, apply or communicate the findings.
Diverse pathways will be effective in different circumstances; effectively reaching diverse audiences for the work enhances the use of well targeted high quality research.
16
Source: MPR Survey of AHRQ-Funded Principal Investigators.
Note: Responses based on 97 responding PIs (70% of grantees surveyed).
Percentage of Respondents Reporting
Major Focus Minor Focus
Publications (any)Journal article(s)Research report/working paperUser-focused research brief/issue paperChapter Book
97912912 6 3
25
2719180
Conference Presentations (any)Paper or poster at research conferencePresentation at policy-/user-focused meeting
705734
282832
Briefings (any)Policymaker briefingsManagerial briefingsInterest group briefings
27181111
26191718
Other Forms of Dissemination (any)Mass mediaFederal or state testimonyExpert witnessOther
95203
2723834
PI Survey Shows Preference for Publication as Dissemination Strategy but Use of Multiple Modes
PI Survey Shows Preference for Publication as Dissemination Strategy but Use of Multiple Modes
17
Amount of Assistance (Reported %)
Type of Assistance Substantial Moderate LimitedNone/Did Not Use Resource
Press office for interaction with media
Newsletters reporting on key findings from research
Established working paper series
Established series of research/issue briefs
Funds available to develop user-oriented materials
Information or training on how to understand and interact with potential users of research
9%
10%
3%
6%
0%
1%
28%
17%
11%
7%
2%
3%
24%
20%
7%
14%
4%
13%
40%
54%
79%
74%
93%
83%
Source: MPR Survey of AHRQ-Funded Principal Investigators.
Notes: N = 91. These analyses exclude 6 respondents who did not respond to any questions about dissemination support.
Support from Host Institution for Dissemination Limited
Support from Host Institution for Dissemination Limited
18
Source: MPR Survey of AHRQ-Funded Principal Investigators.
Mean Percentage of Time PIs Report Interacting with Policymakers
Mean Percentage of Time PIs Report Interacting with Policymakers
19
The Impact of Research on Policy - IThe Impact of Research on Policy - I
Research is one of several influences on decision-making. Importance of research likely to vary with the topic, user, environment and other factors.
Some research may have an immediate impact but most accumulates and gets applied to relevant topics as issues arise.
Accumulation of research contributes to a “research reservoir”—use more likely if findings readily available to potential users and applicability of findings clear.
Research is one of several influences on decision-making. Importance of research likely to vary with the topic, user, environment and other factors.
Some research may have an immediate impact but most accumulates and gets applied to relevant topics as issues arise.
Accumulation of research contributes to a “research reservoir”—use more likely if findings readily available to potential users and applicability of findings clear.
20
The Impact of Research on Policy - IIThe Impact of Research on Policy - II
Accumulation of findings across multiple studies adds weight to findings.
Some forms of communication are likely to be stronger in reaching users than others.
Researchers can enhance usefulness by thinking carefully about the key findings and identify the main “message” of the study (“elevator test”).
Accumulation of findings across multiple studies adds weight to findings.
Some forms of communication are likely to be stronger in reaching users than others.
Researchers can enhance usefulness by thinking carefully about the key findings and identify the main “message” of the study (“elevator test”).
21
Source: MPR Survey of AHRQ-Funded Principal Investigators.
Note: N = 85.
PI Perception of Outcomes Relevant to Their Grants
PI Perception of Outcomes Relevant to Their Grants
22
Lessons from Case Studies on Ways to Enhance Use
Lessons from Case Studies on Ways to Enhance Use
Develop relationships with potential users and involve them early
Be aware of how results may be relevant to different policy decisions and the timing of those decisions
Identify where research “fits” in stream of research
Develop expertise—and reputation for it
Develop relationships with potential users and involve them early
Be aware of how results may be relevant to different policy decisions and the timing of those decisions
Identify where research “fits” in stream of research
Develop expertise—and reputation for it
23
What Federal Research Translators Told Us
What Federal Research Translators Told Us
They make extensive use of this type of research
They are familiar with ongoing work and consult the literature as time allows
They value timeliness but also place a high value on quality and objectivity
Regulatory entities find general studies of markets useful in providing context for case specific analysis
They see critical gaps in the existing body of research in this area
They make extensive use of this type of research
They are familiar with ongoing work and consult the literature as time allows
They value timeliness but also place a high value on quality and objectivity
Regulatory entities find general studies of markets useful in providing context for case specific analysis
They see critical gaps in the existing body of research in this area
24
KEY FINDINGS: WHAT IS AHRQ’S ROLE IN CONTEXT OF NIH, HCFO,
AND OTHER FUNDERS?
KEY FINDINGS: WHAT IS AHRQ’S ROLE IN CONTEXT OF NIH, HCFO,
AND OTHER FUNDERS?
25
General Findings Across the ProjectGeneral Findings Across the Project Researchers view AHRQ as a major source of funding for this
research but have turned elsewhere as AHRQ funds have become less available. Low funding levels for this kind of research at AHRQ was the most common problem cited by PIs.
PIs view AHRQ’s oversight over grants more positively than the agency’s work on dissemination and communication.
While NIH processes mirror those of AHRQ, HCFO has more emphasis on dissemination, and researchers score HCFO higher than other funders on technical assistance
Federal translators perceive AHRQ as more a primary resource for databases and clinical research than for this type of research.
Researchers view AHRQ as a major source of funding for this research but have turned elsewhere as AHRQ funds have become less available. Low funding levels for this kind of research at AHRQ was the most common problem cited by PIs.
PIs view AHRQ’s oversight over grants more positively than the agency’s work on dissemination and communication.
While NIH processes mirror those of AHRQ, HCFO has more emphasis on dissemination, and researchers score HCFO higher than other funders on technical assistance
Federal translators perceive AHRQ as more a primary resource for databases and clinical research than for this type of research.
26
Major Source of Funding
Moderate Funding
Little Funding
No Funding
No Opinion/ Refused
Organizations
Agency for Healthcare Research and Quality (AHRQ)
National Institutes of Health (NIH)
Robert Wood Johnson Foundation’s Healthcare Financing and Organization (HCFO) program
Centers for Medicare & Medicaid Services (CMS)
Robert Wood Johnson Foundation, and other programs
Other federal or state government
Other private foundations
40%
40%
39%
7%
19%
4%
4%
30
30
34
13
43
15
24
27
24
19
62
25
13
16
3
3
1
6
4
5
3
0
3
7
13
9
64
53
Source: MPR Survey of AHRQ-Funded Principal Investigators.
Note: N = 96.
Share of Respondents Who Consider Various Funding Organizations to be Major, Moderate, Minor, or Not a Source of Funding for Research on Health
Care Costs, Productivity, Organization, and Market Forces (percentages unless otherwise noted)
Share of Respondents Who Consider Various Funding Organizations to be Major, Moderate, Minor, or Not a Source of Funding for Research on Health
Care Costs, Productivity, Organization, and Market Forces (percentages unless otherwise noted)
27
Ratings Among Those Who Consider Task Part of AHRQ’s Job
Consider Part of AHRQ’s Job
(Number) Mean Score
Percent Who Say Excellent or Very
Good
All RespondentsPre-award guidanceGrant awardGrant managementResearch methodsDissemination and communication of findingsLinkages with others interested in the topic of your research
82868040
73
62
2.12.02.32.5
2.7
2.9
68%65%59%53%
40%
44%
Source: MPR Survey of AHRQ-Funded Principal Investigators.
Note: Mean scores calculated based on respondent ratings of AHRQ technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor
Ratings of AHRQ Technical SupportRatings of AHRQ Technical Support
28
Source: MPR Survey of AHRQ-Funded Principal Investigators.
Note: Mean scores calculated based on respondent ratings of funders’ technical support, where 1 = Excellent; 2 = Very Good; 3 = Good; 4 = Fair; 5 = Poor.
Number Reported Funding Source and
Provided Rating Mean Score (SD)
Robert Wood Johnson Foundation’s Healthcare Financing and Organization (HCFO) program
Agency for Healthcare Research and Quality (AHRQ)
Robert Wood Johnson Foundation, other programs
Other private foundations
National Institutes of Health (NIH)
Other federal or state government
Centers for Medicare & Medicaid Services (CMS)
35
88
45
28
60
23
16
1.8 (0.9)
2.3 (1.1)
2.3 (0.9)
2.4 (0.8)
2.5 (1.1)
2.6 (1.2)
3.4 (1.0)
Comparison of Technical Assistance Provided by AHRQ, NIH, HCFO, and Others
Comparison of Technical Assistance Provided by AHRQ, NIH, HCFO, and Others
29
KEY FINDINGS: ACTIONS THAT COULD ENHANCE USE OF
RESEARCH FINDINGS
KEY FINDINGS: ACTIONS THAT COULD ENHANCE USE OF
RESEARCH FINDINGS
30
Conclusions From the ProjectConclusions From the Project AHRQ currently is not highly visible as a source of research
on health care costs, productivity, organization and market forces despite a legislative mandate for it and $81 million spent on grants for this kind of research from 1998 through 2006.
Our findings show that the research AHRQ has funded addresses critical aspects of health system performance that are important to decision-makers.
Research AHRQ has funded has had an impact but mainly project by project at the initiative of the investigator.
AHRQ can better track its work and leverage its investments to make its research more visible, make findings more accessible, and enhance their use.
AHRQ currently is not highly visible as a source of research on health care costs, productivity, organization and market forces despite a legislative mandate for it and $81 million spent on grants for this kind of research from 1998 through 2006.
Our findings show that the research AHRQ has funded addresses critical aspects of health system performance that are important to decision-makers.
Research AHRQ has funded has had an impact but mainly project by project at the initiative of the investigator.
AHRQ can better track its work and leverage its investments to make its research more visible, make findings more accessible, and enhance their use.
31
Recommendations for AHRQ - IRecommendations for AHRQ - I
Use our study as basis for ongoing tracking of research.
Create a visible place on the Agency website to communicate AHRQ’s efforts in this area.
Create internal capacity for monitoring grant progress to identify emerging outcomes that may be of interest; reward staff for support in this area.
Communicate with PIs why AHRQ would value ongoing reports of dissemination and use after grants end and develop systems to support such monitoring.
Use our study as basis for ongoing tracking of research.
Create a visible place on the Agency website to communicate AHRQ’s efforts in this area.
Create internal capacity for monitoring grant progress to identify emerging outcomes that may be of interest; reward staff for support in this area.
Communicate with PIs why AHRQ would value ongoing reports of dissemination and use after grants end and develop systems to support such monitoring.
32
Recommendations for AHRQ - IIRecommendations for AHRQ - II
Use our project to develop an annotated bibliography with abstracts of publications from AHRQ funded research in this area.
Work with others on targeted syntheses of findings on topics which AHRQ has invested.
Help others contact experts on given research topics through online contact information on the website or other means.
Offer researchers a “webinar” on ways of thinking about, developing, and communicating “messages” from the study.
Use our project to develop an annotated bibliography with abstracts of publications from AHRQ funded research in this area.
Work with others on targeted syntheses of findings on topics which AHRQ has invested.
Help others contact experts on given research topics through online contact information on the website or other means.
Offer researchers a “webinar” on ways of thinking about, developing, and communicating “messages” from the study.
33
Recommendations for AHRQ (and Others) - III
Recommendations for AHRQ (and Others) - III
Elicit more feedback from users on their needs to identify areas that can be supported by indepth, high quality research.
Work with others to better articulate what this type of research is and its relevance to understanding health system performance and how to enhance outcomes.
Elicit more feedback from users on their needs to identify areas that can be supported by indepth, high quality research.
Work with others to better articulate what this type of research is and its relevance to understanding health system performance and how to enhance outcomes.