Spring 2015 Cycle: Large PragmaticStudies to Evaluate Patient-Centered Outcomes

Applicant Town Hall

June 10, 2015

Your letter of intent (LOI) was reviewed and you have been invited to submit a full application……..

CONGRATULATIONS!

AgendaWelcome

Overview

Merit Review Criteria

Other topics

What to think about before you apply

Questions?Submit questions via the chat function in Meeting Bridge.

Ask a question via phone (an operator will standby to take your questions).

Overview

Overview

David Hickam, MD, MPHProgram DirectorClinical Effectiveness Research

Purpose of the Pragmatic Studies PFATo assist applicants to prepare strong proposals in response to this funding announcement:

Applicants proposing “clinical comparative effectiveness research (CER)”Applicants proposing “improving healthcare systems (IHS) CER”Applicants proposing “CER to reduce or eliminate health and health care disparities”

What is a Pragmatic CER Study?

Answers a practical, real world comparative effectiveness research question. Assesses whether two or more options differ in effectiveness when administered as they are in real lifeProject is conducted in a clinical setting that is as close as possible to a real world setting. The methodological approach (including study design, outcome measures, and follow-up) is as simple as possible without sacrificing scientific rigor.

What is a Pragmatic CER Study? (Cont.)

Pragmatic CER trials addressing populations at risk for disparities may: Require higher levels of outcome data collection. Require tailoring of intervention (e.g., language, culture,

access). Consist of multi-component, multi-level interventions

(e.g., targeting the patient, provider, and systems), as evidenced by disparities literature.

Justification for the Design Elements of a Large Pragmatic Study

Suggest reviewing pragmatic–explanatory continuum indicator summary (PRECIS) toolConsider tradeoffs Eligibility criteria Flexibility of intervention Range and types of outcomes Follow up intensity Adherence Etc.

Source: A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers. Thorpe, et al. CMAJ 2009; 180:E47-E57.

Research Activities Not Supported in this PFA

Studies of decision aidsEfficacy trialsEvidence synthesesCost-effectiveness analysisResearch that aims to compare the overall costs of care between two or more alternatives and use the results to determine the preferred alternative

Merit Review CriteriaImpact and Potential to Improve Care

Steven Clauser, PhD, MPAProgram DirectorImproving Healthcare Systems

Merit Review Criteria

Impact of the condition on health of individuals and populationsPotential for the study’s results to improve care and outcomesTechnical meritPatient-centerednessEngagement with key patient (including caregiver), clinician, and/or other stakeholder organizations

Impact of the Condition

In the US population, or in underserved sub-populations:

Is the condition or disease associated with a significant burden - in terms of prevalence, mortality, morbidity, costs to society, individual suffering, or loss of productivity?

When the Condition is a Healthcare System Problem

In the US population, or in underserved sub-populations:

Is the healthcare system problem associated with poor access to care, suboptimal care quality, high burden to society or to healthcare systems, individual suffering, mortality, morbidity, and/or loss of productivity?

Addressing Impact of the Condition or System Problem

Clinical CER State the prevalence of the condition Discuss the impact of the condition on the individual

Improving Healthcare Systems CER Describe the magnitude of the healthcare system

problem Discuss the impact of the system problem on healthcare

access and quality – and on individual and population suffering, morbidity, mortality, productivity and costs

Potential for the Study’s Findings to Improve Care and Outcomes

What knowledge gap underlies the problem?How does that gap manifest in practice?Would positive study findings close the gap and improve practice and outcomes?Would such findings be adopted widely?

Addressing Potential for Study Findings to Improve Care and OutcomesProvide evidence of an important knowledge gap: A recent systematic review or clinical guidelines pointing to a

research need. An update of previous systematic review is also acceptable.

Provide evidence of substantial variation in practice, resulting from the knowledge gap, if known.Provide indications that the proposed CER has a substantial potential to improve practice and patient outcomes? Should refer to existing efficacy or effectiveness studies IHS CER should refer to prior evidence of efficacy or

effectiveness of components of current intervention, or of effectiveness of the interventions in smaller studies or other settings

Addressing Potential (cont.)

Summarize ongoing studies of the interventionDiscuss the likelihood that positive study findings related to improvements in practice and patient outcomes would be implemented widely and quickly Breadth of current use in existing practice Extent of patient, practice or organizational barriers and facilitators to

rapid adoption by patients, clinicians and relevant health organizations

Applicability to diverse delivery systems Endorsement of the current study by key patient, physician and other

stakeholder groups.Describe how the partners in your proposal (e.g., national and/or regional stakeholder organizations) would help disseminate the study’s findings to potential adopters

Merit Review CriteriaTechnical Merit and Methodology Standards

David Hickam, MD, MPHProgram Director, Clinical Effectiveness Research

Technical Merit

A clear research plan with rigorous methods that adhere to PCORI’s Methodology Standards and accepted best practicesA clear justification for the study designA carefully constructed and realistic timeline that includes specific scientific and engagement milestones A research team with documented appropriate expertise and experience - and a rational organizational structureAn excellent research environment, including the appropriate delivery system(s) to host the study, adequate resources, and strong support of the proposed research

Addressing Technical Merit

Describe how all relevant PCORI Methodology Standards are addressedJustify the proposed sample size estimates, including the anticipated effect size and standard deviation, and all other relevant parametersDescribe anticipated rates of and feasibility of recruitment Justify the proposed study designDescribe the validity of the proposed measures of patient outcomesDiscuss proposed analytic methods, including subgroup analysesDiscuss proposed approach to consent and protection of human participants

PCORI Methodology Standards

47 standards in 11 groups.The Methodology Standards do not address all issues related to study designs and methods.Note that PCORI is not using a specific set of methodological standards for “pragmatic studies.” Consider design tradeoffs (e.g., blinding vs not blinding) Refer to other respected sources for additional guidance. View here: http://www.pcori.org/assets/2013/11/PCORI-

Methodology-Report.pdf

Sample Size Estimate and Subgroups

Document previous relevant studies to justify use of the proposed effect size.Assure appropriate adjustments based on the particular study circumstances, including the study population, expected dropout and crossover, and other parameters.Demonstrate sufficient power to analyze important pre-specified subgroups; these subgroups should be based on patient attributes with strong a priori pathophysiological or empirical justification.

Cluster Randomized Trial Design

Need justification for the use of this design State the benefits and disadvantages of a cluster design

vs. individual randomization, based on the clinical issues and the interventions being studied.

Sample size estimates need to account for reasonable estimates of intraclass correlation.Maximize the number of clusters that are randomized.

Merit Review Criteria Patient-Centeredness and Patient Engagement

Sue Sheridan, MIM, MBA, DHL

Program Director, Patient Engagement

Patient-Centeredness

Is the research focused on questions that affect outcomes of interest to patient and their caregivers? Does the research question address choices that are

important to - and faced frequently by - patients, their caregivers, or clinicians?

Is the study powered on outcomes that are important to patients?

Does the research address one or more of the key questions mentioned in PCORI’s definition of patient-centered outcomes research?

Addressing Patient-Centeredness

Provide evidence that the research question(s) and outcomes are important to patients (and/or their caregivers) Have you obtained the endorsement and participation of

major patient/caregiver organizations that would use the research?

Describe your strategy for measuring outcomes that are important to patients.

Selecting Important Patient-Centered Outcomes

Do proposed outcomes include outcomes based on input from patients and other stakeholders?What is the validity of this outcome measure?What is a minimal important difference in this outcome measure?Have you considered the use of a previously validated patient reported outcome (PRO)?

Patient and Stakeholder Engagement

Evidence that patients, caregivers, patient and caregiver organizations, clinician organizations, and other stakeholders have been and will be engaged in: Formulating the research questions Defining the characteristics of study participants, comparators and

outcomes Selecting the important outcomes to be assessed Monitoring study conduct and progress Designing plans for dissemination of study results

Clear statement of the roles and the decision-making authority of all patient and stakeholder research partners Demonstration of the principles of reciprocal relationships, co-learning, partnership, trust, transparency, and honesty

Patient-Centeredness vs. Patient Engagement

Patient-centeredness is about whether the project aims to answer questions or examine outcomes that matter to patients/caregivers.Patient engagement is about having patients/caregivers as partners in research, as opposed to merely being recruited as study participants.

The Engagement Rubric

The rubric is intended to provide guidance to applicants, merit reviewers, awardees, and engagement/program officers (for creating milestones and monitoring projects) regarding engagement in the conduct of research. It is divided into four segments:

Planning the Study

Conducting the Study

Disseminating the Study Results

PCOR Engagement Principles

Addressing Engagement

Several approaches to engagement can succeedPCORI provides many engagement resources: PCORI’s “The Engagement Rubric”

• http://www.pcori.org/sites/default/files/PCORI-Engagement-Rubric-with-Table.pdf

Sample Engagement Plans • http://www.pcori.org/sites/default/files/announcement-

resources/PCORI-Sample-Engagement-Plans.pdf Engagement in Research website page

• http://www.pcori.org/content/engagement-research PCORI’s Methodology Standards PC-1 to PC-4

• http://www.pcori.org/assets/PCORI-Methodology-Standards1.pdf

Other Topics (EHRs, Recruitment, Research Protections, & Dissemination)

Anne Trontell, MD, MPHSenior Program Officer, Clinical Effectiveness Research

Use of Electronic Health Records and Other Computerized Data Sources

Pragmatic studies should take advantage of existing electronic clinical and demographic data whenever possible.Discuss any proposed uses of existing electronic data in the approach section: Cohort identification and recruitment Collection of covariate and outcome data

Provide evidence of the validity and completeness of available data (e.g., that all follow-up and outcomes of interest are captured) PCORI Standards on Data Networks and Registries

Recruitment

Discuss past experiences with recruitmentProvide best available evidence to support your estimated recruitment successConsider barriers to recruitment – and how you plan to overcome themSome strategies for successful recruitment Engaged clinical sites Clinician advocates for the study Proactive, experienced research coordinator Minimize disruptions in clinical care processes Protocol flexibility, within reason

Study Participant Protection

Provide a Data and Safety Monitoring Plan that operates under the auspices of your institution

Assure that key personnel are educated on human subjects protections

Assure appropriate informed consent Establish procedures to minimize risks to participants Establish procedures to protect privacy and maintain

confidentiality If you anticipate seeking waiver of individual informed

consent, provide the rationale Refer to NIH standards for research involving human

subjects

Use of FDA-Regulated Medical Products

All drugs, biologics, devices, and diagnostics used in your study must be FDA-approved

• Indicate within your application whether investigational new drug (IND) or other regulatory approval will be needed to conduct your study

For example, if your study administers an approved medical product in an off-label manner (e.g. a dose, route, frequency, indication, or patient population that is unapproved by FDA) you may need an IND

• If uncertain, seek guidance from FDA or your legal counsel

Disseminating and Implementing Research Findings

PCORI does not expect applicants to disseminate and implement findings at this time. Applicants should describe the potential for dissemination and implementation.

Describe the potential for disseminating and implementing the study results in other settings.Describe possible barriers to dissemination and implementation.

What To Think About While Completing Your Application

Iris Giggetts, MSW, CRAContracts Associate, Pre-Award

Letter of Intent and Application

You were invited to submit a full application based on the information provided in the LOI, changes after the LOI require PCORI approval.

Show stoppers include:

Changes to the PI

Changes to the

Institution

Changes to the Study

Design

Changes in Engagement

PCORI Online: Application

► PI and Contact Information► Project Information► Key Personnel► Milestones► Templates and Uploads► Save and Review

Public Abstract

Project Information

Technical Abstract

Project Narratives

Milestones

► PI and Contact Information► Project Information► Key Personnel► Milestones► Templates and Uploads► Save and Review

PCORI Monitors Projects

Deliverables:Measurable and verifiable

outcomes or objects that a project team must create and deliver

according to the contract terms

Milestones:Significant events or

accomplishments within the project; may have deliverables

associated with them

Example of Milestone Schedule

Templates and Uploads

► PI and Contact Information► Project Information► Key Personnel► Milestones► Templates and Uploads► Save and Review

People and Places Template: Biosketch

You may use the NIH biosketch or PCORI’s formatBiosketches are required for all key personnelList all partners within the Key Personnel sectionPatient/Stakeholder Biosketch

PageLimit

5 Per person

People and Places Template:Project / Performance Site(s)

Demonstrate that the proposed facilities have the appropriate resources required to conduct the project to plan, within budget, and on time.

Provide a description of the facilities that will be used during the project, including capacity, capability, characteristics, proximity, and availability to the project.

PageLimit15

Professional Profile/Biosketch

Budget Templates

Three budget sections must be submitted as part of the online application process:

Detailed Budget Budget Summary Budget Justification

NOTE:A detailed budget is needed for each year of the project. Complete each budget section for the prime applicant and any/each subcontractor.

Budget and Period Limitations

Funds & BudgetDirect costs up to $10 million over the life of the projectIndirect costs: up to 40% capped for the prime and sub contractsPrime organization can only charge up to 40% IDC on first $25,000 for all sub contracts combinedInstitutional base salary up to $200,000The limit for Scientific Travel is $10,000 over the duration of the project. There is no cap on Programmatic Travel.

Period of Performance Maximum of 5 yearsRequests to extend are not permitted during any stageDo not anticipate receiving a cost OR no-cost time extension.Propose realistic timelines accounting for the burdens associated with obtaining IRB approval

Budget JustificationProvide a narrative that fully supports and explains the basis for the information found in the Detailed Budget

Provide sufficient detail to understand the basis for costs, the reason that the costs are necessary, and an explanation for major cost variances

– Provide justification as to why the costs are reasonable for the work to be performed

Provide a breakdown of costs proposed for each consortia or contractor

Must specify any other sources of funding that are anticipated to support the proposed research project

Provide quotes, indirect cost rate letter, and fringe benefit policy

Costs of Interventions

PCORI will not cover costs for clinical care alternatives that are being compared in the project.PCORI will consider covering costs for ancillary tasks necessary in the implementation or monitoring of a clinical intervention or strategy as part of the research program. Examples include costs for obtaining consent,

collecting data, or monitoring that would not normally be performed in routine care

Support for the study by the involved healthcare delivery systems must be documented.

Guidelines for Intervention Cost/Coverage

Costs for study interventions must be covered by delivery system, payer, manufacturer or developer of the intervention.The willingness of one or more of the stakeholder groups to cover treatment costs, even when one of the proposed intervention arms is not currently covered by insurance, will be taken as strong endorsement of the study by the health system or payer and of the likelihood that they will implement or use the study‘s findings if definitive.This material support for the study by host delivery system, payer or developer should therefore be discussed in the application.In exceptional cases, PCORI may consider coverage of the co-payment or coinsurance costs of participating patients when that is necessary to preserve blinding in a study or to assure access to the study for vulnerable populations. Contact PCORI with questions.

Letters of SupportAll letters of support should be addressed to the PI and institution to demonstrate the commitment of key personnel and supporting organizations to your proposed project.

Letters of support should clearly reflect the substantive involvement and material contribution to be provided by the signatory parties, and are meant to substantiate the commitment of collaboration of all forms.

Please be sure to use the Letters of Support Table provided in the funding center.

What happens to your application after you submit it?

Administrative Screening

Applicants must follow administrative requirements set in PCORI’s Application Guidelines.

► Exceeding page limits, budget, or time limitations

► Not using PCORI’s required templates► Submitting incomplete sections or applications

Missing the Mark

General Guidance

Refer to the Pragmatic Studies page in our Funding Center: http://www.pcori.org/funding-opportunities/announcement/large-pragmatic-studies-evaluate-patient-centered-outcomes-spring

Use the applicant checklist.Adhere to the formatting, page limit, and template requirements.Convert all documents to PDF and upload to PCORI Online using the appropriate filing convention.Use Biosketches for all key personnel (scientists and patient/stakeholder team members).Address Letters of Support to the PI’s institution.

Common Application Errors

Using the wrong browser, access PCORI Online via Chrome or Safari browsers

Not entering information into all required fields in the system

Not clicking the ‘Save and Review’ button

Having multiple people working on the application at the same time

Having the incorrect file extension, only PDF files can be uploaded

Not choosing the correct document type from the drop-down menu

AO is unable to view the application

Resources

Refer to the Pragmatic Studies page in our Funding Center for the following resources

PFA and Application GuidelinesPCORI Online User ManualsSample Engagement PlansGeneral Applicant FAQs: Large Pragmatic Studies Applicant FAQs: PCORI Online: https://pcori.fluxx.io/Research Methodology: http://www.pcori.org/node/4020

Submission and Key Dates

What When

Application Deadline (by invitation only)

July 31, 2015 by 5:00pmET

Merit Review Dates October 29-30, 2015

Awards Announced January 2016

Earliest Start Date March 2016

Learn More

www.pcori.orginfo@pcori.org

Questions and Answers

Submit questions via the chat function in Meeting Bridge

Ask a question via phone (press 7)

Contact Us:• Programmatic Questions: E-mail sciencequestions@pcori.org• Schedule a call at http://bit.ly/programmatic_inquiry• Call 202-627-1884 • Administrative Questions: E-mail pfa@pcori.org or call 202-627-1885

Thank You