An Expert’s PerspectiveA conversation with experts on evidence and the

role of government

An Expert’s Perspective

Moderated byWendy Spencer, CEO, Corporation for National and

Community Service

Panelists includeRon Haskins, Co-Director of the Brookings Center on

Children and Families and Budgeting for National Priorities Project, The Brookings Institution

Dr. Kathryn Newcomer, Director of the Trachtenberg School of Public Policy and Public Administration, George Washington University

Kathy Stack, Vice President of Evidence-Based Innovation, Laura and John Arnold Foundation

Ron Haskins

Co-Director of the Brookings Center on Children and Families and Budgeting for National Priorities

Project, The Brookings Institution

Show Me the Evidence: The Culture of Evidence is Growing

Growth of an Evidence-Based Culture• Obama’s evidence-based initiatives; 1,400 local projects

• Clearinghouses with lists of evidence-based programs

• OMB emphasizes evaluation by agencies

• “Nudge” unit in White House

• Growth of Pay for Success; more than 30 projects

• Results First (Pew and MacArthur; Clearinghouse)

• Results for America

• Coalition for Evidence-Based Policy

• Foundation support: Pew; MacArthur; Grant; Arnold; Annie E.

Casey; Casey Family Programs; Edna McConnell Clark; Gates;

others

Click to edit Master title style

Dr. Kathryn Newcomer

Director of the Trachtenberg School of Public Policy and Public Administration, George Washington University

newcomer@gwu.edu

Click to edit Master title style

“Evidence-based Policy,”

“Data-Driven Decision-making”–

the New Normal?

Click to edit Master title styleSetting the Stage…

• Program Evaluation: application of systematic analytical (social science) methods to address questions about program operations and results, including:

– Performance Measurement (Monitoring): routine measurement of program inputs, outputs, intermediate/ long-term outcomes attributed to a program

Both involve measurement + judgment!!

Click to edit Master title styleWhat are the Opportunities for Evidence to Inform Policymaking?

• Analyses of “performance” data collected by agencies (or delegated service delivery agents such as grantees)

• Implementation, Outcome and Impact evaluations performed by other agents for agencies

• Manipulations of services in experiments by agencies – “behavioral economics”

• Syntheses or systematic reviews of impact evaluations by external agents, e.g. websites like “What Works”

• Other?

8

Click to edit Master title styleWhat are Challenges for Evidence to Inform Policymaking?

• Expectations– What constitutes evidence?

– How transferable is evidence?

– Do we underestimate the role played by the “impactees?”

– Where is the capacity to support both the demand and supply of evidence?

9

Click to edit Master title styleWhy isn’t There Agreement About the Quality of Evidence?

• Differing professional standards and “rules” or criteria for evidence, e.g., lawyers, accountants, engineers, economists

• Disagreements about methodologies within professional groups, e.g., RCTs

• The constancy of change in problems and the characteristics of the targeted impactees

Click to edit Master title style

11

We Overstate the Certainty of the Evidence we Can Collect

Perceptions of the certainty of “evidence” have changed.

Click to edit Master title styleWe Overstate the Ease of Flow of Evidence

It plays a wide (enough) causal

role

Study conclusion: It plays a causal role there

Policy prediction: It will play a causal role

here

Source: Cartwright, N. (2013). Knowing what we are talking about: why evidence doesn't always travel.

Evidence & Policy: A Journal of Research, Debate and Practice, 9(1), 97-112.

Click to edit Master title styleWhy don’t RCT results alone make for well-supported effectiveness predictions?

It will play a positive causal role here

It can play a positive causal role here

It plays a positive causal role there

(and there)

R

C

T

R

C

T

Source: Cartwright, N. (2013). Knowing what

we are talking about: why evidence doesn't

always travel. Evidence & Policy: A Journal of

Research, Debate and Practice, 9(1), 97-112.

Click to edit Master title styleWhat is needed for a well-supported effectiveness (impact) prediction?

It will play a positive causal role here

The support factors for it are w, y, z

? ? ? ?

We have w, y, zhere

? ? ?

It can play a positive causal role here

It can play It plays a the same positive causal

role as role there (and there there)

S R RD

? T C CY

U T T

Source: Cartwright, N. (2013). Knowing what we are talking about: why evidence doesn't always travel.

Evidence & Policy: A Journal of Research, Debate and Practice, 9(1), 97-112.

Click to edit Master title styleWe Underestimate the Role of Volition Among

Impactees and their Own Heuristics

EFFECTSStructural and long term change

PREMISES

Based on information and evidence we decide that...

IMPLEMENTATION PLAN

IF we implement successfully certain activities...

ISSUEOpportunity

NeedProblem

MECHANISM

…THEN we trigger desired behaviors and

processes in a target group...

INPUTSMoneyPersonnelOther resources

ACTIONSOperationsProceduresProcesses

OUTPUTSInfrastructureServicesInformationDisincentives IncentivesChoice architecture

DETAILED PREMISES

FactsResearch result

InformationEarlier

experiencesOpinions

UNDERLYING THEORIESOn what premises do we base our

decisions?

THEORY OF IMPLEMENTATIONHow do we want to use inputs to produce

desired outputs?

THEORY OF CHANGEHow will positive change be produced?

Reaction of subjects

CHANGE

…AND THAT leads to positive, sustainable

change

Context

Heuristics and biases

Click to edit Master title styleWe Overstate The Current Evaluation Capacity in the U.S. Federal Government

Click to edit Master title styleEvaluation Capacity = Both Demand and Supply

• Who is asking for the evidence?

• Is there a clear understanding between providers and requestors for what evidence is needed?

• Are there sufficient resources within agencies to respond to demand?

• What about the lack of interaction and synergies among the different potential providers of evidence - such as GPRA/GPRAMA reporting staff, internal evaluation staff, external evaluation contractors, SBST, data.gov teams, etc.!

Click to edit Master title styleCurrently Fragmented and Non-Communicative

Performance Measurement

Impact Evaluation

Behavioral Economics

Click to edit Master title styleTo Strategic Use and Synergies!

Evaluation

Performance Measurement

Impact Evaluation

Implementation Evaluation

Behavioral Economics

Click to edit Master title styleAnswers? Apply Program Evaluation Tools

• Apply an evaluative mind set as well as apply appropriate skills

• Key elements of evaluation practice of special use in providing and assessing evidence include:

– Develop more realistic Theories of Change

– Frame appropriate questions

– Match designs to questions

20

Click to edit Master title styleProgram Logic Models Typically Ignore Context

Focus of Evaluation

Formative (Process/Implementation) Evaluation

OutcomeEvaluation

ImpactEvaluation

Desired Impact Long-term Outcomes

IntermediateOutcomes

Short-term Outcomes

OutputsActivitiesResources/Inputs

WHYHOW

Click to edit Master title styleA More Elaborate Theory of Change?

Desired Impact Long-term Outcomes

IntermediateOutcomes

Short-term Outcomes

OutputsActivitiesResources/Inputs

Contextual Factors outside of the control of program staff affect the extent to which programs and policies achieve desired outcomes!

Click to edit Master title styleIdentifying Key Contextual Factors is Especially Important in Understanding Our Theory of Change

• Assumptions about linkages, e.g., e.g., the teachers received the appropriate training to implement the new curriculum

• “Unobservables” in clients or beneficiaries- like motivation to continue in the mentoring program

• External barriers like lack of public transportation for clients to get to the service delivery site

• And some may reflect forces of nature or changes in political will, e.g., poor weather limits the amount of nature walks that are supposed to be included in the summer camping experience for inner city youth

Click to edit Master title styleFrame Useful Evaluation Questions

• The type of question – and type of desired evaluation –matter for determining appropriate methods:

– Descriptive Questions – describe input levels, outputs, contextual variables and/or measurable outcomes

– Normative Questions – assess levels of compliance for outputs or outcomes with criteria in law, regulation, or equity norms

– Impact Questions – measures of effectiveness or “impact”

– Explanatory Questions – explain implementation (fidelity) and processes

Click to edit Master title styleMatch Design to Research Questions

Principles of Evaluation Design

1. Frame the most appropriate questions to address

2. Clear and answerable evaluation questions drive the design decisions

3. Design decisions are made to provide appropriate data and comparisons needed to address the evaluation questions

4. Decisions about measurement and design are made to bolster the methodological integrity of the results

5. During design deliberations careful consideration should be given to strengthening inferences about findings

6. Goal to report that evaluation design and reporting decisions were characterized by strong methodological integrity

Click to edit Master title styleExamples

Objective Illustrative Questions Possible Design#1: Describe

program

(or regulatory)

activities

How extensive and costly are the program activities?

How do implementation efforts vary across sites,

beneficiaries, regions?

Has the program been implemented sufficiently to be

evaluated?

Performance Measurement

Exploratory Evaluations

Evaluability Assessments

Multiple Case Studies

#2: Probe

targeting &

implementation

How closely are the protocols implemented with fidelity

to the original design?

What key contextual factors are likely to affect intended

outcomes?

What feasibility or management challenges hinder

successful implementation of the program?

Multiple Case Studies

Implementation or Process

evaluations

Performance Audits

Compliance Audits

#3: Measure

impact of policy

or regulation

What are the average effects across different

implementations of the intervention?

Has implementation of the intervention produced results

consistent with its design (espoused purpose)?

Is the implementation strategy more (or less) effective in

relation to its costs?

Experimental Designs/RCTs

Non-experimental Designs: Difference-in-difference,

Propensity score matching, etc.

Cost-effectiveness & Benefit

Cost Analysis

Systematic Reviews & Meta-

Analyses

#4 : Explain how/

why produce

(un)intended

effects

How/why did the program have the intended effects?

To what extent has implementation of the program had

important unanticipated negative spillover effects?

How likely is it that the program will have similar effects

in other communities or in the future?

Impact Pathways and Process

tracing

System dynamics

Configurational analysis,

Click to edit Master title style

Kathy Stack

Vice President of Evidence-Based Innovation, Laura and John Arnold Foundation

An Expert’s Perspective – Q&A

Moderated byWendy Spencer, CEO, Corporation for National and

Community Service

Panelists includeRon Haskins, Co-Director of the Brookings Center on

Children and Families and Budgeting for National Priorities Project, The Brookings Institution

Dr. Kathryn Newcomer, Director of the Trachtenberg School of Public Policy and Public Administration, George Washington University

Kathy Stack, Vice President of Evidence-Based Innovation, Laura and John Arnold Foundation

Thank you for joining us for the 2015 CNCS Research Summit:

Focus on Evidence!

To learn more about research, evaluation and evidence at CNCS

visit us at

nationalservice.gov/evidence