douglas j. besharov school of public policy university of maryland
DESCRIPTION
The Intersection of Performance Measurement and Program Evaluation: Searching for the Counterfactual Moscow 2011 . Douglas J. Besharov School of Public Policy University of Maryland. Performance Management. Efficiency studies (“outputs”) How much does the program cost? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/1.jpg)
The Intersection of Performance Measurement and
Program Evaluation:Searching for the Counterfactual
Moscow 2011
Douglas J. BesharovSchool of Public PolicyUniversity of Maryland
![Page 2: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/2.jpg)
Performance Management
4 Efficiency studies (“outputs”)4 How much does the program cost?
4 Monetary, nonmonetary, and opportunity costs4 Could it be delivered more efficiently?
4 Effectiveness studies (“outcomes” and “impacts”)4 Does the program achieve its goals?4 Could it be more effective?
4 Both require a comparison, or a “counterfactual”
8Douglas J. Besharov, June 2011
![Page 3: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/3.jpg)
Appearances Can Be DeceivingGiving children a “Head Start”
Douglas J. Besharov, June 2011 2
![Page 4: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/4.jpg)
It Matters How Children Are Raised
Douglas J. Besharov, June 2011 3
![Page 5: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/5.jpg)
Ineffective Early Childhood Education ProgramsIHDP
1985-1988
• Low-birth weight, pre-term infants and their parents
• Home visits, parenting education, and early childhood education
• $20,400 per child per year
• No significant impacts, initial IQ gains fade
CCDP1990-1995
• Poor children under age 1 and their parents
• Case management, parenting education, early childhood education, and referrals to community-based services
• $19,000 per family per year ($60 million annually)
• No significant impacts
Early Head Start1996-2008
• Poor children ages 0-2 and their parents
• Child development, parenting education, child care, and family support services
• $18,500 per child per year ($700 million annually)
• No significant impacts
Douglas J. Besharov, June 2011 4
![Page 6: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/6.jpg)
Douglas J. Besharov (October 21, 2008)
“The closest thing to immortality on this Earth is a federal government program.”
– Ronald Reagan
Program Improvement, not Program Dismantling
![Page 7: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/7.jpg)
Performance Management
4 Leadership,
4Management, and
4Measurement
6Douglas J. Besharov, June 2011
![Page 8: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/8.jpg)
Performance Management
4 Leadership,
4Management, and
4Measurement
7Douglas J. Besharov, June 2011
![Page 9: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/9.jpg)
Point #1
Counterfactuals are needed for accurate performance measurement.
9Douglas J. Besharov, June 2011
![Page 10: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/10.jpg)
4 Head Start Impact Study (2010): 10 years and running
4 Moving to Opportunity Study (1994): 17 years and running
4 Employment Retention and Advancement evaluation (1998): 13 years and running
4 Building Strong Families Project (2002): 9 years and running
4 National Job Corps Study (1993): 15 years to complete
Impact Evaluations Take Too Long
to Manage Performance
Douglas J. Besharov, June 2011 10
![Page 11: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/11.jpg)
Logic Model for Job Training Programs
Inputs
Training facilities
Staff
Funding
Client characteristics
Activities
Job search/job readiness training
Classroom instruction
Job skills training
Outputs
Hours of training instruction
Hours of practice
Staff admin
Skill certificates
Outcomes
Job search skills
Technical job skills
Interpersonal skills
ProximalImpacts
Earnings Employment
UI/Welfare Receipt
Crime
DistalImpacts
Higher lifetime
earnings/employment
Lower poverty
Stronger economy
Theory: If government provides job training to the unemployed, than the unemployed will receive job skills necessary for good jobs, increased earnings, and a stronger economy
Design: (1) Job search/job readiness training, (2) skills training, (3) in a classroom.
External Community and Societal Context
Problem: Some unemployed do not have the necessary skills to obtain and keep well-paying employment, leading to lower income, greater use of government benefits, and a weaker economy.
![Page 12: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/12.jpg)
Point #2
Carefully applied, a measured outcome coupled with a logic model’s theory of change
—often buttressed by other evidence—can serve as a more timely and more
useful performance measure than a formal evaluation of long-term impacts.
12Douglas J. Besharov, June 2011
![Page 13: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/13.jpg)
4 There is no output, so no positive outcome can be reasonably predicted.
4 The output itself is sufficiently suggestive of a likely outcome.
4 The output is produced at such a prohibitively high cost, that, regardless of its likely outcome, it does not meet cost-effectiveness or cost-benefit tests.
When “Outputs” Imply “Outcomes”
Douglas J. Besharov, June 2011 13
![Page 14: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/14.jpg)
4 Evaluations of on-going programs4 Rolling randomized experiments
4 Pre-post studies (with embedded counterfactual)
4 Regression-discontinuity designs
4 Evaluations of specific program “improvements”4 Randomized experiments
4 Pipeline studies (or rolling implementation)
4 Interrupted time series studies
Feasible “Outcome” Evaluations
Douglas J. Besharov, June 2011 14
![Page 15: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/15.jpg)
A Clear Interrupted Time Series
Douglas J. Besharov, June 2011 15
![Page 16: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/16.jpg)
Circling the Wagons
Douglas J. Besharov, June 2011 16
![Page 17: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/17.jpg)
4 Top-down administrative and funding incentives
-- together with --
4 Bottom-up voucher-like programs
Accountability Systems
Douglas J. Besharov, June 2011 17
![Page 18: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/18.jpg)
Appearances Can Be DeceivingGiving children a “Head Start”
Douglas J. Besharov, June 2011 2
![Page 19: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/19.jpg)
![Page 20: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/20.jpg)
When “Outcomes” Imply “Impacts”
4When the desired impact is reasonably predicted to follow from the measured outcome
Douglas J. Besharov, June 2011 10
![Page 21: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/21.jpg)
![Page 22: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/22.jpg)
![Page 23: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/23.jpg)
![Page 24: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/24.jpg)
![Page 25: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/25.jpg)
![Page 26: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/26.jpg)
![Page 27: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/27.jpg)
![Page 28: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/28.jpg)
![Page 29: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/29.jpg)
Teen Pregnancy in Anson County, NCTeen Pregnancy in Anson County, NC20012001--20082008Scale: 0Scale: 0--100100
0
20
40
60
80
100
2001 2002 2003 2004 2005 2006 2007 20080
20
40
60
80
100Rate per 1,000
Adolescent Parenting Program
Number of pregnancies
![Page 30: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/30.jpg)
![Page 31: Douglas J. Besharov School of Public Policy University of Maryland](https://reader035.vdocuments.mx/reader035/viewer/2022062520/56815e43550346895dccb1aa/html5/thumbnails/31.jpg)
Ineffective Job Training ProgramsJob Corps
• Low-income youth
•
•
JTPA1987-1994
• Low-income adults, dislocated workers, and out-of school youth
• Classroom training, on-the-job training, job search assistance, adult basic education, and other services
• $2,400 per participant for 3-4 months ($60 million annually)
• Women: Small initial gains in earnings, employment, and GED receipt fade by 5 yrs
• Men: Small initial gains in earnings fade by 5 yrs, no other impacts
• Youth: On significant impacts
WIA (dislocated)2003-2005
• cts
Douglas J. Besharov, June 2011 2