return on investment and performance measurement of workforce development programs
Post on 21-Jul-2015
54 Views
Preview:
TRANSCRIPT
Return on Investment and Performance
Measurement of Workforce Development Programs
Kevin HollenbeckVice President, Senior Economist
W.E. Upjohn Institute for Employment Research
hollenbeck@upjohn.org
February 25, 2015
Presented at
OERC Conference: Workforce Outcomes: Informing Policy & Practice
through Education & Labor Market Data, Columbus, Ohio
Outline
I. Causality – How do we know what we
know?
II. Net Impact of Washington’s Workforce
Development System
III. Return on Investment
IV. Other States’ Results
V. Employer Satisfaction Performance
Standard
VI. Conclusions
Causality
Washington Legislation
• The Workforce Board is required by Washington
State RCW 28C.18.060(10) to:
Every two years administer scientifically based outcome evaluations
of the state training system, including, but not limited to, surveys of
program participants, surveys of employers of program participants,
and matches with employment security department payroll and
wage files. Every five years administer scientifically based net-
impact and cost-benefit evaluations of the state training
system. (Emphasis added.)
Causality
(Scientifically based)
Physical Science: If A B
Social Science/Education: If A usually B
How do you verify/falsify?
1) Large sample
2) Counterfactual (Absence of A) – Do you usually
get B anyway?
Causality
State of the Art:
1) Randomized Control Trials (RCTs)
2) Quasi-experimental (Statistical Matching)
requires an assumption about nonobservables
3) Econometric Models
Washington Net Impact Results
Treatment Comparison
Exited from a program in
fiscal years 2005/06 or
2007/08
Encountered Job Service in
fiscal years 2005/06 or
2007/08
Table 1 Short-Terma Net Impacts of Washington’s Education and Training System, by Program
Program
Net
Employment
Impact
(in percentage
points)
Net Hourly
Wage Impacts
(‘05 $)
Net Hours
Employed per
Quarter Impacts
Net Quarterly
Earnings
Impacts
(‘05 $)
Net Impact on
the Rate of
Receiving UI
(in percentage
points)
WIA Adults 12.8%*** $1.50*** 73.6*** $1,559*** –1.7%***
WIA Dislocated Workers 10.1%*** –$0.96*** 42.4*** $ 44 –3.1% ***
WIA Youth 8.0%*** –$0.59** 46.5*** $ 359*** 0.3% ***
Comm. College Job Prep 6.6%*** $2.75*** 59.8*** $1,856*** –4.7%***
Comm. College Worker Retraining 8.8%*** –$0.15*** 26.6*** $ 367*** –0.1%***
Comm. College ABE –2.1%*** $0.22*** 15.4*** $ 210*** –0.5%***
IBEST 3.9%*** $1.12*** 35.3*** $ 526** –1.2%***
Private Career Schools –2.7%*** $0.94*** 20.8*** $ 558*** –3.2%***
Apprenticeships 7.8%*** $7.17*** 46.9*** $4,216*** 0.7%***
High School Career Technical Ed. 6.0%*** –$0.21*** 19.2*** $ 173*** –0.2%***
Vocational Rehab. 12.8%*** $0.05*** 1.5*** $ 286** –1.8%***
Div. of Services to Blind na na na na naaDefined as third full quarter after quarter of exit. na -- not available for this draft.
***,**,*Statistically significant at the 0.01, 0.05, 0.10 levels
NOTE: Estimates are preferred specifications as “boxed” in Hollenbeck and Huang (2006) tables. They are regression-adjusted estimates of impact levels for 2007/2008 cohorts
of WIA Dislocated Workers, WIA Youth, Community College Worker Retraining, and High School Career and Technical Education. They are regression-adjusted difference-in-
difference net impact estimates for all other programs.
Washington Net Impact Results
Table 2 Long-Terma Net Impacts of Washington’s Education and Training System, by Program
Program
Net
Employment
Impact
(in percentage
points)
Net Hourly
Wage Impacts
(‘05 $)
Net Hours
Employed per
Quarter Impacts
Net
Quarterly
Earnings
Impacts
(‘05 $)
Net Impact on
the Rate of
Receiving UI
(in percentage
points)
WIA Adults 10.8%*** $1.46*** 43.6*** $ 952*** –1.9%***
WIA Dislocated Workers 4.7%*** $1.28*** 28.4*** $ 756*** –3.2%***
WIA Youth 4.3%*** $0.10*** 30.8*** $ 429*** 2.2%***
Comm. College Job Prep 10.1%*** $2.91*** 59.5*** $1,976*** –5.7%***
Comm. College Worker Retraining 7.5%*** $0.91*** 23.5*** $ 627*** –0.9%***
Comm. College ABE –3.9%*** –$0.06*** 15.1*** $ 189*** –5.6%***
Private Career Schools 3.4%*** $0.56*** 27.1*** $ 470*** –4.4%***
Apprenticeships 9.8%*** $7.97*** 26.4*** $4,019*** 5.6%***
High School Career Technical Ed. 8.4%*** $0.32*** 32.2*** $ 450*** 2.1%***
Vocational Rehab. 12.4%*** –$0.18*** 47.4*** $ 305** 0.6%***
Div. of Services to Blind na na na na naaDefined as average over quarters 9 - 12 after quarter of exit. na -- not available for this draft.
***,**,*Statistically significant at the 0.01, 0.05, 0.10 levels
Note: Estimates are preferred specifications as “boxed” in Hollenbeck and Huang (2006) tables. They are regression-adjusted estimates of impact levels for 2005/2006 cohort of
WIA Dislocated Workers, WIA Youth, Community College Worker Retraining, and High School Career and Technical Education. They are regression-adjusted difference-in-
difference net impact estimates for all other programs.
Washington Net Impact Results
Washington Net Impact Results
What Does State Do with Results?
Return on Investment (Theory)
Benefits Participants Taxpayers Society
Higher productivity 0 + +
Higher earnings + 0 +
Fringe benefits + 0 +
Less unemployment/turnover + + +
Lower income maintenance transfers - + 0
Higher taxes - + 0
Costs
Tuition, fees -/0 - -
Forgone wages - 0 -
Total (Net Benefits) + ?? +/??
Return on Investment (Actual
Framework)
Benefits Participants Taxpayers Society
Higher productivity 0 + +
Higher earnings + 0 +
Fringe benefits + 0 +
Less unemployment/turnover + + +
Lower income maintenance transfers - + 0
Higher taxes - + 0
Costs
Tuition, fees -/0 - -
Forgone wages - 0 -
Total (Net Benefits) + ?? +/??
Note: Entries in the table that are highlighted are omitted from empirical estimation.
Return on Investment – Secondary CTE
Table 4 Participant and Taxpayer Benefits and Costs per Completer in Secondary CTE Programs
Benefit/Cost
First 2.5 years Lifetime (until 65)
Participants Taxpayers Participants Taxpayers
Benefit
Earnings
Fringe Benefits
Taxes
2,753
551
−475
0
0
475
34,603
6,920
−5,969
0
0
5,969
Transfers
UI
TANF
FS
Medicaid
16
−91
−73
−27
−16
91
73
27
168
−442
−380
−169
−168
442
380
169
Costs
Forgone earnings
Program costs
−28
0
0
704
−28
0
0
704
NOTE: ’00 $. Discount rate of 3% real.
SOURCE: Hollenbeck and Huang (2006).
Return on Investment – Postsecondary
CTE
Table 3 Participant and Taxpayer Benefits and Costs per Participant in Postsecondary CTE Programs
Benefit/Cost
First 2.5 years Lifetime (until 65)
Participants Taxpayers Participants Taxpayers
Benefit
Earnings
Fringe Benefits
Taxes
10,386
2,077
−1,792
0
0
1,792
79,239
15,848
−13,669
0
0
13,669
Transfers
UI
TANF
FS
Medicaid
−2,137
351
107
45
2,137
−351
−107
−45
−2,629
933
331
161
2,629
−933
−331
−161
Costs
Forgone earnings
Program costs
2,100
3,519
0
6,877
2,100
3,519
0
6,877
NOTE: ’00 $. Discount rate of 3% real.
SOURCE: Hollenbeck and Huang (2006).
Return on Investment
Table 5 Benefits, Costs, and Annual Rates of Return for Postsecondary and Secondary CTE Programs
over the First 2.5 Years and Lifetime for the Average Participant
Benefit/Cost
Secondary CTE Postsecondary CTE
First 2.5 years Lifetime (age 65) First 2.5 years Lifetime (age 65)
Participants
Benefits
Costs
irr
2,654
-28
na
34,731
-28
na
9,037
5,619
20.93%
80,204
5,619
8.81%
Taxpayers
Benefits
Costs
irr
650
704
-3.14%
6,792
704
4.70%
3,431
6,877
-24.28%
14,873
6,877
2.48%
Society
Benefits
Costs
irr
3,304
676
88.64%
41,523
676
25.35%
12,468
12,496
-0.09%
95,077
12,496
6.65%NOTE: Table entries are for average participant. Benefits include earnings, fringe benefits, and income-related transfer
payments. Costs include tuition and fees (if any), foregone earnings, and public program costs per participant. $ figures
are in real 2000$. na means that irr could not be calculated because of 0 or negative costs. Discount rate is 3% real.
Return on Investment
• Secondary and postsecondary CTE have huge
payoffs for participants, especially in short-run
• State/taxpayers eventually get positive ROI, but
payback takes slightly more than 2.5 years for
secondary and considerably longer for
postsecondary
• “Social” ROI quite positive for secondary and for
postsecondary
Results from Other States
Table 5 A Comparison of Findings Across States
Outcome WIA-Adult
WIA-Dislocated
Workers WIA-Youth TAA
Postscondary
Education
Employment in 3rd quarter (%)
Indiana 14.8*** 17.0*** 3.4 3.2 17.9***
Washington 9.7*** 8.7*** 4.2** na 10.3***
Virginiaa, c 3.4*** −3.9** −5.9*** 2.8***
Quarterly earnings in 3rd quarterb ($)
Indiana 549** 410** 24 −122 1,490***
Washington 711*** 784*** 66 na 1,275***
Virginiaa, c 146*** 62 −154*** 1,539***
NOTES: *, **, *** impact estimate is significant at 0.10, 0.05, and 0.01 level. aWIA-Adults and WIA-Dislocated Workers were combined in the study.bIndiana results in 2008$; Washington and Virginia in 2005/2006$.cVirginia results are for the 4th quarter after exit.
Results from Other States
Table 6 Net Impacts, Disaggregated by Training Status
Outcome
WIA-Adults WIA-Dislocated Workers
All Training No Training All Training No Training
Employment, 3rd Qtr (%) 14.8*** 19.2*** 9.5*** 17.0*** 15.4*** 18.3***
Employment, 7th Qtr (%) 13.7*** 18.2*** 8.2*** 16.5*** 15.9*** 17.0***
Earnings, 3rd Qtr ($) 549*** 751*** 339*** 410*** 482*** 354***
Earnings, 7th Qtr ($) 463*** 692*** 221*** 310** 394*** 245***
UI benefits, 3rd Qtr ($) −15 −17 −12 −53** −70** −39**
UI benefits, 7th Qtr ($) 10 1 21 3 −20 21
NOTES: *, **, *** impact estimate is significant at 0.10, 0.05, and 0.01 level.
Results from Other States
Table 7 Quarterly ROIs Using Lifetime Earnings Flow Estimates
WIA-Adult
WIA-
Dislocated
Workers WIA-Youth TAA Postsecondary
Individual program participant (%) 16.32 2.64 13.27 −0.93 29.87
Government (%) −0.04 1.50 −1.73 5.01 1.82
Society (takes into consideration
individual and government) (%)7.60 2.13 0.22 −0.40 9.66
Results from Other States
Table 2. Net Impact Indicators for Virginia’s Workforce Programs, FY2005
Program
Short-term
employment/
in school rate
Long-term
employment/
in school rate
Short-term
earnings
level
Long-term
earnings
level
Credential
completion rate
DOE and VCCS programs
AEL (DOE)
Postsecondary CTE (VCCS)
−9.58***
0.49
−9.07***
2.81***
289***
1,213***
-21
1,539***
65.48***
22.68***
DRS and DBVI programs
DRS
DBVI
17.63***
17.37***
16.17***
25.00***
429***
1,948***
241***
1,318**
8.79***
6.78***
DSS programs
FSET
TANF/VIEW
−9.29***
−2.22***
−9.55***
−2.30***
−404***
414***
−529***
175*
−0.41***
0.46***
VEC and Senior Advisor programs
TAA (VEC)
WIA Adultsa (Senior Advisor)
WIA Youth (Senior Advisor)
−6.15***
4.75***
−2.91*
−5.88***
3.39***
−3.88**
−210***
442***
480***
−154***
146**
62
65.03***
53.96***
76.12***
W/P (VEC) post–pre impactsb −0.36 0.09 −250*** 38* na
aIncludes WIA Dislocated Workers.bW/P program impacts were computed using a post-pre methodology as described in text. na means not available because credential completion is not meaningful in a post-pre
methodology.
Significance levels (one-tailed test): * 10%; ** 5%; *** 1%.
Results from Other States
Other Studies:
• Heinrich, Mueser, Troske (2008) – 12 States
WIA Adult and Dislocated Worker Titles
• Hollenbeck, Schroeder, King, Huang (2005) – 7
states
WIA Adult and Dislocated Worker Titles
Employer Satisfaction Performance
Standard
WIOA (Title I, Chapter 4, Section 116 (b)(A)(iv))
(iv) INDICATOR FOR SERVICES TO EMPLOYERS.—Prior to the
commencement of the second full program year after the date of
enactment of this Act, for purposes of clauses (i)(VI), or clause
(ii)(III) with respect to clause (i)(IV), the Secretary of Labor and the
Secretary of Education, after consultation with the representatives
described in paragraph (4)(B), shall jointly develop and establish, for
purposes of this subparagraph,1 or more primary indicators of
performance that indicate the effectiveness of the core programs in
serving employers.
Employer Satisfaction Performance
StandardTable 1. Gross Impact Indicators for Virginia’s Workforce Programs, FY2005
Program
Short-term
employment/
in school rate
Long-term
employment/
in school rate
Short-term
earnings
level
Long-term
earnings
level
Credential
completion
rate
Percent of
repeat
employer
customers
DOE and VCCS programs
AEL (DOE) 59.99 59.38 3,572 3,824 66.19 50.46
Postsecondary CTE (VCCS) 72.76 73.01 5,426 6,064 25.48 45.15
DRS and DBVI programs
DRS 54.85 53.98 2,822 3,052 20.19 46.07
DBVI 38.13 40.13 4,176 4,083 19.73 6.98
DSS programs
FSET 54.93 53.16 2,717 2,961 0.20 68.59
TANF/VIEW 63.34 61.85 3,211 3,410 1.11 58.04
VEC and Senior Advisor programs
TAA (VEC) 67.93 68.46 4,366 4,965 65.02 45.40
W/P (VEC) 71.15 70.14 4,257 4,729 0.65 84.58
WIA Adultsa (Senior Advisor) 75.98 74.12 4,439 4,733 54.91 51.86
WIA Youth (Senior Advisor) 71.61 67.04 1,637 1,761 81.39 49.20
aIncludes WIA Dislocated Workers.
NOTE: The workforce programs vary considerably in size. Table C.2 shows the number of records of exiters that were supplied to us by program.
Conclusions
Have we identified causality? To RCT purists, the
answer is no.
But suppose we use a “probable cause” standard,
then consistency across study results may meet
that standard:
• JTPA/WIA works for disadvantaged adults
• Postsecondary CTE works
• Voc. Rehab. works
• Apprenticeship works
Conclusions (Continued)
• Adult basic education “struggles” (labor market
outcomes may not be most important)
Conflicting Results
• Dislocated workers
• WIA Youth
Conclusions (Continued)
My advice:
• Never make a decision based on a single study (too
many parameters, other sources of uncertainty)
• However, (administrative) data can:
i) Be used for program improvement by looking at
trends and subgroups
ii) Be used for accountability and performance
measurement because they can be used, at minimal
expense, to examine trends and subgroups.
Thank you!
top related