return on investment and performance measurement of workforce development programs

Return on Investment and Performance

Measurement of Workforce Development Programs

Kevin HollenbeckVice President, Senior Economist

W.E. Upjohn Institute for Employment Research

[email protected]

February 25, 2015

Presented at

OERC Conference: Workforce Outcomes: Informing Policy & Practice

through Education & Labor Market Data, Columbus, Ohio

mailto:[email protected]

Outline

I. Causality – How do we know what we

know?

II. Net Impact of Washington’s Workforce

Development System

III. Return on Investment

IV. Other States’ Results

V. Employer Satisfaction Performance

Standard

VI. Conclusions

Causality

Washington Legislation

• The Workforce Board is required by Washington

State RCW 28C.18.060(10) to:

Every two years administer scientifically based outcome evaluations

of the state training system, including, but not limited to, surveys of

program participants, surveys of employers of program participants,

and matches with employment security department payroll and

wage files. Every five years administer scientifically based net-

impact and cost-benefit evaluations of the state training

system. (Emphasis added.)

Causality

(Scientifically based)

Physical Science: If A B

Social Science/Education: If A usually B

How do you verify/falsify?

1) Large sample

2) Counterfactual (Absence of A) – Do you usually

get B anyway?

Causality

State of the Art:

1) Randomized Control Trials (RCTs)

2) Quasi-experimental (Statistical Matching)

requires an assumption about nonobservables

3) Econometric Models

Washington Net Impact Results

Treatment Comparison

Exited from a program in

fiscal years 2005/06 or

2007/08

Encountered Job Service in

fiscal years 2005/06 or

2007/08

Table 1 Short-Terma Net Impacts of Washington’s Education and Training System, by Program

Program

Net

Employment

Impact

(in percentage

points)

Net Hourly

Wage Impacts

(‘05 $)

Net Hours

Employed per

Quarter Impacts

Net Quarterly

Earnings

Impacts

(‘05 $)

Net Impact on

the Rate of

Receiving UI

(in percentage

points)

WIA Adults 12.8%*** $1.50*** 73.6*** $1,559*** –1.7%***

WIA Dislocated Workers 10.1%*** –$0.96*** 42.4*** $ 44 –3.1% ***

WIA Youth 8.0%*** –$0.59** 46.5*** $ 359*** 0.3% ***

Comm. College Job Prep 6.6%*** $2.75*** 59.8*** $1,856*** –4.7%***

Comm. College Worker Retraining 8.8%*** –$0.15*** 26.6*** $ 367*** –0.1%***

Comm. College ABE –2.1%*** $0.22*** 15.4*** $ 210*** –0.5%***

IBEST 3.9%*** $1.12*** 35.3*** $ 526** –1.2%***

Private Career Schools –2.7%*** $0.94*** 20.8*** $ 558*** –3.2%***

Apprenticeships 7.8%*** $7.17*** 46.9*** $4,216*** 0.7%***

High School Career Technical Ed. 6.0%*** –$0.21*** 19.2*** $ 173*** –0.2%***

Vocational Rehab. 12.8%*** $0.05*** 1.5*** $ 286** –1.8%***

Div. of Services to Blind na na na na naaDefined as third full quarter after quarter of exit. na -- not available for this draft.

***,**,*Statistically significant at the 0.01, 0.05, 0.10 levels

NOTE: Estimates are preferred specifications as “boxed” in Hollenbeck and Huang (2006) tables. They are regression-adjusted estimates of impact levels for 2007/2008 cohorts

of WIA Dislocated Workers, WIA Youth, Community College Worker Retraining, and High School Career and Technical Education. They are regression-adjusted difference-in-

difference net impact estimates for all other programs.


Table 2 Long-Terma Net Impacts of Washington’s Education and Training System, by Program

Program

Net

Employment

Impact

(in percentage

points)

Net Hourly

Wage Impacts

(‘05 $)

Net Hours

Employed per

Quarter Impacts

Net

Quarterly

Earnings

Impacts

(‘05 $)

Net Impact on

the Rate of

Receiving UI

(in percentage

points)

WIA Adults 10.8%*** $1.46*** 43.6*** $ 952*** –1.9%***

WIA Dislocated Workers 4.7%*** $1.28*** 28.4*** $ 756*** –3.2%***

WIA Youth 4.3%*** $0.10*** 30.8*** $ 429*** 2.2%***

Comm. College Job Prep 10.1%*** $2.91*** 59.5*** $1,976*** –5.7%***

Comm. College Worker Retraining 7.5%*** $0.91*** 23.5*** $ 627*** –0.9%***

Comm. College ABE –3.9%*** –$0.06*** 15.1*** $ 189*** –5.6%***

Private Career Schools 3.4%*** $0.56*** 27.1*** $ 470*** –4.4%***

Apprenticeships 9.8%*** $7.97*** 26.4*** $4,019*** 5.6%***

High School Career Technical Ed. 8.4%*** $0.32*** 32.2*** $ 450*** 2.1%***

Vocational Rehab. 12.4%*** –$0.18*** 47.4*** $ 305** 0.6%***

Div. of Services to Blind na na na na naaDefined as average over quarters 9 - 12 after quarter of exit. na -- not available for this draft.

***,**,*Statistically significant at the 0.01, 0.05, 0.10 levels

Note: Estimates are preferred specifications as “boxed” in Hollenbeck and Huang (2006) tables. They are regression-adjusted estimates of impact levels for 2005/2006 cohort of

WIA Dislocated Workers, WIA Youth, Community College Worker Retraining, and High School Career and Technical Education. They are regression-adjusted difference-in-

difference net impact estimates for all other programs.



What Does State Do with Results?

Return on Investment (Theory)

Benefits Participants Taxpayers Society

Higher productivity 0 + +

Higher earnings + 0 +

Fringe benefits + 0 +

Less unemployment/turnover + + +

Lower income maintenance transfers - + 0

Higher taxes - + 0

Costs

Tuition, fees -/0 - -

Forgone wages - 0 -

Total (Net Benefits) + ?? +/??

Return on Investment (Actual

Framework)

Benefits Participants Taxpayers Society

Higher productivity 0 + +

Higher earnings + 0 +

Fringe benefits + 0 +

Less unemployment/turnover + + +

Lower income maintenance transfers - + 0

Higher taxes - + 0

Costs

Tuition, fees -/0 - -

Forgone wages - 0 -

Total (Net Benefits) + ?? +/??

Note: Entries in the table that are highlighted are omitted from empirical estimation.

Return on Investment – Secondary CTE

Table 4 Participant and Taxpayer Benefits and Costs per Completer in Secondary CTE Programs

Benefit/Cost

First 2.5 years Lifetime (until 65)

Participants Taxpayers Participants Taxpayers

Benefit

Earnings

Fringe Benefits

Taxes

2,753

551

−475

0

0

475

34,603

6,920

−5,969

0

0

5,969

Transfers

UI

TANF

FS

Medicaid

16

−91

−73

−27

−16

91

73

27

168

−442

−380

−169

−168

442

380

169

Costs

Forgone earnings

Program costs

−28

0

0

704

−28

0

0

704

NOTE: ’00 $. Discount rate of 3% real.

SOURCE: Hollenbeck and Huang (2006).

Return on Investment – Postsecondary

CTE

Table 3 Participant and Taxpayer Benefits and Costs per Participant in Postsecondary CTE Programs

Benefit/Cost

First 2.5 years Lifetime (until 65)

Participants Taxpayers Participants Taxpayers

Benefit

Earnings

Fringe Benefits

Taxes

10,386

2,077

−1,792

0

0

1,792

79,239

15,848

−13,669

0

0

13,669

Transfers

UI

TANF

FS

Medicaid

−2,137

351

107

45

2,137

−351

−107

−45

−2,629

933

331

161

2,629

−933

−331

−161

Costs

Forgone earnings

Program costs

2,100

3,519

0

6,877

2,100

3,519

0

6,877

NOTE: ’00 $. Discount rate of 3% real.

SOURCE: Hollenbeck and Huang (2006).

Return on Investment

Table 5 Benefits, Costs, and Annual Rates of Return for Postsecondary and Secondary CTE Programs

over the First 2.5 Years and Lifetime for the Average Participant

Benefit/Cost

Secondary CTE Postsecondary CTE

First 2.5 years Lifetime (age 65) First 2.5 years Lifetime (age 65)

Participants

Benefits

Costs

irr

2,654

-28

na

34,731

-28

na

9,037

5,619

20.93%

80,204

5,619

8.81%

Taxpayers

Benefits

Costs

irr

650

704

-3.14%

6,792

704

4.70%

3,431

6,877

-24.28%

14,873

6,877

2.48%

Society

Benefits

Costs

irr

3,304

676

88.64%

41,523

676

25.35%

12,468

12,496

-0.09%

95,077

12,496

6.65%NOTE: Table entries are for average participant. Benefits include earnings, fringe benefits, and income-related transfer

payments. Costs include tuition and fees (if any), foregone earnings, and public program costs per participant. $ figures

are in real 2000$. na means that irr could not be calculated because of 0 or negative costs. Discount rate is 3% real.

Return on Investment

• Secondary and postsecondary CTE have huge

payoffs for participants, especially in short-run

• State/taxpayers eventually get positive ROI, but

payback takes slightly more than 2.5 years for

secondary and considerably longer for

postsecondary

• “Social” ROI quite positive for secondary and for

postsecondary

Results from Other States

Table 5 A Comparison of Findings Across States

Outcome WIA-Adult

WIA-Dislocated

Workers WIA-Youth TAA

Postscondary

Education

Employment in 3rd quarter (%)

Indiana 14.8*** 17.0*** 3.4 3.2 17.9***

Washington 9.7*** 8.7*** 4.2** na 10.3***

Virginiaa, c 3.4*** −3.9** −5.9*** 2.8***

Quarterly earnings in 3rd quarterb ($)

Indiana 549** 410** 24 −122 1,490***

Washington 711*** 784*** 66 na 1,275***

Virginiaa, c 146*** 62 −154*** 1,539***

NOTES: *, **, *** impact estimate is significant at 0.10, 0.05, and 0.01 level. aWIA-Adults and WIA-Dislocated Workers were combined in the study.bIndiana results in 2008$; Washington and Virginia in 2005/2006$.cVirginia results are for the 4th quarter after exit.


Table 6 Net Impacts, Disaggregated by Training Status

Outcome

WIA-Adults WIA-Dislocated Workers

All Training No Training All Training No Training

Employment, 3rd Qtr (%) 14.8*** 19.2*** 9.5*** 17.0*** 15.4*** 18.3***

Employment, 7th Qtr (%) 13.7*** 18.2*** 8.2*** 16.5*** 15.9*** 17.0***

Earnings, 3rd Qtr ($) 549*** 751*** 339*** 410*** 482*** 354***

Earnings, 7th Qtr ($) 463*** 692*** 221*** 310** 394*** 245***

UI benefits, 3rd Qtr ($) −15 −17 −12 −53** −70** −39**

UI benefits, 7th Qtr ($) 10 1 21 3 −20 21

NOTES: *, **, *** impact estimate is significant at 0.10, 0.05, and 0.01 level.


Table 7 Quarterly ROIs Using Lifetime Earnings Flow Estimates

WIA-Adult

WIA-

Dislocated

Workers WIA-Youth TAA Postsecondary

Individual program participant (%) 16.32 2.64 13.27 −0.93 29.87

Government (%) −0.04 1.50 −1.73 5.01 1.82

Society (takes into consideration

individual and government) (%)7.60 2.13 0.22 −0.40 9.66


Table 2. Net Impact Indicators for Virginia’s Workforce Programs, FY2005

Program

Short-term

employment/

in school rate

Long-term

employment/

in school rate

Short-term

earnings

level

Long-term

earnings

level

Credential

completion rate

DOE and VCCS programs

AEL (DOE)

Postsecondary CTE (VCCS)

−9.58***

0.49

−9.07***

2.81***

289***

1,213***

-21

1,539***

65.48***

22.68***

DRS and DBVI programs

DRS

DBVI

17.63***

17.37***

16.17***

25.00***

429***

1,948***

241***

1,318**

8.79***

6.78***

DSS programs

FSET

TANF/VIEW

−9.29***

−2.22***

−9.55***

−2.30***

−404***

414***

−529***

175*

−0.41***

0.46***

VEC and Senior Advisor programs

TAA (VEC)

WIA Adultsa (Senior Advisor)

WIA Youth (Senior Advisor)

−6.15***

4.75***

−2.91*

−5.88***

3.39***

−3.88**

−210***

442***

480***

−154***

146**

62

65.03***

53.96***

76.12***

W/P (VEC) post–pre impactsb −0.36 0.09 −250*** 38* na

aIncludes WIA Dislocated Workers.bW/P program impacts were computed using a post-pre methodology as described in text. na means not available because credential completion is not meaningful in a post-pre

methodology.

Significance levels (one-tailed test): * 10%; ** 5%; *** 1%.


Other Studies:

• Heinrich, Mueser, Troske (2008) – 12 States

WIA Adult and Dislocated Worker Titles

• Hollenbeck, Schroeder, King, Huang (2005) – 7

states

WIA Adult and Dislocated Worker Titles

Employer Satisfaction Performance

Standard

WIOA (Title I, Chapter 4, Section 116 (b)(A)(iv))

(iv) INDICATOR FOR SERVICES TO EMPLOYERS.—Prior to the

commencement of the second full program year after the date of

enactment of this Act, for purposes of clauses (i)(VI), or clause

(ii)(III) with respect to clause (i)(IV), the Secretary of Labor and the

Secretary of Education, after consultation with the representatives

described in paragraph (4)(B), shall jointly develop and establish, for

purposes of this subparagraph,1 or more primary indicators of

performance that indicate the effectiveness of the core programs in

serving employers.

Employer Satisfaction Performance

StandardTable 1. Gross Impact Indicators for Virginia’s Workforce Programs, FY2005

Program

Short-term

employment/

in school rate

Long-term

employment/

in school rate

Short-term

earnings

level

Long-term

earnings

level

Credential

completion

rate

Percent of

repeat

employer

customers

DOE and VCCS programs

AEL (DOE) 59.99 59.38 3,572 3,824 66.19 50.46

Postsecondary CTE (VCCS) 72.76 73.01 5,426 6,064 25.48 45.15

DRS and DBVI programs

DRS 54.85 53.98 2,822 3,052 20.19 46.07

DBVI 38.13 40.13 4,176 4,083 19.73 6.98

DSS programs

FSET 54.93 53.16 2,717 2,961 0.20 68.59

TANF/VIEW 63.34 61.85 3,211 3,410 1.11 58.04

VEC and Senior Advisor programs

TAA (VEC) 67.93 68.46 4,366 4,965 65.02 45.40

W/P (VEC) 71.15 70.14 4,257 4,729 0.65 84.58

WIA Adultsa (Senior Advisor) 75.98 74.12 4,439 4,733 54.91 51.86

WIA Youth (Senior Advisor) 71.61 67.04 1,637 1,761 81.39 49.20

aIncludes WIA Dislocated Workers.

NOTE: The workforce programs vary considerably in size. Table C.2 shows the number of records of exiters that were supplied to us by program.

Conclusions

Have we identified causality? To RCT purists, the

answer is no.

But suppose we use a “probable cause” standard,

then consistency across study results may meet

that standard:

• JTPA/WIA works for disadvantaged adults

• Postsecondary CTE works

• Voc. Rehab. works

• Apprenticeship works

Conclusions (Continued)

• Adult basic education “struggles” (labor market

outcomes may not be most important)

Conflicting Results

• Dislocated workers

• WIA Youth

Conclusions (Continued)

My advice:

• Never make a decision based on a single study (too

many parameters, other sources of uncertainty)

• However, (administrative) data can:

i) Be used for program improvement by looking at

trends and subgroups

ii) Be used for accountability and performance

measurement because they can be used, at minimal

expense, to examine trends and subgroups.

Thank you!