wia net impacts: preliminary estimates adare project briefing #3 w. e. upjohn institute kevin...

WIA Net Impacts: Preliminary EstimatesADARE Project Briefing #3

Kevin Hollenbeck, W. E. Upjohn InstituteW. E. Upjohn InstituteChristopher T. King & Daniel Schroeder,Ray Marshall Center, University of Texas-AustinRay Marshall Center, University of Texas-Austin

May 7, 2003W.E.Upjohn Institutefor Employment Research

W.E.Upjohn Institutefor Employment Research

Overview

• Background & WIA Context

• Approach

• Preliminary Impact Estimates

• Discussion

• Next Steps


Background & Context

Key to assessing program impacts: Key to assessing program impacts: answer two questions:

1. What happened to those who participated in the program (the ‘treatment group’‘treatment group’)?

2. What would have happened to them if they had not participated?

First question is straightforward, but the second is more complicated…



• The second question is answered by creating a group of people who are as similar as possible to the treatment group, except that they do not receive the treatment.

• Outcomes for the treatment group and this ‘comparison group’‘comparison group’ are tracked over time.

• Program impactProgram impact is the difference between the outcomes for the two groups.



• Ideal Approach:Ideal Approach: Experimental design, which uses random assignment of participants to groups to ensure the similarity of the treatment and control groups.

• This is the “gold standard,” but it can be expensive to implement, and is not always feasible.



• An experiment is not possibleexperiment is not possible for WIA impact estimation because eligible clients cannot be denied service.

• Next-Best Approach?Next-Best Approach? Quasi-experimental design, with a comparison group selected to ensure similarity to the treatment group.

• But, must be carefully done to get at true program impacts, and not just differences that already existed between the groups.


Background ...

History of quasi-experiments:History of quasi-experiments:Multiple, complicated answers to the impact

question are not useful to policymakers.Examples: CETA & CLMS, 1st Job Corps

Evaluation —> 1985 JTLS Advisory Committee —> National JTPA Study, 2nd Job Corps Evaluation, & state evaluations (e.g., TX, UT, WA)


Background ...

WIA Context—WIA Context—• Service Sequencing:Service Sequencing: Core, Intensive &

Training Services

• Young, highly devolved programYoung, highly devolved program


Approach

Data:Data: WIASRD files for FL, GA, IL, MD, MO, TX, & WA; ES registrant data for two states only; and linked UI wage records for all states.

Impacts estimatedImpacts estimated for WIA Adults & Dislocated Workers, not Youth.


Approach ...

Treatment group =Treatment group = those who received intensive or training services and exited WIA in PY2000.

Comparison group =Comparison group = those who received core services only, and exited in PY 2000.

Thus, this design estimates the incremental impact of intensive/training, above and beyond the receipt of core services.


Approach ...

Impact estimates will be presented for these Labor Market Outcomes—Labor Market Outcomes—

• Employment in 4th post-exit quarter• Earnings in 4th post-exit quarter

Note: States de-identified until state clearance is granted. Criterion for employment is a minimum of $100 quarterly earnings.


Preliminary Impact Estimates

Outcomes for 6 states: employment & (conditional) earnings in 4th post-exit quarter.

Employment, 4th Post-Exit Quarter—• Adults:Adults: participation —> positive impacts in 3 states, negative impacts in 3 states.

• Dislocated Workers:Dislocated Workers: participation —> positive impacts in 4 states, negative in 2 states.


Impact Estimates ...

Impact on employment in 4th quarter after exit

-15.0%

-10.0%

-5.0%

0.0%

5.0%

10.0%

15.0%

State A State B State C State D State E State F

Qu

art

erl

y e

mp

loy

me

nt

imp

ac

t in

p

erc

en

tag

e p

oin

ts

Adults

Dislocated Workers



Earnings, 4th Post-Exit Quarter—

• Adults:Adults: participation —> positive impacts in 4 states, zero impact in 1 state, negative impact in 1 state.

• Dislocated Workers:Dislocated Workers: participation —> positive impacts in 2 states, negative in 4 states.



Impact on conditional earnings in 4th quarter after exit

-$1,500

-$1,000

-$500

$0

$500

$1,000

State A State B State C State D State E State F

Qu

arte

rly

earn

ing

s im

pac

t

Adults

Dislocated Workers


Discussion

[Still editing from here on…. -DGS][Still editing from here on…. -DGS]

Considerable impact variation across Considerable impact variation across subgroups (i.e., adults, dislocated workers) subgroups (i.e., adults, dislocated workers) and across statesand across states

Small sample sizes, especially for WIA core Small sample sizes, especially for WIA core servicesservices

Alternative matching techniques, e.g., caliper Alternative matching techniques, e.g., caliper matching and othersmatching and others

Local stratificationLocal stratification


Next Steps

Use ES registrant data to construct comparison Use ES registrant data to construct comparison groups for all 7 current ADARE states (groups for all 7 current ADARE states (andand add add new states)new states)

Further refine and test alternative statistical Further refine and test alternative statistical matching techniquesmatching techniques

Estimate impacts using PY2001 WIA participant Estimate impacts using PY2001 WIA participant data & longer-term UI wage recordsdata & longer-term UI wage records

Vet methods and results more widelyVet methods and results more widely


For More Information

Kevin Hollenbeck 269/[email protected]

Chris King 512/471-2186

[email protected]

Daniel Schroeder 512/[email protected]

wia net impacts: preliminary estimates adare project briefing #3 w. e. upjohn institute kevin...

Documents