Performance Measurement Under Performance Measurement Under Title 1-B of the Workforce Title 1-B of the Workforce

Investment ActInvestment Act

Regional TrainingRegional TrainingRichard WestRichard WestSocial Policy Research AssociatesSocial Policy Research Associates

Topics to Be Covered• WIA Performance Measures

– Definitions and calculations

• Challenges– Working with wage records

– Supplemental data

– When outcomes occur and are measured

– Understanding performance reports

– Implementing new data collection processes

– What about those JTPA exiters included in WIA performance measures?

• Separate measures for funding streams and youth age groups – Adults

– Dislocated workers

– Youth under 19 at registration

– Youth 19 to 21 (under 22) at registration.

• Individual served with multiple funding streams included in all relevant measures.

• (Nearly) Common measures for adults, dislocated workers, and older youth.

17 Performance Measures

• For adults, dislocated workers, older youth:– Entered employment rate– Employment retention rate

– Earnings change (Adults, Older Youth) orearnings replacement rate (DWs)

– Credential attainment

• For younger youth:– Goal attainment– Diploma attainment– Retention

• Customer satisfaction– Participants– Employers

3x4=12

3

217

Performance Measures Keyed to the Exit Quarter

• Exit quarter is the quarter containing the exit date.

• Individuals exit if:– Formally withdraw from or complete services OR– There is no service for 90 days (WIA or tracked

partner-funded services)

• Exit date is the date of withdrawal or completion or the date of the last service.

WIA ExitWIA Exit

WIA Participation No services

Last Service

End of 90-day period

WIA Exit Date

Time Periods Used for the Measures

Measures that use UI wage records. Exiters during the last 3 quarters of the

previous PY and the first quarter of the current PY. October 1 to September 30

Applies to all other measures

Measures that use "real time" data.Measures that use "real time" data. Exiters who leave during the program year.Exiters who leave during the program year. Applies to measures that do not use UI wage Applies to measures that do not use UI wage

records:records: Younger youth goal attainment Younger youth diplomaCustomer satisfaction

Exit Periods for Performance

1 4321 432

Previous PY Current PY

Real Time Measures

Wage Record Measures

EE 2Q Lag

Challenge:Working with Wage Records

Sources of Employment Data• Wage records the primary source for entered

employment and retention; • Wage records do not include:

– Individuals in uncovered employment, such as self-employment

– Federal government employees and others outside the state UI system

• May use out-of-state wage records– Bilateral agreements with nearby states– Wage Record Interchange System (WRIS)

When Are Wage Records Good Enough?• Measures sensitive to incomplete wage records• Example:

– Wage records 90% complete– “True” employment is 70%– Then measured employment is:

0.9 x 70 = 63%

A seven percentage point bias

• Really want records close to 99% complete• Typically a one or two quarter lag before wage records are

good enough

WRIS• Enables states to obtain out-of-state wage records

efficiently and cost-effectively.

• WRIS is currently fully operational with 20 pilot states

• By July 1, 2001 all states will be able to join.

• Expected to be self supporting– Cost of participation has not been determined.

– An initial pricing study is available on • WorkforceATM.org, as is other more information on WRIS

How Does WRIS Work?

• State submits a list of SSNs for matching

• WRIS matches those SSNs against an index to identify the states with records for those SSNs.

• Wage request is sent by WRIS to only those states with records for an SSN.

• Receiving states send wage records back to WRIS.

• Requesting states can download the information from WRIS.

Challenge:Developing a Realistic Process

for Collecting Supplemental Data

Sources of Employment Data 2

• Supplemental data sources may be used if not person not found in wage records. – Case management, follow-up services, participant

surveys.– Records sharing or automated record matching with

other administrative databases.• Note: records that provide information on quarterly

earnings will be considered as wage records,

– Must be collected within 30 days after individual determined to not be in wage records

Supplemental Data “By the Book”

• State gets wage record data

• Local area is told who is not in wage records

• Local area collects supplemental data within 30 days

• Supplemental data is reported to state

1 9876432

Quarters After Exit

Exit

EE 2Q Lag

2Q Lag

SD30 days to get supplemental

5

Can get supplemental early during follow-up services

RTN

Follow-up Services

Get Wage Records

SD

Data Collection Timing for Employment

“Real World” Supplemental Data• Local areas collect supplemental data during follow-up

services– For everyone– For those likely to not be in wage records:

• Self-employed, other noncovered jobs• Out of state employment

• State gets wage record data and tells local areas who is not in wage records

• Use supplemental for those not in wage records• Supplemental data reported to state

Calculating Performance Measures

Global Exclusions

• Exclusions from all performance measures:– Adults & dislocated workers who received only self-

service and informational core services– Institutionalized at exit (Item 622 = 1)– Health/medical exits (Item 622 = 2)– Deceased (Item 622 = 3)– National Emergency Grants only– Rapid Response only (except additional assistance)– Some statewide programs only

What Statewide Programs Can Be Excluded?

• Very specialized statewide programs with their own specific goals, for example:– Incumbent worker training authorized under

Section 134(a)(3)(A)(iv)(I), or

– A project for chemically dependent TANF recipients.

• These statewide programs are also excluded from WIASRD reporting (if they are the only program)

Youth Statewide? Item 307 = 1 YouthYES

Displaced Homemaker Item 308 = 1 Adult or

YouthYES

Incumbent Worker Item 309 = 1 only ExcludedYES

Other Statewide Item 310 = 1 Adult or

Dislocated WorkerYES

Rapid Response onlyItem 311 = 1

ExcludedYES

Rapid Response Additional Assistance Item 312 = 1 Dislocated Worker YES

In What Performance Groups do Statewide Participants Count?

Displaced Homemaker Item 308 = 1

Youth

Other Statewide Item 310 = 1

Adult

Ambiguous Cases--Suggested Approach

• If already in a performance group, ignore these statewide programs

Has Dislocation Date? Item 125or also in NEG grant Item 313a

Dislocated Worker

NO

YES

YES NO YES

NO Age < 18?Age >= 22?

YES

NO

Youth Services?

YES

YESNO

Entered Employment Rate

– Wage records the primary data source– Supplemental data may be used for persons not

found in wage records– Exclude adults, older youth employed at registration– Exclude older youth in advanced training/

postsecondary education (but not employed) in quarter after exit.

Number Employed in Quarter after Exitdivided by

Number of Exiters

YESYES

YES

NO

Entered Employment RateAdults and Dislocated Workers

Employed at Registration?Item 115 = 1

Excluded

Employed in Quarter After Exit ?Item 601 = 1 Numerator

DenominatorNO

Dislocated Worker Measure?

YES

NO

YESYES

NO

YES

Entered Employment Rate Older Youth

Excluded

Numerator

Denominator

Excluded

YES

In Advanced Training orPostsecondary Education

in Quarter After Exit Item 623 = 1

NO

Employed in Quarter After Exit Item 601 = 1

NO

Employed at Registration?Item 115 = 1

Employment Retention

– Based on those employed in quarter after exit– Wage records the primary data source– Can use supplemental data if not in wage records.– Excludes older youth in advanced training/

postsecondary education (but not employed) in 3rd quarter after exit.

Of those employed in the quarter after exit:

Number Employed in 3rd Quarter after Exit

divided byNumber Employed in Quarter after Exit

NO

YES

YES

Employment Retention RateAdults and Dislocated Workers

Excluded

Numerator

DenominatorNO

Employed in 3rd Quarter After Exit

Item 608 = 1

YES

Employed in Quarter After Exit Item 601 = 1

YESYES

NO

YES

NO

YES

Employment Retention RateOlder Youth

Employed Quarter After Exit?Item 601 = 1 Excluded

Employed 3rd Quarter After ExitItem 608 =1

Numerator

Denominator

Excluded

NO

In Advanced Training orPostsecondary Education in 3rd Quarter After Exit

Item 624 = 1

Earnings ChangeAdults and Older Youth

– Wage records only • Exclude if supplemental data used to determine employment:

– in either the 1st or 3rd quarter after exit.

• Exclude if preprogram quarters too far back to get data

– Exclude older youth in advanced training/ postsec. education (but not employed) in 3rd quarter after exit.

divided byNumber employed in quarter after exit

Of those employed in the quarter after exit:

minusEarnings in 2nd and 3rd Quarters

after Exit

Earnings in 2nd and 3rd Quarters before

Registration

Earnings ReplacementDislocated Workers

– Wage records only • Exclude if supplemental data used to determine employment:

– in either the 1st or 3rd quarter after exit.

• Exclude if preprogram quarters too far back to get data

• Use 2nd and 3rd quarters before registration if no dislocation date or dislocation date after registration date

– Add earnings over all included exiters before dividing

Of those employed in the quarter after exit:

Earnings in 2nd and 3rd Quarters after Exitdivided by

Earnings in 2nd and 3rd Quarters before Dislocation

NO

NO

Earnings Change--AdultsEarnings Replacement--DWs

Employed in Quarter After Exit Item 601 = 1 Excluded

Numerator and

Denominator

YES

Supplemental Data Used Q1 Item 602 = 1 or 2

Supplemental Data Used Q3 Item 609 = 1 or 2

YES

YE

S

NO

YES

In Advanced Training orPostsecondary Education in 3rd Quarter After Exit

Item 624 = 1

YES

Yes

NO

NO

Earnings Change Older YouthEmployed Quarter After Exit?

Item 601 = 1Excluded

Supplemental Data Used Q1Item 602 =1 or 2

Included

NO

Employed 3rd Quarter After ExitItem 608 =1

Supplemental Data Used Q3

Item 609 =1 or 2

YES

NO

YE

S

NO YE

S

YES

Credential Rate• Flexible definition of credential

– Standard credentials– Certificates that recognize the successful completion of training

services ….

• Credential attainment by end of 3rd qtr after exit• Positive outcome is attain credential and:

– Enter employment (adults, dislocated workers)– Enter employment or in advanced training/postsec. educ. in quarter

after exit (older youth)

• Includes:– Adults/DWs who received training.– All older youth

Credential--Adults and DWsOf those who receive training:

Credential--Older YouthNumber who enter employment and or

are in advanced training or postsecondary education in the quarter after exit

divided byNumber of exiters

Number who enter employment and attain credential divided by

Number of exiters

YES

NO

YES

Employment and Credential RateAdults and Dislocated Workers

Excluded

Numerator

Denominator

NO

NO

YES

Attained a credentialItem 621 = 1, 2 ,3, 4, 5, or 6

Employed in Quarter After ExitItem 601 =1

YES

Received Training?Item 333 is a valid date

YES

YES

Credential Rate--Older Youth

Employed in Quarter After ExitItem 601 =1

Denominator

NO

YES

YES

Numerator

NO

NO

In Advanced Training or Postsecondary Education in

Quarter After Exit Item 623 = 1

Attained a credential

Item 621 = 1, 2 ,3, 4, 5, or 6

Younger Youth Goal Attainment• Outcome attained during participation• 3 goal areas:

– Basic skills• A basic skills goal must be set if the youth is deficient in basic skills

– Work readiness skills– Occupational skills

• Flexibility in defining goals• Youth has a year to attain the goal• Maximum of 3 goals set per year (for performance)

– First year begins on registration date and ends a year later.– Goals used for performance must be identified up front

Younger Youth Goal Attainment-2

• Add numerator and denominator over all youth before dividing.

• Include all in-school youth and out-of-school youth assessed as needing basic skills, work readiness skills, or occupational skills

Number of goals attaineddivided by

Number of goals attained or reached anniversary/exit date without attaining

HS Diploma/GED Attainment

• Diploma can be attained during participation or by end of quarter after exit

• Exclude youth still in secondary school at exit

Of youth who were not HS grads at registration:

Number of exiters who attained diplomadivided by

Number of exiters not still in school at exit

YESYES

NO

YES

NO

NO

Younger Youth Diploma Rate

HS grad at registration?Item 123 > 11; Item 130 = 2 or 4 Excluded

Attained Diploma?Item 673=1 or 2

Numerator

Denominator

Excluded

YES

In Secondary School at Exit? Item 673 = 3

Younger Youth Retention• In one or more qualifying outcomes in 3rd quarter

after exit– Postsecondary education (Item 676 = 1)– Advanced training (Item 676 = 2)– Military service (Item 676 = 3)– Qualified apprenticeship (Item 675 = 4)– Unsubsidized employment (Item 676 = 5)

• Exclude youth in secondary school at exit• Not related to outcome in quarter after exit

YES

YES

YES

NO

NO

Younger Youth Retention Rate

In Secondary School at Exit? Item 673 = 3

Numerator

Denominator

Excluded

Retained in 3rd Quarter After Exit

Item 676=1,2,3,4, or 5

ChallengeUnderstanding When Outcomes

Occur and When they Are Measured

Q1 Q2 Q4Q3Q2Q1Q4Q3

PY 2000 PY 2001

Entered Employ.

Exit

EE

Retention RTN

Credential

12 Mo

Youth Diploma

12 Mo. Retention

When outcome occurs

=When Measured

When Outcomes Occur

ChallengeUnderstanding Performance

ReportsWhat you get when you ask for “the most recent data available”

1 4321 432

Previous PY Current PY

Real Time Measures

Wage Record Measures

EE

Exiters

RTN

Earnings

Yth Diploma

Look at Data Available May 15

Yth Goals

Credential

Cust Sat

Challenge:Implementing New Data

Collection Processes

Outcome Data Collection Activities• Wage records

– Collect quarterly (both in and out of state)

• Supplemental employment data– Collect during follow-up services– Collect quarterly after not found in wage records if necessary

• Credential– Collect during participation– Update after the 3rd quarter after exit

• Diploma – Collect during participation– Update after the quarter after exit

Challenge:What about those JTPA Exiters Included in WIA Performance

Measures?

JTPA exiters from October 1, 1999 to June 30, 2000 are included in :– All adult, dislocated worker, and older youth

measures– Younger youth retention measure.

• Are you getting wage records for them?• Are you collecting supplemental employment data?

(Do you need it to meet your goal?)• Are you collecting credential data?• Are you collecting younger youth retention outcomes?

It’s not too late now, but ….. .