performance measurement under title 1-b of the workforce investment act regional training
DESCRIPTION
Performance Measurement Under Title 1-B of the Workforce Investment Act Regional Training Richard West Social Policy Research Associates. Topics to Be Covered. WIA Performance Measures Definitions and calculations Challenges Working with wage records Supplemental data - PowerPoint PPT PresentationTRANSCRIPT
Performance Measurement Under Performance Measurement Under Title 1-B of the Workforce Title 1-B of the Workforce
Investment ActInvestment Act
Regional TrainingRegional TrainingRichard WestRichard WestSocial Policy Research AssociatesSocial Policy Research Associates
Topics to Be Covered• WIA Performance Measures
– Definitions and calculations
• Challenges– Working with wage records
– Supplemental data
– When outcomes occur and are measured
– Understanding performance reports
– Implementing new data collection processes
– What about those JTPA exiters included in WIA performance measures?
• Separate measures for funding streams and youth age groups – Adults
– Dislocated workers
– Youth under 19 at registration
– Youth 19 to 21 (under 22) at registration.
• Individual served with multiple funding streams included in all relevant measures.
• (Nearly) Common measures for adults, dislocated workers, and older youth.
17 Performance Measures
• For adults, dislocated workers, older youth:– Entered employment rate– Employment retention rate
– Earnings change (Adults, Older Youth) orearnings replacement rate (DWs)
– Credential attainment
• For younger youth:– Goal attainment– Diploma attainment– Retention
• Customer satisfaction– Participants– Employers
3x4=12
3
217
Performance Measures Keyed to the Exit Quarter
• Exit quarter is the quarter containing the exit date.
• Individuals exit if:– Formally withdraw from or complete services OR– There is no service for 90 days (WIA or tracked
partner-funded services)
• Exit date is the date of withdrawal or completion or the date of the last service.
WIA ExitWIA Exit
WIA Participation No services
Last Service
End of 90-day period
WIA Exit Date
Time Periods Used for the Measures
Measures that use UI wage records. Exiters during the last 3 quarters of the
previous PY and the first quarter of the current PY. October 1 to September 30
Applies to all other measures
Measures that use "real time" data.Measures that use "real time" data. Exiters who leave during the program year.Exiters who leave during the program year. Applies to measures that do not use UI wage Applies to measures that do not use UI wage
records:records: Younger youth goal attainment Younger youth diplomaCustomer satisfaction
Exit Periods for Performance
1 4321 432
Previous PY Current PY
Real Time Measures
Wage Record Measures
EE 2Q Lag
Challenge:Working with Wage Records
Sources of Employment Data• Wage records the primary source for entered
employment and retention; • Wage records do not include:
– Individuals in uncovered employment, such as self-employment
– Federal government employees and others outside the state UI system
• May use out-of-state wage records– Bilateral agreements with nearby states– Wage Record Interchange System (WRIS)
When Are Wage Records Good Enough?• Measures sensitive to incomplete wage records• Example:
– Wage records 90% complete– “True” employment is 70%– Then measured employment is:
0.9 x 70 = 63%
A seven percentage point bias
• Really want records close to 99% complete• Typically a one or two quarter lag before wage records are
good enough
WRIS• Enables states to obtain out-of-state wage records
efficiently and cost-effectively.
• WRIS is currently fully operational with 20 pilot states
• By July 1, 2001 all states will be able to join.
• Expected to be self supporting– Cost of participation has not been determined.
– An initial pricing study is available on • WorkforceATM.org, as is other more information on WRIS
How Does WRIS Work?
• State submits a list of SSNs for matching
• WRIS matches those SSNs against an index to identify the states with records for those SSNs.
• Wage request is sent by WRIS to only those states with records for an SSN.
• Receiving states send wage records back to WRIS.
• Requesting states can download the information from WRIS.
Challenge:Developing a Realistic Process
for Collecting Supplemental Data
Sources of Employment Data 2
• Supplemental data sources may be used if not person not found in wage records. – Case management, follow-up services, participant
surveys.– Records sharing or automated record matching with
other administrative databases.• Note: records that provide information on quarterly
earnings will be considered as wage records,
– Must be collected within 30 days after individual determined to not be in wage records
Supplemental Data “By the Book”
• State gets wage record data
• Local area is told who is not in wage records
• Local area collects supplemental data within 30 days
• Supplemental data is reported to state
1 9876432
Quarters After Exit
Exit
EE 2Q Lag
2Q Lag
SD30 days to get supplemental
5
Can get supplemental early during follow-up services
RTN
Follow-up Services
Get Wage Records
SD
Data Collection Timing for Employment
“Real World” Supplemental Data• Local areas collect supplemental data during follow-up
services– For everyone– For those likely to not be in wage records:
• Self-employed, other noncovered jobs• Out of state employment
• State gets wage record data and tells local areas who is not in wage records
• Use supplemental for those not in wage records• Supplemental data reported to state
Calculating Performance Measures
Global Exclusions
• Exclusions from all performance measures:– Adults & dislocated workers who received only self-
service and informational core services– Institutionalized at exit (Item 622 = 1)– Health/medical exits (Item 622 = 2)– Deceased (Item 622 = 3)– National Emergency Grants only– Rapid Response only (except additional assistance)– Some statewide programs only
What Statewide Programs Can Be Excluded?
• Very specialized statewide programs with their own specific goals, for example:– Incumbent worker training authorized under
Section 134(a)(3)(A)(iv)(I), or
– A project for chemically dependent TANF recipients.
• These statewide programs are also excluded from WIASRD reporting (if they are the only program)
Youth Statewide? Item 307 = 1 YouthYES
Displaced Homemaker Item 308 = 1 Adult or
YouthYES
Incumbent Worker Item 309 = 1 only ExcludedYES
Other Statewide Item 310 = 1 Adult or
Dislocated WorkerYES
Rapid Response onlyItem 311 = 1
ExcludedYES
Rapid Response Additional Assistance Item 312 = 1 Dislocated Worker YES
In What Performance Groups do Statewide Participants Count?
Displaced Homemaker Item 308 = 1
Youth
Other Statewide Item 310 = 1
Adult
Ambiguous Cases--Suggested Approach
• If already in a performance group, ignore these statewide programs
Has Dislocation Date? Item 125or also in NEG grant Item 313a
Dislocated Worker
NO
YES
YES NO YES
NO Age < 18?Age >= 22?
YES
NO
Youth Services?
YES
YESNO
Entered Employment Rate
– Wage records the primary data source– Supplemental data may be used for persons not
found in wage records– Exclude adults, older youth employed at registration– Exclude older youth in advanced training/
postsecondary education (but not employed) in quarter after exit.
Number Employed in Quarter after Exitdivided by
Number of Exiters
YESYES
YES
NO
Entered Employment RateAdults and Dislocated Workers
Employed at Registration?Item 115 = 1
Excluded
Employed in Quarter After Exit ?Item 601 = 1 Numerator
DenominatorNO
Dislocated Worker Measure?
YES
NO
YESYES
NO
YES
Entered Employment Rate Older Youth
Excluded
Numerator
Denominator
Excluded
YES
In Advanced Training orPostsecondary Education
in Quarter After Exit Item 623 = 1
NO
Employed in Quarter After Exit Item 601 = 1
NO
Employed at Registration?Item 115 = 1
Employment Retention
– Based on those employed in quarter after exit– Wage records the primary data source– Can use supplemental data if not in wage records.– Excludes older youth in advanced training/
postsecondary education (but not employed) in 3rd quarter after exit.
Of those employed in the quarter after exit:
Number Employed in 3rd Quarter after Exit
divided byNumber Employed in Quarter after Exit
NO
YES
YES
Employment Retention RateAdults and Dislocated Workers
Excluded
Numerator
DenominatorNO
Employed in 3rd Quarter After Exit
Item 608 = 1
YES
Employed in Quarter After Exit Item 601 = 1
YESYES
NO
YES
NO
YES
Employment Retention RateOlder Youth
Employed Quarter After Exit?Item 601 = 1 Excluded
Employed 3rd Quarter After ExitItem 608 =1
Numerator
Denominator
Excluded
NO
In Advanced Training orPostsecondary Education in 3rd Quarter After Exit
Item 624 = 1
Earnings ChangeAdults and Older Youth
– Wage records only • Exclude if supplemental data used to determine employment:
– in either the 1st or 3rd quarter after exit.
• Exclude if preprogram quarters too far back to get data
– Exclude older youth in advanced training/ postsec. education (but not employed) in 3rd quarter after exit.
divided byNumber employed in quarter after exit
Of those employed in the quarter after exit:
minusEarnings in 2nd and 3rd Quarters
after Exit
Earnings in 2nd and 3rd Quarters before
Registration
Earnings ReplacementDislocated Workers
– Wage records only • Exclude if supplemental data used to determine employment:
– in either the 1st or 3rd quarter after exit.
• Exclude if preprogram quarters too far back to get data
• Use 2nd and 3rd quarters before registration if no dislocation date or dislocation date after registration date
– Add earnings over all included exiters before dividing
Of those employed in the quarter after exit:
Earnings in 2nd and 3rd Quarters after Exitdivided by
Earnings in 2nd and 3rd Quarters before Dislocation
NO
NO
Earnings Change--AdultsEarnings Replacement--DWs
Employed in Quarter After Exit Item 601 = 1 Excluded
Numerator and
Denominator
YES
Supplemental Data Used Q1 Item 602 = 1 or 2
Supplemental Data Used Q3 Item 609 = 1 or 2
YES
YE
S
NO
YES
In Advanced Training orPostsecondary Education in 3rd Quarter After Exit
Item 624 = 1
YES
Yes
NO
NO
Earnings Change Older YouthEmployed Quarter After Exit?
Item 601 = 1Excluded
Supplemental Data Used Q1Item 602 =1 or 2
Included
NO
Employed 3rd Quarter After ExitItem 608 =1
Supplemental Data Used Q3
Item 609 =1 or 2
YES
NO
YE
S
NO YE
S
YES
Credential Rate• Flexible definition of credential
– Standard credentials– Certificates that recognize the successful completion of training
services ….
• Credential attainment by end of 3rd qtr after exit• Positive outcome is attain credential and:
– Enter employment (adults, dislocated workers)– Enter employment or in advanced training/postsec. educ. in quarter
after exit (older youth)
• Includes:– Adults/DWs who received training.– All older youth
Credential--Adults and DWsOf those who receive training:
Credential--Older YouthNumber who enter employment and or
are in advanced training or postsecondary education in the quarter after exit
divided byNumber of exiters
Number who enter employment and attain credential divided by
Number of exiters
YES
NO
YES
Employment and Credential RateAdults and Dislocated Workers
Excluded
Numerator
Denominator
NO
NO
YES
Attained a credentialItem 621 = 1, 2 ,3, 4, 5, or 6
Employed in Quarter After ExitItem 601 =1
YES
Received Training?Item 333 is a valid date
YES
YES
Credential Rate--Older Youth
Employed in Quarter After ExitItem 601 =1
Denominator
NO
YES
YES
Numerator
NO
NO
In Advanced Training or Postsecondary Education in
Quarter After Exit Item 623 = 1
Attained a credential
Item 621 = 1, 2 ,3, 4, 5, or 6
Younger Youth Goal Attainment• Outcome attained during participation• 3 goal areas:
– Basic skills• A basic skills goal must be set if the youth is deficient in basic skills
– Work readiness skills– Occupational skills
• Flexibility in defining goals• Youth has a year to attain the goal• Maximum of 3 goals set per year (for performance)
– First year begins on registration date and ends a year later.– Goals used for performance must be identified up front
Younger Youth Goal Attainment-2
• Add numerator and denominator over all youth before dividing.
• Include all in-school youth and out-of-school youth assessed as needing basic skills, work readiness skills, or occupational skills
Number of goals attaineddivided by
Number of goals attained or reached anniversary/exit date without attaining
HS Diploma/GED Attainment
• Diploma can be attained during participation or by end of quarter after exit
• Exclude youth still in secondary school at exit
Of youth who were not HS grads at registration:
Number of exiters who attained diplomadivided by
Number of exiters not still in school at exit
YESYES
NO
YES
NO
NO
Younger Youth Diploma Rate
HS grad at registration?Item 123 > 11; Item 130 = 2 or 4 Excluded
Attained Diploma?Item 673=1 or 2
Numerator
Denominator
Excluded
YES
In Secondary School at Exit? Item 673 = 3
Younger Youth Retention• In one or more qualifying outcomes in 3rd quarter
after exit– Postsecondary education (Item 676 = 1)– Advanced training (Item 676 = 2)– Military service (Item 676 = 3)– Qualified apprenticeship (Item 675 = 4)– Unsubsidized employment (Item 676 = 5)
• Exclude youth in secondary school at exit• Not related to outcome in quarter after exit
YES
YES
YES
NO
NO
Younger Youth Retention Rate
In Secondary School at Exit? Item 673 = 3
Numerator
Denominator
Excluded
Retained in 3rd Quarter After Exit
Item 676=1,2,3,4, or 5
ChallengeUnderstanding When Outcomes
Occur and When they Are Measured
Q1 Q2 Q4Q3Q2Q1Q4Q3
PY 2000 PY 2001
Entered Employ.
Exit
EE
Retention RTN
Credential
12 Mo
Youth Diploma
12 Mo. Retention
When outcome occurs
=When Measured
When Outcomes Occur
ChallengeUnderstanding Performance
ReportsWhat you get when you ask for “the most recent data available”
1 4321 432
Previous PY Current PY
Real Time Measures
Wage Record Measures
EE
Exiters
RTN
Earnings
Yth Diploma
Look at Data Available May 15
Yth Goals
Credential
Cust Sat
Challenge:Implementing New Data
Collection Processes
Outcome Data Collection Activities• Wage records
– Collect quarterly (both in and out of state)
• Supplemental employment data– Collect during follow-up services– Collect quarterly after not found in wage records if necessary
• Credential– Collect during participation– Update after the 3rd quarter after exit
• Diploma – Collect during participation– Update after the quarter after exit
Challenge:What about those JTPA Exiters Included in WIA Performance
Measures?
JTPA exiters from October 1, 1999 to June 30, 2000 are included in :– All adult, dislocated worker, and older youth
measures– Younger youth retention measure.
• Are you getting wage records for them?• Are you collecting supplemental employment data?
(Do you need it to meet your goal?)• Are you collecting credential data?• Are you collecting younger youth retention outcomes?
It’s not too late now, but ….. .