wia performance measures and standards: the wiasrd, common measures and standards negotiation...
TRANSCRIPT
WIA PERFORMANCE MEASURES AND STANDARDS:The WIASRD, Common Measures and
Standards Negotiation Challenges
Christopher T. KingRay Marshall Center for the Study of Human Resources
University of Texas, [email protected]
512/471-2186
David W. StevensThe Jacob France Institute
University of [email protected]
410/837-4729
April 22, 2003
BRIEFING TOPICS
1. Highlights from PY 2000 program outcome information in theWIASRD files from the seven ADARE Project states,focusing on the quality of the data elements.
2. Negotiated, actual and actual minus negotiated difference in PY 2000 performance data for the seven ADARE Project states.
3. Observations about the proposed common measures.
4. WIA performance standards negotiation challenges andopportunities (including pros and cons of regression modeling).
5. Other challenges that will follow reauthorization.
EMPLOYED IN QUARTER AFTER EXIT QUARTER
The data element code choices are: yes, no and not yet available
Georgia, Illinois and Missouri did not use the not yet available code.
The four ADARE Project states that used the not yet available codeused it the following percent of the time:
Florida 44 percent
Maryland 73 percent
Texas 23 percent
Washington 50 percent
USE AND SOURCE OF SUPPLEMENTAL DATA
The data element code choices are: used case management files and record sharing/matching
Florida, Missouri and Washington did not report any use ofsupplemental data sources.
Georgia reported only three instances of supplemental data use.
Texas reported using supplemental data one percent of the time.
Illinois and Maryland reported using supplemental data threepercent of the time.
OCCUPATIONAL CODEof any job held since exit
This information is to be reported if the individual is reported as employedin the quarter after exit.
The information can be based on information derived from case managementfiles, follow-up services or other sources.
It is not necessary to wait until information on employed in quarter after exitis available.
Florida, Georgia and Maryland used only the nine-digit DOT code.
Illinois and Texas used only the five-digit OES code.
Washington used both the DOT and OES coding taxonomies.
Missouri used the five-digit or six-digit O*Net98 code.
ENTERED TRAINING RELATED EMPLOYMENT
Two-thirds of the yes or no entries for this data element were recorded as a yes.
The range of affirmative entries was from a low of 29 percent forMaryland to a high of 94 percent for Florida.
The reported method used by Florida, Maryland, Texas and Washingtonto determine training related employment was ‘other appropriate method’.
The reported method used most often by Georgia, Illinois and Missouriwas ‘a comparison of the occupational codes of the training activityand the job’, but each of these three states also used ‘a comparison ofthe industry of employment with the occupation of training usingan appropriate crosswalk’.
ENTERED NONTRADITIONAL EMPLOYMENT
The nontraditional employment designation can be based on either local or national data.
Six percent of the yes or no entries for this data element were reportedas a yes.
The range of affirmative entries among the seven ADARE Project stateswas from a low of one percent to a high of fifteen percent.
Texas did not report yes or no entries for this data element.
TYPE OF RECOGNIZED EDUCATIONAL/OCCUPATIONALCERTIFICATE, CREDENTIAL, DIPLOMA OR DEGREE
ATTAINED
Seven codes are provided. States and localities have flexibility in choosing the methods used to collect data documenting this data element.
Each of the seven ADARE Project states reported award of some credentials in each of the six type of credential categories.
PY 2000 CORE MEASURES OF PERFORMANCESEVEN ADARE PROJECT STATES
The four Adult and Dislocated Worker performance measures are covered.
Entered employment rate.
Employment and credential rate.
Retention rate.
Earnings change
Each of the four charts that follow ‘flies in’ PY 2000 negotiated, actualand actual minus negotiated performance measure values for theseven ADARE Project states.
QUESTIONS TO ASK WHEN LOOKING AT THECHARTS THAT FOLLOW
Do I know enough about the criteria for specifying each negotiated performance measure value to interpret the observed differencesin these negotiated values among the seven ADARE Project states?
Do I know enough about the data sources that were used to calculatethe actual performance measure values to interpret the actual minusnegotiated differences in these values among the seven ADARE Projectstates?
What management and/or policy conclusions can I reach based onmy answers to the previous two questions?
Can I be confident in making incentive awards and imposing sanctionsbased on actual minus negotiated value differences?
Program Year 2000 (July 2000-June 2001): Entered Employment Rate
-10%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
FL GA IL MD MO TX WA
NEGOTIATED ACTUAL DIFFERENCE
Program Year 2000 (July 2000-June 2001): Employment And Credential Rate
-60%-50%-40%-30%-20%-10%0%10%20%30%40%50%60%70%80%90%100%
FL GA IL MD MO TX WA
NEGOTIATED ACTUAL DIFFERENCE
Program Year 2000 (July 2000-June 2001): Retention Rate
-10%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
FL GA IL MD MO TX WA
NEGOTIATED ACTUAL DIFFERENCE
Program Year 2000 (July 2000-June 2001): Earnings Change
-$500
$0
$500
$1,000
$1,500
$2,000
$2,500
$3,000
$3,500
$4,000
$4,500
$5,000
FL GA IL MD MO TX WA
NEGOTIATED ACTUAL DIFFERENCE
REVISITING THE QUESTIONS ASKEDHAVING LOOKED AT THE
CHARTS
Do I know enough about the criteria for specifying each negotiated performance measure value to interpret the observed differencesin these negotiated values among the seven ADARE Project states?
Do I know enough about the data sources that were used to calculatethe actual performance measure values to interpret the actual minusnegotiated differences in these values among the seven ADARE Projectstates?
What management and/or policy conclusions can I reach based onmy answers to the previous two questions?
Can I be confident in making incentive awards and imposing sanctionsbased on actual minus negotiated value differences?
COMMON MEASURE ISSUESPerformance Measure Quality
ENTERED EMPLOYMENT RATE
Registration date Employed or not employed at registration Exit date Entered employment by the end of the first quarter after exit
ISSUES
Staff decision whether and when to register a customer Quality of ‘employed or not employed at registration’ data element Unintended consequences of this measure Staff decision when to enter or allow automatic entry of exit date Use of supplemental data sources
COMMON MEASURE ISSUESPerformance Measure Quality
EMPLOYMENT RETENTION RATE
Employed first quarter after exit (regardless of employment statusat time of registration)
Employed second quarter after exit Employed third quarter after exit
ISSUES
Stakeholder interest in this measure Drill-down questions that will be asked Use of supplemental data sources Timeliness of availability for intended uses
COMMON MEASURE ISSUESPerformance Measure Quality
EARNINGS INCREASE
Earnings in second quarter prior to registration Employed in first quarter after exit Earnings in first quarter after exit Earnings in third quarter after exit
ISSUES
Stakeholder interest in this measure Drill-down questions that will be asked Number of ‘pays’ in each reference quarter Use of supplemental data sources Timeliness of availability for intended uses
COMMON MEASURE ISSUESPerformance Measure Quality
EFFICIENCY
The dollar amount specification to serve as the numerator The number of participants figure to serve as the denominator
ISSUES
Stakeholder interest in this measure Drill-down questions that will be asked Quality of data elements Unintended consequences
COMMON MEASURE ISSUESPerformance Measure Quality
PLACEMENT IN EMPLOYMENT OR EDUCATION
Registration date Enrolled in secondary education at registration Exit date Not enrolled in post-secondary education at registration Not employed at registration Enrolled in secondary education at exit Employed in first quarter after exit In military service in first quarter after exit Enrolled in post-secondary education in first quarter after exit Enrolled in advanced training/occupational skills training in
first quarter after exit
CONTINUED……
COMMON MEASURE ISSUESPerformance Measure Quality
PLACEMENT IN EMPLOYMENT OR EDUCATION
CONTINUED….
ISSUES
Stakeholder interest in this measure Drill-down questions that will be asked Quality/uniformity of data definitions and sources Cost of data collection Access to education records Timeliness of data availability for intended uses Unintended consequence—proliferation of
credentials
COMMON MEASURE ISSUESPerformance Measure Quality
ATTAINMENT OF A DEGREE OR CERTIFICATE
Registration date Enrolled in education Exit date Attained a diploma, GED, or certificate by the end of the third
quarter after exit
ISSUES
Stakeholder interest in this measure Drill-down questions that will be asked Access to education records Quality/uniformity of data definitions and sources Timeliness of data availability Unintended consequences
COMMON MEASURE ISSUESPerformance Measure Quality
LITERACY OR NUMERACY GAINS
?
COMMON MEASURE ISSUESPerformance Measure Quality
FIVE ISSUES ARE OF PARTICULAR IMPORTANCE AND CONCERN:
The accuracy and probable unintended consequences associated with the employed or not employed at registration data element
The integrity and value-added of supplemental data use
Selection of denominator and numerator definitions for the proposed efficiency measure
The complexity and value-added of the placement in employmentor education measure
Expected unintended consequences associated with the attainment of a degree of certificate measure
PERFORMANCE STANDARD ISSUES
THREE TOPICS ARE OF PARTICULAR IMPORTANCE:
State and Local Workforce Area Benchmarking
The Census Bureau LEHD Program as a potential source of localdemographic and economic activity information for discretionaryuse in negotiation of state and local performance standards
Benchmarking of own performance over time
Benchmarking against other ‘similar’ states or Local Workforce Areas
Return to regression modeling? Pros and cons
CONTINUED….
PERFORMANCE STANDARD ISSUES
Challenges Associated with Pursuing Continuous Improvement
Integrity of state and local management information systems over time
Continuity of data source availability and content over time
Expected unintended consequences
PERFORMANCE STANDARD ISSUES
Vulnerability to Unintended State and Local Actions
Discretionary opportunities to define selection in criteria, assignment to service components criteria (including whetherand when to use partner services) and timing of exit criteria
Investment in staff development can reduce the frequency of some of the unwanted behaviors that will otherwisefollow introduction of the common measures
OTHER CHALLENGES
Occupations in demand
Required registration of some customers
Stakeholder interest in number of customers served