1

Developing Earmark Grant Developing Earmark Grant Performance Measures:Performance Measures:

Grant Proposal Section 3Grant Proposal Section 3

Deanna KhemaniDeanna Khemani

May 2005 2

Session ObjectivesSession Objectives Participants will be able to:Participants will be able to:

– Understand the context within which your Understand the context within which your grant operates and is funded grant operates and is funded

– Identify performance measures that relate Identify performance measures that relate to your grant’s purpose & discuss to your grant’s purpose & discuss measurement parametersmeasurement parameters

– Understand the importance of having a Understand the importance of having a data collection & information systemdata collection & information system

May 2005 3

Session OverviewSession Overview

Setting the ContextSetting the Context

Grant Proposal Requirements & Grant Proposal Requirements & Key TerminologyKey Terminology

Identifying Performance Measures Identifying Performance Measures & Other Things to Consider& Other Things to Consider

May 2005 4

The ContextThe Context

Your grant is funded within an existing system, Your grant is funded within an existing system, is affected by that system, and draws is affected by that system, and draws requirements from that systemrequirements from that system

– Government Performance and Results Act (GPRA)Government Performance and Results Act (GPRA)

– The Workforce Investment Act of 1998 (WIA)The Workforce Investment Act of 1998 (WIA)

– The President’s Management AgendaThe President’s Management Agenda

– The Labor, Health and Human Services and Education The Labor, Health and Human Services and Education Appropriations ActAppropriations Act

May 2005 5

The ContextThe Context GPRA or “Results Act”GPRA or “Results Act”

– Federal agencies should stop preoccupation with Federal agencies should stop preoccupation with measuring processes and activities and should focus on measuring processes and activities and should focus on bottom line resultsbottom line results

– ETA will use GRPA performance targets to evaluate the ETA will use GRPA performance targets to evaluate the reasonableness of earmark grantee’s expected levels of reasonableness of earmark grantee’s expected levels of performanceperformance

WIAWIA

– Seven key themes, one of the primary principles is the Seven key themes, one of the primary principles is the need for increased accountability at the federal, state and need for increased accountability at the federal, state and local-levels, including grantee-levellocal-levels, including grantee-level

May 2005 6

The ContextThe Context President’s Management AgendaPresident’s Management Agenda

– Federal agencies need to evaluate their programs and Federal agencies need to evaluate their programs and grants, and to show achievement of desired results & ties grants, and to show achievement of desired results & ties budget decisions to performance (consistent with GPRA)budget decisions to performance (consistent with GPRA)

– ““What matters in the end is completion. Performance. What matters in the end is completion. Performance. Results. Not just making promises, but making good on Results. Not just making promises, but making good on promises.”promises.”

Appropriations ActAppropriations Act

– Contains specific language requiring DOL to assess Contains specific language requiring DOL to assess earmark grant performanceearmark grant performance

May 2005 7

Who Cares?Who Cares? Congress and the American people require results Congress and the American people require results

and a return on the federal investmentand a return on the federal investment– PEC is required to develop a Notice of Grant Award statement PEC is required to develop a Notice of Grant Award statement

that outlines the project’s intended outcomes/resultsthat outlines the project’s intended outcomes/results

Earmark grantees must develop a performance Earmark grantees must develop a performance management system that:management system that:– Contains performance measures that relate to the stated goals Contains performance measures that relate to the stated goals

of the projectof the project

– Collects, analyzes and reports information so that the project Collects, analyzes and reports information so that the project can continuously improve and show resultscan continuously improve and show results

May 2005 8

Performance ManagementPerformance Management Performance management is a systematic process that Performance management is a systematic process that

involves:involves:

– Monitoring activities and subcontracts for results;Monitoring activities and subcontracts for results;

– Collecting, analyzing and reporting program and fiscal Collecting, analyzing and reporting program and fiscal information;information;

– Using data to influence program decision-making and resource Using data to influence program decision-making and resource allocation; and allocation; and

– Communicating results to advance organizational learning and Communicating results to advance organizational learning and inform stakeholdersinform stakeholders

Performance management involves the entire organizationPerformance management involves the entire organization

May 2005 9

Grant Proposal ElementsGrant Proposal Elements

Your earmark grant proposal guide Your earmark grant proposal guide references the following:references the following:

– GoalsGoals

– Performance MeasuresPerformance Measures

– Expected Levels of PerformanceExpected Levels of Performance

– Actual Levels of PerformanceActual Levels of Performance

May 2005 10

Grant Proposal ElementsGrant Proposal Elements

PERFORMANCE MEASURE EXPECTED

LEVEL OF

PERFORMANCE

ACTUAL LEVEL OF

PERFORMANCE

The Proposal Template requests that grantees submit performance measurement information using the table above

Grantees should provide information in the first two columns:

(1) Performance Measure and (2) Expected Level of Performance

May 2005 11

Grant Proposal ElementsGrant Proposal Elements GoalGoal

– A broad, comprehensive statement that clearly A broad, comprehensive statement that clearly defines the purpose of the grant.defines the purpose of the grant.

– In the grant proposal, you need to provide a In the grant proposal, you need to provide a narrative statement identifying your project goalsnarrative statement identifying your project goals

» Example 1:Example 1: To provide high-tech manufacturing training to 50 To provide high-tech manufacturing training to 50 individuals.individuals.

» Example 2:Example 2: To analyze whether current office equipment on the To analyze whether current office equipment on the market is accessible to persons with disabilities and to disseminate market is accessible to persons with disabilities and to disseminate research findings.research findings.

May 2005 12

Grant Proposal ElementsGrant Proposal Elements Performance MeasurePerformance Measure

– The name given to what the grantee will measure. The name given to what the grantee will measure. What is it that you are measuring?What is it that you are measuring?

» Example 1:Example 1: Training Completion Rate is the “measure” Training Completion Rate is the “measure” of individual achievement in the training curriculum.of individual achievement in the training curriculum.

» Example 2:Example 2: Developing a report outlining accessibility Developing a report outlining accessibility issues is an obvious performance measure, but also issues is an obvious performance measure, but also Presentation Attendance could be a “measure” of Presentation Attendance could be a “measure” of whether the message is reaching the “right” peoplewhether the message is reaching the “right” people

May 2005 13

Grant Proposal ElementsGrant Proposal Elements Performance measure specifics, also Performance measure specifics, also

referred to as measurement parametersreferred to as measurement parameters– How the performance measure “works”How the performance measure “works”

» Who is included in the measure?Who is included in the measure?

» Who is excluded in the measure?Who is excluded in the measure?

» What is the time period for measurement?What is the time period for measurement?

» What data sources will be used in the measurement? What data sources will be used in the measurement? Implies that you have a system in place to capture the Implies that you have a system in place to capture the information and report it.information and report it.

» What level of performance is sufficient to count in the What level of performance is sufficient to count in the measure?measure?

May 2005 14

Example Grant Proposal Table 1Example Grant Proposal Table 1

PERFORMANCE MEASURE EXPECTED LEVEL OF

PERFORMANCE

ACTUAL LEVEL OF

PERFORMANCE Training Completion Rate is the number of individual who complete training by the 6th quarter of the project divided by the total number of participants that entered training.

Measurement Parameters:

- Only those participants enrolled in training through the project are included in the denominator.

- Participants must complete training by the 6th quarter to be counted in the numerator.

- Training completion means the individual received a certificate from the training vendor showing that they completed training satisfactorily. Satisfactory training completion means the individual received at least a C average.

- The primary data source is training vendor records.

May 2005 15

Example Grant Proposal Table 2Example Grant Proposal Table 2

PERFORMANCE MEASURE EXPECTED LEVEL OF

PERFORMANCE

ACTUAL LEVEL OF

PERFORMANCE Presentation Attendance Rate refers to the number of employer associations (previously identified by the project) that attend a report findings presentation prior to the last day of the project divided by the total number of employer associations on the distribution list.

Measurement Parameters:

- Only those employer associations previously identified by project staff are included in the denominator.

- Only those employer associations that attend a presentation prior to the last day of the project will count in the numerator.

- Employer association representatives must complete the 3-hour presentation training in order to be included in the numerator.

- The primary data source will be attendance records collected at the training sessions.

May 2005 16

Grant Proposal ElementsGrant Proposal Elements

Expected Level of PerformanceExpected Level of Performance

– The numeric performance target that the The numeric performance target that the grantee expects to achieve on each grantee expects to achieve on each performance measure; sometimes referred to performance measure; sometimes referred to as a “performance standard.” as a “performance standard.”

– ETA will use the GPRA levels to assess the ETA will use the GPRA levels to assess the reasonableness of your performance targetsreasonableness of your performance targets

May 2005 17

Performance-Related TerminologyPerformance-Related Terminology

Actual Performance LevelActual Performance Level

– The numeric performance level that the The numeric performance level that the grantee achieved on each performance grantee achieved on each performance measure. measure.

– How you performed on the performance How you performed on the performance measure relative to the Expected Level of measure relative to the Expected Level of Performance.Performance.

May 2005 18

Example Grant Proposal Table 1Example Grant Proposal Table 1PERFORMANCE MEASURE

EXPECTED LEVEL OF

PERFORMANCE

ACTUAL LEVEL OF

PERFORMANCE Training Completion Rate is the number of individual who complete training by the 6th quarter of the project divided by the total number of participants that entered training.

Measurement Parameters:

- Only those participants enrolled in training through the project are included in the denominator.

- Participants must complete training by the 6th quarter to be counted in the numerator.

- Training completion means the individual received a certificate from the training vendor showing that they completed training satisfactorily. Satisfactory training completion means the individual received at least a C average.

- The primary data source is training vendor records.

88% of participants (i.e., 44 participants) will complete training by the end of the grant

period.

78% or 28 participants

completed training by the end of the

grant period.

May 2005 19

Example Grant Proposal Table 2Example Grant Proposal Table 2

PERFORMANCE MEASURE EXPECTED LEVEL OF

PERFORMANCE

ACTUAL LEVEL OF

PERFORMANCE Presentation Attendance Rate refers to the number of employer associations (previously identified by the project) that attend a report findings presentation prior to the last day of the project divided by the total number of employer associations on the distribution list.

Measurement Parameters:

- Only those employer associations previously identified by project staff are included in the denominator.

- Only those employer associations that attend a presentation prior to the last day of the project will count in the numerator.

- Employer association representatives must complete the 3-hour presentation training in order to be included in the numerator.

- The primary data source will be attendance records collected at the training sessions.

78% of employer associations (i.e., 28 associations) will attend the 3-hour

presentation training by the end of the grant

period.

75% or 21 employer associations attended the presentation

training.

May 2005 20

Grant Proposal ElementsGrant Proposal Elements Grantees are also required to provide information on Grantees are also required to provide information on

who is responsible for submitting reportswho is responsible for submitting reports ( (if different from if different from

grantee organizationgrantee organization)) Grantees must describe how they will maintain, track Grantees must describe how they will maintain, track

and report program data?and report program data?

– What types of information will be collectedWhat types of information will be collected

– How the information is secured?How the information is secured?

– How often reports will be generated?How often reports will be generated?

– How information collected will be used to manage the How information collected will be used to manage the project?project?

May 2005 21

Developing Performance MeasuresDeveloping Performance Measures Select appropriate performance measures (Select appropriate performance measures (see Handout Isee Handout I))

– Many different kinds of measures (process, output, outcome)Many different kinds of measures (process, output, outcome) Assess where information on the measure will originateAssess where information on the measure will originate ( (see see

Handout IIIHandout III))– What is evidence of success? Do you have access to the information? What is evidence of success? Do you have access to the information?

How will data be collected? How will data be collected? Track informationTrack information

– How will you know how you are doing (e.g., data collection system)? How will you know how you are doing (e.g., data collection system)? What information is needed? How will it be collected and tracked? What information is needed? How will it be collected and tracked?

Develop reportsDevelop reports– How will you process and aggregate the data? How will you process and aggregate the data?

May 2005 22

Where Do I Start?Where Do I Start?

Before deciding on indicators and Before deciding on indicators and performance measures, always start performance measures, always start with your grant.with your grant.– What’s the purpose?What’s the purpose?

– Why were you funded?Why were you funded?

– What are you expected to accomplish?What are you expected to accomplish?

May 2005 23

Using a Logic ModelUsing a Logic Model

INPUT PROCESS OUTPUT OUTCOME IMPACT

The Federal Government focuses here

But many things happen before “the outcome” that can be measured. Interim or intermediate measurements might even predict the ultimate

outcome!

May 2005 24

Understanding Intermediate MeasuresUnderstanding Intermediate Measures

Input Measures: Input Measures: “What You Invest”“What You Invest”– Measures related to outreach and recruitmentMeasures related to outreach and recruitment– Enrollment ratesEnrollment rates– Measures related to percentage of accepted referrals Measures related to percentage of accepted referrals

from other partnersfrom other partners

Process Measures: Process Measures: “What You Do”“What You Do”– Attendance rates (particularly for youth)Attendance rates (particularly for youth)– Extent of partnering/referrals or co-enrollmentsExtent of partnering/referrals or co-enrollments– Timeliness of reportsTimeliness of reports

May 2005 25

Understanding Intermediate MeasuresUnderstanding Intermediate Measures Output Measures: Output Measures: “What you accomplish as a “What you accomplish as a

result of the product or activity”result of the product or activity”– Completion ratesCompletion rates– % successful completions% successful completions– # of exits with positive outcomes by ‘x’ time period# of exits with positive outcomes by ‘x’ time period

Outcome Measures: Outcome Measures: “What difference does the “What difference does the project make”project make”– % of Participants Placed in Employment, Customer % of Participants Placed in Employment, Customer

Satisfaction, and Diploma Attainment RateSatisfaction, and Diploma Attainment Rate– Change in policymakingChange in policymaking

May 2005 26

Understanding Intermediate MeasuresUnderstanding Intermediate Measures

Impact Measures:Impact Measures: “What is the effect of the “What is the effect of the project”project”

– Societal benefitsSocietal benefits

– Reduced welfare dependency (i.e., self-sufficiency)Reduced welfare dependency (i.e., self-sufficiency)

– Fewer incarcerationsFewer incarcerations

You need to have a mixture of process and output You need to have a mixture of process and output measures so that you can measures so that you can manage performancemanage performance in in order to achieve order to achieve desired outcomesdesired outcomes

May 2005 27

Appropriate Earmark MeasuresAppropriate Earmark Measures Demonstration/Pilot Projects; Multiservice Demonstration/Pilot Projects; Multiservice

Projects; or Multistate Projects Projects; or Multistate Projects – Curriculum Development Rate (output measure)Curriculum Development Rate (output measure)– Training Completion Rate (output measure)Training Completion Rate (output measure)– Skill Attainment RateSkill Attainment Rate– Placement in Employment RatePlacement in Employment Rate– Employer and/or Participant-Customer Satisfaction Employer and/or Participant-Customer Satisfaction

RatesRates– Earnings Change or Percent Increase in WagesEarnings Change or Percent Increase in Wages– Retention RatesRetention Rates– Cost Per ParticipantCost Per Participant

May 2005 28

Appropriate Earmark MeasuresAppropriate Earmark Measures

Research ProjectsResearch Projects– Report completedReport completed

– Community Awareness or Penetration Community Awareness or Penetration RateRate

– Employer and/or Customer-Satisfaction Employer and/or Customer-Satisfaction RatesRates

May 2005 29

ConclusionConclusion Just remember that your Just remember that your performance measuresperformance measures

need to be need to be tiedtied directly to your project’s directly to your project’s goalsgoals and and that the measures should have that the measures should have quantified quantified performance targets performance targets

Each performance measures should answer one of Each performance measures should answer one of the following questions:the following questions:

– How well did you succeed in providing customer value?How well did you succeed in providing customer value?

– How well did you do the services/activities, which How well did you do the services/activities, which support the creation of customer value?support the creation of customer value?

May 2005 30

Example 1Example 1 A project provides high-tech manufacturing A project provides high-tech manufacturing

training to 50 individuals in order to help them training to 50 individuals in order to help them gain high-paying employment.gain high-paying employment.

Based on this goal statement, what are possible Based on this goal statement, what are possible performance measuresperformance measures??

How could the How could the performance measure be definedperformance measure be defined?? What would be appropriate What would be appropriate performance parametersperformance parameters?? What What data sourcesdata sources could be used to assess performance? could be used to assess performance?

May 2005 31

Example 2Example 2

This project will develop a distance learning This project will develop a distance learning program that utilizes computer-based training program that utilizes computer-based training in order to access students in rural areas.in order to access students in rural areas.

Based on this goal statement, what are possible Based on this goal statement, what are possible performance measuresperformance measures??

How could the How could the performance measure be definedperformance measure be defined?? What would be appropriate What would be appropriate performance parametersperformance parameters?? What What data sourcesdata sources could be used to assess performance? could be used to assess performance?