espms presentation aug 28
TRANSCRIPT
-
7/30/2019 Espms Presentation Aug 28
1/25
1
Employment Service
Performance Management System
Presentation to
Employment Ontar io Service Delivery Advisory Group
August 28 2008
-
7/30/2019 Espms Presentation Aug 28
2/25
2
Purpose
Status Report on the Development of the Performance
Management Framework/System for Employment
Services
-
7/30/2019 Espms Presentation Aug 28
3/25
3
The Performance Management Framework and System for
Employment Services builds on Employment Ontario principles
Principles
-
7/30/2019 Espms Presentation Aug 28
4/25
4
Best Practices - International
The Performance Management Framework and System
incorporate characteristics of effective program management
systems practices
Results measured against three dimensions of service delivery success Effectiveness, Efficiency and Customer service
Dimensions of service delivery defined by core measures and weightedaccording to their value to overall service quality
Measures that quantify input, through/output and impact or results
Service Quality Standards that establish the minimum level of acceptableperformance
-
7/30/2019 Espms Presentation Aug 28
5/25
5
Best Practices - OPS Directives on
Performance ManagementThe Performance Management
Framework and System align
with OPS Directives
Key Elements:
Accountability
Transparency
Customer-centric
Integrated Service Delivery
business planprocess
servicedelivery
evaluationand planning
data collectionand reporting
service leveldetermination
PerformanceManagementSystem
-
7/30/2019 Espms Presentation Aug 28
6/25
6
Stakeholder Engagement
Delivery Stakeholder Reference Group participated in
developing The Performance Management Framework and
System
External Delivery Stakeholder Reference Group for EmploymentServices, a sub-committee of SDAG
The PMF/S has been developed in collaboration with other keyelements of the implementation and delivery such as contract qualityassurance, resource planning, information systems development,
business supports, etc.
-
7/30/2019 Espms Presentation Aug 28
7/25
7
Components of the Performance Management
Framework/SystemService Quality Dimensions, Measures, Values
Service Funding Decision Matrix
Funding Model
Performance Incentive (at maturity)
Performance Management Standards
Stage I Interim
Stage II Maturity
Implementation Contract Management
Awards and Recognition
-
7/30/2019 Espms Presentation Aug 28
8/25
8
Service Quality Dimensions, Measures, Values
-
7/30/2019 Espms Presentation Aug 28
9/25
9
Compliance
with MTCU
Contract
OverallService
QualityStandardAchieved
Evidence ofOrganizatio
nalCapacity
Reached
Improvement
Targets
MTCU Response
Success Funding approval, Performance Incent ive and/or Award
Potent ial
Organization is in compliance with contract and program guidelines and hasachieved or surpassed Overall Service Quality Standard for service delivery
OC review demonstrates ability for sustained and/or improved results Organization has attained improvement targets identified in last contract
Official Review
Funding Ap proval with Immediate Compliance Direct ive Organization is NOT in compliance with contract despite achieving or
surpassing Overall Service Quality Standard for service delivery and mustsubmit an action plan for achieving compliance
OC review demonstrates little ability for sustained or improved results Organization did not meet improvement targets from last contract.
Directed Improvement - Funding Ap proval with Compliance and In Year
Improvements to Overal l Serv ice Quali ty Standard
Organization is NOT in compliance with contract/guidelines and must submit anaction plan for achieving compliance
Organization has not achieved Overall Service Quality Standard for service
delivery nor improvement targets from last contract OC review demonstrates there is capacity and potential for immediateimprovement
Terminat ion Notice of contract terminat ion, procedures to supp ort c l ient
serv ice transfer
Organization did not achieve in year service improvements to standard and/or is not in compliance with contract/guidelines Organization does not demonstrate necessary OC to support improvement
Service Funding Decision Matrix
-
7/30/2019 Espms Presentation Aug 28
10/25
10
Funding ModelEmployment Services
Operating funds provides funding to:
Deliver contracted services meet provincial service
quality standards
Funding Unit Ranges
Activity Range
Minimum Unit
Funding
Maximum
Unit
Funding
Operating Budget
(Range)
Employment & Training
Placement Incentive
& Supports (Range)
up to 399 950 2800 95,000-500,000 20,000-80,000
400-899 880 2000 352,000-1,000,000 80,000-180,000
900-1499 795 1100 715,500-1,500,000 180,000-300,000
1500-1999 750 950 1,125,000-1,895,000 300,000-400,000
750 2800 95,000-1,895,000 20,000-400,000
Employment & TrainingPlacement Incentives
Up to $8000 for on-the-jobtraining placement
Flow through funds
Employment & TrainingSupports
Provision of financialsupports (up to $500) to
remove barriers to job search(i.e. certification,transportation, work boots,temporary childcare)
Available for participantsbased on need and LICOlevel
Flow through funds
-
7/30/2019 Espms Presentation Aug 28
11/25
11
Funding Performance Incentive at
Maturity
Operating Budget
AllocationRecognizes:
o Results that achieveoverall service quality
standard,
Operating Budget Allocation
+ Performance Incentive
Level B (up to 3%)
Recognizes: Results that surpass overall
service quality standard within
the Level B range of results
(must exceed results from
84% of organizations SQS
results from previous year
eg. for 2008-09: 7.5)
Operating Budget Allocation +
Performance Incentive
Level A (up to 5%)
Recognizes: Results that surpass service
delivery quality standard within
the Level A range of results
(must exceed SQS results from
95% of organizations results
from previous year eg. for
2008-09: 8.0)
(2013/14)
-
7/30/2019 Espms Presentation Aug 28
12/25
12
Funding Performance Incentive at Maturity
A Performance Management System tracks outcomes and holds organizationsaccountable to achieving a provincial service quality standard. The process and
evaluation is transparent, accountable and responsive. It allows for a quantitative
recognition of organizations who surpass quality standards and sets the stage for
continuous improvement.
To encourage, recognize and reward stellar performance, it is proposed that aperformance incentive be integrated into the Employment Service Funding model.
The proposed performance incentive will:
be available to organizations based on full-year results. The organization mustachieve required results levels in each year to receive the performance incentivein subsequent years
be calculated against the organizations program operating budget allocation, butwill not be included in the organizations base operating allocation
recognize and reward two levels of results in overall service quality that exceedsprovincial standard
be recalculated every year based on changes from previous years results
-
7/30/2019 Espms Presentation Aug 28
13/25
13
Performance Management Standards
Stage I: (09/10 10/11) Interim Overall Service Quality Target8.25
(linked to sustained funding)Dimension Measure Weighting Interim Target Factor
CustomerService(40%)
Customer satisfaction(participant/employer)
40% 85% 3.40
Effectiveness(50%) Employed/Career Pathand/or Training/Education 50% 79% 3.95
Efficiency(10%)
Intake/Activity 10% 90%* 0.90
Interim targets are based on Job Connect 07-08 data with the exception of Customer Satisfaction (CS)which is consistent with OPS standards. Current JC data exceeds proposed CS standard, reflecting a
mature system
Starting in 09/10 service providers will collect and report data related to all of the core measures at themature state, but at the interim stage, will be held accountable to performance on the above 3.
* Cannot exceed 100% in contribution to Overall Service Quality
-
7/30/2019 Espms Presentation Aug 28
14/25
14
Performance Management Standards
Stage II: (11/12)
Proposed Overall Service Quality and Core Measure Standards
Dimension Core Measure Weighting Example of Core
Measure Standard*
Factor
CustomerService(40%)
Customer satisfaction(participant/employer)
15% 85% 1.28
Service Coordination 25% 30% 0.75Effectiveness
(50%)Employed/Career Path 25% 69% 1.73
Training/Education 10% 10% 0.10
Participant Suitability 15% 25% 0.38
Efficiency(10%)
Intake/Activity 5% 100% 0.50
Workshop Activities/Information Sessions 5% 100% 0.50
Proposed Overall Service Quality Standard (Link to sustained funding) 5.24
*Actual standards will be established based on full year data collected on EOIS in 10/11
-
7/30/2019 Espms Presentation Aug 28
15/25
15
ImplementationContract ManagementStage I: (09/10)
Set Interim Provincial Targets for initial 3 core measures using 07/08 full yearJob Connect data as proxy
Hold organizations accountable to proposed Interim Overall Service QualityTarget, based on commitments in 3 core measures (as specified on theirSchedule B)
Identify 09/10 data reporting requirements and approach to, at a minimum,capture and report aggregate data to support the 3 core measures for 2009 -2011
Stage II: (10/11)Ensure EOIS has data fields to collect all necessary data starting in 2010.
Integrate additional core measures/standards into OSQS and Schedule B asactual data is available and verified
-
7/30/2019 Espms Presentation Aug 28
16/25
16
Implementation - Awards and Recognition
Stage 1: (09/10-12/13)
Establish Ministers Award for Service Innovation and Service Results based on
interim targets and data collection and verification methodologies
Stage 2: (13/14)
Develop full Ministers Award to include Award for Service Quality and align
standards with fully evolved PMF. At this point, all awards based on ServiceQuality Standards and validated EOIS data.
Set Performance Incentive Funding levels and identify recipients based on datacollected in 12/13.
-
7/30/2019 Espms Presentation Aug 28
17/25
17
Appendix 1 - Measures, Definitions and Data
Indicators for Customer ServiceDimension Measure Definitions Data Indicators
Customer Service
40%
CustomerSatisfaction
15%
The % of individuals andemployers who reportsatisfaction with theemployment services
On a 1-5 scale, respondentswho rate satisfaction withservice at 4 or 5 level.
ServiceCoordination
25%
The provision of supportedand timely referral andaccess to services andsupports within EO and tocommunity services outsideof EO
Referred in from othercommunityorganizations/services
Registered foreducation/training to completehigh school or equivalent
Registered/participating in
trainingRegistered or confirmedreceiving services with othercommunity resources thatsupport employment
-
7/30/2019 Espms Presentation Aug 28
18/25
18
Customer Service (40%)
Service Coordination (25%)
Client Service Planning and Coordination indicators recognize, as part of a clients
service plan development, the support an agency provides to help them access andparticipate in education, training, and/or other types of community services either whilethey are engaged in employment services or at exit from employment services.
A formalized referral (a planned, supported and co-ordinated arrangement) is made bythe employment service provider on behalf of the client to the other organization(education/ training/community services). It also recognizes formalized referrals madeto the employment service provider from another community organization on behalf of aclient.
Customer Satisfaction (15%)
The Customer Satisfaction indicators recognize the value attributed to the service by its
customers. Employment service participants both individuals and employers aresampled at exit from service, on a 1-5 scale, on how likely they are to recommendthese employment services to someone looking for similar services. Service providersare assessed on the percentage of participants sampled who return a 4 or 5 (out of 5)rating.
-
7/30/2019 Espms Presentation Aug 28
19/25
19
Measures, Definitions and Data Indicators for
EffectivenessDimension Measure Definitions Data Indicators
Effectiveness
50%
ServiceInputEmployed/on CareerPath
25%
Success at enteringthe labour marketand/or on a careerpath based ongoals/qualifications
experience
Employed (with definition)Self-employedBoth employed and intraining/educationEmployed and a registered apprentice
Employed in a more suitable jobconsistent with individuals career pathEmployed in professional/tradeoccupation or profession/trade relatedoccupation consistent witheducation/training qualifications
Service
Input -Education/Training
10%
Registered in
education/recommended toregister in training tosupport careergoals/path
Individual has returned to school full
time or part-time at exit (includes 4breakout definitions)Individual has been referredto/recommended for registration in EOtraining (includes of 8 breakoutdefinitions)
-
7/30/2019 Espms Presentation Aug 28
20/25
20
Effectiveness (cont)
Dimension Measure Definitions Data IndicatorsEffectiveness
50%
ParticipantSuitability
15%
Indicators of participantneed (barriers toemployment) foremployment services
-Age - Under age 20/Over age 44-
-
7/30/2019 Espms Presentation Aug 28
21/25
21
Effectiveness (50%)Effectiveness is captured by two key measures which are linked together. The first,
Participant Suitability, is an activity indicator that gathers data on the clientsaccessing employment services from the service provider. It examines andquantifies identified barriers to employment facing the clients. These indicators ofParticipant Suitability work in conjunction with the outcome indicators of serviceimpact to acknowledge the outcomes achieved by the service providers accordingto their client group. This weighted calculation ensures that the service providersare appropriately recognized for service delivery to the most disadvantaged clients.
Participant Suitability (15%)
The Participant Suitability indicators have been identified in employment literatureas potential barriers to successful labour market attachment. Service providerscollect this information at intake (entrance) to the full suite employment service.
Service Impact (35%)
The Service Impact performance indicators are collected at exit from the full suiteemployment service. Service Impact Outcome indicators will be tracked @ exit and3, 6 and 12 months after exit from service.
-
7/30/2019 Espms Presentation Aug 28
22/25
22
Measures, Definitions and Data Indicators for
EfficiencyDimension Measure Definitions Data Indicators
Efficiency 10% FundedActivity
5%
The # of individuals forwhich the organization willdevelop EmploymentService plans
Start date at which the organizationbegins providing employmentservices to the individual
Informationsessions/workshopactivity
5%
The # ofworkshop/informationsession participants.
Total # of individuals whoparticipate in workshop/informationsessions (see definition forworkshop/information session)
-
7/30/2019 Espms Presentation Aug 28
23/25
23
Efficiency (10%)Funded activity captures activity measures relating to economies of scale and
organizations ability to meet contracted service levels. It recognizes the organizationsactivity to provide full suite employment services and resource and information to self-directed individuals.
Intake/Activity (5%)
Intake/Activity compiles the number of individuals for which the organization develops afull suite employment service plan and calculates it as a percentage of their contracted
service level. It is expected all organizations will meet or exceed this level. It does notinclude individuals for whom the organization refers on to another service withoutengaging in development of service plan. Organizations will not receive additionalrecognition for exceeding 100% of funded activity levels.
Information Sessions/Workshop Activity (5%)
Information Sessions and Workshop Activity is a measure that serves as a proxy foractivity relating to individuals who do not require full suite services and wish to access
resources and information. This measure collects the # of workshop participants.Orientation workshops/information sessions are defined as a structured learningsession that must contain comprehensive and relevant information which can beconducted in group sessions (on or off site), e-learning sessions or one-to-one (whenappropriate) these sessions can be up to 2 days in duration.
-
7/30/2019 Espms Presentation Aug 28
24/25
24
Appendix 2:
Analysis to Support Performance Incentive
In 2007-08, organizations under contract to delivery JC services (132 sites) were heldaccountable to a minimum Provincial Service Quality Standard of 6.0.
All but one met and surpassed this level.
Assessment was completed on the distribution of the remaining 131 contracts andfoundthe average result was 6.937, with a standard deviation of .493
71 (54%) agencies posted scores between minimum Provincial Service QualityStandard and the arithmetical average; 41 agencies were between average and 1ststandard deviation15 agencies (12%) scored between 1st and 2nd standard deviations and 4 agencies
(3%) met Provincial Quality Standard of 8 or above.
-
7/30/2019 Espms Presentation Aug 28
25/25
25