rapid and efficient implementation of a large-scale performance evaluation initiative for the us...
DESCRIPTION
Over five years Khulisa and their partners conducted over 400 performance assessments for PEPFAR using Khulisa’s Performance Assessment Tool, that measured each PEPFAR programme's efficiency, effectiveness and impact.TRANSCRIPT
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Rapid and Efficient Implementation of a Large-Scale Performance
Evaluation Initiative
for the US Government in South Africa
In: Proper Use of Evaluation to Impact Funders and Society
AEA Annual Meeting, October 17, 2013
Susan J. Griffey, DrPH, BSN, FNP
Social & Scientific Systems, Inc.
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Outline
• A little background
• Key points
• What we did and what resulted
• Implications
• How you can apply this
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
A Little Background
Large-Scale Performance Evaluation Initiative
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
27 Evaluators +
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Using Standardized Performance Assessment (PA) Implementation
• Streamlined Performance Assessment (PA) methodology
• Training
• Data collection
• Reporting for the PA and for the portfolio
• Operations and logistics
• Business and administrative processes
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Assessed 60 Organizations
• Not-for-profit, NGO
• For profit
• Academic/research institutes
• Partnerships and consortia
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
On 3 Programmatic Dimensions
• Fiscal management (FM)
• Systems strengthening (SS)
• 1 or more technical program areas [TPA]), each with several sub-areas
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
at 263 Locations across South Africa…
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Over 10 Months
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Produced Rapid Performance Results
• Minimal burden on grantees and funders
• Met funder requirements for accountability and compliance
• Able to give clients relevant results
– Rapidly
– In an immediately-useful way
• Value added from portfolio-level results
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Standardize, Streamline, Automate for Large-Scale Evaluation
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Streamlined Performance Assessment (PA) Methodology
• Focused on critical items only
• Time-limited data collection at each organization
– 1 day each at Headquarters and 2 service sites
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Streamlined evaluator training and Peer Support
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Data CollectionSample Data Collection Worksheet
Availability and awareness ofpolicies, plans, procedures,and regulations
Human CapacityDevelopment & SystemsStrengthening
Organisational Administrationand Fiscal Management
Programme Management,Implementation, andLeadership
Service delivery
Referrals, Linkages,Integration
Excellent
Stro
ng
Som
e im
pro
vem
en
t n
ee
de
d
Imm
edia
te a
tten
tio
n
req
uir
ed
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Report Templates
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Operational and Logistical
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Business and Administrative
Planning/SOW
becomes invoice
Expense Claim =
Reimbursement
Request
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Scheduling Software and UNavailability Calendar
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Findings
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Portfolio: Strong Overall in all 3 Programmatic Dimensions
Technical Program Areas
2.57/3.00
Fiscal Management
2.69/3.00
Systems Strengthening
2.63/3.00
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
% with STRONG Scores
• Overall, FM much stronger
• SS and TPA were similar: 66% and 64%
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
We did more with less
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Implications
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Implications
“[The] report has provided [funder program manager] with clear indications as to what were some of the weaknesses with the just-concluded grant and has also provided them with clear guidance and the way forward with the new grant that just started in 2013.”
Funder
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Standardize, Streamline, Automate for Large-Scale Evaluation
Managers with multiple programs of any budget level – large or small
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Click to Edit Master Title Style
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
K h u l i s a Management Services (Pty)Ltd Reg No.: 99 09520/07
Contact Us!
Sue Griffey, DrPH, BSN
Vice President and Director, Evaluation Center
Social & Scientific Systems, Inc.
– [email protected] @SueMentors
– www.s-3.com
Mary Pat Selvaggio, MPH, BS, Director for Research and Health
Edna Berhane, MPH, MIA, Associate Director
Khulisa Management Services
– [email protected], [email protected]
– www.khulisa.com