Innovations in AccountabilityFederal Government Directions inStrengthening Accountability and theCentral Role of Evaluation

Presented by Terry HuntHalifax, August 11, 2004

Canadian Evaluation SocietyNova Scotia Chapter

2

Outline of Presentation

• Evaluation Function inthe Government ofCanada

• TBS’ Results-basedAgenda

• Role of DeputyMinisters, Heads ofEvaluation andEvaluators

• Evaluation Process andEffective Practices

• Monitoring ofEvaluations – ERIC

• Evaluation Quality• Future Challenges for

Evaluation• Update on CEE

Activities

3

Prime Minister’s CommitmentsEnhance transparency and accountability bystrengthening oversight and sound management of publicresources by:

• Managing for results• Enhancing the comptrollership function and oversight

of government expenditures• Committing to the continuous reallocation of spending

42

All Other Ministers-- Federal Departments /AgenciesDeputy Heads

Parliament of CanadaHouse of Commons

Prime MinisterCabinet (Privy Council)

Parliamentary Committees

President of the Treasury BoardTreasury Board Secretariat (TBS)

Auditor General(appointed)

Conducts independent audits of government operationsProduces periodic oversight reports on the conduct of evaluationPromotes accountability and best practicesReports directly to Parliament

Treasury Board (TB)Cabinet Committee

President of the Treasury Board• Government’s management board

(financial, management and human resources)

Expenditure Review CommitteeCabinet Committee

President of the Treasury Board Review government program expenditures (value for money / reallocation to priority programs Strengthen management, oversight and effective delivery of programs and services

Evaluations submitted to support TB policy and funding submissions, and ERC reviewEvaluation units support implementation and monitoring of departmental Management Accountability FrameworkDevelopment of Program Activity Architecture organizational strategic outcomes

Evaluation supports development and monitoring of departmental Program Activity ArchitectureInternal accountability and reporting External accountability and reporting to TBS and Parliament (annual Departmental Performance Reports)

Results-based Management Directorate & Centre of Excellence for Evaluation

Capacity building Policies and standards

Deputy heads accountable for the application of Evaluation Policy within their departmentsHeads of evaluation implement policy as per TBS standards and guidelinesUse evaluation to make allocation/ reallocation decisions

Parliamentary reportingLinks evaluation and performance measurement

• Support assessment of RMAFs and Evaluation in support of TB policy andfunding submissions, and ERC review

Evaluation Function Within Government of Canada

5

Background: Evaluation Practicein the Federal Government

Formal evaluation policy in Canada for over 30 yearsEvaluation seen as key enabler of results-basedmanagementRecognized need to strengthen and repositionevaluation in government

6

Context for Change

Building Evaluation CapacityNew Policy and Standards took effect April 1, 2001

Greater TBS support and leadershipManagement standards for evaluationInvestment strategy for functional units in departments

However, we still have a long way to go:Budget 2004: Strengthening Public SectorManagement

a modern expenditure management and oversight systemCommitment to re-establishment of Office of ComptrollerGeneral at TBS responsible for the oversight of allgovernment expendituresincreased evaluation activity and capacity

7

Results-based Practicesin Support of Evaluation

Management Accountability Framework (MAF):– requiring deputy heads to put in place risk-based evaluation

plans and a strong evaluation system– Evaluation provides information to address MAF expectations– Departmental evaluation units to support department heads’

ability to monitor implementation and use of MAFProgram Activity Architectures:– Department-level logic model approach, linking programs to

strategic outcomes– Assessing achievement of strategic outcomes, using

established measuresExpenditure Review Committee Process:– ERC assessment of individual programs will require the analysis

provided by program evaluation

8

Management AccountabilityFramework MAF

• Requirement for annual public reporting on MAFexpectations

• Implications for Management Performance Accords

• Key MAF Expectations include:• Management frameworks aligned with outcomes• Relevant information on results (internal, service &

program) is gathered and used to make departmentaldecisions

• Public reporting is balanced, transparent, and easy tounderstand

9

TBS Management Accountability Framework

Public Service Values

By their actions departmental leaders continually reinforce the importance of PS Values and Ethics in the delivery of results toCanadians (e.g.: democratic, professional, ethical and people values).

Learning, Innovation and Change Management

The department manages through continuous innovation and transformation, promotes organizational learning, values corporateknowledge, and learns from its performance.

Results andPerformance

Relevantinformation onresults (internal,service &program) isgathered and usedto makedepartmentaldecisions, andpublic reportingis balanced,transparent, andeasy tounderstand.

Stewardship

The departmental control regime(assets, money, people, services, etc.)is integrated and effective, and itsunderlying principles are clear to allstaff.

Policy and Programs

Departmental research and analyticcapacity is developed and sustainedto assure high quality policyoptions, program design and adviceto Ministers.

Accountability

Accountabilities for results areclearly assigned and consistentwith resources, and delegations areappropriate to capabilities.

Citizen Focused Service

Services are citizen-centred,policies and programs aredeveloped from the ‘outside in’,and partnerships are encouragedand effectively managed.

People

The department has the people, workenvironment and focus on buildingcapacity and leadership to assure itssuccess and a confident future for thePublic Service of Canada.

Risk Management

The executive team clearly definesthe corporate context and practicesfor managing organizational andstrategic risks proactively.

Governance& StrategicDirection

The essentialconditions –internalcoherence,corporatediscipline andalignment tooutcomes -- arein place forprovidingeffective strategicdirection, supportto the Ministerand Parliament,and the deliveryof results.

10

Program Activity Architecture

Integrated Performance Measurement Strategies

Mapping programs to Strategic Outcomes:“Telling the performance story”Links program operational data (e.g.: financial) tostrategic outcomesSufficient detail for strategic decision-makingAccountability and transparency

Long-term expectation for ‘real-time’ data on programplans, performance and resources

11

Expenditure ReviewCommittee

• Extensive review of allgovernment spending toreport in Autumn 2004;planned implementationin 2005 budget

• Evaluation to play criticalrole in in supportingDeputy Ministers withdecisions regarding re-allocation

12

• Clear public performancereporting

• Demonstrates stewardship

Benefits: Strengthened Transparency,Accountability and Decision-making

• Monitor, report and managebased on results

• Assessment of relative priority ofprograms and activities

CANADIANS / PARLIAMENTARIANS

EXECUTIVE / CABINET COMMITTEES

DEPARTMENTS / AGENCIESPrograms, Stakeholders & Beneficiaries

13

Role of FederalEvaluation Function

Embeds principles of evaluation into managementpractice (“life cycle” of programs)

Evaluation as a facilitator and advocate for results-based management

Broadens scope beyond programs to policies &initiatives; cross-jurisdictions

Strategic use for evaluation

Commitment to transparency and management in “fullpublic view”

Investment strategy for functional units in departments

Centre of Excellence for Evaluation (CEE) created

14

The Life Cycle Approach toManaging for Results

Learn & Adjust

StrategicAnalysis

Reporton

ResultsPlan forResults

Monitor,Measure,Evaluate

Implement

Analysis of thecurrentenvironment,pastperformance,emergingpriorities andsignificantrisks toachievement ofdesired results

DevelopingStrategic andOperationalPlans –identify andplan to achievekey resultsand mitigaterisk

Action taken toachieve results

DeliveringDeliveringResults forResults forCanadiansCanadians

Ongoingperformancemeasurementand periodicevaluation todetermineprogress andallow forcorrectiveaction

ProvidingintegratedFinancial andnon-financialinformation on:results andaccountabilityfor internal andexternal use

Integrated Risk Management

Public Service and Organizational Values

15

Evaluation Policy Objective

Policy Objective: “ensuring government has timely,strategically focussed, objective and evidence-basedinformation on the performance of its policies, programsand initiatives to produce better results for Canadians”.

Bottom-line: Evaluation is a key decision-making toolfor that can help shape the way Treasury BoardMinisters and individual departments design, deliver,and report on programs.

16

Standards for Evaluation

• Evaluation Planning andIssues

• Competency

• Objectivity and Integrity

• Consultation and Advice

• Measurement and Analysis

• Reporting

Risk-based evaluation plansCoverage of full range of evaluationissues

Knowledge, skills and experience in,research design and data analysis

Perform free of impairments to objectivityAct with integrity in their relationships

Consult with major stakeholdersPeer review groups

Timely evaluation productsRelevant and useful findings

Concise and clearRecommendation flow from findings

17

Key Activities RequiredDeputy Ministers responsible for supporting evaluationfunction within department:-- has responsibility for evaluation unit and Head of Evaluation-- chair Departmental Audit and Evaluation Committee-- use evaluation findings to guide decision-making

Departmental Head of Evaluation must provideleadership and direction to the practice of evaluation:– strategic, risk-based evaluation plans– work with managers to help improve program design and

performance measurement– inform senior management of issues of concern– make reports publicly available– apply evaluation standards

18

Key Activities Required, cont’d.

Departmental program managers mustmanage for results:– draw on the organization's evaluation capacity– ensure that they have reliable, timely, objective and

accessible information for decision-making andperformance improvement

19

New Roles for Evaluators

Traditional skill set still important� methodologies� measurement� analytical� logic model approach

Facilitation role emerging� working directly with managers to facilitate the identification/articulation

of key results and associated performance measures and datagathering strategies

Promoting results-based culture� closer ongoing interface with managers� Workshops� Orientation/Education/Training sessions� Advisor� Linking horizontal/cross-jurisdictional issues

20

Federal Evaluation Process

1. Results-based Management and AccountabilityFrameworks

Blueprint for results-based management

Provides information on what a policy, program or initiativeis expected to achieve

Applies evaluation methodology to strengthen accountabilityand results-based planning practices in the design ofprograms

Requires identifying performance reporting needs at thefront end

2. Departmental Audit and Evaluation Committee

3. Risk-based Departmental Audit and Evaluation Plan

21

Effective Evaluation Practices

Organizational structure– Environment conducive to results-based management– A direct line to departmental decision makers– Level of resources (activity and capacity)

Planning– Incorporate evaluation planning in corporate planning process– Evaluation incorporated in program design

Quality control– Client satisfaction surveys– Formal tracking of implementation of evaluation

recommendations

Ensuring timeliness of evaluation products– Use of briefing deck to report preliminary findings– Quick contracting practices

22

Interim Evaluation of the EvaluationPolicy: Progress in Implementation

HRSDCandSDC

CSCCSCF&OF&OHCHCJCJC

NRCanNRCanPCHPCH

CICCICECEC

FAITFAITFin/TBSFin/TBS

ICICINACINACNEBNEBNFBNFBSGSG

VACVAC

•Implementation varied across departments and agencies; greatestprogress in those departments with a mature evaluation function in placeprior to 2001

•approximately one-half of departments and agencies (49%) reportedthat they had an evaluation plan approved

1

16

25

StrongCapacity butrepositioning

issues

ModerateCapacity butstill requireadditionalresources

BuildingCapacity from

a weak /minimalistposition

23

Growth in Evaluation Community

1 5 0

2 0 0

2 5 0

3 0 0

3 5 0

4 0 0

2 0 0 0 - 2 0 0 1 2 0 0 2 - 2 0 0 3 R e q u ir e d

P o l i c y E v a l u a t i o n F i n d i n g s i n d i c a t e 3 9 1 F T E s r e q u i r e d i n d e p a r t m e n t s

C A C S t u d y E s t i m a t e o f E v a l u a t i o n F T E s

P o l i c y E v a l u a t i o n D a t a F i n d i n g s i n d i c a t e 2 8 6 E v a l u a t i o n F T E s c u r r e n t l y i n t h e s y s t e m

2 8 6

3 9 1

2 3 0

• Increase of 56 Evaluation FTEs over two years• 50 Evaluators will leave system by end of 2004-05• Current HR Gap estimated at 105 FTEs – a 40% increase in trained evaluators still

needed• Distribution of resources, and lack of capacity in smaller Agencies, remain issues

24

Practical Applications:Capturing Evaluation Findings

Evaluation Review Information Component (ERIC)

CEE currently developing a database to captureinformation from evaluation findings to supportExpenditure Review policy “tests”

Provide CEE with the ability to respond in a timelymanner to Expenditure Review Committee and otherinquiries on evaluation findings

25

Information Captured in ERIC

Accountability indicators

Results - categorized using the 20 societal indicators ofCanada’s Performance Report

Coverage of ERC 7 Policy Tests

High-level quality of evaluation indicators

26

Samples of ERICInformation Output

Drawn from CEE analysis of 28 reports

27

ERC Policy Test #1: Public Interest

28

ERC Policy Test #2: Role ofGovernment

29

ERC Policy Test #3: Federalism Test

30

ERC Policy Test #4: Partnership Test

31

ERC Policy Test #5: Value for Money

32

ERC Policy Test #6: Efficiency Test

33

ERC Policy Test #7: Affordability Test

34

Achieving Objectives per ResultsCategory: economy

35

Achieving objectives: environment,cont’d.

36

Achieving objectives: communities

37

Data Collection Methods Utilized

38

Quality Assessment #1

39

Quality Assessment #2

40

Quality Assessment #3

41

Quality Assessment #4

42

Weaknesses of Evaluation Reports (n = 115)Minority of evaluation designs included a comparison group (13%); baseline measures(14%) or a comparison to norms, literature or some other benchmark (22%)

Only one-third of evaluations presented findings on whether the program duplicates orworks at cross purposes with other programs (0f these, 18% were found to beinadequate)

Only one third discussed unintended outcomes

Only 26% presented findings on alternative, potentially more cost-effective approaches

Unable to assess appropriateness of analysis in 50% of evaluations (ie: degree to whichanalysis is supported by data as determined by significance tests, response rates, etc).Where information was given, 32% were rated as inadequate

24% inadequate in provision of objective-based conclusions related to relevance, successand/or cost-effectiveness.Only 26% of reports identified alternative scenarios

Only 48% of evaluations included management response

A substantial proportion f evaluations reports were rated as inadequate with respect to thefair presentation of data, including numbers and sources (33%) the appropriatepresentation of technical information (30%) and the effective use of tables and charts25%)

43

Strengths of Evaluation Reports (n = 115)Most evaluation reports considered adequate (45%); more than adequate(32%), excellent(8%); although 23% judged as inadequate

Most reports provide good presentation of program being evaluated and underlyingassumptions

Over half of the evaluations provided a presentation of findings related to the continuingneed for relevance of the program in question

87% reported findings demonstrating whether or not the program was producing resultsthat supported its continuation or renewal (Although 26% of these were rated asinadequate)

Quality of evaluations improved after 2002 (32% found inadequate prior to April 2002,versus 18% after April 2002 – 37% found more than adequate after April 2002 versus 22%prior to this date)

No clear pattern to differences in the overall assessment as a function of organizationalsize

44

Future Challengesfor the Evaluation Function

Addressing multiple needs

Cost-effectiveness information

Moving beyond formative assessments to address the centralquestions: What impact did the program have? Did it make adifference?

Focus on quality of evaluations:design

reporting

objectivity

Building capacity

45

Future Challengesfor the Evaluation Function

Following up on use of evaluations

Making findings accessible

Learning Strategy

Integration of the evaluation function within ourorganizations – Cultural issues

Address questions of identity of evaluation andcertification standards

Profile of evaluation profession relative to audit

Results-based budgeting

46

New Products from CEE

Visit the CEE Website:http://www.tbs-sct.gc.ca/eval/eval_e.asp

New Products• CEE Newsletter• Guide for the Review of Evaluation Reports• Updated RMAF guidance• Evaluation Guidebook for Small Agencies• Results-based Management E-learning Tool• Recruitment Genie (to expedite recruitment and

staffing)

47

Learning Strategy for EvaluationCommunity

• Development Program for New Evaluators– Evaluation conducted of pilot in 2003; redesign and

renamed (formerly Internship Program for Entry-LevelEvaluators)

– Courses offered:• Essential Skills Series -- CES• RMAFs• Writing Evaluation Reports• Good Contracting Practice• Surveys/Data Analysis• Effective Interviewing• Working in Teams• Values and Ethics for Evaluators

48

New Products from CEE

Projects Underway• Guide for Program Managers on How to Use Evaluation• Effective Practices Report• Guidance on Developing Departmental Evaluation Plans• Policy Paper on Cost-Effectiveness• Meta-evaluations by sector• Learning Agenda (focusing on intermediate and senior

levels)• Research and policy development:

– Certification for evaluators– Costing of evaluations– Cost-benefit analysis– Conducting horizontal evaluations

49

Contact Information

Terry HuntActing Senior DirectorCentre of Excellence for Evaluation (TBS)L'Esplanade Laurier, 9 West Tower300 Laurier Avenue WestOttawa, OntarioCanada K1A 0R5Telephone: (613) 952-7447Hunt.Terry@tbs-sct.gc.cawww.tbs-sct.gc.ca/eval

50