principles of results framework document development of a

Development of a Long-term Policy

Document on provision of citizen-centric

administrative services by central state

institutions

Principles of Results

Framework

Document

October 2015

Development of a Long-term Policy Document on provision of citizen-centric administrative services by central state institutions | Principles of Result Framework

PwC 2


PwC 3

Document Control

Project Title

Development of a Long-term Policy Document on provision of citizen-centric administrative services by central

state institutions.

Engagement Reference

Contract for professional services No. # 06/2015 between UNDP and PricewaterhouseCoopers Audit sh.p.k.

Document Title

Principles of Results Framework (RF).


PwC 4

Disclaimer

This report (“the Report”) has been prepared in accordance with the terms of our engagement and for no other

purpose. The contents of the Report as well as any other information or comments made thereafter are cannot

be published, distributed, copied or given to any third party. The recipient of this Report has been informed

regarding the confidentiality of the information and is aware that it cannot be used for any other purposes but

those of the specific assignment. We do not accept or assume any liability or duty of care for any other purpose

or to any third party to whom this document is shown.

This Report contains information that has been obtained from various sources as indicated in the Report. We

have sought to establish the reliability of these sources as far as possible, however, no representation or

warranty of any kind (whether express or implied) is given by PwC to any person as to the accuracy or

completeness of the information. The scope of our services was not designed to enable us to express an opinion

and therefore PwC does not express an opinion regarding the presented information.

This is a Report, based on which we expect to receive feedback and comments. The comments obtained will be

taken into account and presented in the final Report.


PwC 5

Contents

Purpose of this document 7

Introduction 9

Key Principles of Results Framework 13

Benefits of Results Framework 19

Key Challenges 21

Managing the RF Process 24

Way Forward 27


PwC 6

Glossary

Abbreviation Meaning

GoA Government of Albania

ICT Information and Communication Technology

IT Information Technology

KPI Key Performance Indicator

MDA Ministry, Department, Agency M&E Monitoring and Evaluation

RF Results Framework

RFD Result Framework Document

ToR Terms of Reference

UNDP United Nations Development Program

VMO Vision, Mission & Objectives

WP Work Plan


PwC 7

Purpose of this document

PricewaterhouseCoopers Audit sh.pk (PwC) has been appointed by UNDP for consultancy for ‘Development of a

long-term Policy Document on provision of citizen-centric administrative services by central state institutions

in Albania; as per Letter of Award dated 29 July 2015, RFP dated 04 May 2015 and our proposal dated 22 May

2015. One of the key deliverables includes drafting Principles of Results Framework (RF) for the policy

document.

The RF provides a summary of the most important results that a specific program or initiative expects to

achieve during a defined period of time, which is typically a financial year. The RF serves two main purposes:

Move the focus of the Government from process-orientation to result-orientation

Provide an objective and fair basis to evaluate the program’s overall performance at the end of the

defined period

These principles will help in the creation of the Results Framework Document (RFD) template for the

Government, which will need to be filled up by the respective Ministries, Departments and Agencies (MDA1s)

individually, based on their roles, responsibilities, functional mandate and programs. This report identifies and

discusses the key principles associated with the development and adoption of a RF within the Government of

Albania.

Once finalized and adopted, these principles will serve as a reference guideline for any Government Ministry,

Department and Agency or institute for the development and implementation of its Result Framework

Document.

The objective of this document is to provide for the reference principles and framework for adoption and implementation of the RF within the GoA. The document has been intentionally kept at a high level as it only discusses the key principles.

1 MDA refer to the Ministry, Department and Agencies and institutes within the GoA


PwC 8

Introduction


PwC 9

Introduction

Government institutions deliver services essential to the well-being and development of the nation. To ensure

that the public service delivery is as efficient and economical as possible, all Government institutions are

required to formulate strategic plans, allocate resources to the implementation of those plans, and monitor and

report the results. In order to achieve its objectives various projects, programs and initiatives are undertaken by

the Government. However, it is essential to focus the attention of the public and oversight bodies on whether

Government institutions are performing against their objectives, goals, service delivery plans or there is need to

alert decision makers to areas, where corrective action is required.

It is difficult to know if programs have succeeded or failed if the expected results are not clearly articulated. An

explicit definition of results - precisely what is to be achieved through the project or program and by when—

keeps measurable objectives in sight, helps monitor progress toward those objectives, and assists with

adjustment and management of program implementation. Making the best use of available data and knowledge

is crucial for improving the execution of the Government's mandate.

A results framework serves as a key tool in the development landscape, enabling Government officials as end-

users to discuss and establish strategic development objectives and then link interventions to intermediate

outcomes and results that directly relate to those objectives.

The RF provides a summary of the most important results that a specific program or initiative expects to

achieve during a defined period of time, which is typically a financial year. The RF serves two main purposes:

Move the focus of the Government from process-orientation to result-orientation

Provide an objective and fair basis to evaluate programs overall performance at the end of the defined

period

The introduction of a Result Framework system in a public institution or on a country-wide basis presents a

significant challenge, which requires both organizational changes as well as cultural changes within the

Government MDAs. This is even more significant considering that different MDAs are at different levels of

readiness, maturity and capacity and have their specific mandates and visions.

To be effective, RF needs to be positioned as far more than a technical instrument for change. It is not enough

to simply create a highly trained evaluation capacity and expect that organizations and systems will eventually

become more effective. A result framework that incorporates the best elements of design, standardizing

different mandates and good governance principles with necessary checks and balances will form the

foundations of an acceptable evaluation system. There is a need to also address the institutional capacity and

indeed the drivers that are generating the demand for evaluation.

The following diagram depicts the key enablers, which are essential for the development, implementation and

sustainability of a Result Framework.


PwC 10

Vision

This refers to the understanding as to how the Result Framework information can assist public sector managers

and decision makers. This of course requires strategic leadership as well as a clear understanding of the basic

concepts and potential uses of the Result Framework.

Enabling Environment

This is a fundamental requirement for ensuring a commitment to not only launch a Result Framework exercise

but also to sustain it over the long-term. This translates into a commitment to resource such an exercise, as well

as providing an enabling environment to allow it to develop and mature. Given that the introduction of a

Monitoring and Evaluation (M&E) system might challenge the current culture and the way of doing things

within Government organizations, political will and leadership are essential to support the values and ethics

that underlie a successful Result Framework exercise; that is, transparency, objectivity, accountability and a

commitment to a ‘results’-orientation and good governance.

Participation Planning

It is important to identify intended participant and measures to ensure participation. There is a need to develop

a schedule and plan out the process for engaging partners and stakeholders.

Conduct RFD development sessions with intended participants

Develop a preliminary results framework and hold sessions with key counterparts to present the draft

strategy and obtain feedback

Capacity and Infrastructure

Capacity includes both the existence of credible and relevant data and information-gathering systems as well as

the skilled personnel to gather, analyze and report on the performance of Government policies and programs.

Demand and Use Result Framework Information

Capacity to ‘use’ RF information requires both a clarity of expectations re where and how RF information is

intended to be used within Government organizations (e.g. planning, policy or program development; decision-

making; budgeting), as well as the capacity within Government institutions to actually incorporate and use the

RF information as part of the normal process of business.


PwC 11

Appropriate Approach

Structured workshops and session may be conducted to construct the results framework. Focus should be on

drawing out the ideas of the group and translating them into the results framework.

Timing

It is very important to assess when the results frameworks should be developed. Success of the activity depends

upon the amount of program information and data available to confidently construct a results framework. So it

is important to identify the time to initiate, considering the available data.


PwC 12

Key Principles of

Results Framework


PwC 13

Key Principles of Results Framework

This section discusses the key principles of the Results Framework (RF). As depicted in the below diagram,

there are broadly six key principles of RF. These principles become the basis of the formulation of the Results

Framework Document (RFD) for any MDA. The RFD has to be developed by the MDAs at an individual level

and once approved at the Government level, this RFD becomes the basis for monitoring and assessing the

performance and the outcomes of the programs and initiatives for the respective MDA.

Core Values

Core Values refer to the foundation principles for any Government

MDA or institute including Vision, Mission, Objectives (VMO) and

functions.

Core Values should capture the ‘Vision’ to identify the big

picture for the MDA or institute as envisaged by the

Government leadership

o Vision statement will be clear and long term, resulting

from existence of MDA

o Vision should have a time horizon of 5-10 years

It is important to include the ‘Mission’ as part of the RFD

because this ‘Mission’ in a way defines the purpose of existence of the MDA or institute

Objectives include the overall goals, purpose and mission of a business that have been established by its

management and communicated to its employees

o Objectives of a Department typically focus on its long range intentions for operating and its overall

governing philosophy that can provide useful guidance for stakeholders and beneficiaries

Core Values

Targets and Performance Indicators

Assess Past Performance

Delivery Commitments

Outcome and Impact

Evaluation Mechanism

Key Principles of RF

Core Values




Outcome and Impact



PwC 14

o Objectives should be linked and derived from the Departmental Vision and Mission statements and

should remain stable over time

Functions should be consistent with the allocation of business rules for the MDA or institute and they

cannot be changed in the RFD unless there is a change in the basic functions


It is essential to assess clearly how the Government is able to

deliver on its mandate and this requires identifying performance

indicators, against which assessment can be done. It is challenging

to specify indicators that measure things that are useful from a

management and accountability perspective. There is a need to be

selective when defining indicators. Defining a good performance

indicator requires careful analysis of what is to be measured.

First step is to shortlist key objectives that would be the focus for

the current RFD. It is important to be selective and focus on the

most important and relevant objectives only:

Objectives may be prioritized according to the degree of

significance

For each specific objective, the department must specify and map the required policies, programs, schemes

and projects

Performance Indicators: This helps in ascertaining level of how well a Department is meeting its aims and

objectives, and which policies and processes are working. Defining a good performance indicator requires

careful analysis of what is to be measured. The following steps can be followed while defining or identifying the

performance indicators as part of the RFD:

Core Values




Outcome and Impact



PwC 15

Agree on the problem those needs to address

Understand the desired changes to define a clear set of outcomes and impacts

Need to assess what it needs to deliver to achieve the set outcome

Specify the outputs, activities and inputs the institution needs to do to achieve the desired outcomes and

impacts

With the help of the desired outcomes and impacts, identify and shortlist the indicators which are required

to service delivery related aspects such as process efficiencies, etc. Select the most important indicators

needed to measure key aspect of service delivery and outputs

Following aspects should be considered while finalizing the indicators:

o The selected indicator communicates whether the Department is achieving the strategic goals and set

objectives

o The indicator is clear and easily understood by stakeholders who will be using it

o The data is readily available for the selected indicator

It is important to have a thorough understanding of the nature of the input, key activities, desired outcomes and

impact along with all the relevant definitions and standards used in the field or domain of the MDA and hence it

is crucial to involve subject matter experts and line managers in the above process.

Targets: Once a set of suitable

indicators has been defined for a

program or a project, the next step

is to specify the level of

performance the institution and its

employees will strive to achieve.

Communicate what will be

achieved if the current policies

and programs are maintained

Enable performance to be

compared at regular intervals -

on a monthly, quarterly or

annual basis as appropriate

Facilitate evaluations of the

appropriateness of current

policies and programs

One of the Performance

frameworks adopted globally, which

defines a useful set of criteria for

selecting performance targets, is the

“SMART” criteria, depicted with

the help of the adjacent diagram.


PwC 16


In order to be realistic and analyze the present performance, it is

important to thoroughly assess past performance information for

comparison. For every success indicator and the corresponding

target, RFD must include values for the past few years and also

projected values for the future. The period of past and future

performance may be selected considering data availability.

The inclusion of data for the past years vis-a-vis the projection for

the next few years will help in assessing the target for the current

year. If an action is being initiated in the current year, then data

for the previous year is not required.


In a Government set up it is common that there is a lot of flow of

information & data among various Ministries and Departments. In

some cases the service delivery may involve efforts from multiple

departments. Hence it is important to capture information that

impacts the department’s performance and are critical for

achievement of the selected Success Indicator. Apart from the

Delivery Commitments of the government MDA, which are

identified in line with its mandate, VMO and annual plans, it is also

important to understand and ascertain the requirements,

dependencies and expectations from other MDAs, which directly or

indirectly impact the performance of the institute. Such

requirements, dependencies and expectations should be mentioned

in quantifiable, specific, and measurable terms as part of the RF

under this component.

Outcome and Impact

Though the evaluation through RF will be done against the

identified targets, it is essential to understand expected outcome

and impact for which MDA is responsible. This helps in reminding

us about not only the purpose of the existence of the MDA but also

the rationale for undertaking the RFD exercise. The whole point of

RFD is to ensure that the department/ministry serves the purpose

for which they were created in the first place.

A mechanism needs to be established to assess the broad outcomes

and the impact of the programs, projects or initiatives of the MDA.

It should capture and closely map with the mandate of the MDA.

Core Values




Outcome and Impact


Core Values




Outcome and Impact


Core Values




Outcome and Impact



PwC 17


Evaluation is defined as the systematic collection and objective

analysis of evidence on public policies, programs, projects,

functions and organizations to assess issues such as relevance,

performance (effectiveness and efficiency), return of investment

(RoI), impact and sustainability. Evaluation essentially involves

comparing achievement against a target.

The Government stakeholders should agree on the target (specific

to the MDA), against which the quality of the RF will be judged.

Here, it should also be considered as to how closely the RF

guidelines were followed at the time of the formulation of the

institute specific RFD.

The identified Evaluation Mechanism should be institutionalized

across the Government. The participating MDAs should own up the responsibility to incorporate the evaluation

mechanism as part of their management function including:

adequate allocation of budget towards implementation and adoption of RFD

identification of dedicated resources

implementation of appropriate measures based on the outcomes of the evaluation

Some of the key guiding principles, which can be considered for the evaluation, are as follows:

Evaluation should be development-oriented and should address key development priorities of the

Government and the citizens

Evaluation should be undertaken ethically and with integrity

Evaluation should be utilization-oriented

Evaluation methods should be comprehensive

Evaluation should advance and promote the Government’s transparency and accountability agenda

Evaluation should be undertaken in a way which is inclusive and participatory

Evaluation must promote learning for all the participating stakeholders

Core Values




Outcome and Impact



PwC 18

Benefits of Results

Framework


PwC 19

Benefits of Results Framework

The RF provides for a well-defined framework to monitor and measure progress as well as the intended

outcomes for the program. A well-constructed RFD facilitates effective monitoring, management, and

evaluation of various Government programs. It provides meaningful information to ascertain the success ratio

of the adopted programs and policies of the institute, which in a way also reflects the success achieved by the

institute in terms of fulfilling the Government mandate.

The RFD also provides a mechanism for any institute to achieve the following:

Create focus on specific expected outcomes

Establish an evidence-based approach to monitoring and evaluation

Provide effective tool to assess program implementation with respect to progress in achieving results at the

outputs, outcomes and impact levels

Achieve strategic objectives, which is the ultimate driver for any program

Provide a systematic approach for decision makers to plan their strategies and to select interventions that

are most likely to address targeted problems

Guide corrective actions to activities, reallocating resources and reevaluating targeted objectives

Clearly identify how progress toward the targeted objective(s) will be measured

Clarify definitions and standards for performance information

Promote accountability and transparency with timely, accessible and accurate performance

Assess what approaches or interventions contribute most effectively to achieving specific development

objectives, a process that helps identify good practices for replication.


PwC 20

Key Challenges


PwC 21

Key Challenges

Successful execution of the RFD requires greater engagement and ownership among stakeholders

Involvement of multiple MDAs for various services adds to complexity in performance assessment

Process, documentation, maturity level may differ among MDAs and lack of a larger view of standardization

across country

Insecurity and apprehensions about outcome may lead to providing inaccurate data for showing better

results

Reluctance to performance evaluation of people may lead to ineffective outcome

Ensuring adherence to timelines by various institutions is a must for meaningful outcome of the exercise

and to have larger view at country level

It is important to have a buy-in of the process among the users and to communicate the importance of the

exercise

Effects of interventions can be difficult to measure fully as the set of indicators carries the risk of not fully

capturing achievements and progress and of missing unintended consequences

Results frameworks can become overly complicated and therefore importance, relevance, cost, timeliness

and utility are key considerations for determining which set of indicators should be included.

Involving program staff in the evaluation process could bias results measurement, since implementers are

motivated to collect data that reflect positively on an intervention and its results

As part of the addressing these challenges, the following action points may be considered by the key

stakeholders responsible for the implementation and adoption of RF within the GoA:

Establish an institutional structure for the RF development, implementation and monitoring. An indicative

structure has been discussed in the next section titled ‘Managing the RF Process’. It needs to be assessed

further in terms of the capacity and preparedness of the existing agencies within the GoA for the role of the

Nodal Agency (NA) for the RF initiative. ADISA under the supervision of MIPA can be one of the options

which can be considered by the Government

Setup regular and periodic review meetings involving all the key stakeholders and participating MDAs at

the highest level of Government possible to ensure commitment and ownership at the MDA level. These

reviews can be undertaken by an independent agency to avoid any conflict at the MDA level. For example,

Delivery Unit (DU) can be one of the options to play the role of the independent reviewer for RF

implementation who will consolidate periodic findings on the RF initiative and submit to the Nodal Agency

for necessary action

Undertake change management initiatives to address any concerns around the RF

Undertake stakeholder consultation workshops to inculcate a sense of ownership amongst the MDAs with

senior leadership participation from Government to communicate the seriousness and commitment to the

cause of RF adoption

Define easy to understand and clear measurable as part of the RF and map them at the initiative level as

well


PwC 22

Undertake training sessions and if necessary, capacity building programs to support and handhold the

participating MDAs at least during the first year of the RF implementation. In the context of Albania, this

responsibility can be undertaken by ADISA

Develop, implement and upgrade infrastructure to support the RF implementation

Understanding these challenges and the underlying mitigation measures is even more important in the context

of Albania since a concept like RF will be implemented for the first time and there are bound to some teething

issues initially. However, with proper planning and preparation, and team work amongst key agencies of GoA

such as MIPA, ADISA, NAIS, DU, etc., the implementation of RF can be done with considerable success.


PwC 23

Managing the RF

Process


PwC 24

Managing the RF Process

Results Framework should be integrated within existing management processes and systems. Key points to be

kept in perspective to ensure this include:

Proper and adequate documentation focusing on the following:

o Integration of RF within existing management processes and systems

o Definitions and technical standards of all the information to be collected by the institution

o Processes for identifying, collecting, collating, verifying and storing information

o Use of information in managing for results

o Publication of performance information

Adequate capacity to manage this activity

Appropriate systems to collect, collate, verify and store the information

Consultation processes that ensure the information needs of different users are taken into consideration

when specifying the range of information to be collected

Processes to ensure the information is appropriately used for planning, budgeting and management

Ensure that responsibility for managing performance information to individual’s role

Identified set of performance indicators for reporting for oversight purposes

Providing public access to Government-held information

o Wide range of information collected by Government can help decision-making in the private sector,

civil society and the general public if placed in the public domain

Just like other countries, even in the context of Albania, to ensure successful adoption of the RF within the

government framework, it is extremely important that this entire agenda is driven from the top leadership of

the government to ensure proper and timely implementation. Here, it is important to understand the different

roles involved for managing and implementing the RF which has been described in the following table.

S. No.

Role Responsibility Remarks

1 Key Sponsor

A prime stakeholder or an entity who is really interested and has important stakes in getting the RF implemented since it will benefit this entity and the associated eco-system. This role is typically played by the highest level Government leadership to ensure that the RF is taken seriously and is implemented in all its earnestness by the various government MDAs.

The key responsibility includes regular review, reinforcing the importance of RF to all the stakeholders,

Provide support for resolution of all the challenges, issues and difficulties

In the case of Albania, the role of the Key Sponsor could be played an entity at the top level of the Government such as the Office of the Prime Minister or Office of the Deputy Prime Minister.


PwC 25

S. No.

Role Responsibility Remarks

Provide guidance and supervise the overall RF implementation

2 Nodal Agency (NO)

Refers to the agency or the entity responsible for managing the day to day operations from the perspective of implementing the RF in a country. This agency should ideally have some exposure and understanding on the monitoring aspects of the government systems and functions. The key responsibilities of the NO include:

Develop and prepare the policy and guidelines for the implementation of the RF

Finalize the RF template

Help and supervise the design and development of the online system for RF

Conduct the change management and capacity building activities

Undertake stakeholder consultations for finalization of the RF framework and the underlying template to be followed

Function as the custodian of the data and documents to be received from the participating MDAs

Setup a help desk and provide handholding support to the participating MDAs with the development of their respective RFD

In case of Albania, the role of the NO can be played by an agency who ideally is also the owner of the forthcoming Long Term Policy Document such as ADISA.

3 Implementing Agency (IA)

Refers to the agency which will be responsible for the implementation of the online system for managing the RF program including online forms, etc.

In case of Albania, this role could be played by NAIS (or AKSHI) given the scope of the activity for design and development of the system.

4 Participating MDA

Refers to the Government entities including Ministries, Departments and Agencies who will participate and develop their respective RFD. The key responsibility of the MDA will include:

Develop, define and finalize the MDA specific RFD

Establish internal institutional mechanism to support the development and implementation of the RFD

Regularly update the RFD

Provide timely and accurate response on the letters

It is important to establish an internal framework and institutional mechanism as to which units or individuals will be responsible for the development, adoption and update of the RFD for the respective MDA.

5 Working Group

Refers to the cross ministerial or agency entity which has adequate representation from different MDAs. The key responsibility of the Working Group will be to build consensus on the RFD template since a common template needs to be adopted across all the MDAs.

In the context of Albania, one or a combination of more than one working groups from the existing set can be considered for this role

The above tabular structure is indicative in nature and may undergo change at the time of the RF finalization

and adoption by the Government.


PwC 26

Way Forward


PwC 27

Way Forward

This document has described the concept behind the RF and along with the key principles, components,

benefits, challenges and the RF management process. As explained in the initial section of this document, the

intent of this document was only to provide a high level view of RF and the underlying principles. Going

forward, once the Long Term Policy Document has been finalized and adopted by the GoA and its MDAs, this

document can be used as a reference for the purposes of preparing and detailing out the RFD for each MDA at

an individual level.


PwC 28

principles of results framework document development of a

Documents