who reform stage 2 evaluation final report appendices · who reform stage 2 evaluation - final...

WHO Reform Stage 2 Evaluation Final report Appendices

18 November 2013

WHO Reform Stage 2 Evaluation - Final report Appendices

PwC 2

Table of content

Appendix A- Inception Report 3

Appendix B- Addendum to Inception Report 40

Appendix C- Evaluation questions 61

Appendix D- List of interviews 67

Appendix E- List of documentation 71

Appendix F- Analysis of staff survey 74


PwC 3

Appendix A- Inception Report

With offices in Aarau, Basel, Berne, Chur, Geneva, Lausanne, Lugano, Lucerne, Neuchâtel, Sitten, St. Gallen, Thun, Winterthur, Zug and Zurich, PricewaterhouseCoopers SA is a provider of auditing services and tax, legal and business consultancy services. PricewaterhouseCoopers AG is a member of a global network of companies that are legally independent of one another; the network is represented in some 150 countries throughout the world.

PricewaterhouseCoopers SA Avenue Giuseppe-Motta 50 P.O Box 2895 1211 Geneva 2 Phone +41 58 792 91 00 Fax +41 58 792 91 10 www.pwc.ch

Office 7121 Bid Ref: IOS Stage two Attn: Mr David Webb, Director, Office of Internal Oversight Services World Health Organization 20, Avenue Appia CH-1211 Geneva 27 Switzerland Geneva, 8 July 2013 Dear Sir / Madam, Thank you for giving PricewaterhouseCoopers (“PwC”) the opportunity to be selected as the providers for the Stage 2 Independent Evaluation (“the evaluation”) of the World Health Organization (“WHO”). We understand the strategic importance of this evaluation and look forward to working closely together to make it a successful assignment. This document sets out our inception report which has been prepared in accordance with the Letter of Agreement for the Independent evaluation of the WHO reform: Stage 2. The purpose of the Inception report is to present our detailed approach to conducting the Stage 2 Evaluation of the WHO reform and presents our project management plan including timeline and milestones of the project. We remain at your disposal for any further information or clarification on our inception report and look forward to hearing from you. Yours faithfully, PricewaterhouseCoopers SA

Gill Sivyer Antoine Berthaut

Engagement Partner - Geneva Engagement Director–Geneva


PwC 5

1. Understanding of the project scope

1.1. Understanding the WHO reform

The proposed evaluation is part of a far reaching reform agenda initiated by the WHO in 2010. The aim of the

reform is to improve the overall performance and accountability system of the organisation to better address

the changing public health needs of the population going forward into the 21st century.

One of the major reasons for kick starting the process of reform has been the continued unpredictability of

funding and the difficulty WHO has had to secure financing for its priority activities and programmes.

Additionally, and as highlighted by the report called ‘Review of management, administration and

decentralisation in the WHO of the Joint Inspection Unit’ (January 2013), WHO has been slow to reform in the

areas of internal and external governance, organisational effectiveness and transparency.

Some of the main challenges identified were around the:

Need for better internal governance and alignment between global and regional bodies

Difficulty of allocating resources across various layers of governance structures

Lack of predictability of funding and associated challenges with priority setting

Weak resource mobilisation capacity at all levels of the organisation

Ballooning of administrative and management costs

Rise of other global health actors and the role of the WHO in a changing environment

The High-level implementation plan of the WHO reform, referred to as Implementation plan in this report,

was developed through an extensive consultation process initiated in 2011 with the discussion of the future

financing of the WHO that has resulted in the definition of four reform areas (programmatic, governance,

managerial and change management) and associated outcomes and related outputs. Whilst the reform areas

and outcomes have been validated by the Secretariat and Member States and agreed upon as the set direction

for the reform, the Implementation plan is a flexible document that continuously aims to improve the quality of

its outputs and responsibility framework.

1.2. WHO reform evaluation process

As part of the above, the Executive Board at its special session in November 2011 requested a two stage

independent evaluation of the WHO reform.

The Stage 1 evaluation of the reform was conducted by WHO’s external auditors, the Comptroller and Auditor General of India, between February and March 2011, with the aim to validate the completeness, comprehensiveness and adequacy of WHO’s reform proposals centred around governance, management and programmatic reform. The recommendations elaborated by the Stage 1 evaluation have included:

Strengthening linkages between governing bodies at the regional and HQ level for greater organisational coherence;

Strengthening the accountability systems throughout all three levels of the organisation (i.e. HQ, regional and country level) which could be supported by a more robust results-based and performance management system;

Focusing on change management through the development of a communication strategy on reform that clearly identifies the champions of the reform at WHO;


PwC 6

Implementing a sound monitoring and evaluation framework that accompanies the reform and is supported by clearly defined outcomes and outputs indicators.

The modalities for Stage 2 of the evaluation were endorsed by the Executive Board at its 132nd session in January 2013 to assess the readiness of the WHO to fully implement the reform agenda and to identify the necessary conditions for success.

1.3. Objectives of the Stage 2 Evaluation

This independent evaluation has at its core the objective of supporting the reform process and assisting the WHO in identifying the mechanisms and tools required to unlock some of the areas of the reform which have been lagging behind. Member States have demonstrated increasing confidence in the evaluation process accompanying the reform and have several expectations in relation to seeing the reform hit the ground. The primary audience for the evaluation will be the Executive Board, represented by the Evaluation Management Group. This evaluation represents an opportunity to validate current reporting to Member States and to foster trust and confidence in the reform process. The secondary audience will be Secretariat management and staff involved in Reform activities. The evaluation will favour exchange of knowledge between the Independent Evaluation Team and the Secretariat and will contribute to paving the way for efficient, effective and sustainable change.

As several reform initiatives defined in the Implementation plan are underway and at different stages of implementation, this independent evaluation will equally cover all aspects of the reform while bringing more depth into areas which the Independent Evaluation Team will have identified has being critical to the success of the reform program and the WHO as a whole.

Also as the Reform is still in its early day, the evaluation will not seek to assess the impact or outcome of the Reform.

The evaluation will follow UNEG norms and standards for evaluations, as well as ethical guidelines.

1.3.1. Assess WHO reform status

The Terms of References for the evaluation, as well as discussions the Independent Evaluation Team has had with members of the WHO Office of Internal Oversight Services (IOS), the Reform Team responsible for overseeing and reporting on the execution of the reform agenda, and a limited number of WHO Member States, indicate a preference for a recommendation-oriented evaluation that addresses the following objectives:

Assess the status of actions taken on the Stage 1 recommendations: In this context the evaluation will in the first place assess whether the recommendations mentioned in Section 1.2 above made in the context of the Stage 1 Evaluation have been addressed and are integrated as part of the Implementation plan.

Assess progress made to date on the WHO reform implementation: The evaluation will assess the status of the reform agenda which will involve reviewing and confirming progress made to date on outcomes, outputs and deliverables defined in the Implementation plan.

Assess the completeness of the WHO reform Implementation plan: Stage 1 of the evaluation validated whether the overall objectives of the reform agenda were in line with the challenges that the WHO needed to address. In contrast, Stage 2 focuses on validating whether the implementation strategy supporting these initiatives addresses all required elements needed to effect change and achieve the intended results. A specific focus relates to assessing whether the theory of change or result-chain of the reform is robust and adequately translated into reform operational plans. This comes down to validating the completeness of the Implementation plan.


PwC 7

1.3.2. Assess the modalities of implementing the reform proposal and the sufficiency of the change management strategy

The second objective of the evaluation relates to validating whether the enabling conditions for success of

the reform are met in order for the reform in general and each initiative in particular to succeed. This includes:

Prioritisation of various components of the reform proposal;

Identification of change agents;

Capability of accountability structures to support the reform process;

Resource requirements for the reform proposal;

Timelines defined for the implementation of the reform proposal;

Performance indicators defined to measure movement towards reform process;

Strategy to deal with hindrances, enablers and dependencies;

Changes in internal procedures and structures to implement the reform process.


PwC 8

Structure (incl. internal governance and accountability)

Strategy

Process (incl. transparency)

People (incl. role)

Technology (incl. Instruments)

Programme Management

Change Management (incl.barriers to change )

Assess & Strategize

Rolling-out new ways of working

Constructing, testing and validating detailed changes

Designing blueprints and detailed execution plans

Building the case for reform, assessing options and validating reform strategies

Operating the reformed organization and ensuring continuous improvement

Ch

an

ge

E

na

ble

rs

De

liv

er

ing

Ch

an

ge

Design Construct Implement Operate & Review

Outcome

Imp

ac

t

Relevance

I A O I A O I A O I A O

4

3

2

1 Sustainability

Efficiency/Effectiveness

I A OInput Activity Output

2. Proposed evaluation approach

2.1. Conceptual framework

Transformational change occurs when there is a fundamental shift in strategy resulting in significant changes in the way that an organisation operates, is structured or sets out to satisfy its stakeholders. Transformational change is deep and pervasive, affecting the interplay of strategy, process, people, technology and structure often requiring changes in culture (mind-set and organisational behaviour). This is the essence of WHO’s reform.

Change is often both misunderstood and also underestimated by organisational leaders because capturing its benefits requires insight into how this change impacts almost every other part of the enterprise, from strategy and structure to people, process and technology. At its heart, change is about people. It is about values and culture, the “soft stuff” that many leaders find so chronically hard to address.

The conceptual framework used for the evaluation builds on best practices for transformation programmes. It provides a logical and systematic framework to cover the full scope of the practical challenges in shifting an organisation away from addressing incremental change to implementing lasting transformation.

The framework takes into account the following analytical elements:

1. UNEG Norms and Standards for Evaluation in the UN System

2. Dimensions of a transformation programme

3. Stages of execution of a transformation programme and result-chain or theory of change to drive impact

4. Enablers for a transformation programme

It is summarised in the picture below:

Figure 1: Conceptual framework


PwC 9

2.1.1. UNEG Norms and Standards for Evaluation in the UN System

The evaluation incorporates the UNEG Norms and Standards for Evaluation in the UN System with a particular focus on the criteria of efficiency, effectiveness and sustainability. Effectiveness will be measured against the qualitative benchmark of whether the WHO is undertaking the right steps towards meeting its reform objectives and if the activities laid out in the Implementation plan adequately address the goals of the reform. The core question here is: How well is the WHO doing in reaching its objectives? When assessing the efficiency of WHO’s reform process, we will assess whether the inputs placed towards the activities detailed in the Implementation plan include the right resources, offer the best value for money and are executed according to plan. The core question here is: Is the WHO doing this in the right way in the set timeframe? Finally, the sustainability of the WHO reform program will be addressed by assessing if it is designed to bring about deep institutional and cultural changes at the WHO. To ensure sustainability, all elements of the change vision need to be present, with strong interaction of those responsible for change leadership and management. The core question here is: Is the solution being implemented clear, supported by appropriate tools, reinforcers and ownership that by those who operate the solution in order to bring about institutional and culture change at the WHO?

Each evaluation questions will clearly identify to which of the above mentioned criteria they relate to.

2.1.2. Dimensions of a transformation programme

Transformation encompasses five dimensions which are considered essential for delivering change within an organisation. These dimensions provide a structure for systematically analysing an organisation’s internal working model, structure, processes and culture.

A strategy that provides the strategic direction and vision for the organisation. The criteria to assess this stage mostly relate to relevance and effectiveness.

A clear organisational structure that defines reporting lines between governing bodies and the various levels of the organisation which will directly affect internal governance and accountability systems. The criterion to evaluate this stage is that of effectiveness.

A set of processes, tools and guidelines which are readily available and incorporated into the day to day work of WHO staff for greater transparency. The criterion to assess this stage is that of effectiveness.

A pool of qualified, adequately incentivised human resources with clearly defined roles and terms of references, supporting by an enabling culture. The criterion to assess this stage is that of effectiveness.

Technology, systems and data which support the direction of change. The criterion to assess this stage is that of effectiveness.

The resulting questions for the evaluation are as follows:

Domain Evaluation questions Criteria

Overall Have all five dimensions of transformation been considered in the reform implementation strategy?

Relevance

Effectiveness

Strategy Has the overall reform plan been translated into coherent sub-strategies for each of the 12+1 initiatives under review?

Is the result-chain robust?

Relevance

Effectiveness

Structure Does the reform adequately address factors relating to internal governance, reporting lines between governing bodies, accountabilities, organisational structure, geographical footprint and relationships with external stakeholders?

Effectiveness


PwC 10

Does each reform initiative take the required enabling structure into account?

How is reform cascaded down into the various levels of the organisation (HQ, Region and country)?

Processes Are processes consistent and standard across the organisation, at the HQ, regional and country level?

Are strategies and policies supported by adequate processes and procedures? Have these been documented? Are these fit-for-purpose?

Is the required data available to support the new processes? How is data quality ensured (availability, accuracy, completeness, timeliness and confidentiality)?

Have internal controls and standardisation of processes and data been considered?

Are tools and guidelines readily available and incorporated into the day to day work of WHO staff?

Effectiveness

Efficiency

People Are governance mechanisms and accountability documented and clear?

Do leaders own change efforts, or are they delegated to others?

Are the skill-sets of staff adequate to meet the objective of each reform initiative?

Has a performance management system and incentives been considered to support the targeted changes?

Are the culture and individual behaviours aligned with and supportive of the reform strategy?

Effectiveness

Technology Are IT enablers such as data management and IT infrastructure been considered to support the direction of change?

How does technology enable the reform and its initiatives?

Does the IT strategy appear to be aligned with the reform Implementation plan? Is technology appropriately funded in the context of the reform?

Effectiveness

The strategy dimension shapes the other four, and is often the starting point for a dialogue that results in

identifying improvement opportunities that are connected across the five dimensions. While each reform

initiative will be analysed through the lens of these five dimensions the relevance of the dimensions per reform

initiative will be considered throughout the evaluation.

2.1.3. Stages of execution of a transformation programme and Result-chain to drive impact

In order for change to be affected, a structured approach to change delivery is required. Successful

transformations are typically conducted through five following stages. These include:

Assess and Strategize - used to understand the strategy and business needs that drive the change initiative and envision the future state and expected benefits at a high level; set-up the required programme management arrangements; develop the change vision and define the best-fit change approach and integrated change strategy to support the change initiative. The criteria to assess this stage mostly relate to relevance and effectiveness.

Design - used to design the changes to the organisation's operating model with a focus on optimising the delivery of benefits. The criterion to assess this stage mostly relates to effectiveness.

Construct - used to build and test the changes to the organisation's operating model to ensure they deliver the expected benefits. The criteria to assess this stage mostly relates to effectiveness and efficiency.

Implement - used to successfully implement the operating model changes and plan for subsequent benefits delivery. The criteria to assess this stage mostly relates to efficiency and sustainability.

http://transform.pwc.com/Lifecycle/Method/Transform?a=strategyandassess

http://transform.pwc.com/Lifecycle/Method/Transform?a=design

http://transform.pwc.com/Lifecycle/Method/Transform?a=construct

http://transform.pwc.com/Lifecycle/Method/Transform?a=implement


PwC 11

Operate and Review - used to hand over the operating model changes to business as usual, realise the

expected benefits and set the foundation for continuous improvement, assess the success of change, finalise

the sustainability plan and complete knowledge transfer and hand over. The criteria to assess this stage are

mostly that of sustainability and impact.



Strategize and Assess

Was the current state of affairs analysed and the baseline created? Is there a clear definition of the problem being addressed?

Has there been a process to evaluate and prioritise areas for improvement? How was the prioritisation conducted? How were the criteria for prioritisation defined?

Has a vision/strategy/concept note been developed to address this area of challenge? How has it been endorsed by the organisation?

Relevance

Effectiveness

Design Have all requirements been collected to define what needs to be changed to deliver the outcome and the expected impact?

Have blueprints been developed?

Has a roadmap been produced, defining broad objectives, timeline, responsibilities, KPIs, budget and reporting structure?

What has been the process for approving this roadmap?

Effectiveness

Construct How is the design/general concept translated into details? Is it comprehensive?

How have all relevant stakeholders been engaged in the process?

How are the new ways of operating tested prior to roll-out?

Has a detailed Implementation plan been developed prior to roll out?

Has a specific body/team been set-up to manage and oversee the implementation?

Effectiveness

Efficiency

Implement Which of the activities part of the Implementation plan have started?

Has sufficient training been provided to impacted stakeholders?

How are corrective actions, where required, determined, discussed and agreed?

How is the transition into Business as Usual planned?

Efficiency

Sustainability

Operate and review

How is the knowledge transferred and how is the hand-over performed from reform to business as usual?

Has ownership of the new way of working been transferred?

Has a post-implementation review been completed?

Sustainability

Impact

The completion of each of these stages is a milestone usually materialised by a sign-off on specific deliverables

and outputs, these outputs becoming the input into the activities taking place in the next stage. In order for the

outcome of a specific change initiative to be achieved, a sequence of input/activity/outputs relating to each

stage of the transformation is required.

Also best practice transformation seeks to have a phased delivery of benefits (outcomes), with a focus on

securing early success to build trust and engagement. This means that the component initiatives of a change

programme will move through the five stages at different pace. A challenge then becomes ensuring

dependencies between initiatives are managed. Finally, experience shows that most programs fail at the

construct and implementation phases.

http://transform.pwc.com/Lifecycle/Method/Transform?a=operateandreview


PwC 12

2.1.4. Enablers for a transformation programme

Creating sustainable change requires more than simply focusing on discrete part of the transformation. Firstly change initiatives need to be setup, directed, managed and controlled if they are to deliver their intended benefits through disciplined programme management arrangements in the form of a Programme Management Office (PMO) or equivalent that enables the change initiative to be effectively managed and controlled.

Secondly change management and communication is critical in turning the potential failure of a change initiative into success, accelerating the delivery of benefits, increasing "stickiness" of changes and ensuring their sustainability. This is achieved through the application of a number of change management and communication activities and techniques.

These cross cutting activities span the whole life cycle of the transformation.



Programme Management

Engaged stakeholders

Have stakeholders been identified, assessed and are they managed on an on-going basis throughout the reform process to ensure a high level of commitment?

Are they appropriately informed to contribute to the success of the programme?

Effectiveness

Clear scope Is the scope of the programme well defined, complete, communicated and agreed?

Does the scope support the objectives of the WHO High-level implementation strategy?

Effectiveness

Managed risks and opportunities

Are effective risk identification processes in place and are the key risks mitigated and opportunities taken?

Effectiveness

Delivery-enabling plans

Are the plans in place realistic, achievable, understood and bought into by key stakeholders executing the reform and those affected by it?

Efficiency

Effectiveness

Focused benefits management

Has a realistic business case been developed subjected to an appropriate level of challenge?

Are benefits clearly defined, owned and tracked for program success?

Efficiency

Effectiveness

Sustainability

High performing teams

Is the programme team highly motivated with the right blend of skills and personalities?

Are team members given enough time to deliver on the reform initiatives in addition to their current workload?

Does the organisation support the team to deliver?

Efficiency

Effectiveness

Smart financing Is the budget and associated policies, processes and reporting standards adequate for effective cost estimation and programme financial management?

Efficiency

Integrated suppliers

Have the right partners been selected?

What is the approach for ensuring suppliers understand the reform and help deliver change?

Effectiveness


PwC 13

Active quality management

Has an agreed quality plan been developed based on appropriate standards to monitor and evaluate the quality of the work performed?

Effectiveness

Embedded lifecycle assurance and learning

Has a clear assurance plan been defined which outlines the nature, timing and extent of planned assurance, quality reviews and embeds learning?

Effectiveness

Agile change control

What change control arrangements are in place to manage changes to the scope, requirements, deliverables and intended benefits that impact the project baseline, time or cost?

Efficiency

Effectiveness

Strong governance and reporting

What are the governance arrangements supporting the reform agenda?

How are outputs reported and is reporting targeting all stakeholders to the reform?

Effectiveness

Change Management

Business case, leadership, alignment and action

Does a change vision exist?

Is there agreement on the objectives of the reform?

Is the case for reform actively pushed by leaders?

How are leadership activities executed and monitored?

Relevance

Culture and behaviours

Does the WHO promote a culture conducive to reform?

Sustainability

Communication & stakeholders management

Has a change readiness assessment been carried out?

Is the communication delivered adequate and containing the right type of messages for different stakeholders of the WHO reform?

Are stakeholders identified, mapped and is there a stakeholder management plan?

Has a communication and engagement plan been defined? How is it executed and reported against?

Effectiveness

Sustainability

Change network and change management knowledge transfer

Has a change agent network been identified?

Are change agent network activities taking place on an on-going basis?

How is the change agent network supported?

Effectiveness

Sustainability

Clarity of working practices and organisational design

Has an organisational impact assessment been carried out for the reform overall and for reform initiatives specifically?

Is there clarity on the working practices, process documentation, organisational design and clear policies to ensure WHO staff understand the purpose of the reform and their role?

Effectiveness

Efficiency

Tools and guidance

Are systems, tools and guidance adapted to support the reform? Efficiency

Alignment of reward processes

Are personal objectives aligned?

How are benefits of the reform demonstrated?

How is the informal and formal reward process managed?

Sustainability

Demand from management

What are the mechanisms in place to coach champions and managers?

Does management regularly report on performance?

Sustainability

Training design and delivery

Are trainings set-up to ensure acquisition and transfer of knowledge and new skills?

Sustainability

Monitoring How is the success of the reform defined? Efficiency


PwC 14

How is it assessed?

How were KPIs defined?

What is the process and mechanism in place for monitoring and reporting?

How does the Reform Team respond to findings?

Sustainability Have learnings from pilots (initial implementations) been applied to full implementation?

Are processes, policies and rules being changed where necessary to support the reform?

Sustainability

2.2. Execution approach

2.2.1. Overall approach

In line with the Terms of Reference for the evaluation, we have tailored a structured and systematic approach that lays down the criteria for evaluation, instruments for delivering on the evaluation, and ultimately a socialisation approach culminating with the final report to be presented at the WHO Executive Board 134th Session in January 2014.

A ‘Mobilise’ Phase will allow us to agree with you on the scope, approach, evaluation framework and project management to quickly get started with the assignment. The product of the ‘Mobilise’ Phase consists of this Inception Report and has been developed on the basis of limited consultations with WHO stakeholders to gather initial views on the areas of focus of the evaluation and expectations with regards notably to the spirit in which work should be conducted.

An ‘Evaluate’ Phase will consist of the core of our work. We will review the entire reform process engaged by the WHO from the adequacy of its Implementation plan, to the execution of the activities under the Implementation plan. Section 2.2.3 below provides a detailed explanation of how we intend to conduct the evaluation and against which evaluation standards the reform will be evaluated. Section 3 outlines which instruments (desk review, interviews, observations and staff survey) will be used to support this analysis.

A ‘Synthesize and Report’ Phase will consist of consolidating the findings gathered in the previous phase, conducting a gap analysis, drawing conclusions and recommendations and delivering on the preliminary and final evaluation report. This phase also relates to socialisation of findings and presentation of the final report.

The overall approach is presented in the next page in Figure 2.


PwC 15

A

sses

sE

valu

ate

Synt

hesi

sean

d R

epor

t

Conduct a gap analysis

Assess WHO reform and current status

Confirm scope, objectives and approach for the evaluation

Conduct key stakeholder interviews

Review current practices

Draft reports

Present at Board Meeting

Agree and validate on the project plan

Mob

ilise

Implementation strategy• Quality and linkage to objectives

• Feasibility of timelines

• Resourcing, role and responsibilities

•Change management and program

delivery

Assess Phase• Conclusions from Stage One report

• Gaps in reform areas

• Solutions to fill gaps

Review plan execution• Operating elements

• Activities in progress and outputs

• Challenges and barriers to change

• Monitoring and Performance system

• Inception report• Confirmed scope and evaluation framework

• Preliminary findings and conclusions

• Preliminary report

• Final report

•Presentation to Executive Board

DeliverablesApproach

Ju

ly 20

13

Analysedocumenta-tion and data

Draw conclu-sion from findings

Discuss prelimi-nary results with EMG

Conduct a survey to assess change readiness

Incorpo-rate feedback from EMG

Synthesize and draw conclusion

Develop recommendations

Final reports

Ja

nu

ary 2

014

A substantial number of outputs of the reform are expected to be generated between July and December 2013,

i.e. during the duration of the evaluation. In order to provide timely feedback to the organisation despite a long

lead time to the final evaluation report presentation, our approach draws on real-time evaluation (RTE),

an approach in which a primary objective is to provide feedback in a participatory way in real time (i.e. during

the evaluation fieldwork) to those executing and managing the initiative under evaluation.

The aim is to improve the programme’s design and performance. In order to achieve this, we will contemplate frequent interactions with the Reform team, IOS, the internal Steering Committee for the evaluation and the Evaluation Management Group (EMG). Where deficiencies against evaluation criteria are identified, the evaluators will provide recommendations, tools or methods throughout the assignment that can support unlocking bottlenecks and addressing identified weaknesses.

2.2.2. The Mobilise Phase

This phase has consisted of engaging with the WHO Project Manager for this evaluation, David Webb, Director of IOS; Ian Smith, Advisor to the Director General Office and head of the Reform Team and -given the tight timeline to produce the inception report- a limited number of Member States who were present in Geneva for the Financing Dialogue on 24 June 2013 and available to meet.

The objectives of the meetings were to:

Get an initial understanding of the structure of the reform and a high level understanding of progress made so far since the start of the reform process;

Obtain a clearer picture of the activities in progress and delivered to date;

Figure 2: Overall approach


PwC 16

Identify specific expectations with respect to the spirit in which the evaluation work should be conducted.

We also conducted an initial collection of data and documentation relevant to the reform agenda. A desk review tool has been set-up and will continue to be populated as the assignment advances. It is presented in Annex A.1 of this inception report.

The backbone for the discussions was the ‘WHO reform story’, a synoptic one-pager document, developed by the Reform Team, which summarises the history of the reform process, its vision, objectives, priority areas and outcomes. A snapshot of the 12 initiatives is displayed for reference in Figure 3 below.

Figure 3: WHO Reform Story


PwC 17

The initial discussions with IOS and the Reform Team have allowed us to validate the objectives, expectations, scope, timelines and milestones for this assignment.

The outcome of this short mobilisation phase is contained in this inception report:

A conceptual framework that defines our evaluation standards and guiding questions. This is presented in Section 2.1.

A refined methodology to execute against the terms of reference of the evaluation. This is presented in section 2.2 below.

A data and information collection approach, including documents to be reviewed, interviews to be carried out, meetings to attend as observers and a staff survey to be conducted. This is presented in Section 3 below.

A schedule for the work execution. This is presented in Section 4.

Clarified roles and responsibilities for the evaluation. This is presented in Section 5. This report is now shared with the Evaluation Management Group (EMG) for feedback and endorsement during the week of the 15th July 2013.

2.2.3. The Evaluate Phase

Based on the consultation of the ‘Mobilise’ Phase and the conceptual framework presented in Section 2.1, we have refined the approach suggested in PwC’s initial proposal. It presents the details of how the evaluation will be executed and the tools or accelerators that will enable us to conclude on each step defined. The methodological approach will be broken down into two steps corresponding to the two objectives of the review:

Assess the status of WHO reform, and

Evaluate whether conditions for success of the reform are met. These two steps will drive the ‘Evaluate’ Phase and will be described in the below section. Each of the reform outcome defined in the Implementation plan will be evaluated against evaluation standards (derived from the Transformation dimensions and stages) considered essential to produce sustainable change within an organisation. We will also consider international best practices in the relevant reform areas, i.e. governance, oversight, monitoring and evaluation etc. that will act as standards of quality for each reform initiative. The evaluation will be guided by the meta-evaluation and overarching questions listed below, and as presented

in Section 3.2.2 of the Terms of Reference for this evaluation.

Principal evaluation question: - Is the WHO properly prepared to implement the reform agenda? Overarching questions: - Are the necessary conditions present to ensure an appropriate level of organizational preparedness

(institutional framework: structures, roles, instruments, procedures and guidelines, support systems for change process, communication, incentives)?

- What has been the progress made by the WHO in implementing the reform since 2011 and what has been the effectiveness, efficiency and sustainability of the measures undertaken?

- What have been the barriers to change and key bottlenecks that have slowed down the process of reform? Where issues have arisen have they been understood and tackled through appropriate actions?


PwC 18

2.2.3.1. Assess the Status of WHO reform

Assess status of action taken on Stage 1 recommendations

At this stage, we will review the recommendations issued by the Stage 1 Evaluation and assess if and how they have been incorporated into the Implementation plan. We will:

Review the process of prioritising the Stage 1 recommendations and assessing their feasibility of implementation;

Compare the versions of the Implementation plan and log the inclusions/exclusions of the Stage 1 recommendations;

Advise on Stage 1 recommendations to be considered by the Reform Team for presentation to the Member States.

This will be achieved through:

Desktop review (WHO meeting notes, progress updates, Implementation plans);

Interviews with the WHO Reform Team;

Interviews with Business Owners and for each reform output.

Assess progress made to date on the WHO reform Implementation plan

This stage will consist of assessing the progress made in each of the 12+1 reform initiatives against the expected

outcomes, outputs and deliverables presented in the Implementation plan. We will:

Review each outcome, output and deliverable and define at which stage of the transformation process

each reform initiative sits (i.e. Strategize & Assess, Design, Construct, Implement, Operate & Review)

presented in Section 2.1.3;

Having defined the level of maturity of the transformation, assess if the relevant tools and mechanisms

are in place for achieving the intended targets;

Assess the interdependencies of the reform initiatives affecting their timely execution;

Define gaps and bottlenecks in the effective implementation while also highlighting good practices;

Recommend international best practices (in the area of oversight, monitoring and evaluation, human

resources, accountability and risk management and communication) most relevant to the reform

initiatives under review to unlock bottlenecks.


Desktop review;


Interviews with Business Owners and for each reform output.

Following preliminary and limited discussions with the Reform Team on the status of each reform output, we have created an initial status of the progress made so far towards reaching the objectives of the Implementation plan. In a nutshell, this tool was developed considering three analytical elements:

The reform agenda was broken down by reform initiative, outcome and related outputs;

Each reform output was assessed against each transformation stage as described in the conceptual framework in Section 2.1.3 to understand its level of advancement;

A progress status was identified per stage and per output and rated 1) completed, 2) in progress or 3) initiated.

This tool will be the accelerator supporting the analysis at this stage.


PwC 19

In the tool presented in Figure 4, each reform output has been assessed against the definitions of the stages of advancement of a reform process and developed into a wheel similar to the one the ‘WHO Reform story’. This initial assessment will be challenged and reviewed with a broader set of stakeholders during the course of the evaluation. Our future findings will allow us to update and refine this ‘WHO wheel’. Our initial findings show us that levels of advancement in terms of execution vary from one reform initiative to the other and within a reform initiative the status of progress of each outcome and deliverables also varies. Based on interviews and high level document review and through preliminary consultation with the Reform Team, it shows that most progress has been achieved in the programmatic initiative (1.1) while the initiatives linked to stakeholder engagement (2.5), streamlining national reporting (2.4) and providing effective support to Members States (3.1) are at the initial stages of the reform. From our initial consultations, advancing on the financing and the human resources initiatives of the reform are considered important for the success of the WHO reform.

Assess & Strategise

Stage

Design

Construct

Implement

Operate & Review

Not initiated

In progress

Completed

Status

Legend

L&D

Conflict of interestManagerial accountability,

transparency and risk

management

Figure 4: WHO wheel and preliminary status on reform


PwC 20

This stage of the evaluation will aim to answer key questions presented in Section 3.2.2 of the Terms of

Reference for this evaluation which include:

Does the Implementation plan adequately address all the elements of delivering on effective governance To

what degree have appropriate strategies, including resource mobilisation and allocation, and corporate strategies such as human resources management and gender mainstreaming, been identified to implement the reform proposals?

Assess the completeness of the WHO reform Implementation plan At this stage, we will review the adequacy and comprehensiveness of the results chain and theory of change presented in the Implementation plan and assess if this is adequately translated into reform operational plans. To achieve this we will:

Assess if all dimensions of a transformation, i.e. Strategy, Structure, Processes, People and Technology have been addressed by the Implementation plan (see Section 2.1.2) and what type of outputs are defined per dimension;

Assess the robustness of the results chain and assumptions underlying the theory of change- this will be done by reviewing the causal linkages between the inputs, outputs, outcomes and deliverables and contextual/external factors which may affect the achievement of the targets. We will notably verify if outputs are specified for all stages of transformation;

Assess the translation of the Implementation plan into operational plans for each reform initiative;

Assess if the completeness of the inputs (consider all relevant dimensions) and are adequate to achieve outputs in the set timeframe;

Identify gaps in the theory of change and in the operational plans;

Assess the robustness of the monitoring framework and accountability systems in place to ensure reporting on the results of the Implementation plan.


Desktop review;


Interviews with Business Owners and for each reform output;

Interviews with WHO Senior Management Team and Member States.

This stage of the evaluation will aim to answer key questions presented in Section 3.2.2 of the Terms of

Reference for this evaluation which include:

Does the Implementation plan adequately address all the elements of delivering on effective governance

and change- strategy, structure, process, people, technology? Is it detailed and comprehensive enough to allow for adequate implementation? Is a new design proposed for one or all of the elements? If so, how adequate and feasible is the new design in terms of execution and timelines?

Are indicators and timelines sufficient, feasible and appropriate to effectively measure (monitor and evaluate) performance and report on the desired outcomes of the reform process?

2.2.3.2. Evaluate whether conditions for success of the reform are met

This second step of the Evaluate Phase is linked to the second objective of the evaluation and contemplates the

enablers of a transformation programme. Firstly, we will assess the current capacity of the Reform Team and

other actors in charge of the reform and review how they currently manage the reform agenda. Concurrently,

we will assess if the elements of successful change management are present. This aspect of the evaluation will

be supported by the elements and questions presented as part of Section 2.1.4.


PwC 21

Assess programme management

At this stage, we will

Evaluate the program management component of the reform, as presented in Section 2.1.4;

Review the project management and excel tools in use by the Reform Team for monitoring reform results;

Assess the adequacy of the stakeholder management strategy, risk management process, reporting/ monitoring frameworks and financial management;

Conduct a gap analysis and identify the critical enablers that are missing for the reform to have in depth and transformative results


Review of project management tools;



Assess change management

At this stage, we will:

Evaluate if the change management practices currently in place are supported by the adequate balance of leadership, communication, availability of tools, trainings and monitoring of change, as presented in Section 2.1.4;

Review the structure of the communication strategy for reform and its implementation;

Assess WHO’s culture alignment with the change vision;

Review the content of change leadership and network activities and assess their adequacy;

Conduct a gap analysis and identify the critical enablers that are missing for the reform to have in depth and transformative results


Desktop review;



Interviews with WHO Senior Management Team and Member States;

Representatives from Regions and Country offices;

Staff survey.

Assess staff perception and readiness for change

In order to cross-check interview and assess whether conditions for successful change implementation are

present, we complement the above assessment with a change readiness assessment focusing on staff feedback

and perception. This will allow us to test how effective management approach to change has been to date.

The instrument to achieve this will be an electronic survey sent to all staff. It is described in Section 3.4 below.

This stage of the evaluation will also contemplate key questions presented in Section 3.2.2 of the Terms of

Reference for this evaluation including:

What is the efficacy of internal and external instruments and modalities to implement the reform proposals, including related risk management plans and risk mitigation actions?

To what degree is there coherence in the governance mechanisms to inform, monitor and evaluate the implementation of the WHO reform package? To which extent are these mechanisms and processes adequate to prioritise the various components of the reform proposal?


PwC 22

What are the specific strategies to address change management? How well are the pertinent issues addressed?

2.2.4. The Synthesize and Report Phase

This phase of the evaluation relates to the finalisation of the evaluation preliminary and final reports,

their dissemination and the broader socialisation of the evaluation findings and recommendations.

The following is planned:

Drafting of the preliminary report;

Discussion of findings and recommendations with WHO Steering Committee and WHO Senior

Management Team to socialise preliminary findings, test conclusions and adjust as required prior to

submitting the preliminary evaluation report to the EMG;

Distribution of preliminary report to the EMG on 2 September 2013 for review, feedback and guidance;

Update of report based on the above EMG feedback received by 30 September 2013 and finalisation;

Distribution of the final report to the EMG and PBAC in October 2013;

Preparation and finalisation of summary presentation of findings;

Presentation to the Audit Committee IEOAC at the November session;

Presentation to PBAC at their January meeting;

Report of the PBAC to the Executive Board in January 2014.

Ad-hoc sessions will be organised with the Reform Team and Business owners as questions arise. We will have

regular contact with the Project Manager, David Webb, the Reform Team, the Steering Committee, the EMG

and senior WHO stakeholders throughout the course of the assignment.

Close proximity with the WHO will ensure that our evaluation results meet the needs of the Executive Board.


PwC 23

3. Data collection instruments

Four primary means of data collection will be used to collect data pertaining to the terms of reference of the evaluation:

Desk review

Interviews with key stakeholders

Observation

Staff survey on the reform Details are provided in the sections below.

3.1. Desk review

We will collect data through the review of key documentation and secondary evidence on the WHO reform process such as:

The Independent evaluation report: Stage One, External Auditor WHO

WHO reform documentation including Board minutes

Any project documentation linked to the reform agenda

Reports from the UN Joint Inspection Unit including the ‘Decentralization of Organisations within the United Nations System- Part III: the World Health Organisation’ and the ‘Review of management and administration in the WHO’

Programme and Budget for the Biennium

Detailed organisational charts

Internal strategies including HR policies

The list of current available documents can be found in Annex A.1.

We will use a desk review tool to be populated as the assignment advances, in collaboration with the WHO

counterparts. All documents will be made available on the PwC secure workspace that is specially created by the

Independent Evaluation Team for sharing documents with the WHO focal points.

3.2. Interviews

All interviews will be conducted by a minimum of two members of the Independent Evaluation Team, thus ensuring adequate capturing and validation of findings. Data from interviews and meetings will be uploaded into a database and mapped against each area of the reform in order to structure evidence effectively for analysis purposes. Table 1 below presents the list of interviewees.


PwC 24

Table 1: List of interviewees

High-level WHO Secretariat HQ

Margaret Chan, Director General

Anarfi (AB) Asamoa-Baah, Deputy Director-General, HQ/ODG Office of the Director-General

Andrew Cassels, Director (Strategy), HQ/ODG Office of the Director-General

Hans Troedsson, Executive Director, HQ/ODG Office of the Director-General

Bersabel Ephrem, Senior Adviser to the ADG/GMG, HQ/GMA GMG ADGO Office of the Assistant DG

Dr. Mohamed Jama, Assistant Director-General, HQ/GMA GMG ADGO Office of the Assistant DG

Regional

Dr Luis G. Sambo, Regional Director, AFRO

Dr Carissa Etienne, Regional Director, AMRO

Ms Zsuzsanna Jacab, Regional Director, EURO

Dr Samlee Plianbangchang, Regional Director, SEARO

Dr. Shin, Regional Director, WPRO

Dr. Ala Alwan, Regional Director, EMRO National:

TBD

Business Owners:

Najeeb Al Shorbaji, Director, HQ/KMS Knowledge Management and Sharing

Gian Luca Burci, Legal Counsel, HQ/LEG Office of the Legal Counsel

Christy Feig, Director, HQ/DCO Department of Communications

Nicholas Jeffreys, Comptroller & Director, HQ/FNM Department of Finance

Daniel Lopez-Acuna, Advisor to Director-General, HQ/ODG Office of the Director-General

Elil Renganathan, Director, HQ/PRP Planning Resource Coordination and Perf Monitoring

Mari-Andree Romisch, Director, HQ/CCO Country Collaboration

Cecilia Rose-Oduyemi, HQ/GBS Dept for Governing Bodies and External Relations

Mahen Sandrasagren, Coordinator/ acting Director, HQ/HRQ HQ HR Management

Gaudenz Silberschmidt, Senior Advisor to the Office of the Director-General, HQ/ODG Office of the Director-Genera

Ian Smith, Advisor to Director-General, HQ/ODG Office of the Director-General

David Webb, Director HQ/IOS Office of Internal Oversight Services

Additional informants of reform initiatives will take place on an as-needed basis

Member States

Gabrielle Jacob, First Secretary (Health), Permanent Mission of Ireland

Peter Mamacos, Multilateral Branch Chief, US Department of Health and Human Services

Colin Mciff, Health Attaché, US Permanent Mission, Geneva

Cath Patterson, Minister Counsellor (Health), Australian Permanent Mission to the UN, Geneva

Magali Girod, Policy Adviser, , UK Mission to the United Nations in Geneva

John Stuppel, Health and UN Cross Systems Team Leader, DFID, UK

Mr B. Kuemmel, Adviser, Federal Ministry of Health, Germany

Elissa Golberg, Ambassador Canada

Onoda Tomoko, Second Secretary Japan

Jan Knutson, Ambassador, Sweden

Sverre Berg Lutnaes, Counsellor, Norway

Maria Luisa Escorel De Moraes, Minister Counsellor, Brazil

Alexey Kulikov, Third Secretary, Russia

Anastacia Osundwa, Kenya

Other Geneva-based Members States representing various groupings- TBD

EMG

Ms Jane Halton, Australia (Chair)

Ms Precious Matsoso, South Africa

Dra Zelibeth Valverde, Panama

Prof. Mohammad Hossein Nicknam, Islamic Republic of Iran

Prof. Ogtay Shiraliyev, Azerbaijan

Prof. Pe Thet Khin, Myanmar

Dr Sukai Prom-Jackson, External, Member (Inspector, Joint Inspection Unit)

Steering Committee

Matshidiso Moeti, Director, WHO, AFRO

Jon Kim Andrus, Deputy Director, WHO, AMRO

Samir Ben Yahmed, Director, WHO, EMRO

Imre Hollo, Director, WHO, EURO

Herbert Tennakoon, Assistant Regional Director, WHO, SEARO

Tieru Han, Director, WHO, WPRO


PwC 25

Interviews will be conducted in a phased approach, starting with Business Owners, EMG and Steering Committee members and WHO HQ and Regional Directors. Member States representatives will be interviewed throughout the interview period pending on availabilities.

3.3. Observation

Another mean for data collection will consist of attending specific WHO meetings as observers. The following meetings are considered as important avenues for better understanding the context of the reform agenda and governance structure.

Financing Dialogue, Taking steps towards a fully-funded programme budget 2014-2015, 24 June 2013.

The Independent Expert Oversight Advisory Committee 10th meeting, Sessions on WHO reform and Financing Dialogue, 3 July 2013.

Regional Director’s Program Management (DPM) network meeting, Sessions on national reporting, 9 July 2013.

3.4. Survey

After drawing preliminary conclusions from the interviews described above, the Independent Evaluation Team recommends undertaking a Change Readiness Assessment which covers key elements of change management and will aim to test our conclusions with a wider audience using an electronic survey. This will allow us to test how staff perceives the reform process, how projects are heading, and if they feel adequately equipped to implement them. The survey will be an opportunity to reach out to WHO regional and country offices. By using a secure survey platform, we will administer the survey on an anonymous basis to all WHO staff for a

period of three weeks. Results will be collected by the Independent Evaluation Team. We will develop survey

questions/statements that can be easily rated, from a scale of 1 to 5, for the purpose of the analyses.

Appendix A-2 presents an initial list of questions that we intend to ask, against the following change

management themes:

Awareness of WHO reform;

Impact of the reform on the person;

Impact of the reform on the Workplace;

Communication on the reform;

Engagement with the reform process;

Leadership;

Evidence of Impact of the Reform;

Organisational Climate.


PwC 26

3.5. Methodology Limitations

In defining the conceptual framework and execution methodology, the following limitations have been

identified and mitigated:

Domain Limitation Mitigation

Conceptual Framework

The conceptual framework, although logically sound, cannot be used as straight jacket – it needs to be fined tuned to every situation

Initial framework has been fined-tuned to WHO specific context

Evaluators will assess whether there are elements of the framework that do not apply to a specific reform initiative

Execution approach

Moving picture: the evaluation is taking place as reform progresses. Situation may be changed between the initial interviews and the final evaluation

Be clear that we are assessing the reform as it stand early July 2013. Should elements change, these will be mentioned in the final report.

WHO Management to also issue their own report on progress since the evaluation for the January PBAC meeting

Timing: the evaluation is planned to be performed during July and August 2013, a period when most northern hemisphere stakeholders are likely to be away and may not be available to the Independent Evaluation Team. This could create biases.

Early communication to all stakeholders to be interviewed

Extension of staff survey period over July and August

Timing: reform is mostly in the design/implementation stages, which do not allow to evaluate the overall impact of the Reform

Clearly exclude assessment of impact from evaluation scope

Consider Stage 3 evaluation in 3-4 years

Data collection instruments

Desktop review could provide a false sense of comfort that reform is happening because documented

Challenge the status of reform in interviews. Cross check with result of staff survey

Interviews could be instrumentalised to provide a biased picture of current state (positive or negative)

Clear assessment criteria

Cross-check responses from various groups to the same questions

Cross-check with other instruments

Survey may not be statistically meaningful and only provides “perception”

Ensuring adequate sampling and response rate through reminders

Focus the survey on the very perception (“I feel”, “I think”), as opposed to looking for factual elements

Ensure translation in WHO working languages

Ensure anonymity

Observation. The presence of the observer may result in the events being observed, not being conducted as they would otherwise be. Also given the timeframe we can only observe events taking place in the July-August period.

Cross-check with other instruments

Be discrete during events under observation


PwC 27

4. Project Management Plan

As described in the Gantt chart, the table below sets out the expected deliverables, milestones and feedback

expected by the EMG.

Deliverables Deadlines Inception report 8 July 2013 EMG feedback on inception report 15 July 2013 Preliminary report 2 September 2013 Feedback on preliminary report 30 September 2013 Final Evaluation report 9 October 2013 Summary PowerPoint presentation 20 December 2013 Presentation to 134th session of the Executive Board 20 January 2013 (TBC)

We have planned for monthly EMG meetings which we suggest should include discussion of initial findings,

approach and method, survey, key stakeholder involvement, reporting and any other important project

elements. We assume that EMG feedback on the inception report and preliminary report will be provided by 15

July 2013 and 30 September 2013 respectively. We also understand that the date proposed for the 134th session

of the Executive Board is 20 January 2013, date at which we will provide our presentation of the evaluation

findings. The IOS Secretariat will be arranging and scheduling meetings.

We have planned bi-monthly meetings with the Steering Committee. These meetings will cover reviews of

evaluation tools and protocols, socialising findings and offer technical guidance. The IOS Secretariat will be

arranging and scheduling meetings.

We have planned bi-weekly meetings or phone calls with IOS to discuss updates on the assignment and resolve

any operational issue.

We have planned for regular interactions with the Reform Team to ensure timely scheduling of interviews with

key informants.

The Gantt chart below presents the detailed workplan that will guide the execution of the Evaluation work.


PwC 28

2013 2014

Planning (# weeks) 24/06 01/07 08/07 15/07 22/07 29/07 05/08 12/08 19/08 26/08 02/09 09/09 16/09 23/09 30/09 07/10 14/10 21/10 16/12 23/12 30/12 06/01 13/01 20/01

Mobilise 2 weeks

1. Collect information 1 week

1.1 Collect data

2. Assess WHO reform status (nb Excel model and notes) 1 week

2.1 Confirm scope and objectives

2.2 Conduct initial interviews with key stakeholders

2.3 Analyse progress made towards 12 reform areas

2.4 Review key documentation

3. Reporting 1 week

3.1 Define detailed approach to evaluation

3.2 Produce inception report

Evaluate 6 weeks

4. Conduct key stakeholders interviews 4 weeks

4.1 Plan and organize interviews

4.2 Conduct interviews

5. Desk review 1 week

5.1 Collect data

5.2 Review key documentation

6. Conduct survey 4 weeks

6.1 Define scope

6.2 Plan survey

6.3 Roll-out survey

6.4 Analyse results

Synthesize & Report 12 weeks

7. Data analysis 4 weeks

7.1 Conduct gap analysis

7.2 Synthesize findings and results from survey

8. Preliminary report 3 weeks

8.1 Report writing

9. Inclusion of EMG feedback 2 weeks

10. Final report 2 week

10.1 Report writing

11. Socialisation & Presentation at Board meeting 3 weeks

11.1 Presentation writing

11.2 Socialisation

11.3 Presentation at Board meeting

TOTAL 20 weeks

Key milestones

Deliverables

Deadlines

Resource management

EMG

Steering Committee

IOS

Reform Team

Final Pres at

EB

20.01.2014

Inception

report

EMG

feedback

15.07.2013

Preliminary

report

02.09.2013

EMG

feedback

30.09.2013

Final report

09.10.2013

Final Pres

20.12.201308.07.2013

Key risk associated with availability of stakeholders during the months of July-August 2013


PwC 29

WHA

MemberStates

EB

DGO IOSEvaluation

Management Group

Steering

Committee

WHO

Secretariat

Oversight

of Stage 2 Evaluation

Engagement Partner

Project

Director

Independent

Evaluation Team

Governance

Reform

TeamDG/DDG

Regional

Directors(6)

RegionalPolicy Group

Business

Owners

Consultant

SME

5. Roles and responsibilities

5.1. Project management structure

Gill Sivyer is the responsible Partner for this evaluation. She is at the disposal of the EMG and the IOS Project

Manager for any remarks or suggestions that they may have and will ensure the quality of our services and the

supervision of the Independent Evaluation Team. She will also take part in key meetings.

The main PwC contact for any project work on a day to day basis is Antoine Berthaut, Project Director.

Antoine Berthaut is supported by:

Josephine Pallandt, Senior Manager, who will be responsible for governance aspects of the evaluation;

Sarah Des Rosiers, Senior Consultant, who has a background in public health, and who will work under the guidance of Antoine and Josephine to conduct the analysis.

The team has adequate gender balance and cultural diversity representation (American, French, Dutch,

Canado-Haitian).

Additional consultants from PwC will be mobilised on an as needed-basis (e.g. to address specific technical

questions to complement the core team described above). An external Subcontractor maybe involved as Subject

Matter Expert, prior to approval by WHO.

The below figure demonstrates the linkages between the Independent Evaluation Team and the WHO entities

to be involved in the Stage 2 Evaluation as well as the planned interactions between both entities.

Figure 5: WHO Stage 2 reform organigram


PwC 30

5.2. Role of WHO

The below table presents the roles and responsibilities of the WHO counterparts described above and sets out

who will be involved with either the day-to-day management of the Independent Evaluation Team, the

oversight of the execution of the evaluation or the facilitation of the key informant interviews throughout the

course of the evaluation.

WHO role Nature of involvement Estimated workload

IOS/Project Manager Mr David Webb, Director, IOS Dr Maria Santamaria, Coordinator Evaluation, IOS

Day to day oversight of the Independent Evaluation Team and responsible for all aspects concerning contracting

Facilitation of access to interviewees and documentation

Focal point for progress updates and provide input into evaluation approach

Focal point for providing technical oversight over the assignment and focal point for progress updates

Advising of key informants and facilitating introductions

Facilitate, organise and participate in EMG and steering committee meetings

2 days/week

Reform Team Ian Smith, Advisor to the Office of the Director-General Gaudenz Silberschmidt, Senior Advisor to the Office of the Director-General Ms Linda Larsson (WHO Reform documentation) Ms Michelle Dean (logistics arrangements)

Provide documentation on the WHO reform

Advise on list of key informants

Provide logistical arrangements involved in the organisation of interviews and focused group meetings

Provide ad-hoc explanations related to the WHO reform

Participate in assessment of WHO reform status

Support definition of survey participants

1-2 days/week

Evaluation Management Group Ms Jane Halton, Australia (Chair) Ms Precious Matsoso, South Africa Dra Zelibeth Valverde, Panama Prof. Mohammad Hossein Nicknam, Islamic Republic of Iran Prof. Ogtay Shiraliyev, Azerbaijan Prof. Pe Thet Khin, Myanmar Dr Sukai Prom-Jackson, External Member (Inspector, Joint Inspection Unit)

Direct the evaluation work and make key decisions on the evaluation work

Ensure the project's scope aligns with the Terms of Reference for the evaluation and represent Member States interests

Provide direction and guidance as work progresses to ensure objectives are met and risks mitigated

Provide technical guidance on the evaluation approach and quality enhancement

Validate findings and discuss recommendations

Validate and approve key deliverables

Ensure quality and timeliness in decision-making

0.5 day every 2 weeks

Steering committee Dr Anarfi Asamoa-Baah, DDG Hans Troedsson, Executive Director, Office of the Director-General Ian Smith, Reform Team David Webb, IOS Mohamed Jama, Assistant Director General, General Management 6 Regional Nominees

Follow-up on the execution of the evaluation and act as a sounding board

Guide and provide feedback on data collection and tools

Address technical questions

Review initial findings and facilitate the socialisation of findings

Ensure transfer of knowledge of the evaluation work within the WHO

0.5 day/week


PwC 31

5.3. Quality assurance

Quality assurance is integrated throughout the evaluation and is based on the four eyes principle. The following checkpoints will be built into the key phases of the evaluation:

Mobilise: Review of evaluation terms of reference by Gill Sivyer and the respective subject matter expert.

Evaluate: - review of the detailed work programme to ensure appropriate scope and depth of coverage - review of workpapers at completion of fieldwork to ensure all necessary steps have been completed

and documented - review of issues raised to ensure consistency of evaluation, justification and relevance prior to

discussion and validation with evaluees.

Reporting: Review of the written report by both Gill Sivyer and PwC internal quality assurance to ensure the evaluation meets the requirements of WHO and to ensure that all issues are fact based and recommendations are practical.

Closure: At the conclusion of the evaluation a formal satisfaction review will be undertaken to ensure that

lessons learned are identified and built into subsequent reviews of a similar nature."


PwC 32

Appendix A. - Data collection

A.1. Desk review tool

The list of documents available in this table has been provided largely by the Reform Team and will be

populated during the course of the evaluation as we enter into the ‘Evaluate’ Phase.

Reform Agenda Documentation

Programmatic reform

Programmatic priorities Draft twelfth general programme of work, SIXTY-SIXTH WORLD HEALTH ASSEMBLY,19 April PROPOSED PROGRAMME BUDGET 2014–2015, SIXTY-SIXTH WORLD HEALTH ASSEMBLY,19 April Twelfth General Programme of Work, 2014–2019, SIXTY-SIXTH WORLD HEALTH ASSEMBLY, 24 May 2013 Programme budget 2014–2015, SIXTY-SIXTH WORLD HEALTH ASSEMBLY, 24 May 2013

Governance reform

Oversight by the governing bodies

Governing Body minutes of meetings and other relevant gatherings and discussions

Revised terms of reference for the Programme, Budget and Administration Committee of the Executive Board, EB131.R2, 28 May 2012

Harmonisation and alignment of governance processes

Project governance documentation, governance Principles , project pans and charters, project agreements, project minutes

Enhanced strategic decision-making by governing bodies

Agenda’s and minutes of relevant governing body meetings, founding documents, charters.

Streamlined reporting of and communication with Member States

Current reports and communications with Member States about the reform initiatives, relevant outputs and tools, Reform team & other key stakeholder meeting minutes

Strengthened effective engagement with other stakeholders

Current engagement initiatives /plans / terms of references, minutes of relevant external stakeholders meetings, outputs and tools used to date

Managerial reform

Effective technical and policy support for all Member States

Review of management, administration and decentralisation in the World Health Organization, report by the Joint Inspection Unit, EB132/5 Add.6, 4 January 2013 Report of the Taskforce on the roles and functions of the three levels of WHO

Staffing matched to needs at all levels of the Organisation

Amendments to the Staff Regulations and Staff Rules1, EB132/40, 11 January 2013

Financing and resource allocation aligned with priorities

Amendments to the Financial Regulations and Financial Rules, A66/57, 20 May 2013 WHO reform, Financing of WHO, A66/48, 13 May 2013 WHO reform, Financing of WHO, A66/48, 20 May 2013

Managerial accountability, transparency and risk management


PwC 33

Strengthened culture of evaluation

Formal evaluation policy, B131

Improved strategic communications

Change management

ADG meeting notes ADG Meeting notes, 2012/2013

Communications Change at WHO, Newsletter on Reform, November 2012 and April 2013 WHO reform Engagement and Communication Strategy WHO reform story

GPG GPG meeting and retreat notes, 212/2013

IEOAC IEOAC Annual Report 2012 IEOAC Agenda, Feb, July, Nov 2012, March 2013 IEOAC, July-November 2012 PBAC IEOAC 7th session final report, 31 July 2012 IEOAC, Report of the Sixth Meeting of the Independent Expert Oversight Advisory, February 2013

Project Management Tool Reform tool

Reporting to MS WHO reform High-level implementation plan and report, 20 May 2013 Implementation of WHO reform, 2012, High-level implementation plan, January 2013

Other documentation

External evaluation of the WHO reform

WHO reform, Independent evaluation report: stage one, 18 May 2012 GAO report, World Health Organization, 2012

UN agencies reform

Best practices in reform areas


PwC 34

A.2. Staff survey questionnaire (initial list of questions to

be reviewed)

For each of the statements below, please indicate whether you:

1) Strongly disagree;

2) Disagree;

3) Neither agree nor disagree;

4) Agree;

5) Strongly agree.

Themes # Survey questions

Awareness

1 I am aware of the WHO Reform

2 I understand the vision, objectives and planned outcomes of the WHO Reform

3 I understand clearly what our organisation will be like after the WHO Reform

4 If our organisation does not make this change, we will be affected negatively in the long term

5 I am confident that the WHO Reform will be implemented successfully

6 I feel there is a clear action plan for implementing the changes associated with the WHO Reform

Personal

7 I will personally benefit from the WHO Reform

8 I am motivated to implement reform initiatives in my workplace

9 I understand how the WHO Reform will impact my work activities

10 I understand my role and what I can do in my job to make the Reform a success

11 I understand the skills and knowledge that I will need to demonstrate in order to be successful in the new organisation

12 I am confident I will receive the training I need in order to effectively do my job

13 I understand the benefits that the initiative will bring to my work area

Workplace

14 I think there is a manageable level of change in our work unit

15 My Workplace has embraced the changes

16 There is sufficient cross functional co-operation for this change to succeed


PwC 35

Communication

17 I feel well informed and kept up to date on relevant WHO Reform initiatives

18 I know where to find information about the WHO Reform

19 I usually learn about organisational changes through formal communication channels (e.g., newsletters, presentations, etc.) rather than through informal communication channels (e.g., contacts, “grapevine”, etc.)

20 There are adequate ways for me to inquire, contribute or voice my concerns about to the Reform

21 Communication on reform between HQ and the regions is effective

22 Through which modes of communication would you prefer receiving information about the Reform? [list to be provided]

Engagement 23 I feel involved in the proposed changes

24 I have received timely and adequate guidance and support from my superiors to help me drive reform

Leadership

25 Senior Management considers the impact of change on the field

26 I have participated in discussion with one or more of my supervisors about the WHO Reform

27 Leaders, superiors and managers have the change skills necessary for this WHO Reform

28 My superiors support the WHO Reform and are positive about its arrival

29 My superiors are adopting new processes and contributing to the WHO Reform

30 I think that adequate support across the WHO and resources will be provided to accomplish the WHO Reform

Evidence of Impact

31 I have seen evidence of reform initiatives being implemented

32 I have seen positive results from the WHO reform

Climate 33 The climate for change is positive and supportive in the organisation


PwC 36

Appendix B. - Preliminary and Final report

B.1. Outline of the preliminary and final report

Table of Content

1. Executive Summary

2. Introduction

3. Scope of Evaluation and Questions

4. Evaluation Areas

4.1 Assess the WHO reform

4.1.1 Key findings

4.1.1.1 Action taken on Stage 1 Recommendations

4.1.1.2 Assess progress made to date on the WHO reform Implementation plan

4.1.1.3 Assess the completeness of the WHO reform Implementation plan

4.1.2 Challenges and Opportunities

4.1.3 Gap analysis

4.1.4 Recommendations

4.2 Evaluate whether conditions for success of the reform are met

4.2.1 Key findings (Programme and Change Management)

4.2.2 Results of survey

4.2.3 Challenges and Opportunities

4.2.4 Gap analysis

4.2.5 Recommendations

5. Suggested Action Plan

6. Conclusion and Way forward

7. Appendix- Assessment against evaluation questions


PwC 37

Appendix C. - UNEG Quality Checklist

C.1. UNEG Quality Checklist for evaluation Terms of

reference and Inception Reports

Evaluation Title: Stage 2 Evaluation of WHO Reform

Commissioning Office: Executive Board

1. Evaluation Purpose

1.0 The Terms Of Reference specifies the purpose of the evaluation and how it will be used.

Section 1.3

1.1 The TOR references the mandate for the conduct of the evaluation. Section 1.2

1.2 The purpose of the evaluation identified in the TOR clearly states why the evaluation is being done, including justification for why it is being done at this time.

Section 1.3

1.3 The TOR identifies the primary and secondary audiences for the evaluation and how the evaluation will be useful.

Section 1.3 Introduction

2. Evaluation Objectives2

2.0 The Terms Of Reference includes clearly defined, relevant and feasible objectives. Section 1.3

2.1 The evaluation objective(s) clearly follow from the overall purpose of the evaluation. Section 1.3

2.2 The TOR evaluation objectives are realistic and achievable, in light of the information that can be collected in the context of the undertaking.

Section 1.3

3. Evaluation Context

3.0 The Terms Of Reference includes sufficient and relevant contextual information. Section 1.2

3.1 The TOR adequately describes the particular political, programmatic and governance environment in which the evaluation will be taking place. For example, the most relevant aspects of the economic, social and political context are described.

Section 1.2

3.2 The TOR adequately describes the most relevant programmatic and/or thematic aspects relevant to the evaluation.

Section 1.2

4. Evaluation Scope

4.0 The Terms of Reference includes the scope of the evaluation. Section 1.3

4.1 The TOR explicitly and clearly defines what will and will not be covered, including, for example, the timeframe, phase in the project and/or geographical area to be covered by the evaluation.


4.2 The scope of the evaluation is adequate to meet the stated evaluation objective(s). Yes.

4.3 The scope of the evaluation is feasible given resources and time considerations. Yes – providing stakeholders availability for interviews.

5. Evaluation Criteria

5.0 The Terms of Reference specifies the criteria that will be utilised to guide the evaluation.

Section 2.1, notably 2.1.1

5.1 The TOR specifies the evaluation criteria against which the subject to be evaluated will be assessed, including, for example, relevance, efficiency, effectiveness, impact and/or sustainability.

Section 2.1, notably 2.1.1

5.2 The TOR spells out any additional criteria of relevance to the particular type of evaluation being undertaken, such as evaluations of development, humanitarian

Section 2.1, notably 2.1.2,


PwC 38

response, and normative programmes. 2.1.3 and 2.1.4

5.3 The scope of the evaluation is feasible given resources and time considerations. Yes – providing stakeholders availability for interviews

6. Tailored Evaluation Questions

6.0 The Terms of Reference includes a comprehensive and tailored set of evaluation questions within the framework of the evaluation criteria.

Section 2.1.2, 2.1.3 and 2.1.4

6.1 The TOR contains a set of evaluation questions that are directly related to both the objectives of the evaluation and the criteria against which the subject will be assessed.

Section 2.1.2, 2.1.3 and 2.1.4

6.2 The set of evaluation questions adds further detail to the objectives and contributes to further defining the scope.

Yes.

6.3 The set of evaluation questions is comprehensive enough that they raise the most pertinent evaluation questions, while at the same time being concise enough to provide users with a clear overview of the evaluation’s objectives.

Yes.

6.4 Factoring in the information that will be collected and the context of the evaluation, evidence backed answers to the set of evaluation questions is achievable.

Section 2.2.2 and 3

7. Methodology

7.0 The Terms of Reference specifies the methods for data collection and analysis, including information on the overall methodological design

Section 4 (data collection) and section 2 (Conceptual Framework)

7.1 The TOR contains a clear and accessible methodological plan. Preferably, a standalone section that is clearly delineated from other information contained in the TOR.

Section 2, 3 and 4

7.2 The TOR states the overall methodological approach and design for the evaluation. Section 2, 3 and 4

7.3 The data collection and analysis methods in the TOR are sufficiently rigorous to assess the subject of the evaluation and ensure a complete, fair and unbiased assessment.

Section 4

7.4 The evaluation methodology includes multiple methods (triangulation); preferably with analysis of both quantitative and qualitative data and with a range of stakeholders covered by the data collection methods.

Yes. Refer Section 4

7.5 Logical and explicit linkages are provided between data sources, data collection methods and analysis methods. For example, sampling plans are included.

Section 4

7.6 The evaluation methodology takes into account the overall purpose of the evaluation, as well as the needs of the users and other stakeholders.

Section 2, 3 and 4

7.7 The evaluation methodology explicitly and clearly states the limitations of the chosen evaluation methods.

Section 3.5

7.8 The TOR specifies that the evaluation will follow UNEG norms and standards for evaluations, as well as ethical guidelines.


8. Evaluation Work Plan

8.0 The Terms of Reference includes a work plan

8.1 The TOR work plan states the outputs that will be delivered by the evaluation team, including information on the degree to which the evaluation report will be accessible to stakeholders, including the public.

Section 2.2.1, 2.2.4

8.2 The TOR work plan describes the key stages of the evaluation process and the project time line.

Section 2.2 and 4

8.3 The TOR work plan establishes clear roles and responsibilities for evaluation team Section 5


PwC 39

members, the commissioning organisation and other stakeholders in the evaluation process.

8.4 The TOR work plan describes the evaluation quality assurance process. Section 5.3

8.5 The TOR work plan describes the process, if any, for obtaining and incorporating evaluand comments on a draft evaluation report.

Section 2.2.4

8.6 The TOR work plan includes an evaluation project budget. N/A

9. Gender and Human Rights

9.0 The Terms of Reference specifies how a human rights and gender perspective will be incorporated in the evaluation design.

N/A

9.1 The TOR indicates both duty bearers and rights holders (particularly women and other groups subject to discrimination) as primary users of the evaluation and how they will be involved in the evaluation process.

N/A

9.2 The TOR spells out the relevant instruments or policies on human rights and gender equality that will guide the evaluation process.

N/A

9.3 The TOR includes an assessment of relevant human rights and gender equality aspects through the selection of the evaluation criteria and questions.

N/A

9.4 The TOR specifies an evaluation approach and data collection and analysis methods that are human rights based and gender sensitive and for evaluation data to be disaggregated by sex, ethnicity, age, disability, etc.

N/A

9.5 The TOR defines the level of expertise needed among the evaluation team on gender equality and human rights and their responsibilities in this regard and calls for a gender balanced and culturally diverse team that makes use of national/regional evaluation expertise.

Section 5.1


PwC 40

Appendix B- Addendum to Inception Report

With offices in Aarau, Basel, Berne, Chur, Geneva, Lausanne, Lugano, Lucerne, Neuchâtel, Sitten, St. Gallen, Thun, Winterthur, Zug and Zurich, PricewaterhouseCoopers SA is a provider of auditing services and tax, legal and business consultancy services. PricewaterhouseCoopers AG is a member of a global network of companies that are legally independent of one another; the network is represented in some 150 countries throughout the world.

PricewaterhouseCoopers SA Avenue Giuseppe-Motta 50 P.O Box 2895 1211 Geneva 2 Phone +41 58 792 91 00 Fax +41 58 792 91 10 www.pwc.ch

Office 7121 Bid Ref: IOS Stage two Attn: Mr David Webb, Director, Office of Internal Oversight Services World Health Organization 20, Avenue Appia CH-1211 Geneva 27 Switzerland Geneva, 21 August 2013 Dear Sir / Madam, We thank the IOS Secretariat for facilitating the teleconference with the Evaluation Management Group (EMG) that took place on 8 August 2013. The aim of the teleconference was to discuss EMG’s feedback on the Inception report submitted to the WHO on 8 July 2013 and to obtain EMG guidance on specific items including 1) list of stakeholders, 2) the survey and quantitative approach and 3) the timeline for the assignment. The purpose of this Addendum to the Inception report is to present refinement to our detailed approach to conducting the Stage 2 Evaluation of the WHO reform following feedback received from the EMG. We remain at your disposal for any further information or clarification on our inception report and look forward to hearing from you. Yours faithfully, PricewaterhouseCoopers SA

Gill Sivyer Antoine Berthaut

Engagement Partner - Geneva Engagement Director–Geneva

Undertaking regarding taking into account WHO’s specific context

WHO Reform Stage 2 Evaluation - Final report Appendices 42

1. Undertaking regarding taking into account WHO’s specific context

1.1. EMG Feedback

Following submission of the inception report on the 8 July 2013, the members of the EMG provided their

written feedback to the content of the report, which can be summarised under the main headings as follows:

How to leverage previous WHO evaluations to inform the stage two evaluation

What should be the extent of stakeholder consultation

How this evaluation will prepare the ground for future evaluations of the reform process

Given the complexity of the evaluation subject, is there sufficient capacity in the evaluation team to address all activities to the level demanded by the evaluation.

To respond to the above mentioned feedback, the below approaches were proposed by Independent Evaluation Team (IET) and discussed with the EMG during the teleconference on 8 August 2013.

While the above approaches were generally agreed to and validated by the EMG, the EMG members noted that

it would beneficial for the IET to contextualise the evaluation approach to the content of the WHO reform,

notably the evaluation questions, and to demonstrate how WHO’s current functioning’s will be impacted by the

reform initiatives rather than relying only on a generic transformation model. EMG members requested that

the IET emphasised what needed to be done to demonstrate that this reform would be more successful than

past initiatives.

4

Leveraging previous evaluations of the WHO to inform the Stage 2 evaluation (South Africa)

Assess the level of implementation of Stage 1 recommendations

Previous WHO evaluation

Feedback Evaluation suggestion

1

List of stakeholders is limited to high-level seniormanagement at HQ and regional level (South Africa)Considering both internal and external conditions thatare enablers of reform (JIU)

Broaden stakeholder consultation tocountry level (see next agenda item)

Extend stakeholder consultation toglobal health actors (see next agendaitem)

Extent of stakeholders consultation2

Making explicit how this evaluation will prepare theground for the next phase evaluation (outcome/impact) (JIU)

Include recommendations on the futuremonitoring and evaluation of the reformin evaluation report

Future evaluation of WHO reform3

Capacity put in place by PWC to carry out all activities with level of depth demanded by evaluation as opposed to a performance assessment model (JIU)

Revise timeline to address level of depththrough additional interviews

4 Capacity to cover depth



Finally, EMG emphasised the need for the IET to address the position of WHO in a changing global health

architecture and evolving financial environment and to consider how external factors will be affecting the

success of the reform.

1.2. Independent Evaluation Team Undertaking

In response to the EMG, the Independent Evaluation Team (IET) first reiterates that in selecting the evaluation

questions in the Inception Report, the IET already ensured the relevance of these questions, although of a

generic nature, to WHO context. Building further on this conceptual framework, and to ensure the specific

context of WHO reform is taken into account, the IET have also complemented the set of questions with specific

ones outlined below.

The table summarises the IET undertaking to address the above mentioned points.

Domain of the evaluation framework

Specific factor Evaluation question How this will be taken into account in the evaluation

External to WHO

Strategy, structure

WHO position in the global health architecture

Has progress been made by WHO in clarifying its role in a changing global health architecture? How does reform takes this changing role into account?

Interviews with external key informants on global health

Literature review addressing WHO’s positioning and role in the global health architecture

Review of alignment of WHO’s vision with the 2014-15 Programme budget

Strategy Financial crisis – funding for Health and UN agencies

What is the impact of financial constraints on the implementation of the reform?

Funding inequalities across the three levels of the organisation will be assessed and the question of resources for reform will be addressed in interviews

Specific focus will be placed on financing and result-based management initiatives

Strategy UN Post 2015 development goals and UN-wide organisation

How is WHO positioning itself in the broader UN system and what leadership is it demonstrating in the definition of the post 2015 UN development goals?

Desk review of General Programme of Work

Interviews with Secretariat’s senior management

Interviews with external key informants

Structure Changing nature of world politics

How is WHO taking into account emerging global governance arrangements and the evolution of the balance of power between geographical areas?

Interviews with Member States, Secretariat and key informants

Internal to WHO

Change management, programme management

Past reform initiatives

Have learnings and recommendations issued by previous evaluations of WHO reforms in the 90s’ been factored into the definition of the current reform program and its implementation?

Review of past reforms and their outcomes based on literature review (Chatham House report, prior JIU reports)



Domain of the evaluation framework

Specific factor Evaluation question How this will be taken into account in the evaluation

All domains Relevance of the evaluation questions and conceptual framework to WHO’s specific context

What is the content of each reform initiative and are those initiatives effectively helping to reshape the organisation?

Interviews with the Reform Team, Business Owners and regional level senior management to discuss the substance of the reform initiatives.

Desk review of relevant documentation discussed with interviewees in order to validate and support findings on the status and content of the reform

Structure, programme and change management

Specific WHO governance arrangement in regions

What is the level of alignment of governance arrangements and processes across regions?

Review of regional governing body documentation, reports and resolutions

Interviews with WHO Regional Offices and Chairs of Regional Committees

Complement and updates to the approach and instruments


2. Complement and updates to the approach and instruments

2.1. Engagement with a selection of Members States

2.1.1. Selection of Member States

The feedback from the EMG in relation to the selection of stakeholders was an absolute requirement of a

balanced representation of the various stakeholders’ groups to ensure credible and valid findings. However, it

was also recognized that the options available are limited due to time constraints for conducting the interviews

and consultations; and due to the practicalities of conducting this evaluation through the summer vacation

period.

Selection of Member States should be representative of all countries composing the World Health Assembly and

not necessarily of those who are more vocal or supporters of the reform process. The selection of Member States

should ensure representation of North/South, East/West, BRICS and equal representation of developing

countries. Random sampling of Member States is the recommended methodology for selection. The EMG

concluded that the IET should propose a list of potential informants for validation.

In order to realign with the above feedback, two (2) countries have been randomly selected for

each of the six (6) WHO regions, with a total of 12 countries selected.

The selection aimed for balance in the following criteria:

Income level based on World Bank country classification1

BRICs vs non BRICS

Status of renewal of the Country Cooperation Strategy2

Fragile countries based on World Bank classification3

The selection excluded countries which are solicited in other ways throughout the evaluation:

Evaluation Management Group countries (i.e. Australia, Azerbaijan Islamic Republic of Iran, Myanmar,

Panama, South Africa);

Member States from the Geneva group that will be convened through a focus group discussion by the IET;

The country affiliation of Chairs of Regional Committees;

Countries the IET interviewed during the Mobilise phase of the evaluation (United States, United Kingdom,

Ireland, Brazil, Norway, Canada).

Based on the above, the random selection resulted in the following list of countries presented in the next page.

1 http://data.worldbank.org/about/country-classifications/country-and-lending-groups#Low_income 2 http://www.who.int/countryfocus/cooperation_strategy/listofccs/en/ and WHO country offices websites 3 http://siteresources.worldbank.org/EXTLICUS/Resources/511777-1269623894864/HarmonizedlistoffragilestatesFY14.pdf



Table 1. List of Member States with WHO representation to be consulted

Regions Country selection

AFRO Kenya

Cameroon

AMRO Argentina

Mexico

EMRO Morocco

Egypt

EURO Russia

Turkey

SEARO India

Thailand

WPRO China

Solomon Islands

Changes suggested by the EMG are highlighted in bold.

For each of the 12 countries, the IET will endeavour to interview country representatives from the following

organisations:

Ministry of Health (priority 1)

Ministry of Foreign affairs (priority 2)

Geneva mission (priority 3)

The IET relies on the IOS to provide contacts and facilitate introductions.

In addition to the above selection of Member States, interviews will be conducted with:

the current Chairs of all WHO Regional Committees and the country affiliations to the following Member

States are presented as follows:

o Regional Committee AFRO: Dr José Vieira Van-Dúnem, Angola

o Regional Committee EURO: Dr Joseph R. Cassar, Malta

o Directing Council PAHO: Dr. Franklin Vergara Jaén, Panama

o Regional Committee EMRO: H.E. Mr Bahar Idris Abu Garda, Sudan

o Regional Committee SEARO: H.E. Dr Nafsiah Mboi, Minister for Health, Indonesia

o Regional Committee WPRO: Dr Nguyen Thi Kim Tien, Vietnam

all Member States forming the Geneva Group will be convened to attend a joint meeting organised and

facilitated by the WHO. The Geneva Group is composed of the following Member States: Australia,

Belgium, Canada, France, Germany, Japan, Italy, Mexico, Netherlands, Russia, Republic of Korea, Spain,

Sweden, Switzerland, United Kingdom, United States

The focus of the interviews with Member States will be to address the reform priorities, communication on

reform, governance arrangements and impact of the reform in their countries.



2.2. Selection of non-state actors

For the purpose of this evaluation, the group of non-state actors is broken down into three main categories:

NGOs in official relation with the WHO

Private donors

Key informants on global health

2.2.1. NGOs in official relation with the WHO

Of the 194 NGOs in official relation with the WHO, six (6) NGOs have been randomly selected with the view to

ensure representation from:

Implementing NGO

Professional association

Disease or illness focused organisation

Pharmaceutical focused organisation

The resulting selection resulted in the following NGOs being sampled:

African Medical and Research Foundation

International Federation of Pharmaceutical Manufacturers and Associations

International Council of Nurses

World Medical Association

Medicus Mundi International – International Organisation for Cooperation in Health Care

OXFAM


The focus of the interviews will be to understand NGOs perception of the WHO reform, in particular with

respect to WHO’s position in the global health architecture, its role as a convener on global health challenges

and its relationship with civil society actors.

2.2.2. Private donors

Two (2) private donor representatives have been selected on the basis of their financial support to WHO and

their recent involvement in the Financing Dialogue, June 2013.

The selection includes representatives of the following philanthropies:

Bill and Melinda Gates Foundation

Wellcome Trust


The focus of the interviews will be to understand the private sector’s perception of WHO and its relevance in the

changing global health architecture while also aiming to assess the efforts deployed at creating a space for

dialogue between WHO and the private sector.

2.2.3. Key informants on global health

The evaluation will consider the voices of five (5) external informants on global health, from academia and

government alike, some of whom have been involved in the reform directly. These will include in particular:



Professor Thomas Zeltner, former Special Envoy on WHO Financing Dialogue and current Special Envoy on Engagement with non-state actors

Dr. Sigrun Mogedal, former AIDS Ambassador of Norway

Mr Bezancon, Advisor on FAO reform and author of a 2005 report on The Global Fund’s replenishment

Dr. Ilona Kickbusch, Global Health Diplomacy, Graduate Institute of International and Development Studies

Charles Clift, Chatham House

The focus of the interviews will be to obtain their views on the positioning of the WHO in the evolving global

health architecture and the role it should play in the post-2015 development agenda.

2.3. Engagement with a selection of WHO Representatives at country level

The feedback from the EMG in relation to Member States was that in order to avoid bias, the selection of WHO

representatives (WRs) should be performed randomly, and should not rely on interviewee suggestions.

To that effect, the IET suggests that the selection of WRs reflect the random selection of 12 Member States

selected randomly above (see Table 1.). This will also allow triangulating results from interviews with WRs with

those of interviews with Member States.

The focus of the interviews will be on the regional and country level implementation of the reform agenda and

on the level of engagement of the WRs in the process.

2.4. Engagement with a random selection of WHO staff

Feedback from the EMG emphasised that in order to avoid selection bias and to capture internal staff

perceptions as to how far the reform message has penetrated the organisation, the methodology for selecting

internal WHO staff should use a random selection of staff identified from the phone directory.

Six (6) WHO staff will be randomly selected across all three levels of the organisation, and identified through

the phone directory as mentioned above.

The focus of the interviews will be to better understand how the reform is directly impacting staff in their day-

to-day work.

Quantitative Survey approach


3. Quantitative Survey approach

3.1. EMG feedback

The EMG members signalled that the current questionnaire, as designed, did not capture the necessary information from staff required to answer the question “How reform is translating at their level and what impact it will have on their work”. Following discussions on the purpose and need for the questionnaire the participants agreed on the need to redesign the questionnaire for an all staff survey. The revised questionnaire would:

a) focus on the various domains of the reform; b) include more personal details on the respondent, such as length of service with WHO; c) assess the extent to which the reform changed what the respondent do/working approach in the last 12

months; d) explore the respondent’s perception on how the reform will change WHO in the future.

It was agreed that the IET would re-submit the re-designed questionnaire to the EMG for final clearance before implementation.

3.2. Revised Objectives for quantitative survey

In line with the request of the EMG, the survey questionnaire has been reworked to address the above mentioned objectives. An updated version is presented in Appendix A for EMG feedback. The survey contains four sections. Section 1 collects information on the respondents. Section2 addresses how communication on reform is perceived by respondents. It seeks to understand how staff receives information on reform and how efficient and relevant the existing communication channels are in the eyes of the respondent. Section 3 addresses the respondent engagement with the reform process. It aims at providing a temperature gauge on the engagement of staff in the reform process. Questions seek to assess their level of engagement, from understanding the objectives of the reform, to concretely inputting into reform initiatives and being supportive of the reform process. Section 4 seeks to quantify respondent perceptions of WHO work practices in those areas of reform that the IET deemed to impact staff and work practices the most based on initial interviews with Business Owners:

Overall perception of changes in working practices

Alignment, synergy and collaboration across the organisation

Engagement with other stakeholders

Human Resources

Programmatic priorities

Finances

Accountability, transparency and risk management

Knowledge management

Note that no question will be asked on the topic of “Evaluation” given that a detailed survey has been performed in July 2013 by the Secretariat the results of which will be reviewed by the IET once made available. This section also seeks to understand if staff believes that the WHO reform will have a meaningful impact on improving work practices in the above mentioned areas.



3.3. Addressing challenges encountered in constructing the quantitative survey

An early finding of the investigation to date relates to the weakness of the result-chain or theory of change of the reform. In the absence of a clear articulation of the exact and ultimate impact each reform initiative is aiming to have on work practices, building a survey to assess whether these changes are happening is not trivial. In this context, the IET had to:

Identify which areas of reform or work practices are most likely to impact staff, that staff apply or can observe;

Understand through desk review and interviews with Business Owners the goal and objectives of these initiatives;

Identify some of the missing indicators that would bring evidence of change and success of these reform initiatives;

Check if these indicators have been measured in previous surveys, in which case the same questions could be reused to see if progress has been achieved;

Where no previous question can measure the change in work practices, create new questions to provide an initial baseline.

Asking staff what specific reform related improvements they are observing makes an invalid assumption that staff understand the exact content of each reform initiative and can link these initiatives to change in their daily work. Staff cannot and do not necessarily need to be expected to make the difference between changes or improvements in work practices that are specifically linked to reform initiatives from changes or improvements that may be due to initiatives not related to reform. In this context the IET team took the view that it is better to seek to measure with absolute certainty if changes in work practices are happening than to seek to determine if these changes are unequivocally linked to the reform programme. The premise is that if changes are not seen to be happening on the ground, then the logical conclusion is that reform has yet to address the barriers that need to be addressed and implement the scope of required activities to effect change. The IET also sought to complement the approach by prompting staff on whether they believe that work underway as part of reform will have an impact on these work areas.

3.4. Note on re-use of some of the 2012 JIU survey questions

In reviewing the JIU ‘Review of Management, administration and decentralization in the WHO’ reports, the IET noted that the JIU submitted a survey to staff mid 2012 which contained a number of questions on work practices in the various areas of reform. These can be used as proxies to see if changes are being observed. In building section 4 of the evaluation survey, we leveraged a number of these previously used questions. These are identified by an asterisk (*) in the survey. By comparing the results with the results of the mid-2012 JIU, the IET will be able to identify if respondent’s perception in the various areas of reform have changed in the last year. In cases where no relevant JIU questions were available, e.g. in the area of engagement with non-state actors or risk management, the IET created new questions. These will provide an initial baseline and allow to gauge staff’s perception of WHO’s current performance in these areas.

3.5. Organisation of survey

Timing

The survey will span over a period of three (3) weeks from 2 September to 23 September 2013.

Communication



Communication on the survey will be done through four (4) avenues and supported by the IOS Secretariat and

the WHO Communications Department:

1. The survey will be sent to all staff through a formal initial email communication by the IOS Secretariat

on 2 September 2013 informing them of the purpose of the evaluation and objectives of the survey;

2. A message will be posted on the WHO intranet site on the survey and Stage 2 Evaluation, supported by

the Communications Department;

3. IOS will request Assistant Director Generals and Regional Directors to send a communication to their

staff encouraging them to take the survey;

4. Two reminders will be sent by email to all WHO staff on 9 and 16 September 2013.

Platform

The survey will be administered online. The survey will be anonymous and access to detailed answers and

results will be limited to the IET.

Updated Project Management Plan


4. Updated Project Management Plan

As described in the Gantt chart, the table below sets out the expected deliverables, milestones and feedback

expected by the EMG.

Deliverables Deadlines Inception report 8 July 2013 EMG feedback on inception report 19 July 2013 Preliminary report 14 October 2013 Feedback on preliminary report 29 October 2013 Final Evaluation report 11 November 2013 Summary PowerPoint presentation 20 December 2013 Presentation to 134th session of the Executive Board 17 January 2013

Below is a detail of the EMG check points throughout the assignment:

9 July: Receive inception report

19 July: Provide feedback on inception report

08 August: EMG call on inception report

21 August: Receive survey and list of key stakeholders

28 August: Provide feedback on survey and list of key stakeholders

16 September: Receive key findings in draft form

19 September: EMG debriefing call (key findings)

03 October: EMG debriefing call (TBD)

14 October: Receive preliminary report

29 October: Provide feedback on preliminary report

4 November: EMG call on preliminary report

11 November: Receive final report

20 December: Receive final presentation

The Gantt chart below presents the updated detailed workplan that will guide the execution of the Evaluation

work.

Updated Project Management Plan


T ask Ju neP lanning

( # o f

w e e ks ) 2 4 .0 6 1.0 7 8 .0 7 15 .0 7 2 2 .0 7 2 9 .0 7 5 .0 8 12 .0 8 19 .0 8 2 6 .0 8 2 .0 9 9 .0 9 16 .0 9 2 3 .0 9 3 0 .0 9 7 .10 14 .10 2 1.10 2 8 .10 4 .11 11.11 18 .11 2 5 .11 2 .12 9 .12 16 .12 2 3 .12 3 0 .12 6 .0 1 13 .0 1 2 0 .0 1

Mobilise 6 we e k s

Collect in for m a tion 2 weeks

A ssess WHO r efor m sta tu s 2 weeks

Pr odu ce in cept ion r epor t (IR) 2 weeks

Su bm it in cept ion r epor t

EMG r ev iew of in cept ion r epor t 2 weeks

Receiv e EMG feedba ck on in cept ion r epor t

Hold EMG ca ll on in cept ion r epor t

Evaluate 12 we e k s

Con du ct key sta keh older s in ter v iew s

HQ 9 weeks

Reg ion s 7 weeks

Cou n tr ies 6 weeks

Mem ber Sta tes 10 weeks

Con du ct desk r ev iew 7 weeks

Con du ct su r v ey 7 weeks

Synthesize & Report 2 2 we e k s

Con du ct da ta a n a ly sis 8 weeks

Dr a ft Pr elim in a r y r epor t (PR) 5 weeks

Su bm it pr elim in a r y r epor t

EMG r ev iew of pr elim in a r y r epor t 2 weeks

Receiv e EMG feedba ck on pr elim in a r y r epor t

Hold EMG ca ll on pr elim in a r y r epor t

In cor por a te EMG feedba ck 2 weeks

Su bm it fin a l r epor t

Socia lise fin din g s 6 weeks

Pr epa r e fin a l pow er poin t pr esen ta t ion 2 weeks

Su bm it fin a l pow er poin t pr esen ta t ion

Pr esen t a t Ex ecu tiv e Boa r d

Key milestones

Ja n u a ry 2014Ju ly A u gu st Sept em ber Oct ober Nov em ber Decem ber

Receive EMG

feedback19 July

2013

Submit Inception

report

8 July

2013

Submit final presen-

tation20 Dec

2013

Present at Executive

Board17 Jan 2014

Hold EMG call

Week 8 August

2013

Submit final

report11 Nov

2013

Hold EMG

call 4 Nov

2013

Submit preliminary

report14 Oct

2013

ReceiveEMG

feedback29 Oct

2013

Roles and responsibilities


5. Roles and responsibilities

5.1. Update to description and role of WHO

The EMG raised some concern regarding the governance arrangements for the oversight of the evaluation

process. In particular, the EMG sought clarification on its role in relation to the one of the “Internal Steering

Committee”. The feedback of the EMG on the role of the Steering Committee was that the title of this body is

misleading because it is not its role to provide guidance and advisory support on the evaluation.

The IOS Secretariat confirmed that the internal committee was formed in response to the previous EMG’s

specific request for a WHO representation to institutionalise the evaluation; as well as the IET’s request for a

“sounding board” to facilitate internal communication within WHO so that it could benefit from the experience

of the external consultants in conducting such an evaluation. It was agreed that the internal Steering Committee

will be renamed to better reflect its purpose and avoid the risk of confusion with the EMG’s role. The table

below presents the updated name and composition of the ‘WHO Stage 2 Evaluation Committee’.

WHO role Nature of involvement Estimated workload

WHO Stage 2 Evaluation Committee Hans Troedsson, Executive Director, Office of the Director-General Ian Smith, Reform Team David Webb, IOS Mohamed Jama, Assistant Director General, General Management 6 Regional Nominees

Follow-up on the execution of the evaluation and act as a sounding board

Guide and provide feedback on data collection and tools

Consider initial findings and facilitate the socialisation of findings

Ensure transfer of knowledge of the evaluation work within the WHO

Three planned teleconferences:

Review execution phase plans

Consider preliminary report key findings

Consider recommendations and lessons learned (final report)

Other roles and responsibilities remain unchanged.


Appendix D. - Data collection-Updated Staff survey questionnaire for feedback from EMG

WHO Reform survey

The Executive Board at its special session in November 2011 requested a two stage independent evaluation of

the WHO reform. The modalities for Stage 2 of the evaluation were endorsed by the Executive Board at its 132nd

session in January 2013 to assess the readiness of the WHO to fully implement the reform agenda and to

identify the necessary conditions for success.

As part of this work, the Independent Evaluation Team from PwC is interested in knowing more about what

staff’s involvement in the reform and what effect it is having on your day to day work.

The survey should only take about 15 minutes of your time and your answers will be completely anonymous.

The findings of the evaluation, including the results of this survey, will be presented at the PBAC session prior

to the 134th Executive Board session in January 2014.

We encourage you to participate by answering in your own capacity and sharing with us your personal

perception.

We thank you in advance for your collaboration to ensure this process is successful and beneficial to all.

The Independent Evaluation Team


1. About you This survey is anonymous; however we would appreciate if you could provide answers on your employment at WHO. This will allow us to perform a more meaningful analysis. 1.1 My post is located at:

Region of employment Tick the corresponding box(es)

HQ

AMRO/PAHO Region

AFRO Region

EURO Region

EMRO Region

WPRO Region

SEARO Region

Regional office

Country office

Other

If other, please specify

1.2 My duty station is:

Duty station Tick the corresponding box

Pick from list of all duty WHO stations

1.3 I have been working at WHO for:

Length of service Tick the corresponding box

0-2 years

3-5 years

6-10 years

More than 10 years

1.4 I am hired as:

Contract type Tick the corresponding box

International Professional staff

National Professional officer

General Service staff

Other

1.5 My appointment is:

Contract type Tick the corresponding box

Continuing

Fixed term

Temporary

1.6 My grade/category is:

Grade Tick the corresponding box

D1-D2

P5-P6

P3-P4

P1-P2

G


Other

1.7 My age group is

Age group Tick the corresponding box


18-29

30-39

40-49

50-62

Older than 62


Other

1.8 My gender is:

Gender Tick the corresponding box

Female

Male

1.9 The area I primarily work in is (alphabetical order):

Work area Tick the corresponding box

Communicable diseases

Corporate services/enabling functions

Health systems

Non-communicable Diseases

Preparedness, surveillance and response

Promoting health through the life‐course

2. Communication on the WHO reform This section seeks to understand how you are informed on the WHO reform. 2.1 For each of the statements below please tick the answer that best corresponds to your views:

Statement Don’t

know/ no

opinion

Strongly

disagree Disagree

Neither

agree or

disagree

Agree Strongly

agree

I am aware of the WHO reform

I have been well informed by management of the

WHO reform process*

I believe that management is communicating

adequately with staff about important staff issues

relating to reform (e.g. HR policies)

I believe communication on reform is timely

I believe communication on reform is efficient

I believe communication on reform is adapted to my

needs

There are adequate ways for me to inquire about the

WHO reform programme


2.2 I am primarily informed about the WHO reform through (ranked alphabetically):

Channels Tick the corresponding box, more

than one choice possible Available Governing Body documentation

Informally by other colleagues

Updates in management meetings

My supervisor

Participation in reform related projects or task forces

The Change@WHO Newsletter

The Staff Association

Townhall presentations

WHO Intranet and internet pages on Reform

Other

If other, please precise

2.3 I would prefer receiving information on WHO reform through:

Please select up to 3 sources by order of importance

Preferred Channels 1st

favourite

2nd

favourite

3rd

favourite

Available Governing Body documentation

Informally by other colleagues

Updates in management meetings

My supervisor

Participation in reform related projects or task forces

The Change@WHO Newsletter

The Staff Association

Townhall presentations

WHO Intranet and internet pages on Reform

Other

If other, please precise

3. Your engagement in the WHO reform In this section, we seek to understand your involvement and perception of the WHO reform 3.1 For each of the statements below please tick the answer that best corresponds to your views:

Statement Don’t

know/ no

opinion

Strongly

disagree Disagree

Neither

agree or

disagree

Agree Strongly

agree

I believe I understand the vision, objectives and

planned outcomes of the WHO reform

I believe the WHO reform will be important to make

the organisation more ‘fit for purpose’ in the future

I believe the WHO reform will have a positive impact

on health outcomes and national health systems

I understand what is expected from me to make the

WHO reform successful

I believe I can personally contribute to the

achievement of the WHO reform objectives

I have participated in taskforces or working groups

specifically set-up in my

Cluster/Department/Regional office/Country office

to address the WHO reform


4. Evolution of work practices This section lists a number of work areas related to reform and seek to understand how reform is impacting your work practices. Note: statements with an asterisk (*) indicate questions you may have previously been asked in the JIU 2012 survey as part of the ‘Review of Management, administration and decentralization in the WHO’. We are asking these questions again to identify if any change has occurred in the last year. 4.1 For each of the statements below please tick the answer that best corresponds to your views:

Statement

Don’t

know/ no

opinion

Strongly

disagree Disagree

Neither

agree or

disagree

Agree Strongly

agree

Overall change

In the last 12 months, I have seen changes in my day

to day work linked to the WHO reform

Alignment, synergy and collaboration across

the organisation

I have a clear perspective on the division of

responsibilities in my area of work between HQ, the

regional offices, and the country offices*

I think there is sufficient coordination and

cooperation within clusters/divisions/units within

the following entities:*

WHO Headquarters

Regional offices

Country offices

I think the level of coordination and cooperation

between the following WHO entities is adequate*:

WHO Headquarters and regional offices

Regional offices and country office

WHO Headquarters and country offices

There is adequate alignment between WHO’s country

presence and country needs and priorities

Regional and country offices are adequately staffed to

play their role with Member States

I think the work underway on reform will improve

the effectiveness of the organisation*

Engagement with other stakeholders

I think the WHO will be better at convening health stakeholders after the reform

Guidance on engaging with non-state actors to help improve the public health outcomes of my programme is readily available

I think the role of WHO, among other actors working

on global health, in my programme area is clearly

defined

I think WHO has the available capacities to coordinate, convene and direct the various actors in global health

I think WHO’s role in global health governance will be strengthened through the reform process

Human Resources

I think the recruitment process is fair and

transparent in WHO*

I think I will move to another duty station in the next

3 years

Training received is adapted to my career

development needs*

My training is linked to my individual performance

appraisal system in line with WHO priorities*

I think the system of administration of justice in


WHO ensures an adequate and fair treatment of staff

complaints*

I think the work underway on reform will improve

the human resource capacity of the organisation to

deliver against its mandate

Programmatic priorities

WHO sets out clear priorities for its work

I am familiar with the concept of RBM and its

implementation in WHO*

I think WHO activities are measured with clearly

defined performance indicators*

I think the WHO reform will improve the focus of the

organisation in my programme area.

Finances

The allocation of funds across WHO is transparent

I am seeing more alignment between WHO priorities

and how resources are allocated across the

organisation

I conduct resource mobilisation activities (if any) as

part of a coordinated whole of WHO approach

There is an adequate level of coordination and

cooperation in resource mobilisation activities across

the different levels of the organisation

My unit receives necessary resources in order to meet

its goals*

I think the WHO reform will enhance the ability of

my programme area to have more predictable and

sustainable financing

Accountability, transparency and risk

management

I think managers are held accountable for the

authority delegated to them*

I have a clear idea of what are my tasks and

responsibilities and how I will be held accountable

for those*

Our policies and procedures for managing conflict of

interest are effective

The key risks in the unit I work for have been

identified and documented

My unit has defined mechanisms to mitigate these

major risks facing the achievement of my units

objectives

My superior actively supports and drives the use of

risk management in our work

I think work underway as part of the reform on

accountability, transparency and risk management

will build Member State’s trust in WHO

Knowledge management

Information-sharing in my unit happens in support

to my work*

The aims and scope of knowledge management in

WHO are clear to me*

I can regularly benefit from knowledge management

in my area of work*

4.2 In your view, what are top three challenges and barriers that need to be addressed for the reform to be a success? Do you have any suggestions on how to address this? (open question)

Free text form


Appendix C- Evaluation questions

The Terms of Reference for the ‘Independent evaluation of the WHO reform: stage two’ proposes a set of

questions to be addressed by the evaluation, and which are set out in the table below. These are linked to the

key issues the evaluation should address, as approved by the EB during its 132nd session4 (EB 132/5 Add 7). The

IET has supplemented those questions with a number of more detailed related questions, as set out in this

Appendix, which have guided the evaluation and interview process. These are set-out according to the two key

issues agreed to by the EB, (i) the status of action taken on the recommendations of stage one and (ii) the

modalities of implementing the reform proposal and the sufficiency of the change management strategy.

The proposed questions are presented in bold, while the detailed evaluation questions are presented in italic.

(i) Status of action taken on the recommendations of stage one.

(a)What is the status of implementation of the recommendation relating to the reform process compared with the expected (planned) progress and achievements?

(b)Have there been unintended effects in the reform process and how have these impacted the overall result of WHO reform?

(c)What factors have been identified (from progress to date) that will enhance moving forward with the reform effectively?

(ii) Modalities of implementing the reform proposal and the sufficiency of the change management strategy including:

(d) To which degree is there coherence in the governance mechanisms to inform, monitor and evaluate the implementation of the WHO reform package? To which extent are these mechanisms and processes adequate to prioritize the various components of the reform proposal?

Has there been a process to evaluate and prioritise areas for improvement? How was the prioritisation conducted? How were the criteria for prioritisation defined?

Inception report

Have governance bodies been set-up to advise/act as sounding board on this particular area? What has been the representation of these governance bodies?

Inception report

What has been the process for approving this roadmap? Inception report

How are the expectations of WHO Secretariat and Member States managed? Inception report

How are results reported and how is the quality of the deliverables monitored? Inception report

Has a realistic business case been developed subjected to an appropriate level of challenge?

Inception report

Has an agreed quality plan been developed based on appropriate standards to monitor and evaluate the quality of the work performed?

Inception report

4 WHO, Modalities for the independent evaluation of the WHO reform: stage two, Report by the Director-General, EB132/5 Add.7, 11 January 2013.


What are the governance arrangements supporting the reform agenda? Inception report

How are leadership activities executed and monitored? Inception report

Does the WHO promote a culture conducive to reform? Inception report

Has a change readiness assessment been carried out? Inception report

Is the communication delivered adequate and containing the right type of messages for different stakeholders of the WHO reform?

Inception report

Are stakeholders identified, mapped and is there a stakeholder management plan? Inception report

Has a communication and engagement plan defined? How is it executed and reported against?

Inception report

Has a change agent network been identified? Are change agent network activities taking place on an on-going basis?

Inception report

(e)What are the specific strategies to address change management? How well are the pertinent issues addressed?

Who has been involved in developing the implementation plan? Inception report

How have relevant stakeholders been engaged in the process? Inception report

How is the knowledge transferred and how is the hand-over performed from reform to business as usual?

Inception report

Have stakeholders been identified, assessed and are they managed on an on-going basis throughout the reform process to ensure high level of commitment?

Inception report

Are they appropriately informed to contribute to the success of the programme? Inception report

Has the nature of stakeholders been considered, their roles and relationship to one another defined?

Inception report

What change control arrangements are in place to manage changes to scope, requirements, deliverables and intended benefits that impact the project baseline, time or cost?

Inception report

Does a change vision exist? Inception report

Is there agreement on the objectives of the reform? Inception report

Is the case for reform actively pushed by leaders? Inception report

How are benefits of the reform demonstrated? Inception report

How is the informal and formal reward process managed? Inception report

What are the mechanisms in place to coach champions and managers? Inception report

(f)Have the significant barriers, challenges, systematic constraints, and risks for the reform been identified (including engagement of staff in the process)? How relevant and adequate are the proposed approaches for dealing with these barriers, challenges, systematic constraints,


and risks in relation to achieving the outcomes expected from the reform process? How efficient are the approaches to address them to ensure sustainability?

Are effective risk identification processes in place and are the key risks mitigated and opportunities taken?

Inception report

Have learnings from pilots (initial implementations) been applied to full implementation?

Inception report

Has progress been made by WHO in clarifying its role in a changing global health architecture? How does reform takes this changing role into account?

Addendum

Have learnings and recommendations issued by previous evaluations of WHO reforms in the 90s’ been factored into the definition of the current reform program and its implementation?

Addendum

(g) What is the efficacy of internal and external instruments and modalities to implement the reform proposals, including related risk management plans and risk mitigation actions?

How are performance issues addressed? Inception report

Has a clear assurance plan been defined which outlines the nature, timing and extent of planned assurance, quality reviews and embeds learning?

Inception report

Are personal objectives aligned? Inception report

How does the Reform Team respond to findings? Inception report

What is the level of alignment of governance arrangements and processes across regions?

Addendum

(h)Are indicators and timelines sufficient, feasible and appropriate to effectively measure (monitor and evaluate) performance and report on the desired outcomes of the reform process?

Have has a roadmap been produced, defining broad objectives, timeline, responsibilities, KPIs, budget and reporting structure?

Inception report

Has a detailed implementation plan been developed that integrates new ways of working?

Inception report

Which of the activities part of the implementation plan have started? Inception report

Has a post-implementation review been completed? Inception report

Is the scope of the programme well defined, complete, communicated and agreed? Inception report

Does the scope support the objectives of the WHO High-level implementation strategy? Inception report

Are the plans in place realistic, achievable, understood and bought into by key stakeholders executing the reform and those affected by it?

Inception report

How are outputs reported and is reporting targeting all stakeholders to the reform? Inception report

Does management regularly report on performance? Inception report

How is the success of the reform defined? Inception report

How is it assessed? Inception report


How were KPIs defined? Inception report

What is the process and mechanism in place for monitoring and reporting? Inception report

(i) To what degree have appropriate strategies, including resource mobilization and allocation, and corporate strategies such as human resources management and gender mainstreaming, been identified to implement the reform proposals?

What are for you the strategic drivers of the reform? What has been the process to identify and agree on the major challenges?

Inception report

Was the current state of affairs analysed and the baseline created? Inception report

Has a strategy/concept note been developed to address this area of challenge? Inception report

Has an approach been designed to address the area for improvement? Has a proposal been developed?

Inception report

Is the programme team highly motivated with the right blend of skills and personalities? Inception report

Is the budget and associated policies, processes and reporting standards adequate for effective cost estimation and programme financial management?

Inception report

What are WHO’s core competencies? Inception report

What are the processes for formulating strategy? What are the processes for measuring and managing the execution of strategy?

Inception report

Does the current structure of the client support the overall strategy? Inception report

Has the strategic plan been translated into coherent sub-strategies for each of the 12+1 initiatives under review?

Inception report

How well are the sub-strategies for IT, operations, finance, and risk management aligned to the overall strategy?

Inception report

What is the impact of financial constraints on the implementation of the reform? Addendum

How is WHO positioning itself in the broader UN system and what leadership is it demonstrating in the definition of the post 2015 UN development goals?

Addendum

How is WHO taking into account emerging global governance arrangements and the evolution of the balance of power between geographical areas?

Addendum

What is the content of each reform initiative and are those initiatives effectively helping to reshape the organisation?

Addendum

(j)Are the necessary conditions present to ensure an appropriate level of organizational preparedness (institutional framework: structures, roles, instruments, procedures and guidelines, support systems for change process, communication, incentives)

Has a specific body/team been set-up to manage and oversee the implementation? Inception report

Has an organisational impact assessment been carried out for the reform overall and for reform initiatives specifically?

Inception report

Is there clarity on the working practices, process documentation, organisational design and clear policies to ensure WHO staff understand the purpose of the reform and their role?

Inception report

Are systems, tools and guidance adapted to support the reform? Inception report


Are trainings set-up to ensure acquisition and transfer of knowledge and new skills? Inception report

Are processes, policies and rules are being changed where necessary to support the reform?

Inception report

Does the strategy call for centralised control and commonality of process? Inception report

Have factors relating to internal governance, reporting lines between governing bodies, accountabilities, organizational structure, geographical footprint and relationships with external stakeholders been adequately considered?

Inception report

Which processes could be candidates for outsourcing? If processes are currently outsourced, what has been the WHO's experience?

Inception report

Are processes consistent and standard across the organisation, at the HQ, regional and country level?

Inception report

What does the process documentation tell us about WHO’s model? Inception report

Is there a clear end-to-end process view? Inception report

Is process performance measured? Inception report

What processes are internal and which have been outsourced? Inception report

Have internal controls and standardization of processes and data been considered? Inception report

Are tools and guidelines readily available and incorporated into the day to day work of WHO staff for greater transparency?

Inception report

Are governance mechanisms and accountability documented and clear? Does the WHO have a mechanism for whistleblowing?

Inception report

How do decisions get made? What is the span of control and reporting? Where within the WHO are the power brokers that influence change?

Inception report

Do leaders own change efforts, or are they delegated to others? Inception report

What is the DNA of the culture? Is it consistent across the WHO? Inception report

Are organisational structures, culture, and individual behaviours aligned with and supportive of strategy?

Inception report

Are the skill-sets of staff adequate? Inception report

Has a performance management system and incentives been considered? Inception report

Have agents of change within the organisation been identified? Inception report

What processes are in place to connect the IT function to the organisation to support the reform?

Inception report

Does the IT strategy appear to be aligned with the implementation plan? Is technology appropriately funded in the context of the reform?

Inception report

Has technology such as data management and IT infrastructure, in order to support the direction of change?

Inception report


Have all five dimensions been considered in the reform implementation strategy? Inception report


Appendix D- List of interviews

Name Position Stakeholder category

Al Shorbaji, Najeeb Director, HQ/KMS Knowledge Management and Sharing WHO HQ

Alwan, Ala Regional Director, EMRO WHO Regional Office

Andrus, Jon Kim Deputy Director, AMRO/PAHO WHO Regional Office

Arnold, Dianne DAF, SEARO WHO Regional Office

Asamoa-Baah, Anarfi Deputy Director-General, HQ/ODG Office of the Director-General

WHO HQ

Aylward, Bruce Assistant Director-General - Polio, Emergencies and Country Collaboration

WHO HQ

Balladelli, Pier Paolo WHO Representative, Argentina WHO Country Office

Bekedam, Hendrik Jan WHO Representative, Egypt WHO Country Office

Belghiti Alaoui, Abdelali Secretary of Health, Ministry of Health, Morocco Member States

Ben Yahmed, Samir DPM, EMRO WHO Regional Office

Benton, David Chief Executive Officer, International Council of Nurses NGO

Birmingham, Maureen WHO Representative, Mexico WHO Country Office

Branca, Francesco Director, Department of Nutrition for Health and Development (NHD)

WHO HQ

Burci, Gian Luca Director, Legal Counsel, HQ/LEG Office of the Legal Counsel WHO HQ

Cassels, Andrew Director (Strategy), HQ/ODG Office of the Director-General WHO HQ

Chan, Margaret Director General WHO HQ

Chestnov, Oleg Assistant Director-General - Family, Women's and Children's Health

WHO HQ

Deynout, Chantal Management Officer, NMH WHO HQ

Emiroglu, Nedret Deputy Director, Communicable diseases, health security, EURO

WHO Regional Office

Ephrem, Bersabel Senior Adviser HQ/GMA GMG ADGO Office of the Assistant DG

WHO HQ

Escorel De Moraes, Maria Luisa

Minister Counsellor, Permanent Mission of Brazil Member States

Etienne, Carissa Regional Director, AMRO/PAHO WHO Regional Office

Feig, Christy Director, HQ/DCO Department of Communications

WHO HQ



Fleischl, Juliet WHO Representative, Solomon Islands WHO Country Office

Fukuda, Keiji Assistant Director-General - Health Security and Environment

WHO HQ

Garrett, Laurie Journalist and writer Global Health Advisor

Girod, Magali Policy Adviser, Specialised Agencies and Humanitarian Team, UK Mission to the United Nations in Geneva

Member States

Golberg, Elissa Ambassador and Permanent Representative, Permanent Mission of Canada

Member States

Guerma, Teguest Director, AMREF NGO

Guner, Oner DG for Foreign Affairs & EU Member States

Halton, Jane Secretary of the Department of Health and Ageing, Department of Health, Australia

Evaluation Management Group

Han, Tieru DPM, WPRO WHO Regional Office

Hollo, Imre DAF, EURO WHO Regional Office

Hossein Nicknam, Mohammed

Acting Minister for International Affairs, International Affairs Department, Ministry of Health and Medical Education, Iran


Jacab, Zsuzsanna Regional Director, EURO WHO Regional Office

Jacob, Gabrielle First Secretary (Health), Permanent Mission of Ireland Member States

Jama, Mohamed Assistant Director-General, HQ/GMA GMG ADGO Office of the Assistant DG

WHO HQ

Jee, Youngmee Scientist, WP/RGO/DCC/EPI WHO Regional Office

Jeffreys, Nicholas Comptroller & Director, HQ/FNM Department of Finance WHO HQ

Kickbush, Ilona Director of the Global Health Programme at the Graduate Institute of International and Development Studies, Geneva

Global Health Advisor

Kieny, Marie-Paule Assistant Director-General - Health Systems and Innovation WHO HQ

Kloiber, Omar Director, World Medical Association NGO

Kobza, Jeffrey DAF, WPRO WHO Regional Office

Landry, Steve Deputy Director, Vaccine Delivery, Bill & Melinda Gates Foundation

Donors

Lopez-Acuna, Daniel Advisor to Director-General, HQ/ODG Office of the Director-General

WHO HQ

Lutnaes, Sverre Berg Counsellor, Permanent Mission of Norway Member States

Maina, Emmanuel Chief of the Division of Cooperation of the Ministry of Public Health, Cameroon

Member States

Mamacos, Peter Multilateral Branch Chief, US Department of Health and Human Services

Member States



Mandlhate, Custodia WHO Representative, Kenya WHO Country Office

Mason, Elizabeth Mary Acting in the absence of Flavia Bustreo, Assistant Director-General - Family, Women's and Children's Health

WHO HQ

Matsoso, Precious Director-General of Health, Ministry of Health, South Africa Evaluation Management Group

Mciff, Colin Health Attaché, Permanent Mission of the United States Member States

Menabde, Nata WHO Representative, India WHO Country Office

Mercado, Julio Counsellor, Permanent Mission in Geneva, Argentina Member States

Migliorini, Luigi WHO Representative, Russia WHO Country Office

Moeti, Matshidiso Deputy Regional Director (acting DPM), AFRO WHO Regional Office

Mogedal, Sigrun Former Norwegian AIDS Ambassador Global Health Advisor

Mourain-Schut, Françoise

Senior Legal Officer, HQ/DGO/DGD/LEG/IAL WHO HQ

Nakatani, Hiroki Assistant Director-General - HIV/AIDS, TB, Malaria and Neglected Tropical Diseases

WHO HQ

Ndiaye, Charlotte WHO Representative, Cameroon WHO Country Office

Ndihokubwayo Jean, Bosco

Regional Advisor, AF/RGO/HSS/BLT WHO Regional Office

Nzang, Andres Acting DAF, AFRO WHO Regional Office

Oduyemi, Cecilia Rose HQ/GBS Dept for Governing Bodies and External Relations WHO HQ

Oedo, Cecilia Finance assistant, HQ/HSS/Essential Medicines and Health Products (EMP) Department

WHO HQ

Osundwa, Anastacia Third Secretary, Permanent Mission of Kenya Member States

Ottiglio, Mario Director, IFPMA NGO

Padilla, Liliana Counsellor, Permanent Mission of Mexico Member States

Patterson, Cath Minister Counsellor (Health), Australian Permanent Mission to the UN, Geneva

Member States

Pendergast, Scott PRP Planning Resource Coordination and Performance Monitoring

WHO HQ

Peruga, Armando Acting in the absence of Dr Douglas Bettcher, Director, Prevention of Noncommunicable Diseases (PND)

WHO HQ

Plianbangchang, Samlee Regional Director, SEARO WHO Regional Office

Profili, Maria Cristina WHO Representative, Turkey WHO Country Office

Quiroz Ávila, Alberto Chief, Secretariat of External Relations, Ministry of Foreign Affairs, Mexico

Member States



Raviglione, Mario Director of the Global TB Programme WHO HQ

Ren, Minghui Director-General, Department of International Cooperation, National Health and Family Planning Commission, Ministry of China

Member States

Renganathan, Elil Director, HQ/PRP Planning Resource Coordination and Perf Monitoring

WHO HQ

Ruiz Vargas, Ruth Dafne Director of Multilateral Affairs, Secretariat of Health, Ministry of Health, Mexico

Member States

Sambo, Luis G. Regional Director, AFRO WHO Regional Office

Sandrasagren, Mahen Coordinator/ acting Director, HQ/HRQ HQ HR Management WHO HQ

Saphaisal, Kanita First Secretary (Health), Permanent Mission of Thailand Member States

Saunders, Philipa Consultant, OXFAM NGO

Schwartlander, Bernhard WHO Representative, China WHO Country Office

Schwartz, Thomas Director, Medicus Mundi NGO

Shin, Young-soo Regional Director, WPRO WHO Regional Office

Silberschmidt, Gaudenz Senior Advisor, HQ/ODG Office of the Director-Genera WHO HQ

Smith, Ian Advisor to Director-General, HQ/ODG Office of the Director-General

WHO HQ

Souteyrand, Yves WHO Representative, Morocco WHO Country Office

Stuppel, John Health and UN Cross Systems Team Leader, DFID Member States

Sukai, Jackson Inspector, Joint Inspection Unit of the United Nations System


Tegegn, Yonas WHO Representative, Thailand WHO Country Office

Tennakoon, Herbert Assistant Regional Director, SEARO WHO Regional Office

Troedsson, Hans Executive Director,HQ/ODG Office of the Director-General WHO HQ

Webb, David Director HQ/IOS Office of Internal Oversight Services WHO HQ

Wibulpolprasert, Suwit Vice Chair, International Health Policy Program Foundation (IHPF), Ministry of Public Health, Thailand

Member States

Yalanazan, Irem National Programme Officer, Mental Health and Disability, EU/ACO/TUR

WHO Country Office

Zeltner, Thomas Former Director-General of the Swiss National Health Authority and Secretary of Health of Switzerland, Special envoy to WHO

Global Health Advisor


Appendix E- List of documentation

Barry R. Bloom, WHO needs change, Nature, Vol 473, 12 May 2011.

Change at WHO, Newsletter on Reform, November 2012 and April 2013

Chatham House, the Role of the World Health organization in the International system, quoted in World Health Organization, The Third Ten Years of the World Health organization 1968-1977.

DFID. Multilateral Aid Review, Ensuring maximum value for money for UK aid through multilateral organisations, March 2011.

Garrett, Laurie. Existential Challenges to Global Health, Center on International Cooperation, New York University, August 2013.

Health In the post-2015 agenda, Report of the Global Thematic Consultation on Health, April 2013

IEOAC, Report of the Sixth Meeting of the Independent Expert Oversight Advisory, February 2013

IHP+. Progress in the International Health Partnership & Related Initiatives (IHP+), 2012 Annual Performance Report.

Ilona Kickbusch, Wolfgang Hein, and Gaudenz Silberschmidt. Addressing Global Health Governance Challenges through a New Mechanism:The Proposal for a Committee C of the World Health Assembly, Journal of law, medicine & ethics, 2010.

Joint Inspection Unit, Review of Enterprise risk management in the United Nations System, Benchmarking framework, 2010.

Joint Inspection Unit of the United Nations, Review of management, administration and decentralisation in the World Health Organization (WHO), 2012.

Ministry of Foreign Affairs of Norway, Cooperation for Health Development, WHO 's support to programmes at country level, September 1997.

Mopan Common Approach 2010: World Health Organization, January 2011.

Morrison JS. The end of the golden era of global health? Center for Strategic and International Studies; 17 April 2012.

Paul Lincoln et al, Conflicts of interest and the UN high-level meeting on non-communicable diseases, Lancet, Vol 378 November 12, 2011.

United States Government Accountability Office, United Nations, Improved Reporting and Member States’ Consensus Needed for Food and Agriculture Organization’s Reform Plan, September 2011.

United States Government Accountability Office, World Health Organization, Reform Agenda Developed, but U.S. Actions to Monitor Progress Could be Enhanced, July 2012.

WHO Accountability Compact between Assistant Director General and Dr Margaret Chan, Director General, World Health Organization, 2013.

WHO and Grayling, Stakeholder Perception Survey Global Communication Strategy Review, January 2013.

WHO Guidance Note, Operational Planning 2014-15 for Reform, Identification and inclusion of reform-specific products/services within 2014-15 operational plans, August 2013.

WHO reform, Consolidated report by the Director-General, 65th World Health Assembly, 25 April 2012.

WHO reform, Financing of WHO, A66/48, 13 May 2013

WHO reform, High-level implementation plan and report, 66th World Health Assembly, 10 May 2013.

WHO reform, High-level implementation plan and report, Report of the Programme, Budget and Administration Committee of the Executive Board to the 66thWorld Health Assembly, 20 May 2013.


WHO reform, Independent evaluation report: stage one, 65th World Health Assembly, 18 May 2012.

WHO Regional Committee for South-East Asia, Report of the sixty-fifth session, Indonesia, 5-7 September 2012.

WHO Regional Office for Europe, Rules of Procedure of the Regional Committee for Europe and of the Standing Committee of the Regional Committee for Europe, 2011.

WHO Regional Office for the Americas, A conceptual framework for the adoption of enterprise risk management (ERM) in PAHO, 2011.

WHO WPRO, Moving Forward, Making a difference’ Reform experiences of WHO Regional Office for the Western Pacific (2011–2012), Interim Report February 2012

WHO WPRO, Transforming Human Resources Management in the WHO Regional Office for the Western Pacific, 2013.

WHO, Amendments to the Financial Regulations and Financial Rules, Report of the Programme, Budget and Administration Committee of the Executive Board to the Sixty-sixth World Health Assembly, 66th World Health Assembly, 20 May 2013.

WHO, Corporate risk register, Organization-wide strategic risk management in WHO, Report by the Secretariat, Executive Board, 133rd Session, 17 May 2013.

WHO, Country Cooperation Strategy (CCS) WHO – Switzerland, 2013.

WHO, Draft twelfth general programme of work, 66th World Health Assembly,19 April 2013

WHO, Draft workplan for study on strengthening and institutionalization of Category Networks, June 2013.

WHO, Final report of the Task Force on Resource Mobilization and Management Strategies, May 2013.

WHO, Financing Dialogue, Investing in the World's Health Organization, 24 June 2013.

WHO, Financing Dialogue, WHO's budget: What is behind the numbers?, 24 June 2013.

WHO, Follow-up to the Political Declaration of the High-level Meeting of the General Assembly on the Prevention and Control of Non-communicable Diseases, 66th World Health Assembly, 27 May 2013.

WHO, Governance: options for criteria for inclusion, exclusion or deferral of items on the provisional agenda of the Executive Board, Report of the Director-General, Executive Board, 133rd session, 8 May 2013.

WHO, Human Resource Annual Report, Report of the Programme, Budget and Administration Committee of the Executive Board to the Sixty-sixth World Health Assembly, A66/60, 20 May 2013.

WHO, Human resources: annual report, Report by the Secretariat, 66th WHA, 14 May 2013.

WHO, Human resources: annual report, Report by the Secretariat, 66th World Health Assembly, 14 May 2013.

WHO, Modalities for the independent evaluation of the WHO reform: stage two, Report by the Director-General, EB132/5 Add.7, 11 January 2013.

WHO, Notes and Decisions from the Global Policy Group Retreat, 12-13 March 2013 , Chiang Mai, Thailand.

WHO, Notes and Decisions from the Global Policy Group Retreat, 28 May 2013 , Geneva.

WHO, Options to streamline the reporting of and communication with Member States,EB132/5 Add.4, 18 January 2013.

WHO, Programme budget 2014–2015, 66th World Health Assembly,19 April 2013

WHO, Real estate, Report by the Director-General, Executive Board, 132nd session, 7 December 2012.

WHO, Reforms in the WHO Western Pacific Region, Progress reports, June 2011-December 2012 and January-June 2013


WHO, Report of the regional committees to the Executive Board, Executive Board, EB 132/4, 18 January 2013.

WHO, Resolutions and decisions, 65th World Health Assembly, WHA65/2012/REC/1, 21-26 May 2012.

WHO, Review of management, administration and decentralization in the World Health Organization, report by the Joint Inspection Unit, EB132/5 Add.6, 4 January 2013

WHO, Revised terms of reference for the Programme, Budget and Administration Committee of the Executive Board, EB131.R2, 28 May 2012

WHO, Social determinants of health, Report by the Secretariat, Executive Board, 132nd session, 23 November 2012.

WHO, Streamlining of the work of the governing bodies and harmonization and alignment of the work of regional committees, Executive Board, 132nd session, 14 December 2012.

WHO, The Overarching Roles and Functions of the Three Levels of the Organization, presented and discussed at the Global Policy Group Retreat, March 2013.

WHO, WHO Country Cooperation Strategy 2008-2011 and 2012-2016, Thailand.

WHO, WHO evaluation practice handbook, OIOS.

WHO, WHO reform at country level, Country Coordination and United Nations Collaboration, October 2013.

WHO, WHO reform Programme Management Approach- Reporting, 2013

WHO, WHO reform, towards a new policy of WHO engagement with NGOs, consultation with NGOs 18 October 2012.

WHO, WHO’s arrangements for hosting health partnerships and proposals for harmonizing WHO’s work with hosted partnerships, Report by the Director-General, Executive Board, 132nd session, 23 November 2012.

WHO, WHO’s Financing Dialogue, Mission briefings, IEOAC, 16 October 2013.

WHO, WHO’s role in global health governance, Report by the Director-General, Executive Board, 132nd session, 18 January 2013.


PwC 74

Appendix F- Analysis of staff survey

who reform stage 2 evaluation final report appendices · who reform stage 2 evaluation - final...

Documents