Methodology for

POLICY IMPACT

ANALYSIS AND EVALUATION

Paolo Roberti (Istat) Roma, RRT Meeting July 6, 2006

PIA*: State of the Art

Discussions on PIA are constantly on the increase at the EU level:

– EU “Guidelines on Impact Assessment” published on June 15, 2005;

– The use of PIA methodologies is increasingly considered as a necessary condition for “good policies” (and good projects)

– The investment in PIA methodologies occupy a central position in the new FP7

* Policy Impact Analysis

Is PIA a new fad?

NO

• PIA methodologies were first developed systematically in the 1950s as a tool “to tell the public what progress has been made” (US Office of Management and Budget)

• PIA interests all policy areas; the outlook may be macro, meso o micro

• PIA may consist of numbers, coefficients, or indicators which in turn may be quantitative or qualitative (simple or compound) factual or contrafactual and derived from complex models.

What does measuring the impact of policy on innovation mean ?

• Measuring the impact of policy: effective and/or potential; static and/or dynamic, etc.;

• Assessment of the effects and results obtained by policies: have the expected objectives been attained? Was the impact proportional to the effort and the resources employed for their implementation? etc.;

• Generalization of the results: may the cost-efficient policies be replicated and implemented in other countries/regions with a “guarantee of success”?

PIA can mean different things. Evaluation criteria may be

demand, use-oriented o expert-driven

During the assessment one must remember that

• Every policy is the product of a complex and circular process that: Continues until it is abolished; Produces results that are not necessarily

satisfactory; Must be monitored and evaluated.

“ROAMEF” CYCLE OF POLICY ANALISIS

In the policy analysis jargon ROAMEF stands forRationale, Objectives, Appraisal, Monitoring, Evaluation and Feedback

The Cycle begins with a … Problem

• Which at the political level may - or may not - be considered a problem that should - or should not - be faced and resolved;

• It is important to remember that even a “non policy” is a political choice;

Every policy has objectives and reasons: this means results that must be reached and

situations which must be acted upon

1° Principle of Policy Appraisal and Evaluation

“Impact and Evaluation Analysis can not be improvised ex post, it must be planned and programmed ex ante and must be directly related to the administrative and operative processes”

If PIA methodologies are not planned, developed, structured and observed, the evaluation will be weak and uncertain

Four Policy Models

Model 1: “Throwing money” on specific programmes, hoping that the

expected results, general and/or specific, will somehow be reached;

Model 2: Financing programmes that guarantee the attainment of precise

objectives and the implementation of planned activities, hoping that the

expected results will somehow be reached;

Model 3: Financing programmes that entail the enactment of specific

activities that are defined against specific and quantifiable objectives.

Model 4: Mod (3) + activities that can be quantified + monitoring and

evaluation ex-ante, in itinere ed ex-post (therefore during the entire cycle)

against both specific and general, measurable objectives and results.

Figure 1 – Typical Programme with no defined objectives

Management (Regulations and

Guidelines)

Resources (Allocation)

Action (Delivery)

[A]

Evaluation

Does [A] comply with Regulations and Guidelines?

What are the characteristics of [A]?

Figure 4 – Typical evaluation with measurable programme objectives

Policy (Goals and programme objectives)

Budget

Human Resources

Management (strategy &

supervision)

Administration (development &

control & intended impact

Data collection compliance check &

Feedback from stakeholders

Support & service & Analysis

Operations & delivery

(measurable objectives)

Outcomes /

Objectives

Data collection

Monitoring

Evaluation Testable assumptions, measures of progress

& feasibility

What may be evaluated?

Model 1: Respect of rules and procedures

Model 2: The attainment of the established objectives: have all steps been implemented? In compliance with norms and procedures? (i.e. expense flows, costs, timing)

Model 3: Did the implemented activities allow the attainment not only of the specific, but also of general objectives?

Model 4The evaluation of Results may be “Good/Bad” and “Efficient/inefficient” on the basis of criteria that may require:

• EVALUATION OF PROCESSES (INDICATORS)• EVALUATION OF RESULTS (INDICATORS)• EVALUATION IN TERMS OF SPECIFIC OBJECTIVES

Did the innovation policies favour R&D investments? (INDICATORS & ANALYSIS)

• EVALUATION IN TERMS OF GENERAL OBJECTIVES Did the innovation policies incentivise R&D investments? Did they have an impact on economic growth? (ANALYSIS & MODELS)

What Criteria and Which Tools may be used to perform evaluations?

In one word, the answer is: Many

Facts can be evaluated with quantitative and qualitative criteria (but which facts?);

The criteria depend on the objectives, on the time and on the aspects that must be evaluated;

“Indicators” are used for rankings, scoreboards and benchmarking analysis, as well as models (simple, complex, static and dynamic), and more in general the set of tools that in the USA are referred to as “Systematic Analysis”