Policy Evaluation

“..changing a policy does not guarantee improved performance.”

Kraft (2007)

Policy Evaluation

Goal is to determine whether a social intervention has produced the intended result.

Results are not always well received.

Probably the most difficult type of research to do well

Evaluate• Effectiveness: the policy works (?)

• Efficiency: the policy is not wasteful

• Other values– Equity– Fairness– Democratic– These values may often conflict– i.e. affirmative action, increases equality,

assuming it is effective, but is it fair?

POLICY EVALUATION

1. Specifying Goals

2. Identifying Unintended Consequences

3. Identifying Who Benefits/Who Suffers

4. Developing Methods and Measures of Evaluation: Qualitative, Quantitative, Mixed-Method

5. Examine both outputs and outcomes

6. Sabatier: probably takes 10 or more years to figure out the impact of a policy

Types of Evaluation

• Needs

• Evaluability Assessment

• Process

• Impact Analysis (summative evaluation)

• Efficiency Analysis– Cost-benefit Analysis– Cost-effectiveness Analysis

Types of Evaluation Research Designs:

Experimental designs Control/treatment group

Quasi-experimental designs Time-series design Correlational design

Qualitative evaluations Focus groups, interviews, Historical

Comparative

Ethical Issues: Social interventions being evaluated may raise ethical issues.

Identification of drug users, sex offenders

Deciding who gets treatment and who doesn’t. (A study, published 11/23/10 by the New England Journal of Medicine, found men taking Truvada, a common combination of two antiretroviral drugs, were 44 percent less likely to get infected with HIV than an equal number taking a placebo)

Research may be a mask for unethical behavior, i.e. Alabama treatment of black men with syphilis – Tuskegee (1932-1972)

Are Evaluations simply used to legitimize a social intervention?

How are the evaluations used?

THE POLITICS OF POLICY EVALUATION 1. Who Evaluates?

Elites: Government Agencies, Public Universities, Think Tanks

Interest Groups: AAA, ACLU, AMA, Business groups, etc.

Citizens: Complaints; Whistle Blower laws, Public Opinion; Voting; Recalls

Some agencies may be shielded from evaluations, some may have repeated evaluations thrust upon them.

2. Evaluation and Bureaucratic Oversight

(at least) Two reasons for evaluation:

Are the policy solutions working

Are the policy solutions really being implemented

3. How is Evaluation done?

Haphazardly: takes time and money

Fire alarm v. policy patrol model of oversight

4. Why do governments usually fail at their own efforts at policy evaluation?

5. Why are government programs rarely terminated, even when evaluations show that the programs are ineffective?

6. Interpretations of Evaluations (i.e., is the EPA doing a good job?)

WHY EVALUATIONS MAY BE INGORED

Too Complicated: Implications may not be presented in a way that nonresearchers can understand.

Results can sometimes be counter-intuitive = Results sometimes contradict deeply held beliefs.

Vested interest in a program.

Who evaluates government policies? Answer: often the government evaluates itself, i.e.

GAO.

Environmental Policy: Is it a Success?

Conservative view: - government over regulates

- government placates to the public

- public hypersensitive to risk

Liberal view: measures of success?

- government turns a blind eye

- government ignores public concerns

- public ill informed about risk

• Political evaluations of policies are not just based on effectiveness or efficiencies

• Often based on perception of immediate and personal risk or on the measurability of success

• Global warming (not an immediate threat)• Lead in consumer goods (immediate threat)• Endangered Species Act (not a personal threat,

not easy to measure effects)• Does support for environmental policy change if

the policy effects prices of consumer goods (cars, gas, paper, lumber) or is the public willing to pay higher prices to protect the environment.

“Being Dismal in Policy Analysis or Learning to Love Policymaking”

• Max Neiman (2000). Defending Government: Why Big Government Works.

• Modern democratic policymaking is an ugly process, but….

• Brings in lots of different perspectives and ideas.

• Forces compromise and deliberation