**** policy analysis for sustainable development - complexities and methodological response

8/3/2019 **** Policy Analysis for Sustainable Development - Complexities and Methodological Response

1/20

PaperfortheWorkshoponComplexityandPolicyAnalysis,Cork,Ireland 2224June2005

Policyanalysisforsustainabledevelopment

Complexitiesandmethodologicalresponses

HensRunhaar1,CarelDieperink,PeterDriessen2

AbstractPolicymakerswhoareinvolvedinthepromotionofsustainabledevelopmentoperate inahighlycomplex

policycontext.Apartfromcomplexitiesthatarerelated totheconceptofsustainabledevelopmentandto

theknowledgeavailableontheissue,complexitiesareraisedduetothedependencyonstakeholderswithin

thestate,

market,

and/or

civil

society.

In

short,

policy

makers

have

to

develop

policy

alternatives

that

are

bothsatisfactory intermsofproblemsolvingandthatcancountonsufficientsupportfromstakeholders.

This implies thatpolicyanalysts,whosupportpolicymakers,mustbeable toproduceknowledge that is

scientificallyvalid,relevant to thepolicydebate,andacceptedbystakeholders.Thispaperfocuseson the

policy analysis toolbox that environmental social scientists should have available.We discuss how this

toolboxshouldlookandhowitshouldbeutilised.

Keywords: the philosophy of policy analysis; sustainable policy development; robust policy

analysis.

1. IntroductionComplexitiesrelatedtosustainabledevelopment

The Brundtland Commission defined sustainable development as development that

meetstheneedsofthepresentwithoutcompromisingtheabilityoffuturegenerationsto

meet their own needs (WCED, 1987). This entails various complexities that policy

makerswillhavetodealwithtoattainthisobjective.Thefirstcomplexityresidesinthe

needtooptimisethreevalues(i.e.ecological,social,andeconomic)atthesametime.It

also resides in the observation that humanbehaviour (including public policy) often

affectseach

of

these

values

simultaneously.

1 Corresponding author. Contact data: dr.H.A.C. Runhaar, Environmental Studies& Policy,

CopernicusInstitute forSustainableDevelopmentandInnovation,UtrechtUniversity,P.O.Box

80115, 3508 TC Utrecht, phone: ++31302533050; fax: ++31302532746; email:

[email protected],dr.C.Dieperink,andProf.dr.P.P.J.Driessen,EnvironmentalStudies&

Policy,CopernicusInstituteforSustainableDevelopmentandInnovation,UtrechtUniversity.

1
mailto:[email protected]:[email protected]


2/20


A second source of complexity is causedby knowledge gaps that exist in the policy

domain of sustainable development (see Kates et al., 2001, for an overview) and

structuraluncertaintiesinthisarea(theongoingdebateabouttheextenttowhichmen

contributes to globalwarming is a typical example see for instanceMcIntyre and

McKitrick,2005a;2005b).

A third source of complexity is the need to operate in a multiactor policy context.

Typically, the resources required for successful policy implementation (e.g. money,

information, support) tend tobe spread over actors other than (central) government

(Glasbergen,1998).Someoftheseactorsarefoundwithinthestateapparatus,duetothe

typical organisation of public administration into distinct policy domains. Other

relevant actors are found in other domains of society (themarket and civil society).

Giventhestatesdependencyontheseactors,governmentsoftenhavetonegotiatewith

them and engage them in processes of policy development, implementation, and

evaluation (Rhodes, 1997; Van de Riet, 2003). Yet the actors from various domains

typicallyperceive

agiven

problem

differently.

Moreover,

they

have

the

potential

to

block problemsolving activities. Policymakers thus face the challenge of finding a

balance between policy alternatives that are both satisfactory in terms of problem

solvingandthatcancountonsufficientsupportfromstakeholders(Bovensetal.,2001).

Governingsustainabledevelopment

The multiactor policy context implies that strategies for sustainable development

usually have to relate to a form of governance: a nonhierarchical form of steering,

wherestateandnonstateactorsparticipate in the formulationand implementationof

publicpolicy(Rhodes,1997).Thissteeringmodelcontradictsthetraditionalhierarchical

modelof

government.

In

the

latter,

the

decisive

power

is

centralised,

located

in

one

policymakerorgroupofpolicymakerswithsimilarobjectives.

Governancehasmany faces.Dependingon the typeofactors thatare (orhave tobe)

involved,adistinctioncanbemadebetweenmultilevelgovernance(highlightingthe

varioustiersofgovernment:local,regional,national,orsupranational)andmultiactor

governance (reflecting the involvement ofboth public and private actors) (Driessen,

2005). Inboth cases multisector governance is required aswell: the integration of

environmentalobjectives innonenvironmentalpolicydomains (LaffertyandHovden,

2003). In practice, the various forms of governance have become concrete in non

hierarchicalarrangements

such

as

agreements,

covenants,

or

partnerships

between

publicandprivatepartiesandamongmarketpartiesandcivilorganisations.

Policyanalysisforsustainabledevelopment

In the context of governance for sustainable development, environmental social

scientists supportpolicymakers through their traditional roleof informationprovider

bymeansofpolicyanalysis.Policyanalysisreferstotheuseofanalyticaltechniquesand

2


3/20


knowledgeforand inpolicymaking(Parsons,1995:xvi).Itsaimistosupportpolicy

makers by producing and transforming policyrelevant information (Dunn, 1981;

Fischer,1997;VandeRiet,2003).Governanceforsustainabledevelopmentplacessome

specificdemandsonpolicyanalysis.Oneoftheseisrelatedtotheobservationthatoften

stakeholderseitherdonotaccepttheresearchoutcomesortheyusetheseoutcomesina

strategicway

(see

for

instance

Bras

Klapwijk,

1999).

In

part,

this

can

be

attributed

to

uncertainties that cannotbe addressed adequatelyby science alone (Funtowicz and

Ravetz,1993).Anotherexplanation is thatmanypolicyanalyticalstudiesarebasedon

singlevaluemethodsthatfocusononeorafewoperationalcriteria(e.g.efficiencyand

effectiveness)and that largely ignoreothercriteriaorvalues (costbenefitanalysis isa

typical example). In thisway themethods do not always reflect the values that are

involvedinapolicyarea.Thismayprovokeoppositionfromgroupsofstakeholders(De

Jong, 1999;Van deRiet, 2003). Policy analysis for sustainable development therefore

requiresspecificcompetencies.

Goaland

structure

of

the

paper

In thispaperwefocusonthepolicyanalyticalcompetencies thatenvironmentalsocial

scientists are expected to possess. In particular, we discuss the basic methods that

shouldbeinthetoolboxofenvironmentalsocialscientists.InSection2wepresentfive

methods thatprovideanswers to themainpolicyquestions regardinggovernance for

sustainable development. These methods are described in more detail in Sections 3

through7.Notonlywillwediscussthestrengthsandweaknessesofthesemethods,we

considerthe implicationsofthemultiactorpolicycontextfortheirapplicationaswell.

WeoffersomeconcludingremarksinSection8.

2. BasicmethodsofpolicyanalysisforsustainabledevelopmentMethodsofpolicyanalysis:definition

Ingeneral terms,policyanalysiscanbedefinedas analysis forpolicy (Parsons,1995:

xvi).Thisdefinitionisratherbroad;itcoversnumerousmethodsfromawiderangeof

disciplines. In thispaperwenarrow itdownby applying an additional criterion: the

researchhastohaveaclearlinkwithpolicy.Thisleavesoutstudieson,forinstance,the

functioningofecosystemscommissionedbyadepartmentofagriculture.Butitincludes

research into the effectiveness of a policy that aims at restoring ecosystems in a

particulararea.

Methods,ingeneralterms,arewaystofindanswerstoquestions(thetermmethodis

derived from the Greek word methodos, which literally means a way or path of

transit). In the search for answers several choices have tobemade, e.g. on research

design,data collection,data analysis, and reporting.Amethodnotonly specifies the

researchquestions;italsodescribesaparticular,coherentsetofdecisions(orrules)that,

according tocertainobjectivesorcriteria,willyield thebest results (Swanborn,1987).

3


4/20


Methods thus are standardised procedures for producing answers to particular

questions.

Theterms methodand researchstrategyareoftenusedassynonyms.Yin(1994),for

instance, defines a research strategy as a distinct way of collecting and analysing

empiricalevidence

in

alogical

sequence

that

connects

the

empirical

data

to

astudys

initial researchquestionsand,ultimately, to itsconclusions (Yin,1994:3,19).Typical

research strategies used in social science include experiments, surveys,histories, case

studies, and analyses of archival information (Yin, 1994). Other authors, however,

considermethods as distinct types of research strategies or specific operationalisations of

elements thereof (e.g. Swanborn, 1987: 41). The boundaries between methods and

researchstrategies tend toblur.Asaruleof thumb,weconsiderresearchstrategiesas

beingmore general thanmethods.We subsequently distinguishmethods from data

collection techniques, which are specific ways to gather empirical data (examples

includeinterviewsandobservation).Often,amethodcanbebasedon,orcomplemented

by,multiple

(alternative)

data

collection

techniques.

Again,

the

boundaries

between

methodsanddatacollectiontechniquesarenotalwayssharp(seeFigure1).

Figure1: Researchstrategies,methodsanddatacollectiontechniquesinperspective

Researchquestions

Research

strategy

Method MethodAnswers

Data Data Data

collection collection collection

technique technique technique

Empiricaldata

Keyresearchquestionsandmethodsofpolicyanalysisforsustainabledevelopment

Identification of the basic methods of policy analysis in the domain of sustainable

developmentrequiresinsightintothebasicresearchquestionsinthisfield.Inlinewith

LeroyandNelissen (2000)wedistinguishbetween five relevant themesaroundwhich

researchquestionscanbeformulated,namely:

4


5/20


Policy content:Someexamplesofquestionsare:Whatare thepolicyobjectives?(e.g. What is meant by decoupling or integrated policy?); What problem

definitions underlie policy objectives? (e.g. Why is global warming

problematic?);andHowvalidaretheassumptionsuponwhichthepolicyrests?

(e.g. Is thereanyevidence that roadpricinghasa substantial impacton travel

behaviour?).

Policyprocess:e.g.WhatistheinfluenceofNGOssuchasGreenpeaceonpoliticalagendasetting?,andHowhavepolicyprocessesevolvedovertime?

Policy organisation: e.g. Which policy domains are involved in the issue ofsustainabledevelopment?andHowistheimplementationofEUenvironmental

directivesorganisedinthevariousmemberstates?

Policyeffects:e.g.Hasthepolicyresultedintherealisationofitsobjectives?,Arethere any (severe) sideeffects?, Do stakeholders evaluate the policy effects

similarlyornot?,andWhatexplainssuccessorfailureofthepolicyunderstudy?

Policycontext:e.g.Howisthepolicycontentaffectedbypolitical,economic,andcultural

developments?

(Think

of

the

impact

of

the

trend

of

mass

individualisationonenvironmentalpressure,ortheplace andtimespecificityofparticularpolicyproblems,suchasclimatechange.)

Environmentalscientistsshouldbeequippedwithatoolboxofmethodsthatcoverthe

main questions related to the above themes.Over the years, numerousmethods of

policyanalysishavebeendeveloped(Geurtsetal.,1989;Mayeretal.,2004).However,in

ordertoanswerthetypesofquestionslistedabove,thefollowingfivemethodsformthe

minimalmethodologicalequipmentforenvironmentalsocialscientists:

Reconstructionofpolicytheory Stakeholderanalysis Impactassessment Costbenefitanalysis Discourseanalysis

Table1showshowthefivemethodsrelatetotheabovementionedthemes.Sometimes

themethods overlap;both stakeholder analysis and discourse analysis, for instance,

provide insight into the problem perceptions of stakeholders (albeit from a different

perspective). Inaddition, the fivemethodsarecomplementary.Areconstructedpolicy

theoryandastakeholderanalysis,forinstance,provideelementsthatcanbeusedinan

impactassessmentinordertomeasureandexplainthe(side)effectsofapolicy.Inturn,

animpact

assessment

can

be

an

input

for

acost

benefit

analysis.

5


6/20


Table1: Policythemesaddressedbythefivemethodsofpolicyanalysis

Theme1(content)

Theme2(process)

Theme3(organisation)

Theme4(effects)

Theme5(context)

Reconstructionofpolicytheory

Stakeholderanalysis

Impactassessment

Costbenefitanalysis

Discourseanalysis

Inthefollowingsections,webrieflydiscusseachofthesemethods.Weaddressamong

otherthingstheirstrengthsandweaknessesfromascientificandpragmaticperspective

(e.g. reliability,validity, and costliness) and their recommended application inmulti

actorpolicycontexts.

3. ReconstructionofpolicytheoryFocusofthemethod

Apolicytheorycanbedefinedasthetotalofcausalandotherassumptionsunderlying

apolicy(Hoogerwerf,1990:285)andrepresentstheconceptionofwhatmustbedone

tobringabouttheintendedsocialbenefits(Rossietal.,2004:134).Reconstructionofa

policytheoryfocusesonelucidatingthelogicorreasoningbehindapolicyprogramme

byexamining(Hoogerwerf,1990:286):

Finalrelations:relationsbetweenobjectivesandmeans Causalrelations:relationsbetweencausesandeffects,mainlyrelatedtothepolicy

problem. For instance: the climate problem is partlybased on the (assumed)

impactofhumanactivitiesonglobalwarming

Normativerelations:relationsbetweenprinciplesandnormsmutuallyorbetweenprinciplesandnormsontheonehandandexistingorexpectedsituationsonthe

other. Normative relations will, among other things, explain the problem

perceptionsuponwhichthepolicyrests

Themethod typically focuses on the content of a policy (theme 1 from Section 3). It

addresses questions such as the following:What argumentation underlies the policy

problemdefinition?,

How

did

the

policy

makers

envisage

that

the

instruments

chosen

wouldsolvetheproblem?,andWhatassumptionsdidtheyhave?Theanswerstosuch

questions arenot only interesting in themselves,butmay also explain the success or

failureofapolicy, i.e. itseffects (theme4).Often, incorrectassumptionsexplainwhya

policy fails (Hoogerwerf, 1990).Themethod can alsobeused to explain otherpolicy

aspects,suchas thechoiceof instruments (the impact theyaresupposed tohave,will

6


7/20


providesomejustification)and thepolicygoal (whichwillbe related to theperceived

policyproblem).

Avariantofthemethodforreconstructingpolicytheories,aselaboratedbyRossietal.

(2004),takes intoconsiderationnotonlythepolicycontentbutalsosomeorganisational

aspects(theme

3):

Howwill the targetpopulationofapolicymeasurebe reached?Andhowandwhenwilltheprogrammebeterminated?

Which programme resources, personnel, administration, and generalorganisationareneededinordertoimplementtheprogramme?

Strengthsandweaknesses

Strengthofthemethodisthatitisanimportantresourceforinterpretingandexplaining

thequalityofapolicyaswellasitseffectiveness(Hoogerwerf,1990;Rossietal.,2004).A

more pragmatic strength is that the method allows the parties to gain a basic

understandingof

acomplex

matter

(i.e.

apolicy

programme)

in

arelatively

short

period

oftime.

However, reconstructionofpolicy theoryoverestimates the rationalcharacteristicsof

policymakingandunderestimatesaspectsofpolitics,socialinteraction,andconflicting

rationalitiesand interests. Inaddition, thepolicy theoryof thepoliticaladministrative

elitemaydifferfromthepolicytheoryofimplementerswhoputthepolicyintopractice

(Hoogerwerf,1990).Themethodmayfailtorevealtothepublicexpostrationalisation

ofdecisionsthathavebeentakeninanirrationalway(e.g.viacompromises).Finally,

theassessmentofthequalityofapolicytheoryneedsfurtherelaboration(Hoogerwerf,

1990).Scientific

standards

(validity,

reliability)

may

not

be

appropriate

in

light

of

the

subjective approach fromwhich reconstruction of policy theory starts. In addition,

highqualitypolicytheoriesmaybeunsuccessfulintheirimplementation.

Recommendedapplicationinmultiactorpolicycontexts

Themethodlargelyignoresstakeholdersotherthanthepoliticaladministrativeelites.If

themethod is applied in order to gain insight into the chance of success of a policy

programme,anditisreasonabletoexpectthatthepublicauthoritiesinquestioncannot

command implementationof thepolicy ina topdownmanner, thepolicy theorywill

notbeasufficientsourceof information.Thisproblemcanbeovercomebycombining

themethod

with

astakeholder

analysis

(see

Section

4),

which

gives

insight

into

relevant

stakeholders,theirproblemperceptions,andsupportforthepolicyprogramme.

Ifthemethodisappliedinamultiactorsettinginordertosupportpolicymakers(e.g.

foranexanteassessmentoftheassumptionsofaproposedpolicy),theproblemmaybe

thatstakeholdersdonotaccept theoutcomesof thestudy (inparticular if theydonot

recognisetheirownperceptionsofthepolicyissue).Thisproblemcouldbeovercomeby

7


8/20


stakeholder participation during the research. (For an overview of various forms of

stakeholderparticipation,seeMayer(1997).)Stakeholders inputcanbeusedtosetthe

boundariesofthepolicyunderstudyortospecifyevaluationcriteria.

4.Stakeholder

analysis

Focusofthemethod

Stakeholderanalysisaimsateliciting themainactors thathaveastakeor interest ina

particularpolicyproblem.Actorsbecomestakeholderswhentheycontributetoapolicy

problem, are needed for solving the problem, or are affected by problemsolving

activities.

Usually, stakeholder analysis focuseson aspects such as the interestsof stakeholders,

theirperceptionsoftheproblem,theirpositionsandrelationshipswithotheractors,and

theresources

that

they

control.

The

method

offers

answers

to

research

questions

such

as

thefollowingones:

Howarestakeholdersinvolvedintheproblemarea?,Whataretheirinterestsinsolvingormaintainingtheproblem?,andHowdotheyperceivetheproblem?

How relevant are these actors (in terms of critical resources: power, support,authority,andthelike)?

Whichpolicyalternativeissupportedbymoststakeholders? Whateffectsdoordidthestakeholdersexpectfromthepolicyprogrammeorits

alternatives?

Howare the stakeholders related?Which (coalitionsof)actorsmay supportorblock

problem

solving

activities?

The answers to these questions shed some light onwhy the apriorimost effective

solution available was not chosen. This may simply be due to lack of support. In

addition,astakeholderanalysisprovidesbackgroundinformationforunderstandingthe

dynamicsinpolicyprocesses.Shiftingpolicygoalsmaybeunderstoodinthecontextof

stakeholders interestsandpower.Astakeholderanalysismayalsoprovideanexante

evaluationoftheeffectivenessofapolicyprogrammebyassessingtheextenttowhich

the programme and the interests of stakeholders coincide (see for instance Crvers,

2001). Finally, and in linewith this, a stakeholder analysismaybe used in order to

improveaprogrammeintermsoflegitimacyaswellasquality.Suchimprovementscan

bemade

if

the

values,

perceptions,

or

experiences

of

stakeholders

are

incorporated

(Grimble, 1998). The method therefore covers all the five themes of policyrelated

questionslistedinSection3.

Stakeholderanalysiscanbeconducted inastaticway(asinglesnapshot)or inamore

dynamicway (by analysing changes in an actor network over a particular period of

time).Different levels of analysis canbe employed. Traditional stakeholder analysis

8


9/20


focuses on attributes of individual stakeholders. (Policy) network analysis focuses on

networksofinterdependentactors,givingspecialattentiontorelationshipsandpositions

ofactorsinthesenetworks(e.g.Kickertetal.,1997;Rowley,1997).

An important issue is identifying the boundaries of the network (or group of

stakeholders)under

study

(Rowley,

1997).

In

many

situations,

the

whole

world

(includingfuturegenerations)mayhaveastakeinthepolicyproblemunderstudy(e.g.

exhaustionofnatural resources). Inpart, thenetworkwillbedemarcatedonpractical

grounds.Itsboundarieswilldependonthepurposeofthestudy(e.g.toexplainpolicy

choices or explore potential problems during policy implementation by a certain

agency).


Strength of themethod is itsbroad focus: it canbeused to address awide range of

questions (seeabove).Theadvantagesofusing theresultsofastakeholderanalysis in

policyprocesses

have

been

described

above.

An

additional,

pragmatic

advantage

is

that

astakeholderanalysiscanberelativelycheapintermsofmoneyandtime.

Some characteristics of stakeholder analysis and policy network analysis are the

following:

No uniform definition of stakeholders and policy networks is given in theliterature(Mitchelletal.,1997;VanTatenhoveandLeroy,1995)

Interpretationofdataissubjective.Supportfororobjectiontoaparticularpolicycanhardlybemeasured

Although a stakeholder analysis may provide some explanation for policychoices,

the

link

between

these

choices

and

the

stakeholder

analysis

is

difficult

to

establish unambiguously. An option is to compare the ideal policy goal

formulation or the optimal policy instruments (in light of the fundamental

causesofthepolicy issue)withtheactualones;anydiscrepanciesmaypointto

compromisestostakeholders


A stakeholder analysis is based on the assumption that the actors in the policy

environmentareimportantfor(understanding)policymaking.Themethodthusiswell

suited for use in multiactor policy contexts. Stakeholder participation during the

researchprocess

or

interviews

with

stakeholders

are

valuable

for

identifying

relevant

stakeholders and their interests and perceptions. Not only does this source of

informationrequirelessinterpretationonthepartoftheresearcher,itmayalsoresultin

ahighercommitmentofstakeholderstotheoutcomesoftheanalysis.

9


10/20


5. ImpactAssessmentFocusofthemethod

ImpactAssessment(IA)coversawiderangeofmethodsfordeterminingtheeffectsofa

policy

programme

(theme

4

from

Section

3).

The

overall

question

that

is

addressed

by

themethod is,Dopoliciesactuallyproduce the intendedeffects?Policyeffectscanbe

measuredintwoways.Oneistomeasurethedifferenceinasituationbeforeandaftera

policy intervention.Theother is toassess the situationafter the interventionwith the

situation thatwouldhaveoccurred if thepolicyhadnotbeen implemented (thebase

caseor businessasusualscenario).Figure2 illustratesbothapproaches.The latter is

mostinformativeabouttheactualpolicyeffects.Butitleadstopracticaldifficulties,asit

is hard to know for certain how thingswould havebeen if the policy had notbeen

implemented(Mohr,1992).

Figure2: Twoapproachesformeasuringeffectsforecastedtrend

Outcome withoutpolicy

variable

effect

actualtrendwithpolicy

effect

Beforepolicy Policyintervention Afterpolicyintervention

Time

ThereareseveralvariantsofIA.Somevariantsfocusonparticulartypesofimpacts(e.g.

social impact assessment, environmental risk assessment, and environmental impact

assessment). In addition, a distinction can be made between ex ante and ex post

evaluation (depending on whether or not the policy under study has been

implemented). In the former case, not the actual policy effects are assessedbut the

expectedor forecastedeffects.Lookingat thescientificrigourofthesemethods,which

largelydeterminesthevalidityoftheresearchresults,adistinctioncanbemadebetween

two types of IA: randomised experiments, and quasiexperimental research designs.

Randomised

experiments

use

groups

of

participants

that

are

randomly

sorted

into

at

leasttwogroups.Oneisthecontrolgroup,anditdoesnotreceiveanintervention(ora

placebo); the other (the intervention group) does. Outcomes are then observed and

differencescanbecompletelyattributedtotheintervention.Themaincontrolcondition

ofrandomisedexperimentsisthatparticipantsarerandomlyassignedtothecontroland

intervention groups. In this way the groups are similar in terms of composition,

predispositions,andexperiences;thentheonlythingthatdiffersistheintervention.All

theremaining impactassessmentdesignsconsistofnonrandomisedquasiexperiments

10


11/20


inwhich the intervention group and controlgroups are onlypresumed tobe similar

(Rossietal.,2004).Thecontrolgroupmayforinstanceconsistofpopulations,firms,or

whatever from abroad, or the target group before the intervention. Designs using

nonrandomised controls universally yield less convincing results thanwellexecuted

randomised field experiments (Rossi et al., 2004). Nevertheless, in practice quasi

experimentsare

often

conducted

due

to

practical

considerations

like

time,

money,

or

ethics.


AgeneralstrengthofIAisthatitisexplicitlyaimedatassessingcausalityandcanthus

beusedtoassesstheeffectivenessofpolicy.Inaddition,IAhasarigorousbasis;overthe

yearsthemethodologyhasbeenrefinedinordertomaximisethevalidityandreliability

ofthemethod.Finally,ideallyanIAstudyisolatestherelativeimpactofvariablesandin

thiswayprovidesagoodbasisforadjustingpolicyprogrammes.

Thereare

also

several

weaknesses.

One,

in

practice,

is

that

randomised

experiments

in

policy studies are often impossible to conduct due to time ormoney restrictions or

ethical considerations. Another is that assessments cannot (always) be made with

certainty, onlywith varying degrees of plausibility (Rossi et al., 2004).An important

reasonisthatitisnotalwayspossibletoassessthebusinessasusualscenarioagainst

whichtheeffectsoftheprogrammeunderstudyareassessed(Rossietal.,2004).Thirdly,

IAoftendoesnotassesspolicyeffectsinthelongterm,andthereforethesustainability

ofpolicyeffects isnotalwaysknown.Amorepracticalproblem isthatgoalsareoften

vaguelydefined,contradictory,orshiftingduringpolicy implementation,whichposes

operationalisationproblemsfortheresearcher(VerschurenandZsolnai,1998).


Due to its explicit focus on policy outcomes, IA implicitly overestimates the goal

orientationofpolicymakersandpoliticiansandusuallyunderestimates theprocessof

policymaking.The intrinsicvalueofdevelopingaprogramme incollaboration isthus

ignored. In addition, itpresumesuniformity in goals andmotives among thepeople

involved.Butthatassumptionisnotrealistic,particularlynotinmultiactorcontextsor

insituationswhereduringtheimplementationchangesaremadeinordertocopewith

unforeseen problems (Verschuren and Zsolnai, 1998). In addition, a focus on policy

goalsmay draw the attention of the researcher away from sideeffects (e.g. injustice

causedby

the

policy)

or

from

the

basic

question

of

whether

or

not

the

policy

goals

are

appropriateinlightofsocialnormsandvalues(Fischer,1997;VerschurenandZsolnai,

1998).ThesecharacteristicsmakeanIAinitsclassicalformlessappropriateforuseina

multiactorcontext.Yet,thisdoesnotmeanthemethodshouldbeignored.Itsrigorous

methodologicalbasemakesitanadequatemethodforassessingpolicyeffects.Itismore

adequatethanastakeholderanalysis,whichcanalsobeusedtoexplorehowanongoing

policyprogrammemayworkoutinfuturebutwhichrequires(more)interpretationby

11


12/20


theresearcher.Forasuccessfuluseinamultiactorcontext(i.e.whentheresultsareto

be accepted), some adaptations are required.One is that a stakeholder analysis can

precedeanIAinordertoprovideanoverviewofthemainactorsthatwillbefacedwith

the effects of a policy and to show how they perceive that. These impacts can be

specified and quantified in an IA. In thisway the (side)effects for stakeholdersmay

becomeclear.

A

second

adaptation

is

that

stakeholder

participation

during

the

IA

study

mayresultinasetofpossible(side)effectsthatareconsideredrelevantbystakeholders

(thismay for instance include the effects on collaborationbetween government and

otherstakeholders).Stakeholdersmayalsobeasourceofinformationwhenitcomesto

the operationalisation (or estimation) of effects (e.g.bymeans of aDelphi survey or

focusgroup).Stakeholderparticipationmaysubsequentlyresult incommitment to the

study results aswell as to the problemsolving activity that isbeing evaluated (e.g.

Hisschemlleretal.,2001).

6. CostBenefitAnalysisFocusofthemethod

Costbenefitanalysis(CBA)focusesonevaluating(i.e.attachingavalueto)theeffectsof

(proposed)policyprogrammes (theme 4 fromSection 3). It canbeused todetermine

whetherornotthebenefitsofan investmentorapolicyoutweighitscosts.CBAhasa

verybroadscope,sinceitmayexpressallpositiveandnegativeeffectsofanactivityina

common unit, namely money, from a social as opposed to a firms point of view

(Wrisberg and Udo de Haes, 2002). Typically, costs and benefits are expressed in

consumerpreferences, specifically in thewillingnesstopay for the goodsor services

that

are

evaluated.

CBA can be used in order to find out whether or not policy choices have been

worthwhile,whetherornotthemostefficientpolicyprogrammeshavebeenchosen,or

howsimilar,futureprogrammescanbetransformedintomoreefficientones(Rossietal.,

2004).Thisisusuallydoneonanexpostbasis(i.e.afterpolicyimplementation).Exante

analysiscanbeusedtoprioritiseprogrammesaccordingtotheirbenefittocostratio.


The main strength of CBA is that it allows for evaluating various types of effects

(tangibleand

intangible,

direct

and

indirect)

by

translating

them

into

one

(monetary)

unit. In addition, over the yearsmany sophisticated procedures forCBA for specific

applications have been developed. In the Netherlands, for instance, a standardised

procedure for conducting CBA hasbeen developed which is compulsory for every

investmentintransportinfrastructurethatisunderconsideration(seeEijgenraametal.,

2000,orDeJongandGeerlings,2003,foradiscussion).Thesestandardisedprocedures

preventasmuchaspossiblemethodologicalerrorssuchasdoublecountingandmake

surethattheoutcomesofdifferentCBAsarecomparable.

12


13/20


CBAhas severalmethodologicalaswellaspragmaticweaknesses.Forone thing, it is

verydata intensive.Often, theunavailabilityor inaccuracyofdatawillplaceaserious

limitationontheusefulnessofthemethod(Rossietal.,2004;WrisbergandUdodeHaes,

2002).Byextension,whenmajorcostsorbenefitsaredisregardedbecausetheycannotbe

measured or monetised (e.g. the extinction of a particular animal or income

redistributionaleffects),

the

project

may

appear

less

or

more

efficient

than

it

is

(Rossi

et

al., 2004). Therefore, such costs and benefits should be added to the list, albeit in

qualitativeterms.Thirdly,CBAisnotappropriatewhenaprogrammeisnotyetbeyond

thedevelopmentstateorwhenuncertaintyremainsabouttheeffects(Rossietal.,2004).

Fourthly,thetreatmentofindirecteffectsissubjecttodebate,sinceitisoftendifficultto

assess towhat extent they are included in themeasurement of direct effects (which

raises the risk ofdoublecounting). Finally, andmore fundamentally, it is questioned

whetherconsumerpreferencesareaproperbasisforthevaluationofeffectsonnature

(e.g.theextinctionofspecies)(VanWee,2003;Fischer,1997).

Recommendedapplication

in

multi

actor

policy

contexts

CBAiswellsuitedforsingleactorpolicycontextsbecause,initstraditionalform,ithas

onetarget(efficiency).Themethod ignoresvaluesotherthanefficiency,suchasequity

(welfaredistribution)orthequestionofwhetherornotapolicyhasbeeneffective,i.e.has

reached its targets or not (Fischer, 1997). The neglect of other valuesmay endanger

support for the outcomes of a CBA. (This is particularly so when the CBA is an

important input to decisionmaking on investments, as in the case of transport

infrastructure in theNetherlands.) Yet this is not the only reasonwhyCBAmaybe

problematicinmultiactorpolicycontexts.CBAisalsoacomplexmethodandnoteasy

fornoneconomiststounderstand (WrisbergandUdodeHaes,2002;VanWee,2003).In

addition,CBA

overemphasises

the

importance

of

policy

outputs

as

compared

to

the

processofpolicymaking(Fischer,1997).Anotherdifficultyisthatnotalleffectscanbe

monetised(e.g.becausemarketpricesarelackingorbecausemonetisationissubjectto

criticism;thinkofquantifyingthevalueoflife).

Inpart,theabovedrawbackscanbeovercomebymethodologicaladaptations.Incases

where the benefits of a policy programme cannot be valued unambiguously, cost

effectiveness analysis (CEA) offers a good alternative. CEA calculates which policy

option produces or produced the desired beneficial effects (e.g. reduction in CO2

emissions) at the lowest costs. Only the costs are monetised. Regarding its lack of

transparencyto

non

economists,

clearer

communication

with

stakeholders

is

required.

The focusoncostsandbenefits(moregenerally,onpolicyoutputs)canbejustifiedon

thebasisthattaxpayersdesirevalueformoney.Bymeansofstakeholderparticipation,

stakeholders can contribute to thedefinitionof relevant costs andbenefits and to the

wayinwhichtheyareincorporatedintheanalysis(operationalisation,monetisationor

not).Finally,theequityconsequencesofaprojectcanbemadevisiblebyspecifyingcosts

andbenefitredistributionaleffectsforvariousgroupsofstakeholders(forthispurpose,

astakeholderanalysismayprecedetheCBA).

13


14/20


7. DiscourseanalysisFocusofthemethod

Discourse analysis focuses on theways inwhich (groups of) actors givemeaning to

particular

phenomena

(e.g.

a

policy

programme

or

the

causes

of

a

policy

problem).

The

methodisbasedonthepremisethatthewaypeopletalkisnotaneutralreflectionofour

world, identities, and social relations. Rather, it plays an active role in creating and

changing them.Within aparticularworldview, some forms of actionbecomenatural

whereas others become unthinkable. Different social understandings of the world

therefore lead to different social actions (Jrgensen and Phillips, 2002). Discourse

analysisexplorespatternsinwrittenorspokenstatementsaswellasrelatedpracticesin

order to identify the representationsof reality thatare employed. It also explores the

socialconsequencesofdifferentrepresentationsofreality(JrgensenandPhillips,2002;

Hajer,2005).

Thedefinitionofproblemsisnottakenforgranted.Infact,theaimofdiscourseanalysis

istoelucidatethevariousproblemperceptionsinvolvedinanissue(Hajer,2005).Thus,

themethodcanbeusedtoreconstructvisionson(social)problemsaswellasthecontext

inwhich theyemerged.Discourseanalysis canalso shed lightoncontroversiesabout

certainissues.Itcanexplainwhyaparticularunderstandingofaproblematsomepoint

gains dominance and is seen as authoritative, while other understandings are

discredited(Hajer,1995).

Therearetwodistinctformsofdiscourseanalysis:

Thelinguisticorientedtradition,whichfocusesonlanguageandwhatlanguageis

used

for

(see

for

instance

Georgakopoulou

and

Goutsos,

1997)

Thebroadertradition,focusingondiscoursesinamoregeneralsense.Here,theanalysis focusesbothon thewaysof thinking and arguing on specific (social)

themesorissuesandontherelatedpractices,structures,andinstitutions(seefor

instanceHajer,2005)

Intheareaofpolicyanalysis,thelatterformofdiscourseanalysisdominates.Itcanbe

usedtoanswerquestionssuchasthefollowing:Howhavepolicyprocessesevolvedand

why?Whatargumentsunderlie thedecisionsmade?Whatmeanings,arguments,and

lines of reasoning did the various (groups of) actors involvedbring forward?What

advocacycoalitions

can

be

discerned?

And

what

were

controversies

or

conflicts

about?

Thesequestionscanbegroupedunderthethemes1,2,4,and5fromSection3,which

illustratesthatthemethodissuitableforabroadsetofquestions.


A discourse analysis can reveal opinions and attitudes and underlying value

judgements.Thus,itisveryappropriateforanalysingthesubjectivityofcertaintextsor

14


15/20


sources.Forthispurpose,variousstandardisedprocedureshavebeendevelopedinthe

linguisticoriented tradition of discourse analysis. Another strength of the broader

tradition of discourse analysis is that it can structure complex scientific or political

debates.However, in this traditiona standardisedprocedure fordiscourseanalysis in

policy issues is largely lacking, which results in the danger of subjectivity. The

researchertherefore

should

be

explicit

and

clear

in

his

or

her

selection

of

material

and

interpretation(Burman,2003).Inaddition,theanalystshouldtrytotriangulateasmuch

aspossible.

The limitedgeneralisabilityof the resultsof thediscourseanalysiscanbe regardedas

anotherweakness.Thepostmodern tradition inwhichdiscourseanalysis fitssuggests

theabsenceofuniversaltruths;representationsoftheworldarethereforeunique.This

implies that theopportunities todraw (policy) lessons fromdiscourseanalysesareby

definition limited. Apart from this, according to the postmodernists, there are no

criteriabywhichtodistinguishthegoodresearchfromthebad.Thevaliditycriterion

commonlyemployed

in

science

is

not

applicable

in

discourse

analysis.

The

reason

is

that

it is based on the assumption that knowledge can reflect reality without bias, an

assumptionthatisrejectedaprioribydiscourseanalysis.

More practical questions include the following: At what level does one define a

discourse?, How can one delineate single discourses?, and How does one identify

discourses thatcompete in the samedomain? (JrgensenandPhillips,2002)A related

issueishowtodeterminewhenaparticulardiscourseisdominant.


Themethod

is

appropriate

for

studying

policy

making

processes

in

multi

actor

policy

contexts.That isbecause itpresumes the existence ofdifferentperceptions, opinions,

and,more generally, discourses that relate to amultitude of actors. Since discourse

analyses are primarily conducted for scientific reasons, the problems with policy

analysis in multiactor contexts described in Section 2 are not typical for discourse

analysis. The outcomes of discourse analyses do not have to be accepted by

stakeholders.Butitisprudenttodiscussthemwiththosestakeholderswhoseopinions

areexaminedinordertocheckiftheanalysisofthediscoursemakessense(Hajer,2005).

8. ConclusionsIn this paper we argued that policymakers who are responsible for promoting

sustainabledevelopmentoperate inahighly complexpolicy environment.Oneof the

sources of this complexity is the fact that policymakers typically need support from

stakeholders elsewherewithin the government orwithin civil society or themarket.

Policy analysts in this context often need to produce knowledge that is scientifically

valid,relevanttothepolicydebate,andacceptedbystakeholders.

15


16/20


17/20


BrasKlapwijk,R.(1999),Adjustinglifecycleassessmentmethodologyforuseinpublic

policydiscourse,Ph.D.dissertationDelftUniversityofTechnology,DelftUniversityof

Technology,Delft.

Burman,E. (2003),Discourse analysismeans analysingdiscourse: some commentson

Antaki,Billig,

Edwards

and

Potter

Discourse

analysis

means

doing

analysis:

acritique

of six analytical shortcomings, Discourse Analysis Online (Available

http://extra.shu.ac.uk/daol/;viewedMay27,2004).

Crvers, R.J.M. (2001), Netwerksturing bij natuurontwikkeling. Bestuurskundige

consequentieanalyse van gebiedsgerichte projecten (Network steering in nature

development. Public administrative impact assessment of areaspecific projects; in

Dutch),Ph.D.dissertationUtrechtUniversity,ShakerPublishing,Utrecht.

Driessen, P.P.J. (2005), Restructuring the Dutch countryside: limits of a governance

strategy,Planning,

Practice

&

Research

(forthcoming).

Dunn,W.N.(1981),Publicpolicyanalysis,PrenticeHall.London.

Eijgenraam,C.J.J.,C.C.Koopmans,P.J.G.Tang,andA.C.P.Verster(2000),Evaluationof

infrastructural projects; guide for costbenefit analysis (Part 1: Main report), CPB

NetherlandsBureauforEconomicPolicyAnalysis/NetherlandsEconomicInstitute,The

Hague/Rotterdam.

Fischer,F.(1997),Evaluatingpublicpolicy,NelsonHallPublishers,Chicago.

Funtowicz,S.O.andJ.Ravetz (1993),Science for thepostnormalage,Futures,Vol25,

No7,pp.739755.

Georgakopoulou, A. and D. Goutsos (1997), Discourse analysis, an introduction,

EdinburghUniversityPress,Edinburgh.

Geurts,J.L.A., S.L.Hart, andN.S.Caplan (1989), Beleidsanalytische technieken van

sociaalwetenschappelijk onderzoek (Methods of policy analysis in social science; in

Dutch), in:J.GeurtsandJ.Vennix (Eds.),Verkenningen inbeleidsanalyse.Theorie en

praktijkvanmodelbouwensimulatie,Kerkebosch,Zeist,pp.345361.

Glasbergen, P. (ed.)(1998), Cooperative environmental governance. Publicprivate

agreementsasapolicystrategy,Kluwer,Dordrecht.

Grimble,R. (1998),Stakeholdermethodologies innaturalresourcemanagement.Socio

economic methodologies, best practices guidelines, Natural Resources Institute, The

UniversityofGreenwhich,Chatham.

17
http://extra.shu.ac.uk/daol/http://extra.shu.ac.uk/daol/


18/20


Hajer,M.A. (1995),Thepoliticsof environmentaldiscourse.Ecologicalmodernization

andthepolicyprocess,ClarendonPress,Oxford.

Hajer,M.A.(2005),Coalitions,practicesandmeaningsinenvironmentalpolitics:from

acidraintoBSE,in:J.Torfing(Ed.),DiscourseanalysisandEuropeanpolitics,Palgrave,

London(forthcoming).

Hisschemller,M., R.S.J. Tol, and P. Vellinga (2001), The relevance of participatory

approachesinintegratedenvironmentalassessment,IntegratedAssessment,Vol2,No2,

pp.5772.

Hoogerwerf,A.(1990),Reconstructingpolicytheory,Evaluationandprogramplanning,

Vol13,No3,pp.285291.

Jong, W.M. de (1999), Costeffective use of evaluation models: an empirical cross

nationalinvestigation,

International

Journal

of

Technology

Management,

Vol

19,

No

3/4/5,

pp.368383.

Jong,M.deandH.Geerlings (2003),Exposingweaknesses in interactiveplanning: the

remarkable return of comprehensive policy analysis in The Netherlands, Impact

AssesmentandProjectAppraisal,Vol21,No4,pp.281291.

Jrgensen,M. and L. Phillips (2002),Discourse analysis as theory andmethod, Sage

Publications,London.

Kates,R.W.,W.C.Clark,R.Corell,J.M.Hall,C.C.Jaeger, I.Lowe,J.J.McCarthy,H.J.

Schellnhuber, B. Bolin, N.M. Dickson, S. Faucheux, G.C. Gallopin, A. Grubler, B.

Huntley,J.Jager,N.S.Jodha,R.E.Kasperson,A.Mabogunje,P.Matson,andH.Mooney

(2001),Sustainabilityscience,Science,Vol.292Issue5517,pp642.

Kickert,W.J.M.,E.H.Klijn,andJ.F.M.Koppenjan(1997),Introduction:amanagement

perspectiveonpolicynetworks, in:W.J.M.Kickert,E.H.Klijn,andJ.F.M.Koppenjan

(Eds.),Managingcomplexnetworks.Strategiesforthepublicsector,SagePublications,

London,pp.113.

Lafferty,W.M. and E.Hovden (2003), Environmental policy integration: towards an

analyticalframework,

Environmental

Politics,

Vol

12,

No

3,

pp.

122.

Leroy, P. and N.J.M. Nelissen (2000), Milieubeleid en beleidswetenschappen

(Environmental policy and policy science; in Dutch), in: P.P.J. Driessen and P.

Glasbergen (Eds.), Milieu, samenleving en beleid, Elsevier Bedrijfsinformatie, The

Hague,pp.7193.

18


19/20


Mayer, I. (1997),Debating technologies.Amethodological contribution to the design

and evaluation ofparticipatory policy analysis,Ph.D.dissertationTilburgUniversity,

TilburgUniversityPress,Tilburg.

Mayer,I.S.,C.E.vanDaalen,andP.W.G.Bots(2004),Perspectivesonpolicyanalyses:a

frameworkfor

understanding,

International

Journal

of

Technology,

Policy

and

Management,Vol4,No2,pp.169191.

McIntyre, S., and R. McKitrick (2005a), Hockey sticks, principal components, and

spurioussignificance,GeophysicalResearchLetters,Vol32,No3,L03710.

McIntyre, S. and R. McKitrick (2005b), The M&M critique of the MBh98 Northern

HemisphereClimateIndex:updateandimplications,EnergyandEnvironment,Vol16,

No1,pp.69100.

Mitchell,R.K.,

B.R.

Agle,

and

D.J.

Wood

(1997),

Toward

atheory

of

stakeholder

identificationand salience:defining theprincipleofwhoandwhat reallycounts,The

AcademyofManagementReview,Vol22,No4,pp.853886.

Mohr,L.B.(1992),Impactanalysisforprogramevaluation,SagePublications,Newbury

Park.

Parsons,W. (1995),Publicpolicy:an introduction to the theoryandpracticeofpolicy

analysis,EdwardElgar,Aldershot.

Rhodes, R.A.W. (1997), Understanding governance. Policy networks, governance,

reflexivityandaccountability,OpenUniversityPress,Buckingham/Philadelphia.

Riet,O.vande(2003),Policyanalysisinmultiactorpolicysettings.Navigatingbetween

negotiatednonsenseandsuperfluousknowledge,Ph.D.dissertationDelftUniversityof

Technology,EburonPublishers,Delft.

Rossi,P.H.,M.W.Lipsey,andH.E.Freeman(2004),Evaluation,asystematicapproach,

7thed.,Sage,ThousandOaks.

Rowley, T.J. (1997), Moving beyond dyadic ties: a network theory of stakeholder

influences,The

Academy

of

Management

Review,

Vol

22,

No

4,

pp.

887

910.

Swanborn, P.G. (1987), Methoden van sociaalwetenschappelijk onderzoek (Social

sciencemethodology;inDutch),Boom,Meppel.

Tatenhove,J.vanandP.Leroy (1995),Beleidsnetwerken:eenkritischeanalyse (Policy

networks:acriticalassessment;inDutch),Beleidswetenschap,Vol9,No2,pp.128145.

19


20/20


Verschuren,P.J.M. andL.Zsolnai (1998),Norms,goals, and stakeholders inprogram

evaluation,HumanSystemsManagement,Vol17,No2,pp.155160.

WCED(1987),Ourcommonfuture,OxfordUniversityPress,Oxford.

Wee,B.

van

(2003),

Transportbeleid

en

logistieke

organisatie.

Een

technisch

bestuurskundige visie op kostenbatenanalyse (Transport policy and logistical

organisation. A technology, policy, and management perspective on CostBenefit

Analysis;inDutch),inauguraladdress,DelftUniversityofTechnology,Delft.

Wrisberg,N. andH.A.Udo deHaes (2002)(eds.),Analytical tools for environmental

design and management in a systems perspective, Kluwer Academic Publishers,

Dordrecht.

Yin,R.K.(1994),Casestudyresearch.Designandmethod.2nded.,Sage,ThousandOaks.

20

**** policy analysis for sustainable development - complexities and methodological response

Documents