what kind of theory should drive theory- driven evaluation? · 2017-05-23 · 1 what kind of theory...
TRANSCRIPT
1
What Kind of Theory Should Drive Theory-driven Evaluation?
Program Logics and Realist Program Theory in the Evaluation of a Dairy Extension Program
G ElsworthG DrysdaleD GoodrickM Walton
C.I.R.C.L.E
2
Target 10A state-wide dairy industry development program in Victoria, AustraliaInvolves both government and industry agenciesPrimary aim is to improve the productivity and sustainability ofdairy farm businesses and their role in rural communitiesCommenced in 1992• Early focus on improving grazing management• Later extended to other feed-base management programs• Capacity building programs in business planning also introduced• Later emphasis on developing farmer capacity to adapt to
change by encouraging a ‘learner-centered’ culture in the industry
• Evaluation framed by Bennett’s outcomes hierarchy
3
A Typical Representation of Bennett’s Outcomes Hierarchy
(From ‘Targeting Outcomes of Programs (TOP)’ http://deal.unl.edu/TOP/english/contentsf.html)
4
Outcomes HierarchyTypically, a ordered set of component program outputs and outcomes that must be attained before achievement of a longer-term outcome is possibleIn Educational Psychology an early influential outcomes hierarchy was Gagne’s Learning Hierarchy• Learning tasks can be organised according to complexity• Pre-requisite lower-order learning tasks need to be
completed to facilitate attainment of higher-order tasks• Served as the basis for sequencing instruction and led to
the idea of ‘chaining’ – the connection of a set of Ss R components into a sequence
Program outcomes hierarchies appear to be based on very similar principles
5
Is an Outcomes Hierarchy an Appropriate Theory for
Theory-driven Evaluation?
6
Program Theory in EvaluationHistorical Overview
Suchman (1967)• Two reasons for program failure: implementation failure; theory
failureWeiss (1972)• Introduced the idea that an evaluation could be based on a
process theory of the programWholey (1979, 1983)• Elucidation of program theory as part of evaluability assessment:
Prior to the start of a formal evaluation, analyse the program logicto ascertain whether the program, as designed, should be able tobring about the desired outcomes
Chen & Rossi (Chen, 1990)• ‘Theory-driven evaluations’: an evaluation should be based on a
prior explication of the program theory, which is then tested by the evaluation study. This should be a social science theory, not simply a series of ad hoc logical premises about how the programelements fit together (‘program logic’). Importantly, it contains an account of the mechanisms that bring about program effects
7
Program Theory in Theory-driven Evaluation
Two IssuesWhat is an appropriate epistemological grounding for a program theory in Theory-driven Evaluation?
What evaluative purpose should a program theory serve in Theory-driven Evaluation?
8
Epistemological Grounding
Realism vs Empiricism
9
Situating Scientific Realism
result/regularity
model-building
transcendental idealismempirical-testingreal
(3)
events; sequences; invariances
imagined/imaginary
(1) classical empiricism
(2)
transcendental realism
generative mechanisms in models
---------------------------------------------------------------------
---------------------------------
[Taken from Bhaskar (1978) 145]
The Logic of Scientific Discovery
10
‘Scientific’ (or ‘Transcendental’or ‘Critical’) Realism
Two significant differences when realism is
compared with the ‘classical’ empiricist
(‘positivist’) paradigm of program evaluation
• A radically different model of causal explanation
• A radically different view of the relationship between
‘facts’ and ‘values’
11
Different Models of Causal ExplanationClassical Empiricism
‘Covering law’ from which the facts or (subsidiary) laws to be explained can be deduced
‘Constant conjunction of events’ is both necessary and sufficient evidence to establish a causal relationship
Constant conjunctions occur within ‘closed systems’
12
Arguably Results in a Limiting Model of Causality in Complex Social Programs
Causal phenomena in the human domain of social programs are
More complexMultiply determinedMultiply determiningNon-linearConditional on specific contextsFrequently, more radically unanticipated
Than a simplified ‘closed system’ model can adequately portray
13
Different Models of Causal ExplanationScientific Realism
Begins with an ontological assumption - the phenomenon to be explained are the overt manifestations of the activation (‘triggering’) of an (unobserved) generative structure (Fararo, 1989, p. 39) – that is, reality is stratifiedObservation of a constant conjunction of events is neither necessary or sufficient to establish causalityIn an open system, observed outcomes are radically context dependent, such that a mechanism, if triggered, will only result in observable change in the appropriate contextCausal explanation typically involves a hypothetical model of the structure, built by analogy with a known structure (the positionadvocated, particularly, by Harré – Fararo, 1989, p. 40)
14
The ‘Ontological Assumption’ of an Embedded Reality – Bhaskar’s Model
Real Actual Empirical
Structures ×
Events × ×
Experiences × × ×
15
The ‘Ontological Assumption’ of an Embedded Reality – Harré’s Model
Realm 1 - entities that are available to immediate sensory observationRealm 2 - entities that are potentially available to direct sensory observationRealm 3 - entities that are real but will never be available to sensory observation (entities that can be known, but might never be shown to exist)Realist explanations of social events frequently entail entities that appear to be in Harré’s Realm 3 (structures and mechanisms that are inferred from experiences but cannot be directly observed)
16
Thus Pawson & Tilley (1997, pp. 68-72) Contrast Various ‘Successionist’ (Empiricist)
Models of CausalityX Y
X
Z
Y
X
Z
Y
A directsuccessionist model
A model in whichZ ‘mediates’ the relationshipbetween X and Y
A successionist modelsuggesting thatX and Y are‘spuriously’ causedby Z
(Pawson, R., & Tilley, N. (1977). Realistic Evaluation. London: Sage. p. 68)
17
With a Model in Which a Program is Viewed as ‘Triggering’ the Action of a Generative Structure
An action is only causal if it triggers, in Context, a Mechanism that results in the Regularity (outcome).(Pawson, R., & Tilley, N. (1977). Realistic Evaluation. London: Sage. p. 72)
Context
Mechanism
Regularity
18
Is Reconciliation Between Empiricist and Realist Theories Possible?
Fararo (1989) presented a compelling two-step argument for the reconciliation of ‘positivist’ theory with realism in his account of ‘general theoretical sociology’Step 1• Mathematical axiomatics (e.g. Suppes) is embedded within an
instrumemtalist ‘meaning hierarchy’ theory structure (e.g. Toulmin) as a desirable but not necessary attribute of the latter
Step 2• Meaning hierarchy theory structures are construed as being
compatible with realism if the entities proposed by the theory are believed to really exist (i.e. are not simply regarded as usefulfictions)
19
Evaluative Purpose
Normative vs Causal Theory
20
Program TheoryH. T. Chen’s Definition
(A program theory is) “a specification of what must be done to achieve the desired goals (of the proposed program, or the program to be evaluated) what other important impacts may also be anticipated, and how these goals and impacts would be generated.”
(Chen, H. T. (1990). Theory-driven Evaluations. Thousand Oaks, CA: Sage. p.43)
21
Thus Chen Proposed Two Kinds of Program Theory
Normative (Prescriptive) Program Theory: “a specification of what must be done to achieve the desired goals”
Causative (Descriptive) Program Theory: “a specification … of how these desired goals would be generated”
22
The Three ‘Domain Theories’ of Normative Program Theory
Treatment Theory• Specifies what the nature of the program
treatment should beImplementation Environment Theory• Specifies the nature of the context within
which the program should be implementedOutcome Theory• Specifies what the nature of the program
outcomes should be
23
The Three ‘Domain Theories’ of Causative Program Theory
Impact Theory• Specifies the causal link between the treatment and
the outcomeIntervening Mechanism Theory• Specifies how the underlying intervening processes
operateGeneralisation Theory• Specifies the generalisability of evaluation results to
the topics and circumstances of interest to stakeholders
24
Chen Thus Sees Two Clear-cut Elaborations of Program Theory
‘Purely’ Normative• Where the axioms describe how people should
evaluatively orient to the world
‘Purely’ Empirical• As is a psychological theory where the axioms of the
theory enable the derivation of claims that can be empirically falsified
(The distinction between ‘purely normative’ and ‘purely empirical’ elaborations of a theory was made by Fararo, T. J. (1989). The Meaning of General Theoretical Sociology. Cambridge: Cambridge UIniversity Press. P 20.)
25
This Categorical Distinction Appears to Lead to Two Potential Confusions in Chen’s Structure of Program Theory
26
In Addition to Arguing for an Integration of ‘Positivist’ and Realist Theory Structures,
Fararo (1989) Recognised that ‘Hybrid’ Elaborations of a Theory are Possible
A hybrid elaboration “… recognizes that the axioms might well be falsified in careful empirical investigations, yet it regards them as defining an idealized model of action as a starting point for theoretical explanations in a variety of contexts.”
(Fararo, T. J. (1989). The Meaning of General Theoretical Sociology. Cambridge: Cambridge UIniversity. P. 20.)
27
From an Empiricist Perspective We Have an ‘Outcomes Hierarchy’ Derived From ‘Axioms’ About
the Contingent Relationships Between Events Which Yields
Explicit hypotheses about the temporal ordering of events duringprogram implementation
An initial template for refining specific evaluation questions and designing observations and measurement
A caution that longer-term outcomes may not be attainable (and hence observable) within the duration of the proposed evaluation
And hence, a model for positing that short and medium term outcomes provide some evidence that longer-term outcomes may be achieved
28
From a Realist Perspective We Have
A focus on the causal mechanisms that are triggered or enabled by a programThe proposition that program theories should highlight both micro (individual) and macro (social or community level) mechanisms and the causal relationships and interactions between the two
29
Thus a ‘Normative’ Program Theory Should, Critically, Contain Accounts of
The ways that the reasoning and subsequent choices of the program participants are expected to change as a result of participation in the programThe ways that the capacities and resources of the social group(s) will be developed and enhanced to support these ‘micro-level’ changes
30
And a ‘Causative’ Program Theory Should Contain
accounts of how these mechanismswere actually generated and how they led to the desired program outcomes
31
Conclusion
A Proposed Hybrid Structure for Program Theory
(With Embedded Elements of an Outcomes Hierarchy)
32
A Revised Structure for Program Theory
In what other general contexts does the program succeed?
What other contexts will the program successfully operate in?
A GeneralisationTheory
What causal mechanisms were actually triggered by the program? In which contexts?
How will the program bring about the desired outcomes?
A Theory of Causal Mechanisms
What were the actual implementation contexts?Which were associated with the intended outcomes?
What contexts is the program designed to operate in?
A Theory of the Implementation Environment
What structures /strategies were actually implemented?
What structures /strategies will constitute the program?
A Theory of Intended Treatments
What are the actual outcomes?
What are the desired outcomes?
A Theory of Intended Outcomes
What client needs did the program actually meet?
What client needs is the program designed to meet?
A Theory of NeedCausativeNormative