how complexity influences evaluation a presentation to the australasian evaluation society...
TRANSCRIPT
![Page 1: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/1.jpg)
How complexity influences evaluationA presentation to the Australasian Evaluation Society Conference
Sydney, August 2011
Julie McGeary
![Page 2: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/2.jpg)
Presentation outline
• Complex systems• Complex interventions• Role of program theory in complex evaluations• Audience expectations• Future Farming Strategy evaluation planning
![Page 3: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/3.jpg)
• Non linear• Emergent• Dynamical• Adaptive• Uncertain• Co-evolutionary
Complex systems
Patton, M. (2010). Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press, NY, p. 8.
![Page 4: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/4.jpg)
Victorian agriculture system
![Page 5: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/5.jpg)
• Governance and implementation• Simultaneous causal strands• Alternative causal strands• Non-linearity and disproportionate outcomes• Emergent outcomes
Complex interventions
Rogers, P. (2008). Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation, 14(1), p. 32.
![Page 6: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/6.jpg)
The Future Farming Strategy (FFS)
![Page 7: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/7.jpg)
Interactions
Complex interventions are events in systems.
It is not the intervention itself but the context or setting into which it is introduced and with which it interacts which is the most significant aspect of the complexity.
Hawe, P., Shiell, A. and Riley, T. (2009). Theorising interventions as events in systems. American Journal of Community Psychology, 43, 267-276.
![Page 8: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/8.jpg)
Role of program theory
![Page 9: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/9.jpg)
Role of program theory
Inputs Outputs
Transformational potential
![Page 10: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/10.jpg)
Future Farming Strategy logic model
![Page 11: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/11.jpg)
![Page 12: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/12.jpg)
Audience expectations
•Judgements of adequacy•‘Global’ net benefits•Causal inference – level of confidence•Quantification
“The tension between the systematic evaluation approaches proposed by evaluation experts and the impact measurements required by clients for accountability purposes is not going to be solved easily.”Molas-Gallart, J. and Davies, A. (2006). Toward theory-led evaluation: The experience of European science, technology, and innovation policies. American Journal of Evaluation, 27(1), p. 78.
![Page 13: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/13.jpg)
Evaluation planning
Purpose
• Examine if FFS has made progress towards improving the productive, competitive and sustainable status of Victorian farm businesses
• Assess the benefits of Victorian Government investment in the FFS
• Provide opportunities for learning and improvement in whole-of-government strategy implementation and delivery to guide future policy and program development.
![Page 14: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/14.jpg)
Evaluation planning
Implications of complexity
• Outcomes definition
• Alignment between Action and strategy level outcomes
• Forms of evidence
• Emergent evaluation
![Page 15: How complexity influences evaluation A presentation to the Australasian Evaluation Society Conference Sydney, August 2011 Julie McGeary](https://reader035.vdocuments.mx/reader035/viewer/2022062314/56649f315503460f94c4d13e/html5/thumbnails/15.jpg)
Everything should be made as simple as possible, but not simpler.
Albert Einstein, 1879 – 1955
Reality is complex. Nicoletta Stame, 2004
The art of dealing with the complicated and complex real world lies in knowing when to simplify and when, and how, to complicate.
Patricia Rogers, 2008
. . . and finally . . .