rsd4 1c moore - evaluation at the frontiers of systemic design

Click here to load reader

Post on 14-Apr-2017

471 views

Category:

Design

1 download

Embed Size (px)

TRANSCRIPT

PowerPoint Presentation

Evaluation at the Frontiers of Systemic DesignPresentation at RSD4 Banff, AB, August 1, 2015*Marah Moore, (Claire Nicklin, Keith Miller)

This presentations . . .Systems perspective for complex problemsR&D as a human centered design processFrom product to outcomes

Paradigm ShiftsWhat can weInternational R&Dlearn from design practitioners?

What can design practitioners learn from evaluators?

Wicked problems (hunger, nutrition, agricultural livelihoods for smallholder farmers) beg for a systems approachthis is a paradigm shiftAt its best, Ag R&D is a design process; in our projects, that is a paradigm shiftThere is a need to move away from a focus on product design towards a focus on outcomesthis includes elements of service design (research as a service) this is a paradigm shift

Evaluation can support these paradigm shifts

We need to learn from the design field about the best practices in systemic designPerhaps the design field can learn from evaluation about how to continue systematic learning and adaptation after design is donehow to move to an systems-oriented outcomes-based approach

2

High Andean Cropping systems

W Africa: Millet- and sorghum- based cropping systems

E/H of Africa: Crop Improvement

Southern Africa: Integrating legumes in cereal-based systems

CCRPThe McKnight Foundations Collaborative Crop Research Program (CCRP): Quick Overview

3Research related to cropping systems in 4 regions, 12 countries

Diverse research topicsbreeding, agronomic practices, nutrition, connection to markets . .. .

Program level evaluationsupport evaluation capacity for projects and analysis for regions. Synthesize findings across program.

Works at all levels; presentation will focus on the grantee/project level4

Communities grow local nutritious food, sustainably

Research and Development as a Design Process

6

Traditional R&D Design:

Linear Problem Solving Approach Product Design

Researcher-led, farmer as beneficiary7

SoilPestsClimate

Researcher works on isolated problems. Sometimes looks at multiple issues at the same time. If youre lucky, the researcher might ask the farmer for input.

Goal is to find the silver bullet that will solve the identified problem for all, or at least many, farmers.8

SocialContextPoliticalContextHistoricalContextCulturalContextInstitutionalContextEnvironmentalContext

SoilPestsClimate

But its not so simple . . . Not only are there multiple issues that affect what and how something will work in a given context, but there are different farmers with different needs, preferences, and resources.9

Zoom in to a concrete example of a complex system around a particular issuesoil health . . .10

Rethinking R&D as HC Design:

A Systems Approach (R&D as a service to design outcomes)

In CCRP we are trying to rethink this approach. What if the researcher thought of the their job as providing a service to the farmer? They are there to assist the farmer in solving their problems. How can this service be designed to best meet the needs of the farmers?11

Complex Adaptive Systems . . . dynamic systems able to adapt in and evolve with a changing environment.

There are a lot of different systems theories. Complex Adaptive Systems theory provides a framework for the CCRP work that is relevant and useful.

Because of this, we need to find ways not only to support the design of products, and the design of services, but also a way to learn about how change happens, what the underlying principles are that drive the systems, and how we can build capacity to adapt as the systems changes

12

I M E P(Integrated Monitoring Evaluation and Planning)

CCRPs Integrated Evaluation, Monitoring, and Planning (IMEP) process is our attempt to do this

13

The Role of EvaluationTraditional evaluation mirrors traditional R&D in many ways:

Top downNarrow focusIsolated resultsOr, sometimes only process . . .

[email protected]

In Michael Quinn Patton, Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guilford Press, June 2010

15

Design EvaluationCollect DataShape PracticeMake Meaning from Data

Traditional Evaluation Framework

#

These are the phases of a traditional evaluation. When you are lucky, you can see the link between evaluation and practice. When you are really lucky, you get a chance to redesign the evaluation based on what you learn.

This more traditional evaluation works ok in static, predicable systems.

As we saw earlier, that is not CCRP16

Collect Data

Make Meaning from Data

Design Evaluation

SHAPE PRACTICE

Complex Systems-Oriented View of Phases of Evaluation

#

In a complex systems, the evaluations are most effective if phases of evaluation overlap, and practice is shaped at every point.

In other words, shaping the systems within which the evaluation is working happens well before the data is analyzed and interpreted.17

Systems Levers

what is what could be

Here is another way to visualize the potential of a systems-oriented evaluationit gets under the tip of the iceberg and helps to articulate the deeper reasons that things are the way they are, and what some of the levers are to influence change.

18

An Alternative: Developmental EvaluationEvaluation to support the development of innovation in complex situations

Sounds a lot like designing solutions to wicked problems . . .

This is the approach that we took in developing an evaluation system for CCRP.

Ill talk about three specific challenges that the evaluation was attempting to address

19

CCRP DE Challenges:Fostering Adaptive Capacity

Building Coherence: Articulating and Testing Assumptions About Change

Going to Scale in a Complex Environment

CCRP DE Challenge # 1: Fostering Adaptive Capacity Paradigm ShiftLEARNING

We saw the need for a paradigm shift in the research

Movement towards an understanding of the systemsnot only as context, but are a set of inter-related and interacting parts that all need to be considered in the research.Movement towards a design mindsetwhat are the innovations, the new ideasboth technological and social?

AND, then what can we learn about how change happens, how to influence change, both during the research process and as we evaluate the product and outcomes? Need for a systematic way to feed learning about outcomes back into the system to influence the way we approach the research (design) process.21

Adaptive Action: iterative cycles, design mindset

Deceptively simpleembedded adaptive action cycles across levelsprogram, regions, projects.

Used in in all meetings, all reports, most interactions

Over time began to influence the way people thought, the way they worked, their expectations about the impact22

Example: Soils, Northern Andes

What?So What?Now What?

Project of Mark Caufield, PHD student at Wageningen student. The site is in the Chimborazo province of Ecuador.

The what is that the State had money from oil and prefers to buy goods instead of investing incapacity or people. So has bought tractors for many towns.

So What: Because of the slope, tractors should not be used in this kind of environment. The light brown color in the circle shows how the black top soil is eroding within weeks.

Now What? We are supporting research that tests which kind of cover crops, once they are incorporated into the soil, slow the erosion.23

CCRP DE Challenge #2: Building Coherence, Articulating and Testing Assumptions about Change

Expanded vision

Contextualized R&D

Focus on outcomes

To build coherence of concepts we worked with the program to develop an adaptive theory of change. Theories of change (ToC) are a tool that can be used to conceptualize systems change and design interventions. In a complex systems, these ToCs are not static, and must have room to evolve alongside the program evolution.

We found that the theory of change work helped to bound the creative energy so that the conceptualizing fed a more coherent process. With ToCs in place, results of the project-level reflections now feed up to inform the regional reflections; likewise, the regional reflections inform the program-level reflections.

24

1.Make Hypotheses ExplicitTheories of change at the project, region, and program levelsTargeted research and evaluation questionsProject Level

25

1.Make Hypotheses ExplicitAdaptive Action in complex systems3.Collaborative interpretation, planning, and design

2. Collect cross-cutting and deep data

Adaptive action requires bounding the inquiry the TOC provides a way to do that

In terms of design ToCs are important to not focus on the output, but rather the outcome. So if a particular technology is not producing the intended outcome, back to the drawing board, even if the output exists and works

26

Going Deeper: Roles/jobs of women and menProductive (10 tasks) (18 tasks)Agriculture:Place seed during sowingWork in sowing and harvesting

Animals:Pasturing sheep and cows on the hillsBring cows to hillUse of local technologyWeaving and SpinningPut on the yokeBuild the house

Temporary migration to the cityGive money to the mothersLeave to earn money in the cityServices (14 tasks) (14 tasks)Outside the home:Purchase clothes and necessitiesTake sick children to the hospital or traditional healerBring firewood to the house

In the home:Prepare and serve food to the familyToast quinoa for pito drinkClean the houseWash clothesHelp to make breadFeed children

Acculturation Raise childrenTeach our childrenTeach children how to workEducate children

Reproductive Breastfeed childrenCarry children in the wombMake woman pregnant and form the family

Slide: WNB

This is a product of the type of diagnosis that comes of participatory action research, where men and women discussion groups talk about their respective roles (red is women, blue is man)

This is trying to understand how to improve the frequency and quality of child feeding, see how parents are allocating their time and where things could be nudged, for example getting men to make bread more often.

But also making sure that any solution does not go completely against the culture of the community and wont be adopted.27

When men learned that malnutrition effects mental capacity not just growth, they became much more interested in improving it

The support of men and mothers-in-law has been critical in increasing the frequency and quality of feedings

Northern Potosi in Bolivia, the World Neighbors project.

So here, in a complex wicked problem like malnutrition, the key is not one technology or food, or even education, it is about perceptions.

The men were never worried when nurses said malnutrition leads to small children, the men are small and they feel stong.

But in the discussion groups it emerged that the men feel like they dont have good memories and feel limits in their mental capacity and this was connected to malnutrition, this made them very motivated to innovate and change and behavior. So the change in perception was key.28

CCRP DE Challenge #3: Going to scale in complex environments

29

Replication Scale

From the few to the manyCommon understanding is that you pin down a model and then replicateFor CCRP that might mean developing a new variety, or farming practice, and then having it adopted in different locations by different populations

30

Scale: Farmers need to make and apply compost to their fields to increase fertility

31

Make a bin: wire cylinder that is 3-4 feet in diameterAdd 6 inch layer of brown organic material to the bottomAdd 2-3 inch layer of green organic matterRepeat layers until pile is 4-5 feet highMix the layers in 2 days

**From Compost for dummies

UniversalistJim Hancock, FAOEvaluation TasksFidelityOutcomes/Impact

Universalist: What are generalizable simple rules and techniques that can be applied on a large scale and in a wide set of contexts?

Evaluation involves looking at fidelity to the model, and outcomes at particular sites. Might also include understanding how best to promote the model.33

In complex systems

(i.e. the real world)

this usually looks a bit different ...

Differences in environment matter:

CCRPdifferences include microclimates, soils, pests ad disease, cultural practices and preferences, market access and structures, policies governing seed systems and subsidiesand more

34

The real world . . .

In the real world, different conditionsenvironmentscall for differences in approaches35

Scaling models

UniversalistContextualistJim Hancock, FAO

(Universalist: What are generalizable simple rules and techniques that can be applied on a large scale and in a wide set of contexts?)

Contextualist: What are good processes that help take important elements and adapt them in new contexts, or influence large scale processes?

36

Compost in the real world?

It depends

37

What does this look like on the ground?

UNIVERSALISTCONTEXTUALISTCase of Soil Fertility (complex system)

Policies and practices at scaleInnovation NetworksInspirationAdaptationInfluence

Options = success at small scaleCompostGreen ManureLegumesFallowsDo nothingChemical fertilizer

Farmer typologies

+=

In the case of work with soil fertility (to take an example of a complex system)Reminder: universalist is saying everyone needs compost and with specific recipe

Contextualized scaling says lets understand farmer typologies to provide better options for contexts and lets get people together and networked to influence practice and policy at scale, not all farms and farmers are the same so need a lot of options the farmers can choose from and adapt, they also need to be involved in the development of options to build on their knowledge and make sure the options are relevant.

Research also has something to offer by understanding underlying principles and connecting farmers to other experiences.

Sometimes there will be universalist type materials and paradigms, but the point is to foment innovation around this starting point

38

Jim Hancock, FAOEvaluation TasksOutcomes/ImpactUnderstanding ContextContextualist

Evaluation tasks

Here there is also a focus on outcomes and possibly impact

However, instead of fidelity, the task is to look at the influence of context39

Jim Hancock, FAOExamples of Evaluation QuestionsContextualistFactors for success or failure?

Understanding drivers: what starts something, and helps it grow?

Constraints, bottlenecks?

Context, or spaces: understanding the enabling environment

External triggers? How can these be harnessed?

Evaluation Questions

Factors for success or failure (essential ingredients?)climate constraints? Population characteristic? Resource availability?

Understanding drivers: what starts something, and helps it grow? Charismatic leadership? Resources? Incentives?

Constraints, bottlenecks? Cost issues?

Context, or spaces: understanding the enabling environment. Norms and belief systems? Policy environment? Risk aversion?

External triggers? Death of a child. Funding changes. Dramatic weather conditions. Drastic changes in government systems. Changes in markets.

40

The deep learning that comes after the innovation is out in the communityParadigm ShiftLEARNING

We saw the need for a paradigm shift in the research

Movement towards an understanding of the systemsnot only as context, but are a set of inter-related and interacting parts that all need to be considered in the research.Movement towards a design mindsetwhat are the innovations, the new ideasboth technological and social?

AND, then what can we learn about how change happens, how to influence change, both during the research process and as we evaluate the product and outcomes? Need for a systematic way to feed learning about outcomes back into the system to influence the way we approach the research (design) process.41

Now What?

What design can teach the development/research worldIf people dont use it, its not their fault, its our fault

Participation: be creative about getting into peoples hearts and minds understanding what they need, want and will useiterative

Not just following a protocolyou have to get creative, observe, set up testsyou need good human capital

Other?

There are no silver bullets, one size fits all, if people dont use it, its not their fault, its our fault

Being participatory is not a token thing we cross off the list, we have to be creative about getting into peoples hearts and minds understanding what they need, want and will use, this will be iterative

Its not about saying just you followed a protocol (interviewing, surveys, samples etc), you have to get creative, observe, set up tests, you need good human capital

43

What the evaluation world can teach designThere is use and then there is dis-use. Where are the incentives to track if it is useful over time, for who and why?

To what depth are designers really understanding the systems, and using a contextual approach to scale?

Other?

Marah Moore, [email protected]

45