panel session 2: all in the process
DESCRIPTION
Building Evidence in Education: Conference for EEF Evaluators 11 th July: Theory 12 th July: Practice www.educationendowmentfoundation.org.uk. Panel session 2: All in the process. Pleasure and pain of process July 2013. Introductions and purpose. Becky Clarkson - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/1.jpg)
Building Evidence in Education:Conference for EEF Evaluators
11th July: Theory12th July: Practice
www.educationendowmentfoundation.org.uk
![Page 2: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/2.jpg)
Panel session 2: All in the process
![Page 3: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/3.jpg)
Pleasure and pain of process
July 2013
![Page 4: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/4.jpg)
Introductions and purpose
Becky Clarkson
Look at some of the issues I’ve encountered with process evaluations, and hopefully some of the positive aspects.
![Page 5: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/5.jpg)
Overview of the five transition RCTsName The partner How many
schools / pupils
Length of delivery
Who is actually delivering
What the intervention involves
Chatterbooks Academic at Coventry Uni
12 / 577 10 weeks Researchers hired especially
2 arms – 1 enhancedBook club style meeting once a week
R4R Academic and cellist
6 / 421 10 weeks The cellist! 10 min session once a week
Vocab Enrichment
Bolton Council not finalised 6 months Normal teachers ?
Speaking and Listening
Greenford High
not finalised 8 months TAs hired specifically
?
One to one tuition
Perry Beeches Academy
not finalised full school year
Graduates hired specifically
Five sessions per fortnight
![Page 6: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/6.jpg)
What the process evaluations consist of
• Observations of training sessions• Observations of intervention sessions• Interviewing deliverers• Teacher log
![Page 7: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/7.jpg)
The first pain
TIME
![Page 8: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/8.jpg)
The second pain (the biggy really!)
RELATIONSHIPS• non-contactable• too interested• over-friendly
![Page 9: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/9.jpg)
The pleasures
Personal interest
Continuity
![Page 11: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/11.jpg)
Some Issues in Process EvaluationPresentation to the EEF evaluators conference
Caroline SharpJuly 2013
![Page 12: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/12.jpg)
What is a process evaluation?
Process evaluation: documents and analyses the development and implementation of a programme, assessing whether strategies were implemented as planned and whether expected output was actually produced. Bureau of Justice Assistance (1997) quoted in EEF (2013)
![Page 13: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/13.jpg)
What does process evaluation involve?It:• explores the implementation and/or development of
a programme• considers how the programme has been
implemented and/or developed• can assess whether the programme has been
implemented and/or developed as planned• considers which activities/outputs have been
implemented.
![Page 14: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/14.jpg)
Why implementation is important
Interventions are rarely implemented as designed and, crucially, variability in implementation is related to variability in the achievement of expected outcomes (Lendrum and Humphrey, 2012, p.635).
14
Because we need to know why and how an initiative has achieved its outcomes and impact in order to draw valid conclusions and improve practice.
![Page 15: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/15.jpg)
Some issues and challenges• Process evaluation may be confused with qualitative
evaluation• It can be difficult to establish intended activities and
outputs• How to respond to new/developing initiatives?• Difficult to achieve good measures of resources and costs• How to deal with issues of fidelity versus responsiveness
to context?• How to make good judgements about scaleability?
![Page 16: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/16.jpg)
What helps?• A clearly defined intervention (rather than a loosely
defined ‘programme’)• A more established programme model• A clear logic model or Theory of Change• Participation in key events (e.g. observing staff
training; attending project meetings)• Good communication between the programme
managers and the evaluation team(s)• Understanding the broader context of going to scale.
![Page 17: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/17.jpg)
Process evaluation of Changing Mindsets: challenges and lessons
![Page 18: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/18.jpg)
Process evaluation of Changing Mindsets: challenges and lessons
Changing Mindsets•Primary school intervention (Year 5)•Managed and delivered by Portsmouth University and partners•In Portsmouth and Hampshire•Testing theory of ‘fixed’ and ‘growth’ mindsets•2 separate interventions: Pupil; Teacher Inset•RCT with tests in English, Maths, and Mindset measures
![Page 19: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/19.jpg)
Process evaluation of Changing Mindsets: challenges and lessons
Process evaluation – general objectives•To establish whether the programme was delivered as planned•To identify:
• factors which could explain why the intervention did/didn’t work
• contextual reasons for variation• Issues which should inform plans for future roll-out
![Page 20: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/20.jpg)
Process evaluation of Changing Mindsets: challenges and lessons
Design of the process evaluation•Include experiences and perspectives of participants (project workers, teachers, head teachers, project partners)•Cover all key components of the intervention•Keep to the ‘light touch’ EEF principle•Use project’s own evaluation
![Page 21: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/21.jpg)
Process evaluation of Changing Mindsets: challenges and lessonsProcess evaluation of Changing Mindsets – challengesNot over-burdening the project or participantsGaining cooperation without jeopardising projectAvoiding confusion between project team and evaluator teamFitting in with the project: timing and placeKeeping up with changes to the projectStaying objective
![Page 22: Panel session 2: All in the process](https://reader035.vdocuments.mx/reader035/viewer/2022062301/56815ec3550346895dcd4b0e/html5/thumbnails/22.jpg)
Process evaluation of Changing Mindsets: challenges and lessons
Process evaluation: some lessons from Changing MindsetsWork with the project team Don’t duplicate data collection•Use project’s own evaluationKeep it ‘light’ – don’t over-collect targeted visits & observationskeep interviews short & focusedFlexible methods - phone rather than face to face, email responses