using formative and summative evaluation to improve field trip programs kimberly m. burtnyk amgen...
TRANSCRIPT
Using Formative and Summative Evaluation to Improve Field Trip
Programs
Kimberly M. Burtnyk
Amgen Center for Science Learning
California Science Center
Pre-/Post-visit classroom activities• Curriculum compatible• K8 • CA standards
Gallery worksheets (atypical)• “Chaperone Sheets”
Available free from our website
ThinkSCIENCE! PathwaysComponents
Help teachers implement curriculum-compatible field trips
Increase educational guidance for field trips
Increase interactions with staff
Help chaperones facilitate
Increase concept understanding through chaperone facilitation
ThinkSCIENCE! PathwaysGoals of the program
Formative Evaluation Spring 2002
Focus: Access, Use, Chaperone/student behavior
Summative Evaluation Timeline Spring 2003
Focus: Cognitive gains
ThinkSCIENCE! PathwaysEvaluation Plan
Recruiting• Groups with existing reservations
Questionnaires• Pathways usage issues• Field trip planning and behavior• Chaperone roles
Observations• Stopped/didn’t stop• Time spent--we want Pathways groups to stay longer• Panels Read, Discussions, Gestures
Formative Evaluation Method
Recruiting • 37 Schools recruited from March-June
Questionnaires• 82 Returned (37 Teacher 45 Chaperone)
Observations• 15 schools tagged for observation (8 PW, 7 Non-PW)• 72 Chaperones observed
Formative Evaluation Results Summary
Only 3 of 8 PW recruits used it
Reasons Technical difficulties• Materials looked too complicated and long• Confused about where to find them Unaware of the materials• Timing
Formative Evaluation Questionnaire Highlights
StoppedDid not
stopTotals
PW Users 13 4 17
Non-PW Users 22 33 55
Totals 35 37 72
Fisher’s exact p<0.012 Sig. at p<0.05
72 Chaperones observed
Formative Evaluation Observations Highlights
Stopped/Did not Stop at Capsules
Non-PW Chaperones PW ChaperonesN=55 N=17
0
10
20
30
40
50
60
70
80
Perce
nt of
Chap
erones
0 50 100 150 200 250 300 350
Time(s)
41
9
4 1 0 00
10
20
30
40
50
60
70
80
0 50 100 150 200 250 300 350
5
2
6
1 1 2
Time(s)
Mean t: 66 s Mean t: 138 s
Mann Whitney U p<0.002
Formative Evaluation Observations Highlights
Time spent at Capsules
QualityFactor
Pathways(n=13)
Non-Pathways
(n=22)Reading 20 incidents 9Discussions 12 12Gestures 18 29
Reading: Mann Whitney U: p<0.0095
Of the 35 chaperones who stopped...
Discussion: p<0.12
Gestures: p<0.64
Formative Evaluation Observations Highlights
Reading, Discussion, Gestures
Pathways usage was limited due to:• Access to computers/printers• Length and perceived complexity of the materials Communication gaps between school administrators
and teachers
BUT...it did produce some significant behavioral
changes in chaperones who used it
Formative Evaluation Conclusions
How Did Formative Evaluation Impact the Program?
In-class/On-site activities separated
Shortened• Map removed• One activity removed (Gallery incomplete)
More direct link created• quick
• easy
• obvious
Summative Evaluation Method
Recruiting• Talked directly to lead teacher• Materials sent to groups• In-class visit prior to field trip
Pre/Post Test
Questionnaires--dropped
Observations• Gestures dropped
Summative Evaluation Results Summary
298 Students Recruited• 217 had in-class visit
254 Pre- and Post-tests completed• 28 PW• 226 Non-PW
Observations• 26 chaperones observed ONLY 4 CHAPERONES USED PATHWAYS
Summative Evaluation Conclusions
Jury still out on cognitive impact of Pathways
• The numbers are too small• Will resume attempts next year
Pre/Post test shows impact of In-class Visit
• Micrometeoroid question Chi Square p<0.004 for those who had visit vs those who did not
1. Even with intensive intervention and support, teachers did not use PW
There were special circumstances in the A&S Gallery And timing, curriculum, age groups were an issue tooBUT: Are such materials worth spending $$ developing? What do teachers really want/need, if anything? We
don’t yet know.
2. Whether or not PW improves learning outcomes as the lit. suggests is still unknown
Warrants further testing and refinement of good and USEFUL field trip practices.
What did Formative and Summative Evaluation Teach us About Pathways?
3. We really need to learn more about teachers and chaperones
Can’t assume that teachers have access to net sources
Chaperones might be willing facilitators
4. We can apply statistics to tell us about the use and effectiveness of the program
Further refining statistical techniques Learning curve Exciting prospects
What did Formative and Summative Evaluation Teach us About Pathways?