evaluation research and engineering education lesley jolly for aaee [email protected] erm wiki...
TRANSCRIPT
Evaluation Research and Engineering Education
Lesley Jolly
For AaeE
ERM wiki at www.aaee-scholar.pbwiki.com
What is evaluation research?
• Periodic assessment of results– Appropriateness, effectiveness, efficiency,
impact, sustainability
• Identifies intended and unintended effects
• Identifies what worked and what didn’t
• Provides level of judgement about overall worth of intervention
What has it got to teach us?
• Systematic approach to what we’re doing already
• Extracting lessons learned
• Legacy of sustained evaluation frameworks for ongoing data collection
• Arguments against ‘popularity contest’ course and program evaluation
Step 1: describe the program
• Aims and objectives– Need to be clearly articulated– Does everyone involved share the same aims
• Program logic diagram– Describes how we think the program
produces results– May be called logframe
Program logic identifies what we need to evaluate
Evaluating PBL tutor training
INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACTS
Training
materials
Staff time
Staff support
workshops
Rooms/ space
Library support
[no
Administrative
support]
Published
research
Compiling
materials
Putting
Materials
online
Running
workshop
Staff research
Review and
edit materials
Staff reflection
Numbers of
new tutors
Numbers of
Returning
tutors
Level of
satisfaction
Growing
library of
examples and
materials
Tutors facilitate
rather than
inform
Tutors
Understand
PBL
Tutors develop
Knowledge
independently
Tutors make
PBL -
Appropriate
contributions toCurriculum
review
Students take
Responsibility
for learning
Students learn
to focus
research and
Synthesise
results
Monitoring facilitates data collection
• Can be process (outputs) or impact (short to medium outcome) monitoring
• Need to develop indicators of progress• Targets may be included in indicators or
separate e.g.– In 2010 50% of 2nd yrs will have used the new facility
for more than 20 hrs– OR percentage of students using facility (indicator)– 50% in 2010 (target)
PBL Tutor training
• GOAL: provide timely well-placed supportive guidance to encourage tutors to scaffold and facilitate student learning
• OBJECTIVES: at the end of training the successful tutor will be able to
• articulate a good understanding of the objectives and methods of PBL
• guide student learning through providing appropriate support and guidance rather than information
• contribute to curriculum development within a staff team
Monitoring Tutor TrainingFOCUS PERFORMANCE
INDICATORSDATASOURCES
DATACOLLECTIONMETHODS
RESPONSIBILITY FOR COLLECTION
TIMEFRAME
Objective 1: tutors articulate a good understanding of the objectives and methods of PBL
OutcomesTutors facilitaterather thaninform
Tutor contributions to discussion
OnlineDiscussionlists
Content analysis of lists
Course co-ordinator Twice persemester
OutputsLevel ofsatisfaction
Likert: training hasclarified purposes of
PBL
Exit survey Paper survey post training
Trainer Everysemester
Objective 2: tutors guide student learning through providing appropriate support and guidance rather than information
OutcomesTutors facilitaterather thaninform
Students takeresponsibility forlearning
Tutor contributions to discussion
Student contributions to discussion
OnlineDiscussionlists
OnlineDiscussionlists
Content analysis of lists
Content analysis of lists
Course co-ordinator
Course co-ordinator
Twice persemester
Twice persemester
Evaluation asks formative and summative questions
• Are the questions cohesive and logical?
• Do evaluation questions link to monitoring data?
• Have ethical issues been addressed?
• What mechanisms are in place to gather the learnings generated by evaluation?
• What needs to be retained from this evaluation process in future years?
Vary questions to suit program
QUESTIONS SOURCES OF INFORMATION FROM MONITORING
SOURCES OFINFORMATIONFROM EVALUATIONDATA COLLECTION
DATA ANALYSISAND REPORTINGMETHODS
Is the program as implemented appropriate, and is it being appropriately monitored?
Has the purpose of the program been achieved?
Is the program being implemented in the most efficient and cost effective way possible?
Has the program produced sustainable results and outcomes?
PBL Tutor Training• Appropriateness
• Did the training model the target behaviour?
• Has the purpose of the training been achieved?
• Indicator 1: changes in Tutor behaviour indicating deeper knowledge of and commitment to PBL
• Indicator 2: changes in student behaviour
• Efficiency• Was the time invested by staff good value?
• Sustainability• what needs to retained as core material from year
to year to retain benefit of training
Owen, J. (2006) "Program Evaluation" Allen & Unwin: Crows Nest
Making use of evaluation
Dissemination and reporting
• Findings may be communicated throughout project to multiple audiences– Evidence, conclusions, judgements,
recommendations.
• Different occasions call for different styles– Oral, interactive workshops, posters, reports,
summaries, papers, conference presentations
• Must be well timed
Developing capacity
• “process use of evaluation”– Taking part develops skills– Taking part sensitises staff to issues
• Improved communication– Ongoing 360 degree dissemination– Development of local discourse