evaluating the new technologies ann sefton faculties of medicine and dentistry university of sydney
TRANSCRIPT
![Page 1: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/1.jpg)
Evaluating the New Technologies
Ann Sefton
Faculties of Medicine and Dentistry
University of Sydney
![Page 2: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/2.jpg)
Outline
• what is our educational purpose?• levels of decision-making• evaluating local strategies• evaluating others’ “solutions”• designing evaluations• using IT for program evaluation• teaching students to evaluate• conclusions
![Page 3: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/3.jpg)
Levels of decision-making have different implications• institution• faculty• department• unit of study• individual staff member
– whole program, unit of study, “lesson”?– undergrad, postgrad, continuing
education?
![Page 4: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/4.jpg)
Evaluation of local strategies
• against goals and expectations• measuring learning processes• comparative• program improvement• consider quality, costs,
accessibility
![Page 5: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/5.jpg)
Meeting goals:example values - degree program:
• student-centred, independent learning
• reflection and self-evaluation• cooperation in groups• evidence-based decision making• effective skills– clinical – IT
![Page 6: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/6.jpg)
Meeting goals: example goals - degree program• basic and clinical science
– critical reasoning for medical practice
• patient and doctor – effective communication, clinical skills
• community and doctor – community concerns and population issues
• personal & professional development – ethics/law, humanities, reflective practice
![Page 7: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/7.jpg)
Meeting goals: learning package examples
• understand new concept(s)• solve problems(s)• learn a skill• simulate an experiment• access new information incl.
images • formative self-assessment
![Page 8: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/8.jpg)
Evaluation of local strategies• against goals and expectations• understanding learning
processes– evaluation vs research
![Page 9: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/9.jpg)
Evaluation of local strategies• against goals and expectations• understanding learning processes
– evaluation vs research
• comparative studies– evidence-based education, cohort
studies– uncontrolled variables, ethical
dilemmas
![Page 10: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/10.jpg)
Evaluation of local strategies
• against goals and expectations• understanding learning processes
– evaluation vs research
• comparative studies– evidence-based education, cohort
studies– uncontrolled variables, ethical
dilemmas
• quality improvement– structures in place for revision
![Page 11: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/11.jpg)
Evaluation of local strategies
• against goals and expectations• understanding learning processes
– evaluation vs research
• comparative studies– evidence-based education, cohort
studies– uncontrolled variables, ethical
dilemmas
• quality improvement– structures in place for revision
• consider quality, costs, accessibility
![Page 12: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/12.jpg)
Evaluation of imported “solutions”• to compare • to adopt “as is”• to adapt or modify• consider quality, accessibility,
costs
![Page 13: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/13.jpg)
Goals and expectations - learning package:
• learn a new concept• apply knowledge already learned• offer access to information,
databases• solve problems• rehearse a skill • provide feedback on learning• access/simulate/replace an
experiment
![Page 14: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/14.jpg)
Match: IT and educational aims• encourages active learning?• stimulates problem-solving?• triggers “what if” speculation?• stimulates student-student
discussion?• supports further exploration?• offers quizzes and/or feedback?
![Page 15: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/15.jpg)
Is the content appropriate?
• material relevant? important?• specific skills essential? • level of knowledge/skill right?• enhances useful generic skills?• well-matched to assessment?• outcomes consistent with
program goals?
![Page 16: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/16.jpg)
Design considerations
• well constructed and structured?• are screens clear/acceptable to users?• easily navigated?• instructions
clear/intuitive/consistent?• can the user exit easily?• too slow/fast?• does it include feedback?
![Page 17: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/17.jpg)
Quality and relevance
• is the issue/task/learning important?• is the information accurate?• is the approach up-to-date?• are examples appropriate/engaging?• are illustrations clear and relevant?• if a simulation, how "real" is it?
![Page 18: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/18.jpg)
Students’ needs
• what are their expectations?• what are their specific learning needs?• what are their generic learning needs?• is the IT interactive and time-
effective?• is the IT consistent with assessments?• does it offer helpful feedback?
![Page 19: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/19.jpg)
Student and staff reports
• Student views:– exciting, interesting? useful? boring,
useless?– easy to use? impenetrable?– what aspects are valued for learning?
• Objective measures:– does it stimulate discussion?– do they seek it out frequently?– can improved learning be measured?
![Page 20: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/20.jpg)
Is the program effective and cost-effective?
• is the students' learning significantly increased?
• do they enjoy the new learning?• do they rate it as high quality?• are they motivated to learn more?• are there cheaper but effective
alternatives?
![Page 21: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/21.jpg)
Using computers for evaluation of programs and units
• feedback can be automated, interactive
• feedback can be frequent (but avoid overload)
• response times are shortened• BUT appropriate mechanisms
are needed for implementation
![Page 22: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/22.jpg)
Teaching students skills of evaluation
• enhances collegiality• stimulates evidence-based practice • engaging students can provide insights
for them on program expectations• encouraging participation important
BUT they must see results!• “learning for life”: information literacy
![Page 23: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/23.jpg)
Effective computer-based learning:• meets a clear educational need• is consistent with program goals • is cost-effective and timely• avoids errors, misconceptions • is set at an appropriate level• interests and motivates students• engages students actively in
learning• is well-designed, easy to use• encourages collaboration
![Page 24: Evaluating the New Technologies Ann Sefton Faculties of Medicine and Dentistry University of Sydney](https://reader036.vdocuments.mx/reader036/viewer/2022070413/5697bfe71a28abf838cb5ecd/html5/thumbnails/24.jpg)
Good evaluation
• is essential to meeting educational need
• must justify costs (time, money)• is difficult, expensive: many
variables• depends on explicit program goals• is time-consuming and may not
yield conclusive results