elo-based supplemental course evaluations teaching academy assessment subcommittee david baum &...
TRANSCRIPT
![Page 1: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/1.jpg)
ELO-based supplemental course evaluations
Teaching Academy Assessment Subcommittee
David Baum & Janet Batzli
![Page 2: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/2.jpg)
The Assessment Grid
![Page 3: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/3.jpg)
The Assessment Grid
![Page 4: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/4.jpg)
The Assessment Grid
![Page 5: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/5.jpg)
The Assessment Grid
![Page 6: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/6.jpg)
Goals• Campus wide, quantitative course evaluations– Not to replace departmental evaluations
• Address Essential Learning Outcomes– Track student learning– Assess courses/programs– Educate students about the ELOs
• Post summary data on the web– Incentivize participation– Help students select courses– Be open with stakeholders
![Page 7: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/7.jpg)
Why should instructors care?
• Know how we are doing• Be judged on what student’s gain from the
course (not charisma)• Get students to reflect on learning rather than
enjoyment
![Page 8: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/8.jpg)
UW Essential Learning Outcomes
http://www.learning.wisc.edu/welo2010.pdf
![Page 9: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/9.jpg)
UW Essential Learning Outcomes
![Page 10: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/10.jpg)
Approach
• Initial questions based on ELO language• Survey Center conducted three rounds of
focus groups to sequentially improve questions
• Also piloted data presentation
![Page 11: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/11.jpg)
Initial question
How much did this course enhance your knowledge of the human or natural world?
![Page 12: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/12.jpg)
Final Recommendation
In general, how much did this course enhance your knowledge of the world, such as knowledge of human cultures, society, or science? [Not at all, A little, Somewhat, Quite a bit, A great deal] – While students were not always sure what “..of
the world” meant, with the defining words, they got the intended idea
![Page 13: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/13.jpg)
Initial question
How much did this course help you develop intellectual and practical skills, such as critical and creative thinking, written and oral communication, teamwork, and problem-solving?
Combining intellectual and practical/pre-professional skills was confusing to students – split into two questions
![Page 14: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/14.jpg)
Final Recommendations
How much did this course help you develop intellectual skills, such as critical or creative thinking, quantitative reasoning, and problem solving?
How much did this course help you develop professional skills, such as written and oral communication, computer literacy, and working in teams?
[Not at all, A little, Somewhat, Quite a bit, A great deal]
![Page 15: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/15.jpg)
Initial question
How much did this course affect your values and sense of personal and social responsibility, for example by increasing your knowledge of policy issues, engagement in community and civic affairs, intercultural knowledge, or ability to reason ethically?
![Page 16: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/16.jpg)
Final RecommendationsHow much did this course increase your sense of
social responsibility, that is increased your knowledge of cultures or provided you with opportunities for civic or community involvement?” [Not at all, A little, Somewhat, Quite a bit, A great deal] • There was some confusion between feeling vs. acting
responsibly• The question mixes an impact on a student with contents
of a course• We tried “reason ethically” but this confused students
– More editing might be needed
![Page 17: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/17.jpg)
Initial question
How much did this course advance your ability to integrate diverse areas of knowledge?
![Page 18: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/18.jpg)
Final Recommendations
How much did this course improve your ability to combine knowledge or skills from different fields of study? [Not at all, A little, Somewhat, Quite a bit, A great deal] • “I thought it just meant was the course
interdisciplinary or not.” “Yeah, I thought the same thing. It didn’t have to do with my ability, but more the class itself.”
• If we interpret the ELO to mean interdisciplinarity within a course, then the question is working
![Page 19: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/19.jpg)
Initial question
How would you rate this course for its overall quality and educational impact?
Quality and impact were found to be different.
![Page 20: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/20.jpg)
Final RecommendationHow would you rate the overall educational value
of this course, that is the extent to which the course improved your all-around education or prepared you for the future?” [Very poor, Poor, Fair, Good, Very good]– Students understood: “How much you gain and take
out of the course in general;” “How much you learned or how much you were challenged by the course and the subject;” “I also considered the practicality of it, for using it in the future and if it will benefit you in some way;” “Yeah, I was thinking just how happy you were that you took the course.”
![Page 21: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/21.jpg)
Final RecommendationHow would you rate the overall quality of this
course, that is the extent to which it was structured and taught in order to maximize its educational value?” [Very poor, Poor, Fair, Good, Very good]– Unclear to some students if they should assess the
professor or course structure– We want the focus to be on the course rather
than the instructor– Maybe need to narrow, split, or drop this question
![Page 22: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/22.jpg)
Final Recommendation
Students felt it would be helpful to know whether a student was or was not “a major”
Question added (at the start of the survey):At the time you enrolled, did you primarily take
this course to fulfill a requirement for your major? [yes/no]– Students seemed to understand this question, as
intended
![Page 23: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/23.jpg)
General student reactions to the survey
• The survey made them reflect on their courses in ways they hadn’t before and thought the survey asked about aspects of courses where they did gain something
• One said you can appreciate classes for different reasons and “these questions reflected that.”
• Some students had mentally compared the answers they gave from one course to another
• ELOs became clearer• The overall reception of the utility of the survey
was very positive
![Page 24: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/24.jpg)
Ideal implementation
• After each semester, students receive a link to a personalized survey covering all the courses they took in the previous semester
• Data are collated and made available to students and the public (CouseGuide?)
![Page 25: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/25.jpg)
Sample Data Presentation
• The students liked the idea of having such data available– it would definitely help because there isn’t currently a
way to know if a class is good or bad besides “anecdotes”
– Better than “RateMyProfessor.com” - those ratings don’t necessarily tell you what the class is going to be like
• But they are realistic: – …A student shouldn’t put all his “trust into what the
graph might say.”
![Page 26: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/26.jpg)
5 Imaginary studies 101 (331 students; 105 majors)
Imaginary studies 102 (183 students; 80 majors)
Imaginary studies 202 (89 students; 33 majors)
Imaginary studies 401 (23 students; 12 majors)
Imaginary studies 402 (19 students; 17 majors)
Imaginary studies 201 (113 students; 83 majors)
Mean for nonmajorsMean for majors
Sample data presentationNeeds work!
![Page 27: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/27.jpg)
Other things student wanted (but we don’t expect to provide)
• How much work• How easy it was to get an A• Correlation between the amount of work
students put in and how much they feel they actually learned from the course
• Data for specific professors or TAs
• Nobody mentioned wanting to know how hot the professor was [Take that RmP!]
![Page 28: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/28.jpg)
Data from pilot (8 students 3 courses each)
1 = Not at all2 = A little3 = Somewhat4 = Quite a bit5= A great deal
71% of courses received 4 or more on at least one ELO (Q2-6)
42% course received 5 on at least one ELO (Q2-6)
100% of students gave at least one 5 across their three courses (Q2-6)
![Page 29: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/29.jpg)
Good correlation between overall impact and the other measures
![Page 30: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/30.jpg)
But not for “overall quality”
![Page 31: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/31.jpg)
Next steps
• Run a pilot survey on a large sample of students• Use focus groups to improve data presentation
style• Consult with central administration on whether
funds would be available to establish ongoing surveys and posting of data for all courses
• Seek faculty/staff buy-in (allowing for instructors to opt-out)
![Page 32: ELO-based supplemental course evaluations Teaching Academy Assessment Subcommittee David Baum & Janet Batzli](https://reader036.vdocuments.mx/reader036/viewer/2022062715/56649da65503460f94a92129/html5/thumbnails/32.jpg)
Thanks
• Janet Batzli• TA Executive Committee and Assessment
Subcommittee• UW Survey Center– John Stevenson; Jennifer Dykema; Jaime Faus;
Tara Piche
• Mo Bischoff• UW Assessment Council