a familiar(ity) problem: assessing the impact of prerequisites ......mol bio •very familiar...
TRANSCRIPT
![Page 1: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/1.jpg)
A familiar(ity) problem: Assessing the impact of prerequisites and content
familiarity on student learning
Brian Sato
University of California, Irvine
![Page 2: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/2.jpg)
![Page 3: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/3.jpg)
UC System: Teaching Professor(Lecturer SOE)
• Tenure-track
• Expectations of quality:
– Teaching– Professional
Development
– Service
60%
30%
10%
![Page 4: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/4.jpg)
![Page 5: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/5.jpg)
![Page 6: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/6.jpg)
Typical STEM curriculum
Bio 1 Bio 2 Bio 3
Bio 1
Bio 2
![Page 7: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/7.jpg)
Why do we have prerequisites?
![Page 8: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/8.jpg)
Student Perceptions of Prerequisites
Positive Attributes Fraction of Interviews
Background knowledge 89.3%
Acts as a safety net for students 35.7%
Responsible for future success 25.0%
Contributes to interest in subject material 21.4%
Positively impacts how instructors teach 14.2%
Improves student behaviors 7.1%
Improves overall quality of students 7.1%
Scheduling 7.1%
![Page 9: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/9.jpg)
Student Perceptions of Prerequisites
Negative Attributes Fraction of Interviews
Scheduling 51.7%
Waste of student’s time or money 37.9%
Not used as intended by faculty 31.0%
Student’s are prepared without the prerequisite 17.2%
Not used as intended by students 13.8%
![Page 10: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/10.jpg)
Why do we have prerequisites?
• Teach students content/skills necessary for success in future courses
• Students need time to mature to succeed in later courses
• Logistically easier to schedule future courses
Potential benefits are rarely assessed
![Page 11: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/11.jpg)
How does your campus assess prereqs?
• Prereq = Exam/Course Grades
• Prereq course grade ~ Later course grade
![Page 12: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/12.jpg)
Is prerequisite completion beneficial?
It depends!
Prerequisite Later scenario
assessed
Does prereq impact
later success?
Reference
Premed
Anatomy/Histolog
y
Anatomy/Histology
(Medical School) Yes
Forester et al.
2002
Premed Organic
Chemistry
Biochemistry in
Chiropractic SchoolYes
McRae. 2010
General Ed
Courses
ETS Major Fields Test
for BusinessYes
Ritchie et al. 2011
Organic
Chemistry
Course
Intro Biochemistry
Course No
Wright, et al. 2009
# of Business
School Prereqs
MBA Program GPANo
Christensen et al.
2011
Intro Business
Courses
Intermediate
Business CoursesNo
Jones et al. 2013
![Page 13: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/13.jpg)
• Examine concept overlap on exam questions
Novel Means to Assess Value of Prerequisites - Familiarity
Genetics (Prerequisite)
Mol Bio
• Very Familiar – Students should be able to answer the Mol Bio exam question based on the Genetics prereq
• Familiar – This was discussed in the Genetics prereq
• Not Familiar – This was not discussed in the Genetics prereq
![Page 14: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/14.jpg)
• Examine concept overlap – Familiarity
Novel Means to Assess Value of Prerequisites
Hypothesis: Students will perform better on MolBio exam questions covering more familiar concepts (VF > F > NF)
Genetics (Prerequisite)
Mol Bio
![Page 15: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/15.jpg)
How to assign question familiarity?
Genetics Lecture Slides Genetics Instructor Genetics Students
Using:
![Page 16: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/16.jpg)
How to examine the impact of familiarity on exam performance?
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10
Assign Familiarity (Genetics)
Mol Bio Exam
Questions
![Page 17: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/17.jpg)
How to examine the impact of familiarity on exam performance?
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10
VF F NF
Calculate % correct for questions in each category
85% 75% 65%
Assign Familiarity (Genetics)
Mol Bio Exam
Questions
![Page 18: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/18.jpg)
Do the familiarity assignment methods agree with each other?
0
20
40
60
80 Agree
Slightly Agree
Disagree
Methods Compared
Students & Slides
Students & Instructor
Slides & Instructor
Perc
enta
ge o
f Ex
am Q
ues
tio
ns
![Page 19: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/19.jpg)
Exam Performance and Concept Familiarity
Familiarity Designation: Genetics Lecture Slides
*** p < 0.001
Shaffer et al. 2016
*** ***
VF F NF0
20
40
60
80
100
Perc
ent
Co
rrec
t
![Page 20: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/20.jpg)
Familiarity Designation: Genetics Instructor
Shaffer et al. 2016
VF F NF0
20
40
60
80
100
Exam Performance and Concept FamiliarityPe
rcen
t C
orr
ect
![Page 21: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/21.jpg)
Familiarity Designation: Genetics Students
Shaffer et al. 2016
VF F NF0
20
40
60
80
100
Exam Performance and Concept FamiliarityPe
rcen
t C
orr
ect
![Page 22: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/22.jpg)
• Hypothesis:
VF > F > NF
Exam Performance and Concept Familiarity
At best, students do better on VF questions
![Page 23: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/23.jpg)
But how can we control for differences in question type?
• Bloom’s taxonomy
![Page 24: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/24.jpg)
Similar conclusions when controlling for Bloom’s level
Shaffer et al. 2016*p < 0.05 **p < 0.01 *** p < 0.001
![Page 25: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/25.jpg)
Conclusions
• At best VF > F and VF > NF, but not F > NF
• Similar conclusions with:
• Human Physiology (prereq)/Human Anatomy
• Microbiology Lecture (prereq)/Lab
![Page 26: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/26.jpg)
So what do we take away from this?
• Breadth vs. Depth
• Important to assess prerequisites
• Need for increased communication between faculty teaching related courses
![Page 27: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/27.jpg)
Transforming Courses – Breadth vs. Depth
• Topics covered:– DNA/RNA/protein structure– DNA Replication– DNA Repair– Transcription– Prokaryotic Txn Regulation– Eukaryotic Txn Regulation – Chromatin, Txn Factors– mRNA processing, export– Non-coding RNAs– Genetic Code– Translation – Initiation, Elongation, Termination– Viruses
Mol Bio
Instructors A/B/C
Instructors D/E/F
Instructors G/H/I
Goal: Revamp the course to re-align the 3 sections
![Page 28: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/28.jpg)
Transforming Courses – Breadth vs. Depth
Mol Bio
Instructors A/B/C
Instructors D/E/F
Instructors G/H/I
Goal: Revamp the course to re-align the 3 sections
![Page 29: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/29.jpg)
Transformation of Mol Bio
• High Structure Format
– Increased active learning in lecture
• Clickers
• Student group work
– Problem-based discussion sections
– Emphasis on work outside of class
• Online textbook quizzes before each lecture
• Weekly online HWs
– Increased testing
• 3 “midterms”
• 1 cumulative final
Decrease content “covering”Increase depth of coverage
![Page 30: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/30.jpg)
Impact of Course Transformation
• 19 question end of quarter student assessment
10
20
30
40
50
0
Perc
ent
Co
rrec
t
Low Structure Course
High Structure Course
*** p < 0.001
***
![Page 31: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/31.jpg)
What next?
• Expand prerequisite analysis to additional courses/disciplines
• Would this work in your discipline?
• Can you think of a pair of courses (prereq/later course) to analyze?
![Page 32: A familiar(ity) problem: Assessing the impact of prerequisites ......Mol Bio •Very Familiar –Students should be able to answer the Mol Bio exam question based on the ... Shaffer](https://reader034.vdocuments.mx/reader034/viewer/2022051809/601278a97a12b03e1154042b/html5/thumbnails/32.jpg)
Acknowledgements
• Justin Shaffer
• Pavan Kadandale
• Usman Alam
• Samantha Dacanay
• Jennifer Dang
• Amanda Lee
• Michael Dennin
• Sarah Eichhorn
Questions?