a framework for decision-making: developing an outcomes-based evaluation plan for adaptive learning
TRANSCRIPT
Katrice A. Hawthorne, PhD
Karen Vignare, PhD
Cristi Ford, PhD
A Framework for Decision-Making:
Developing an Outcomes-Based
Evaluation Plan for Adaptive Learning
dLRN 2015 Conference
October 16 - 17, 2015
Stanford, CA
◼Undergrad & Grad Enrollment: 87,000◼Demographics:
▪47% of the total enrollment comprised of minority students (34% African American)
▪20% of students are Pell eligible
◼Largest public online open access university ◼Primarily online, but serving 50% military
including on bases throughout world
University of Maryland University College
◼74% of undergraduate students enrolled in UMUC’s stateside programs work full-time.◼54% of students are working parents.◼Median age for students enrolled in UMUC’s
stateside undergraduate programs is 31 years old.
University of Maryland University College
“ An approach to instruction and remediation that uses technology and accumulated data to provide customized program adjustments based on an individual student’s level of demonstrated mastery.”
Waters, J. K. (2014, April 16). The great adaptive learning experience. Campus Technology. Retrieved from http://www.campustechnology.com
Defining Adaptive Learning
◼2010, Carnegie grant with CMU for Open Learning Initiative (cognitive tutors)
◼2014, launched three vendor tools for piloting (Cogbooks, Realizeit, and EdReady)
◼2014, Next Gen Courseware subgrantee to Stanford OLI and Lumen Learning
UMUC History with Adaptive Learning
6
UMUC Adaptive Research Scope
“The systematic acquisition and assessment of information to provide useful feedback about some object” (program, policy, technology, etc.).
Defining Evaluation
Trochim, W. M. K. (2006). Introduction to Evaluation. Retrieved from http://www.socialresearchmethods.net
Continuous Improvement Cycle
Evaluation Logic Model
What We THINK We Know
10
• Accurately predict persistence
• Influence persistence through interventions
• Predict which students need more help
• Non-cognitive factors are an important predictor of success/retention
Results - Course Completion
Results - Grade Distribution
Study Time and Student Achievement
◼Moderate correlation between study time and score gain in solving equations, r(26) = .458, p = .019.
◼Moderate correlation between study time and score gain in percents and real numbers, r(25) = .472, p = .017.
◼Strong correlation between study time and score gain in whole numbers, fractions, and mixed numbers, r(26) = .501, p = .009.
Score Gain
Student Perceptions of Adaptive
Student Comments
“I know so much more than I did before and this is an accomplishment for me in math. I have always stayed away from math until this class. This class helped me to move through math in a way I did not see me doing just weeks ago.”
“I really appreciated the A.I. It assesses my knowledge level as I complete questions and adjust accordingly allowing me to progress at my own pace.”
“It presented the lecture in bite size, easy to digest chunks of information and provided quizzes to assess comprehension with immediate feedback on answers.”
What We Have Learned
◼Learning is not linear
◼Combine learning science and data science
◼Effective, flexible faculty and rich feedback are integral
◼Not enough content in some areas
◼Mapping and course organization are important
What We Want to LEARN
Questions & Contact
Katrice A. Hawthorne, Associate Director, Evaluation and ResearchCenter for Innovation in Learning and Student SuccessUniversity of Maryland University [email protected]
Karen Vignare, Vice ProvostCenter for Innovation in Learning and Student SuccessUniversity of Maryland University [email protected]