open academic analytics initiative - campus technology innovator award presentation
DESCRIPTION
The Open Academic Analytics Initiative (OAAI) has developed an open-source academic early alert system using Sakai and Pentaho, an open-source Business Intelligence tool, designed to identify students who are at risk to not complete their courses? successfully and then deploy an intervention intended to help the student succeed. The system includes a predictive model which has been released under an open-source license using a standard markup Language to facilitate use and enhancement by others. The system has been deployed to over 2200 students across four different institutions. Based on these pilots, research on critical scaling factors such as the ?portability? of such predictive models and success of intervention strategies has been conducted. Our presentation will update the community on this initiative and our latest research findings as well as discuss future work.TRANSCRIPT
![Page 1: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/1.jpg)
OAAI: Open Academic Analytics
InitiativeJosh Baron
Senior Academic Technology Officer
Marist College
![Page 2: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/2.jpg)
36%
• Reference: Integrated Postsecondary Education Data System (IPEDS)
![Page 3: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/3.jpg)
13%Four Graduation Rate for
Savannah State University (2010)
![Page 4: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/4.jpg)
![Page 5: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/5.jpg)
Before overviewing the project, a brief word on terminology…
![Page 6: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/6.jpg)
Quick Note on TerminologyAcademic Analytics Learning Analytics
A process for providing higher education institutions with the data
necessary to support operational and financial decision making*
The use of analytic techniques to help target instructional, curricular, and support resources to support the
achievement of specific learning goals*
Focused on the business of the institution
Focused on the student and their learning behaviors
Management/executives are the primary audience
Learners and instructors are the primary audience
* - Analytics in Higher Education: Establishing a Common Language
![Page 7: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/7.jpg)
The Open Academic LearningAnalytics Initiative (OAAI)
Project Overview
![Page 8: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/8.jpg)
• EDUCAUSE Next Generation Learning Challenges (NGLC)
• Funded by Bill and Melinda Gates Foundations
• $250,000 over a 15 month period• Goal: Leverage Big Data concepts to create
an open-source academic early alert system and research “scaling factors”
Open Academic Analytics Initiative
![Page 9: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/9.jpg)
Student Attitude Data (SATs, current GPA, etc.)
Student Demographic Data (Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive ModelScoring
Identifies students “at risk” to not complete
course
SIS
Dat
aLM
S D
ata
OAAI Early Alert System Overview
Intervention Deployed“Awareness” or Online
Academic Support Environment (OASE)
“Creating an Open Academic Early Alert System”
Academic Alert Report (AAR)
Model DevelopedUsing Historical Data
Step #1: Developed model using historical data
![Page 10: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/10.jpg)
OAAI Goals and Milestones • Build “open ecosystem” for Learning analytics
– Sakai Collaboration and Learning Environment• Secure data capture process for extracting LMS data
– Pentaho Business Intelligence Suite• Open-source data mining, integration, analysis &
reporting– OAAI Predictive Model released under open license
• Predictive Modeling Markup Language (PMML)• Researching learning analytics scaling factors
– How “portable” are predictive models?– What intervention strategies are most effective?
![Page 11: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/11.jpg)
Research Design• Deployed OAAI system to 2200 students
across four institutions– Two Community Colleges– Two Historically Black
Colleges and Universities• One instructors teaching 3 sections
– One section was control, other 2 were treatment groups
![Page 12: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/12.jpg)
Research Design
Each instructor received an Academic Alert Report (AAR) three times during the semester at 25%, 50% and 75% into the semester.
![Page 13: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/13.jpg)
OAAI Predictive ModelDevelopment and Portability
Research Findings
![Page 14: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/14.jpg)
Predictive Model Training
![Page 15: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/15.jpg)
Predictors of Student Risk
Some predictors were discarded if not enough data was available.
LMS predictors were measured relative to course averages.
![Page 16: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/16.jpg)
Predictive Power
![Page 17: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/17.jpg)
Initial Portability Research Findings
Compared predictive elements and correlations in the Marist data to what was found by Dr. John Campbell in his dissertation research at Purdue.
![Page 18: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/18.jpg)
Institutional Profiles
![Page 19: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/19.jpg)
Spring ’12 Portability Findings
![Page 20: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/20.jpg)
Fall ’12 Portability Findings
![Page 21: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/21.jpg)
Intervention Strategies
Design, Deployment and Research Findings
![Page 22: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/22.jpg)
Student Attitude Data (SATs, current GPA, etc.)
Student Demographic Data (Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive ModelScoring
Identifies students “at risk” to not complete
course
SIS
Dat
aLM
S D
ata
OAAI Early Alert System Overview
Intervention Deployed“Awareness” or Online
Academic Support Environment (OASE)
“Creating an Open Academic Early Alert System”
Academic Alert Report (AAR)
Model DevelopedUsing Historical Data
Step #1: Developed model using historical data
![Page 23: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/23.jpg)
Awareness Messaging
“Based on your performance on recent graded assignments and exams, as well as other factors that tend to predict academic success, I am becoming worried about your ability to successfully complete this class.
I am reaching out to offer some assistance and to encourage you to consider taking steps to improve your performance. Doing so early in the semester will increase the likelihood of you successfully completing the class and avoid negatively impacting on your academic standing.”
![Page 24: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/24.jpg)
Online Academic Support Environment (OASE)
![Page 25: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/25.jpg)
OASE Design Framework
• Follows online course design concepts– Learner – Content Interactions– Learner – Facilitator Interactions– Learner – Mentor Interactions
• Leverages Open Educational Resources
![Page 26: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/26.jpg)
Design Frame #1Learner-Content Interactions• Self-Assessment Instruments
– Assist students in self-identifying areas of weakness
• OER Content for Remediation– Focus on core subjects (math, writing)
• OER Content for Learning Skills• Time management, test taking strategies, etc.
![Page 27: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/27.jpg)
Design Frame #2Learner - Facilitator Interaction
• Academic Learning Specialist role– Connecting learners to people & services– Promoting services and special events– Moderates discussions on pertinent topics
• Example: “Your first semester at college”
![Page 28: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/28.jpg)
Design Frame #3Learner - Mentor Interactions• Online interactions facilitated by student
“mentor”– Facilitates weekly “student perspective”
discussions• Online “student lounge” for informal
interactions– Let others know about study groups, etc.
![Page 29: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/29.jpg)
• Analysis showed a statistically significant positive impact on final course grades– No difference between
treatment groups
• Saw larger impact in spring then fall
• Similar trend amount low income students
Intervention Research Findings Final Course Grades
Awareness OASE Control50
60
70
80
90
100
Fin
al G
rade
(%
)
Mean Final Grade for "at Risk" Students
![Page 30: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/30.jpg)
• Student in intervention groups were statistically more likely to “master the content” then those in controls.– Content Mastery =
Grade of C or better
• Similar for low income students.
Intervention Research Findings Content Mastery
Yes No Yes No0
200
400
600
800
1000
Content Mastery for "at Risk" Students
Control Intervention
Fre
quen
cy
![Page 31: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/31.jpg)
• Students in intervention groups withdrew more frequently than controls
• Possibly due to students avoiding withdrawal penalties.
• Consistent with findings from Purdue University
Intervention Research Findings Withdrawals
Yes No Yes No0
200
400
600
800
1000
Withdrawal rates for "at Risk" Students
Control Intervention
Freq
uenc
y
![Page 32: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/32.jpg)
Instructor Feedback"Not only did this project directly assist my students by guiding students to resources to help them succeed, but as an instructor, it changed my pedagogy; I became more vigilant about reaching out to individual students and providing them with outlets to master necessary skills.
P.S. I have to say that this semester, I received the highest volume of unsolicited positive feedback from students, who reported that they felt I provided them exceptional individual attention!
![Page 33: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/33.jpg)
Future Research Interests• Factors that impact on intervention effectiveness
– Intervention Immunity – Students who do not respond to first intervention never do
– Student Engagement – How can we increase the level of engagement with OASE?
• Can predictive models be customized for specific delivery methods and programs/subjects?
• Can Learning Analytics identify “at risk” students who would otherwise not be identified?
![Page 34: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/34.jpg)
Social Networks Adapting Pedagogical Practice (SNAPP)
![Page 35: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/35.jpg)
![Page 36: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/36.jpg)
LEARNING ANALYTICS AND KNOWLEDGE CONFERENCE (LAK) THE INTERNATIONAL JOURNAL OF THE SOCIETY FOR LEARNING ANALYTICS RESEARCHSOLAR FLARES – REGIONAL CONFERENCESSOLAR STORMS – DISTRIBUTED RESEARCH LABMOOCS ON LEARNING ANALYTICS
http://www.solaresearch.org/
![Page 38: Open Academic Analytics Initiative - Campus Technology Innovator Award Presentation](https://reader035.vdocuments.mx/reader035/viewer/2022070318/557bf760d8b42a72228b496d/html5/thumbnails/38.jpg)
38
• “The obligation of knowing” – John Campbell• If we have the data and tools to improve student
success, are we obligated to use them?• Consider This > If a student has a 13% chance of passing a
course, should they be dropped? 3%?
• Who owns the data, the student? Institution?• Should students be allowed to “opt out”?
• Consider This > Is it fair to the other students if by opting out the predictive model’s power drops?
• What do we reveal to students? Instructors?• Consider This > If we tell a student in week three they have a
9% chance of passing, what will they do?• Will instructors begin to “profile” students?
LEARNING ANALYTICS ETHICS