learner analytics: hype, research and practice in moodle
DESCRIPTION
Presentation by John Whitmer, Michael Haskell (Cal Poly SLO), and Hillary Kaplowitz (CSU Northtridge) at US West Coast Moodle Moot 2012. “Learner Analytics” has captured the attention of the media and is the topic of much debate in professional and academic circles. What lies behind the hype? In this presentation, we will discuss the state and limits to current in research in LMS Learner Analytics. We will then look at examples of Learner Analytics in Moodle, including tools for faculty and reports for reporting across the entire instance.TRANSCRIPT
US West Coast Moodle Moot 2012
John Whitmer, CSU Chico (& Office of the Chancellor)Michael Haskell, Cal Poly San Luis Obispo
Hillary Kaplowitz, CSU Northridge
Learner Analytics: Hype, Research and Practice in Moodle
Download slides at: http://bit.ly/QttGnd
2
“But everything we know about cognition suggests that a small group of people, no matter how intellingent, simply will not be smarter than the larger group. ... Centralization is not the answer. But aggregation is.”
- J. Surowiecki, The Wisdom of Crowds, 2004
Outline
1. Hype & Promise of Learner Analytics
2. Campus Case Studies– Getting Started w/Institutional Reporting (Mike)– Analytics at work in the classroom (Hillary)– Evaluating course redesign (John)
3. Q & A
1. HYPE & PROMISE OF LEARNER ANALYTICS
John Goodlad’s Place-Based Research
Classroom-based research: “What is schooling?”
1,000 classrooms, 27,000 individuals
14 foundations needed to support
Fundamental changes to understanding of educational practice
Steve Lohr, NY Times, August 5, 2009
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.
8
Source: jisc_infonet @ Flickr.com
Source: jisc_infonet @ Flickr.com
Learner Analytics
“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)
Fundamental Questions behind Learner Analytics
1. How are students using technology?
2. Does it matter (re: achievement, engagement, learning)?
3. How does this relationship vary (by student, by course, by goal)?
4. What should we do? – Changes in student behavior? – Changes in faculty/program?
L.A. Empirical Research Findings
2. CAMPUS CASE STUDIES
Download slides at: http://bit.ly/QttGnd
Getting Started with Institutional Reporting
Michael Haskell
Cal Poly, San Luis Obispo
We perform Analytics, but are we doing Learning Analytics?
?
What can we do in the meantime…
How far away is Learning Analytics?
Sounds like it’s about 2-3 Years out…
Can we wait that long?
Institutional Reporting
What information is available?
Where is it?
How can we use it?
Types of Information
Individual Behavior
Content Population Behavior
Location of Information
Web Server Logs Population
Moodle Log Table (mdl_log)IndividualContent Populatio
n
Google Analytics Population
Moodle Database Structure Content Individual
http://docs.moodle.org/dev/Database_schema_introduction
Moodle Database Structure
Modules by Course SQL: https://gist.github.com/3203120
How do we utilize this information?Foster collaboration between Faculty
“ Top 10 Instructors TabIn this section, the data was further categorized to find the top 10 instructors in each college who “used the most modules” and “created the most of each module”. The first two graphs show the top 10 instructors from all the colleges.In the first graph, the instructors who used the most modules (8 modules) were X and Y, who are from the College of Engineering and College of Ag, Food and Env respectively. In that same section, Z from the College of Science and Math is listed down three times for classes in the top 10. ”- Student Researcher
How do we utilize this information?To keep a pulse on adoption
How do we utilize this information?
Percentage of Activated Courses by College (Spring 2012)
16%
84%
College of Agriculture, Food and Environmental Science
29%
71%
College of Architecture & Envi-ronmental Design
53%47%
Orfalea College of Business
21%
79%
College of Engineering
26%
74%
College of Liberal Arts
37%
63%
College of Science & Mathematics
To keep a pulse on adoption
How do we utilize this information?To learn how instructors leverage Moodle.
Determine where developer time is best spent.
How do we utilize this information?
Moodle Admin: There’s a problem with Module X.
Instructional Designer: The problem will be fixed soon, but in the meantime I have a workaround I’d like to communicate to instructors. Hmm… I don’t want to reach out to every instructor. Can you provide a list of all the instructors who use Module X?
Moodle AdminNo Problem.
Informed Communication
Conclusions
• Current• Manual Exploration• A lot of Small Wins
• Future• Automate reporting of top tens• Open up the data to a wider audience• Take action on data we have• Keep an eye on LA Tools for faculty and
students
How can data help teachers and students work better
together?
Hillary Kaplowitz
Instructional Designer, Faculty Technology Center
Part-Time Faculty, Cinema and Television Arts Department
California State University, Northridge
Case #1
“I'm not upset that you lied to me, I'm upset that from now on I can't believe you.”
Friedrich Nietzsche
“Hey Professor,
I just looked at my assignments and realized that my Chapter 11 summary did not get submitted, which I'm having trouble believing that I didn't submit it... especially because I see that I did it, and I always submit my assignments as soon as I finish them.”
Now the hard part….
Do I believe him?
If I only I could check…
And it was all his idea…
The student suggested that I check Moodle and if that didn’t work told me how to check the Revision History in GoogleDocs with step-by-step directions!
Case #2
“Life isn't fair. It's just fairer than death, that's all.”
William Golding
“The quiz is unfair”
Hybrid Course Weekly Structure
1. Watch lectures
2. Read textbook
3. Online chat and tutoring
4. Post questions and take practice
quiz
5. Class meets
6. Aplia quiz
But the story was not that simple…
• Reports on Moodle painted a different picture• Student was watching the lectures at 10:00 p.m.• Then immediately taking quiz
Enabled constructive feedback…
Advised the student how the structure of the course was designed to enhance learning
Student revised their study habits Improved grades and thanked the instructor!
What we can do with data now
Use Reports in Moodle to verify student claims Review participant list to see last access time Empower students to review their own reports Analyze usage and advise students how to study better Review quiz results to find common misconceptions
Could we help improve student learning outcomes if we knew the effect of…
Coffee
Sequencing
Amount
Textbook
LMS Access
LMS Activities
Mobile
Attendance
43
EVALUATING COURSE REDESIGN: INTRO TO RELIGIOUS STUDIES 180
44
Front-end: What? Why? Evaluation for Program Assessment• Year-long faculty course redesign program
• Case: Intro to Religious Studies: increased enrollment from 80 to 373 students first semester: 250,000 course website hits
• Outcome: increased mastery course concepts AND increased number D/W/F students
• Why? (and for whom?)
• What is the relationship between LMS actions, student background characteristics and student academic achievement? (6 million dollar question)
45
Back-end: How?
• Integrated data from LMS log files, student enrollment records, and course grade
• LMS logfiles are “data exhaust” for server analysis
• Filtering and cleaning reduced 250K records to 71k
• Analysis tools: Excel, Tableau (visualization), Stata (statistical analysis)
46
grade
A A- B+ B B- C+ C C- D+ D F W
0K
5K
10K
15K
20K
25K
30K
Avg. total_dwell0
20
40
60
80
100
120
140
160
180
200
220
240
260
280
Avg. total_hits
Measure Names
Avg. total_dwell
Avg. total_hits
47
Grades by Hits & Dwell Timegrade
A A- B+ B B- C+ C C- D+ D F W
0K
5K
10K
15K
20K
25K
30K
Avg. total_dwell
0
20
40
60
80
100
120
140
160
180
200
220
240
260
Avg. total_hits
Measure Names
Avg. total_dwell
Avg. total_hits
Pell v. Non-Pell: Grades by Hit/Dwellgrade
A A- B+ B B- C+ C C- D+ D F W
0K
10K
20K
30K
Avg. total_dwell
0
50
100
150
200
250
300
Avg. total_hits
Pell Eligible
Pell-Eligible
Not Pell-Eligible
49
Content: the Time Differentialgrade / pelleligible
A B+ C C-
Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible Pell-Eligible Not Pell-Eligible
0K
5K
10K
15K
20K
25K
30K
35K
Value
Content
Content
Engage
Engage
Assess
Assess
Admin
Admin
Content
Content
Engage
Engage
Assess
Assess
Admin
Content
Content
Engage
Engage
Assess
Assess
Content
Content Engage
Engage
Assess
Assess
Admin
Admin
Measure Names
Admin
Assess
Engage
Content
Call to Action
1. You’re *not* behind the curve, this is a rapidly emerging area that we can (should) lead ...
2. Metrics reporting is the foundation for Analytics
3. Don’t need to wait for student characteristics and detailed database information; LMS data can provide significant insights
4. If there’s any ed tech software folks in the audience, please help us with better reporting!
52
Q&A and Contact Info
Download slides at: http://bit.ly/QttGndResources Googledoc: http://bit.ly/HrG6Dm
Contact Info: • John Whitmer ([email protected])• Michael Haskell ([email protected])• Hillary C Kaplowitz ([email protected])
53
Works CitedAdams, B., Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing Teaching Learning through Educational Data Mining and Learning Analytics: An Issue Brief. Washington, D.C.: U.S. Department of Education, Office of Educational Technology.Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1). Bousquet, M. (2012). Robots Are Grading your Papers. Retrieved from http://chronicle.com/blogs/brainstorm/robots-are-grading-your-papers/45833Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics: A New Tool for a New Era. EDUCAUSE Review, 42(4), 17. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.LaValle, S., Hopkins, M., Lesser, E., Shockley, R., & Kruschwitz, N. (2010). Analytics: The new path to value. Findings from the 2010 New Intelligent Enterprise Global Executive Study and Research Project: IBM Institute for Business Value and MIT Sloan Management Review.Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Hung Byers, A. (2011). Big data: The next frontier for innovation, competition, and productivity.Parry, M. (Producer). (2012, 5/14/2012). Me.edu: Debating the Coming Personalization of Higher Ed. Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/wiredcampus/me-edu-debating-the-coming-personalization-of-higher-ed/36057Siemens, G. (2011, 8/5). Learning and Academic Analytics. Retrieved from http://www.learninganalytics.net/