personal analytics: getting off the deficit path - alasi... · video tutorial of surgery and...

Personal Analytics: Getting off the

deficit path

Professor Gregor Kennedy The University of Melbourne

What are Personal Analytics?

http://birdsontheblog.co.uk/getting-fitter-and-healthier-with-the-fitbit/


https://gigaom.com/2011/11/07/is-klout-crossing-the-line-when-it-comes-to-privacy/

Big Data = Analytics

https://www.linkedin.com/today/post/article/20140312180810-246665791-the-future-of-big-data-and-analytics

Learning Analytics is all the Rage

With Great Promise

• Detect potential “at risk” students

• Formative and summative feedback to students on their learning processes and outcomes

• Assist with evidence-based resource allocation

• Improve institutional decision-making and responsiveness to known challenges

• Promote a shared understanding of institutional successes and challenges

• Academic research and development

(Long & Siemens, 2011)

Defining Analytics

Society for Learning Analytics Research (2011)

Learning Analytics is the measurement, collection, analysis and reporting of data about learners and their contexts for the purposes of understanding and optimizing learning, and the environments in which it occurs. Academic Analytics is the improvement of organizational processes, workflows, resource allocation, and institutional measurement through the use of learner, academic, and institutional data.

Managers, Administrators, Funders

Learners, Educators, Teachers

THIS BIT IS NOT NEW

Two Traditions

Intelligent Tutoring Systems

Interactivity Research

Two Traditions


Taxonomies of Interaction

Taxonomies and Classifications e.g. Thompson & Jorgenson (1989)

Reactive Interactive Proactive


Taxonomies and Classifications e.g. Schwier & Misanchuk (1993)

Reactive Proactive Mutual


Taxonomies of Interaction


Concerns about the past

• Often use fairly raw metrics, simple student measures and inputs (e.g. MCQs, simple access counts).

• Largely descriptive (useful) but often fails to complete the feedback loop to students and/or teachers.

Two Traditions



Two Traditions


Student Model

Pedagogical Model

Domain Knowledge

Feedback

Two Traditions

Intelligent Tutoring Systems Flag the Error

Explain the Error

Give a Hint

Show a worked example

(Mike Timms)


• “ITS were recognised as narrow and brittle” (Cumming & McDougall, 2000) • … heavily reliant on educational programs and applications that

had defined or discrete stages and steps.

• They were often tied to a program and were not generalisable.

Concerns about the past


Student Smart

System

Assess

Diagnose

Recognise Personal

Adaptive


Two Traditions Combined

Two Traditions



Drill and Practice

Procedural Simulation

Conceptual Simulation

Drill and Practice

Student Path

A B C

Content Content Content Content Content

A B C

A B C

A B C

A B C

Feedback Content

X

Procedural Simulation

Implicit Feedback

Explicit Feedback

X Student Path

Conceptual Simulation

X

Student Path

Implicit Feedback

Explicit Feedback

Back to (Today’s) Analytics

Detect “At Risk” Students

for Retention

Teaching & Learning Research, Evaluation

& QA

Personalised or Adaptive Feedback

for Learning

How Today’s Analytics are Used

✓

✓

✓

• Purdue University’s “Signals”

• Used to predict students who are “at risk”

• Individual student risk is predicted using an algorithm based on data from four sources:

− Performance … “points earned in a course to date”

− Effort … interaction with the learning management

system as compared to peers

− Academic history … e.g. GPA, prior academic history

− Student characteristics … e.g. residency, age

(Arnold & Pistilli, 2012)

#1 … “At Risk” Analytics

• Post to students’ LMS • Email or text students • Refer them to an advisor • Call for a chat

(Arnold & Pistilli, 2012)

“At Risk” Analytics

How Today’s Analytics are Used

Analytics = Diagnosing Deficit Path

Preferred Parameters or Pathway

Student Path

Feedback

Assess, diagnose and recognise “deficit”

The field of educational technology has always been interested in using students’ digital traces to assess and diagnose when they move away from preferred learning pathways.

X

This approach has useful pedagogical applications … but

Macro: Attrition

Micro: Drill & Practice

Deficit Pathways

So what?

Is this a problem?

The Promise of Learning Analytics

How can we …

• harness different data analysis techniques

• for the provision of more meaningful feedback

• to students on their learning processes

• in real time

• for genuinely personlised learning environments?

A core promise of learning analytics is to improving students’ micro learning processes in order to enhance their learning outcomes

Getting off the Deficit Path

From Personal Deficit Analytics … … to Personal Learning Analytics

Example 1: Surgical Skills Simulation

James Bailey Professor, Computing & Information Systems

Ioanna Ioannou Research Fellow, Otolaryngology

Stephen O'Leary Professor, Otolaryngology

Patorn Piromchai PhD Student, Otolaryngology

Sudathi Wijewickrema Research Fellow, Otolaryngology

Yun Zhou PhD Student, Computing & Information Systems

Example 1: Surgical Skills Simulation

O'Leary, S., et al. (2008). Validation of a networked virtual reality simulation of temporal bone surgery. The Laryngoscope, 118(6), 1040-1046.

Metrics from the Simulator

• Tool position, orientation and force metrics - e.g. current force applied by the drill • Burr metrics - e.g. radius of the current burr • Anatomical structure metrics - e.g. distance of the drill tip to the closest point of one of three key anatomical structures • Bone specimen metrics - e.g. rotation of the bone

----- 15 records of 48 metrics generated per second -----

• A sequence of points containing a continuous drilling motion

• The end of a stroke is reached - when drilling ceases; or - when there is an abrupt change in the direction of drilling

• Once a way of identifying strokes has been determined a

range of “stroke metrics” can be calculated from the data stream output by the simulator (e.g. stroke duration, stroke length, average stroke speed, minimum distance of stroke to structures, etc.)

A Key Metric: Stroke

(Hall, Rathod, et al., 2008)

Data Mining for Personal Feedback

• We needed to provide personalised feedback to trainees across multiple dimensions or features in an open, complex, procedural simulation.

• Not just deficit feedback about manifest error or procedural stage - “You hit the facial nerve” - “You should have completed X before Y”

• For example:

- force used - stroke length - stroke smoothness - distance to critical structures, etc.

• Prototype 1: Hidden Markov Models built to discriminate patterns of novice and expert behaviour on a single association rule.

• Prototype 2: A range of analysis techniques used to develop models to provide feedback on multiple features:

- A random forest model to determine expert/novice behaviour

- Nearest neighbour techniques along with a random forest model to generate feedback in the case of novice behaviour

- An independent feedback system (application) was built

Data Mining for Personal Feedback

Feedback Parser

Feedback Generator

Simulator Stroke Detector Simulator Metrics

Proximity Triggers

Feedback

Stroke Metrics

Stroke Metrics Technique Feedback

A Personal Feedback System

A Personal Feedback System

• 24 medical students - 12 were provided with automated feedback - 12 were not

• Knowledge of anatomy but not surgery;

video tutorial of surgery and simulator familiarisation.

• Two group comparison of students’ performance on a cortical mastoidectomy - Effectiveness of technique feedback - Accuracy of feedback - Usability of system

Feedback System Test

Effectiveness of Technique

% of Expert Stokes

With Feedback

Without Feedback

M (SD) M (SD) F p

61.59 (16.19) 38.86 (13.11) 14.29 <.001

Effectiveness of Technique

• A surgeon undertook a post hoc analysis of the feedback provided by the system

- False Positives: feedback was provided when stroke technique was acceptable

- False Negatives: feedback was not provided when technique was unacceptable.

- Wrong Feedback: participants’ technique was accurately classified as “trainee” but the content of the feedback was inaccurate.

Accuracy of Feedback

# of Feedback Messages

Percentage (of Total)

False Positives 39 6.8%

False Negatives 69 11.4%

Wrong Feedback 52 9.0%

Total Feedback 576

Accuracy of Feedback

Overwhelmingly positive feedback from participants on the basis of post-test interviews “it reminded me to be gentle near structures”

“particularly helpful was changing burr size and whether or not to zoom in”

“it gave me the confidence to go faster”

Usability

Example 2: Cognition and Interaction

Barney Dalgarno Charles Sturt University

Sue Bennett University of Wollongong

Dalgarno, B., Kennedy, G., & Bennett, S. (in press). The impact of students’ exploration strategies on discovery learning using computer-based simulations.Educational Media International. Accepted Oct 2014.

Example 2: Cognition and Interaction

How does the design of interactive multimedia impact on students’ learning strategies and cognition?

Study

• Two learning conditions (observation & exploration)

• Two content areas (global warming & blood alcohol concentration)

• Each participant (n=158) completed:

– the observation condition in one content area – the exploration condition in the other – a pre and post-test of knowledge in each content area

• Students’ actions were logged.

Exploration Condition • Content screens providing background to the area and

some terminology explanation, but no explanation of key concepts.

• A series of screens allowing students to manipulate the simulation parameters and asking them to “predict, observe, explain”.

Observation Condition • The same series of background content screens. • A series of simulation output screens each showing the

effect of pre-set manipulations of input parameters.

Multimedia Design

Blood Alcohol Concentration

• No effect of learning condition for Global Warming

• Modest main effect of learning condition for Blood Alcohol

Mean Post-Test

Observation

Mean Post-Test

Exploration

F (1,155) p

Global Warming 1.42 1.72 2.40 0.13

Blood Alcohol 3.42 3.93 5.52 0.02

*ANCOVA using pre-test as a covariate

Results: Observation v Exploration

• We noticed that the variance in post-test scores for exploration participants was quite high.

• Eyeballing the logs showed some students seemed more systematic in their exploration of the simulation than others.

End of Story?

Students’ learning behaviours, strategies and approaches were characterised using various

heuristics as well as cluster analysis.

Cluster Analysis [not shown here!]

• time spent on the background material before simulation

• total time spent on the simulation

• number of cycles in which exactly one variable was changed from the previous cycle

• number of cycles in which exactly one variable was changed from the provided base values

• number of cycles where at least one variable was changed from the previous cycle

• the sum of the number of variables changed per cycle across all cycles

Classifying Students Approaches

Systematic Exploration • Students who completed 4 or more simulation cycles with

only one variable changed from previous cycle; or

• Students who completed 4 or more simulation cycles with only one variable changed from pre-set values

Non-Systematic Exploration • All other “exploration” students

Observation

• Students who completed the Observation learning condition (no simulation)

Classifying Students Approaches

• Significant main effect of condition in both content domains

Post-Test Observation

Non-Systematic Post-Test

Exploration

Systematic Post-Test

Exploration

F

p

Global Warming 1.42 (1.29) 1.33 (1.52) 2.48 (2.20) 4.17 .017

Blood Alcohol 3.42 (1.31) 3.51 (1.30) 4.56 (1.33) 8.69 <.001

Observation = Non Systematic < Systematic

Outcome by Approach

Getting Off the Deficit Path

From Personal Deficit Analytics … … to Personal Learning Analytics

Implications …

• Each study presents a simulation-based, digital learning environment: one procedural, one conceptual.

• Each employs analytic approaches which are very much framed by the learning design of these environments and the pedagogical intent of the task:

– what we wanted students to do and learn.

• But the approaches attempt to move away from a narrow “deficit” model of analytics based on

– assessing what students know; – how much they fail to participate; and/or – how much they are “at risk” of disengaging from the task (or dropping out of the course)

Implications …

• These studies show how learning analytics can be used to uncover complex patterns of students’ on-task learning behaviour which are:

– indicative of distinct approaches to learning; – correspond to adaptive (and maladaptive) learning or “thinking” processes; and – are associated with good (or poor) learning outcomes.

Analytics that are used to determine students’ adaptive learning patterns and processes with digital learning tasks, which can be used as the basis for individualised, personal feedback, to improve their learning processes and, ultimately, their learning outcomes.

What are Personal Learning Analytics?

PLA : Simply a Matter of Perspective?

http://www.marquette.edu/magazine/recent.php?subaction=showfull&id=1357063200

Deficit

Personal

PLA : Simply a Matter of Perspective?

A Work in Progress

Deficit Analytics in T&L Personal Analytics in T&L

Assessing what you know Determining how you are coming to know

Determining how much you don’t participate and access

Determining the way in which you participate

Macro: often multidimensional Macro: does not speak to macro

Micro: based on simple knowledge assessments

Micro: based on approaches and responses to the learning context

Micro: profiles of simple access to learning resources and activities

Patterns of interactions with learning activities and tasks


Is there a role for Fit Bit for students? • Tracking engagement with University

• “macro” level interactions with

learning activities and resources

Conclusion

• The holy grail … and the somewhat elusive promise of learning analytics is to create genuinely adaptive and personalised online environments to improve individual student’s learning.

• Identifying when students’ transgress and step off the defined

learning path gets us some of the way …

• … but understanding how we can use students’ adaptive patterns of engagement with specific learning tasks is an important next step …

PBC Directions : eLearning Incubator

PBC Bid Director of eLearning

eLI

CSHE

Support ITS

Academics (F&GS)

Learning Environments

Business Development

Thanks

[email protected]

personal analytics: getting off the deficit path - alasi... · video tutorial of surgery and...

Documents