![Page 1: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/1.jpg)
Temporally Coherent Clustering of Student Data
Severin Klingler, Tanja Käser, Barbara Solenthaler and Markus Gross
ETH Zurich, Switzerland
![Page 2: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/2.jpg)
Clustering sequential data to detect behavior patterns
2
Clustering in EDM
![Page 3: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/3.jpg)
Clustering sequential data to detect behavior patterns
3
Clustering in EDM
Peckham& McCall, 2012
![Page 4: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/4.jpg)
Clustering sequential data to detect behavior patterns
4
Clustering in EDM
Peckham& McCall, 2012Perera et al., 2009
![Page 5: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/5.jpg)
Clustering sequential data to detect behavior patterns
5
Clustering in EDM
Peckham& McCall, 2012Perera et al., 2009Martinez-Maldonado et al., 2006
![Page 6: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/6.jpg)
Clustering sequential data to detect behavior patterns
6
Clustering in EDM
Peckham& McCall, 2012Perera et al., 2009Martinez-Maldonado et al., 2006Bergner et al., 2014
![Page 7: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/7.jpg)
Clustering sequential data to detect behavior patterns
7
Clustering in EDM
Peckham& McCall, 2012Perera et al., 2009Martinez-Maldonado et al., 2006Bergner et al., 2014 Herold et al., 2014
![Page 8: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/8.jpg)
8
Evolution of behavior patterns
Session 1 Session 2 Session t
![Page 9: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/9.jpg)
Entire sequence
e.g. [Bergner et al., 2014 ], [Martinez-Maldonado et al., 2013], [Herold et al., 2013 ]
Evolutionary analysis
[Kinnebrew et al., 2013]
9
Related work
Complete training Session 1 Session 2
![Page 10: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/10.jpg)
Temporal consistency Cluster changes (e.g. size and numbers)
10
Challenges of evolutionary clustering
vs. vs.
![Page 11: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/11.jpg)
Complete processing pipeline for evolutionary clustering based on AFFECT algorithm[Xu et al., 2014]
We propose several extensions to tailor method for educational data sets
Pipeline can be used as a black box for any ITS
11
Contribution
![Page 12: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/12.jpg)
12
Pipeline overview
![Page 13: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/13.jpg)
Input to our system
The only part that is system dependent
13
Action sequences
Input, Backspace, Change View, Invalid Input
Input, Input, Input, Change View
Invalid Input, Backspace, Invalid Input, Backspace
event mapping
![Page 14: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/14.jpg)
Action sequences
provide rich temporal information
exhibit considerable amount of noise
Markov chains
actions = statestransition probabilities = frequencies
14
Action processing
![Page 15: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/15.jpg)
15
Pipeline overview
![Page 16: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/16.jpg)
Suitable similarity measure between students?
16
Similarity computation
![Page 17: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/17.jpg)
17
Similarity computation
Action sequence basedLongest common subsequences [Bergner et al., 2014]
Levenshtein distance [Desmarais & Lemieux, 2013]
I N T E * N T I O N
* E X E C U T I O N
![Page 18: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/18.jpg)
Markov chain basedEuclidean distance [Köck & Paramythis, 2011]
18
Similarity computation
Action sequence basedLongest common subsequences [Bergner et al., 2014]
Levenshtein distance [Desmarais & Lemieux, 2013]
I N T E * N T I O N
* E X E C U T I O N
![Page 19: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/19.jpg)
Markov chain basedEuclidean distance [Köck & Paramythis, 2011]
Jenson-Shannon
Hellinger
19
Similarity computation
Action sequence basedLongest common subsequences [Bergner et al., 2014]
Levenshtein distance [Desmarais & Lemieux, 2013]
I N T E * N T I O N
* E X E C U T I O N
![Page 20: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/20.jpg)
How to cluster the pairwise similarity matrices?
20
Clustering
𝑾𝟐 𝑾𝑻𝑾𝟏
users
use
rs
usersu
sers
use
rs
users
![Page 21: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/21.jpg)
How to cluster the pairwise similarity matrices?
21
Clustering
𝑾𝟐 𝑾𝑻𝑾𝟏
users
use
rs
usersu
sers
use
rs
users
Standard clustering at each time step?
![Page 22: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/22.jpg)
How to cluster the pairwise similarity matrices?
22
Clustering
𝑾𝟐 𝑾𝑻𝑾𝟏
users
use
rs
usersu
sers
use
rs
users
Standard clustering at each time step? Does not use temporal information
![Page 23: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/23.jpg)
Assumption
23
AFFECT clustering [Xu et al.,2014]
observed similarities
![Page 24: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/24.jpg)
Assumption
24
AFFECT clustering [Xu et al.,2014]
observed similarities
true similarities
![Page 25: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/25.jpg)
Assumption
25
AFFECT clustering [Xu et al.,2014]
observed similarities
true similarities
random noise
![Page 26: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/26.jpg)
Assumption
Smoothed similarity matrix proposed
26
AFFECT clustering [Xu et al.,2014]
![Page 27: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/27.jpg)
Assumption
Smoothed similarity matrix proposed
27
AFFECT clustering [Xu et al.,2014]
previous best estimate of similarities
![Page 28: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/28.jpg)
Assumption
Smoothed similarity matrix proposed
28
AFFECT clustering [Xu et al.,2014]
previous best estimate of similarities noisy observation
![Page 29: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/29.jpg)
Assumption
Smoothed similarity matrix proposed
29
AFFECT clustering [Xu et al.,2014]
previous best estimate of similarities noisy observation
controls amount of smoothing
![Page 30: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/30.jpg)
Optimal 𝛼 alpha as a trade-off:
30
AFFECT clustering [Xu et al.,2014]
estimated noise
Amount of new information
![Page 31: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/31.jpg)
Based on the estimates Ψ𝑡 we apply static k means clustering
31
AFFECT clustering [Xu et al.,2014]
Ψ1 = Ψ2 = Ψ𝑡 =
![Page 32: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/32.jpg)
32
Pipeline overview
![Page 33: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/33.jpg)
We expect clusters to change over time
• growth and shrinkage
• dissolving and forming
33
Model selection
Determine the number of clusters at each time step
![Page 34: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/34.jpg)
34
Model selection
𝐴𝐼𝐶𝑐 = −2 ln 𝐿𝐿 + 2𝑃 +2𝑃(𝑃 + 1)
𝑛 − 𝑃 − 1
![Page 35: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/35.jpg)
35
Model selection
𝐴𝐼𝐶𝑐 = −2 ln 𝐿𝐿 + 2𝑃 +2𝑃(𝑃 + 1)
𝑛 − 𝑃 − 1Likelihood [Pelleg & Moore, 2000]
spherical Gaussians
based on empirical variance
![Page 36: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/36.jpg)
36
Model selection
𝐴𝐼𝐶𝑐 = −2 ln 𝐿𝐿 + 2𝑃 +2𝑃(𝑃 + 1)
𝑛 − 𝑃 − 1
Number of parameters
based on effective dimensionality[Krikpatrick, 2000]
![Page 37: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/37.jpg)
37
Model selection
𝐴𝐼𝐶𝑐 = −2 ln 𝐿𝐿 + 2𝑃 +2𝑃(𝑃 + 1)
𝑛 − 𝑃 − 1
Correction for finite sample size[Burnham & Anderson, 2002]
![Page 38: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/38.jpg)
Synthetic experiments
• Performance evaluation of our method based on ground truth
• Robustness to noise
Exploratory data analysis
• Cluster extraction on real world data
• Comparison across ITS
38
Evaluation
![Page 39: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/39.jpg)
Synthetic experiments
• Performance evaluation of our method based on ground truth
• Robustness to noise
Exploratory data analysis
• Cluster extraction on real world data
• Comparison across ITS
39
Evaluation
![Page 40: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/40.jpg)
80 students over 50 sessions solving 20 tasks (8 steps)
Rasch model for probability of correctly solving a task
40
Synthetic data generation
𝑑𝑖 𝑦𝑛,𝑖 𝜃𝑛
items 𝑰
students 𝑵
item difficulties student abilities
![Page 41: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/41.jpg)
41
Synthetic data generation
correct step?
emit correct
emit incorrect
emit help
emit new task
steps done? help?
yes
nono
no
yes
yes
according to Rasch model
![Page 42: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/42.jpg)
42
Synthetic data generation
correct step?
emit correct
emit incorrect
emit help
emit new task
steps done? help?
yes
nono
no
yes
yes
did we attempt all 8 sub steps?
![Page 43: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/43.jpg)
43
Synthetic data generation
correct step?
emit correct
emit incorrect
emit help
emit new task
steps done? help?
yes
nono
no
yes
yesBernoulli sample
![Page 44: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/44.jpg)
44
Synthetic data generation
correct step?
emit correct
emit incorrect
emit help
emit new task
steps done? help?
yes
nono
no
yes
yes adjusted to overall
probability
![Page 45: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/45.jpg)
We simulated four student groups with different behavior
45
Synthetic data generation
Good performance Frequent help request 𝜽 𝒑𝑯
-1 0.05
1 0.05
-1 0.2
1 0.2
![Page 46: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/46.jpg)
We compare to the following methods (cross-validated)
LCS_KM* Longest common subsequence -> k-means
MC_EUC_KM** Markov chains -> Euclidean dist. -> k-means
Ours_HD Markov chains -> Hellinger dist. -> AFFECT clustering
Ours_SD Markov chains -> Shannon div . -> AFFECT clustering
Ours_EUC Markov chains -> Euclidean dist. -> AFFECT clustering
* [Bergner et al., 2014] **[Köck & Paramythis, 2011]
46
Clustering Quality & Robustness
![Page 47: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/47.jpg)
47
Clustering Quality & Robustness
![Page 48: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/48.jpg)
48
Clustering Quality & Robustness
Little noise, P = 0.82
P = 0.53
![Page 49: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/49.jpg)
49
Clustering Quality & Robustness
Noisy observations P = 0.45
P = 0.34
![Page 50: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/50.jpg)
Temporal stability over 50 simulated sessions
50
Stability
![Page 51: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/51.jpg)
Temporal stability over 50 simulated sessions
51
Stability
Very unstable (despite no cluster change
![Page 52: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/52.jpg)
Temporal stability over 50 simulated sessions
52
Stability
stable clustering (exploit temporal information)
![Page 53: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/53.jpg)
Identification of cluster numbers and sizes
53
Interpretability
![Page 54: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/54.jpg)
Identification of cluster numbers and sizes
54
Interpretability
Our pipeline correctly
identifies cluster events.
![Page 55: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/55.jpg)
Synthetic experiments
• Performance evaluation of our method based on ground truth
• Robustness to noise
Exploratory data analysis
• Cluster extraction on real world data
• Comparison across ITS
55
Evaluation
![Page 56: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/56.jpg)
Clustering student interactions in two different ITS
56
Exploratory data analysis
![Page 57: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/57.jpg)
Clustering student interactions in two different ITS
57
Exploratory data analysis
Calcularis
Data from 134 studentsIntelligent tutoring systemChildren with difficulties in mathematics
![Page 58: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/58.jpg)
Clustering student interactions in two different ITS
58
Exploratory data analysis
Orthograph
Data from 106 studentsComputer-based training Children with dyslexia
![Page 59: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/59.jpg)
59
Example: Navigation behavior
![Page 60: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/60.jpg)
60
Navigation behavior
![Page 61: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/61.jpg)
61
Navigation behavior
Beginning 7 different clusters
![Page 62: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/62.jpg)
62
Navigation behavior
Children spent more than 50% off task
![Page 63: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/63.jpg)
63
Navigation behavior
Cluster dissolves
![Page 64: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/64.jpg)
64
Navigation behavior
![Page 65: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/65.jpg)
65
Navigation behaviorVery focused on training (80% in training)
![Page 66: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/66.jpg)
66
Navigation behavior34% in shop, high transition probabilities
![Page 67: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/67.jpg)
67
Navigation behavior
Very likely to immediately return to game
![Page 68: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/68.jpg)
68
Navigation behavior
Equal transition probabilities for next view
![Page 69: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/69.jpg)
69
Navigation behavior
Still focused, 76% in training
![Page 70: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/70.jpg)
70
Navigation behavior
Still focused, 76% in training
47% off task
![Page 71: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/71.jpg)
71
Navigation behavior
Still focused, 76% in trainingMore frequent transitions to shop (58% )
than performance (17%)
47% off task
![Page 72: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/72.jpg)
72
Help Seeking Behavior
![Page 73: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/73.jpg)
73
Help Seeking Behavior
![Page 74: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/74.jpg)
74
Help Seeking Behavior
People try out a lot of different
strategies in the beginning
Large variance in student
behavior
![Page 75: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/75.jpg)
75
Help Seeking Behavior
People try out a lot of different
strategies in the beginningDiversity disappears
![Page 76: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/76.jpg)
76
Help Seeking Behavior
Experimentation with help systems
Almost no use of help functions
![Page 77: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/77.jpg)
77
Help Seeking Behavior
Students more likely to perform
help requests (13%)
Students less likely to perform
help requests (3%)
![Page 78: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/78.jpg)
Enforcing temporal coherence beneficial for
• detection of student behavior
• stable detection of cluster events
Exploratory analysis demonstrated
• Reveal interesting properties about student behavior
• Pipeline can be used as a black box for any ITS
78
Conclusion
![Page 79: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/79.jpg)
79
Thank you.
![Page 80: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/80.jpg)
[Bergner et al., 2014] Y. Bergner, Z. Shu, and A. A. Von Davier. Visualization and Confirmatory Clustering of Sequence Data from a Simulation-Based Assessment Task. In Proc. EDM, 2014.
[Köck & Paramythis, 2011] M. Köck and A. Paramythis. Activity sequence modelling and dynamic clustering for personalized e-learning. UMUAI, 2011.
[Desmarais & Lemieux, 2013] M. Desmarais and F. Lemieux. Clustering and visualizing study state sequences. In Proc. EDM, 2013.
[Xu et al., 2014] K. S. Xu, M. Kliger, and A. O. Hero Iii. Adaptive evolutionary clustering. Data Mining and Knowledge Discovery, 2014.
[Peckham& McCall, 2012] T. Peckham and G. McCalla. Mining Student Behavior Patterns in Reading Comprehension Tasks. In Proc. EDM, 2012.
[Perera et al., 2009 ] D. Perera, J. Kay, I. Koprinska, K. Yacef, and O. R. Zaıane. Clustering and sequential pattern mining of online collaborative learning data. TKDE, 2009.
[Martinez-Maldonado et al., 2013 ] R. Martinez-Maldonado, K. Yacef, and J. Kay. Data mining in the classroom: Discovering groups’ strategies at a multi-tabletop environment. In Proc. EDM, 2013.
[Herold et al., 2014] J. Herold, A. Zundel, and T. F. Stahovich. Mining meaningful patterns from students’ handwritten coursework. In Proc. EDM, 2013.
[Krikpatrick, 2000] M. Kirkpatrick. Patterns of quantitative genetic variation in multiple dimensions. Genetica, 2006.
[Burnham & Anderson, 2002] Burnham, K. P.; Anderson, D. R. (2002), Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.), Springer-Verlag
[Pelleg & Moore, 2000] D. Pelleg and A. Moore. X-means: Extending K-means with Efficient Estimation of the Number of Clusters. In Proc. ICML, 2000.[Kinnebrew et al., 2013] J. S. Kinnebrew, D. L. Mack, and G. Biswas. Mining temporally-interesting learning behavior patterns. In Proc. EDM, 2013.
80
References
![Page 81: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/81.jpg)
Selected features cover broad range of characteristics
in accordance with the literature on DD
High sensitivity (0.91) and specificity (0.91)
Good construct validity
Reliability
81
Discussion
approximation only
![Page 82: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/82.jpg)
82
Parameter influence
Analysis of parameter effects
Study effect on performance using linear regression
Variables
All 6 model parameters
additionally: correct ratio and average number of tasks
![Page 83: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/83.jpg)
Predictive performance
Task outcome Knowledge state Parameter spaceConvergence
propertiesModel
robustness
BKT
IRT
FAST
LFKT
83
Parameter space
![Page 84: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/84.jpg)
Parameters configuration designed to match real world conditions
84
Experimental setup
Parameter Configurations Reference
BKT parameters Sampled around clusters [Ritter et al. 2009]
Student abilities N(0,1) [Harris 1989]
Range of item difficulties [0,3] [Harris 1989]
Feature weights [0,1.5] According to item difficulties
![Page 85: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/85.jpg)
Clustering sequential data to detect behavior patterns
85
Clustering in EDM
Peckham& McCall, 2012Perera et al., 2009
Martinez-Maldonado et al., 2006
Herold et al., 2014
Bergner et al., 2014
![Page 86: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/86.jpg)
Optimal 𝛼 alpha as a trade-off:
86
AFFECT clustering [Xu et al.,2014]
estimated noise
Amount of new information
Problem: var(n𝑖𝑗) and 𝜓𝑡𝑖𝑗
are unknown
[Xu,2014] propose estimate
![Page 87: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/87.jpg)
87
Navigation BehaviorVery focused on training 80% in training
34% in shop, high transition probabilities
Very likely to immediately return to game
Equal transition probabilities for next view
Still focused, 76% in trainingMore frequent transitions to shop (58% )
than performance (17%)
47% off task
![Page 88: Temporally Coherent Clustering of Student Data - Educational Data Mining · 2016-07-03 · Temporally Coherent Clustering of Student Data Severin ... 2013] R. Martinez-Maldonado,](https://reader033.vdocuments.mx/reader033/viewer/2022042309/5ed5c5aa08214b19be6a1b5f/html5/thumbnails/88.jpg)
Similarity computations and clustering is O(n^2)
Investigation of different clustering algorithms
More analysis tools (such as integrated PCA)
88
Limitations