big data and learning - uc engageucengage2014.ucop.edu/pdfs/session-1-data-analytics-how... ·...

Big Data and Learning Analytics for decisions and discovery Zachary A. Pardos, PhD Assistant Professor School of Information & Graduate School of Education UC Berkeley

Background

K K K

Q Q Q

P(T|H) P(T|H)P(L0)

P(G)P(S)

Student

Student Skill Interaction for P(T)

Node statesK , Q, H = Two state (0 or 1)Student = Multi state (1 to N)(Where N is the number of students in the training data)

H

P(H|Student)(multistep method – step 2)

Model ParametersP(L0) = Skill probability of initial knowledgeP(T|H) = Skill probability of learning given high or low individual student learning rate, HP(G) = Skill probability of guessP(S) = Skill probability of slip

Intelligent Tutoring Systems K-12 Platforms

10/24/14 4

BerkeleyX MOOC Data

•  518,000 registered learners since Spring 2013 •  Producing 350M logged events •  35% report already having a Bachelor's degree •  25% report already having a Master’s degree

Median age = 28 Std = 10

Not the typical residential student population

Research Questions

10/24/14 5

•  What are students learning from? •  How can we recommend better pathways to learning? •  How can we provide formative feedback to teachers?

–  How well aligned are the assessments with the content –  What pedagogy is working –  For whom

10/24/14 6

Answering problems on the platform, receiving feedback and help

Students are learning from… Students are learning from…

The data Student par+cipa+on •  154,000 enrolled •  108,000 entered class •  7,000 received cer5ficate

Course components •  434 lecture videos •  37 homework problems •  105 lecture problems •  1009 book pages •  12 labs •  145 tutorial videos •  2 exams •  Wiki •  Discussion board

course interface

The Platform

Zach Pardos UC eNGAGE

User Res Time Resp1 Resp2 Count1 Count2

Learning pathway example Learning objective: Answer homework problem correctly

User Res Time Resp1 Resp2 Count1 Count2 sarah Lec1p1.5 1m52s incorrect - - 1 - -

User Res Time Resp1 Resp2 Count1 Count2 sarah Lec1p1.5 1m52s incorrect - - 1 - - sarah S1V8 0m58s - - - - - - - -

User Res Time Resp1 Resp2 Count1 Count2 sarah Lec1p1.5 1m52s incorrect - - 1 - - sarah S1V8 0m58s - - - - - - - - sarah Lec1p1.5 0m22s incorrect incorrect 2 1

User Res Time Resp1 Resp2 Count1 Count2 sarah Lec1p1.5 1m52s incorrect - - 1 - - sarah S1V8 0m58s - - - - - - - - sarah Lec1p1.5 0m22s incorrect incorrect 2 1 sarah Book.p27 0m38s - - - - - - - -

User Res Time Resp1 Resp2 Count1 Count2 sarah Lec1p1.5 1m52s incorrect - - 1 - - sarah S1V8 0m58s - - - - - - - - sarah Lec1p1.5 0m22s incorrect incorrect 2 1 sarah Book.p27 0m38s - - - - - - - - sarah Book.p28 1m56s - - - - - - - -

User Res Time Resp1 Resp2 Count1 Count2 sarah Lec1p1.5 1m52s incorrect - - 1 - - sarah S1V8 0m58s - - - - - - - - sarah Lec1p1.5 0m22s incorrect incorrect 2 1 sarah Book.p27 0m38s - - - - - - - - sarah Book.p28 1m56s - - - - - - - - sarah Lec1p1.5 0m22s correct - - 3 - -

User Res Time Resp1 Resp2 Count1 Count2 sarah Lec1p1.5 1m52s incorrect - - 1 - - sarah S1V8 0m58s - - - - - - - - sarah Lec1p1.5 0m22s incorrect incorrect 2 1 sarah Book.p27 0m38s - - - - - - - - sarah Book.p28 1m56s - - - - - - - - sarah Lec1p1.5 0m22s correct - - 3 - - sarah Lec1p1.5 0m11s correct - - 2


What resource was most effective? How do we model knowledge in this scenario?


Evidence representation {0 incorrect, 1 correct} Basic Model: Allowing for multiple responses in a time slice

K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 0 0 1 1


K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 - 0 1 1 1 - - - - 0 0 - 0 - 0 0 0 1 -

- 1 - 0

- - - 1

- - - -

Train parameter with EM on training data


K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 - 0 1 1 1 - - - - 0 0 - 0 - 0 0 0 1 -

- 1 - 0

- - - 1

- - - -


? - ? ? ? ? - - - -

- ?

- -

- -

Use trained parameters to iteratively predict test set sequences


K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 - 0 1 1 1 - - - - 0 0 - 0 - 0 0 0 1 -

- 1 - 0

- - - 1

- - - -


0.24 - ? ? ? ? - - - -

- ?

- -

- -

Use trained parameters to iteratively predict test set sequences

Pred Act 0.24


K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 - 0 1 1 1 - - - - 0 0 - 0 - 0 0 0 1 -

- 1 - 0

- - - 1

- - - -


0 - ? ? ? ? - - - -

- ?

- -

- -

Update inferred probability of knowledge given observation:

Pred Act 0.24 0

𝑃(𝐾|Q)= 𝑃𝑄 𝐾 𝑃(𝐾)/𝑃(𝑄) 


K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 - 0 1 1 1 - - - - 0 0 - 0 - 0 0 0 1 -

- 1 - 0

- - - 1

- - - -


0 - 0.11 ? ? ? - - - -

- ?

- -

- -

Predict next response

Pred Act 0.24 0 0.11


K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 - 0 1 1 1 - - - - 0 0 - 0 - 0 0 0 1 -

- 1 - 0

- - - 1

- - - -


0 - 1 ? ? ? - - - -

- ?

- -

- -

Repeat…

Pred Act 0.24 0 0.11 1

10/24/14 33


K

p(L0)

Kp(T)

Q1 Q2 Q1 Q2

Kp(T)

Q1 Q2

Kp(T)

Q1 Q2p(G)p(S)

0 - 0 1 1 1 - - - - 0 0 - 0 - 0 0 0 1 -

- 1 - 0

- - - 1

- - - -


Calculate accuracy metric on predictions

Pred Act 0.24 0 0 0.11 1 0 0.67 1 1 0.95 1 1 … …

Pardos, Bergner, Seaton, Pritchard (EDM, 2013)

10/24/14 34

Pred Act 0.24 0 0 0.11 1 0 0.67 1 1 0.95 1 1 … …

Pedagogical efficacy measurement

Pardos, Xu, Johnson (in preparation)

Resource type can be: 1: learning from answering 2: learning from other prob 3: video 4: wiki 5: discussion 6: tutorial 7: book

K

Q Q Q

P(L0)

P(G)P(S)

R

K

Q Q Q

P(T|R)

R

Resource model: Measure the learning value of resources

(Pardos & Heffernan, JMLR W & CP, In Press)

ensemble (¿Physisists?) collaborative filtering probabilistic graphical models

10/24/14 35

What is the appropriate level of granularity to measure knowledge?

course chapter assignment problem

Analytics Platform

Ethical Framework

•  Asilomar Education Convention

Topic [3/3] http://cs.berkeley.edu/~zp/mooc

10/24/14 38 http://cs.berkeley.edu/~zp/mooc

http://cs.berkeley.edu/~zp/mooc

http://cs.berkeley.edu/~zp/mooc

Additional analytics will display a listing of the most and least effective learning objects per problem allowing for iterative data driven improvement of course materials and learning object recommendation (Pardos & Kao, in preparation).

Ethical Framework

10/24/14 41

http://asilomar-highered.info •  Respect for the rights

and dignity of learners •  Beneficence •  Justice

•  Openness •  The humanity of learning •  Continuous consideration

Thank you collaborations desired Zachary A. Pardos Assistant Professor UC Berkeley http://cs.berkeley.edu/~zp [email protected]

10/24/14 42

From the Classroom to Research: Learning Analytics and the IRB Rebecca Armstrong, DVM, PhD Director, Research Subject Protection Office for Protection of Human Subjects UC Berkeley

1)  Why is the IRB part of this teaching with technology?

2)  What are key topics or discussion points with “your” IRB?

3)  Some possible approaches and solutions?

Why is the IRB Involved?

q  Intent of “activity”?

q  Research or teaching/curriculum improvement?

q  Or just plain good teaching with improved access?

q  Cross-linkage with other databases; privacy and confidentiality of learners; voluntary participation in research….

10/23/14 46

Scenario – 1 •  Professor offers

interesting course via MOOC

•  50,000 students sign-up

•  Commercial-based platform standard agreement of use

•  Data is collected

•  Prof notices some trends and communicates to learners with suggestions that may improve learning

Scenario – 2 •  Same set-up

•  But, Prof want to test the trends noticed to see if learning outcomes are improved

•  Interventions are planned based & data obtained during course

•  Prof does whole course analysis at end of course

•  Was this voluntary participation in research?

Scenario – 3 •  Same set-up as #1

•  Prof talks with colleague about course

•  Colleague wants access to data to do research using learning analytics…

•  Can s/he get the data?

•  Under what circumstances?

•  Who “owns” the course data? Prof or UC or platform company?

10/23/14 47

When to involve the IRB?

Before you put on your “researcher” hat!

10/23/14 48

Discussion points with your IRB •  Work through whether proposed activity is

NHSR? Exempt? Expedited or full Committee review?

•  Are waivers of elements of consent or other strategies appropriate in relation to informed consent?

•  Ask for consent to use data at the end of a course?

•  Is this incomplete disclosure?

10/23/14 49

IRB discussion points continued… •  Are data held in a repository? De-identified?

•  Privacy and confidentiality issues?

•  Is it (resulting data & collection process) minimal risk to subjects (learners)?

•  Will learning data be linked to other databases?

•  What funding supports this research activity?

•  Minors or adults? Do you know?

10/23/14 50

UCB IRB’s Guidance Documents ü  Data Security Guidelines and Matrix

ü  Exempt Research

ü  Informed Consent

ü  Internet-based Research

ü  Recruitment

ü  Secondary Analysis of Existing Data

And, more are forthcoming… http://cphs.berkeley.edu/guideline.html

10/23/14 51

Education Research Today and Tomorrow

•  Changing landscape of teaching with technology locally and “at a distance” (MOOCs)

•  UC System Policy (FERPA, C&G Manual)

•  AAAHRPP Accreditation

•  Unchecking “the box” on FWA

•  Flexibility enhanced http://cphs.berkeley.edu/faqs.html#threeyear

10/23/14 52

UC System IRB Directors

Ø Developed and refined a MOU process for IRB Review across the System

Ø UCOP supported online Registry

Ø Developing standardized expanded exempt categories for non-federally funded studies *

Ø UCB leading effort to protect subjects & facilitate research

Contemporary technology has created unprecedented opportunities to create radical improvements in learning and educational achievement, but also conditions under which information about learners is collected continuously and often invisibly. For these reasons, collection and aggregation of evidence to pursue learning research must proceed in ways that respect the privacy, dignity, and discretion of learners. http://asilomar-highered.info/

10/23/14 54

Thank you for your attention today. Becky Armstrong Director Office for Protection of Human Subjects [email protected]

10/23/14 55