assessment embedded in step- based tutors (sbts) cpi 494 feb 12, 2009 kurt vanlehn asu
TRANSCRIPT
Assessment embedded in step-based tutors (SBTs)
CPI 494 Feb 12, 2009
Kurt VanLehn
ASU
Outline
Benefits of assessment embedded in SBT Synthesis of several common ITS assessment
methods– Terminology– Where does assessment fit into an ITS?– A synthesis
Why should step-based tutors (SBT) be better than multiple choice tests (MCT) for assessment?
Why is a 30 minute task (SBT) better than a 1 minute task (MCT)?– Authenticity, validity, prediction of valued tasks– “teaching to the test” becomes valuable
Why is seeing the steps better than seeing just the answer?– Assess problem solving strategy– Assess meta-cognitive strategy (e.g., use of hints)
Why is embedding assessment in daily homework or seatwork better than monthly in-class testing?
Frees up time for teaching Can’t be absent on day of test “doesn’t test well” and “only good at tests” are
no longer excuses No test anxiety No cramming Instructors can see changes in competence daily
Homework is unsupervised but testing is supervised. How does that affect assessment?
What if students refer to textbook, web, friends or instructor?– Sequestered problem solving is inauthentic– Assessment may improve if tutor can “see” such
references What about cheating?
– Technology can thwart most copying– Would have to hire a friend to do almost all of one’s
homework
Outline
Benefits of embedding assessments in SBTs Synthesis of several common ITS assessment
methods– Terminology– Where does assessment fit into an ITS?– A synthesis
Next
Definition: Knowledge component
A piece of knowledge Generic – can be applied in to multiple tasks Any form: rule, principle, concept, fact,
procedure… Examples:
– Two angles are called “supplementary” if….– To buckle your seat belt, insert the …– The gravitational constant of earth, g, is 9.8 m/s2.
Definition: Mastery of a knowledge component (KC)
No more training needed– E.g., The ITS need no longer pick problems that
involve it Typical signs of mastery
– Applies the KC when should– Applies the KC only when should– Applies without referring to text, notes, …– Can explain the KC, when to apply, etc.
Main goal of assessment
Never have enough data – E.g., typically do not have student’s explanations
Must estimate P(mastery) for every knowledge componentgiven only steps as data
Definition: The response pattern of a student’s step
How a step was accomplished by the student E.g., 4 response patterns:
(S = student; T = tutoring system)
1. S: Correct.
2. S: Error; T: Hint; S: Correct.
3. S: Help; T: Hint; S: Correct.
4. S: Help; T: Hint; S: Help; T: Hint; S: Help; T: Bottom out hint; S: Correct
Definition: The relevant knowledge components of a student’s step
The knowledge components involved in deriving the correct step– given the information available at the time the step
was done The knowledge the student should apply to
generate this step The topic of the hints (if any were given)
Outline
An introduction to a common type of ITS– Terminology– Three examples
Synthesis of several common ITS assessment methods– Terminology (KC, mastery, response pattern, KCs
relevant to a step)– Where does assessment fit into an ITS?– A synthesis
Next
The 3 major computations of intelligent tutoring systems
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response pattern for each student step
Expert
Assessor
Helper
The expert’s computation Expert can be humans
or an expert system Solve the problem
in all acceptable ways Record steps taken Record knowledge
components used at each step
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response patterns for each student step
Expert
Assessor
Helper
The helper’s computation
When the student enters a step, match it to a correct step
Give feedback & hints as necessary
Record response pattern
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response patterns for each student step
Expert
Assessor
Helper
The assessor’s computation Given
– Reponse patterns for each step taken by the student
– Old P(mastery) for each knowledge component
Calculate– New P(mastery)
P(mastery) for each knowledge component
Problem to be solved
All correct steps in all orders
Response patterns for each student step
Expert
Assessor
Helper
Outline
An introduction to a common type of ITS– Terminology– Three examples
Synthesis of several common ITS assessment methods– Terminology– Where does assessment fit into an ITS?– A synthesis Next
Some assessment methodsKnowledge tracing (Corbett & Anderson)-- Only two response patterns: “S:Correct” & “other”-- Only one relevant knowledge component per step
SMART (Shute)-- Multiple response patterns
Andes (VanLehn et al.)-- Multiple knowledge components per step
Today: A synthesis of the above
DT tutor (Murray)-- Dynamic Bayesian networks
The problem: Given step responses & prior P(mastery), calculate posterior P(mastery)
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1
S: error; T: Hint;
S: Correct
Step 2
S: Correct
Step 3
S: Help;T: Hint;S: Help;T: Hint;S: Help;
T: B.O. Hint;S: Correct
.05
.05
.11
.10
.05
.75
.12
.50
.99
.05
Prior P(mastery) for each knowledge component (KC)
Posterior P(mastery) for each knowledge component
A dynamic Bayesian network represents that P(mastery) changes during steps
Bayesiannetwork
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Bayesian network
Step 1 Step 2
P(mastery) of knowledge component 4 at time 1
P(mastery) of knowledge component 4 at time 2
Assume P(mastery) changes only for relevant knowledge components
Bayesian network
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4Bayesian network
Step 1Involves only KC2 and KC3
Step 2Involves only KC1 and KC4
Knowledge tracing: Assumes just one relevant knowledge component per step
Bayesian network
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4 Bayesian network
Step 1Involves only
KC2
Step 2Involves only
KC4
Knowledge tracing: Network topology and conditional probability tables
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Step responseKC2 mastered: Yes No
S: Correct .92 .10
Other .08 .90
Was mastered: Yes No
Mastered 1 .47
Not mastered 0 .53
P(Step response not correct | KC2 mastered)
Called “slip parameter”
P(Step response correct | KC2 not mastered)
Called “guess parameter”
P(Mastered | Was not mastered, relevant to step) Called “acquisition parameter”
Need to allow multiple relevant knowledge components per step
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Use AND node to represent that the step response depends only on
whether or not all relevant knowledge components are mastered
The student’s meta-cognitive strategy also affects the student’s step response
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Meta-cognitive strategy
Conditional probability table for the first “Step response” node
Meta-cognitive strategy Learn Perform
Both KC2 and KC3 mastered? Yes No Yes No
S: Correct .99 .50 .99 .10
S: Error; T: Hint; S: Correct .01 .35 .00 .01
S: Help;… T: Bottom out hint; S: Correct .00 .01 .01 .85
All other response patterns .00 .14 .00 .04
P(step response was “S:Correct” | Meta-cognitive strategy was “Learn”,
Not all knowledge components mastered before step
Strong tendency to abuse help when the meta-cognitive strategy is “Perform” and some relevant knowledge component is not mastered yet.
Need to assume new P(mastery) depends on the step response as well as the old P(mastery)
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 1 Step 2
Step response
Meta-cognitive strategy
Conditional probability table for a “knowledge component” node
Step response: S:Correct 1 hint Bottom out hint Other
Mastered before? Yes No Yes No Yes No Yes No
Mastered after step 1.0 .50 .95 .30 .90 .01 .55 .05
Not mastered after step 0.0 .50 .05 .70 .10 .99 .45 .95
How the network should be used
Given log file – all steps taken by the student Construct the whole network – all steps in all problems Enter prior P(mastery) on knowledge components at
time 1 Clamp all step responses Update – a huge computation Read out P(mastery) on knowledge components after
last step.
How the network is actually used Step 1
– Construct network– Enter prior P(mastery)– Clamp step 1 response– Update– Read out P(mastery)
Step 2– Construct network– Enter prior P(mastery)– Clamp step 2 response– Update– Etc.
KC1
KC3
KCn
KC2
KC4
Step response
KC1
KC3
KCn
KC2
KC4
Step 1
Meta-cognitive strategy
KC1
KC3
KCn
KC2
KC4
KC1
KC3
KCn
KC2
KC4
Step 2
Step response
Meta-cognitive strategy
Future workKnowledge
tracingSMART Andes Today
Design Implementation Calibration Validity Sensitivity
Summary
An introduction to a common type of ITS– Steps replace answers as main student response
A synthesis of several Bayesian assessment methods– Many knowledge components– Probability of mastery of each knowledge component– One (or more) knowledge components per step– Can use step response patterns to infer mastery– Global traits e.g., meta-cognitive strategy