Stereotypes, student models and Scrutability

Judy Kay

Basser Department of Computer ScienceMadsen Building, F09

University of Sydney NSW 2006AUSTRALIA

email judy@cs.usyd.edu.au

Overview

• Stereotypes– Pervasive– Natural– um community but not ITS/AIED

• Scrutability and student models• Stereotypes and scrutability

Example: Stereotype for ITS’2000 attendee Friday 9am

• Expert in education

• Expert in AI

• Stayed up too late last night

• Saturated with great new insights in ITS

• Happy with `intelligent’ in ITS

• Unaware of Fiji coup

More……..

• Have not read my paper in the proceedings

• Very interested in student modelling

• Had not heard term scrutability before this week

• Unconvinced of merit of scrutability of student/learner models

• Unconvinced of merit of stereotypes

Classic stereotypes

• Beginner vs advanced• Default is beginner• Stereotypes about beginners• A little initial information about the

student allows inference of a rich initial default student model

• May well be very rough (cf nothing)• ?

Rich:A stereotype represents a collection of attributes that often co-occur in people. ...they enable the system to make a large number of plausible inferences on the basis of a substantially small number of observations. These inferences must however, be treated as defaults which can be overridden by specific observations.

Example:athletic person

motivated by excitement (0.7)strength and perseverance (0.8)interested in sports (0.63)

Elements:• Initial interaction• Activate stereotypes• DAG of stereotypes• On-going refinement

Double stereotypes(Chin, 1989)

• Reason from user actions– Observed actions– Assume actions known– Infer expertise classification

• Reason from expertise classification– Infer huge number of actions known

Tightening the Stereotype?

• Triggers• Active stereotype v inactive stereotype• Retraction condition• Essential triggers• High fanout of inferences

{trigger function activate stereotype}

{retraction function deactivate stereotype}

{failed essential trigger deactivate stereotype}

active stereotype Many inferences

Statistical Character

• Validity is statistical (ie not individual)• Threshold probability for each inference

Can be based upon empirical evidence:Active (expert) Knows (reg exp) 0.87

What is not a Stereotype?

B prerequisite of A

Know loops (0.6)

Knows (loops) Knows (variables) (0.99)

Eg.

Knows (A)

Stereotype and classic student modelling

• Overlays– ‘expert’

• Differential models– ‘plausibly ideal student’

• Buggy models– ‘classic errors’

Stereotype for ITS’2000 attendee Friday 9.22am

• Expert in education

• Expert in AI

• Stayed up too late last night

• Saturated with great new insights in ITS

• Happy with `intelligent’ in title ITS

More……..

• Very interested in student modelling

• Had not heard term scrutability before this week

• Unconvinced of merit of scrutability of student/learner models

• Unconvinced of merit of stereotypes

Building Stereotypes

• Handcrafted stereotypes– Eg beginner, advanced– Local stereotypes

• Empirically-based stereotypes– Machine learning– Statistical analysis– Cliques, communities

Scrutability

Jonathan uses a text editorHe does not know about its undo.A coach tells him about it, why it is useful and how to use it.Jonathan tries it.He likes it.

A Scenario

What Jonathan might like to know?

How did coach know that I didn’t know about undo?What else does coach think I know?Or don’t know?Why did coach tell me about undo?How can I tell coach what I want to know?Why did coach explain undo that way?Does coach explain it differently to other people?

Is this the sort of inquisitive student you want?

Should we want students to question, explore, …?

sam and the Basser Study (Kay and Thomas, CACM 1995)

• Monitor data as basis for building scrutable student models

• Data on growth of expertise

• 10 years, ~600 new users per year initially

• Field trial of users in their 4th semester

basics

minimal

go_k

scroll 2

scroll_1_3

no_scroll_b

basics

minimal

go_k

scroll 2

scroll_1_3

no_scroll_b

Go_kShow evidenceExplainSet value to: TrueSet value to: FalseSet value to: Maybe

Explanation of go_k

Clicking anywhere in the current window moves the cursorTo that place in that window. Start typing and characters Appear right of the cursor.

Clicking on a window other the current window makes that new window (and that file/buffer) current.

Explanation of go_k

Once you have selected a window with one click on Button 1, the text you type goes where the cursor is.

To move the cursor in the current window, click button 1Where you want to type.

For example, if you type `hollo world’ when you meantto type `hello world’, you click (with button 1) at the pointbetween the `o’ and the `l’ in `hollo’, type the backspacekey and an `e’, then click again after `there’ to continuetyping where you left off.

Scrutability

• Ontological

• Values of student model components

• Bases

• Reasoning mechanisms

• Big student models

Stereotypes and Scrutability

• Am I a beginner?• What are the implications of being a beginner?• What would be different if I were an expert?• How can I let the system model me as a

beginner, but have it recognise some of the more advanced things I know?

Buggy Stereotype as Learning Objects

Scenario OneBeginner Python programmer tries to write a program.Student hits a problem.ITS diagnoses the difficulty.

Scenario TwoBeginner Python programmer tries to write a program. Student hits a problem.Student consults list of stereotypic errors.(For beginners in this task, at this stage).

Stereotype Companions

eg Hietala Niemirepo 1998

• Young children learning mathematics• Several artificial learning companions• Able (stereotype?)• Weak (stereotype?)

Conclusions on scrutability of student models

• Reflection

• Learner control and responsibility

• Metacognition

• `Correctness’ of model

• Humans may be inscrutable to machine but

• Machines and people are different

• Vive la difference!

Conclusions on stereotypes and scrutability

• Triggers (essential and other)

• Large fanout of inferences – Learner control of probability allowed– Tuning selected inferences

• Activation and retraction– User control

Final conclusions

• ITS’2000 attendees at 9.50• Feel they knew scrutability all their lives• Believe scrutable student models may be ok• Feel familiar with the notion of stereotypes

in student models• Are likely to think about scrutable

stereotypes in their own systems• Are very anxious for morning tea

Thank you!

Over to youfor questions