problematising assessment

Problematising Assessment (as if it needed it)

James Atherton11 March 13

Problematising Assessment (as if it needed it)

James Atherton11 March 13

Balloons in this colour are

additional notes for the online version

3.3: Understand theories, principles and applications of formal and informal assessment

This is the outcome to which the

session relates

3.3: Understand theories, principles and applications of formal and informal assessment

And if I were teaching Ofsted style I should

now recite the objectives...

Confused

And for once I will. At the end of this

session you should be–

Confused

...but at a higher level than before

Confused

...but at a higher level than before

Probably from Kelley, 1951, but attributed to various sources

It is frowned upon for you to

confuse your students.

Confused

...but at a higher level than before

Probably from Kelley, 1951, but attributed to various sources

It is frowned upon for you to

confuse your students.

...which may well be the biggest

limitation on your teaching.

Confusion can be constructive in teaching—

like ploughing before planting

The Problem of Proxies

1: The Problem of Proxies

...or surrogates, or substitutes, or

stand-ins for the real thing

1: The Problem of Proxies

...or surrogates, or substitutes, or

stand-ins for the real thing

Assessment is rife with them, and diluted by their use—but we are

stuck with them

This is the essence of intuitive heuristics:

when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution

Kahneman 2011: 12

Thinking Fast and Slow, Penguin

This is the essence of intuitive heuristics:

when faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution

Kahneman 2011: 12

And this is exactly what we do in assessment

Content Assess-ment

Content Assess-ment

In principle our teaching is governed by content, and the

assessment is just to check that it has been learned

Content Assess-ment

Content Assess-ment

In practice, the demands of the assessment can all

to easily take over

Content Assess-ment

In practice, the demands of the assessment can all

too easily take over

“Will we be tested on this?”

AspectsForms

Purposes

AspectsForms

Purposes

Here are some traditional

perspectives on assessment...

• Diagnosis

• Feedback

• Standards

AspectsForms

Purposes

• Diagnosis

• Feedback

• Standards

AspectsForms

Purposes

Pre-teaching

• Diagnosis

• Feedback

• Standards

AspectsForms

Purposes

During teaching

• Diagnosis

• Feedback

• Standards

AspectsForms

Purposes

After teaching

• Validity

• Reliability

• Fairness

• Security

AspectsForms

Purposes

• Validity

• Reliability

• Fairness

• Security

AspectsForms

Purposes

Traditional criteria for evaluating assessment

• Criterion-referenced

• Norm-referenced

• Ipsative

AspectsForms

Purposes

• Criterion-referenced

• Norm-referenced

• Ipsative

AspectsForms

Purposes

Judging against fixed pre-specified criteria

• Criterion-referenced

• Norm-referenced

• Ipsative

AspectsForms

Purposes

Judging against other people’s performance

• Criterion-referenced

• Norm-referenced

• Ipsative

AspectsForms

Purposes

Judging against your own prior

performance: personal best

• Formative

• Summative

AspectsForms

Purposes

...etc. I could now test you on your

knowledge of assessment, but

• 12 items of jargon

See what I’ve done? I’ve reduced the whole topic to

Validity

• Does it do what it says on the tin?

• Is it really assessing the outcome?

What the area of practice

actually requires

Let’s look at the whole process of assessment drift.

What the area of practice

actually requires

Let’s look at the whole process of assessment drift.

Based on the work of Howard Becker and

Etienne Wenger, among others

What the course sets out to teach

What the course sets out to teach

There’s about 80% overlap—never a

perfect fit

What the course actually does

teach

What the course sets out to assess

What the course actually does

assess

What the course actually does

assess

What the area of practice

actually requires

That’s all the overlap left

What the course actually does

assess

What the area of practice

actually requires

And if you don’t pass very well...

2: False positives and false negatives:

the inherent limitations of testing

2: False positives and false negatives:

the inherent limitations of testing

I got into some trouble in this section!The maths are correct, but the problem comes with the labelling of the False Positives (or Type 1 errors) and what happens if you try to

eliminate them simply by making the assessment stricter (rather than by targeting it more precisely), so to avoid unnecessary extra

confusion, I’ve taken that out of this version.

Take a hundred people and train them for something....

In the real world, 80% are competent at it, and 20% aren’t

Competent (80%)Not

com

pete

nt

(20%

)

In the real world, 80% are competent at it, and 20% aren’t

But we’re not in the real world—we’re in a college—and we have to devise a test to determine who can be let loose on the public

Accurate (80%)

Inaccurate (20%)

... but tests aren’t always good predictors. You devise the best you can, but it may be only, say, 80% accurate.

Competent (80%)

Accurate (80%)

Inaccurate (20%)N

ot c

ompe

tent

(2

0%)

So the 80% the test passes are not the same as the 80% who are genuinely competent

True + (64%)True –16%

False – (16%)False +(4%)

True + (64%)True –16%

False – (16%)False +(4%)

These are the “true positives”—they passed

the test, and so they should have

True + (64%)True –16%

False – (16%)False +(4%)

These are the true negatives: they

failed and so they should have done.

True + (64%)True –16%

False – (16%)False +(4%)

These are the unfortunates: the test failed them, but it was wrong. That is technically a ‘Type 2’ error.

True + (64%)True –16%

False – (16%)False +(4%)

These are the ‘Type 1’ errors: they should have failed, but the test passed them.

This test will always be 20% wrong. So you can only reduce the False

Positives at the cost of increasing the False Negatives.

See the notes for more on this.

So I hope you are now confused at a higher level

than before...

• Becker H (1963) “Why school is a lousy place to learn anything in” reprinted in R J Burgess (ed.) Howard Becker on Education Buckingham; Open University Press, 1998

• Kahneman D (2011) Thinking, fast and slow London; Penguin

• Kay J (2011) Obliquity; why our goals are best achieved indirectly London; Profile Books

• Wenger E (1998) Communities of Practice; learning, meaning and identity Cambridge; C.U.P.

www.bedspce.org.uk/cbc