1

Research and DevelopmentFrom CEM

CEM conference: Improving Pupil Assessment

London 7th June 2011

2

CEM’s Achievements

The largest educational research unit in a UK university (70 staff)

1.1 million assessments are taken each year

More than 50% of UK secondary schools use one or more CEM system

CEM systems used in over 40 countries Largest provider of computerised adaptive

tests outside US

3

Changes in proportion gaining five A*-Cs at GCSE122

.6

22.9

23.5

23.7

23.7

24.0

25.0

26.1

26.2

26.7

26.9

26.7

26.4 29

.9 32.8 34

.5 36.8 38

.3 41.2 43

.3

43.5

44.5

45.1

46.3 47

.9 49.2

50.0 51

.6 52.9

53.7 56

.3 58.5 60

.9

64.8

0

10

20

30

40

50

60

70

197

519

76

197

719

78

197

919

80

198

119

82

198

319

84

198

519

86

198

719

88

198

919

90

199

119

92

199

319

94

199

519

96

199

719

98

199

920

00

200

120

02

200

320

04

200

520

06

200

720

08

Pe

rcen

tag

e o

f '1

5 ye

ar o

lds'

2

Rising standards

4

Grade slippage at A level

Average grade achieved by students with the same ability (ITDA=50)

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

A le

vel g

rade

BiologyEnglish (Lit)FrenchGeographyHistoryMathsWeighted avg of 40 subjs

U

E

D

C

B

-----

------

Cu

rricu

lum

200

0 an

d ne

w TD

A -

------

------

------

---

5

Performance of England in international surveys

Maths (age 10, TIMSS)

Maths (age 14, TIMSS)

Reading (age 11, PIRLS)

Science (age 10, TIMSS)

Science (age 14, TIMSS)

Reading literacy (age 15, PISA)

Mathematical literacy (age 15, PISA)

Scientific literacy (age 15, PISA)

480

490

500

510

520

530

540

550

560

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

Year

Sta

nd

ard

ise

d t

es

t s

co

re

6

Maths performance from 1976-2008

Hodgen et al, 2009

7

Can existing research help you to improve your school?

8

School Effectiveness Research (= Lists of characteristics of ‘effective’ schools)o Can we really identify effective schools?o Can we change these characteristics?o Will this lead to real improvement?

School Improvement Research (= Descriptions of change efforts)o Did important outcomes really improve?o Do we know why?o Are the changes (and any real effects) replicable?

Existing research

9

Set high expectations Are good at planning Employ a variety of teaching strategies Have a clear strategy for pupil

management Manage time and resources wisely Employ a range of assessment methods Set appropriate homework Keep pupils on task

Effective Teachers (according to Hay McBer, 2000)

(For which the DfEE paid £3m)

10

The secret of success(according to J. Paul Getty)

Rise early Work late Strike oil

(This advice was given for free)

11

Wait for a bad year and/or choose a bad school to start with. Things can only get better.

Take on any initiative, and ask everyone who put effort into it whether they feel it worked. No-one wants to feel foolish.

Define ‘improvement’ in terms of perceptions and ratings of teachers. DO NOT conduct any proper assessments – they may disappoint.

Only study schools that recognise a problem and are prepared to take on an initiative. They’ll improve whatever you do.

How to produce ‘school improvement’ (1)

12

Conduct some kind of evaluation, but don’t let the design be too good – poor quality evaluations are much more likely to show positive results

If any improvement occurs in any aspect of performance, focus attention on that rather than on any areas (or schools) that have not improved or got worse

Put some effort into marketing and presentation of the school. Once you start to recruit better students, things will improve.

How to produce ‘school improvement’ (2)

13

Use high-quality assessment and monitoring systems to track a range of valued outcomes

Take account of the best available research evidence about the effectiveness of different approaches

Experiment and adapt to local needs, contexts and capacities, with rigorous evaluation.

What would an evidence-driven school do?

14

15

‘Best buy’ strategies

Cost per pupil

Effe

ct S

ize

(m

on

ths

ga

in)

£00

10

£1000

Feedback

Meta-cognitive

Peer tutoring Pre-school

1-1 tutoringHomework

ICT

AfLParental

involvementSports

Summer schools

After school

Individualised learning

Learning styles

ArtsPerformance

payTeaching assistants

Smaller classes

Ability grouping

16

Probably worth investing ino Feedbacko Meta-cognitive & self-regulation strategieso Peer tutoring

Less effective (or not good value)o More teachers / TAs (smaller classes)o 1-1 support (ECaR, ECC)o Setting / streamingo PRP

If our aim is to promote learning

17

CEM’s Aim

To help educators improve educational outcomes, througho Assessments that support learningo Monitoring and feedback systems for

self-evaluationo Rigorous evaluation of the impact of

different approacheso Promotion of evidence-based practices

and policies

18

Use high-quality assessment and monitoring systems to track a range of valued outcomes

Take account of the best available research evidence about the effectiveness of different approaches

Experiment and adapt to local needs, contexts and capacities, with rigorous evaluation.

What would an evidence-driven school do?