the centre for research on lifelong learning and education (cele) nordic comparative and...

The Centre for Research on Lifelong Learning and Education (CELE) Nordic Comparative and International Education Society (NOCIES)

Symposium

Educarium Building University of Turku, Turku, Finland May 21-22, 2013

Comparative research and

fallacious causal attributions

Jón Torfi Jónasson, School of Education, University of Iceland [email protected]

mailto:[email protected]

Examples of massive, perhaps influential documents;who has got the time and energy to go through them systematically and

critically?

ILSA (international large-scale assessments) – PISA, TIMSS, We also have Talis,...• OECD. ( 2009). Creating Effective Teaching and Learning Environments: First Results from TALIS

Derived analysis, see e.g. from PISA 2009 (1+4 volumes)– PISA 2009 Results: Overcoming Social. Background EQUITY IN LEARNING OPPORTUNITIES AND

OUTCOMES. VOLUME II - with policy implications– PISA 2009 Results: Learning to Learn. STUDENT ENGAGEMENT, STRATEGIES AND PRACTICES. VOLUME

III - with policy implications

Descriptive studies (e.g. the background documents for the “summits”), all with policy implications

• OECD. (2013), Teachers for the 21st Century: Using Evaluation to Improve Teaching, OECD Publishing.• Schleicher, A. (2012), Ed., Preparing Teachers and Developing School Leaders for the 21st Century:

Lessons from around the World, OECD Publishing.• OECD. (2011). Building a High-Quality Teaching Profession. Lessons from around the world.

• McKinsey. (2007). “How the world’s best performing school systems come out on top”• McKinsey. (2010). “How the World’s Most Improved School Systems Keep Getting Better”

Jón Torfi Jónasson CELE NOCIES Symposium Turku 2013 Fallacious inferences 2

A recent class of comparative studies:ILSS – International large scale studies

What is the basic idea? : To compare!?

What is their presumed relevance? : If there is a difference, it calls for change by those who are behind; the culprit is normally “the system”

Various types, assessments (inviting ranking, PISA, TIMSS), surveys (Talis), interviews (McKinsey), ...

And then what is their presumed use? and what methodological design demands does this make?

A very neglected field for discussion and debate?


Overview

• Comparative education; what for?

• Research based policy discourse

• Analysis of the problem

• Formal methodological issues

• Discussion of the problem


Comparative education; what for? on what grounds? with reference to which questions?

It may be interesting to compare competence, curriculum, organization and systems, dispositions, aspirations ...

• One may want to understand what similarities there might be comparing certain aspects of education (e.g. in the drop-out patterns), despite notable system differences (or vice versa)

• One may want to learn from other systems (or cultures) about ways of– doing things, perhaps arguing on qualitative grounds– not doing things, which is common, and sometimes quite dramatic (don't

emulate us, please!)


Research based policy discourse – we must be able to see the wood rather than only the

trees

“One way we'll know we're succeeding in changing China's schools is when those PISA scores come down.”

2010 JIANG XUEQIN, deputy principal of Peking University High School, and director of its International Division. http://online.wsj.com/article/SB10001424052748703766704576008692493038646.html

Here, neither PISA as such nor China are the issue, but the relationship in general between various tests (e.g. PISA), education, schools and their function in society.

And this reminds us also of the more general question, what kind of evidence is relevant for educational decisions, and how do we use it?

Note the title of: Bridges, D., Smeyers, P., & Smith, R. (Eds.). (2009). Evidence-Based Education Policy. What Evidence? What Basis? Whose Policy? : John Wiley And Sons Ltd.


http://online.wsj.com/article/SB10001424052748703766704576008692493038646.html

Research based policy discourse (cont.)

• What issues can in principle be solved and perhaps not be solved by research? (A neglected but crucial question in an evidence based ethos.)

– The aim(s) of education, will in principle not be determined by research?

– The “best preparation for the future”, can at any given time not be determined by research even though it could perhaps in principle be answered at a later time? (E.g. research on the long-term effects of medical interventions.)



General conceptual issues. Distinguish between different classes of questions, such as (and please stop to think how different these are:)

• Type A questions: What systems, methods and content will best serve “our” aims of education? Or even, particular subsets of aims?– Then we must determine what kind of research design might be best suited to

respond to these questions; PISA as an example, might or might not be good way of doing it.

• Type B questions: How can we use existing data (e.g. ILSAs such as PISA, TIMSS, ..) to clarify the operation of our education systems?– Then we would explore what methods would be most appropriate to survey the

data in order to tease out the informative patterns; without - in most cases – being able to deduce any causal relationships



It might be that people start out with class A questions, collect data that then allows responding to a series of class B questions (and do that quite well) and then feel legitimised talking as if class A questions had been asked.

Please note in the following an attempt to convey the questions we think we should be asking and then speculate at what level our questions really are?

Then stop for a moment to think how you might map, e.g. the PISA endeavour onto the (some) general overarching aims of education.



It is suggested here that the principal validity problem is, however, not simply a validity problem, because it is very or totally unclear what the principal questions are, and thus what the constructs at issue are. Much more time should be spent on this problem than is normally done.

What are the principal questions we want to answer? And as soon we have determined these we must enter the validity discussion.

In the following we suggest a number of levels (and there are more) at which we might approach the problem. We seem to be normally at the lowest level (or lower) but the discussion is often as if we were at the highest level.


Formal methodological issues: approaching validity

• What kind of society do we want for the future and what should – or might the education system contribute to its formation? (What do we want?)

– What characteristics and knowledge do we desire from our emerging generations? What kind of metrics would be sensible to use to gauge these? (How do we measure this?)

• To which extent would we expect the important characteristics and knowledge to emerge from within our educational systems or what role would we want these systems to play? (What should the education systems do?)

– To the extent the education system is expected to serve the goal of preparing the new generations for the future work life, what kinds of skills or dispositions or cultures would be most sensible? (Education and the world of work.)


Deconstructing the aims of education and relating to, e.g. PISA


PISA

The aims of education

For the individual, skills, well being, social functioning …

For society, world of work, survival, democratic and cultural participation, …

Assessment stuies, e.g. PISA

A huge amount of data is collected; but what are the fundamentally important questions we should be asking (even) before we start analysis?

What does the variance mean? In real terms?

Examples:

a) How do we compare a high group in one system to a low group in another system? What are the system implications of that comparison

b) Why on earth does the nation state demand such an attention; what about different regions within it? (Note e.g. Canada, but the examples abound; what does it mean to compare the U.S. and China?)

c) Why are the differences within a normal class not the most interesting focus of attention?



PISA 2009

Variation of reading performance within countries

Figure II.1.1

PISA 2009 Results:

Overcoming SocialBackgroundEQUITY IN LEARNING OPPORTUNITIESAND OUTCOMES

VOLUME II


Variation in Literacy Skillsamong Canadian Provinces:Findings from the OECD PISAJ. Douglas WillmsUniversity of New BrunswickPublished by authority of the Minister responsible for Statistics Canada© Minister of Industry, 2004

Formal methodological issues: validity

• Construct validity, internal validity, (external validity)– To a large extent the validity issue centres around the definition of

the problem; what are the questions at the heart of the studies?

• Internal validity, causal inferences; design demands– I. Randomized experiments

– II. Non-randomized designs (and the problems they entail)

• Quasi experiments (static- groups, various versions of (interrupted) time-series designs)

• Correlational research (various statistical analysis; regression, path-analysis, ...

• Survey research

What methodology does allow evidence based policy borrowing?


Formal methodological issues: alternative approaches

What system or content would be most appropriate in order to

• a) achieve equality within society?• b) build a democratically competent nation?• c) form a creative population?

• Why are these not the most relevant questions; how do we design studies to address those?

• But of course existing studies might be very helpful in gauging the problem or assessing the situation.


A Nordic model – Nordic issues?

What are we talking about when discussing a Nordic model?

A common system?A common history?A common set of values?A common culture?How important is the nation state as a unit of analysis in the this context?

What are the criteria for being Nordic (a question about approach) a) being unique? No probably not? or not necessarilyb) sharing something, perhaps also with others? Yes, probablyc) sharing something valuable also with others? Yes, definitely

Consider some examples from this perspective. What is Nordic about those? Start with the general importance of equality.


Jón Torfi Jónasson CELE NOCIES Symposium Turku 2013 Fallacious inferences

International Monetary FundFinance & Development, September 2011, Vol. 48, No. 3Andrew G. Berg and Jonathan D. Ostry

“Do societies inevitably face an invidious choice between efficient production and equitable wealth and income distribution? Are social justice and social product at war with one another?In a word, no.”

“That experience brought home the fact that sustainable economic reform is possible only when its benefits are widely shared. “

20

http://www.imf.org/external/pubs/ft/fandd/2011/09/berg.htm




Percentage of variance within and between schools Figure II.5.1

Variation in reading performance between and within schools

Expressed as a percentage of the average variance in student performance in OECD countries

Variance in student performance explained by the index of economic, social and cultural status of students and schools

Fig

ure

II.5

.1

Var

iati

on

in

rea

din

g p

erfo

rman

ce b

etw

een

an

d w

ith

in s

cho

ols

Exp

ress

ed a

s a

perc

enta

ge o

f th

e va

rianc

e in

stu

dent

per

form

ance

acr

oss

OE

CD

cou

ntrie

s

100

8060

4020

020

4060

8010

0

Qat

ar

154

Bulg

aria

1

48

Trin

idad

and

Tob

ago

1

47

Isra

el

144

Arg

entin

a

135

Dub

ai (U

AE)

1

31

Luxe

mbo

urg

1

24

New

Zea

land

1

22

Belg

ium

1

20

Japa

n

116

Aus

tria

1

16

Alb

ania

1

15

Pana

ma

1

14

Uru

guay

1

14

Aus

tral

ia

113

Kyrg

yzst

an

113

Swed

en

112

Peru

1

12

Sing

apor

e

110

Uni

ted

Stat

es

108

Icel

and

1

06

Italy

1

06

Uni

ted

King

dom

1

05

Gre

ece

1

05

Irel

and

1

05

Ger

man

y

104

Braz

il

102

Switz

erla

nd

101

Mon

tene

gro

1

00

Czec

h Re

publ

ic

98

Nor

way

9

6

Kaza

khst

an

96

Slov

enia

9

5

Jord

an

95

Cana

da

94

Slov

ak R

epub

lic

94

Hun

gary

9

4

Rom

ania

9

4

Russ

ian

Fede

ratio

n

93

Pola

nd

92

Net

herl

ands

9

1

Croa

tia

89

Spai

n

88

Port

ugal

8

7

Colo

mbi

a

87

Lith

uani

a

86

Finl

and

8

6

Chin

ese

Taip

ei

86

Tuni

sia

8

4

Mex

ico

8

3

Hon

g Ko

ng-C

hina

8

1

Serb

ia

81

Den

mar

k

81

Esto

nia

8

0

Liec

hten

stei

n

80

Chile

7

9

Turk

ey

77

Shan

ghai

-Chi

na

74

Latv

ia

74

Kore

a

72

Mac

ao-C

hina

6

7

Aze

rbai

jan

6

6

Thai

land

6

0

Indo

nesi

a

51

OEC

D av

erag

e65

%

OEC

D av

erag

e 42

%

Vari

ation

wit

hin

scho

ols

Vari

ation

bet

wee

n sc

hool

s

Tota

l var

ianc

eas

a pr

opor

tion

of th

e O

ECD

varia

nce

21


Variation in reading performance explained by students' and schools' socio-economic background

Figure II.5.4

Expressed as a percentage of the average variance in student performance in OECD countries

Variation in reading performance explained by students' and schools' socio-economic background

Exp

ress

ed a

s a

perc

enta

ge o

f th

e av

erag

e va

rianc

e in

stu

dent

per

form

ance

in O

EC

D c

ount

ries

Not

e:

Cou

ntrie

s ar

e ra

nked

in a

scen

ding

ord

er o

f th

e pe

rcen

tage

of

over

all v

aria

nce

in r

eadi

ng p

erfo

rman

ce e

xpla

ined

by

the

PIS

A

inde

x of

eco

nom

ic,

soci

al a

nd c

ultu

ral s

tatu

s of

stu

dent

s an

d sc

hool

s.

Sou

rce

: O

EC

D P

ISA

200

9 da

taba

se,

Tab

le I

I.5.

2.

Var

iati

on

in

rea

din

g p

erfo

rman

ce e

xpla

ined

by

stu

den

ts' a

nd

sch

oo

ls' s

oci

o-e

con

om

ic b

ackg

rou

nd

Fig

ure

II.5

.4

010

2030

4050

6070

8090

100

Luxe

mbo

urg

Uni

ted

King

dom

New

Zea

land

Uni

ted

Stat

esSw

eden

Den

mar

kU

rugu

ayCo

lom

bia

Pola

ndPe

ruA

ustr

alia

Mon

tene

gro

Turk

eyLi

echt

enst

ein

Chile

Shan

ghai

-Chi

naBe

lgiu

mBu

lgar

iaCz

ech

Repu

blic

Ger

man

ySi

ngap

ore

Hun

gary

Port

ugal

Irel

and

Trin

idad

and

Tob

ago

Arg

entin

aSl

ovak

Rep

ublic

OEC

D a

vera

geA

lban

iaKo

rea

Chin

ese

Taip

eiSp

ain

Kyrg

yzst

anLa

tvia

Braz

ilSw

itzer

land

Aus

tria

Japa

nLi

thua

nia

Serb

iaCr

oatia

Pana

ma

Isra

elCa

nada

Esto

nia

Net

herl

ands

Russ

ian

Fede

ratio

nIta

lySl

oven

iaG

reec

eKa

zakh

stan

Rom

ania

Dub

ai (U

AE)

Mex

ico

Mac

ao-C

hina

Nor

way

Finl

and

Icel

and

Jord

anTh

aila

ndIn

done

sia

Hon

g Ko

ng-C

hina

Qat

arTu

nisi

aA

zerb

aija

n

Perc

enta

ge o

f var

ianc

e in

read

ing

perf

orm

ance

exp

lain

ed b

y th

e PI

SA in

dex

of e

cono

mic

, soc

ial

and

cultu

ral s

tatu

s of

stu

dent

s an

d sc

hool

s

Varia

tion

in p

erfo

rman

ce e

xpla

ined

by s

tude

nts'

socio

-eco

nom

icba

ckgr

ound

w

ithin

scho

ols

Varia

tion

in p

erfo

rman

ce e

xpla

ined

by

scho

ols'

socio

-eco

nom

icba

ckgr

ound

be

twee

n sc

hool

s

22

300

350

400

450

500

550

Finl

and

Hong

Kon

g-Ch

ina

Sing

apor

e

Cana

da

New

Zeal

and

Aust

ralia

Net

herla

nds

Belg

ium

Nor

way

Esto

nia

Switz

erla

nd

Uni

ted

Stat

es

Liech

tens

tein

Swed

en

Ger

man

y

Irela

nd

Fran

ce

Denm

ark

Uni

ted

King

dom

Hung

ary

OEC

D av

erag

e

Port

ugal

Mac

ao-C

hina

Italy

Slov

enia

Gre

ece

Spai

n

Czec

h Re

publ

ic

Croa

tia

Isra

el

Luxe

mbo

urg

Aust

ria

Duba

i (U

AE)

Russ

ian

Fede

ratio

n

Serb

ia

Mex

ico

Trin

idad

and

Toba

go

Braz

il

Mon

tene

gro

Jord

an

Arge

ntina

Kaza

khst

an

Qat

ar

Pana

ma

Azer

baija

n

Kyrg

yzst

an

All students Students without an immigrant background Second-generation students First-generation students

Mean score


Reading performance, by immigrant status

23

Formal methodological issues: meta-analysis

• The lessons of meta-analysis; show substantial variation between studies. The problem of relying on single studies, single methodologies, and very homogeneous criteria is probably more serious than is often appreciated.


Two references

Critical discussion of the McKinsey reports:Coffield , Frank (2012). Why the McKinsey reports will not improve school systems. Journal of Education Policy, 27(1), 131-149

Critical discussion of the political use of ILSAsEngel, Laura, Williams, James, Feuer, Michael. (April 2012).The Global Context of Practice and Preaching: Do High-Scoring Countries Practice What U.S. Discourse Preaches? School of Education and Human Development, George Washington University. Working paper 2.3


Conclusion

A firm conceptual and technical methodology for relating comparative studies to any form of policy action must be (re-) established. There is a serious lack of rigour, not less at the conceptual level, than the technical level (noting that the problem has little to do with statistics which is may be carried out at a sophisticated level – but that is another debate).

The most serious problems are those related to validity of the studies, vis-à-vis the questions they are in fact intended to answer and what inferences, causal or otherwise can be drawn related to those questions. This relates to all aspects of validity, not just to internal validity.

Therefore, let us briefly return to the meta-question, what are the questions we are seeking answers to?


Formal methodological issues: approaching validity

• What kind of society do we want for the future and what should – or might the education system contribute to its formation? (What do we want?)

– What characteristics and knowledge do we desire from our emerging generations? What kind of metrics would be sensible to use to gauge these? (How do we measure this?)

• To which extent would we expect the important characteristics and knowledge to emerge from within our educational systems or what role would we want these systems to play? (What should the education systems do?)

– To the extent the education system is expected to serve the goal of preparing the new generations for the future work life, what kinds of skills or dispositions or cultures would be most sensible? (Education and the world of work.)


Thank you


the centre for research on lifelong learning and education (cele) nordic comparative and...

Documents

school of education

fallacious inferences

comparative research

policy implications

policy implications

overview comparative

oecd publishing

policy discourse analysis