22 april 2009 benefits and limitations of international comparative research in science and...
TRANSCRIPT
22 April 2009
Benefits and limitations of international
comparative research in science and
mathematics education
AMSTEL – SeminarPauline Vos
Universiteit van Amsterdam
AMSTEL Instituut
2
International comparison Use identical instruments in different countries and look at differences between countries.
My own international experience: Went to school in NL, D and F worked in NL, Zimbabwe, Mozambique &
South Africa Worked at the Dutch research centre for
TIMSS-1999 (University of Twente) PhD on mathematics curriculum in NL using
TIMSS-instruments
3
Many variations
Bilateral studies
Affect in science: ROSE (Relevance of Science Education), 40 countries
Teacher competencies: TEDS(M), 17 countries
4
Two abbreviations…PISA = Project for International Student Assessment
organised by OECD(Organisation for Economic Cooperation and Development)
TIMSS = Trends in International Maths and Science Studypreviously: Third Internat. Maths & Science Study
organised by IEA (Internat. Assoc. for the Evaluation of Educational Achievement)
5
Translate and define
Mathematics Basic calculations and further
Science physics, chemistry, biology, physical
geography, astronomy, ecology, etc
Students all students (not just pre-university)
6
NL USA Fla Jap Maroc Russ.
38 49 38 47 16 37
7
NL USA Fla Jap Maroc Russ.
45 44 46 31 16 38
8
NL USA Fla Jap Maroc Russ.
82 76 76 77 33 74
9
NL USA Fla Jap Maroc Russ.
44 36 66 80 31 55
10
NL 36
USA 19
Fla 21
Jap 44
Maroc
2
Russ. 9
11
NL USA Fla Jap Maroc Russ.
81 74 71 78 45 76
12
Some history…1964 A comparison of 12 countries, Husén & Bloom1971 FISS1980-82 SIMS 1984 SISS
1995 TIMSS (later known as “TIMSS-1995” )
1999 TIMSS-Repeat (later known as “TIMSS-1999”)
2003 TIMSS-2003
2007 TIMSS-20072008 TIMSS Advanced
TIMSS-2011
2000 PISA-2000
2003 PISA-2003
2006 PISA-2006
PISA-2009
22 April 2009
To what extent do countries differ in the education of science and mathematics?
How can you do research on the question:
14
Hours of instruction for science (grd 4)
15
Teacher characteristics math (grd 8)
22 April 2009
Both studies focus on a test
17
Complex studies ..Assessment frameworkLarge number of itemsLong design process for instrumentsLarge random sampleQuality control trend items (retained) coding and re-coding excluding countries when criteria were
not met
18
Pisa framework for science
19
Pisa framework for science competencies
20
Pisa framework for science
21
Pisa framework for science
22
Pisa science item ‘Acid Rain’
23
Pisa science item ‘Acid Rain’
24
Pisa science item ‘Acid Rain’
25
Complex studies… (1)Not just measuring country scores in one discipline, but also: Sub-disciplines (algebra, ecology, ..) Sub-competencies
(reproduce, explain, communicate, ..) Background variables
Not just measuring students’ scores, but also trying to explain: Differences between countries Differences between schools Differences between students
26
Complex studies… (2)PISA: 85 exercises distributed over 13 test booklets
PISA-2000 Main topic: Literacy (reading) Additional: science, maths
PISA-2003: Main topic: Mathematical Literacy Additional: literacy, science, problem solving
PISA-2006 Main topic: Science Additional: literacy, maths
27
Complex studies… (3)TIMSS: 300 exercises distributed over 12 booklets
TIMSS-95: grades 3, 4, grades 7, 8 Performance Assessment (= test in a lab-like environment) grade 12: math/science literacy; advanced maths/science video study (USA, Jap, Germ.)
TIMSS-99: grade 8 TIMSS-R Video study (USA, Jap, NL, Cz, Switz, Austr, HongK.)
TIMSS-2003 grade 4, grade 8
TIMSS-2007 grade 4, grade 8
TIMSS-2008 grade 12
28
Percentage Distribution Math Items PISA - TIMSS
15-year-olds
PISAGrade 8
TIMSSNumber 22% 33%
Measurement 18% 14%
Geometry 12% 18%
Data 40% 11%Algebra 11% 23%Classified to multiple strands 2% 0%
29
Math Item Formats
Multiple Choice
Short Constructed
Response
Extended Constructed
Response
TIMSS 71% 16% 13%
PISA* 33% 42% 25%
*PISA definitions:Multiple Choice includes regular and “complex” multiple-choice items.Short Constructed Response includes “short answer” and “closed response” items.Extended Constructed Response includes “open response” items.
30
Mathematical Complexity
LowModerat
eHigh
TIMSS 51% 46% 3%
PISA 29% 64% 7%
31
PISA-2006 science country ranking
32
PISA-2006 science country ranking
33
TIMSS-2007country ranking
Maths grade 4
34
TIMSS-2007 multi-country comparison
35
Horse race - thinkingboth in PISA and TIMSS
Ranking is simplisticSample: intrinsically inaccurateScores of some countries are very close. With s.e.>3 are small differences not significantNL position in 2003: math: on shared positions 2-7 in PISA
and shared positions 6-9 in TIMSS science: on shared positions 4-8 in PISA
and shared positions 7-11 in TIMSS
36
Differences PISA and TIMSSPopulation:
PISA: age cohort: 15-year olds (indep. of grade) TIMSS: grade (indep. of age)
Countries: PISA: economically developed (OECD) TIMSS: whole world (incl. Saudi Arabia, Botswana)
Cyclus: PISA: 3-yearly with different thematic foci TIMSS: 4-yearly with different populations
Content: PISA: applicable knowledge (situations): reading, math.
literacy, scientific literacy TIMSS: school knowledge (US curriculum): mathematics,
scienceJustification of the study:
PISA: knowledge has economical importance TIMSS: knowledge about education is needed
37
Country score and GDP
38
39
TIMSS-1999 Video StudyGrade 8seven countries: Australia, Hong Kong, Japan, NL, Czech Rep, USA, SwitzerlandAim: describe patterns in lessonsquestion: what is typical for the math lessons in a certain country?
in each country 85 random lesson filmed, transcribed & coded
Points of analysis:•Structure of the lesson: e.g. who speaks? How many exercises?•Mathematical content: complexity, procedures•Char of exercise: symbols, contexts
40
TIMSS-Video study: use of materials:Percentage lessons in which a calculator is used
41
TIMSS-Video study: structure of lessonsPercentage lessons in which a goal is stated
42
TIMSS-Video study: structure of lessonsPercentage lessons, in which a summary statement is made
43
TIMSS-Video study: exercisesPercentage exercises per lesson with a context of symbols only
44
TIMSS-1999 Video Study
USA
video USA
Grade 8 (14 yrs) Heterogeneous
non-problem segment:the teacher shows the class examples of angles formed by secants and tangents intersecting inside, on, or outside a circle.
45
TIMSS-1999 Video Study
Netherlands
video NL
Grade 8 (14 yrs) Homogeneous (pre-university)
non-problem segment:the teacher briefly presents historical background related to the Pythagorean theorem (i.e., Pythagoras was a Greek gentleman). She also connects ideas learned previously (i.e., the definitions of various types of triangles) to the present lesson.
46
TIMSS-1999 Video Study
Hong Kong: grd 8, heterogeneousExplanation of a new concept.
video HongKong
Pre-knowledge: students can solve linear equations
What preceded the scene:Solving2x+10=2(x+5)2x+10=2x+10 0 = 0Then different values were insterted for x .
left hand sideright hand side“expanded form”“factorized form”
47
Next in this theatre…
TEDS(M)report in December 2009
TIMSS-2008 Advancedreport in December 2009
Pisa 2009report in 2010
48
References for PISA and TIMSSwww.pisa.oecd.org
http://timss.bc.edu
49
remains the question….
what are the benefits and limitations?