evolution of the portuguese basic and secondary...

Evolution of the Portuguese Basic and Secondary Education:Through the Lens of the International Large Scale Student Assessments

João Marôco, Ph. D.

IAVE, I. P.

Evolution of the Portuguese Basic and Secondary Education: Through the Lens of the International Large Scale Student Assessments

Summary

1. Why should we care about ILSSA?

2. Concurrent Validity of ILSSA vs. National Exams

3. Evolution of the Portuguese Basic and Secondary Education: Mathematics &

Language

4. ILSSA Driven Policy and Practices Changes

5. Final thoughts…


Data from international standardized assessments can be useful in research on causal/correlational factors within or across education systems (Rey, 2010).

“TIMSS has contributed to the practice of evidence-based decision making in today’s shifting landscape of education policy”

Ina S. Mullis, TIMSS & PIRLS Study Center

S. Breakspear (2012) on PISA:

• Policy-makers in most participating countries see PISA as an important indicator of system performance;

• PISA reports impact policy problems and set the agendas for national policy debate;

• Policymakers accept PISA as a valid and reliable instrument for internationally benchmarking system performance and changes over time;

• Countries have started policy reforms in response to PISA reports.

“Your education today is your economy tomorrow!”Andreas Schleicher, OECD

Rey, Olivier. 2010. The use of external assessmentsand the impact on education systems. In Beyond Lis-bon 2010 : Perspectives from Research and Develop-ment for Education Policy in Europe. Sheila M. Stoney (Ed.). Slough: NFER.Breakspear, S. (2012). The Policy Impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance, OECD Education Working Paper, n. 71

459

501

470

498

454

492

400

420

440

460

480

500

520

540

2000 2003 2006 2009 2012 2015

Sco

re o

nth

ePIS

A s

cale

Ano

SCIE READ MATH1000

0

OCDE

Portugal

442

532541

452

522

508

400

450

500

550

600

1995 1999 2003 2007 2011 2015

Sco

re o

n t

he T

IMSS s

cale

MATH SCIE1000

=

=

20 years of TIMSS 4 results15 years of PISA results

42 SCIE

28 READ

38 MATH 66 SCIE

99 MATH


ca. 1 schooling year ca. 2 schooling years

Three major players for basic and secondary education: PISA, TIMSS & PIRLS

What is PISA?

Programme for International Student Assessment: “(…) assesses the extent to which 15-year-old students, near the end of their compulsory education, have acquired key knowledge and skills that are essential for full participation in modern societies.”

(OECD, 2016)

SCIENCE LITERACY:

“the ability to engage with

science related issues, and with

the ideas of science, as a

reflective citizen”.

READING LITERACY:

“understanding, using, reflecting

on and engaging with written

texts, in order to achieve one’s

goals, knowledge and potential,

and to participate in society”

MATHEMATICAL LITERACY:

”capacity to formulate, employ

and interpret mathematics in a

variety of contexts; reasoning

and using mathematical

concepts, procedures, facts

and tools to describe, explain

and predict phenomena.”

COLLABORATIVE PROBLEM

SOLVING:

“ability to work with two or

more people to solve a

problem.”



What is TIMSS?

Trends in International Mathematics and Science Study: International Math and Science curricula assessment of 4th and 8th grade students. For example, for TIMSS 4th Grade:

APPLYING(38%)

Cognitive Domains:

REASONING(21%)

KNOWING

(41%)

Numbers

53%

Geometric

Shapes and

Measures

33%

Data Display

14%

MA

TH

EM

AT

ICS

Life Sciences

46%

Physical

Sciences

35%

Earth Sciences

19%

SC

IEN

CE

Content Domains:

(IEA, 2016)



What is PIRLS?

Progress in International Reading Literacy Study: Reading Literacy assessment of 4th grade students.

“the ability to understand and use those written language forms required by society and/or valued by the individual. Readers can construct meaning from texts in a variety of forms. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment.”

Focus &

Retrieve

Info

20%

Make

Strawightfoward

inferences

30%

Interpret &

Integrate Info

30%

Evaluate &

Critique

20%

Processes of Comprehension:Purposes for Reading:

Literay

Experience

50%

Acquire and Use

Information

50%

(IEA, 2017)



Do the tests measure the knowledge / skills of the students they are supposed to measure?

• Framework established by panel of (international) educational experts; Curricula overlap analysis;

• Rigorous translation, adaptation and cross-cultural validation - countries and consortia;

• Appropriate (statistical) estimation methods for large-scale sampling studies with planned missingness test design; Estimation of abilities by IRTs; Estimation of plausible values for student performance by latent regression models;

• Strong item-total correlation.

Are the tests reliable?

• Coding of responses under strict quality control (Coding Guides; Training; Monitoring of coding reliability);

• Non-public trend items;

• Anchoring of results to a T0.

Do ILSSA show Concurrent Validity with National High Stake Exams?

Lower Sec.

Education

Primary

Education

ISC

ED

3IS

CE

D 2

ISC

ED

1IS

CE

D 0

Grade 2

Grade 1

Grade 5

Grade 4

Grade 6

1st Cycle Basic EducationGrade 3

Pre-primary Education

2nd Cycle Basic Education

96 %

Grade 7

Grade 8

Grade 9

3rd Cycle Basic Education 87%

Grade 10

Grade 11

Grade 12

4

3

7

6

8

5

9

10

11

12

13

14

General Secondary Education

Scientific-Humanistic Courses

Vocational/Prof. Secondary Education

Tech.; Arts; Vocation.; Edu & train.; Non-dual voc.15

16

17

Upper Sec.

Education

75 %

Technological Spec. Courses

Polytechnic

Education

Bachelor’s

Master’s

Doctorate

ISC

ED

7

ISC

ED

8

ISC

ED

5

ISC

ED

6

ISC

ED

4

17

18

19

20

21

23

24

25

Ad

ult E

duca

tion

88 %

87 %

Higher

Education

50 %

2. Concurrent Validity of ILSSA vs. National ExamsThe Portuguese Educational System (2016)

The 2015 Mathematics case-study

National Exams vs. TIMSS Grade 4, PISA (grades 9 & 10) and TIMSS Advanced (Grade 12)

Mathematics grade 4 Mathematics grade 9Mathematics grade 12

r = 0,71 0,01

(n = 3 904)r = 0,64 0,01

(n = 4 178)r = 0,71 0,01

(n = 2 934)


No perfect fit between national high stake exams vs. ILSSA!

Why ?


País OCDEPaís não-OCDE

Mathematics G4 TIMSS 4National

Exams

Content Domains % %

Numbers 50.0 44.0

Geometric Shapes

and Measures35.0 43.0

Data Display 15.0 13.0

Cognitive

Domains% %

Knowing 40.0 43.0

Applying 40.0 36.0

Reasoning 20.0 21.0

TIMSS

ADV

National

Exams

Content Domains % %

Algebra 35.0 20.0

Calculus 35.0 32.5

Geometry 29.0 27.5

Probabilities and

Combinatorics— 20.0

Cognitive Domains % %

Knowing 29.0 22.5

Applying 41.0 57.5

Reasoning 30.0 20.0

PISA

2015

National

Exams

Content Domains % %

Quantity

(PRT - Numbers and Operations)25.0 12.0

Space and Shape

(PRT - Geometry and Measure)25.0 40.0

Change and relationships

(PRT - Algebra)25.0 35.0

Uncertainty and data

(PRT - Data Display)25.0 13.0

Cognitive Domains % %

Knowing - 37.0

Formulating 25.0 -

Applying 50.0 38.0

Interpreting/Reasoning 25.0 25.0

a. No ‘perfect’ match of content domains…


Mathematics G9 Mathematics G12

b. Different purposes…

LARGE SCALE STUDENTS ASSESSMENTSPORTUGUESE NATIONAL EXAMS

VOLUNTARY

• Schools are invited to participate.

• 4th grade and 9th grade parents give consent.

• Age based (PISA grades 7-11) or grade based (TIMSS)

NON-PUBLIC & SINGLE-DOMAIN ITEMS

• Trend items from past cycles to evaluate trends in education

systems.

• Items evaluate single cognitive/content domains.

AIM TO COMPARE COUNTRIES’ EDUCATIONAL SYSTEMS

• Diagnose/evaluate students knowledge and skills according to

international standards.

MANDATORY

• 4th grade & end of ISCED 1 (6th grade)

• At the end of ISCED 2 (9th grade).

• At the end of ISCED 3 (12th grade).

• Account for 25-30% final grade.

PUBLIC & HYBRID ITEMS

• Items become public after being applied.

• Hybrid items (evaluate content and cognitive domains).

CERTIFICATION

• Certify students’ knowledge and skills by the end of school cycles.

• 12th grade exams’ results are used for students admission to

higher education.

• ILSSA and National Exams have common normative references on the national mathematics curriculum for grades 4 and 12.

• Mobilization of knowledge, applying and reasoning competencies.


c. Students’ expectations & Teachers’ participation.

Disciplinary climate

Teacher support

Inquiry-based teaching an learning

Students’ expected occup.nal status

Grade Repetition

Achieving motivation

Test Anxiety

(…)

ESCS

b = 0.12

b = -0.20

R2 = 0.15***

S2SDR = 0.848***

b = 0.08

HLM

MATH G9 PRT – PISA MATH LITERACY

Class Size

(..)

Educational leadership

Teacher participation

School autonomy

b = -0.51

R2 = 0.60ns

S2SDR = 0.398ns


3. Evolution of the Portuguese Basic & Sec. Education

454

470

443

522

534

499

476

519

477

528

493

489

y = 3,1615x + 455,69

R² = 0,1485

400

425

450

475

500

525

550

2000 2003 2006 2009 2012 2015M

ath

em

atics

Sco

re N

atio

nal Exa

ms

Year

1000

0

454 466 466

487487 492

y = 2,6095x + 453,15

R² = 0,9094

400

425

450

475

500

525

550

2000 2003 2006 2009 2012 2015

Math

em

atics

Sco

re P

ISA

Year

1000

0

Portugal

OECD

National ExamsPISA


… as assessed by PISA Mathematics Literacy and Portuguese Mathematics Grade 9 high stake exams

470478 472

489488

498

y = 1,781x + 467,36

R² = 0,8333

400

425

450

475

500

525

550

2000 2003 2006 2009 2012 2015

Read

ing

Litera

cy

Sco

re P

ISA

Year

1000

0

Portugal

OECD

National ExamsPISA


… as assessed by PISA Reading Literacy and Portuguese Language Grade 9 high stake exams

512

457

594

584

541

545

509

524

492

541

553

y = 0,7866x + 523,44

R² = 0,0044

400

425

450

475

500

525

550

575

600

625

2000 2003 2006 2009 2012 2015Po

rtug

uese

Sco

re N

atio

nal Exa

ms

Year

1000

0


On Exams’ format:

• Item types and Coding/grading…

TIMSS 4th Grade Mathematics National Exam Grade 4 Mathematics


On Teaching Practices and Curriculum…

• Mathematics Grade 4:

Definition of Goals and Objectives for Math Learning, e.g. recommendation to add more teaching hours to “numbers and fractions”…

• Advanced Mathematics Grade 12:

(…) analysis of these elements [TIMSS Advanced framework], as well as curricula from other countries notparticipating in TIMSS Advanced, reveals that the inclusion in the curriculum of some fundamental themes,currently absent from the Secondary Education in Portugal, contributes decisively to the alignment of nationalcurricular options with the international plan. As an example, refer to those in the Primitive and Integral Calculusdomain. Some topics have also been reinforced, such as use of trigonometry solve triangles problems or thestudy of limits of successions and functions (…). (p. 4)

DGE (2015). Programa e Metas Curriculares Matemática do Ensino Secundário. Lisboa: Ministério da Educação. Retrieved from http://www.dge.mec.pt/sites/default/files/ficheiros/programa_metas_curriculares_matematica_a_secundario.pdf

“

”

http://www.dge.mec.pt/sites/default/files/ficheiros/programa_metas_curriculares_matematica_a_secundario.pdf


454

466 466

487 487

492

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

Math

em

atics

Sco

re P

ISA

M & P

4,6 & 9

On External Assessments Policies

(Census/Random samples;

Low Stakes/High Stakes)

M & P

4

M & P

4,6 & 9M & P

4,6 & 9

M & P

4,6 & 9M & P

4,6 & 9

M & P

4,6 & 9M & P

4,6 & 9

M & P

4,6 & 9

M & P

9

M & P

4,6 & 9

M & P

4,6,& 9

M & P

4,6 & 9

M & P

4 & 6,9M & P

4,6 & 9

M & P

4 & 6

5

21

12

Sample low-stakes Census low-stakes Census high-stakes Low-stakes exams High-stakes exams

Note:

PISA 2000 Math Literacy

without grades 5 and 6:

457

M & P & X

9 & 2, 5, 8


• ILSSA give valid and reliable indicators of a National Education System performance per comparison to international frameworks

Assuming all standards for sampling, testing and coding are respected.

• ILSSA allow for National Education Trends in Students’ achievement

Trend items to anchor results between cycles – contrary to what happens with National Public Exams.

• ILSSA frameworks are established by international well known experts Can serve as guidelines or offer suggestions for national curricula improvement.

• ILSSA allows for Assessment knowledge building (e.g. Items’ types and Coding practices) National Assessment Agencies can use this knowledge to improve validity and reliability of national and local assessments.

• ILSSA promote Education as a Worldwide Asset? Can ILSSA promote a Worldwide valid curricula?

Thank You

email: [email protected]

evolution of the portuguese basic and secondary...

Documents