School of Education

Investigating influences on post-16 progression in science in England using quantitative national data

Matt Homer

UK Science Education Research Conference –

Researchers sharing with researchers 2-4 July 2012

National Science Learning Centre,

University of York

School of Education

Some background issues…

• In the UK we have a lot of national assessment data.

• How can and should it be used?– Researchers, policy makers, schools, others?

• What are the limitations of findings from such data?

• The context: what does post-16 science participation look like – what can national data tell us about key influences on this?

– How might curriculum reform change patterns of participation?

2

School of Education

Overview of talk

• Policy and project (EISER) background• National data sources – the NPD• Known influences on post-16 science participation• Appropriate statistical methods• Results: Descriptive analysis – gender/SES/14-16

pathway• Results: Modelling participation• Conclusion/Discussion

– implications for policy/further research?

33

School of Education

Policy reformA new science curriculum for 14-16 year olds introduced in England in 2006

• Flexibility: a greater variety of ‘routes’ through 14-16 (KS4) science.

• A focus on teaching about the nature of science and socio-scientific issues – How science works

• Enhanced presence of vocational science courses (‘applied sciences’).

Also,

• ‘Entitlement’ to Triple award (separate sciences at KS4) from 2008 – big increases in students

4

School of Education

http://www.education.leeds.ac.uk/research/projects/enactment-and-impact-of-science-education-reform-eiser

Our project: Enactment and Impact of Science Education Reform (EISER)Mixed methods, 2008-2011: jointly funded by the Gatsby Charitable Foundation and the Economic and Social Research Council

This study examines school responses to this major curriculum reform. A particular focus is teacher enactment of the science curriculum in the classroom. The study is also investigating the initial impact of these reforms on student achievement, attitudes towards science education and participation in post-compulsory science courses.

Document analysis, interviews with teachers and students, National data analysis

Jim Ryder, Indira Banner, Matt Homer, Jim Donnelly…

5

School of Education

Immediate Longer Term

Increase student interest in their science education

Improve student attainment as measured through external examinations

Support students in engaging effectively with science-related issues as citizens

Increase post-compulsory participation in science education

Ensure adequate supply of scientists/engineers

Increase the employability of students

Improve social mobility and inclusion

Multiple aims of reform

Ryder and Banner (2011)

6

School of Education

Data source: the National pupil database (NPD)What is in it?

•All pupils in state-funded schools in England.

•Pupil level assessment data (primary to post-16)

•Pupil level personal data (e.g. gender, ethnicity, SES...).

•School level data (phase, selection policy, governance,…)

•Files linked by a unique pupil ID number.

•Large files: e.g. 600, 000 per cohort so KS4 results file – 6 million records for a particular year – 1 per qualification (GCSE…), i.e. multiple records per pupil.

•A powerful resource – the national picture and how it is changing over time.

7

School of Education

Use of NPD datasets in EISER• What are the patterns of participation and

attainment across KS4 and KS5 science courses?• How is this changing over time? • What are the influences on participation and

attainment?

Five successive KS4 cohorts• Two pre-2006 reform: 04-06, 05-07• Three post-reform: 06-08, 07-09,08-10

• Initial focus on KS4 – longitudinal change

Have recently started focussing on post-16…

8

School of Education

Our post-16 analysis is limited for two main reasons:

•NPD AS data is problematic (~20% missing due to cashing/not cashing in issues) – fixed(?) in newest data

•As of Jan 2011, we had only one post-reform full A-level cohort of data – hence it is probably too early to see longitudinal ‘impacts’ of the reforms on post-16 participation and attainment.

•Have investigated first post-reform cohort...(KS4: 06-08, A-Level 2010)

•Work in progress…

Problems with post-16 data

9

School of Education

Post-16 science – influences on participation

These are some known factors in uptake:•attainment at 16•14-16 pathway – Triple award vs. Dual award vs ‘Other’•Gender – e.g. biology vs. physics•Socio-economic status – lower groups less likely...

•(Plus teachers, student identity, school policy…)

But…•What are the relative ‘sizes’ of these effects?•How do they vary across the sciences (biology, chemistry, physics) and against other subjects?•To what extent are supposed ‘science’ problems with participation specific to science?

10

School of Education

Methods: progression to eight courses compared

Name of qualification Type of qualification More details of qualification types

Science courses

A-levels (formally Advanced Level General Certificate of Education) are usually studied over a two-year period, and entrance to university is normally predicated on sufficiently high achievement in these qualifications.

Applied A-levels are A-levels with a vocational emphasis.

BTEC National Awards are equivalent to A-levels in terms of academic value, but have a vocational focus.

Biology A-level

Chemistry A-level

Physics A-level

Single Award Applied Science

Applied A-level –vocationally orientated

BTEC National Award Applied Science

A vocationally orientated qualification

Comparator courses

Mathematics A-level

Psychology A-level

History A-level

11

School of Education

Methods – two approaches1. Descriptive statistics: comparing participation rates by

• Gender, SES and 14-16 pathway separately• Also, controlling for prior attainment at 16 in science

2.Multi-level modelling: predicting participation for each course based on gender, SES and 14-16 pathway plus:

• attainment in science and maths at 16, • socio-economic status (2 measures - free school meal, FSM;

and Income Depravation affecting children index, IDACI)• whether 14-16 school also teaches to 18

This modelling gives measures of the INDEPENDENT effect of each variable, and also gives an indication of how important schools are in determining participation.

12

School of Education

Descriptive results: overall participation post-16

13

Number and percentage of full cohort at 16 taking each qualification

Main messages

• Participation rates are generally quite low (a few per cent)

• The vocational routes are small

(A-levels etc in 2010)

School of Education

Red: all students doing the course

Blue: all students doing the course who achieved a (mean) grade ‘B’ in science at 16

Main messages

• Wide variation in participation rates across both sciences and non-sciences – e.g. biology/physics, maths/psychology

• The role of KS4 attainment – it certainly does not account for the differences

– within courses red bars similar to blue in length

Descriptive results: participation by genderPercentage of female students within courses

14

School of Education

Main messages

• Red: wide variation in participation rates across both sciences and non-sciences – but generally low – vocational courses the exception

• Blue: Controlling for prior attainment partly ameliorates the under-representation. But not entirely.

So FSM students are generally less likely to participate (e.g. physics) but this is not a ‘science’ problem per se (e.g. history).

Descriptive results: participation by FSMPercentage of FSM-eligible students within courses

15

School of Education

Main messages

• Red: As we would expect, TA students are heavily represented in ‘traditional’, but not in vocational, sciences (esp. physics)

• Blue: Controlling for prior attainment makes little difference.

Descriptive results: participation by pathwayPercentage of TA students within courses

16

School of Education

Multi-level modelling results

• Coefficients are odds ratio of participating compared to not

• An odds ratio of 1 means no effect on participation , >1 implies more likely to participate…

• All significant at 5% level except for shaded cells

Post-16 course

Odds ratios estimates for participation

Percentage of additional

variation at school level

Mean prior attainment in science at 16

Prior attainment in maths at 16

Gender: Male

Measures of socio-economic

status

14-16 science pathway 14-16

school teaches up to age 18FSM IDACI TA Other

Biology 1.237 1.008 0.652 1.052 1.281 2.550 0.132 1.077 6.6Chemistry 1.260 1.068 1.177 1.313 2.702 3.435 0.138 0.963 11.3Physics 1.188 1.133 7.683 0.971 1.076 2.275 0.168 1.239 9.9Single Award Applied Science

1.074 1.007 0.849 0.846 0.824 0.298 2.106 2.020 69.3

BTEC National Award Applied Science

1.068 0.998 0.838 1.107 2.044 0.708 1.094 1.381 74.1

Mathematics 1.061 1.402 2.368 1.049 1.600 1.370 0.660 1.075 10.2Psychology 1.076 1.034 0.292 0.856 0.780 0.715 0.453 0.967 8.4History 1.115 1.010 0.924 0.653 0.391 0.778 0.501 1.680 9.2

Odds ratio estimates are significantly different from 1 at 5% level except for those shaded. 17

School of Education

Implications for policy?

Odds ratio estimates are significantly different from 1 at 5% level except for those shaded.

• Gender: Girls are ‘missing’ from physics but so too are boys (a little) from biology – does it matter?

• TA tends to encourage participation in science but is this weakening over time? More to do longitudinally

• Students from lower SES backgrounds are not discriminated against in terms of progression to the sciences (once prior attainment is accounted for).– Other subjects are worse e.g. psychology, history

• Vocational sciences are of a different character – and schools play a much bigger role in promoting progression

More to do…what influences attainment?

18

School of Education

Finally…•The NPD is undoubtedly very useful in terms of assessing the macro picture following reform, and tracking longitudinally.

•Also need a mixed methods approach to find out what (and the why) is going on in schools, attitudes science etc.

•More changes working their way through the system

– EBacc, National curriculum review(?), changes to accountability measures (league tables, what counts and how much), O-levels (!)

19

School of Education

References• Banner, I; Donnelly, J; Homer, M; Ryder, J (2010) The impact of recent reforms in

the key stage 4 science curriculum In: School Science Review 92 (339) pp. 101 – 109

• Homer, M., Ryder, Jim & Donnelly, J., 2011. Sources of differential participation rates in school science: the impact of curriculum reform. British Educational Research Journal, pp.1-18.

• Homer, M., Ryder, Jim & Donnelly, J., 2011. The use of national data sets to baseline science education reform: exploring value-added approaches. International Journal of Research & Method in Education, 34, pp.309-325.

• Ryder, J; Banner, I. (2011) Multiple aims in the development of a major reform of the national curriculum for science in England In: International Journal of Science Education 33 (5) pp. 709 – 725

Thank you – questions?

Matt Homer: m.s.homer@leeds.ac.uk http://www.education.leeds.ac.uk/people/staff/academic/homer

20