investigating the factors affecting turkish students’ pisa ... · 1.3. factors affecting...

ISSN: 1300-5340 http://www.efdergi.hacettepe.edu.tr/

Hacettepe Üniversitesi Eğitim Fakültesi Dergisi (H. U. Journal of Education) 32(3): 544-559 [2017]

doi: 10.16986/HUJE.2017026950

Investigating the Factors Affecting Turkish Students’ PISA 2012

Mathematics Achievement Using Hierarchical Linear Modeling*

PISA 2012 Matematik Başarısını Etkileyen Faktörlerin Hiyerarşik

Lineer Model Kullanılarak İncelenmesi

Eren Halil ÖZBERK**

Kübra ATALAY KABASAKAL***

Nagihan BOZTUNÇ ÖZTÜRK****

Received: 13.10.2016 Accepted: 21.03.2017 Published: 31.07.2017

ABSTRACT: When Turkey's mathematical literacy performance in PISA 2003 and 2012 is examined, it is seen that

it has an increase of approximately 25 points between the two assessments. Despite this increase, the fact that the

mathematics performance is below the OECD average leads to the need to determine what factors affect the

mathematics performance of Turkish students. This study is aimed to investigate the factors affecting mathematics

performance of Turkish students at school and school level. Initially Turkish sample consisted a total of 4848

students from 170 schools. As a result of examining the assumptions of the two-level hierarchical linear model; the

data of 4236 students from 128 schools were included in the analysis. The results indicate that 64 % of the variability

in mathematics achievement was found between schools. Variables associated with attitudes towards mathematics

have significant effects on mathematics achievement. Mathematics self-efficacy has the most significant impacts on

mathematics achievement after controlling remaining variables. For the school-level variables, proportion of

mathematics teachers was found to be a strong predictor of a school’s average mathematics achievement. However,

it’s been found that student-teacher ratio was the only negative predictor of mathematics achievement at school-level.

School-level variables explained 44.1% of the variance in the between school difference in mean mathematics

achievement.

Keywords: PISA 2012, mathematics achievement, hierarchical linear modeling

ÖZ: Türkiye’nin PISA 2003 ve 2012 uygulamalarındaki matematik okuryazarlığı performansı incelendiğinde; iki

uygulama arasında yaklaşık 25 puanlık bir artışa sahip olduğu görülmektedir. Bu artışa rağmen matematik

performansının OECD ortalamasının altında yer alması, Türk öğrencilerinin matematik performansını etkileyen

faktörlerin neler olduğunun belirlenmesi ihtiyacını doğurmaktadır. Bu çalışmada da; Türk öğrencilerin matematik

performansını etkileyen öğrenci ve okul düzeyindeki faktörlerin incelenmesi amaçlanmıştır. PISA 2012 Türkiye

örneklemini 170 okuldan 4848 öğrenci oluşturmuştur. İki düzeyli hiyerarşik lineer modelin varsayımların incelenmesi

sonucunda; çalışmaya 128 okuldan 4236 öğrencinin verileri analize dâhil edilmiştir. Çalışmanın sonuçlarına göre;

matematik başarısındaki değişiminin %64’ü okullar arasındaki farklılıklardan oluşmaktadır. Matematiğe yönelik

tutumlar ile ilişkili olan değişkenlerin matematik başarısı üzerine önemli etkileri olduğu bulunmuştur. Analizde ele

alınan değişkenler kontrol edildikten sonra, matematik öz yeterliliğinin matematik başarısı üzerinde en çok etkiye

sahip olan değişken olmuştur. Okul düzeyinde değişkenlere bakıldığında; matematik öğretmenlerinin oranın, bir

okula ait ortalama matematik başarısının güçlü bir yordayıcısı olduğu görülmektedir. Öğrenci-öğretmen oranı ise

okul düzeyindeki matematik başarısının tek negatif yordayıcısı olmuştur. Okul düzeyi değişkenleri, ortalama

matematik başarısındaki okullar arasındaki farklılığın %44,1’ini açıklamıştır.

Anahtar sözcükler: PISA 2012, matematik başarısı, hiyerarşik lineer model

1. INTRODUCTION

1.1. PISA Background

The Programme for International Student Assessment (PISA) is the most representative

research within international comparison research which seeks to measure the academic

* This study was presented as an oral presentation at the International Congress on Education for the Future: Issues

and Challenges May, 2015 Ankara. ** Dr., Hacettepe Üniversitesi, Rektörlük, Ankara-Türkiye, e-posta: [email protected] *** Yard. Doç. Dr., Hacettepe Üniversitesi, Eğitim Fakültesi, Ankara-Türkiye, e-posta: [email protected] **** Dr., Hacettepe Üniversitesi, Rektörlük, Ankara-Türkiye, e-posta: [email protected]

Investigating the Factors Affecting Turkish Students’ PISA 2012 Mathematics Achievement Using Hierarchical...545


performance of 15-year-old students in mathematics, reading and science (OECD, 2004). There

are 65 countries and regions participating in PISA 2012. PISA was first performed in 2000 and

then repeated every three years regularly. PISA allows national policy makers to define,

monitor, and measure the end-goals of their education systems and student performance over

time and to compare them with those of other countries.

Turkey first joined PISA in 2003. Results from that assessment showed that, with mean

mathematics performance at around 423 points and more than half of the students performing

below baseline Level 2, Turkey’s 15-year-olds were performing far below the OECD average.

In 2006, where scientific literacy was the major domain of assessment, the picture was similar.

As shown in Figure 1, Turkey improved its mathematics performance by more than 25

points between 2003 and 2012. Turkey outperformed even in mathematics and other domains

compared to other OECD countries, including Greece, Spain and Italy. As a low performing

country, the factor associated with the mathematics achievement of Turkish students would

have important implications for mathematics education. Current study aims to examine various

factors at student and school levels using hierarchical linear modeling (HLM) in order to

monitor mathematics performances of Turkish students in PISA 2012.

Figure 1. Turkey’s Mathematics Literacy Performance by Year

1.2. Mathematics Literacy

The main purpose of PISA mathematic literacy is to discover whether students have been

well prepared mathematically for future challenges in life and work (Stacey and Turner, 2015).

It has been expected from students to analyze, reason and connect ideas, formulate solve and

interpret mathematical problems in a variety of situation (Thomson at al., 2013). PISA defines

mathematical literacy as:

“…an individual’s capacity to formulate, employ, and interpret mathematics in a

variety of contexts. It includes reasoning mathematically and using mathematical

Eren Halil Özberk, Kübra Atalay Kabasakal, Nagihan Boztunç Öztürk


546

concepts, procedures, facts and tools to describe, explain and predict phenomena. It

assists individuals to recognize the role that mathematics plays in the world and to

make the well-founded judgments and decisions needed by constructive, engaged and

reflective citizens. (OECD, 2013b, p.25)”

In this conception, mathematical literacy is expressed through not only using concepts of

mathematics but also making judgements and understanding the effectiveness of mathematics in

other domains.

1.3. Factors Affecting Mathematics Performance

Mean mathematic performance scores are generally known as the main referenced

variable when determining students’ mathematic performance in PISA. However, there are

multiple factors that might influence students’ mathematics performance. PISA uses

questionnaires for both students and schools to measure various variables.

PISA 2012 had put a great emphasis on the assessment of content variables toward

mathematics. Only a few studies have been done on mathematics performance of Turkish

students with student and school level factors. There are many constructed indices which are

helpful to discover precise performance of PISA scores.

The index of openness to problem solving (OPENPS) was constructed using student

responses over handling a lot of information, understanding things quickly; seeking

explanations for things, linking facts together easily and solving complex problems (OECD,

2013a). This definition basically related to students who are willing to engage with complex

problems and ready to solve expected or unexpected situations. Students who are more open to

problem solving generally perform at higher levels in mathematics (OECD, 2016).

The indexes related to mathematics attitudes mainly focus on the effects of constructs on

mathematics achievement. Self-efficacy and self-concepts are two constructs measuring how

students perceive their abilities in mathematics learning in PISA 2012. The mathematics self-

concept was constructed using student responses to question over being good at mathematics,

getting good grades in mathematics, learning mathematics quickly, understanding the most

difficult works and so on (OECD, 2013a). A common finding across numerous studies has been

that mathematical self-concept and mathematics achievement are positively related (Wang,

2007; Radišic, Videnovic and Baucal, 2015). The index of mathematics self-efficacy was

constructed using student responses over the extent whether they are feeling confident about

using mathematical equations and are aware of mathematics which might help to get a job and

improve their career. Considerable evidence from previous research suggests that, mathematics

achievement gaps diminish with the increase in mathematics self-efficacy (Kitsantas, Cheema

and Ware, 2011; Liang, 2010).

The socio-economic status (SES) of Turkish students has created segregation among

other OECD countries (Aydın, Sarıer and Uysal, 2012). A study by the Ministry of National

Education (MEB, 2010) about the PISA studies showed that the differences between the schools

and the school types are evident, and the effect of socioeconomic condition on the schools are

clear.

1.4. PISA Mathematics Performance of Turkish Students

Different studies have been attempted to explore what contributes to Turkey’s performance

in PISA mathematics. İş Güzel and Berberoğlu (2010) studied the effects of numerous variables

on students’ mathematics literacy performance in PISA 2003 using structural equation modeling



analysis. They found that the greatest relationship was found between self-efficacy in

mathematics and mathematical literacy. Other significant relationships with Mathematical

Literacy were found with the latent variables Interest in and Enjoyment of Mathematics,

Anxiety in Mathematics, and Disciplinary Climate for Mathematics Lessons. Güzeller and Akın

(2011) studied the relationship between the amount of time spent on homework in all subjects,

the time spent on mathematics homework, confidence in doing mathematics homework and the

mathematics achievement variables in PISA 2003 using multiple regression analysis. They

found positive relationships between mathematics achievement and confidence in doing

mathematics homework. Özer and Anıl (2011) studied factors effecting students’ science and

mathematics achievement in PISA 2006 using structural equating modeling analysis. They

found positive relationships between mathematics achievement and confidence in doing

mathematics homework. Researchers found that learning time has a positive effect on both

science and mathematics literacy.

Usta (2014) studied the effects of student and school level associated with mathematical

literacy performance of Finn and Turkish students attending PISA 2003 and 2012 through

HLM. The researcher found that size of the place in which the school is located, pre-school

education, mothers occupation, socio-cultural index and domestic educational resources were

meaningful variables for Turkey. Koğar (2015) studied direct and indirect factors effecting

PISA 2012 mathematical literacy performance through mediation model. The researcher found

that gender, economic, social and cultural status index and time allocated for learning

mathematics have a significant influence on mathematical literacy. The researcher also found

that the mediation variable that explains mathematical literacy at the highest level is self-

efficacy. Özbay (2015) investigated the differences mathematics, reading and science literacy

performance in PISA 2012 with respect to school types and different geographical regions using

MANOVA. Findings of the study demonstrated that Turkish students’ performance in

mathematics, reading and science differed significantly across the geographical regions and

school types. Uysal (2015) studied the effect of variables determining mathematics interest,

mathematics self-concept, mathematics anxiety, teacher-student relation, classroom

management and sense of belonging on the Mathematics achievement of Turkish students in

PISA 2012 by using structural equating modeling analysis. The researcher found that

mathematics interest and mathematics self-concept had positive effect on mathematics

achievement. Contrary, mathematics anxiety had negative and medium effect on mathematics

achievement. Moreover, no evidence had been found on mathematics achievement among the

effects of teacher-student relation, classroom management, and sense of belonging variables.

However, there are too many limitations in these studies. Mostly, the researchers include in

their models a limited number of variables that they are interested in, but neglect some other

variables that may be important predictors of mathematics achievement. This situation limits the

interpretations of the factors affecting Turkish students’ performance. Moreover, in PISA 2012

studies rotated design of the student questionnaire and missing data problem were not

mentioned properly. One of the main reasons why PISA 2012 uses a rotated design or content

questionnaires (CQ), was to extend the content coverage of the student questionnaire. And this

rotation presents massive missing data problems.

Based on the literature review, the purpose of study is to examine various factors at student

and school levels using hierarchical linear modeling (HLM) in order to monitor mathematics

performances of Turkish students in PISA 2012. The factors include not only those that have

been found significantly associated with performance in previous studies, such as mathematics

self-efficacy and mathematics self-concept, but also some neglected but potentially important

student and school factors, such as mathematics anxiety, familiarity with mathematical

concepts, Home educational resources, information and communication technology (ICT)



548

familiarity and various school educational resources. This study will explore the following

questions:

1. According to the PISA 2012 results, how much do Turkish schools vary in their

mathematics achievement? Does the mathematics achievement of students vary by

school?

2. Which student factors have an effect on the difference of student mathematics

achievement?

3. Which school characteristics have an effect on the student mathematics achievement?

Understanding variables that predict how student and school level factors affect

mathematic achievement is important. Traditional regression analysis cannot be used because at

a specific point, students are nested within schools. The data required for these analyses consist

of both achievement and measures of students (level 1) and measures of school traits for each

school (level 2) attended by the students. Thus the observations among students are not

independent. Hierarchical linear modeling (HLM) consider the variance among students in the

same school.

2. METHOD

2.1. Sample

The study data was obtained from the PISA 2012 Turkish data set. A two-stage stratified

sampling design was used for the PISA assessment. The first-stage sampling units consisted of

individual schools having 15-year-old students. Schools were sampled among PISA-eligible

schools with probabilities that were proportional to a measure of size. In the second-stage

sampling unit students were sampled within schools. Once schools were selected to be in the

sample, a complete list of each sampled school’s 15-year-old students was prepared (OECD,

2014). Prior to sampling, schools were stratified in the sampling frame by using region,

programme type, school type, gender, urbanicity and funding. The stratified sampling method

ensures the appropriate proportion of each type of school in the sample. Initially Turkish sample

consisted a total of 4,848 students from 170 schools. The sample consisted almost equal

proportion of boys (51,11%) and girls (48,89%). Due to minimum group member assumption,

42 schools were excluded from original sample. The final sampling in this study includes 4236

students from 128 schools.

2.2. Variables

The outcome variable of this study is mathematics performance which is reported as five

plausible variables calculated for each student in the sample by using one-parameter (Rasch)

model for dichotomous items (OECD, 2014). Each plausible value uses a posterior distribution

estimating the possible scores. Five plausible variables (PV1MATH – PV5MATH) of scores

were selected rather than one to facilitate unbiased estimation. The final reliability of PISA

mathematics domain is 0.912 (OECD, 2014).

2.2.1. Student Level Variables

Based on the potential factor affecting mathematics achievement six of attitudes towards

mathematics variables were included in this study: mathematics anxiety (ANXMAT, five items

with Cronbach alpha=0.82), attributions to failure in mathematics (FAILMAT, six items with

Cronbach alpha=0.66), mathematics self-efficacy (MATHEFF, eight items with Cronbach

alpha=0.82), mathematics intentions (MATINTFC, five items with Cronbach alpha=0.77),



mathematics self-concept (SCMAT, five items with Cronbach alpha=0.85) and subjective

norms in mathematics (SUBNORM, six items with Cronbach alpha=0.71).

Another perspective to predict mathematics achievement is to look for potential contents

which might have an influence on mathematics achievement. ICT familiarity, opportunity to

learn concept, problem solving, economic, social and cultural status. For ICT familiarity

perspective ICT entertainment use (ENTUSE, ten items with Cronbach alpha=0.90), attitudes

towards computers: limitations of the computer as a tool for school learning (ICTATTNEG,

three items with Cronbach alpha=0.77) and ICT resources (ICTRES) were included to validate

mathematics achievement. For PISA opportunity to learn concept perspective home educational

resources (HEDRES, seven items with Cronbach alpha=0.66), teacher behavior: student

orientation (TCHBEHSO, four items with Cronbach alpha=0.75), experience with pure

mathematics tasks at school (EXPUREM, three items with Cronbach alpha=0.92), familiarity

with mathematical concept (FAMCON and its adjusted index FAMCONC, thirteen items with

Cronbach alpha=0.87) were included to validate mathematics achievement. One of the last two

variables is commonly used variable is the index of economic, social and cultural status (ESCS)

which reflect a composite measure of parental occupation, parental education and wealth. Last

variable is a new scaled index namely openness to problem solving (OPENPS, five items with

Cronbach alpha=0.78) which was developed in recognition of the increasing importance of

problems solving in the cognitive part of the assessment (OECD, 2014).

2.2.2. School Level Variables

School-level variables used in this study are mathematics extracurricular activities at

school (MACTIV), proportion of mathematics teachers (PROPMATH), quality of school

educational resources (SCMATEDU, six items with Cronbach alpha=0.83), student-teacher

ratio (STRATIO) and student-related factors affecting school climate (STUDCLIM, eight items

with Cronbach alpha=0.87). MACTIV is an index of mathematics extracurricular activities

which asks for additional mathematics lessons, mathematic club activities and competitions.

PROPMATH was computed by dividing the number of mathematics teachers by the total

number of teachers. STRATIO was obtained by dividing the number of enrolled students by the

total number of teachers. STUDCLIM reflects student related aspects of school climate.

SCMATEDU was computed on the basis of six items measuring the school principals’

perceptions of potential factors hindering instruction at school (OECD, 2014).

2.3. Handling Missing Data

In the PISA 2012 student questionnaire, the rotation of context questionnaires (CQ) were

implemented for the first time. While ST01-ST28 items were in all rotated forms, ST29-ST104

items were alternately used in student questionnaire forms (OECD, 2014). As with the rotation

of the cognitive assessments in the PISA 2012 survey, the rotation of the context questionnaires

presents a massive missing data problem. Adams, Leitz, & Berezner (2013) have argued that

rotation of the CQ is not detrimental to plausible value estimation of latent proficiency based on

the full conditioning and population model approach. Kaplan and Su (2016) made a contribution

to missing data problem in the PISA 2012 when they presented findings on the consequences of

matrix sampling of context questionnaires for the generation of plausible values in PISA.

According to the findings of the research, Kaplan and Su (2016) examined several different

PISA 2012 forms of missing data imputation within the chained equations framework by

comparing predictive mean matching, Bayesian linear regression, and proportional odds logistic



550

regression. They found that predictive mean matching accurately reproduced the marginal

distributions of the missing context questionnaire data due to matrix sampling. In this research

single and five imputation predictive mean matching method were compared to the PISA 2012

Turkish data in order to overcome missing data problem. Predictive mean matching, unlike

other imputation methods, uses linear regression not to generate imputed values. Rather, it

builds a construct metric for matching cases with missing data to similar cases with original data

(van Buuren and Groothuis-Oudshoorn, 2011).

Figure 2: Comparison of Kernel Density Plots of Selected Items for Single and Five Imputations

In this research all the questionnaire items and estimated indices are implemented at the

same time by using predictive mean matching (pmm) with single and five imputations. All the

missing data were imputed using the “mice” package (van Buuren & Groothuis-Oudshoorn,

2011) in R (R Core Team, 2014). Figures 2 displays the kernel density plots of the selected

items using pmm for single imputation (the graph on left) and five imputations (the graph on

right), respectively. In Figure 2 for single imputation, all the densities of imputed values using

pmm are closer to the densities of the observed values compared.

Similarly, Figure 3 compare Q-Q plots of selected variables of imputed data. Single

imputation Q-Q plots for item-index data has unbiased estimations rather than five imputations

when compared to original data. This research uses the single imputed data for further analysis.

Figure 3: Comparison of Q-Q Plots of Selected Items for Single and Five Imputations

2.4. Statistical Analysis

This study employed hierarchical linear modeling (HLM) as a more appropriate method for

analyzing PISA data which students are nested within schools (Raudenbush & Bryk, 2002) in

order to examine the effects of student and school factors on mathematics achievement at both



student and school levels. HLM accommodates some of the variance among students attending

the same school, capturing differences in mathematics achievement among schools as well as

between students

2.4.1. Model 1

Model 1 is technically the Null model which is the simplest of the models and called the

fully unconditional model. It basically separates total variance into variance due to student level

and variance due to school level.

Student Level (Level 1):

School Level (Level 2):

The variance of outcome variable is equal to the sum of between variability (τ_00) and

within variability (σ^2). Null model also allows to calculate proportion of variance that is

attributable to school level, which is named intraclass correlation:

2.4.2. Model 2

Model 2 is the extent of Model 1 and named Random Coefficient Model. Random

Coefficient Model includes a covariate at student level with a random effect which has different

effects on school level variables.



.

.

2.4.3. Model 3

The final model, Intercepts-and-Slopes-as-Outcomes Model, allows to model the

variability of the regression coefficient using both intercepts and slopes (Raudenbush and Bryk,

2002).




552


.

.

3. FINDINGS

Table 1 presents the descriptive statistics for the student level, school level and outcome

variables. Most OECD questionnaire scale indices are standardized with a mean of 0 and

standard deviation of 1 for all student population of OECD countries. Thus, negative value

indicates that Turkish students responded less positively than the average response across

OECD.

Table 1: Descriptive Statistics for Selected Variables

Variables N Mean SD

Student Level

Mathematics Anxiety (ANXMAT)* 4236 .25 1.02

ICT Entertainment Use (ENTUSE)* 4236 -.38 1.33

Index of economic, social and cultural status (ESCS)* 4236 -1.38 1.09

Experience with Pure Mathematics Tasks at School (EXPUREM)* 4236 -.07 1.07

Attributions to Failure in Mathematics (FAILMAT)* 4236 .25 1.06

Familiarity with Mathematical Concepts (FAMCON)* 4236 .48 .81

Familiarity with Mathematical Concepts (Signal Detection Adjusted)

(FAMCONC)* 4236 .12 1.03

Home educational resources (HEDRES)* 4236 -.52 1.05

Attitudes Towards Computers: Limitations of the Computer as a Tool for

School Learning (ICTATTNEG)* 4236 .17 1.12

ICT resources (ICTRES)* 4236 -1.33 1.19

Mathematics Self-Efficacy (MATHEFF)* 4236 .01 .93

Mathematics Intentions (MATINTFC)* 4236 .18 .96

Openness for Problem Solving (OPENPS)* 4236 .22 .95

Mathematics Self-Concept (SCMAT)* 4236 -.05 .97

Subjective Norms in Mathematics (SUBNORM)* 4236 .26 1.10

Teacher Behavior: Student Orientation (TCHBEHSO)* 4236 .30 1.03

School Variables

Mathematics Extracurricular activities at school (MACTIV)* 128 1.81 1.33

Proportion of mathematics teachers (PROMATH) 120 .12 .04

Quality of school educational resources (SCMATEDU)* 128 -.03 .90

Student-Teacher ratio (STRATIO) 121 16.57 6.92

Student-Related Factors Affecting School Climate (STUDCLIM)* 128 -.02 1.08

Outcome Variables

PV1MATH 4236 456.38 93.84

PV2MATH 4236 456.52 93.58

PV3MATH 4236 456.53 93.17

PV4MATH 4236 457.36 93.60

PV5MATH 4236 457.55 93.54

*Indices standardized (µ=0, σ=1)



Based on descriptive statistic it is obvious that Turkish schools are slightly under the

OECD average except for extracurricular mathematics activities which is significantly above the

OECD average. For student level indices, economic and socio-economic status (ESCS) and ICT

resources (ICTRES) are significantly under the OECD average. The descriptive statistic and

additional central tendency statistic (skewness and kurtosis) both indicate that five plausible

values of mathematics achievement are roughly normally distributed (Skewness Max=.415,

Min=.381; Kurtosis Max=-.244, Min=-.293).

All factors were analyzed using HLM to explore their associations with mathematics

achievement in PISA 2012. Table 2 presents the results of HLM results for null model, random

coefficient model and intercepts-and-slopes-as-outcomes model. Average school mean

mathematics achievement was statistically different from zero ( =454.732, p=0.000). Given

the mean and variance a confidence interval was calculated to describe a range that includes

95% of all schools’ average mathematics achievement (CI= 454.732±1.96 ). With

95% confident the mean mathematics achievement mean is between 441.67 and 469.36. For the

variance in school means =5584.54, p=0.000, so there were considerable variations in the

school means. Interclass correlation coefficient (ICC) (ρ= / ( ) = 5584.546 /

(5584.546+ 3160.658) =0.64) indicates that 64 % of the variability in mathematics achievement

was between schools (remaining 46 % of variability within school). It implies that, on average,

mean mathematics achievement of Turkish schools vary heterogeneously between schools.

Thus, additional student-level variables were added to try to reduce the variance within schools

and the school-level predictors were added in order to explain between-school variance in the

Model 2 and Model 3.

Table 2: Fixed Effects Estimates and for Models of the PISA 2012 Mathematics Achievement

Fixed Effects Model 1 Model 2 Model 3

Coeff SE p Coeff SE p Coeff SE p

Intercept, 454.732 6.662 <0.001 454.722 6.661 0.000 456.34 5.234 0.000

Student Level

(ANXMAT, -4.312 0.912 0.000 -4.608 0.854 0.000

ENTUSE, 3.639 0.691 0.000 4.028 0.733 0.000

ESCS, 3.799 1.125 0.001 3.504 1.113 0.002

EXPUREM, 3.407 0.893 0.000 4.359 0.856 0.000

FAILMAT, -1.931 0.861 0.027 -1.812 0.894 0.042

FAMCON, 4.453 1.131 0.000 3.591 1.095 0.002

FAMCONC, 5.698 0.898 0.000 5.404 0.853 0.000

HEDRES , 3.515 1.358 0.011 4.5 1.424 0.002

ICTATTNEG, -4.743 0.805 0.000 -5.099 0.777 0.000

ICTRES, -2.714 1.187 0.024 -3.103 1.175 0.009

MATHEFF, 8.707 1.011 0.000 8.659 0.947 0.000

MATINTFC, 2.132 0.915 0.021 2.461 0.888 0.006

OPENPS, 3.872 0.926 0.000 4.286 0.894 0.000

SCMAT, 3.776 1.081 0.001 3.595 1.101 0.001

SUBNORM, -3.757 0.889 0.000 -4.225 0.837 0.000

TCHBEHSO, -5.767 0.866 0.000 -5.622 0.781 0.000

School Variables

MACTIV, 9.990 4.377 0.024

PROMATH, 457.72 21.938 0.000

SCMATEDU, 12.123 5.877 0.041

STRATIO, -2.501 0.829 0.004

STUDCLIM, 19.132 5.599 0.001



554

Intercept variance, 5584.54 5603.14 3122.40

Level 1 variance, 3160.65 2509.47 2733.63

Intraclass correlation, .64 .69 .53

Between-school variance

explained (%) - 1% 44%

Within-school variance

explained (%) - 21% 13%

The results based on Model 2 shows that after including 16 variables as a student-level

predictors of mathematics achievement within school, within school variability reduced by 21%

(3160.658-2509.471)/3160.658 =0.206), relative to the null model. Overall mean mathematics

achievement across schools was still significantly different from zero ( =454.722,p=0.000).

The average effects of all student-level variables on mathematics achievement was significant.

The largest effect has been found on mathematics self-efficacy scores. For each unit increase in

students’ mathematics self-efficacy, there were average 8.707 points increase in mathematics

scores in PISA 2012 across schools. For mathematics anxiety, attributions to failure in

mathematics, attitudes towards computers: limitations of the computer as a tool for school

learning, ICT resources, subjective norms in mathematics and teacher behavior: student

orientation variables there also had significant negative effects on mathematics achievement.

That mean one unit change in these variables decreases mathematics scores in PISA 2012 across

schools. Specifically, for each unit change in students’ attitudes towards computers could

decrease 4.743 points of mathematics scores in PISA 2012 across schools. Table 3 summarizes

the random effects for each model. Table 3 shows that at school-level it has been found that

there was a significant variation in school means in familiarity with mathematical concepts

variable. That means the effect of familiarity with mathematical concepts variable was not same

across schools in mathematics achievement.

Table 3: Random Effects Estimates and for Models of the PISA 2012 Mathematics Achievement

Random Effects Model 1 Model 2 Model 3

Variance SD p Variance SD p Variance SD p

Level 1 Error, 3160.65 56.22 0.000 2509.47 50.09 2733.63 52.28

Level 2 Error, 5584.54 74.73 3122.40 55.87 0.000

Student Level

ANXMAT 7.261 2.695 0.382

ENTUSE 3.787 1.946 >.500

ESCS 25.492 5.049 0.161

EXPUREM 20.581 4.537 0.177

FAILMAT 16.797 4.098 >.500

FAMCON 14.837 3.852 0.032 5.009 2.23 0.116

FAMCONC 9.985 3.160 >.500

HEDRES 57.543 7.586 0.051

ICTATTNEG 10.779 3.283 0.134

ICTRES 26.424 5.140 0.242

MATHEFF 11.270 3.357 0.405

MATINTFC 12.146 3.485 0.195

OPENPS 7.130 2.670 >.500

SCMAT 36.894 6.074 0.413

SUBNORM 25.636 5.063 0.178

TCHBEHSO 10.600 3.256 >.500



Model 3 provides both student-level and school-level predictors for mathematics

achievement. In this model intercept was treated as random with school-level predictors, and the

remaining coefficients were specified as fixed. Relative to the null model, 44.1% of the variance

in the between school difference in mean mathematics achievement was accounted by

mathematics extracurricular activities at school, proportion of mathematics teachers, quality of

school educational resources, student-teacher ratio and student-related factors affecting school

climate ( (Model 1)- (Model 3)/ (Model 1) =44.1%). Overall mean mathematics

achievement across schools was also remained significant from zero ( =456.34, p=0.000).

After controlling for other school-level variables there was a significant negative difference in

mathematics achievement between schools with high student-teacher ratio and low student-

teacher ratio ( =-2.501, p=0.004). The schools with low student-teacher ratio has more

qualified in mathematics achievement rather than high student-teacher ratio. After controlling

rest of school-level variables proportion of mathematics teachers had the largest significant

effect on mathematics achievement between schools ( =457.77, p=0.000). This result shows

that the amount of mathematics teachers in any school had the largest affect in mathematics

achievement in PISA 2012. Schools with more mathematics teachers tend to increase

mathematics achievement in PISA 2012 after controlling other school-level variables.

4. DISCUSSION and RESULTS

By way of HLM analyses at student and school levels, this study has emphasized the most

important factors affecting the mathematics achievement of Turkish students in PISA 2012. The

results of Model 1 indicate that 64 % of the variability in mathematics achievement was found

between schools. Thus, adding student-level variables could help to clarify variability within

schools. At student level factors, it has been found that there are consistent results with prior

research. Variables associated with attitudes towards mathematics have significant effects on

mathematics achievement. Mathematics self-efficacy has the most significant impacts on

mathematics achievement after controlling remaining variables are taken into account.

Consistent with previous researches (Güzel & Berberoǧlu, 2010; Anderson, Milford, & Ross,

2009) there is a positive and significant relationship between mathematics self-efficacy and

mathematics achievement. After controlling remaining variables, one-unit increase in

mathematics self-efficacy has increased mathematics achievement average 8 points in both

Model 2 and Model 3. Mathematics anxiety is another variables determining attitudes towards

mathematics. Unfortunately, In Model 2 and Model 3, it’s been found that mathematics anxiety

has negative significant effect on mathematics achievement. After controlling remaining

variables, one-unit increase in mathematic anxiety decrease mathematics achievement,

respectively. Findings are consistent with prior researches (Güzel & Berberoǧlu, 2010; Koğar,

2015; Uysal, 2015). Uysal (2015) also has found that mathematics anxiety had negative and

medium effect on mathematics achievement.

The index of subjective norms in mathematics has a negative effect on mathematics

achievement which is mostly related to perceptions of family and friends. According to the

findings of this study, it is found that there is a gap between student-related attitudes and

environment related factors with respect to attitudes towards mathematics. Factors related

external variables, such as subjective norms in mathematics and mathematics anxiety have

negative effects on mathematics achievement. Contrary, factors mostly related students’ belief,

awareness and intension have positive effects on mathematics achievement.

Delen and Bulut (2011) suggested that technology usage at school was found to be a

weak predictor of mathematics achievement. In that research, generally speaking, ICT family

variables have negative effect on mathematics achievement except for ICT entertainment use.

Students' use of ICT for entertainment purposes has positive effects on their mathematics



556

achievements which is consistent with the study of Skryabin, Zhang, Liu, and Zhang (2015).

After all, adding 16 student-level variables helped to explain %21 of variability within schools.

For the school-level variables, proportion of mathematics teachers was found to be a

strong predictor of a school’s average mathematics achievement. Mathematics extracurricular

activities at school, quality of school educational resources, student-related factors affecting

school climate are the variables that associate with better performance at school-level. However,

it’s been found that student-teacher ratio was the only negative predictor of mathematics

achievement at school-level. The results show that attendance in extracurricular activities had a

positive effect on students' mathematics achievement. Prior research concluded that this

relationship is common in literature and at the school level, better educational resources could

improve average school mathematics achievement (Schuepbach, 2015; Shelley and Su, 2011).

It’s been found that only student-teacher ratio variable has negative effect on mathematics

achievement. More likely the schools with a higher student-teacher ratio has lower mathematics

achievement because of overcrowded classroom. This results are expected especially when

proportion of mathematics teachers’ variable has the largest effect on mathematics achievement

at school-level. These two variable show somewhat consistency across Model 3 results. As a

conclusion, all these school-levels explained 44.1% of the variance in the between school

difference in mean mathematics achievement.

5. REFERENCES

Adams, R. J., Lietz, P., & Berezner, A. (2013). On the use of rotated context questionnaires in conjunction with

multilevel item response models. Large-Scale Assessments in Education, 1, 5. Retrieved from

http://www.largescaleassessmentsineducation.com/content/1/1/5

Anderson, D. (2012). Hierarchical Linear Modeling (HLM): An Introduction to Key Concepts Within Cross-

Sectional and Growth Modeling Frameworks, (Technical Report # 1308) Eugene: Behavioral Research and

Teaching.

Anderson, J. O., Milford, T., & Ross, S. P. (2009). Multilevel modeling with HLM: Taking a second look at PISA.

Quality Research in Literacy and Science Education: International Perspectives and Gold Standards, 263–286.

http://doi.org/10.1007/978-1-4020-8427-0_13

Aydın, A., Sarıer, Y. & Uysal, Ş. (2012). The Comparative Assessment of the Results of PISA Mathematical Literacy

in terms of Socio-Economic and Socio-Cultural Variables. Education and Science. 37(164), 20-30.

Delen, E., & Bulut, O. (2011). The relationship between students’ exposure to technology. TOJET: The Turkish

Online Journal of Educational Technology. 10(3), 311–318.

Güzeller, C. O., & Akın, A. (2011). An Examination of the Programme for International Student Assessment (PISA)

2003 Turkish Database with the Aim of Exploring the Relationship Between Homework Variables and

Mathematics Achievement. Educational Research and Reviews. 6(13), 793-803.

İş Güzel, Ç., & Berberoğlu, G. (2010). Students’ Affective Characteristics and Their Relation to Mathematical

Literacy Measures in the Programme for International Student Assessment (PISA) 2003. Eğitim Araştırmaları-

Eurasian Journal of Educational Research, 40, 93-112.

Kaplan, D., & McCarty, A. T. (2013). Data fusion with international large scale assessments: a case study using the

OECD PISA and TALIS surveys. Large-Scale Assessments in Education, 1, 6. Retrieved from

http://www.largescaleassessmentsineducation.com/content/1/1/5

Kaplan, D., & Su, D. (2016). On Matrix Sampling and Imputation of Context Questionnaires with Implications for

the Generation of Plausible Values in Large-Scale Assessments. Journal of Educational and Behavioral

Statistics 41(1), 57–80. doi: 10.3102/1076998615622221

Koğar H. (2015) Matematik okuryazarlığını etkileyen faktörlerin aracılık modeli ile incelenmesi, Education and

Science, 40 (179), 45-55. doi: 10.15390/EB.2015.4445

Kitsantas, A., Cheema, J., and Ware, H. W. (2011). Mathematics Achievement: The Role of Homework and Self-

Efficacy Beliefs. Journal of Advanced Academics, 22(2), 10-339.



Lam, T. Y. P., & Lau, K. C. (2014). Examining Factors Affecting Science Achievement of Hong Kong in PISA 2006

Using Hierarchical Linear Modeling. International Journal of Science Education, 36(15), 2463–2480.

http://doi.org/10.1080/09500693.2013.879223

Liang, X. (2010). Assessment Use, Self-Efficacy and Mathematics Achievement: Comparative Analysis of PISA

2003 Data of Finland, Canada and the USA. Evaluation & Research in Education, 23(3), 213-229.

Milli Eğitim Bakanlığı (The Ministry of National Education-TURKEY) (2010). PISA 2009 International Student

Assessment Project, Preliminary National Report. Ankara: Eğitim Araştırma ve Geliştirme Dairesi Yayınları.

Retrieved from: http://www.earged.meb.gov.tr/pdf/pisa2009rapor.pdf

Organization for Economic Cooperation and Development (2004). Learning for Tomorrow’s World: First Results

from PISA 2003. París: OECD.

Organization for Economic Cooperation and Development (2006). Assessing Scientific, Reading, and Mathematical

Literacy: A Framework for PISA 2006. Paris, France: Author.

Organization for Economic Cooperation and Development (2009a). Creating Effective Teaching and Learning

Results: First Results from TALIS. Paris, France: Author.

Organization for Economic Cooperation and Development (2009b). PISA 2006 Technical Report. Paris, France:

Author.

Organization for Economic Cooperation and Development (2013a), PISA 2012 Results: Ready to Learn: Students’

Engagement, Drive and Self-Beliefs (Volume III), PISA, OECD Publishing.

http://dx.doi.org/10.1787/9789264201170-en

Organization for Economic Cooperation and Development (2013b). The PISA 2012 Assessment and Analytic

Framework: Mathematics, Reading, Science, Problem Solving, and Financial Literacy, OECD Publishing.

http://dx.doi.org/10.1787/9789264190511-en

Organization for Economic Cooperation and Development (2014). PISA 2012 Technical Report. Paris, France:

Author.

Organization for Economic Cooperation and Development (2016, January 22). PISA Results from PISA 2012:

JAPAN. Retrieved from https://www.oecd.org/pisa/keyfindings/PISA-2012-results-japan.pdf

Özbay, C. (2015). Investigation of Turkish students’ performance in mathematics, reading and science literacy in the

PISA 2012 data. (Unpublished doctoral dissertation). Middle East Technical University.

Özer, Y., & Anıl, D. (2011). Examining The Factors Affecting Students’ Science and Mathematics Achievement with

Structural Equation Modeling. Hacettepe University Journal of Education. 41: 313-324.

R Core Team. (2014). R: A language and environment for statistical computing [Computer software manual]. Vienna,

Austria. Retrieved from http://www.R-project.org/

Radišic, J., Videnovic, M., & Baucal, A. (2015). Math Anxiety--Contributing School and Individual Level Factors.

European Journal of Psychology of Education, 30(1), 1-20.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods

(2nd ed.). Thousand Oaks, CA: Sage.

Schuepbach, M. (2015). Effects of extracurricular activities and their quality on primary school-age students’

achievement in mathematics in Switzerland. School Effectiveness and School Improvement, 26(2), 279–295.

http://doi.org/10.1080/09243453.2014.929153

Shera, P. (2014). School effects, gender and socioeconomic differences in reading performance: A multilevel

analysis. International Education Studies, 7(11), 28–39. http://doi.org/10.5539/ies.v7n11p28

Skryabin, M., Zhang, J., Liu, L., & Zhang, D. (2015). How the ICT Development Level and Usage Influence Student

Achievement in Reading, Mathematics, and Science. Computers & Education, 85, 49–58. doi:

10.1016/j.compedu.2015.02.004

Usta, G. (2014). PISA 2003 ve PISA 2012 matematik okuryazarlığı üzerine uluslararası bir karşılaştırma: Türkiye ve

Finlandiya. (Unpublished doctoral dissertation). Ankara University.

Uysal, Ş. (2015). Factors affecting the Mathematics achievement of Turkish students in PISA 2012. Educational

Research and Reviews. 10(12), 1670-1678. doi: 10.5897/ERR2014.2067

van Buuren, S., & Groothuis-Oudshoorn, K. (2011). Mice: Multivariate Imputation by Chained Equations in R.

Journal of Statistical Software. 45(3), 1-67.



558

Wang, J. (2007). A trend study of self-concept and mathematics achievement in a crosscultural context. Mathematics

Education Research Journal, 19(3), 33–47.

Uzun Özet

Ekonomik İşbirliği ve Kalkınma Teşkilatı - OECD (Organization of Economic Cooperation and

Development) tarafından finanse edilen Uluslararası Öğrenci Değerlendirme Programı – PISA (The

Programme for International Student Assessment), üç yılda bir OECD üye ve üye olmayan bazı ülkelerin

katılımı ile 2000 yılından itibaren uygulanmaktadır. PISA, 15 yaş grubu öğrencilerin matematik, fen ve

okuma becerilerini günlük hayatta karşılaştıkları problemlerin çözümünde nasıl kullandıklarını ölçen bir

uygulamadır. Bu nedenle; ülkelerin kalkınmasına katkı sağlayacak genç nüfus hakkında ciddi veriler

sağlamaktadır. Elde edilen veriler çeşitli araştırmalara konu olup farklı şekillerde kullanılabilir. PISA

verileri ile ülkelerin eğitim politikalarına yön verecek, veriye dayalı kararların alınması bir ülkenin

geleceğine dair önemli bir katkıdır.

Türkiye PISA uygulamasına ilk olarak 2003 yılında katılmış ve sürekli katılımı devam etmektedir.

Ülkemizin PISA uygulamalarında gösterdiği performanslar incelendiğinde; yıllara göre artan bir başarıya

sahip olmasına rağmen halen OECD ortalamasının altında yer aldığı görülmektedir. Performansa etki

eden faktörlerin incelenmesi, Türk eğitim sistemine getireceği katkıları sebebiyle büyük önem teşkil

etmektedir. Bu sebeple, PISA 2003 uygulamasından itibaren ulusal alan yazında çeşitli araştırmalarla

okuma becerisi, matematik okuryazarlığı ve fen okuryazarlığı performansları incelenmekte ve çeşitli

öneriler sunulmaktadır.

İlgili çalışmalara bakıldığında performanslara etki eden değişkenlerin genel olarak tek düzeyde ele

alındığı görülmüştür. Bu çalışmada ise, PISA 2012 uygulamasına katılmış olan Türk öğrencilerin

matematik performansına etki eden değişkenlerin okul ve öğrenci düzeylerinde incelenmesi

amaçlanmıştır. Bu amaç doğrultusunda PISA 2012 öğrenci anketi ve okul anketlerinde yer alan ve alan

yazında matematik performansına etkisi olduğu düşünülen değişkenler ele alınarak ilişkisel bir desen

oluşturulmuştur.

Matematik performansına etkisi olduğu düşünülen değişkenleri okul düzeyinde 25 indeks, öğrenci

düzeyinde ise 37 indeks oluşturmaktadır. Bu indekslerin matematik performansına olan etkilerinin

incelenmesi için Hiyerarşik Lineer Model (HLM) analiz yöntemi kullanılmıştır. HLM yöntemi ile hem

öğrenci ve okul düzeyindeki değişkenlerin performansa olan etkisi hem de okullar arasındaki

farklılıklardan kaynaklı olan başarı değişiklikleri incelenmiştir. Analize başlamadan önce PISA 2012’ye

katılmış olan 170 okuldan 4848 öğrencinin verileri ile iki düzeyli HLM varsayımları test edilmiştir.

Varsayımların test edilmesi sonucunda analiz 128 okuldan 4326 öğrenci verisi kullanılarak yapılmıştır.

PISA 2012 uygulamasında, daha önceki PISA uygulamalarından farklı olarak, öğrenci anketinde

farklı desenler kullanılmış ve 3 farklı anket kitapçığı oluşturulmuştur. Daha az soru ile daha fazla veri

elde etme amacıyla yapılan bu desenleme sonucunda ciddi bir kayıp veri sorunu ortaya çıkmıştır. Bu

kayıp veri sorununu giderebilmek için yordayıcı ortalama eşleşme (pmm) yöntemi kullanılarak tüm anket

maddeleri ve indeksler aynı anda uygulanmıştır. Kayıp verilerin giderilmesi için R yazılımında “mice”

paketi kullanılmıştır. Kernel yoğunluk grafikleri ve Q-Q grafikleri incelerek kayıp verilerin giderilmesi

için tek bir uygulamanın (imputatiton) yeterli olduğuna karar verilmiştir.

Analiz sonucunda okul düzeyinde 5 indeksin (Okulda ders programı dışındaki matematik

aktiviteleri (MACTIV), Matematik öğretmenlerinin oranı (PROMATH), Okuldaki eğitimsel kaynakların

kalitesi (SCMATEDU), Öğretmen-öğrenci oranı (STRATIO), Öğrenci-okul iklimini etkileyen ilişkili

faktörler (STUDCLIM)), öğrenci düzeyinde ise 16 indeksin (Matematik kaygısı (ANXMAT), Bilgi ve

İşlem Teknolojisinin (BİT) eğlence için kullanımı (ENTUSE), Ekonomik, sosyal ve kültürel statünün

indeksi (ESCS), Okuldaki matematik görevleri deneyinimi (EXPUREM), Matematik başarısızlığı

(FAILMAT), Matematiksel kavramlara aşinalık (FAMCON), Ayarlanmış matematiksel kavramlara

aşinalık (Signal Detection Adjusted) (FAMCONC), evdeki eğitimsel kaynaklar (HEDRES), Bilgisayara

yönelik tutumlar: bir araç olarak bilgisayarın okul öğrenmeleri için sınırlamaları (ICTATTNEG), BİT

kaynakları (ICTRES), Matematik özyeterliliği (MATHEFF), Matematiğin amacı (MATINTFC), Problem

çözmeye açıklık (OPENPS), Matematik benlik kavramı (SCMAT), Matematikteki öznel normlar

(SUBNORM), Öğretmen davranışı: öğrenci oryantasyonu (TCHBEHSO)) matematik performansını

anlamlı bir şekilde etkilediği sonucu elde edilmiştir.



İki düzeyli HLM analizi ile 3 model elde edilmiştir. Model 1’in sonucuna göre; matematik

başarısındaki değişkenliğin %64’ü okullardan kaynaklanmaktadır. Bu sebeple öğrenci düzeyindeki

değişkenlerin eklenmesi ile okul içindeki değişkenliğe bir açıklık getirilmiştir. Matematiğe karşı tutumla

ilişkili olan değişkenler matematik başarısı üzerinde önemli bir etkiye sahiptir. Analizde kalan

değişkenler kontrol edildikten sonra matematik öz yeterliğinin matematik başarısı üzerinde en önemli

etkiye sahip olan değişken olduğu görülmüştür. Matematik öz yeterliğindeki bir birimlik artışın, hem

Model 2 hem de Model 3’te ortalama 8 puanlık bir artış sağladığı sonucu elde edilmiştir. Matematik

kaygısı hem Model 2 hem de model 3’te matematik başarısını olumsuz yönde etkilemiştir. Kalan

değişkenler kontrol edildikten sonra matematik kaygısındaki bir birimlik artış matematik başarısını

düşürmüştür.

Öğrenci düzeyindeki 16 değişkenin modele eklenmesiyle okullar içindeki değişkenliğin %21’i

açıklanmıştır. Okul düzeyindeki değişkenler için, matematik öğretmenlerinin oranı okulun ortalama

başarısı için güçlü bir yordayıcı olarak bulunmuştur. Okulda ders programı dışındaki matematik

aktiviteleri, okuldaki eğitim kaynaklarının kalitesi ve öğrenci-okul iklimini etkileyen ilişkili faktörler,

okul düzeyinde daha iyi bir performans sağlanması ile ilişkili olan değişkenler olmuştur. Öğrenci-

öğretmen oranının okul düzeyinde matematik başarısının tek negatif yordayıcısıdır.

Sadece öğrenci-öğretmen oranı değişkenin matematik başarısı üzerinde olumsuz etkiye sahip

olduğu tespit edilmiştir. Büyük olasılıkla daha yüksek bir öğrenci-öğretmen oranına sahip okullarda

sınıfların kalabalık olması sebebiyle daha düşük bir matematik başarısı vardır. Model 3’e bakıldığında

tüm bu okul düzeylerinin ortalama matematik başarısının okullar arasındaki farkın varyansının %44,1’ini

açıklamıştır.

investigating the factors affecting turkish students’ pisa ... · 1.3. factors affecting...

Documents