class, school, municipal, and state effects on mathematics achievement in argentina: a multilevel...

This article was downloaded by: [Dicle University]On: 07 November 2014, At: 08:29Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

School Effectiveness and SchoolImprovement: An International Journalof Research, Policy and PracticePublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/nses20

Class, school, municipal, and stateeffects on mathematics achievement inArgentina: a multilevel analysisRubén Alberto Cervini aa Department of Social Science , Quilmes National University ,Buenos Aires, ArgentinaPublished online: 22 Jul 2009.

To cite this article: Rubén Alberto Cervini (2009) Class, school, municipal, and state effects onmathematics achievement in Argentina: a multilevel analysis, School Effectiveness and SchoolImprovement: An International Journal of Research, Policy and Practice, 20:3, 319-340, DOI:10.1080/09243450802664404

To link to this article: http://dx.doi.org/10.1080/09243450802664404

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

http://www.tandfonline.com/loi/nses20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/09243450802664404

http://dx.doi.org/10.1080/09243450802664404

Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

Class, school, municipal, and state effects on mathematics achievement

in Argentina: a multilevel analysis

Ruben Alberto Cervini*

Department of Social Science, Quilmes National University, Buenos Aires, Argentina

(Received 6 November 2007; final version received 17 September 2008)

This article analyzes the distribution of mathematics achievement among class,school, municipality, and state in Argentina. Data from the Year 2000, 6th-gradePrimary School Census from the Minister of Education are analyzed usingmultilevel methodology. The results indicate that all levels of the education systemare relevant and must be considered. If all levels are included, the school ‘‘raw’’effect is much less (18%) than if they are not (31%). Prior academic performanceand socioeconomic level of the individual pupil, plus socioeconomic compositionvariables, accounted for most of the performance distribution among school,municipality, and state levels. These kinds of variables do not explain significantinter-class differences inside the school. As a consequence, the class became thesuperior level with the larger proportion of the total unexplained variance (15%),while the school variation represents just 8,6% of that variation. Finally,unexplained inter-school and inter-class variation is still significant making itnecessary to further investigate relevant schooling and classroom factors.

Keywords: mathematics achievement; primary education; school effectiveness;multilevel analysis; educational inequality

Introduction

The central hypothesis of the school effectiveness research (SER) paradigm is thatcertain school characteristics have an effect on pupil school performance, even aftercontrolling for the demographic, academic, and socioeconomic background of theindividual pupil. Numerous studies seem to support this hypothesis (Bosker &Witziers, 1996; Knuver & Brandsma, 1993; Mortimore, Sammons, Stoll, Lewis, &Ecob, 1988; Opdenakker & Van Damme, 2001; Phillips, 1997; Rutter, Maughan,Mortimore, & Ouston, 1979; Sammons, Thomas, & Mortimore, 1997).

Pupil achievement, as commonly measured by standardized testing, is taken asthe effectiveness indicator, and average student achievement by school is assumed asthe school performance indicator. The proportion of the between-school variance inrelation to the total achievement variance is considered as the ‘‘raw’’ school effect,and the school variance that cannot be explained by control variables (‘‘residual’’) isconsidered as the ‘‘net school effect.’’ It is expected that a significant proportion ofthe unexplainable school variation could be potentially explained by specific schoolfactors.

*Email: [email protected]

School Effectiveness and School Improvement

Vol. 20, No. 3, September 2009, 319–340

ISSN 0924-3453 print/ISSN 1744-5124 online

� 2009 Taylor & Francis

DOI: 10.1080/09243450802664404

http://www.informaworld.com

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

As such, the magnitude of the ‘‘net’’ school effect depends on the explanatorypower of the control variables that are considered in the analysis, especially those ofcompositional and contextual characteristics. Additionally, Opdenakker and VanDamme (2000) and, more recently and systematically, Van den Noortgate,Opdenakker, and Onghena (2005) have shown that the estimation of the relativeimportance of school variance is strongly affected by the levels specified in the model,in particular the lower (classroom) and the upper (school district, county, state, etc.)school levels. Again, the estimation of the ‘‘net’’ effect of each level in the modelneeds to be controlled by composition variables at each level.

Nevertheless, studies frequently estimate the net school effect without includingmeasurements such as ‘‘composition’’ or ‘‘context’’, or solely specify two levels(students and school), leading to an overestimate of the school effect, and justifyingcriticisms that recommend including other aggregate levels below (the class) andabove the school (e.g., municipality) (Luyten, 2003; Van den Noortgate et al., 2005),and also taking into account the environment and the composition, not only fromthe school level (Thrupp, 2001b) but also from other aggregate levels, such as ‘‘theimpact of classroom composition’’ (Luyten, Visscher, & Witziers, 2005, p. 259).

Using data from the 2000 sixth-grade Primary School Census, the main objectiveof this article is to evaluate the hypothesis of school effectiveness in primaryeducation in Argentina, taking into account mainly two aspects pointed outpreviously: to specify factors below and above the school level and to include‘‘compositional’’ and ‘‘contextual’’ measurements in the analysis. The objectiveimplies the estimation of the relative importance of each level in explaining thedifferences in test results, before and after controlling for ‘‘contextual’’ effect at eachlevel. Multilevel linear modeling is used for analyzing the data (Aitkin & Longford,1986; Bryk & Raudenbush, 1992; Goldstein, 1995).

For primary education in Argentina, at least four levels are particularly relevant:classroom, school, municipality, and state/provincial. Provinces are responsible forthe administration of basic education, but a significant part of the financial resourcescomes from the national government. At the same time, each province is divided intoeducational districts, whose geographic limits in most cases overlap with themunicipalities. Municipalities, or educational districts, are the main spatial andinstitutional references for both administration and educational policies designed atthe provincial level. This structure becomes particularly relevant for the analysis ofthe decomposition of the achievement variance at different levels of the educationalsystem. However, there are no antecedents for this type of evaluation in Argentina.

Knowing both (a) how homogeneous each relevant level of the educationalsystem is and what is its relative importance in relation to the total variance structureand (b) how much of the variations in each level (‘‘raw level effect’’) is explained bythe uncontrollable ‘‘compositional’’ factors holds relevance for both policymakersand school effectiveness researchers. Educational policies that are designed andimplemented for a homogeneous population will not be effective if such ahomogeneous assumption is not true. If the variation between classrooms insidethe school is significant, policy directed towards the school institution to improveachievement that does not consider such variation has a high probability of notreaching its objectives. In order to form their policies, provincial governments mustknow if there exists significant between-municipalities variation and how much of itis explained by the uncontrollable ‘‘contextual’’ characteristics of the populationsurrounding schools. On the other hand, researchers need to know what is the

320 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

maximum effect that these internal school factors can have along with the otherlevels of the educational systems, to orient their search for educational factors withthe highest potential explicative power. If the variation between classrooms is greaterthan the variance between schools, then in the search for areas of effectiveness itwould be more sensible to look inside the schools rather than just between theschools (Webster & Fisher, 2000).

Conceptual models and theory

The tradition of SER is characterized by the predominant use of quantitativemethodologies (large-scale quantitative methodologies). While trying to identifyfactors that lead to better educational results, SER researchers have worked withgeneral conceptual models to classify the specific measurements included in theirresearch and to show the relationships among them (Scheerens, 1994; Scheerens &Bosker, 1997; Stringfield, 1994). Commonly, the factors are classified into inputs,contexts, processes, and results (Stufflebeam & Shinkfield, 1985), and within thesecategories, there are proposed subcategories that include a further closeness betweenthe conceptual structure and the available measurements. As well, these factors canbe defined at different levels (student, class, school, etc). Finally, the hypothesis isformed and the investigation proceeds with correlations between the availablevariables, based on the conceptual model. This analysis commonly includesadjustments for input (prior student achievement and socioeconomic status) andfor context (socioeconomic school composition).

Obviously, these general conceptual structures cannot be strictly considered astheory. One of the principal criticisms of SER studies is the lack of theoretical bases,not only in selecting analytical measurements but also in explaining why certaindeterminative variables correlate with school effectiveness and which mechanismsunderlie these correlations. These conditions are necessary to be able to infer causalrelations, attainable only using data produced through qualitative and detailedmicrolevel research (Thrupp, 2001b).

Even though the analyses of extensive databases have certain limits, they areindispensable to know the true magnitude and incidence of the facts and processrevealed by the microlevel in the educational system that has been researched. This isto say that the size and the relative weight of the processes detected in the microlevelstudies are marked by the estimations of parameters from large-scale quantitativestudies. In the same way, these analyses, like the interactions between covariates andachievement, inside and between different levels, as well as the direct and indirecteffects, guide or validate the qualitative microlevel studies.

For such estimations to be reliable, it is necessary to meet certain basicrequirements in the multilevel analysis, such as (a) to use criterion-referenced tests asthe indicator for evaluating effectiveness; (b) to specify all levels whose factors couldaffect the learning process; (c) to consider several, different, and adequate individualpupil background characteristics, including prior pupil attainment measurement;and (d) to include ‘‘compositional’’ and contextual characteristics in each aggregatedlevel. Multilevel analysis will estimate, for each level, the variance not explained byvariables in (c) and (d). The ‘‘residual’’ in each level is the maximum variation thatcould be potentially explained by malleable educational specific factors situated inthe corresponding level (maximum probable ‘‘net’’ effect). Still, unmeasured factorsbeyond level (classroom, school, etc.) control may explain just as well part or the

School Effectiveness and School Improvement 321

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

total of those ‘‘residual’’ level variations. From this perspective, it seems moreadvisable to include in the analysis as many control variables (covariates) as possible,rather than limit them in the functioning of restrictive theoretical structures.Moreover, certain conceptual taxonomies, such as economic, cultural, and symboliccapitals, well-founded theoretically (see the sociological theory of Bourdieu, 1977,1984) are sufficiently broad in scope as to permit the use of a great variety ofcovariates.

Given that this study utilizes bureaucratic data at a system level already obtained,it has been decided to analyze all the available measures for the individual pupil thatcan be considered as covariates.

The present study situates itself in this perspective and contemplates the fourconditions established above. The main hypothesis of this research is that the estimatedmagnitude of the school ‘‘net effect’’ on the pupil achievement in Argentina decreasesstrongly when both (a) classroom and municipal levels are specified and (b) severaland different ‘‘compositional’’ measurements at all levels are included in the analysis.On the other hand, it is expected that the extra-school factors do not explain classvariation. As a consequence, the unexplained variation at this level will becomemore important than the school variance. The latter indicates the greater potentialrelevance of the educational class factors in relation to the school factors.

Therefore, the study does not include any analysis of either the interaction andindirect effects of the considered measurements or any particular educationalmeasurement (educational effectiveness factors). The latter would require a relevantand specific theoretical foundation. Creemers and Kyriakides (2006) suggest, from aquantitative approach, that this conceptual framework must (a) be multilevel, (b)include curvilinear analysis, (c) indicate the dimensions (frequency, focus, stage,quality, and differentiation) of the measurement of each factor, and (d) definerelations among effectiveness factors. These are all tasks whose results will beconsidered in future works.

Previous empirical studies

It is well established that both pupil family socioeconomic level and cultural back-ground performance have strong impacts on pupil achievement. Thus, we will focusonly on three other specific aspects: the prior achievement effect, the ‘‘composi-tional’’ effect, and the number of aggregated levels to be specified in multilevelmodels.

Prior achievement and socioeconomic effect on student progress

A general consensus exists that for a just comparison between schools, the grade ofefficacy must be measured by the obtained ‘‘progress’’ by students (‘‘value added’’).But it is not agreed upon whether or not the ‘‘value added’’ is affected by thesocioeconomic variables and what is the relative importance of both types of factors.It has been argued that the explanatory force of the socioeconomic background isalways much smaller than that obtained with prior achievement (Goldstein, 1998;Thomas & Mortimore, 1996). Additionally, the latter completely absorbs the effectof the socioeconomic factors, making its control dispensable (Fitz-Gibbon, 1996).For example, the Tennessee Value-Added Assessment System (TVAAS – USA)neither measures nor controls the differences of race or socioeconomic status for the

322 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

students, given that the statistical analysis has demonstrated that a relationship doesnot exist between annual progress of achievement and ‘‘race, eligibility for free orreduced lunch, or any other of a variety of potentially biasing differences amongstudents’’ (Stone, 2003, p. 6).

This approach has been criticized for tending to conceal the possible effect of thestudent’s social origin and the socioeconomic context of the school (Gibson &Asthana, 1998), even though evidence exists. In the UK and in the USA(D’Agostino, 2000; McCaffrey, Koretz, Lockwood, & Hamilton, 2003; Thomas,2001), there exists evidence where the student’s background predicts his/herprogress, and for this, modeling student background characteristics ‘‘remains anempirical question that must be addressed by each analyst in the context of thesespecific factors’’ (McCaffrey et al., 2003, p. 70). In his review of the 25 ‘‘value added’’studies that included different measurements of students’ socioeconomic back-ground, Cervini (2006) confirmed that only in five of them did such covariables nothave significant effects on the progress of learning. All of the studies but onereferring to the secondary level reveal that the set of student’s prior achievement andsocioeconomic background individual variables have a greater relative effect at theinter-school level than at the inter-student level. On the other hand, the studies aboutprimary education do not arrive at a clear conclusion, perhaps because they do notshow the effect on the classroom level, deal with the 1st year of the primary level, orinclude a report of the teacher as an indicator of prior achievement (Hill & Rowe,1998; Kyriakides, 2004; Strand, 1997).

There are few studies that inform about and distinguish the effects of both typesof variables. At the primary level (Muijs & Reynolds, 2003), prior achievementproduces a large decrease (44%) in the variation at the student level, explainingnearly all of the inter-school variation and more than half of the inter-classroomvariation. The student’s socioeconomic background influences more on the studentlevel (18%) but adds very little to the explanation of the ‘‘residual’’ left by priorachievement. Differently, in Strand (1997), the student’s cognitive background‘‘explains’’ very little of the inter-school variation (4%) but produces almost thesame decrease – 39% – in the intra-school (student) variation, while the otherstudent backgrounds barely contribute to 2% of the explanation at this level. In thesecondary level (Veenstra & Kuyper, 2004), students’ diverse personal backgroundsexplain 6% and 13% of the intra-school and the inter-school variations, respectively.The cognitive background of the student explains 24%, 80%, and 73% of the‘‘residual variance’’ at student, class, and school levels, respectively. Obviously, fromthese studies it is not possible to identify a pattern of behavior.

In middle schools in Argentina (Cervini, 2006), prior achievement explains 17%of the inter-student variance and 45.4% of the inter-school variance, while the othercharacteristics of the students produce a decrease of 50% and 5.5% in the inter-school and intra-school variables, respectively. Both types of variables producesimilar effects at the school level but very different effects at the student level. Themodel with both subsets of variables explains 71.3% of the variation at the schoollevel and 19.6% of the variation of the student level. These results allow us toconclude that previous student achievement has the capacity to explain the intra-school variation far better than other individual variables. If the objectives were toonly ‘‘adjust’’ the achievement averages of the schools, it would disregard thestudent’s prior achievement, whenever it takes into account good measurements ofthe student’s personal and familial characteristics.


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

School contextual effect

The contextual effect is captured by a ‘‘compositional variable’’ that ‘‘measures anaspect of the composition of the school to which the individual student belongs’’(Goldstein, 1995, p. 30). To estimate the contextual effect, the correspondingindividual pupil background characteristics must be specified to ensure that effectsattributed to higher level units are not simply a consequence of poorly specifiedindividual variables (Hauser, 1970). Then the contextual effect is the impact of theschool student composition on individual student achievement, over and above thedirect influence of his/her own individual background.

One of the criticisms made by some authors against the traditional schooleffectiveness studies is that they have not paid enough attention to the effect of theschool’s social context, giving the impression that it functions independently of itsenvironment (Angus, 1993; Coe & Taylor, 1998; Gerwitz, 1998; Hatcher, 1998; Slee& Weiner, 1998; Thrupp, 2001a, 2001b). In fact, several empirical studies havereported an important and significant effect of the socioeconomic and cultural‘‘compositional’’ characteristics on student achievement (Bryk & Raudenbush, 1992;Caldas & Bankston, 1997; Cervini, 2005; Nuttall, Goldstein, Prosser, & Rasbash,1989; Sammons et al., 1997; Strand, 1997). This conclusion has been extended to theacademic or intellectual background composition, whether at the primary educationlevel or the secondary education level (Leiter, 1983; Opdenakker & Van Damme,2001; Resh & Dar, 1992; Strand, 1997; Teddlie & Reynolds, 2000; Tymms, 2001).Some studies report that, if both the school and class levels are specified in the modeland the socioeconomic composition is ‘‘controlled’’, the ‘‘school effect’’ disappearsalmost completely or becomes smaller than the ‘‘class effect’’ (Muijs & Reynolds,2003; Opdenakker & Van Damme, 2000; Opdenakker, Van Damme, De Fraine, VanLandeghem, & Onghena, 2002; Webster & Fisher, 2000). Yet in others, it continuesto be greater (De Jong, Westerhof, & Kruiter, 2004; Luyten & De Jong, 1998).

Briefly, to ignore the ‘‘compositional’’ factors would lead to gross misinterpreta-tion of the school system’s functioning. Still, to establish the existence of the ‘‘com-position’’ effect, it is important to maintain a certain methodological caution. The‘‘composition’’ effect can be an artefact, resulting from the inclusion of poorlymeasuredLevel 1 variables with low validity or from not including some important Level 1covariates (Harker & Tymms, 2004), meaning adequacy and completeness of theindividual attributes in themodel. The ‘‘composition’’ effect has also been considered as‘‘likely to be an artefact of within-class selections’’ by schools for student and familycharacteristics that adversely affect research results (Nash, 2003, p. 442). In reality, ‘‘theresearcher can never be sure about what has been found’’ (Harker & Tymms, 2004, p.195). From here it is possible to infer, from a practical point of view, that it is advisableto include the highest quantity and variety of available measurements that refer tostudent background. This is the adopted focus of this article.

The levels to be specified

Theory and/or the actual structure of the educational system in the country shouldindicate which aggregation levels must be considered in the model. In general terms,any level whose factors could affect the learning process should be included. Van denNoortgate et al. (2005) have shown that the multilevel analysis result depends on thenumber of specified levels in the model. Ignoring a top or intermediate level has aneffect both on the fixed and random parameters and on the corresponding standarderrors. Then, if ‘‘the researcher is interested in a specific level, it is especially

324 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

important that both the lower and the upper level are included’’ (p. 299). In theschool effectiveness paradigm, the focus is obviously on the school, implying thatlevels above and below it must be included as well. However, this condition is notfilled in most studies. None of the multilevel studies referred to by the authorsmentioned above specified an upper school level, meaning that all of them have theschool as the highest level.

Class and school

It is well established that the pupil learns in the classroom. Several studies havereported the importance of class effects (Fraser, Walberg, Welch, & Hattie, 1987;Hextall & Mahony, 1998; Hill & Rowe, 1996; Opdenakker & Van Damme, 2000;Reynolds et al., 1994; Scheerens & Bosker, 1997; Scheerens & Creemers, 1989;Teddlie, 1994). Then, what happens in the classroom actually determines the schooleffectiveness level – ‘‘Effective schools are schools which can achieve effectiveclassroom . . .’’ (Creemers, 1994, p. 201).

Still, there is no consensus on the relative importance of class and school effects.In the Second International Mathematics Study (SIMS) of the InternationalAssociation for the Evaluation of Educational Achievement (IEA), the class effect onthe 2nd-year student achievement noticeably exceeded the school effect (Scheerens,Vermeulen, & Pelgrum, 1989, Table 8.2, p. 794), supporting the hypothesis that ‘‘thecase for ‘effective classrooms’ is somewhat stronger than that for ‘effective schools’’’(p. 798). In his review, Cuttance (1998) concluded that the inter-class/teachervariation reaches 60%, while the inter-school variation oscillates between 8% and19%, which are figures very close to those reported by Rowe, Turner, and Lane(1999) for the student ending high school. With the Trends in InternationalMathematics and Science Study (TIMSS) data for Australia (seventh and eighthgrades), Webster and Fisher (2000) found a class effect (33.9%) superior to theschool effect (7.6%) on mathematics achievement.

On the other hand, in his review, Luyten (2003) concluded that if the classes areboth parallel and with different teachers, the school effect is greater than the ‘‘teachereffect’’ on mathematics but very similar in language, and then, ‘‘the dominance ofteacher effects over school effects is not inevitable’’ (p. 46). Opdenakker and VanDamme (2000) report a clear predominance of the school effect (32.5%) over theclass effect (23.2%) in language but not in mathematics (19.6% and 23.2%,respectively) at the first grade level. During the next year, the school effect is greaterin both subjects, although moderately (Opdenakker et al., 2002). De Jong et al.(2004) found an extreme school effect predominance (30%) over the class effect(10%) in the 1st year of high school in The Netherlands.

In short, while some studies support the hypothesis that classes are moreimportant in determining how pupils perform in schools (Muijs & Reynolds, 2001),the others suggest that the school effect could be equal or stronger than the classeffect, depending on subjects (mathematics or language), education level (primarylevel or middle/high school), and pupil grade level. Therefore, the issue deserves to beinvestigated in different contexts.

Neighborhood

In the last decade, there have been numerous studies about the neighborhoodinfluence on student attainment, most of them finding that students from socially


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

deprived areas will have, on average, lower educational attainment than pupils frommore advantaged neighborhoods (Entwisle, Alexander, & Olson, 1994; Gephart,1997; Mayer, 2002). On the other hand, Gibbons (2002) found that neighborhooddetermines only a small proportion of the variation in individual educationalattainment and that family background plays a larger role. It seems that theoutcomes considered as the educational attainment (years of completed schooling,dropout, graduation, postsecondary schooling), the operational neighborhoodcharacteristics included in the analysis, and the models specifications show a greatvariety. As for the results, estimates of neighborhood effects have varied widely(Ginther, Haveman, & Wolfe, 2000). More recently, some longitudinal studiesusing traditional regression analysis report findings related more specifically tostudent achievement that confirm the neighborhood effect, even after controlling forfamily- and school-related factors (Ainsworth, 2002; Gibbons, 2002; Gordon &Monastiriotis, 2006).

There are some antecedents of multilevel analysis regarding neighborhoodeffects. Bell (2003) investigated the neighborhood effects on the relative educationalprogress (added value) of pupils from age 14 to age 16 in secondary education (meanGCSE score). The study considers two neighborhood (ward) characteristics definedat the school level: a Child Poverty Index and the percentage of very good or betterteaching (as grading by inspectors). No socioeconomic background variable at thepupil level is included. The ward socioeconomic effect is significant, even aftercontrolling for prior achievement. However, the neighborhood is not defined as aspecific level, and, consequentially, there is no estimation of neighborhood effect asbeing different from school effect.

In fact, the main problem with the multilevel analysis of neighborhood effect isthe difficulty in distinguishing school and neighborhood as two different levels. Bothshould be seen as ‘‘compositional’’ effects. The first one is a consequence of‘‘interaction of pupils or parents with peers in the community’’ and the latter comesfrom the ‘‘interaction of pupils with peers in school/class’’ (Gordon & Monastiriotis,2006, p. 5). However, there is an overlap between school and neighborhood, meaningthey are not additively separable. On the contrary, there are substantial interactionsbetween them. In each school, there could be students from different neighborhoods,and vice versa. This data structure is not strictly hierarchical. In the multilevelapproach, it must be analyzed with a cross-classified multilevel model. Theclassification with the greater number of units is specified as a standard hierarchicallevel (random effect), and for the other classification, a dummy variable for each unitis defined (1: if the unit belongs to that unit), whose variances are constrained to beequal (fixed effect) (Goldstein, 1995).

Garner and Raudenbush (1991) adopted this methodology in studying theneighborhood effects on the progress in educational performance that occurs insecondary education from age 11 and 12 to the end of compulsory education. Thevariation between neighborhoods (almost 20%) is substantial, but a majority of it(85%) is explained by prior attainment and family background. Almost 34% of theremaining neighborhood variation is explained by neighborhood deprivation andschool membership. From these results, the authors call attention to the effect ofneighborhoods.

This strategy, however, presents some limitations. First, it is not possible todisentangle and estimate the ‘‘compositional’’ effects of both school and neighbor-hood and to know how much the neighborhood socioeconomic variable would

326 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

explain if the socioeconomic school ‘‘composition’’ were controlled. Second, becausethe school dummy variables are being fit at the student level, they may potentiallycapture part of the neighborhood effect and part of the effect of neighborhooddeprivation (Garner & Raudenbush, 1991). In short, both neighborhood and schoolcannot be considered simultaneously as a standard aggregation level in a multilevelmodel. Then, only one of them has to be chosen. If the basic interest were therelationship between achievement and ‘‘composition’’ characteristics in the differentlevels of the educational system, the schools’ own pupil mixes will have the priority.On the other hand, if only the ‘‘compositional’’ neighborhood characteristic (areamix factor) is specified as a level (random effect) in the model, the estimatedneighborhood effect could be no more than artifacts of non-inclusion of the schoollevel.

The potential ‘‘collinearity’’ magnitude between neighborhood and school effectsdepends on both (a) the operational definition of neighborhood (area scale:extension of aggregated spatial data) and (b) the scale and rigidity of schoolcatchments areas and the degree of schools’ autonomy in selecting pupils. It isreasonable to hypothesize that in certain situations there would be a large overlap ofneighborhood and school ‘‘composition’’. As an example, in Argentina, approxi-mately 75% of the pupils in the sixth and seventh grades of primary schools live nomore than four blocks from the school. In the public schools, these figures may behigher. In a situation such as this, almost the total school ‘‘composition’’ effect couldbe considered simultaneously as ‘‘composition’’ neighborhood effect.

Municipality and state

The immediate upper school level that can be specified is the municipality, withseveral schools belonging to each of them. The municipality effect could reflect notonly its socioeconomic and cultural ‘‘composition’’ incidence but also the action ofeducational and political process factors. As an example of this approach, Caldasand Bankston (1999) reported an analysis with three-level models (student, school,and districts). They included individual and family socioeconomic characteristics,school ‘‘composition’’, and district environmental factors. Unfortunately, they donot present initial variance partition between the three levels (‘‘empty’’ model), nordo they present the final ‘‘residual’’ level distribution for an appropriateinterpretation of the explanatory power of their hierarchical models.

Finally, it makes sense to specify the state as the top level. As in the schooldistrict case, the socioeconomic and cultural inequality between states may explainthe variance in this level, if any. The state is a relevant political jurisdiction. In mostcountries, it provides the funding for and manages the basic education system. InArgentina, states (provinces) not only are responsible for the administration of thebasic education but also provide a significant part of the financial resources to it.Variance at this level could indicate the operation of process factors over whichauthorities have some control.

Only one study (Ramirez, 2007) analyzing the complete educational structure(provinces, municipality, school, classroom, and student) was found. Unfortunately,it does not use multilevel methodology. The one-way analysis of variance (ANOVA)is used to decompose the fourth-grade student achievement distribution in theSIMCE 1999 (Chilean National Assessment System), with these results: 2.9% forRegions/Provinces, 3.5% for communities; 20.8% for school, 4.0% for classes, and


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

68.8% for students within classes. The proportions were adjusted to reflect the effectof each level (e.g., communities) after discounting the effect of the upper-level units(e.g., provinces). Ramirez used ordinary least squares (OLS) regression procedures(with stepwise selection method) in order to identify relevant variables and toestimate their effects. Almost 62% of the differences among schools is explained bysocioeconomic and other external school factors, meaning that the unexplainedschool variance falls to 8% of the total variance.

Method

Data

Data from a standardized mathematics test, a student questionnaire (SQ), and aschool’s principal questionnaire (PQ) gathered during the 2000 sixth-grade PrimarySchool Census are analyzed. The complete dataset was downloaded from http://diniece.me.gov.ar/index.php?m¼1&i¼334 on November 30, 2006. The data are alsoavailable at http://diniece.me.gov.ar/concurso2006/basededatos.html. The evalua-tion was performed at the end of the school year. The questionnaires were self-applied. Additionally, a quality of life indicator at the county level from the 2001Population and Housing Census done by the National Institute of Statistics andCensus is included in the analysis.

Only pupils who took the mathematics test and completed the questionnaire wereconsidered. Classes with fewer than 20 pupils,1 schools with less than two classes,and municipalities with less than two schools were not included in the study. Twostates were excluded because they do not have more than two municipalities. Underthese conditions, the final archive contains 290,988 pupils in 11,140 classes in 4,208schools in 285 municipalities in 22 states. (The state of Buenos Aires is divided intoGreater Buenos Aires [suburbs] and the rest of the state.) It must be noted that thestate level has less than 30 units, the minimum to obtain reliable estimates of the top-level parameters (Snijders & Bosker, 1999; Van den Noortgate et al., 2005). But thislimitation does not affect the conclusions about the main focus – the school level –because an immediate upper school level is specified (municipality), with almost 300units.

Variables

The dependent variable in the analysis is the student achievement (score) on themathematics test constructed by the Minister of Education. The test was criterionreferenced: Each item was tied to the Common Basic Contents (NationalCurriculum).

The independent variables are obtained from both the student and the principalquestionnaires.

(1) As sociodemographic individual student characteristics (Level 1; SQ) areused:(a) the highest educational level of mother or father (from incomplete

primary ¼ 1 to university ¼ 6) (EDUC)2;(b) availability of goods or services at home (17 items) (GOODS);(c) availability of books at home (from none ¼ 1 to more than 100 ¼ 4)

(BOOKS);

328 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

http://diniece.me.gov.ar/index.php?m=1&i=334




http://diniece.me.gov.ar/concurso2006/basededatos.html

(d) physical characteristics of housing (availability of basic amenities:running water, electricity, kitchen, lavatory) (HOUSING);

(e) Occupation density in the household (number of inhabitants per room)(OVERCROWDING);

(f) intensity of child labor (index: daily hours, days weeks, children incomecontributed to home) (CHILD LABOR).

(2) Available ‘‘proxies’’ to prior academic performance of student (Level 1; SQ) are:grade repetition (did not repeat a year ¼ 0; repeated at least one year ¼ 1) and thesum of math and language qualifications at the previous school year (PRIORGRADES). It is expected that these variables will be less significant than theindicators commonly used in ‘‘added value’’ studies (i.e., achievement in a priorstandardized test).

(3) The sociodemographic and prior academic group composition variables arethe average (for measures assumed as intervals) or the proportion (fordummy variables) of each pupil’s individual variables, at each aggregatedlevel. They are given the same acronyms as the individual variables but withthe added ending _c (for class: Level 2), _s (for school: Level 3), or _m (formunicipality: Level 4).

(4) The classroom/school resource measures are constructed from the principal’sanswers to three sets of items (Likert type), referred to:(a) the current physical conditions of school (school building, furniture,

classrooms (in general), library, and bathrooms) (S_RESOURCES);(b) the current conditions of the majority of classrooms (desks, blackboard,

lighting, heating, ventilation, and area/pupils) (C_RESOURCES); and(c) availability and current quality conditions of 12 pedagogic resources at

school (teachers’ books, journals of pedagogic actualization, texts, booksand guides for students, didactic video, projector, retro projector, taperecorder, video, laboratory material) (DIDACTIC). They are includedboth at the school level (original) and at the municipal level (aggregated).

(5) Basic Unsatisfied Needs (BUN) in municipality (Level 4) is the percentage ofpopulation in the home with at least one of the following situations: morethan three persons per room; precarious housing; housing without any kindof toilet, an age 6 to 12 child who is not in school, four or more persons foreach working person, and head of household with less than a third-gradeeducation.

Multilevel analysis strategy

For multilevel analysis models (Aitkin & Longford, 1986; Bryk & Raudenbush,1992; Goldstein, 1995), five levels were specified: pupil, class, school, municipality,and state. All independent pupil variables are centered round the grand mean. Theanalysis is carried out in three stages. First, complete and partial empty models areestimated in order to evaluate the consequence of omitting levels on the results.Second, individual background variables are modeled, starting with the pupil’s prioracademic performance variables and continuing with demographic and socio-economic background variables. Based on an earlier exploration of data, theindicators were ordered by their predictive force (% variance explained andmaximum likelihood test) and introduced in accordance with this, being first themost significant measure. An additional variable is maintained in the model if it is


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

statistically significant. Finally, those ‘‘compositional’’ variables that correspond tothe individual variables maintained in the last models are introduced with the samelogic used with the above individual variables. The analysis starts with class‘‘composition’’ effect and follows sequentially by estimations of school, municipal,and state ‘‘composition’’ effects.

Results

Empty model

The initial division of the achievement variance among the different levels isestimated. In the fixed part, only the overall mean performance (intercept) isestimated without explanatory variables (‘‘empty model’’). The random part consistsof a residual term for each level, indicating the random variation in achievement overeach level. These residuals can be understood as the ‘‘raw’’ effect of each level (Vanden Noortgate et al., 2005).

Besides the complete intercept model (5-level-intercept model: student, class,school, county, and state), five additional models with a different number of levelsbut all containing the school level are fit to the data. The main objective is toevaluate the consequence of omitting one or more levels on the ‘‘raw’’ school effectestimated. Table 1 shows the partitioning (%) of the variance, and in Appendix I theestimates and corresponding standard error of each model are exposed.

In the complete model, the school represents no more than 18% of the totalachievement variation. The two upper levels add 8.6%, while the variation at theclass level is 12.5%. The county and the state seem to be a very important source ofachievement variation. If the state (the highest level) is omitted, both the varianceestimates at the county level and its corresponding standard error increases, whichmay be due to the high standard error and the small number of units at the statelevel. If now the county level is discarded but the state level is maintained, the schoolvariance increases noticeably (from 17.9% to 21.1%) and so does its standard error,but only slightly (Appendix I). The increase of the school ‘‘raw’’ effect is even largerwhen the two upper levels are left out. Now the school level could explain more thanone quarter of the total achievement variance. Then, in order to estimate adequatelythe school effect in primary education of Argentina, it is necessary to specify at leastone of the upper levels, preferably the one with more units and closer to the schoollevel, the municipal level.

Table 1. Distribution (%) of variance components in ‘‘empty’’ models with different specifiedlevels.

RandomLevels

CompleteModel

‘‘Empty’’ models with levels omitted

State County*State þCounty* Class

State þClass

State þCounty* þ Class

State 5.0 – 5.8 – 4.9 – –Municipality 3.6 7.5 – – 3.6 7.6 –School 17.9 18.2 21.1 25.5 23.0 23.3 30.6Class 12.5 12.7 12.5 12.7 – – –Pupil 61.0 61.6 60.6 61.8 68.5 69.2 69.4

Total 100.0 100.0 100.0 100.0 100.0 100.0 100.0

*County ¼ Municipality.

330 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

Omitting the class level has the largest impact. If it is discarded, not only does the‘‘raw’’ school effect increase considerably (from 17.9% to 23%), but also both theinter-pupil (intra-school) variation estimates and its standard error increase notably(from 61% to 68.5%, and from 0.67 to 0.75, respectively). Omitting the class levelproduces an overestimation both of the ‘‘raw’’ school effect and the intra-schoolvariations, meaning an overestimation of the potential effect of the pupil’sbackground on the achievement. Finally, to consider only the school as anaggregated level produces a greatly distorted image of both the relative importanceof the potential factors that could affect achievement distribution and the specific‘‘accountability’’ of each level for the inequalities in the educational system. In thiscase, the magnitude of the potential bias in the school effect estimates seems to beapproximately 71%. In general terms, all these results are consistent with themethodological conclusion by Van den Noortgate et al. (2005) and indicate thatnone of the levels can be neglected.

Individual background variables models

Model A (Table 2) fits a constant plus the two pupil academic background variables.The effects of both measures are statistically significant, even when they arecontrolled by each other. In relation to the complete ‘‘empty’’ model, these variablesnot only explain some 10% of individual variations in math scores but also reduce17.3% the school variance, reflecting some school academic selectivity (someschools, more than others, contained more repeaters and low prior academicperformance pupils). This is not the case with inter-class variance, where the smallreduction (6%) indicates that class academic selectivity, if any, is insubstantial.

Table 2. Estimates and (standard error) in complete ‘‘empty’’ multilevel model andmultilevel models with pupil-level variables.

Multilevel models with pupil-level variables

Academic Socioeconomic (A) þ (B)

Parameter and levels‘‘Empty’’model (A) (B) (C)

(%)‘‘residual’’variance

FIXEDIntercept 57.5 58.5 58.4 59.0PRIORGRADE 3.27* (0.021) 3.11* (0.021)REPEATER 73.94* (0.085) 73.41* (0.084)GOODS 0.11* (0.013) 0.05* (0.013)HOUSING 0.84* (0.033) 0.59* (0.031)OVERCROWDING 70.05* (0.002) 0.03* (0.002)CHILD LABOR 0.35* (0.006) 0.29* (0.006)BOOKS 1.62* (0.036) 1.14* (0.034)EDUCATION 0.17 (0.023) 0.05 (0.022)

RANDOMState 20.4 (7.36) 16.9 (6.17) 12.1 (4.490) 11.9 (4.436) 2.9Municipality 14.8 (2.37) 13.1 (2.08) 10.0 (1.671) 10.2 (1.666) 2.5School 73.5 (2.24) 60.8 (1.92) 52.1 (1.739) 49.2 (1.645) 12.0Class 51.5 (1.04) 48.2 (0.97) 49.5 (1.000) 47.3 (0.948) 11.5Pupil 250.4 (0.67) 225.2 (0.60) 243.7 (0.651) 221.8 (0.593) 54.0

Total variance estimated 410.6 364.2 367.4 340.4 82.9

*Significant at the .001 level.


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

In Model B, the individual and family demographic and socioeconomiccharacteristics are introduced jointly. Initial processing not shown here indicatesthat all variables individually have substantial effects. Now, when these variables areincluded simultaneously in a single model, all except ‘‘the highest educational level ofmother or father’’ maintain significance. The introduction of these variablesproduces a very small decrease in inter-pupil variation (2.7%) but a strong variationsituated at the school level (almost 31%), indicating a marked socioeconomicselectivity, higher than the academic one detected before. Therefore, in Argentina thedifferences between mean school achievements are closely related to the schoolsocioeconomic pupil recruitment. At the municipal level, a significant decrease ofvariance is also observed, advancing a probably strong socioeconomic compositioneffect at this level. Finally, as in the case of academic background, no significanteffect is detected at the class level, reflecting great socioeconomic class homogeneityinside the school.

Finally, in Model C, both subsets of variables at the pupil level are modeledjointly. All variables maintain significance, but some changes can be observed. All ofthe socioeconomic background coefficients decrease, while the two corresponding tothe prior academic performance measures remain practically the same. This isconsistent with the idea that part of the influence of socioeconomic background willoperate through prior academic performance of the pupil. But, at the same time, italso means that those variables do affect the educational progress in mathematicsand, therefore, cannot be omitted.

This model explains 17% of the total variance of mathematic scores. But only6.5% of the variance that remained at Model A has been explained as a consequenceof adding pupil background characteristics to that model. The main changes can beobserved at the school and upper levels. At the student and class levels, thesevariables do not contribute substantially to the explanation. In relation to the‘‘empty’’ model, 11.4% of the inter-student variance has been explained, which issmaller than the 19.6% estimates for Argentina in a prior ‘‘value added’’ study(Cervini, 2006, Table 1). This distance is due to the different prior achievementmeasure used in both studies. As expected, prior grades and repetition of schoolyear, used in this study, are not as good predictors as prior achievement in astandardized test.

Class and school ‘‘composition’’ effects

The results of compositional analysis are shown in Table 3. The new coefficients andthe corresponding standard error of individual pupil variables are not displayedbecause they are not of interest in the following analysis.

As the first step, the class academic and socioeconomic compositional variablesare included (Model D). All indicators are significant but PRIORGRADE. On theother hand, the proportion of repeaters in class (REPEATER) is a strong predictor.The coefficient and its sign indicate that pupil achievement will be lower as theproportion of repeaters in his/her class becomes higher, independently of their otherindividual characteristics. In the same line, the coefficients and signs of thesocioeconomic variables indicate that when a pupil belongs to a class with asocioeconomically advantaged composition, his/her mathematics achievement willbe positively affected, independently of his/her individual characteristics.

332 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

In Model E, the measures included are constructed through school aggregationof individual characteristics. Exploratory initial processing not shown heredemonstrates, as expected, the existence of a strong collineality between class andschool composition measures. As a consequence, one of the two composition modelsmust be chosen. To do this, the above Model D (class composition) can be comparedto Model E (school composition). The difference in the likelihood ratio statisticbetween both models is highly significant and suggests that Model E is the bestadjusted. In fact, all Model E coefficients except GOOD’s are higher than those ofModel D. Nevertheless, it has to be noted that the two models explain significantvariance between schools but not between classes, which indicates a clearhomogeneity between them inside the school. This model contains the classrepeaters composition instead of school repeaters composition because the formerappeared to be a better predictor.

All indicators are significant, and, jointly, they reveal the importance of schoolsocioeconomic composition in explaining the mean school achievement variation(intercept), given that the corresponding individual pupil and family characteristicshave been controlled for. Wealthier socioeconomic school composition predictshigher achievement for pupils belonging to the school, independently of individualpupil characteristics.

By modeling socioeconomic school composition measures further, 46.5% of themean school achievement is explained. In relation to the variance componentestimates at the empty model, Model E not only explains 64.2% of the inter-school(or intra-municipality) variance but also explains 78.4% of the inter-municipality (orintra-state) variance. Thus, differences between municipalities seem to be associatedprincipally with the fact that some municipalities have schools with larger

Table 3. Estimates and (standard error) of multilevel models with class, school, andmunicipality ‘‘compositional’’ variables.

ParameterClass

SchoolCounty

(D) (E) (F) (G)

FIXEDPRIOR GRADE_c 0.54 (0.228) – – –REPEATER_c 73.48* (0.929) 76.88* (0.808) 76.86* (0.808) 76.90* (0.808)GOODS_c/_s 0.94* (0.116) 0.55* (0.198) 0.46* (0.197) 0.42* (0.197)HOUSING_c/_s 0.89* (0.301) 1.30* (0.48) 1.38* (0.48) 1.35* (0.49)OVERCROWDING_c/_s 70.12* (0.017) 70.15* (0.028) 70.13* (0.028) 70.14* (0.028)CHILD LABOR_c/_s 0.45* (0.062) 0.77* (0.116) 0.74* (0.116) 0.73* (0.116)BOOKS_c/_s 3.26* (0.367) 4.48* (0.653) 4.43* (0.652) 4.48* (0.652)DIDACTI_s 0.06* (0.018) 0.06* (0.018)DIDACTI_m 0.16* (0.068)NBI_m 0.002 (0.003)

RANDOMState 7.76 (2.771) 8.63 (3.007) 8.19 (2.865) 8.47 (2.929)Municipality 3.75 (0.763) 3.15 (0.686) 3.00 (0.666) 2.61 (0.620)School 26.15 (1.14) 26.30 (1.128) 26.22 (1.126) 26.26 (1.126)Class 47.58 (0.951) 46.58 (0.934) 46.58 (0.934) 46.58 (0.934)Pupil 221.78 (0.593) 221.78 (0.593) 221.78 (0.593) 221.78 (0.593)

Total variance estimated 306.97 306.44 305.77 305.69Deviance test 2422067 2421908 2421896 2421887

*Significant at the .001 level.


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

percentages of poor families or repeater pupils, or both. Something similar can besaid with respect to the inter-state variance, which is reduced in some to 58%.

Some exploratory processing revealed that, if school composition variables inModel E are substituted by the homologous municipal composition variables, theinter-municipal variance falls to 0.839 (standard error ¼ 0.4510) and becomes notsignificant. But it also implies a clear trade-off against the inter-school variancebecause it maintains a similar magnitude as in Model C (without compositionvariables). As a consequence, no composition municipal variable can be used in theanalysis.

The remaining unexplained variation in each level of the model continues to bestatistically significant. In order to explain the variation, the effects of the threeschool resource indicators (S_RESOURCES; C_RESOURCES; DIDACTIC) wereanalyzed. Initial exploratory processing showed that only the availability/quality ofconditions of pedagogic resources at school (DIDACTIC) is statically significant atthe 1% level. Model F shows all the coefficients recalculated. As one can see, a veryslight change in the variance at each level is registered. The same occurs with thecoefficients of compositional variables. Thus, the didactic resources at school, asevaluated by the school principal, do not reduce an important proportion of theunexplained variances, at least from the practical point of view, and neither reducesthe socioeconomic compositional effects; therefore, they do not mitigate thoseinfluences.

Finally, didactic resources and socioeconomic deprivation at the municipal levelare considered. Both variables had a highly significant effect before being controlledfor school composition variables. In Model G, however, only DIDACTI_mmaintains statistical significance. Thus, educational resources at both school andmunicipal levels explain 17% of the exiguous inter-municipal variance leftunexplained by the school composition variables (Model E). Nonetheless, it wasnot possible to find an explanation for the very small residual variance at this level byusing the social deprivation index.

Conclusion

The main focus of this article was to evaluate the school effect hypothesis for theprimary education in Argentina. To do so, multilevel models with class, school,municipality, and states levels were fit to the mathematics achievement of sixth-gradepupils of primary education in Argentina. There does not exist any antecedent inArgentina, nor very possibly in Latin American countries, of the multilevel analyseswith two levels above the school (municipality, state/province), which are bothparticularly relevant to the presented study.

Several important conclusions can be drawn from the results. Perhaps the mostprominent is that all levels of the educational system have an important effect onmathematics achievement. None of them can be omitted without distorting thecorrect image of the educational system. As a majority of the studies reveal, inter-pupil variation due to individual characteristics is the most extensive of them (60%).This figure, however, depends on class-level inclusion in the models. If omitted, thisvariation increases to nearly 70%. It means that an important proportion of within-school variance indicates differences between teachers – potentially due toeducational factors – instead of differences between individual students – due toextra-school factors. Similarly, the inter-school variation is strongly affected by the

334 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

levels specified. If both an upper (municipality) level and an under (class) level areincluded, the variance estimate at the school level is much less (18%) than if they arenot included (31%). Then, the magnitude of the ‘‘school effect’’ could be overestimatedin the majority of traditional SER studies that have not included lower levels of theschool. Additionally, the data confirm that ‘‘ignoring a top level (county, states)causes an overestimation of the variance belonging to the highest level considered’’(Opdenakker & Van Damme, 2000, p. 107), in our case, the school level.

No five-level analysis (provinces, municipality, school, classroom, and student)was found by the author in the review. Since the number of specified levels in themodels affects the estimations of each level variation, the results obtained here arenot directly comparable to those found in the reviewed literature. Only one studydeveloped a five-layer model (Ramirez, 2003) but it did not use strictly the multilevelanalysis, and, as a consequence, whatever comparative exercise made with its resultscould be invalid.

Individual academic and socioeconomic pupil background explains not onlyintra-class (inter-pupil) variation but, more importantly, inter-school variation,denoting strong academic and socioeconomic school selectivity. The pupil prioracademic performance measures used in this study (prior grades in the previousschool period and repetition of school year) are not as strong predictors as priorachievement in a standardized test. However, when the focus is on the pupil upperlevels, it seems to be a proper proxy, with the evident cost advantage for measuring‘‘value added’’.

Strong ‘‘contextual’’ effects exist. The academic and socioeconomic ‘‘composi-tion’’ effects are quite significant, even after controlling for relevant pupilcharacteristics. The class composition variables explain a high proportion of schoolmean achievement variation but little of the class variation, reflecting a pronouncedinter-class composition homogeneity inside the school. The models, includingacademic and socioeconomic school composition, are by far the most powerful. Theyexplain 64.2% of the inter-school variance and 78.4% of the inter-municipalityvariance. Differences between schools and municipalities seem to be associatedmainly with strong variations in the proportion of socially and academicallydisadvantaged pupils in schools and in the proportion of schools with largerpercentages of poor families or repeater pupils, or both, inside the municipalities.

Analysis revealed that didactic resources both in schools and in municipalities arethe only of three educational resource measures whose influence is significant.However, it explains only 17% of the small inter-municipal variance left unexplainedby the school composition variables. It has a practically imperceptible effect on theinter-school variation.

Finally, no effect of municipal social deprivation index was detected. In fact,socioeconomic ‘‘composition’’ measures are better predictors when contrasted withsuch indicators.

Even though the school ‘‘raw’’ effect (18%) is more important than the class‘‘raw’’ effect (12.5%), after controlling for pupil characteristics and schoolcomposition, the class became the superior level with a larger proportion (15%) ofthe total unexplained variance, meaning the extra-school factors do not explain classvariation. This finding suggests that looking into the potential educational classfactors is a prospective line of research in primary education, which does not intendto deny the important factors that could explain the residual inter-school variation(8.6% from the total unexplained variance).


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

From a policy point of view, the relevance of the informed empirical findings isobvious. Educational programs and policies must take into account the hetero-geneous and contextual effects of extra-school factors at risk of compromising theirefficacy. Still, other aspects must be considered. Neither differential or interactioneffects nor educational process effects were investigated in this article. They will alsobe part of future analysis, in order to deepen the understanding of the primaryeducation system of Argentina.

Notes

1. This condition has to do with the whole project at the National University of Quilmes.Anyway, the main results do not change much when classes with few pupils are alsoincluded.

2. Processing not shown here indicates that EDUC is the most significant predictor fromother optional measures available: educational level of mother and father and averagelevel of parent’s education.

Notes on contributor

Ruben Alberto Cervini received a B.A. in Political and Social Science from the UniversidadNacional de Cuyo in Argentina, an M.A. in Political Science from the FLACSO (LatinAmerican Faculty of Social Studies) in Chile, and an M.A. in Educational Management fromthe joint programme of Universidad del Valle and OAS. He has been a UNESCO consultanton several projects as well as a professor at Chilean, Colombian, and Mexican universities. Hisresearch focus is on student achievement factors. He has been at the Universidad Nacional deQuilmes (National University of Quilmes) since 1995, addressing quality and equity in primaryand secondary education in that country. He has written several articles on these topics.

References

Ainsworth, J.W. (2002). Why does it take a village? The mediation of neighborhood effects oneducational achievement. Social Forces, 1, 117–152.

Aitkin, M., & Longford, N. (1986). Statistical modeling issues in school effectiveness. Journalof the Royal Statistical Society A, 149, 1–42.

Angus, L. (1993). The sociology of school effectiveness. British Journal of Sociology ofEducation, 14, 333–345.

Bell, J. (2003). Beyond the school gates: The influence of school neighborhood on the relativeprogress of pupils. Oxford Review of Education, 29, 485–502.

Bosker, R.J., & Witziers, B. (1996, April). The magnitude of school effects or does it reallymatter which school a student attends? Paper presented at the Annual Meeting of theAmerican Educational Research Association, New York.

Bourdieu, P. (1984). Distinction: A social critique of the judgement of taste (R. Nice, Trans.).London: Routledge.

Bourdieu, P. (with Passeron, J.-C.). (1977). Reproduction in education, society and culture (R.Nice, Trans.). London: Sage.

Bryk, A., & Raudenbush, S. (1992). Hierarchical linear models for social and behavioralresearch: Applications and data analysis methods. Newbury Park, CA: Sage.

Caldas, S., & Bankston, C. (1997). Effect of school population socioeconomic status onindividual academic achievement. Journal of Educational Research, 90, 269–277.

Caldas, S., & Bankston, C. (1999). Multilevel examination of student, school, and district-leveleffects on academic achievement. The Journal of Educational Research, 93, 91–100.

Cervini, R. (2005). The relationship between school composition, school process andmathematics achievement in secondary education in Argentina. International Review ofEducation, 51, 173–200.

Cervini, R. (2006). Progreso de Aprendizaje en la Educacion Secundaria Basica de Argentina:Un analisis multinivel de valor agregado [Progress in learning in basic secondaryeducation in Argentina: A multilevel analysis of added value]. Revista ElectronicaIberoamericana sobre Calidad, Eficacia y Cambio en Educacion, 4, 53–83.

336 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

Coe, R., & Taylor, C. (1998). School effectiveness research: Criticism and recommendations.Oxford Review of Education, 4, 421–438.

Creemers, B. (1994). Effective instruction: An empirical basis for a theory of educationaleffectiveness. In D. Reynolds, B. Creemers, P.S. Nesselrodt, E.C. Schaffer, S. Stringfield, &C. Teddlie (Eds.), Advances in school effectiveness research and practice (pp. 189–203).Oxford: Pergamon.

Creemers, B., & Kyriakides, L. (2006). Critical analysis of the current approaches to modelingeducational effectiveness: The importance of establishing a dynamic model. SchoolEffectiveness and School Improvement, 17, 347–366.

Cuttance, P. (1998). Quality assurance reviews as a catalyst for school improvement inAustralia. In A. Hargreaves, A. Lieberman, M. Fullan, & D. Hopkins (Eds.), Internationalhandbook of educational change (Part Two, pp. 1135–1162) Dordrecht, The Netherlands:Kluwer Publishers.

D’Agostino, J.V. (2000). Instructional and school effects on students’ longitudinal reading andmathematics achievements. School Effectiveness and School Improvement, 11, 197–235.

De Jong, R., Westerhof, K.J., & Kruiter, J.H. (2004). Empirical evidence of a comprehensivemodel of school effectiveness: A multilevel study in Mathematics in the 1st year of juniorgeneral education in TheNetherlands.School Effectiveness and School Improvement, 15, 3–31.

Entwisle, D., Alexander, K., & Olson, L. (1994). The gender gap in math: Its possible originsin neighborhood effects. American Sociological Review, 59, 822–838.

Fitz-Gibbon, C.T. (1996). Monitoring education: Indicators, quality and effectiveness. London:Cassell.

Fraser, B.J., Walberg, H.J., Welch, W.W., & Hattie, J.A. (1987). Syntheses of educationalproductivity research. International Journal of Educational Research, 11, 73–145.

Garner, C., & Raudenbush, S. (1991). Neighborhood effects on educational attainment: Amultilevel analysis. Sociology of Education, 64, 251–262.

Gephart, M. (1997). Neighborhoods and communities as context for development. In J.Brooks-Gunn, G.J. Duncan, & J.L. Aber (Eds.), Neighborhood poverty: Context andconsequences for children (pp. 1–43). New York: Russel Sage Foundation.

Gerwitz, S. (1998). Can all schools be successful? An exploration of determinants of school‘‘success’’. Oxford Review of Education, 24, 439–457.

Gibbons, S. (2002). Neighborhood effects on educational achievement: Evidence from the Censusand National Child Development Study (CEEDWorking paper no. 18). London: Centre forthe Economics of Education.

Gibson, A., & Asthana, S. (1998). School performance, school effectiveness and the 1997White Paper. Oxford Review of Education, 24, 195–210.

Ginther, D., Haveman, R., & Wolfe, B. (2000). Neighborhood attributes as determinant ofchildren’s outcomes: How robust are the relationships? The Journal of Human Resources,35, 603–642.

Goldstein, H. (1995). Multilevel statistical models (2nd ed.). London: Edward Arnold.Goldstein, H. (1998). A response to Gibson and Asthana. Oxford Review of Education, 24,

521–523.Gordon, I., & Monastiriotis, V. (2006). Education, location, education: A spatial analysis of

English secondary school public examination results (Research Papers in Environmentaland Spatial Analysis No. 116). London: London School of Economics.

Harker, R., & Tymms, P. (2004). The effects of student composition on school outcomes.School Effectiveness and School Improvement, 15, 177–199.

Hatcher, R. (1998). Social justice and the politics of school effectiveness and schoolimprovement. Race, Ethnicity and Education, 1, 267–289.

Hauser, R.M. (1970). Context and consex: A cautionary tale. American Journal of Sociology,75, 645–664.

Hextall, I., & Mahony, P. (1998). Effective teacher, effective schools. London: Biddles.Hill, P.W., & Rowe, K.L. (1998). Modelling student progress in studies of educational

effectiveness. School Effectiveness and School Improvement, 9, 310–333.Hox, J. (2002). Multilevel analysis. Techniques and applications. Mahwah, NJ: Lawrence

Erlbaum Associates.Knuver, A.W.M., & Brandsma, H.P. (1993). Cognitive and affective outcomes in school

effectiveness research. School Effectiveness and School Improvement, 4, 189–204.


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

Kyriakides, L. (2004). Differential school effectiveness in relation to sex and social class: Someimplications for policy evaluation. Educational Research and Evaluation, 10, 141–161.

Leiter, J. (1983). Classroom composition and achievement gains. Sociology of Education, 56,126–132.

Luyten, H. (2003). The size of school effects compared to teacher effects: An overview of theresearch literature. School Effectiveness and School Improvement, 14, 31–51.

Luyten, H., & De Jong, R. (1998). Parallel classes: Differences and similarities. Teacher effectsand school effects in secondary schools. School Effectiveness and School Improvement, 9,437–473.

Luyten, H., Visscher, A., & Witziers, B. (2005). School effectiveness research: From a reviewof the criticism to recommendations for further development. School Effectiveness andSchool Improvement, 16, 249–279.

Mayer, S. (2002). How economic segregation affects children’s educational attainment. SocialForces, 8, 153–176.

McCaffrey, D.F., Koretz, D., Lockwood, J.R., & Hamilton, L. (2003). Evaluating value-addedmodels for teacher accountability (Series: Monographs, Document Number: MG-158-EDU). Santa Monica, CA: RAND Corporation.

Mortimore, P., Sammons, P., Stoll, L., Lewis, D., & Ecob, R. (1988). School matters: Thejunior years. London: Open Books.

Muijs, D., & Reynolds, D. (2001). Effective teaching: Evidence and practice. London: PaulChapman.

Muijs, D., & Reynolds, D. (2003). Student background and teacher effects on achievement andattainment in mathematics: A longitudinal study. Educational Research and Evaluation, 9,289–314.

Nash, R. (2003). Is the school composition effect real? A discussion with evidence from theUK PISA data. School Effectiveness and School Improvement, 14, 441–457.

Nuttall, D., Goldstein, H., Prosser, R., & Rasbash, J. (1989). Differential school effectiveness.International Journal of Educational Research, 13, 769–776.

Opdenakker, M., & Van Damme, J. (2000). The importance of identifying levels in multilevelanalysis: An illustration of the effects of ignoring the top or intermediate levels in schooleffectiveness research. School Effectiveness and School Improvement, 11, 103–130.

Opdenakker, M., & Van Damme, J. (2001). Relationship between school composition andcharacteristics of school process and their effect on mathematics achievement. BritishEducational Research Journal, 27, 407–432.

Opdenakker, M., Van Damme, J., De Fraine, B., Van Landeghem, G., & Onghena, P. (2002).The effect of schools and classes on mathematics achievement. School Effectiveness andSchool Improvement, 13, 399–427.

Phillips, M. (1997). What make schools effective? A comparison of the relationships ofcommunitarian climate and academic climate to mathematics achievement. AmericanEducational Research Journal, 34, 633–662.

Ramirez, M.J. (2003). The distribution of mathematics knowledge among Chilean fourthgraders and related explanatory factors. EducarChile, Retrieved May 2003, from http://200.68.0.6/medios/20030604142903.pdf.

Ramirez, M. (2007). Diferencias dentro de las salas de clases: Distribucion del rendimiento enMatematicas [Differences inside classrooms: the distribution of math scores]. EstudiosPolıticos, 106(5–22). Retrieved June 2008, from http://www.cepchile.cl/dms/lang_2/doc_3917.html.

Resh, N., & Dar, Y. (1992). Learning segregation in junior high schools in Israel: Causes andconsequences. School Effectiveness and School Improvement, 3, 272–292.

Reynolds, D., Teddlie, C., Creemers, B., Chen, Y., Dundas, B., Green, B., et al. (1994). Schooleffectiveness research: A review of international literature. In D. Reynolds, B. Creemers,P.S. Nesselrodt, E.C. Schaffer, S. Stringfield, & C. Teddlie (Eds.), Advances in schooleffectiveness research and practice (pp. 25–51). Oxford, UK: Pergamon.

Rowe, K.J., Turner, R., & Lane, K. (1999, November). The ‘‘myth’’ of school effectiveness:Locating and estimating the magnitudes of major sources of variation in students’ Year 12achievements within and between schools over five years. Paper presented at the 1999AARE-NZARE Joint Conference of the Australian and New Zealand Association forResearch in Education, Melbourne Convention Centre, Melbourne, Australia.

338 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

http://200.68.0.6/medios/20030604142903.pdf

http://200.68.0.6/medios/20030604142903.pdf

http://www.cepchile.cl/dms/lang_2/doc_3917.html

http://www.cepchile.cl/dms/lang_2/doc_3917.html

Rutter, M., Maughan, B., Mortimore, P., & Ouston, J. (1979). Fifteen thousand hours.London: Open Books.

Sammons, P., Thomas, S., & Mortimore, P. (1997). Forging links: Effective schools andeffective departments. London: Paul Chapman.

Scheerens, J. (1994). Conceptual framework on school effectiveness: Rational control versuschoice. In D. Reynolds, B. Creemers, P.S. Nesselrodt, E.C. Schaffer, S. Stringfield, & C.Teddlie (Eds.), Advances in school effectiveness research and practice (pp. 207–215).Oxford, UK: Pergamon.

Scheerens, J., & Bosker, R. (1997). The foundations of educational effectiveness. Oxford, UK:Pergamon.

Scheerens, J., & Creemers, B. (1989). Conceptualizing school effectiveness. InternationalJournal of Educational Research, 13, 691–706.

Scheerens, J., Vermeulen, C.J.A.J., & Pelgrum, W.J. (1989). Generalizability of instructionaland school effectiveness indicators across nations. International Journal of EducationalResearch, 13, 789–799.

Slee, R., & Weiner, G. (with Tomlinson, S.) (Eds.). (1998). School effectiveness for whom?London: Falmer Press.

Snijders, T.A.B., & Bosker, R.J. (1999). Multilevel analysis. An introduction to basic andadvanced multilevel modeling. London: Sage.

Stone, J.E. (2003). Value-added assessment: An accountability revolution. Retrieved September4, 2007, from http://www.edexcellence.net/better/tchrs/16.htm.

Strand, S. (1997). Pupil progress during Key Stage I: A value added analysis of school effects.British Educational Research Journal, 23, 471–487.

Stringfield, S. (1994). The analysis of large data bases in school effectiveness research. In D.Reynolds, B. Creemers, P.S. Nesselrodt, E.C. Schaffer, S. Stringfield, & C. Teddlie (Eds.),Advances in school effectiveness research and practice (pp. 55–72). Oxford, UK: Pergamon.

Stufflebeam, D., & Shinkfield, A. (1985). Systematic evaluation. Hingham, MA: KluwerAcademic Publishers.

Teddlie, C. (1994). The study of context in school effects research: History, method, results,and theoretical implications. In D. Reynolds, B. Creemers, P.S. Nesselrodt, E.C. Schaffer,S. Stringfield, & C. Teddlie (Eds.), Advances in school effectiveness research and practice(pp. 85–110). Oxford: Pergamon.

Teddlie, C., & Reynolds, D. (2000). International handbook of school effectiveness research.London: Falmer Press.

Thomas, S. (2001). Dimensions of secondary school effectiveness: Comparative analysesacross regions. School Effectiveness and School Improvement, 12, 285–322.

Thomas, S., & Mortimore, P. (1996). Comparison of value added models for secondary schooleffectiveness. Research Papers in Education, 11, 5–33.

Thrupp, M. (2001a). Recent school effectiveness counter-critiques: Problems and possibilities.British Educational Research Journal, 27, 443–457.

Thrupp, M. (2001b). Sociological and political concerns about school effectiveness research:Time for a new research agenda. School Effectiveness and School Improvement, 12, 7–40.

Tymms, P. (2001). A test of the big fish in a little pond hypothesis: An investigation into thefeeling of seven-year-old pupils in school. School Effectiveness and School Improvement, 12,161–181.

Van den Noortgate, W., Opdenakker, M., & Onghena, P. (2005). The effects of ignoring alevel in multilevel analysis. School Effectiveness and School Improvement, 16, 281–303.

Veenstra, R., & Kuyper, H. (2004). Effective students and families: The importance ofindividual characteristics for achievement in High School. Educational Research andEvaluation, 10, 41–70.

Webster, B.J., & Fisher, D.L. (2000). Accounting for variation in science and mathematicsachievement: A multilevel analysis of Australian data Third International Mathematicsand Science Study (TIMMSS). School Effectiveness and School Improvement, 11, 339–360.


Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

http://www.edexcellence.net/better/tchrs/16.htm

Appendix

1.

Estim

ationofvariance

components

and(standard

error)

of‘‘em

pty’’modelswithdifferentspecified

levels.

Random

Levels

Complete

Model

‘‘Empty’’modelswithlevelsomitted

State

County

Stateþ

County

Class

Stateþ

Class

Stateþ

Countyþ

Class

State

20.4

(7.36)

–24.1

(7.75)

–20.4

(7.39)

––

Municipality

14.8

(2.37)

30.6

(3.78)

––

14.9

(2.39)

30.9

(3.83)

–School

73.5

(2.24)

73.8

(2.25)

87.1

(2.48)

103.1

(2.82)

94.7

(2.23)

95.0

(2.24)

124.2

(2.81)

Class

51.5

(1.04)

51.5

(1.04)

51.5

(1.04)

51.5

(1.04)

––

–Pupil

250.4

(0.67)

250.4

(0.67)

250.4

(0.67)

250.4

(0.67)

282.3

(0.75)

282.3

(0.75)

282.3

(0.75)

Totalvariance

estimated

410.6

406.3

413.1

405.0

412.3

408.2

406.5

340 R.A. Cervini

Dow

nloa

ded

by [

Dic

le U

nive

rsity

] at

08:

29 0

7 N

ovem

ber

2014

class, school, municipal, and state effects on mathematics achievement in argentina: a multilevel...

Documents