1

The impact of national standardized literacy and numeracy testing on children andteachingstaffinremoteAustralianIndigenouscommunities.SusyMacqueenAustralianNationalUniversity,AustraliaUteKnochUniversityofMelbourne,AustraliaGillianWigglesworthUniversityofMelbourne,AustraliaRachelNordlingerUniversityofMelbourne,AustraliaRuthSingerUniversityofMelbourne,AustraliaTimMcNamaraUniversityofMelbourne,AustraliaRhiannaBrickleUniversityofMelbourne,Australia

AbstractAll educational testing is intended to have consequences, which are assumed to bebeneficial,buttestsmayalsohaveunintended,negativeconsequences(Messick,1989).Theissue is particularly important in the case of large-scale standardised tests, such asAustralia’sNationalAssessmentProgram–LiteracyandNumeracy(NAPLAN),theintendedbenefits ofwhich are increased accountability and improved educational outcomes. TheNAPLAN purpose is comparable to that of other state and national ‘core skills’ testingprograms which evaluate cross-sections of populations in order to compare resultsbetweenpopulationsub-groupings.Suchcomparisonsunderpin‘accountability’intheeraof population-level testing.This study investigates the impact ofNAPLAN testingononepopulation grouping that is prominent in the NAPLAN results comparisons and publicreporting: children in remote Indigenous communities. A series of interviews withprincipals and teachers documents informants’ first-hand experiences of the use andeffectsofNAPLANinschools.Intheviewsofmostparticipants,thelanguageandcontentofthe test instruments, the nature of the test engagement and the test washback havenegativeimpactsonstudentsandstaff,withlittlebenefitintermsoftheusefulnessofthetestdata.TheprimaryissueisthefactthatmeaningfulparticipationinthetestsdependscriticallyonproficiencyinStandardAustralianEnglish(SAE)asafirstlanguage.Thisstudycontributes to the broader discussion of how reform-targeted standardised testing fornational populations affects sub-groups who are not treated equitably by the testinstrument or reporting for accountability purposes. It highlights a conflict betweenconsequential validity and the notion of accountability which drives reform-targetedtesting.KeywordsTest impact, literacy testing, numeracy testing,NAPLAN, accountability, global educationreformmovement,remoteIndigenouscommunities,commensuration

2

Introduction

Educationalreformmovementsarisein‘standardlanguagecultures’,wherealanguagevarietyhastaken

holdeconomicallyandisimposedthroughinstitutionalmeans(Milroy,2001,2012).Arguably,thereisno

morepowerfulmeansofimposingandmaintainingastandardvarietythanstandardizedliteracytesting,

a phenomenonwhich has become increasinglywidespread to the extent that comparisons using such

testswithinandbetweenjurisdictionsamountto‘globalpanopticism’(Lingard,Martino,&Rezai-Rashti,

2013,p.539).Atthesametime,studieshavedocumenteddetrimentaleffectsofstandardisedtestingon

school curricula and language minority students (see, for example, David, 2011; Dulfer, Polesel, &

Turnbull,2012;Mathison,2003;Menken,2008).Suchtestsareparticularlyproblematicforassessingthe

academic achievement of childrenwho are not proficient in the standard variety (Genishi&Brainard,

1995;Gutierrez-Clellen,1996).Thepresentpaperisacasestudyintheissuesrepresentedbytheseglobal

developmentsandpractices.ItdealswiththeimpactofnationalstandardizedtestinginAustraliaonthe

education of Indigenous children in remote communities where the first language of the children is

generallynotEnglishandwhereEnglishisfirstencounteredonentrytoschool.1

NationalliteracyandnumeracytestinginAustralia

National standardized literacy and numeracy testing in Australia, known as the National Assessment

Program–Literacy and Numeracy (NAPLAN), was introduced in 2008. The testing program replaced

various State-based standardized tests and was introduced ahead of the new national curriculum.

NAPLAN is designed to systematically measure andmonitor the English literacy and numeracy of all

children in years 3, 5, 7 and 9. 2 The test development and review process, accessible

athttps://nap.edu.au/about/test-development, includes details of the test and item development and

reviewprocess,includingthetriallingwhichisconductedwithrepresentativestudentsamplesacrossall

statesandterritories.Thetestsarealsoreviewedbyanexpertpanel,beforeasecondnationalsampleof

studentstakespartinanadditional‘equatingtrial’whichiscarriedoutpriortothemaintesttoensure

thattheresultscanbereportedonthesameassessmentscale.

Deliveredannuallyovera three-dayperiod, the testsassess four ‘domains’:numeracy,writing,

reading, and language conventions.3 The numeracy test covers numerical and algebraic calculations,

measurement and geometry, and statistics and probability. Thewriting test requires one composition

whichiseitherpersuasiveornarrativewriting.Thereadingtestincludesdifferenttexttypesandsamples

skills ranging from finding directly stated information to identifying the tone of an argument. The

languageconventions test targetsspelling,grammarandpunctuation.Besides thewrittencomposition,

most items across the tests aremultiple choice with some constructed response items, e.g. write the

correct spelling. Detailed descriptions of the tests, and sample papers, can be found at

https://www.nap.edu.au/naplan/the-tests.

1Theterm'Indigenous'referstoAboriginalandTorresStraitIslanderpeoples,theoriginalinhabitantsoftheAustraliancontinent.ItisthetermusedinofficialNAPLAN-relateddocuments.2Correspondingagesareapproximately8,10,12and14yearsold.3https://www.nap.edu.au/naplan(Accessed8December,2016)

3

The rationale for the development of NAPLAN arose from a general series of reforms in the

2000swhich increasingly nationalised curriculum and assessment (for political context see Lingard&

Sellar,2013).Theprimaryimpetusforthedevelopmentofanationalstandardizedtestingprogramwasa

declarationbytheMinistersofEducation in2008which inscribednational testingandanemphasison

literacy and numeracy (MCEETYA, 2008) as keymeans to achieving the broad goals of 1) ‘Australian

schooling promotes equity and excellence’ and 2) ‘all young Australians become successful learners,

confidentandcreativeindividuals,andactiveandinformedcitizens’(n.p.).Prominentinthedeclaration

was the intention to improve outcomes for Indigenous youth (to ‘close the gap’) and the provision of

targetedsupportforthispurpose.Itwasalsoexplicitaboutthegatheringof‘goodqualitydata’aboutthe

performanceofschoolstoenablecomparison,andpublicreportingfor‘strengtheningtransparencyand

accountability’ (n.p.). According to theAustralianCurriculumandTestingAuthority (ACARA), the body

responsibleforNAPLANdevelopment,administrationandreporting,thetestsaredesignedto‘determine

whetherornotyoungAustraliansaremeetingimportanteducationaloutcomes’(ACARAwebsite,2016).

ACARA emphasises ‘two benefits’ of the NAPLAN tests: 1) to ‘help drive improvements in student

outcomes’and2)to‘provideincreasedaccountabilityforthecommunity’(ACARAwebsite,2016).4

Inrelationtothefirstbenefit,‘drivingimprovements’,ACARAstates:

All Australian schools benefit from the outcomes of national testing. Schools can gaindetailed informationabouthowtheyareperforming,and theycan identify strengthsandweaknesseswhichmaywarrantfurtherattention.

Inrelationtothesecondbenefit,‘accountability’,ACARAstates:

Australians can expect education resources to be allocated in ways that ensure that allstudentsachieveworthwhilelearningduringtheirtimeatschool.ThereportedoutcomesoftheNAP[NationalAssessmentProgram]enabletheAustralianpublictodevelopageneralnational perspective on student achievement and,more specifically, an understanding ofhowtheirschoolsareperforming.

‘WhyNAP?’ACARAwebsite5

Theseaims–accountability and improvement–arehallmarksof theGlobalEducationReformMovement

(GERM),whichtypicallyusesstandardizedtestinstrumentstoimplementeducationalpolicy(Chalhoub-

Deville, 2016; Sahlberg, 2015). Prominent in a reform agenda, the concept of ‘accountability’ is

unidirectional: teachers and schools are held accountable to governments and the populace through

standardizedtestresultsthataremadepublicthroughsomemeans.AsstatedintheNAPLANstatement

of benefit above, the test results provide the public with ‘a general national perspective on student

achievement’ with a particular focus on ‘how their schools are performing’. Therefore, the test is a

powerfulpolicyenactmentwhichsteersthe ‘systemsandtheworkofschools’(Lingardetal.,2013,p.

539).

4https://www.nap.edu.au/about/why-nap(Accessed8December,2016)5https://www.nap.edu.au/about/why-nap(Accessed2November,2016)

4

GERM testing samples cross-sections of populations for thepurposes of comparisonsbetween

populationsub-groupings,suchasgeographicalareas,e.g.comparisonofAustralianstatesbyNAPLANin

LingardandSellar(2013)orpre-identifiedsocialcategories,e.g.comparisonsofgenderachievementby

ProgrammeforInternationalStudentAssessment(PISA)andPanCanadianAssessmentProgram(PCAP)by

Martino and Rezai-Rashti (2013). These types of comparisons are enabled through a process of

‘commensuration’ whereby different entities (e.g. schools, districts, states) and categorizations (e.g.

geographicalareas,gender,ethnicity)arecomparedbytransferringtheirqualitiesintoacommonmetric

– the test scores (Espeland & Stevens, 1998; Lewis & Hardy, 2017). Commensuration underpins the

notionof‘accountability’intheeraofpopulation-leveleducationaltesting(Hoffer,2000);comparisonsby

categoryon testmeasuresarehowgovernments ‘do’accountability.Theexplicit categorizationswhich

are used to organise NAPLAN scores for accountability-based comparisons are state, sex, Indigenous

status,LanguageBackgroundOtherthanEnglishstatus(LBOTE),parentaloccupation,parentaleducation,

andgeolocation(metropolitan,provincial,remoteandveryremote).6Thistypeofdatacanbeconsidered

‘catalyst data’ because it brings about the need for explanations for differences in test results across

categories (Lingard & Sellar, 2013, p. 635). Thus, the accountability is ostensibly at the level of

government, as it iswithother testingprograms thatprovide system-level comparisons, such asPISA.

However, the publishing of NAPLAN school results on awebsite known asMySchool ensures that the

results are highly visible at a local level.7One use of the website is to enable ‘fair and reasonable

comparisonsamongschoolswith similar students’ througha complexmeasureknownas the ‘Indexof

Community Socio-Educational Advantage’ (ICSEA)which is calculated using geographical location, the

proportion of Indigenous Students, and parents’ education and occupation. 8 Therefore the

commensurationiscarriedoutatthelevelofindividualschoolsviatheICSEAmeasure,NAPLANscores

andthepublicwebsiteasaportalforaccountability.

For language testers, a more familiar perspective on accountability has its roots in validity

theory,whichhas,fornearly50years, includedtheconsequencesoftestingaspartofthediscussionof

test validation (Bachman, 2005; Chalhoub-Deville, 2009; Cronbach, 1988; Kane, 2013; Lynch, 2001;

McNamara,2008;Messick,1989;Shohamy,2001).Theimpactofhigh-stakestests,definedas‘…anyofthe

effects that a testmay have on individuals, policies or practices,within the classroom, the school, the

educationalsystemorsocietyasawhole’(Wall,1997,p.291),isthefocusofaccountabilityconcernsfor

educationaltesters.Inthisview,thepositiveandnegativeeffectsofthetestarepartofthevalidationof

themeaningfulness and usefulness of test scores (e.g. Fulcher&Davidson, 2007;Messick, 1996). This

educationalmeasurementviewofaccountabilityrelatestothedefensibilityoftheinterpretationanduse

ofthetest instrumentinrelationtoitseffectsonpopulationsoftest-takers.Thoseheldaccountablefor

negative consequences are test providers/developers and policymakers, i.e. thosewho effect the test

instrumentandpolicy(alsoseediscussionofresponsibility inChalhoub-Deville,2009). Incontrast, the

GERM notion of accountability is concerned with the defensibility of school and teaching quality in

6In2016thedecisionwasmadetousetheABSAustralianStatisticalGeographyStandard(ASGS)forNAPLAN(majorcities,innerregional,outerregional,remoteandveryremote)http://www.nap.edu.au/information/glossary(Accessed11February,2017)7https://www.myschool.edu.au8https://www.myschool.edu.au/MoreInformation/About(Accessed9January,2017)http://www.acara.edu.au/_resources/About_icsea_2014.pdf(Accessed9January,2017)

5

relationtothecomparedperformancesofparticularsub-groupsoftest-takers.Inreform-targetedtesting,

accountabilityflowsdownwardsothatthoseheldaccountableareschools,teachersandeventest-takers

andtheirparents, i.e. thosewhoareaffectedby thetest-policy(seealsodiscussion inLingard,Sellar,&

Lewis,2017).

In jurisdictions that implement educational reform using standardized tests, the process of

commensurationprovidesametricwhichcombinespersuasivelywiththemetaphorof‘distance’(Lewis

&Hardy,2017),areadily-invokedpolicyhook,e.g. intheUS, ‘nochildleftbehind’and‘racetothetop’,

and,inAustralia,‘closethegap’,whichreferstothedifferenceinaveragescoresofIndigenousAustralian

children and non-Indigenous children.9An Australian policy application of this process is the current

targettohalvetheEnglishliteracyandnumeracygapbetweenIndigenousandnon-Indigenousstudents

by 2018 using 2008 NAPLAN results as baseline data (Council of Australian Governments, 2009). As

Guentherobserves,NAPLANhasbecomeembeddedinthe‘rhetoricofschooling’;thenotionofthe‘gap’

offersadefinitionofeducationaldisadvantage,which,inturn,risksmanufacturingtheunderstandingthat

‘beingAboriginalorTorresStraitIslanderisthedisadvantage’(2013,pp.157-8).

NAPLANandchildreninremoteIndigenouscommunities

ThelanguageecologiesofAustralianremoteIndigenouscommunities

The Indigenous population of Australia constitutes 3% of the total population. Of this Indigenous

population, 22% live in remote or very remote communities scattered across (the north of)Western

Australia,theNorthernTerritory,andQueensland(AustralianProductivityCommission,2014).Inthese

remote communities, the language ecology is varied and complex with high rates of language shift

underway (Marmion, Obata, & Troy, 2014), and the array of languages being spoken in any remote

communityisoftencomplex.ChildrenmaybelearningastheirL1oneofseveraloptions.Somechildren

willgrowuplearningthetraditionallanguageoftheareainwhichtheylive,althoughonlyaround15-18

traditionallanguagescontinuetobelearnedbychildrenasafirstlanguage.Otherchildrenmaybeliving

in a community where language is undergoing rapid shift and they are active in the process of

developing newmixed languages (e.g. Gurindji KriolMeakins, 2012; and LightWarlpiri O'Shannessy,

2005).Alternatively,childrenmaybeacquiringoneofthemanyvarietiesofKriolastheirfirstlanguage

such as Roper River Kriol, Fitzroy Valley Kriol, Barkly Kriol orWumpurrani English, another creole

languagesuchasYumplatokorYarrieLingo,oroneofanumberofdifferentAboriginalEnglishvarieties

(seediscussionsinMeakins,2014;Sellwood&Angelo,2013).Whatallthesechildrenhaveincommonis

that they are growing up in language environments where Standard Australian English (SAE) is not

spokenandtheyarenotacquiringSAEastheirfirstlanguage.Rather,theirenvironmentismuchmore

akintoanEnglishasaForeignLanguage(EFL)environmentthantothatofmigrantchildrenwholivein

English speaking communities. In these remote communities, SAE is not generally spoken among

membersof the Indigenous community and is, for themostpart, limited to someofficial government

activities, health and justice contexts, the classroomand the local shop. Formost children, their first,9Inthecaseofliteracyandnumeracy,the‘gap’isdeterminedbytestscores,butthemetaphorisusedtodescribearangeofpolicyareas,(e.g.health,employment)eachofwhichhasanassociatedprocessorprocessesofcommensuration.

6

moresustainedexposuretoSAEandtowrittenformsoflanguagecomeswhentheystartschoolat5-6

yearsold.

NAPLANandthemonitoringofIndigenouschildren

Indigenouschildrenareasub-groupingoftheAustralianpopulationwhoareidentifiedinthetestdata

and subject to explicitmonitoring via theNAPLAN tests. Thismonitoring has consistently found that

Indigenous childrenarenot achievingat the same level as the restof thepopulationacrossall states

(Ford,2013).Angelo(2013)pointsoutthatthedisaggregationofIndigenousstudents’scoreshasmade

thisgroup‘highlyvisible’andhencesubjectto‘intensiveeducationalinterventions’onthebasisofpoor

NAPLANresults,butwithnorecognitioninthetestdatathatmanyIndigenouschildrenarealsoEnglish

languagelearners(p.54,seealsoDixon&Angelo,2014).

Thetestresultsfromthefirstadministrationin2008indicatedthatthetestscoresofIndigenous

childrenwerelowerthannon-Indigenouschildren,andthatallchildrenlivinginremoteareasachieved

lower scores than their urban peers (ACARA, 2008). Indigenous children in remote areas, however,

scoredlowerthanallremotestudentsacrossallsubtestsatallgradelevels(ACARA,2008).Mediareports

atthetimefocusedontheperformanceofIndigenouschildren,withheadlinesinmajornewspaperssuch

asOneinThreeIndigenousKidsFailTest (Ferrari,2008)and IndigenousStudentsFallingFurtherBehind

(Adlam,2008).The impactof thispressurewasgreatest in theNorthernTerritory(NT),whichhas the

highestpopulationofIndigenouschildrenlivinginremoteareas,veryfewofwhomarenativespeakersof

StandardAustralianEnglish(SAE).TheNTGovernmentmandatedthatallstateschoolsteachinEnglish

forthefirstfourhoursofeachday–apolicywhicheffectivelysawtheendofthefewremainingbilingual

education programs in place, even if ‘programs had been running fairly successfully with strong

communitysupport’(Devlin,2011,p.263;Devlin,Disbray,&Devlin,2017;Simpson,Caffery,&McConvell,

2009).

Since its introduction, therehavebeentwogovernment inquiries intoNAPLAN(Educationand

EmploymentReferencesCommittee,2014;Education,EmploymentandWorkplaceRelationsReferences

Committee, 2010), each ofwhich attracted submissions concerningNAPLAN and Indigenous students.

The 2010 report recommended that ‘ACARA analyse and report publicly on how NAPLAN tests are

servingdifferentgroupsofLanguageBackgroundOtherThanEnglish(LBOTE)students’asaresponseto

the non-recognition of the language learner status of many Indigenous students (p. 18). However, in

dealing specifically with Indigenous student performance, no mention was made of language learner

statusdespitethereportedcommitteeviewthat‘ACARAshouldinvestigatewaystoprovideaccesstorich

information on lower achievers’ (p. 20), an action which would almost certainly reveal that English

languageproficiencyisaprominentvariableformanyIndigenouschildren(seeAngelo,2013).The2014

inquiry noted that NAPLAN contains material which presumes both English-as-a-first-language

developmental levels and experience of urban environments, but the recommendations focussed

predominantlyontheintroductionofacomputeradaptiveversionofthetestwhichwashopedtosolve

thegamutofcomplexissuesraised.

7

ImpactofNAPLAN

In the case of NAPLAN, the most widely reported effect or washback has been a narrowing of the

curriculumthroughagreaterfocusontesttasktypesandtestpractice(Dulferetal.,2012;Hardy,2015;

Thompson, 2014). A narrowed curriculum echoes the experience of other jurisdictions with national

testing programs (e.g. Berliner, 2011; Reay & Wiliam, 1999), and the narrowing can result in

impoverished instruction for students for whom the tests are in a second language (Creagh, 2016;

Menken, 2006, 2008). Narrowing of the curriculum and test task practice which exceeds the

recommended test format familiarizationhavealsobeen reported in schoolswithmajority Indigenous

children in Australia (Angelo, 2012, 2013; Unsworth, 2013). Unsworth’s (2013, p. iii) broad survey of

teachers’viewsofNAPLANin30remoteschoolsalsoidentifiedamisalignmentbetweenthe‘teachingto

the test and teaching to Indigenous students’ funds of knowledge’. Related to thismisalignment is the

identification of cultural inappropriateness of NAPLAN materials for children in remote communities

(Unsworth,2013;Wigglesworth,Simpson,&Loakes,2011),forsomeofwhomsuchforeigncontentcould

onlycompoundtheeffectoftheforeignlanguageofthetest.

The interactionbetween thewriting test task and the second language learner status of some

Indigenous children has also been the subject of research. An analysis of practice narratives for the

NAPLANwritingtestbychildrenwhoareL1speakersofTorresStraitCreoleshowedthatthechildren’s

secondlanguagedevelopmentallevelimpactstheirtestresponses,andfurther,thateventheabilityto‘fill

up’ overtly-taught narrative structures (e.g. introduction, problem, solution and conclusion) requires a

levelofL2proficiencywhichisbeyondtheSAEcompetenceofsomeofthesechildren(Angelo,2012,p.

50).

Method

Researchaim

InresponsetotheconcernsdocumentedintheliteratureandthehighvisibilityofIndigenouschildrenin

the annual NAPLAN reporting and media response, this study investigates the impact of NAPLAN in

remoteIndigenouscommunities.Specifically,weposedtwobroadquestions:

1. From the perspectives of school staff with experience of NAPLAN in remote Indigenous

communities,whataretheimpactsofNAPLANinremoteIndigenouscommunities?

2. Arethereportedeffectspositiveornegative?

Participants

Theintervieweeswereapurposivesampleof23schoolstaffwhohadcombinedexperienceofNAPLANin

20 Indigenous communities classified as ‘very remote’ in Western Australia, Queensland and the

NorthernTerritory.10Thecategoryof‘veryremote’inAustraliaisdeterminedbydistancetothenearest

cities and towns. A very remote community may be hundreds of kilometres from the nearest urban

centre.

10http://www.nap.edu.au/Information/Glossary#l(Accessed3November,2016)

8

Theparticipants’ teachingexperience ranged from four to thirtyyears;18withexperienceas teachers

and 5 with experience as teachers and principals. The small sample size reflects the proportion of

teacherswithexperienceofNAPLANinremotecommunitieswhichisalsoverysmall(asan indication,

around 3% of all primary school teachers worked in remote or very remote areas in 2001).11 Our

selective samplingprocedurematchedour researchquestionby seeking theperspectives of principals

and teachers,both Indigenousandnon-Indigenous,with first-handexperienceof theuseandeffectsof

NAPLANinremoteIndigenouscommunities.

Procedure

Theinterviewsweresemi-structuredandeachlastedapproximately30minutes.Interviewswerecarried

outbymembersoftheresearchteambyphone,bySkypeorface-to-face.Priortotheinterviews,potential

participantswereinformedthattheinterview

willfocusonwhathappensatyourschoolaroundthetimeofNAPLANtestingandwhetherornotNAPLAN has an effect on aspects of schooling such as curriculum, student attendance, studentmotivation,useofhome languagesandEnglish, teachingandassessmentpracticesandanyotheraspects you think are relevant. We will also be interested to hear how NAPLAN affects thecommunitymoregenerallyandanychangesyoubelieveshouldbemadetothetests.

The interview questions were broadly designed to explore any effects the test had on schooling (see

questionsinAppendixA).Participantswerealsoinvitedtoraiseanytopicstheyfeltwererelevanttothe

researchfocus.Ouruseofinterviewsisasmeansofcollectinginformation,ratherthanasocialpractice,

asdistinguishedinTalmy(2011).Onecleardisadvantageofthisapproachisthatittreatsinterviewdata

asindividuallyconstructedbytheinterviewee,ratherthanthecollaborativeeffortbroughtaboutbythe

interviewcontextandthesemi-structured(non-standardized)natureofthequestions(see,forexample,

discussionsinMann,2011;Talmy,2010).ItshouldalsobenotedthatNAPLANisachargedtopicamong

educationalistsinAustralia,withmanyteachersperceivingittohavenegativeeffectsonvariousaspects

ofschooling(Dulferetal.,2012).Thatsaid,webelieveit is intheinterestofthebroadercommunityto

hear theviews,as faraswecanpresent them,of thosewhohaveexperiencedanational standardized

testingprograminthesecomplexlinguisticenvironments.

TheaudiorecordingsweretranscribedandanalysedthematicallyusingNVivosoftwarebytwo

coders inan iterativeanalyticprocessbasedon theprocesses setout inMiles,HubermanandSaldaña

(2014).Thefirststageinvolveddataannotationsontheinterviewtranscripts,aswellasidentificationof

main themes inherent in the interview questions. These annotations and expected themes formed the

basis of broader themes or ‘pattern codes’ (Miles et al., 2014, p. 73). Pattern codes included practical

considerations (e.g. administration), practices (e.g. test strategies), beliefs or perceptions (e.g. cultural

appropriateness), and more theoretically-oriented topics (e.g. usefulness of data). These themes were

trialedcollaborativelybytwoanalystsonadatasubset,followedbycodingofthewholedatasetbyone

coder.Thenextstageinvolvedasynthesisofthesecodedexceptsinto‘higherlevelanalyticmeanings’by

11 http://www.abs.gov.au/ausstats/abs@.NSF/2f762f95845417aeca25706c00834efa/459c3882fad473a2ca2570eb0083be84!OpenDocument (Accessed 26 October, 2017)

9

thesecondanalyst(Milesetal.,2014,p.95).Inthisstage,thepatterncodeswererelatedtotheoretical

aspectsoftestimpact,(forexampleinCheng,2008;Messick,1996;Wall,1997),whichformedthebroad

categoriesunderwhichwepresentthefindingsbelow.Thesethemesareinevitablyselectiveduetothe

richnessoftheinterviewinsights,theconstraintsofspaceandtheneedtoconsiderthedatainrelationto

thebroaderissueswhichareofrelevancetoothercontextsinwhichnationaltestingiscarriedout.

Findings

Thefindingsarepresentedinthreebroadareasrelatingtotestimpact:i)Construct-irrelevantvariance,

ii)Psychosocialimpactsiii)UseofNAPLANresults.

Constructirrelevantvariance

Presumed English language proficiency. All participants agreed that the English language proficiency

status (i.e. English as a second/foreign language or dialect) of Indigenous students in remote

communities is amajor barrier tomeaningful participation inNAPLAN. As Teacher 4 observed,many

childrenfirstencounterEnglishatschoolwhen:

they’veneverseenbooks,they’veneverseenthelanguagewrittendown,theyhaven’tseentheirown

language written down; they come from an oral background, they come in not even speaking

English,andnowwe’regettingthemtositdownandtryanddoaNAPLANtest.Canyouseehow

absurdthatis?

Principal 1 described the enormity of the test English literacy demands in relation to students’ oral

English proficiency: ‘When you struggle to speak a sentence in Standard Australian English, it’s really

hardtowriteanarrative.’

Thewriting testwas generally perceived as posing the greatest challenge becausemeaningful

participationrequiresahighlevelofwrittenEnglishlanguageproficiency.Lessmeaningfulparticipation

was achieved in some cases through the use of strategic genre formulas for the NAPLAN text types

(persuasiveornarrative).Further,therewassomedisparityreportedbetweentheconditionsandlevelof

thewritingtesttaskandthetypicalprocess-basednatureofclassroomwriting.Forexample,Teacher9

notedthatthewritingtasktopiccanbe‘quitedifficulttoaccess’without‘supportandnegotiatingthefield

of discussion before they get going’. Participants described various writing test strategies which

compensate for students’ inability to provide an engaged response. Teacher 3, for example, described

studentssimplywritingallthewordstheyknew.Teacher8explainedthatshewouldtellthestudentsto

writeasentencetheyhadlearntpreviouslywhich‘mightnothaveanythingtodowiththetopicbutthey

mighthaveafullstopandacapitalletterandthat’ssomethingthattheycanputonthepage’.

Thenumeracy test drew comment in relation to theword-basednature ofmany of the items,

whichwasconsideredabarriertodemonstrationofskills:thenumeracytestis‘almostatestofEnglish

comprehension, rather than mathematical ability’ (Teacher 10). One accommodation is that

administrators may read aloud worded parts of the numeracy test (ACARA, 2015). However, it was

reported that in practice, the accommodation made the test delivery time-consuming and incoherent

10

becausethenumericalcomponentofthequestioncannotbereadaloud:‘You’retryingtoreadittoachild

who’sgotEnglishasasecondlanguagebutleavingbitsout’(Teacher8).

Presumed cultural knowledge. Teachers reported instances in which the texts and items in both the

numeracyandliteracytestspresumeexperienceandknowledgetobecommonacrosschildhoods,butare

‘totallyalien’tothelifeexperienceofchildreninremotecommunities(Principal2).Participantsprovided

manyexamplesofconceptswhichmaybefamiliartochildreninurbanareas,butarerelativelyforeignin

a remote community, for example the topic of ‘recycling’ (Principal 3), ‘hosing things into gutters’

(Teacher 10), tenpin bowling (Teacher 3) and going to the cinema (Teacher 14). Issues of cultural

inappropriateness areperhapsmostproblematic in thewriting testwhere thewhole taskhingeson a

singleconcept.Forexample,‘changingalaw’,whichasTeacher15explained,isacomplexandunfamiliar

topicinthecontextofanunfamiliartypeofactivity(thetest).

Bubble-filling.Manyparticipantsreferredto‘bubble-filling’–themultiple-choiceresponseformatwidely

used across the reading, language conventions and numeracy sections – as a test strategy or even an

activity in and of itself, rather than a format for testing literacy or numeracy. Bubble filling gave the

students something they could do ‘even if they didn’t understand the question, theywere able to put

something’(Principal1).Teacher4reportedtellingthestudents ‘topicktheirfavouriteletterandfill it

in’.Teacher2describedtheflurryofbubble-fillingthatoccurredeven ‘beforeyousay, turnyourpaper

over’.Principal2,intheexcerptbelow,indicatesthatforstudentswithverylimitedEnglishproficiency,

therewaslittleengagementwiththetestbeyondbubble-filling:

It’s just, some of themwould say, ‘what do I do here?’. And you would just say, ‘well, fill in the

bubble’.[…]iftheywerecapableofreadingyouwouldsay,‘readthequestionandfillinabubble’.

[…]iftheyweren’tcapableofreadingIwouldjustsay,‘justfillinabubble’.

Notsurprisingly,therewasageneralperceptionthatthetestresultswere,tosomeextent,aproductof

guesswork.

Washbackonclassroomandcurriculum

Theinterviewdatarevealedagreatdealofvariabilityintheschoolpracticesregardingtestpreparation.

Principal 2 explained that her school did ‘nothing specific’ in the lead up to the test except for

familiarisingthestudentswithsomeofthequestiontypes.Teacher3describedaclearbeliefinnotdoing

specific test preparation tasks because, in her view, test practicewas of ‘no use’ as such tasks do not

engagethestudents’abilities.Similarly,Principal1felttheSAEspeakercompetencethatunderpinnedthe

testmade it unfeasible to prepare for the tests in anymeaningfulway: ‘The tests themselvesweren’t

goingtocater forESL[EnglishasaSecondLanguage]students[…]soyouknow,wewereteachingthe

curriculum,weweren’ttryingtoteachthetest’.Incontrast,Teacher1reportedconsiderableamountsof

testpreparation,withteachers‘veryfocusedonteachingtothetest’,particularlytheNAPLANtexttypes.

11

Principal3describedalotofworkonpersuasivetextsandweeklypracticetestswhichhefelthadlifted

theirresults.

Teacher11observedtwoapproachestoNAPLANpreparation.Onewastodopracticetestsand

theotherwastocreateNAPLAN-formatresourcesthatlinkthetesttotheteachingprogram‘sothey’re

gettinganunderstandingofwhatatestisaboutwiththecontentthatthey’refamiliarwith’.Anexample

of thismore learner-basedapproachwastoadaptapopularESLresourceso that it includedNAPLAN-

style multiple choice questions (Teachers 8, 9 and 10). Not surprisingly, multiple choice format was

widelymentionedasapre-taughtfeatureofthetest.Teachers8and14described‘practisingcolouringin

multiple choice’ so that the students didn’t sit there doing nothing. Ironically, Teacher 16 felt that

NAPLANactually‘damagedtheirthirstforliteracyandreading…becausewe’dhavetostopourregular

literacyprogramthatwasworkingsowell,foraweekleadingupandthendothesetestsandthengoback

toitandwe’vemissedtwoweeksofqualityteachingtime.’ThiswasechoedbyTeacher1whofeltthat

‘theamountoftimethatisspentjustfocusingoncontentthatyouthinkisgoingtobethetestprevents

kidsaccessingallareasoflearningandgettingthoserichlearningexperiences.’

Psychosocialimpactsoftestexperience

Psychological impact on students. There were various perceptions of how students were affected

psychologically by the test. These ranged from having little impact to having strong negative effects,

whichmayresultfrom‘theemphasisthattheschoolputsonthetestandtheindividualteacher’(Teacher

1). Some teachers felt the test was simply irrelevant and an inconvenience. Teacher 17, for example,

explained that for her Year 3s, the experience wasn’t ‘overwhelmingly positive or overwhelmingly

negative,itwaskindofanexperiencethatforthemwasprobablyabitnonsensical;theycouldn’treally

understandwhatwe’reaskingthemtodoorwhywe’reaskingthemtodoit.’

Someteachersobservedstrongnegativeeffectonthestudents,intermsofstressorself-esteem.

Teacher14explainedthatdespiteschoolstryingvariousmethodsofmakingthestudentsfeelsupported

andpromotingthechancesofsuccessonthetest,shecouldnotseeanyaspectofthetestingthatsuited

theremoteIndigenouscontextinwhichshetaught.Forher,theEnglishlanguageproficiencyrequiredfor

testparticipationraisedissuesofstudentwellbeing:

Ididn’tfeeltheyneededthatpressureonthembecausethentheywouldjustshutdowncompletely.I

hadonechildduringNAPLANhidingunderthetable,throwingpencilsout,juststressedcompletely

aboutitandbecausetheydidn’tunderstandit,despiteallthepreparation.[…]Ijustlettheminthe

end,writewhatevertheywantedforthewritingsectionandtheyfeltliketheyweresucceedingand

IletthembelievetheyweresucceedingandtoldthemhowproudIwasofthemandthingslikethat

because to me, the number one thing I’m looking after in the classroom is the wellbeing of the

students[…]itwasjustbeinginEnglishitwastoohardforthem.

AttendanceisreportedlyaffectedbyNAPLAN,withstudentsstayingawayonNAPLANdays,or

experiencing one day of testing and then refusing to return for the other test days and reports that

12

parentskepttheirchildrenawayfromschoolonNAPLANdaysoutofconcernforthenegativeimpactof

the test pressure (Teachers 12, 14).While attendance is an issue throughout the school year inmany

remotecommunities,principalsreportedthattheymobilisedattendanceofficersforNAPLAN(Principal

1)orevencollectedchildrenpersonally(Principal2).Teacher3,aNAPLANadministrator,explainedthat

thesporadicattendanceacrosstheNAPLANweekmeantthat‘itwouldoftentake3or4timestoactually

finishit’.Staffalsowenttosomeefforttolettheparentsknowattendancewasespeciallyimportanton

NAPLANdays,forexamplegoingtothestoreoraroundthecommunitytoalertparents.

Psychologicalimpactonteachers.SometeachersfeltthemselvestobenegativelyaffectedbytheNAPLAN

administration, which was ‘possibly harder for the teachers than it is for the students because, as

teachers,we knowwhat the test is andweknowwhat the implications for the results of the test are’

(Teacher17).Someteachersfelttheythemselveswerebeingjudgedbythetestresults,whiletop-down

pressuresassociatedwithNAPLANresults(describedbyTeacher1below)couldaffectthecultureofthe

school:

IthinkNAPLANaffectstheschoolandthestudentsbyactuallychangingthecultureoftheschool.I

thinkit’sbecausethere’sbeenwhat’ssoldasahigh-expectationagendaoranimprovementagenda

[…] and the only thing that is used tomeasurewhether outcomes are improving for students is

NAPLAN.Soprincipalsgetpressurefromtheirregionswhogetpressurefromthecentralofficewho

getpressurefromwhereverthatcomesfrom,fromabove,andthenthatimpactsonprincipalspass

thatpressuredowntoteacherssothat’salmostchangedthecultureofteaching.

Teacher 5 explained that although ‘it was kind of good to see what would be expected of kids in

mainstreameducation’,itwasalso‘disheartening’toseethatherownstudentswere‘sofarbehind’,even

thoughsheacknowledgedthatEnglishmaybeathirdorfourthlanguageforherstudents.Forsome,this

resultedinaverynegativeemotionalresponse,describedbyPrincipal1below:

…almost every year after the tests have finished […] the teachers are generally feeling soul-

destroyed,allfeelingabsolutelydevastated[…]theyfeelsobadaboutputtingthechildrenthrough

it.

Impactonstudent-teacherrelationships.Thetestexperiencewasperceivedbyseveralteachersashaving

anegative effect on the student-teacher relationship.Teachers felt that their inability tohelp students

duringthetesterodedthetrustthatcharacterisesstudent-teacherrelationships.Theyfeltresponsiblefor

imposing an impossible task on the students at the same time as refusing assistance which would

normallybeprovided.Forexample,Teacher9describeditas‘breakingthatbondinthatyou’regivingthe

studentssomethingthatyouknowtheycan’tdoandyou’renotgivingthemanysupport’.

To counteract the negative effects on the students, there was also an element of protection

expressedwhereteacherstriedtoinsulateandcompensatefortheperceivedrisktostudents’esteem.For

example, teachers gave ‘pep-talks’ (Teacher 10) or ‘played it down’ (Teacher 11) as well as giving

rewardsorpraiseattheendofthetest(Teacher4).Teacher14explainedthatshewas‘neverhonestwith

13

thekids’,preferringtomake‘itseemlikeafunthing’withthembecauseshedidn’twantittoaffecttheir

self-esteem.

UseofNAPLANresults

AlthoughparticipantsindicatedthattheytooknoteofNAPLANresults,itwaswidelyexpressedthatthe

NAPLAN data has little relevance or usefulness for schools or parents. Comparisons between schools

usingtheMySchoolwebsitewere foundtobeunhelpfulduetothesmall,variablecohortsdescribedby

Principal1above.Principal4saidthattheschoolusedsomeoftheresults,but ‘attheendofthedayit

doesn't showusahellofa lot’. Inaddition, theverysmall,variablecohortsmade the interpretationof

dataproblematic,assuggestedinthefollowingexcerpt:

we spenta lot of time lookingat thedata, and I still havenightmares sometimesaboutboxand

whiskerdataanalysiswithgraphs[…]TheYear5sthisyearin2007didthisandin2009, lookat

this improvement, you can work on that! And you’re trying to explain to them, look, they were

completelydifferentkids;wehadawhitekidintheclassin2009andthat’swhytheresults[are]so

muchbetter.ThisisoutofaclassofaboutfourYear7’sorsomething(Principal1)

Therewereinstancesinwhichresultswereusedincurriculumplanning:inordertoaddressweaknesses

(Principal 3) or decidewhich areas ofmaths to cover (Teacher 10). Teacher 1 described sharing and

analysingtheresults,lookingat‘whichyearlevelsdidwellinwhattest[…]andwhatteacherstheythink

hadwhatimpactonwhatresults’.Sheperceivedthisprocessput‘alotofpressure’onteachers.

At the level of the jurisdiction, Teacher 9 described how ‘every year the NT government gets

kicked’ at the release of NAPLAN results. He explained that despite government reviews identifying

bilingualeducationasthebestevidenced-basedpractice,NAPLANresultsandgovernment-setNAPLAN

goalsputdestructivepressureonbilingualprograms.

Discussion

These data, from teachers and principals who have experienced firsthand the effects of NAPLAN in

remoteIndigenouscommunities,showthatthereiscauseforconcernabouttheappropriatenessoftest

useinthesecontextsandtheresultingeffectsofthemismatchbetweenstudentandteacherneedsonthe

onehand,andtest instrumentsandusefulnessofresultsontheother.Onthewhole,staffdidnotthink

thatthetestresultsreflectedtheabilitiesofthestudentsforfourmain,interrelatedreasons,thefirstof

whichisprimary:1)Englishlanguageisnotthefirstlanguageandinsomecases,notthefirstliteracyof

the children 2) the test content is inaccessible or foreign 3) a large proportion of children draw on

construct-irrelevant test strategies such as bubble-filling 4) the tests do not engage the children

cognitively or socioculturally. In the view of most participants, these characteristics of the test

instruments(language/content)andtest-takerbehaviours(testengagement)havenegativepsychosocial

impacts on students and staff,with little or no benefit in termsof the usefulness of the test data. The

reportedwashback on classroom and curriculum amounts to superficial test-survival strategieswhich

14

potentiallyevendamagestudents’literacydevelopment,ratherthanpromoteit(seealsoAngelo,2013).

Intheviewsoftheparticipantsinthisstudy,theNAPLANexperienceandeffectsappeartobe,atbest,an

irrelevance and atworst, the source of considerable distress. This finding echoes other surveys in the

AustraliancontextthathaveconcludedthatmostteachersfeeltheeffectsofNAPLANhavebeenlargely

negative(Dulferetal.,2012;Thompson,2014).

TheseeffectsarecompoundedbytheveryhighvisibilityofIndigenouschildreninthenational

analysis,whichdisaggregatesanddiscusses thisgroup’sperformanceannually.However,arguably,not

includingIndigenouschildreninremotecommunitiesinanationaltestingprogramwouldseparatethis

ethnoculturalgroupinwaysthatarealsopotentiallydetrimental.Itisimportanttonotethatparticipants

werenotopposedtostandardizedtestingperse,butrathertothepresumptionofEnglishnative-speaker

language competence and western urban culture that they perceived to underpin test constructs.

Connectedtothisisthelackofdiscernmentatthelowerendofattainment;asTeacher9explained,his

studentswere‘prettymuchalldeemedtofailequally’eventhoughheknewtheywerenotallatthesame

level.

A fundamental issue is that theNAPLAN test constructsareunderpinnedbyEnglish-as-a-first-

language literacy (Angelo, 2013; Creagh, 2016). Writing from the US context, Solano-Flores (2009)

observesthattheinclusionofEnglishlanguagelearnersinnationaltesting‘hasnotbeenaccompaniedby

effectiveapproachestominimizeconstruct-irrelevantscorevarianceduetolanguageissues’(p.33).He

characterisesthetestingofsuchlearnersasa‘stochasticprocess’duetouncertaintyaboutvariablessuch

as sensitivity to contextual features, and language of the test delivery (Solano-Flores, 2009, p. 43). As

observedby theparticipants in thisstudy,even thenumeracy test is rendered inaccessible through its

heavilyEnglish-languageconstruct.Toexemplifythis,ofthe120itemsonthefourYear3numeracytests

from2008to2011,therewerejusttwoitemscomposedonlyofnumericalsymbolsorshapes.Allother

items require anunderstandingofwrittenEnglish formeaningful engagement and, inmany instances,

thesearecomplicatedsyntacticstructures,suchas‘InwhichgamedidKimscore3moregoalsthanshe

scoredinGame2?’(NAPLANYear3NumeracyTest,2008).TheNAPLANconstructs,therefore,areapoor

fitwiththelanguageecologiesofmanyremoteIndigenouscommunities,wherethereislimitedpresence

ofprint-based literacy inany language. Indigenouschildren in remotecommunitieshavegenerallynot

acquiredanyofthepre-literacyskillsineitherEnglishortheirhomelanguage/sthattheirurban,English-

speaking counterparts are typically equippedwith in readiness for schooling (Kral, 2012). This has a

profoundeffectontheiracquisitionofEnglishacademicliteracy,andinevitably,ontheirperformanceon

NAPLAN.

Messick(1980)remindsus that thequestionof test fairnessariseswhenan individual’sgroup

membershipdeterminesthatheorsheachievessignificantlydifferentlyfromthepopulationasawhole.

AsmeasuredbyNAPLAN,thesignificantlydifferentperformanceofremoteIndigenousstudentshasbeen

sustained and highly visible in reporting and media in every test administration. The quantification

providedbyNAPLANremainsprominentinpublicdiscourseabout‘closingthegap’,withannualrevisits

inthemediaatNAPLANreportingtimewhichporeovertheunclosedgap(e.g.'ReformsFailingtoClose

Indigenous Schools Gap', Ferrari, 2012). In addition to the effects that these participants reported in

relation to students, teachers, classroom and curriculum, the commensuration, i.e. the comparison

15

differentgroupingsonacommonmetric(Espeland&Stevens,1998),providedbyNAPLANhasbecome

absorbed in Australian educational and public discourse; it has become ‘common sense’ (Star &

Lampland, 2009). An easily-comprehendedmetric, NAPLAN belies the nature of schooling in complex

sociolinguistic contexts, thus renderingan ‘empirically intricate realitydeceptively straightforward’ for

the purpose of comparison (Stevens, 2008, p. 102). Thanks to this systemic absorption, ‘the numbers

themselvescometoberegardedascoextensivewiththeunderlyingphenomenatheymeasure’(Stevens,

2008, p. 102). This contribution of test-user perspectives has interrogated this assumption and

highlightedthecomplexityunderpinningthemetric.

Conclusion

StandardizedtestingprogramssuchasNAPLANareintendedtobringaboutchangeintheliteracylevel

(as an indicator and proxy for ‘educational outcomes’) of the population. The test enacts a policy

underpinned by efforts to increase life opportunities at an individual level (e.g. Mahuteau, Karmel,

Mavromaras,&Zhu,2015)andacorrelationbetween literacyandeconomicgrowthatanational level

(e.g.Coulombe,Tremblay,&Marchand,2004).Thebodyresponsible forNAPLAN,ACARA,hasstressed

thattheuseofNAPLANresultsactsasimpetusinthiscausationalchainbelow:

ItshouldbeemphasisedthatNAPLANisatoolto informschool improvement,notanimproverof

educationaloutcomes.Itisnottheteststhatwillimprovestudents’literacyandnumeracyskills,but

thewaystudents’results(includingschool,systemandnationallevelresults)areusedbyteachers,

schools and systems to identify strengths andweaknesses, particularly in teaching practices and

programs,thatwillimprovestudentoutcomes.

(ACARAsubmission,reportedinNAPLANInquiryReport,EducationandEmploymentReferences

Committee,2014,p.58)

Asdiscussedearlier,thetestresultsare‘catalyst’fortheaccountabilityprocesswhichapplies,according

to ACARA, ‘particularly in teaching practices and programs’. Yet, this method of accountability is not

necessarilyademocraticone.Shohamypointsoutthat,althoughstandardizedtesting‘maybebasedon

equal opportunities and conditions of testing’, it does not necessarily follow that ‘all test takers have

equalopportunitytoobtainanequaleducation’(2004,p.87).Indeed,sheliststheimpositionofteststhat

are‘basedononetypeofknowledge’ontest-takersfromdifferentbackgroundsasapossibleviolationof

test-takerrights.InrelationtoIndigenouschildreninremotecommunities,itisunrealistic,asKral(2009,

2012) points out, to assume that the development of academic English literacy can be achieved in a

foreign language through brief pedagogical interventions to a level comparable to that ofmiddle-class

Australia(seealsoAngelo,2013).Arguably,practiceswehavecometorecogniseasindicationsoffairness

ineducationaltestingsuchas‘carefulstandardizationoftestsandadministrationconditions’intheCode

ofFairTestingPracticesinEducation(JointCommitteeonTestingPractices,2004)maywellbethecause

of ill-effect when the context, the educational environment and the test-taker linguistic background

differssomarkedlyfromtheliteracypracticesvaluedbythe ‘standardlanguageculture’(Milroy,2001)

16

and onwhich the construct ismodelled. According toAngelo andDixon (2014), this has resulted in a

widespread ignorance of the value of multilingualism or need for English language learning for

Indigenous children. In their view, Australia’s ‘monolingual mindset’ (Clyne, 2008; see also Shohamy,

2011)hasassistedthesafepassageofNAPLAN‘literacyandnumeracytesting’intoschoolingasifitwere

not in any language and hence fair, even for studentswho are speakers of Indigenous traditional and

contactvarieties,withlittleEnglishproficiency.AsGuenther(2013)concludes,‘theassumptionsbehind

theadministrationoftestinginstruments,thelanguageusedinthemandthevaluesembeddedwithinthe

testsshouldbequestioned’(p.167).Ifanidentifiablesub-populationoftest-takersareunabletoengage

with test content because of their proficiency in the language of delivery, the test has poor

‘interactiveness’, inBachmanandPalmer’s(1996)testusefulness framework.Notbeingable toengage

with the test instrument precludesmeaningful results and thus also the kind of reform envisaged by

policymakers.The‘interactiveness’observedbyourintervieweesintheNAPLANappearedtobelimited

tofillinginthebubbleswithnosenseofwhetherornottheresponsewascorrect.

Thisbringsusbacktothepropositionthataccountabilityineducationalreformrunscounterto

accountabilityinconsequentialvalidity.Shohamy’sview,basedonthenotionofconsequentialvalidityin

educationalmeasurement,isthat ‘thereshouldbeaccountabilitywithregardtothepurposeofthetest,

its practice and its methods’ (2004, p. 87). Those responsible for the design and use of the test –

developers,providers,policymakers–needtoensurethattheinstrumentdoesnoharm(Kunnan,2000,

2004).Itisclear,however,thattheteachersandprincipalsinthisstudyfeeltheonusisonthemtodefend

thepoorperformancesoftheirstudents inaGERMsenseofaccountability.Caughtbetweenapowerful

policy instrumentand theneedsof theirstudents,manyparticipantsalso felt theneed toprotect their

students from theunintendednegative effectsof the test instruments.These findings suggest that this

processofcommensuration—thequantitativecomparisonofthequalitiesofdifferentsub-groups—isnot

informative about or beneficial for children in remote Indigenous communities. For comparison to be

carriedout,thenthereisaclearneedforaninstrumentwhichissensitivetothedevelopmentofliteracy

andnumeracyinEnglishasaforeignlanguagewhichisaccessibleforthesestudents.Atalocallevel,such

aninstrumentshouldbeabletodiscernlearninginthesecontextsinawaythatmightbeinformativefor

teaching andbeneficial for learning.At anational level, an instrument that recognises themultilingual

capacity of many Australian children has the potential to disrupt the myth of monolingualism in

Australia’sclassrooms.

17

References

Adlam,N.(2008).Indigenousstudentsfallingfurtherbehind.NorthernTerritoryNews/Sunday

Territorian.Angelo,D.(2012).Sadstories:ApreliminarystudyofNAPLANpracticetextsanalysingstudents’second

languagelinguisticresourcesandtheeffects.PaperpresentedattheProceedingsofthe42ndAustralianLinguisticSocietyConference-2011.

Angelo,D.(2013).NAPLANimplementation:Implicationsforclassroomlearningandteaching,withrecommendationsforimprovement.TESOLinContext,23(1/2),53.

AustralianCurriculumandReportingAuthority.(2015).NAPLAN2015Nationalprotocolsfortestadministration:Downloadedfrom:http://nap.org.au/naplan/school-support/national-protocols.html(Accesed17November,2015).

AustralianCurriculumAssessmentandReportingAuthority.(2008).2008NationalAssessmentProgramLiteracyandNumeracyAchievementinReading,Writing,LanguageConventionsandNumeracy.RetrievedfromSydney:https://www.nap.edu.au/results-and-reports/national-reports

https://www.nap.edu.au/_resources/2ndStageNationalReport_18Dec_v2.pdfAustralianProductivityCommission.(2014).OvercomingIndigenousdisadvantage:Keyindicators2014.

Canberra:AustralianProductivityCommission.Bachman,L.F.(2005).Buildingandsupportingacasefortestuse.LanguageAssessmentQuarterly,2(1),

1-34.Bachman,L.F.,&Palmer,A.S.(1996).Languagetestinginpractice:Designinganddevelopinguseful

languagetests.Oxford:OxfordUniversityPress.Berliner,D.(2011).Rationalresponsestohighstakestesting:Thecaseofcurriculumnarrowingandthe

harmthatfollows.CambridgeJournalofEducation,41(3),287-302.Chalhoub-Deville,M.(2009).Theintersectionoftestimpact,validation,andeducationalreformpolicy.

AnnualReviewofAppliedLinguistics,29,118-131.Chalhoub-Deville,M.(2016).Validitytheory:Reformpolicies,accountabilitytesting,andconsequences.

Languagetesting,33(4),453-472.Cheng,L.(2008).Washback,impactandconsequences.InN.H.Hornberger&E.Shohamy(Eds.),

EncyclopediaofLanguageandEducation,Volume7:LanguageTestingandAssessment(pp.349-364).Berlin:Springer.

Clyne,M.(2008).ThemonolingualmindsetasanimpedimenttothedevelopmentofplurilingualpotentialinAustralia.SociolinguisticStudies,2(3),347-365.

Coulombe,S.,etal.(2004).InternationalAdultLiteracySurvey:Literacyscores,humancapitalandgrowthacrossfourteenOECDcountries.Ottawa:StatisticsCanada.

CouncilofAustralianGovernments.(2009).NationalIndigenousReformAgreement.RetrievedfromCanberra:http://www.federalfinancialrelations.gov.au/content/national_agreements.aspx

Creagh,S.(2016).AcriticalanalysisoftheLanguageBackgroundOtherThanEnglish(LBOTE)categoryintheAustraliannationaltestingsystem:aFoucauldianperspective.JournalofEducationPolicy,31(3),275-289.

Cronbach,L.J.(1988).Fiveperspectivesonvalidityargument.InH.Wainer&H.I.Braun(Eds.),Testvalidity(pp.3-17).Hillsdale,NewJersey:LawrenceErlbaumAssociates.

David,J.(2011).Whatstudentsneedtolearn:high-stakestestingnarrowsthecurriculum.EducationalLeadership,68(6),78-80.

Devlin,B.(2011).ThestatusandfutureofbilingualeducationforremoteIndigenousstudentsintheNorthernTerritory.AustralianReviewofAppliedLinguistics,34(3).

Devlin,B.,etal.(2017).HistoryofBilingualEducationintheNorthernTerritory(Vol.12).Singapore:SpringerSingapore.

Dixon,S.,&Angelo,D.(2014).Dodgydata,languageinvisibilityandtheimplicationsforsocialinclusion.AustralianReviewofAppliedLinguistics,37(3),213-233.

Dulfer,N.,etal.(2012).Theexperienceofeducation:Theimpactsofhighstakestestingonschoolstudentsandtheirfamilies,Aneducator'sperspective.RetrievedfromSydney:

EducationandEmploymentReferencesCommittee.(2014).EffectivenessoftheNationalAssessmentProgram–LiteracyandNumeracy.RetrievedfromCanberra:http://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Education_and_Employment/Naplan13/Report/index

Education,EmploymentandWorkplaceRelationsReferencesCommittee.(2010).AdministrationandReportingofNAPLANTesting.RetrievedfromCanberra:

18

http://www.aph.gov.au/sitecore/content/Home/Parliamentary_Business/Committees/Senate/Education_Employment_and_Workplace_Relations/Completed_inquiries/2010-13/naplan/report/index

Espeland,W.N.,&Stevens,M.L.(1998).Commensurationasasocialprocess.Annualreviewofsociology,24(1),313-343.

Ferrari,J.(2008,December23).OneinthreeIndigenouskidsfailtest.TheAustralian.Ferrari,J.(2012,December19).ReformsfailingtocloseIndigenousschoolsgap.TheAustralian.Retrieved

fromhttp://www.theaustralian.com.au/national-affairs/education/reforms-failing-to-close-indigenous-schools-gap/news-story/c7b6c260d2d9eee60a49982824382470

Ford,M.(2013).AchievementgapsinAustralia:WhatNAPLANrevealsabouteducationinequalityinAustralia.RaceEthnicityandEducation,16(1),80-102.

Fulcher,G.,&Davidson,F.(2007).Languagetestingandassessment:Anadvancedresourcebook.Abingdon:Routledge.

Genishi,C.,&Brainard,M.(1995).Assessmentofbilingualchildren:Adilemmaseekingsolutions.InE.E.Garcia&B.McLaughlin(Eds.),MeetingtheChallengeofLinguisticandCulturalDiversityinEarlyChildhoodEducation(pp.49-63).NewYork:TeachersCollegePress.

Guenther,J.(2013).ArewemakingeducationcountinremoteAustraliancommunitiesorjustcountingeducation?TheAustralianJournalofIndigenousEducation,42(02),157-170.

Gutierrez-Clellen,V.(1996).Languagediversity:Implicationsforassessment.InK.Cole,P.Dale,&D.Thal(Eds.),AssessmentofCommunicationandLanguage(pp.29-56).Baltimore:PaulH.Brookes.

Hardy,I.(2015).Alogicofenumeration:ThenatureandeffectsofnationalliteracyandnumeracytestinginAustralia.JournalofEducationPolicy,30(3),335-362.

Hoffer,T.B.(2000).Accountabilityineducation.InM.T.Hallinan(Ed.),HandbookoftheSociologyofEducation(pp.529-543):Springer.

JointCommitteeonTestingPractices.(2004).Codeoffairtestingpracticesineducation.Washington,DC:AmericanPsychologicalAssociation.http://www.apa.org/science/programs/testing/fair-testing.pdf(Accessed17January,2017).

Kane,M.T.(2013).Validatingtheinterpretationsandusesoftestscores.JournalofEducationalMeasurement,50(1),1-73.

Kral,I.(2009).TheliteracyquestioninremoteIndigenousAustralia.Canberra:CentreforAboriginalEconomicPolicyResearch,AustralianNationalUniversity.

Kral,I.(2012).Talk,textandtechnology:LiteracyandsocialpracticeinaremoteIndigenouscommunity.Bristol:MultilingualMatters.

Kunnan,A.J.(2000).Fairnessandjusticeforall.InA.J.Kunnan(Ed.),Fairnessandvalidationinlanguageassessment:Selectedpapersfromthe19thLanguageTestingResearchColloquium,Orlando,Florida(Vol.9,pp.1-14).Cambridge:UniversityofCambridge,LocalExamsSyndicate.

Kunnan,A.J.(2004).Testfairness.InM.Milanovic&C.J.Weir(Eds.),EuropeanLanguageTestinginaGlobalContext:ProceedingsoftheALTEBarcelonaConference,July2001(Vol.18,pp.27-48).Cambridge:CambridgeUniversityPress.

Lewis,S.,&Hardy,I.(2017).Trackingthetopological:Theeffectsofstandardiseddatauponteachers’practice.BritishJournalofEducationalStudies,65(2),219-238.

Lingard,B.,etal.(2013).Testingregimes,accountabilitiesandeducationpolicy:commensurateglobalandnationaldevelopments.JournalofEducationPolicy,28(5),539-556.doi:10.1080/02680939.2013.820042

Lingard,B.,&Sellar,S.(2013).‘Catalystdata’:PerversesystemiceffectsofauditandaccountabilityinAustralianschooling.JournalofEducationPolicy,28(5),634-656.

Lingard,B.,etal.(2017).Accountabilitiesinschoolsandschoolsystems.InG.Noblit(Ed.),OxfordResearchEncyclopediaofEducation.Oxford:OxfordUniversityPress.

Lynch,B.K.(2001).Rethinkingassessmentfromacriticalperspective.Languagetesting,18(4),351-372.Mahuteau,S.,etal.(2015).EducationalOutcomesofYoungIndigenousAustralians.Adelaide:Report

submittedtotheNationalCentreforStudentEquityinHigherEducation(NCSEHE),Australia.NationalInstituteofLabourStudies(NILS),FlindersUniversity.

Mann,S.(2011).Acriticalreviewofqualitativeinterviewsinappliedlinguistics.AppliedLinguistics,32(1),6-24.

Marmion,D.,etal.(2014).Community,Identity,Wellbeing:TheReportoftheSecondNationalIndigenousLanguagesSurvey.RetrievedfromCanberra:http://aiatsis.gov.au/publications/products/community-identity-wellbeing-report-second-national-indigenous-languages-survey

19

Martino,W.,&Rezai-Rashti,G.(2013).‘Gaptalk’andtheglobalrescalingofeducationalaccountabilityinCanada.JournalofEducationPolicy,28(5),589-611.

Mathison,S.(2003).Theaccumulationofdisadvantage:Theroleofeducationaltestingintheschoolcareerofminoritychildren.Workplace,10,32-43.

McNamara,T.(2008).Thesocio-politicalandpowerdimensionsoftests.InE.Shohamy&N.H.Hornberger(Eds.),Encyclopediaoflanguageandeducation(2nded.,Vol.7:Languagetestingandassessment,pp.415-427).Dordrecht:Springer.

Meakins,F.(2012).Whichmix—code-switchingoramixedlanguage?—GurindjiKriol.JournalofPidginandCreoleLanguages,27(1),105-140.

Meakins,F.(2014).Languagecontactvarieties.InH.Koch&R.Nordlinger(Eds.),TheLanguagesandLinguisticsofAustralia:Acomprehensiveguide(pp.365-416).Berlin:DeGruyterMouton.

Menken,K.(2006).Teachingtothetest:HowNoChildLeftBehindimpactslanguagepolicy,curriculum,andinstructionforEnglishlanguagelearners.BilingualResearchJournal,30(2),521-546.

Menken,K.(2008).Englishlearnersleftbehind:Standardizedtestingaslanguagepolicy.Clevedon:MultilingualMatters.

Messick,S.(1980).Testvalidityandtheethicsofassessment.AmericanPsychologist,35(11),1012-1027.Messick,S.(1989).Validity.InR.L.Linn(Ed.),EducationalMeasurement(3ed.,pp.13-103).NewYork:

AmericanCouncilonEducationandMacmillan.Messick,S.(1996).Validityandwashbackinlanguagetesting.Languagetesting,13(3),241-256.Miles,M.B.,etal.(2014).Qualitativedataanalysis:Amethodssourcebook(3rded.).ThousandOaks:Sage.Milroy,J.(2001).Languageideologiesandtheconsequencesofstandardization.JournalofSociolinguistics,

5(4),530-555.Milroy,J.(2012).Sociolinguisticsandideologiesinlanguagehistory.Thehandbookofhistorical

sociolinguistics,569-584.MinisterialCouncilonEducation,EmploymentTrainingandYouthAffairs.(2008).Melbournedeclaration

oneducationalgoalsforyoungAustralians.RetrievedfromMelbourne:O'Shannessy,C.(2005).LightWarlpiri:Anewlanguage.AustralianJournalofLinguistics,25(1),31-57.Reay,D.,&Wiliam,D.(1999).’I’llbeanothing’:structure,agencyandtheconstructionofidentitythrough

assessment.BritishEducationalResearchJournal,25(3),343-354.Sahlberg,P.(2015).Finnishlessons2.0:WhatcantheworldlearnfromeducationalchangeinFinland?New

York:TeachersCollegePress.Sellwood,J.,&Angelo,D.(2013).Everywhereandnowhere:InvisibilityofAboriginalandTorresStrait

IslandercontactlanguagesineducationandIndigenouslanguagecontexts.AustralianReviewofAppliedLinguistics,36(3),250-266.

Shohamy,E.(2001).Thepoweroftests:Acriticalperspectiveontheusesoflanguagetests.NewYork:Longman.

Shohamy,E.(2004).Assessmentinmulticulturalsocieties:Applyingdemocraticprinciplesandpracticestolanguagetesting.InB.Norton&K.Toohey(Eds.),Criticalpedagogiesandlanguagelearning(pp.72-92).Cambridge:CambridgeUniversityPress.

Shohamy,E.(2011).Assessingmultilingualcompetencies:Adoptingconstructvalidassessmentpolicies.TheModernLanguageJournal,95(3),418-429.

Simpson,J.H.,etal.(2009).GapsinAustralia'sIndigenouslanguagepolicy:DismantlingbilingualeducationintheNorthernTerritory(0855756713).Retrievedfrom

Solano-Flores,G.(2009).ThetestingofEnglishlanguagelearnersasastochasticprocess:Populationmisspecification,measurementerror,andovergeneralization.InK.Ercikan&W.-M.Roth(Eds.),Generalizingfromeducationalresearch(pp.33-45).NewYork:Routledge.

Star,S.L.,&Lampland,M.(2009).Reckoningwithstandards.InM.Lampland&S.L.Star(Eds.),Standardsandtheirstories:Howquantifying,classifyingandformalizingpracticesshapeeverydaylife(pp.3-24).Ithaca:CornellUniversityPress.

Stevens,M.L.(2008).Cultureandeducation.TheANNALSoftheAmericanAcademyofPoliticalandSocialScience,619(1),97-113.

Talmy,S.(2010).Qualitativeinterviewsinappliedlinguistics:Fromresearchinstrumenttosocialpractice.AnnualReviewofAppliedLinguistics,30,128-148.

Talmy,S.(2011).Theinterviewascollaborativeachievement:Interaction,identity,andideologyinaspeechevent.AppliedLinguistics,32(1),25-42.

Thompson,G.(2014).NAPLAN,MySchoolandaccountability:Teacherperceptionsoftheeffectsoftesting.Internationaleducationjournal:comparativeperspectives,12(2).

Unsworth,P.(2013).StudyingtheeffectsofNAPLANonIndigenouseducationinremoteschools,Australia.(PhDThesis),SchoolofEducation,UniversityofSouthAustralia.

20

Wall,D.(1997).Impactandwashbackinlanguagetesting.InC.Clapham&D.Corson(Eds.),Encyclopediaoflanguageandeducation(Vol.7,pp.291-302).Dordrecht:KluwerAcademicPublishers.

Wigglesworth,G.,etal.(2011).NAPLANlanguageassessmentsforIndigenouschildreninremotecommunities:Issuesandproblems.AustralianReviewofAppliedLinguistics,34(3),320-343.

21

AppendixAInterviewquestions A.INTERVIEWSFORPRINCIPALSThankyoufortakingthetimetoparticipateinthisstudy.AsyouknowwearelookingatNAPLANandwhateffectsithasonschools–thestudents,theteachersandthecommunitymoregenerally.CouldyougivemeageneralpictureofhowNAPLANaffectsyourschoolandthestudents?CouldyoudescribetheNAPLANtestingperiodhere?DoesNAPLANtestinghaveanyeffectonteachingpractices?(Ifso,what…)WhatisyouropinionoftheNAPLANtestpapersandtasks(inrelationtothestudentshere)?WouldyourecommendanychangestoNAPLAN?Doyoufeelpressureaboutyourschool’sperformanceonNAPLAN?HowdoyouthinkyourstudentsandteachersviewNAPLAN?B.FOCUSGROUPSFORTEACHERSCouldyougivemeageneralpictureofhowNAPLANaffectsyourschoolandthestudents?CouldyoudescribetheNAPLANtestingperiodhere?HowdoyouthinkyourstudentsfeelaboutNAPLAN?Doyoufeelpressureaboutyourstudents’performancesonNAPLAN?DoesNAPLANtestinghaveanyeffectonwhatyoudoinclass?(Ifso,what…)WhatisyouropinionoftheNAPLANtestpapersandtasks(inrelationtothestudentshere)?WouldyourecommendanychangestoNAPLAN?

22

Minerva Access is the Institutional Repository of The University of Melbourne

Author/s:

Macqueen, S; Knoch, U; Wigglesworth, G; Nordlinger, R; Singer, R; McNamara, T; Brickle, R

Title:

The impact of national standardized literacy and numeracy testing on children and teaching

staff in remote Australian Indigenous communities

Date:

2019

Citation:

Macqueen, S., Knoch, U., Wigglesworth, G., Nordlinger, R., Singer, R., McNamara, T. &

Brickle, R. (2019). The impact of national standardized literacy and numeracy testing on

children and teaching staff in remote Australian Indigenous communities. Language Testing,

36 (2), pp.265-287. https://doi.org/10.1177/0265532218775758.

Persistent Link:

http://hdl.handle.net/11343/217031

File Description:

Accepted version