the impact of national standardized literacy and numeracy
Post on 16-Oct-2021
4 Views
Preview:
TRANSCRIPT
1
The impact of national standardized literacy and numeracy testing on children andteachingstaffinremoteAustralianIndigenouscommunities.SusyMacqueenAustralianNationalUniversity,AustraliaUteKnochUniversityofMelbourne,AustraliaGillianWigglesworthUniversityofMelbourne,AustraliaRachelNordlingerUniversityofMelbourne,AustraliaRuthSingerUniversityofMelbourne,AustraliaTimMcNamaraUniversityofMelbourne,AustraliaRhiannaBrickleUniversityofMelbourne,Australia
AbstractAll educational testing is intended to have consequences, which are assumed to bebeneficial,buttestsmayalsohaveunintended,negativeconsequences(Messick,1989).Theissue is particularly important in the case of large-scale standardised tests, such asAustralia’sNationalAssessmentProgram–LiteracyandNumeracy(NAPLAN),theintendedbenefits ofwhich are increased accountability and improved educational outcomes. TheNAPLAN purpose is comparable to that of other state and national ‘core skills’ testingprograms which evaluate cross-sections of populations in order to compare resultsbetweenpopulationsub-groupings.Suchcomparisonsunderpin‘accountability’intheeraof population-level testing.This study investigates the impact ofNAPLAN testingononepopulation grouping that is prominent in the NAPLAN results comparisons and publicreporting: children in remote Indigenous communities. A series of interviews withprincipals and teachers documents informants’ first-hand experiences of the use andeffectsofNAPLANinschools.Intheviewsofmostparticipants,thelanguageandcontentofthe test instruments, the nature of the test engagement and the test washback havenegativeimpactsonstudentsandstaff,withlittlebenefitintermsoftheusefulnessofthetestdata.TheprimaryissueisthefactthatmeaningfulparticipationinthetestsdependscriticallyonproficiencyinStandardAustralianEnglish(SAE)asafirstlanguage.Thisstudycontributes to the broader discussion of how reform-targeted standardised testing fornational populations affects sub-groups who are not treated equitably by the testinstrument or reporting for accountability purposes. It highlights a conflict betweenconsequential validity and the notion of accountability which drives reform-targetedtesting.KeywordsTest impact, literacy testing, numeracy testing,NAPLAN, accountability, global educationreformmovement,remoteIndigenouscommunities,commensuration
2
Introduction
Educationalreformmovementsarisein‘standardlanguagecultures’,wherealanguagevarietyhastaken
holdeconomicallyandisimposedthroughinstitutionalmeans(Milroy,2001,2012).Arguably,thereisno
morepowerfulmeansofimposingandmaintainingastandardvarietythanstandardizedliteracytesting,
a phenomenonwhich has become increasinglywidespread to the extent that comparisons using such
testswithinandbetweenjurisdictionsamountto‘globalpanopticism’(Lingard,Martino,&Rezai-Rashti,
2013,p.539).Atthesametime,studieshavedocumenteddetrimentaleffectsofstandardisedtestingon
school curricula and language minority students (see, for example, David, 2011; Dulfer, Polesel, &
Turnbull,2012;Mathison,2003;Menken,2008).Suchtestsareparticularlyproblematicforassessingthe
academic achievement of childrenwho are not proficient in the standard variety (Genishi&Brainard,
1995;Gutierrez-Clellen,1996).Thepresentpaperisacasestudyintheissuesrepresentedbytheseglobal
developmentsandpractices.ItdealswiththeimpactofnationalstandardizedtestinginAustraliaonthe
education of Indigenous children in remote communities where the first language of the children is
generallynotEnglishandwhereEnglishisfirstencounteredonentrytoschool.1
NationalliteracyandnumeracytestinginAustralia
National standardized literacy and numeracy testing in Australia, known as the National Assessment
Program–Literacy and Numeracy (NAPLAN), was introduced in 2008. The testing program replaced
various State-based standardized tests and was introduced ahead of the new national curriculum.
NAPLAN is designed to systematically measure andmonitor the English literacy and numeracy of all
children in years 3, 5, 7 and 9. 2 The test development and review process, accessible
athttps://nap.edu.au/about/test-development, includes details of the test and item development and
reviewprocess,includingthetriallingwhichisconductedwithrepresentativestudentsamplesacrossall
statesandterritories.Thetestsarealsoreviewedbyanexpertpanel,beforeasecondnationalsampleof
studentstakespartinanadditional‘equatingtrial’whichiscarriedoutpriortothemaintesttoensure
thattheresultscanbereportedonthesameassessmentscale.
Deliveredannuallyovera three-dayperiod, the testsassess four ‘domains’:numeracy,writing,
reading, and language conventions.3 The numeracy test covers numerical and algebraic calculations,
measurement and geometry, and statistics and probability. Thewriting test requires one composition
whichiseitherpersuasiveornarrativewriting.Thereadingtestincludesdifferenttexttypesandsamples
skills ranging from finding directly stated information to identifying the tone of an argument. The
languageconventions test targetsspelling,grammarandpunctuation.Besides thewrittencomposition,
most items across the tests aremultiple choice with some constructed response items, e.g. write the
correct spelling. Detailed descriptions of the tests, and sample papers, can be found at
https://www.nap.edu.au/naplan/the-tests.
1Theterm'Indigenous'referstoAboriginalandTorresStraitIslanderpeoples,theoriginalinhabitantsoftheAustraliancontinent.ItisthetermusedinofficialNAPLAN-relateddocuments.2Correspondingagesareapproximately8,10,12and14yearsold.3https://www.nap.edu.au/naplan(Accessed8December,2016)
3
The rationale for the development of NAPLAN arose from a general series of reforms in the
2000swhich increasingly nationalised curriculum and assessment (for political context see Lingard&
Sellar,2013).Theprimaryimpetusforthedevelopmentofanationalstandardizedtestingprogramwasa
declarationbytheMinistersofEducation in2008which inscribednational testingandanemphasison
literacy and numeracy (MCEETYA, 2008) as keymeans to achieving the broad goals of 1) ‘Australian
schooling promotes equity and excellence’ and 2) ‘all young Australians become successful learners,
confidentandcreativeindividuals,andactiveandinformedcitizens’(n.p.).Prominentinthedeclaration
was the intention to improve outcomes for Indigenous youth (to ‘close the gap’) and the provision of
targetedsupportforthispurpose.Itwasalsoexplicitaboutthegatheringof‘goodqualitydata’aboutthe
performanceofschoolstoenablecomparison,andpublicreportingfor‘strengtheningtransparencyand
accountability’ (n.p.). According to theAustralianCurriculumandTestingAuthority (ACARA), the body
responsibleforNAPLANdevelopment,administrationandreporting,thetestsaredesignedto‘determine
whetherornotyoungAustraliansaremeetingimportanteducationaloutcomes’(ACARAwebsite,2016).
ACARA emphasises ‘two benefits’ of the NAPLAN tests: 1) to ‘help drive improvements in student
outcomes’and2)to‘provideincreasedaccountabilityforthecommunity’(ACARAwebsite,2016).4
Inrelationtothefirstbenefit,‘drivingimprovements’,ACARAstates:
All Australian schools benefit from the outcomes of national testing. Schools can gaindetailed informationabouthowtheyareperforming,and theycan identify strengthsandweaknesseswhichmaywarrantfurtherattention.
Inrelationtothesecondbenefit,‘accountability’,ACARAstates:
Australians can expect education resources to be allocated in ways that ensure that allstudentsachieveworthwhilelearningduringtheirtimeatschool.ThereportedoutcomesoftheNAP[NationalAssessmentProgram]enabletheAustralianpublictodevelopageneralnational perspective on student achievement and,more specifically, an understanding ofhowtheirschoolsareperforming.
‘WhyNAP?’ACARAwebsite5
Theseaims–accountability and improvement–arehallmarksof theGlobalEducationReformMovement
(GERM),whichtypicallyusesstandardizedtestinstrumentstoimplementeducationalpolicy(Chalhoub-
Deville, 2016; Sahlberg, 2015). Prominent in a reform agenda, the concept of ‘accountability’ is
unidirectional: teachers and schools are held accountable to governments and the populace through
standardizedtestresultsthataremadepublicthroughsomemeans.AsstatedintheNAPLANstatement
of benefit above, the test results provide the public with ‘a general national perspective on student
achievement’ with a particular focus on ‘how their schools are performing’. Therefore, the test is a
powerfulpolicyenactmentwhichsteersthe ‘systemsandtheworkofschools’(Lingardetal.,2013,p.
539).
4https://www.nap.edu.au/about/why-nap(Accessed8December,2016)5https://www.nap.edu.au/about/why-nap(Accessed2November,2016)
4
GERM testing samples cross-sections of populations for thepurposes of comparisonsbetween
populationsub-groupings,suchasgeographicalareas,e.g.comparisonofAustralianstatesbyNAPLANin
LingardandSellar(2013)orpre-identifiedsocialcategories,e.g.comparisonsofgenderachievementby
ProgrammeforInternationalStudentAssessment(PISA)andPanCanadianAssessmentProgram(PCAP)by
Martino and Rezai-Rashti (2013). These types of comparisons are enabled through a process of
‘commensuration’ whereby different entities (e.g. schools, districts, states) and categorizations (e.g.
geographicalareas,gender,ethnicity)arecomparedbytransferringtheirqualitiesintoacommonmetric
– the test scores (Espeland & Stevens, 1998; Lewis & Hardy, 2017). Commensuration underpins the
notionof‘accountability’intheeraofpopulation-leveleducationaltesting(Hoffer,2000);comparisonsby
categoryon testmeasuresarehowgovernments ‘do’accountability.Theexplicit categorizationswhich
are used to organise NAPLAN scores for accountability-based comparisons are state, sex, Indigenous
status,LanguageBackgroundOtherthanEnglishstatus(LBOTE),parentaloccupation,parentaleducation,
andgeolocation(metropolitan,provincial,remoteandveryremote).6Thistypeofdatacanbeconsidered
‘catalyst data’ because it brings about the need for explanations for differences in test results across
categories (Lingard & Sellar, 2013, p. 635). Thus, the accountability is ostensibly at the level of
government, as it iswithother testingprograms thatprovide system-level comparisons, such asPISA.
However, the publishing of NAPLAN school results on awebsite known asMySchool ensures that the
results are highly visible at a local level.7One use of the website is to enable ‘fair and reasonable
comparisonsamongschoolswith similar students’ througha complexmeasureknownas the ‘Indexof
Community Socio-Educational Advantage’ (ICSEA)which is calculated using geographical location, the
proportion of Indigenous Students, and parents’ education and occupation. 8 Therefore the
commensurationiscarriedoutatthelevelofindividualschoolsviatheICSEAmeasure,NAPLANscores
andthepublicwebsiteasaportalforaccountability.
For language testers, a more familiar perspective on accountability has its roots in validity
theory,whichhas,fornearly50years, includedtheconsequencesoftestingaspartofthediscussionof
test validation (Bachman, 2005; Chalhoub-Deville, 2009; Cronbach, 1988; Kane, 2013; Lynch, 2001;
McNamara,2008;Messick,1989;Shohamy,2001).Theimpactofhigh-stakestests,definedas‘…anyofthe
effects that a testmay have on individuals, policies or practices,within the classroom, the school, the
educationalsystemorsocietyasawhole’(Wall,1997,p.291),isthefocusofaccountabilityconcernsfor
educationaltesters.Inthisview,thepositiveandnegativeeffectsofthetestarepartofthevalidationof
themeaningfulness and usefulness of test scores (e.g. Fulcher&Davidson, 2007;Messick, 1996). This
educationalmeasurementviewofaccountabilityrelatestothedefensibilityoftheinterpretationanduse
ofthetest instrumentinrelationtoitseffectsonpopulationsoftest-takers.Thoseheldaccountablefor
negative consequences are test providers/developers and policymakers, i.e. thosewho effect the test
instrumentandpolicy(alsoseediscussionofresponsibility inChalhoub-Deville,2009). Incontrast, the
GERM notion of accountability is concerned with the defensibility of school and teaching quality in
6In2016thedecisionwasmadetousetheABSAustralianStatisticalGeographyStandard(ASGS)forNAPLAN(majorcities,innerregional,outerregional,remoteandveryremote)http://www.nap.edu.au/information/glossary(Accessed11February,2017)7https://www.myschool.edu.au8https://www.myschool.edu.au/MoreInformation/About(Accessed9January,2017)http://www.acara.edu.au/_resources/About_icsea_2014.pdf(Accessed9January,2017)
5
relationtothecomparedperformancesofparticularsub-groupsoftest-takers.Inreform-targetedtesting,
accountabilityflowsdownwardsothatthoseheldaccountableareschools,teachersandeventest-takers
andtheirparents, i.e. thosewhoareaffectedby thetest-policy(seealsodiscussion inLingard,Sellar,&
Lewis,2017).
In jurisdictions that implement educational reform using standardized tests, the process of
commensurationprovidesametricwhichcombinespersuasivelywiththemetaphorof‘distance’(Lewis
&Hardy,2017),areadily-invokedpolicyhook,e.g. intheUS, ‘nochildleftbehind’and‘racetothetop’,
and,inAustralia,‘closethegap’,whichreferstothedifferenceinaveragescoresofIndigenousAustralian
children and non-Indigenous children.9An Australian policy application of this process is the current
targettohalvetheEnglishliteracyandnumeracygapbetweenIndigenousandnon-Indigenousstudents
by 2018 using 2008 NAPLAN results as baseline data (Council of Australian Governments, 2009). As
Guentherobserves,NAPLANhasbecomeembeddedinthe‘rhetoricofschooling’;thenotionofthe‘gap’
offersadefinitionofeducationaldisadvantage,which,inturn,risksmanufacturingtheunderstandingthat
‘beingAboriginalorTorresStraitIslanderisthedisadvantage’(2013,pp.157-8).
NAPLANandchildreninremoteIndigenouscommunities
ThelanguageecologiesofAustralianremoteIndigenouscommunities
The Indigenous population of Australia constitutes 3% of the total population. Of this Indigenous
population, 22% live in remote or very remote communities scattered across (the north of)Western
Australia,theNorthernTerritory,andQueensland(AustralianProductivityCommission,2014).Inthese
remote communities, the language ecology is varied and complex with high rates of language shift
underway (Marmion, Obata, & Troy, 2014), and the array of languages being spoken in any remote
communityisoftencomplex.ChildrenmaybelearningastheirL1oneofseveraloptions.Somechildren
willgrowuplearningthetraditionallanguageoftheareainwhichtheylive,althoughonlyaround15-18
traditionallanguagescontinuetobelearnedbychildrenasafirstlanguage.Otherchildrenmaybeliving
in a community where language is undergoing rapid shift and they are active in the process of
developing newmixed languages (e.g. Gurindji KriolMeakins, 2012; and LightWarlpiri O'Shannessy,
2005).Alternatively,childrenmaybeacquiringoneofthemanyvarietiesofKriolastheirfirstlanguage
such as Roper River Kriol, Fitzroy Valley Kriol, Barkly Kriol orWumpurrani English, another creole
languagesuchasYumplatokorYarrieLingo,oroneofanumberofdifferentAboriginalEnglishvarieties
(seediscussionsinMeakins,2014;Sellwood&Angelo,2013).Whatallthesechildrenhaveincommonis
that they are growing up in language environments where Standard Australian English (SAE) is not
spokenandtheyarenotacquiringSAEastheirfirstlanguage.Rather,theirenvironmentismuchmore
akintoanEnglishasaForeignLanguage(EFL)environmentthantothatofmigrantchildrenwholivein
English speaking communities. In these remote communities, SAE is not generally spoken among
membersof the Indigenous community and is, for themostpart, limited to someofficial government
activities, health and justice contexts, the classroomand the local shop. Formost children, their first,9Inthecaseofliteracyandnumeracy,the‘gap’isdeterminedbytestscores,butthemetaphorisusedtodescribearangeofpolicyareas,(e.g.health,employment)eachofwhichhasanassociatedprocessorprocessesofcommensuration.
6
moresustainedexposuretoSAEandtowrittenformsoflanguagecomeswhentheystartschoolat5-6
yearsold.
NAPLANandthemonitoringofIndigenouschildren
Indigenouschildrenareasub-groupingoftheAustralianpopulationwhoareidentifiedinthetestdata
and subject to explicitmonitoring via theNAPLAN tests. Thismonitoring has consistently found that
Indigenous childrenarenot achievingat the same level as the restof thepopulationacrossall states
(Ford,2013).Angelo(2013)pointsoutthatthedisaggregationofIndigenousstudents’scoreshasmade
thisgroup‘highlyvisible’andhencesubjectto‘intensiveeducationalinterventions’onthebasisofpoor
NAPLANresults,butwithnorecognitioninthetestdatathatmanyIndigenouschildrenarealsoEnglish
languagelearners(p.54,seealsoDixon&Angelo,2014).
Thetestresultsfromthefirstadministrationin2008indicatedthatthetestscoresofIndigenous
childrenwerelowerthannon-Indigenouschildren,andthatallchildrenlivinginremoteareasachieved
lower scores than their urban peers (ACARA, 2008). Indigenous children in remote areas, however,
scoredlowerthanallremotestudentsacrossallsubtestsatallgradelevels(ACARA,2008).Mediareports
atthetimefocusedontheperformanceofIndigenouschildren,withheadlinesinmajornewspaperssuch
asOneinThreeIndigenousKidsFailTest (Ferrari,2008)and IndigenousStudentsFallingFurtherBehind
(Adlam,2008).The impactof thispressurewasgreatest in theNorthernTerritory(NT),whichhas the
highestpopulationofIndigenouschildrenlivinginremoteareas,veryfewofwhomarenativespeakersof
StandardAustralianEnglish(SAE).TheNTGovernmentmandatedthatallstateschoolsteachinEnglish
forthefirstfourhoursofeachday–apolicywhicheffectivelysawtheendofthefewremainingbilingual
education programs in place, even if ‘programs had been running fairly successfully with strong
communitysupport’(Devlin,2011,p.263;Devlin,Disbray,&Devlin,2017;Simpson,Caffery,&McConvell,
2009).
Since its introduction, therehavebeentwogovernment inquiries intoNAPLAN(Educationand
EmploymentReferencesCommittee,2014;Education,EmploymentandWorkplaceRelationsReferences
Committee, 2010), each ofwhich attracted submissions concerningNAPLAN and Indigenous students.
The 2010 report recommended that ‘ACARA analyse and report publicly on how NAPLAN tests are
servingdifferentgroupsofLanguageBackgroundOtherThanEnglish(LBOTE)students’asaresponseto
the non-recognition of the language learner status of many Indigenous students (p. 18). However, in
dealing specifically with Indigenous student performance, no mention was made of language learner
statusdespitethereportedcommitteeviewthat‘ACARAshouldinvestigatewaystoprovideaccesstorich
information on lower achievers’ (p. 20), an action which would almost certainly reveal that English
languageproficiencyisaprominentvariableformanyIndigenouschildren(seeAngelo,2013).The2014
inquiry noted that NAPLAN contains material which presumes both English-as-a-first-language
developmental levels and experience of urban environments, but the recommendations focussed
predominantlyontheintroductionofacomputeradaptiveversionofthetestwhichwashopedtosolve
thegamutofcomplexissuesraised.
7
ImpactofNAPLAN
In the case of NAPLAN, the most widely reported effect or washback has been a narrowing of the
curriculumthroughagreaterfocusontesttasktypesandtestpractice(Dulferetal.,2012;Hardy,2015;
Thompson, 2014). A narrowed curriculum echoes the experience of other jurisdictions with national
testing programs (e.g. Berliner, 2011; Reay & Wiliam, 1999), and the narrowing can result in
impoverished instruction for students for whom the tests are in a second language (Creagh, 2016;
Menken, 2006, 2008). Narrowing of the curriculum and test task practice which exceeds the
recommended test format familiarizationhavealsobeen reported in schoolswithmajority Indigenous
children in Australia (Angelo, 2012, 2013; Unsworth, 2013). Unsworth’s (2013, p. iii) broad survey of
teachers’viewsofNAPLANin30remoteschoolsalsoidentifiedamisalignmentbetweenthe‘teachingto
the test and teaching to Indigenous students’ funds of knowledge’. Related to thismisalignment is the
identification of cultural inappropriateness of NAPLAN materials for children in remote communities
(Unsworth,2013;Wigglesworth,Simpson,&Loakes,2011),forsomeofwhomsuchforeigncontentcould
onlycompoundtheeffectoftheforeignlanguageofthetest.
The interactionbetween thewriting test task and the second language learner status of some
Indigenous children has also been the subject of research. An analysis of practice narratives for the
NAPLANwritingtestbychildrenwhoareL1speakersofTorresStraitCreoleshowedthatthechildren’s
secondlanguagedevelopmentallevelimpactstheirtestresponses,andfurther,thateventheabilityto‘fill
up’ overtly-taught narrative structures (e.g. introduction, problem, solution and conclusion) requires a
levelofL2proficiencywhichisbeyondtheSAEcompetenceofsomeofthesechildren(Angelo,2012,p.
50).
Method
Researchaim
InresponsetotheconcernsdocumentedintheliteratureandthehighvisibilityofIndigenouschildrenin
the annual NAPLAN reporting and media response, this study investigates the impact of NAPLAN in
remoteIndigenouscommunities.Specifically,weposedtwobroadquestions:
1. From the perspectives of school staff with experience of NAPLAN in remote Indigenous
communities,whataretheimpactsofNAPLANinremoteIndigenouscommunities?
2. Arethereportedeffectspositiveornegative?
Participants
Theintervieweeswereapurposivesampleof23schoolstaffwhohadcombinedexperienceofNAPLANin
20 Indigenous communities classified as ‘very remote’ in Western Australia, Queensland and the
NorthernTerritory.10Thecategoryof‘veryremote’inAustraliaisdeterminedbydistancetothenearest
cities and towns. A very remote community may be hundreds of kilometres from the nearest urban
centre.
10http://www.nap.edu.au/Information/Glossary#l(Accessed3November,2016)
8
Theparticipants’ teachingexperience ranged from four to thirtyyears;18withexperienceas teachers
and 5 with experience as teachers and principals. The small sample size reflects the proportion of
teacherswithexperienceofNAPLANinremotecommunitieswhichisalsoverysmall(asan indication,
around 3% of all primary school teachers worked in remote or very remote areas in 2001).11 Our
selective samplingprocedurematchedour researchquestionby seeking theperspectives of principals
and teachers,both Indigenousandnon-Indigenous,with first-handexperienceof theuseandeffectsof
NAPLANinremoteIndigenouscommunities.
Procedure
Theinterviewsweresemi-structuredandeachlastedapproximately30minutes.Interviewswerecarried
outbymembersoftheresearchteambyphone,bySkypeorface-to-face.Priortotheinterviews,potential
participantswereinformedthattheinterview
willfocusonwhathappensatyourschoolaroundthetimeofNAPLANtestingandwhetherornotNAPLAN has an effect on aspects of schooling such as curriculum, student attendance, studentmotivation,useofhome languagesandEnglish, teachingandassessmentpracticesandanyotheraspects you think are relevant. We will also be interested to hear how NAPLAN affects thecommunitymoregenerallyandanychangesyoubelieveshouldbemadetothetests.
The interview questions were broadly designed to explore any effects the test had on schooling (see
questionsinAppendixA).Participantswerealsoinvitedtoraiseanytopicstheyfeltwererelevanttothe
researchfocus.Ouruseofinterviewsisasmeansofcollectinginformation,ratherthanasocialpractice,
asdistinguishedinTalmy(2011).Onecleardisadvantageofthisapproachisthatittreatsinterviewdata
asindividuallyconstructedbytheinterviewee,ratherthanthecollaborativeeffortbroughtaboutbythe
interviewcontextandthesemi-structured(non-standardized)natureofthequestions(see,forexample,
discussionsinMann,2011;Talmy,2010).ItshouldalsobenotedthatNAPLANisachargedtopicamong
educationalistsinAustralia,withmanyteachersperceivingittohavenegativeeffectsonvariousaspects
ofschooling(Dulferetal.,2012).Thatsaid,webelieveit is intheinterestofthebroadercommunityto
hear theviews,as faraswecanpresent them,of thosewhohaveexperiencedanational standardized
testingprograminthesecomplexlinguisticenvironments.
TheaudiorecordingsweretranscribedandanalysedthematicallyusingNVivosoftwarebytwo
coders inan iterativeanalyticprocessbasedon theprocesses setout inMiles,HubermanandSaldaña
(2014).Thefirststageinvolveddataannotationsontheinterviewtranscripts,aswellasidentificationof
main themes inherent in the interview questions. These annotations and expected themes formed the
basis of broader themes or ‘pattern codes’ (Miles et al., 2014, p. 73). Pattern codes included practical
considerations (e.g. administration), practices (e.g. test strategies), beliefs or perceptions (e.g. cultural
appropriateness), and more theoretically-oriented topics (e.g. usefulness of data). These themes were
trialedcollaborativelybytwoanalystsonadatasubset,followedbycodingofthewholedatasetbyone
coder.Thenextstageinvolvedasynthesisofthesecodedexceptsinto‘higherlevelanalyticmeanings’by
11 http://www.abs.gov.au/ausstats/abs@.NSF/2f762f95845417aeca25706c00834efa/459c3882fad473a2ca2570eb0083be84!OpenDocument (Accessed 26 October, 2017)
9
thesecondanalyst(Milesetal.,2014,p.95).Inthisstage,thepatterncodeswererelatedtotheoretical
aspectsoftestimpact,(forexampleinCheng,2008;Messick,1996;Wall,1997),whichformedthebroad
categoriesunderwhichwepresentthefindingsbelow.Thesethemesareinevitablyselectiveduetothe
richnessoftheinterviewinsights,theconstraintsofspaceandtheneedtoconsiderthedatainrelationto
thebroaderissueswhichareofrelevancetoothercontextsinwhichnationaltestingiscarriedout.
Findings
Thefindingsarepresentedinthreebroadareasrelatingtotestimpact:i)Construct-irrelevantvariance,
ii)Psychosocialimpactsiii)UseofNAPLANresults.
Constructirrelevantvariance
Presumed English language proficiency. All participants agreed that the English language proficiency
status (i.e. English as a second/foreign language or dialect) of Indigenous students in remote
communities is amajor barrier tomeaningful participation inNAPLAN. As Teacher 4 observed,many
childrenfirstencounterEnglishatschoolwhen:
they’veneverseenbooks,they’veneverseenthelanguagewrittendown,theyhaven’tseentheirown
language written down; they come from an oral background, they come in not even speaking
English,andnowwe’regettingthemtositdownandtryanddoaNAPLANtest.Canyouseehow
absurdthatis?
Principal 1 described the enormity of the test English literacy demands in relation to students’ oral
English proficiency: ‘When you struggle to speak a sentence in Standard Australian English, it’s really
hardtowriteanarrative.’
Thewriting testwas generally perceived as posing the greatest challenge becausemeaningful
participationrequiresahighlevelofwrittenEnglishlanguageproficiency.Lessmeaningfulparticipation
was achieved in some cases through the use of strategic genre formulas for the NAPLAN text types
(persuasiveornarrative).Further,therewassomedisparityreportedbetweentheconditionsandlevelof
thewritingtesttaskandthetypicalprocess-basednatureofclassroomwriting.Forexample,Teacher9
notedthatthewritingtasktopiccanbe‘quitedifficulttoaccess’without‘supportandnegotiatingthefield
of discussion before they get going’. Participants described various writing test strategies which
compensate for students’ inability to provide an engaged response. Teacher 3, for example, described
studentssimplywritingallthewordstheyknew.Teacher8explainedthatshewouldtellthestudentsto
writeasentencetheyhadlearntpreviouslywhich‘mightnothaveanythingtodowiththetopicbutthey
mighthaveafullstopandacapitalletterandthat’ssomethingthattheycanputonthepage’.
Thenumeracy test drew comment in relation to theword-basednature ofmany of the items,
whichwasconsideredabarriertodemonstrationofskills:thenumeracytestis‘almostatestofEnglish
comprehension, rather than mathematical ability’ (Teacher 10). One accommodation is that
administrators may read aloud worded parts of the numeracy test (ACARA, 2015). However, it was
reported that in practice, the accommodation made the test delivery time-consuming and incoherent
10
becausethenumericalcomponentofthequestioncannotbereadaloud:‘You’retryingtoreadittoachild
who’sgotEnglishasasecondlanguagebutleavingbitsout’(Teacher8).
Presumed cultural knowledge. Teachers reported instances in which the texts and items in both the
numeracyandliteracytestspresumeexperienceandknowledgetobecommonacrosschildhoods,butare
‘totallyalien’tothelifeexperienceofchildreninremotecommunities(Principal2).Participantsprovided
manyexamplesofconceptswhichmaybefamiliartochildreninurbanareas,butarerelativelyforeignin
a remote community, for example the topic of ‘recycling’ (Principal 3), ‘hosing things into gutters’
(Teacher 10), tenpin bowling (Teacher 3) and going to the cinema (Teacher 14). Issues of cultural
inappropriateness areperhapsmostproblematic in thewriting testwhere thewhole taskhingeson a
singleconcept.Forexample,‘changingalaw’,whichasTeacher15explained,isacomplexandunfamiliar
topicinthecontextofanunfamiliartypeofactivity(thetest).
Bubble-filling.Manyparticipantsreferredto‘bubble-filling’–themultiple-choiceresponseformatwidely
used across the reading, language conventions and numeracy sections – as a test strategy or even an
activity in and of itself, rather than a format for testing literacy or numeracy. Bubble filling gave the
students something they could do ‘even if they didn’t understand the question, theywere able to put
something’(Principal1).Teacher4reportedtellingthestudents ‘topicktheirfavouriteletterandfill it
in’.Teacher2describedtheflurryofbubble-fillingthatoccurredeven ‘beforeyousay, turnyourpaper
over’.Principal2,intheexcerptbelow,indicatesthatforstudentswithverylimitedEnglishproficiency,
therewaslittleengagementwiththetestbeyondbubble-filling:
It’s just, some of themwould say, ‘what do I do here?’. And you would just say, ‘well, fill in the
bubble’.[…]iftheywerecapableofreadingyouwouldsay,‘readthequestionandfillinabubble’.
[…]iftheyweren’tcapableofreadingIwouldjustsay,‘justfillinabubble’.
Notsurprisingly,therewasageneralperceptionthatthetestresultswere,tosomeextent,aproductof
guesswork.
Washbackonclassroomandcurriculum
Theinterviewdatarevealedagreatdealofvariabilityintheschoolpracticesregardingtestpreparation.
Principal 2 explained that her school did ‘nothing specific’ in the lead up to the test except for
familiarisingthestudentswithsomeofthequestiontypes.Teacher3describedaclearbeliefinnotdoing
specific test preparation tasks because, in her view, test practicewas of ‘no use’ as such tasks do not
engagethestudents’abilities.Similarly,Principal1felttheSAEspeakercompetencethatunderpinnedthe
testmade it unfeasible to prepare for the tests in anymeaningfulway: ‘The tests themselvesweren’t
goingtocater forESL[EnglishasaSecondLanguage]students[…]soyouknow,wewereteachingthe
curriculum,weweren’ttryingtoteachthetest’.Incontrast,Teacher1reportedconsiderableamountsof
testpreparation,withteachers‘veryfocusedonteachingtothetest’,particularlytheNAPLANtexttypes.
11
Principal3describedalotofworkonpersuasivetextsandweeklypracticetestswhichhefelthadlifted
theirresults.
Teacher11observedtwoapproachestoNAPLANpreparation.Onewastodopracticetestsand
theotherwastocreateNAPLAN-formatresourcesthatlinkthetesttotheteachingprogram‘sothey’re
gettinganunderstandingofwhatatestisaboutwiththecontentthatthey’refamiliarwith’.Anexample
of thismore learner-basedapproachwastoadaptapopularESLresourceso that it includedNAPLAN-
style multiple choice questions (Teachers 8, 9 and 10). Not surprisingly, multiple choice format was
widelymentionedasapre-taughtfeatureofthetest.Teachers8and14described‘practisingcolouringin
multiple choice’ so that the students didn’t sit there doing nothing. Ironically, Teacher 16 felt that
NAPLANactually‘damagedtheirthirstforliteracyandreading…becausewe’dhavetostopourregular
literacyprogramthatwasworkingsowell,foraweekleadingupandthendothesetestsandthengoback
toitandwe’vemissedtwoweeksofqualityteachingtime.’ThiswasechoedbyTeacher1whofeltthat
‘theamountoftimethatisspentjustfocusingoncontentthatyouthinkisgoingtobethetestprevents
kidsaccessingallareasoflearningandgettingthoserichlearningexperiences.’
Psychosocialimpactsoftestexperience
Psychological impact on students. There were various perceptions of how students were affected
psychologically by the test. These ranged from having little impact to having strong negative effects,
whichmayresultfrom‘theemphasisthattheschoolputsonthetestandtheindividualteacher’(Teacher
1). Some teachers felt the test was simply irrelevant and an inconvenience. Teacher 17, for example,
explained that for her Year 3s, the experience wasn’t ‘overwhelmingly positive or overwhelmingly
negative,itwaskindofanexperiencethatforthemwasprobablyabitnonsensical;theycouldn’treally
understandwhatwe’reaskingthemtodoorwhywe’reaskingthemtodoit.’
Someteachersobservedstrongnegativeeffectonthestudents,intermsofstressorself-esteem.
Teacher14explainedthatdespiteschoolstryingvariousmethodsofmakingthestudentsfeelsupported
andpromotingthechancesofsuccessonthetest,shecouldnotseeanyaspectofthetestingthatsuited
theremoteIndigenouscontextinwhichshetaught.Forher,theEnglishlanguageproficiencyrequiredfor
testparticipationraisedissuesofstudentwellbeing:
Ididn’tfeeltheyneededthatpressureonthembecausethentheywouldjustshutdowncompletely.I
hadonechildduringNAPLANhidingunderthetable,throwingpencilsout,juststressedcompletely
aboutitandbecausetheydidn’tunderstandit,despiteallthepreparation.[…]Ijustlettheminthe
end,writewhatevertheywantedforthewritingsectionandtheyfeltliketheyweresucceedingand
IletthembelievetheyweresucceedingandtoldthemhowproudIwasofthemandthingslikethat
because to me, the number one thing I’m looking after in the classroom is the wellbeing of the
students[…]itwasjustbeinginEnglishitwastoohardforthem.
AttendanceisreportedlyaffectedbyNAPLAN,withstudentsstayingawayonNAPLANdays,or
experiencing one day of testing and then refusing to return for the other test days and reports that
12
parentskepttheirchildrenawayfromschoolonNAPLANdaysoutofconcernforthenegativeimpactof
the test pressure (Teachers 12, 14).While attendance is an issue throughout the school year inmany
remotecommunities,principalsreportedthattheymobilisedattendanceofficersforNAPLAN(Principal
1)orevencollectedchildrenpersonally(Principal2).Teacher3,aNAPLANadministrator,explainedthat
thesporadicattendanceacrosstheNAPLANweekmeantthat‘itwouldoftentake3or4timestoactually
finishit’.Staffalsowenttosomeefforttolettheparentsknowattendancewasespeciallyimportanton
NAPLANdays,forexamplegoingtothestoreoraroundthecommunitytoalertparents.
Psychologicalimpactonteachers.SometeachersfeltthemselvestobenegativelyaffectedbytheNAPLAN
administration, which was ‘possibly harder for the teachers than it is for the students because, as
teachers,we knowwhat the test is andweknowwhat the implications for the results of the test are’
(Teacher17).Someteachersfelttheythemselveswerebeingjudgedbythetestresults,whiletop-down
pressuresassociatedwithNAPLANresults(describedbyTeacher1below)couldaffectthecultureofthe
school:
IthinkNAPLANaffectstheschoolandthestudentsbyactuallychangingthecultureoftheschool.I
thinkit’sbecausethere’sbeenwhat’ssoldasahigh-expectationagendaoranimprovementagenda
[…] and the only thing that is used tomeasurewhether outcomes are improving for students is
NAPLAN.Soprincipalsgetpressurefromtheirregionswhogetpressurefromthecentralofficewho
getpressurefromwhereverthatcomesfrom,fromabove,andthenthatimpactsonprincipalspass
thatpressuredowntoteacherssothat’salmostchangedthecultureofteaching.
Teacher 5 explained that although ‘it was kind of good to see what would be expected of kids in
mainstreameducation’,itwasalso‘disheartening’toseethatherownstudentswere‘sofarbehind’,even
thoughsheacknowledgedthatEnglishmaybeathirdorfourthlanguageforherstudents.Forsome,this
resultedinaverynegativeemotionalresponse,describedbyPrincipal1below:
…almost every year after the tests have finished […] the teachers are generally feeling soul-
destroyed,allfeelingabsolutelydevastated[…]theyfeelsobadaboutputtingthechildrenthrough
it.
Impactonstudent-teacherrelationships.Thetestexperiencewasperceivedbyseveralteachersashaving
anegative effect on the student-teacher relationship.Teachers felt that their inability tohelp students
duringthetesterodedthetrustthatcharacterisesstudent-teacherrelationships.Theyfeltresponsiblefor
imposing an impossible task on the students at the same time as refusing assistance which would
normallybeprovided.Forexample,Teacher9describeditas‘breakingthatbondinthatyou’regivingthe
studentssomethingthatyouknowtheycan’tdoandyou’renotgivingthemanysupport’.
To counteract the negative effects on the students, there was also an element of protection
expressedwhereteacherstriedtoinsulateandcompensatefortheperceivedrisktostudents’esteem.For
example, teachers gave ‘pep-talks’ (Teacher 10) or ‘played it down’ (Teacher 11) as well as giving
rewardsorpraiseattheendofthetest(Teacher4).Teacher14explainedthatshewas‘neverhonestwith
13
thekids’,preferringtomake‘itseemlikeafunthing’withthembecauseshedidn’twantittoaffecttheir
self-esteem.
UseofNAPLANresults
AlthoughparticipantsindicatedthattheytooknoteofNAPLANresults,itwaswidelyexpressedthatthe
NAPLAN data has little relevance or usefulness for schools or parents. Comparisons between schools
usingtheMySchoolwebsitewere foundtobeunhelpfulduetothesmall,variablecohortsdescribedby
Principal1above.Principal4saidthattheschoolusedsomeoftheresults,but ‘attheendofthedayit
doesn't showusahellofa lot’. Inaddition, theverysmall,variablecohortsmade the interpretationof
dataproblematic,assuggestedinthefollowingexcerpt:
we spenta lot of time lookingat thedata, and I still havenightmares sometimesaboutboxand
whiskerdataanalysiswithgraphs[…]TheYear5sthisyearin2007didthisandin2009, lookat
this improvement, you can work on that! And you’re trying to explain to them, look, they were
completelydifferentkids;wehadawhitekidintheclassin2009andthat’swhytheresults[are]so
muchbetter.ThisisoutofaclassofaboutfourYear7’sorsomething(Principal1)
Therewereinstancesinwhichresultswereusedincurriculumplanning:inordertoaddressweaknesses
(Principal 3) or decidewhich areas ofmaths to cover (Teacher 10). Teacher 1 described sharing and
analysingtheresults,lookingat‘whichyearlevelsdidwellinwhattest[…]andwhatteacherstheythink
hadwhatimpactonwhatresults’.Sheperceivedthisprocessput‘alotofpressure’onteachers.
At the level of the jurisdiction, Teacher 9 described how ‘every year the NT government gets
kicked’ at the release of NAPLAN results. He explained that despite government reviews identifying
bilingualeducationasthebestevidenced-basedpractice,NAPLANresultsandgovernment-setNAPLAN
goalsputdestructivepressureonbilingualprograms.
Discussion
These data, from teachers and principals who have experienced firsthand the effects of NAPLAN in
remoteIndigenouscommunities,showthatthereiscauseforconcernabouttheappropriatenessoftest
useinthesecontextsandtheresultingeffectsofthemismatchbetweenstudentandteacherneedsonthe
onehand,andtest instrumentsandusefulnessofresultsontheother.Onthewhole,staffdidnotthink
thatthetestresultsreflectedtheabilitiesofthestudentsforfourmain,interrelatedreasons,thefirstof
whichisprimary:1)Englishlanguageisnotthefirstlanguageandinsomecases,notthefirstliteracyof
the children 2) the test content is inaccessible or foreign 3) a large proportion of children draw on
construct-irrelevant test strategies such as bubble-filling 4) the tests do not engage the children
cognitively or socioculturally. In the view of most participants, these characteristics of the test
instruments(language/content)andtest-takerbehaviours(testengagement)havenegativepsychosocial
impacts on students and staff,with little or no benefit in termsof the usefulness of the test data. The
reportedwashback on classroom and curriculum amounts to superficial test-survival strategieswhich
14
potentiallyevendamagestudents’literacydevelopment,ratherthanpromoteit(seealsoAngelo,2013).
Intheviewsoftheparticipantsinthisstudy,theNAPLANexperienceandeffectsappeartobe,atbest,an
irrelevance and atworst, the source of considerable distress. This finding echoes other surveys in the
AustraliancontextthathaveconcludedthatmostteachersfeeltheeffectsofNAPLANhavebeenlargely
negative(Dulferetal.,2012;Thompson,2014).
TheseeffectsarecompoundedbytheveryhighvisibilityofIndigenouschildreninthenational
analysis,whichdisaggregatesanddiscusses thisgroup’sperformanceannually.However,arguably,not
includingIndigenouschildreninremotecommunitiesinanationaltestingprogramwouldseparatethis
ethnoculturalgroupinwaysthatarealsopotentiallydetrimental.Itisimportanttonotethatparticipants
werenotopposedtostandardizedtestingperse,butrathertothepresumptionofEnglishnative-speaker
language competence and western urban culture that they perceived to underpin test constructs.
Connectedtothisisthelackofdiscernmentatthelowerendofattainment;asTeacher9explained,his
studentswere‘prettymuchalldeemedtofailequally’eventhoughheknewtheywerenotallatthesame
level.
A fundamental issue is that theNAPLAN test constructsareunderpinnedbyEnglish-as-a-first-
language literacy (Angelo, 2013; Creagh, 2016). Writing from the US context, Solano-Flores (2009)
observesthattheinclusionofEnglishlanguagelearnersinnationaltesting‘hasnotbeenaccompaniedby
effectiveapproachestominimizeconstruct-irrelevantscorevarianceduetolanguageissues’(p.33).He
characterisesthetestingofsuchlearnersasa‘stochasticprocess’duetouncertaintyaboutvariablessuch
as sensitivity to contextual features, and language of the test delivery (Solano-Flores, 2009, p. 43). As
observedby theparticipants in thisstudy,even thenumeracy test is rendered inaccessible through its
heavilyEnglish-languageconstruct.Toexemplifythis,ofthe120itemsonthefourYear3numeracytests
from2008to2011,therewerejusttwoitemscomposedonlyofnumericalsymbolsorshapes.Allother
items require anunderstandingofwrittenEnglish formeaningful engagement and, inmany instances,
thesearecomplicatedsyntacticstructures,suchas‘InwhichgamedidKimscore3moregoalsthanshe
scoredinGame2?’(NAPLANYear3NumeracyTest,2008).TheNAPLANconstructs,therefore,areapoor
fitwiththelanguageecologiesofmanyremoteIndigenouscommunities,wherethereislimitedpresence
ofprint-based literacy inany language. Indigenouschildren in remotecommunitieshavegenerallynot
acquiredanyofthepre-literacyskillsineitherEnglishortheirhomelanguage/sthattheirurban,English-
speaking counterparts are typically equippedwith in readiness for schooling (Kral, 2012). This has a
profoundeffectontheiracquisitionofEnglishacademicliteracy,andinevitably,ontheirperformanceon
NAPLAN.
Messick(1980)remindsus that thequestionof test fairnessariseswhenan individual’sgroup
membershipdeterminesthatheorsheachievessignificantlydifferentlyfromthepopulationasawhole.
AsmeasuredbyNAPLAN,thesignificantlydifferentperformanceofremoteIndigenousstudentshasbeen
sustained and highly visible in reporting and media in every test administration. The quantification
providedbyNAPLANremainsprominentinpublicdiscourseabout‘closingthegap’,withannualrevisits
inthemediaatNAPLANreportingtimewhichporeovertheunclosedgap(e.g.'ReformsFailingtoClose
Indigenous Schools Gap', Ferrari, 2012). In addition to the effects that these participants reported in
relation to students, teachers, classroom and curriculum, the commensuration, i.e. the comparison
15
differentgroupingsonacommonmetric(Espeland&Stevens,1998),providedbyNAPLANhasbecome
absorbed in Australian educational and public discourse; it has become ‘common sense’ (Star &
Lampland, 2009). An easily-comprehendedmetric, NAPLAN belies the nature of schooling in complex
sociolinguistic contexts, thus renderingan ‘empirically intricate realitydeceptively straightforward’ for
the purpose of comparison (Stevens, 2008, p. 102). Thanks to this systemic absorption, ‘the numbers
themselvescometoberegardedascoextensivewiththeunderlyingphenomenatheymeasure’(Stevens,
2008, p. 102). This contribution of test-user perspectives has interrogated this assumption and
highlightedthecomplexityunderpinningthemetric.
Conclusion
StandardizedtestingprogramssuchasNAPLANareintendedtobringaboutchangeintheliteracylevel
(as an indicator and proxy for ‘educational outcomes’) of the population. The test enacts a policy
underpinned by efforts to increase life opportunities at an individual level (e.g. Mahuteau, Karmel,
Mavromaras,&Zhu,2015)andacorrelationbetween literacyandeconomicgrowthatanational level
(e.g.Coulombe,Tremblay,&Marchand,2004).Thebodyresponsible forNAPLAN,ACARA,hasstressed
thattheuseofNAPLANresultsactsasimpetusinthiscausationalchainbelow:
ItshouldbeemphasisedthatNAPLANisatoolto informschool improvement,notanimproverof
educationaloutcomes.Itisnottheteststhatwillimprovestudents’literacyandnumeracyskills,but
thewaystudents’results(includingschool,systemandnationallevelresults)areusedbyteachers,
schools and systems to identify strengths andweaknesses, particularly in teaching practices and
programs,thatwillimprovestudentoutcomes.
(ACARAsubmission,reportedinNAPLANInquiryReport,EducationandEmploymentReferences
Committee,2014,p.58)
Asdiscussedearlier,thetestresultsare‘catalyst’fortheaccountabilityprocesswhichapplies,according
to ACARA, ‘particularly in teaching practices and programs’. Yet, this method of accountability is not
necessarilyademocraticone.Shohamypointsoutthat,althoughstandardizedtesting‘maybebasedon
equal opportunities and conditions of testing’, it does not necessarily follow that ‘all test takers have
equalopportunitytoobtainanequaleducation’(2004,p.87).Indeed,sheliststheimpositionofteststhat
are‘basedononetypeofknowledge’ontest-takersfromdifferentbackgroundsasapossibleviolationof
test-takerrights.InrelationtoIndigenouschildreninremotecommunities,itisunrealistic,asKral(2009,
2012) points out, to assume that the development of academic English literacy can be achieved in a
foreign language through brief pedagogical interventions to a level comparable to that ofmiddle-class
Australia(seealsoAngelo,2013).Arguably,practiceswehavecometorecogniseasindicationsoffairness
ineducationaltestingsuchas‘carefulstandardizationoftestsandadministrationconditions’intheCode
ofFairTestingPracticesinEducation(JointCommitteeonTestingPractices,2004)maywellbethecause
of ill-effect when the context, the educational environment and the test-taker linguistic background
differssomarkedlyfromtheliteracypracticesvaluedbythe ‘standardlanguageculture’(Milroy,2001)
16
and onwhich the construct ismodelled. According toAngelo andDixon (2014), this has resulted in a
widespread ignorance of the value of multilingualism or need for English language learning for
Indigenous children. In their view, Australia’s ‘monolingual mindset’ (Clyne, 2008; see also Shohamy,
2011)hasassistedthesafepassageofNAPLAN‘literacyandnumeracytesting’intoschoolingasifitwere
not in any language and hence fair, even for studentswho are speakers of Indigenous traditional and
contactvarieties,withlittleEnglishproficiency.AsGuenther(2013)concludes,‘theassumptionsbehind
theadministrationoftestinginstruments,thelanguageusedinthemandthevaluesembeddedwithinthe
testsshouldbequestioned’(p.167).Ifanidentifiablesub-populationoftest-takersareunabletoengage
with test content because of their proficiency in the language of delivery, the test has poor
‘interactiveness’, inBachmanandPalmer’s(1996)testusefulness framework.Notbeingable toengage
with the test instrument precludesmeaningful results and thus also the kind of reform envisaged by
policymakers.The‘interactiveness’observedbyourintervieweesintheNAPLANappearedtobelimited
tofillinginthebubbleswithnosenseofwhetherornottheresponsewascorrect.
Thisbringsusbacktothepropositionthataccountabilityineducationalreformrunscounterto
accountabilityinconsequentialvalidity.Shohamy’sview,basedonthenotionofconsequentialvalidityin
educationalmeasurement,isthat ‘thereshouldbeaccountabilitywithregardtothepurposeofthetest,
its practice and its methods’ (2004, p. 87). Those responsible for the design and use of the test –
developers,providers,policymakers–needtoensurethattheinstrumentdoesnoharm(Kunnan,2000,
2004).Itisclear,however,thattheteachersandprincipalsinthisstudyfeeltheonusisonthemtodefend
thepoorperformancesoftheirstudents inaGERMsenseofaccountability.Caughtbetweenapowerful
policy instrumentand theneedsof theirstudents,manyparticipantsalso felt theneed toprotect their
students from theunintendednegative effectsof the test instruments.These findings suggest that this
processofcommensuration—thequantitativecomparisonofthequalitiesofdifferentsub-groups—isnot
informative about or beneficial for children in remote Indigenous communities. For comparison to be
carriedout,thenthereisaclearneedforaninstrumentwhichissensitivetothedevelopmentofliteracy
andnumeracyinEnglishasaforeignlanguagewhichisaccessibleforthesestudents.Atalocallevel,such
aninstrumentshouldbeabletodiscernlearninginthesecontextsinawaythatmightbeinformativefor
teaching andbeneficial for learning.At anational level, an instrument that recognises themultilingual
capacity of many Australian children has the potential to disrupt the myth of monolingualism in
Australia’sclassrooms.
17
References
Adlam,N.(2008).Indigenousstudentsfallingfurtherbehind.NorthernTerritoryNews/Sunday
Territorian.Angelo,D.(2012).Sadstories:ApreliminarystudyofNAPLANpracticetextsanalysingstudents’second
languagelinguisticresourcesandtheeffects.PaperpresentedattheProceedingsofthe42ndAustralianLinguisticSocietyConference-2011.
Angelo,D.(2013).NAPLANimplementation:Implicationsforclassroomlearningandteaching,withrecommendationsforimprovement.TESOLinContext,23(1/2),53.
AustralianCurriculumandReportingAuthority.(2015).NAPLAN2015Nationalprotocolsfortestadministration:Downloadedfrom:http://nap.org.au/naplan/school-support/national-protocols.html(Accesed17November,2015).
AustralianCurriculumAssessmentandReportingAuthority.(2008).2008NationalAssessmentProgramLiteracyandNumeracyAchievementinReading,Writing,LanguageConventionsandNumeracy.RetrievedfromSydney:https://www.nap.edu.au/results-and-reports/national-reports
https://www.nap.edu.au/_resources/2ndStageNationalReport_18Dec_v2.pdfAustralianProductivityCommission.(2014).OvercomingIndigenousdisadvantage:Keyindicators2014.
Canberra:AustralianProductivityCommission.Bachman,L.F.(2005).Buildingandsupportingacasefortestuse.LanguageAssessmentQuarterly,2(1),
1-34.Bachman,L.F.,&Palmer,A.S.(1996).Languagetestinginpractice:Designinganddevelopinguseful
languagetests.Oxford:OxfordUniversityPress.Berliner,D.(2011).Rationalresponsestohighstakestesting:Thecaseofcurriculumnarrowingandthe
harmthatfollows.CambridgeJournalofEducation,41(3),287-302.Chalhoub-Deville,M.(2009).Theintersectionoftestimpact,validation,andeducationalreformpolicy.
AnnualReviewofAppliedLinguistics,29,118-131.Chalhoub-Deville,M.(2016).Validitytheory:Reformpolicies,accountabilitytesting,andconsequences.
Languagetesting,33(4),453-472.Cheng,L.(2008).Washback,impactandconsequences.InN.H.Hornberger&E.Shohamy(Eds.),
EncyclopediaofLanguageandEducation,Volume7:LanguageTestingandAssessment(pp.349-364).Berlin:Springer.
Clyne,M.(2008).ThemonolingualmindsetasanimpedimenttothedevelopmentofplurilingualpotentialinAustralia.SociolinguisticStudies,2(3),347-365.
Coulombe,S.,etal.(2004).InternationalAdultLiteracySurvey:Literacyscores,humancapitalandgrowthacrossfourteenOECDcountries.Ottawa:StatisticsCanada.
CouncilofAustralianGovernments.(2009).NationalIndigenousReformAgreement.RetrievedfromCanberra:http://www.federalfinancialrelations.gov.au/content/national_agreements.aspx
Creagh,S.(2016).AcriticalanalysisoftheLanguageBackgroundOtherThanEnglish(LBOTE)categoryintheAustraliannationaltestingsystem:aFoucauldianperspective.JournalofEducationPolicy,31(3),275-289.
Cronbach,L.J.(1988).Fiveperspectivesonvalidityargument.InH.Wainer&H.I.Braun(Eds.),Testvalidity(pp.3-17).Hillsdale,NewJersey:LawrenceErlbaumAssociates.
David,J.(2011).Whatstudentsneedtolearn:high-stakestestingnarrowsthecurriculum.EducationalLeadership,68(6),78-80.
Devlin,B.(2011).ThestatusandfutureofbilingualeducationforremoteIndigenousstudentsintheNorthernTerritory.AustralianReviewofAppliedLinguistics,34(3).
Devlin,B.,etal.(2017).HistoryofBilingualEducationintheNorthernTerritory(Vol.12).Singapore:SpringerSingapore.
Dixon,S.,&Angelo,D.(2014).Dodgydata,languageinvisibilityandtheimplicationsforsocialinclusion.AustralianReviewofAppliedLinguistics,37(3),213-233.
Dulfer,N.,etal.(2012).Theexperienceofeducation:Theimpactsofhighstakestestingonschoolstudentsandtheirfamilies,Aneducator'sperspective.RetrievedfromSydney:
EducationandEmploymentReferencesCommittee.(2014).EffectivenessoftheNationalAssessmentProgram–LiteracyandNumeracy.RetrievedfromCanberra:http://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Education_and_Employment/Naplan13/Report/index
Education,EmploymentandWorkplaceRelationsReferencesCommittee.(2010).AdministrationandReportingofNAPLANTesting.RetrievedfromCanberra:
18
http://www.aph.gov.au/sitecore/content/Home/Parliamentary_Business/Committees/Senate/Education_Employment_and_Workplace_Relations/Completed_inquiries/2010-13/naplan/report/index
Espeland,W.N.,&Stevens,M.L.(1998).Commensurationasasocialprocess.Annualreviewofsociology,24(1),313-343.
Ferrari,J.(2008,December23).OneinthreeIndigenouskidsfailtest.TheAustralian.Ferrari,J.(2012,December19).ReformsfailingtocloseIndigenousschoolsgap.TheAustralian.Retrieved
fromhttp://www.theaustralian.com.au/national-affairs/education/reforms-failing-to-close-indigenous-schools-gap/news-story/c7b6c260d2d9eee60a49982824382470
Ford,M.(2013).AchievementgapsinAustralia:WhatNAPLANrevealsabouteducationinequalityinAustralia.RaceEthnicityandEducation,16(1),80-102.
Fulcher,G.,&Davidson,F.(2007).Languagetestingandassessment:Anadvancedresourcebook.Abingdon:Routledge.
Genishi,C.,&Brainard,M.(1995).Assessmentofbilingualchildren:Adilemmaseekingsolutions.InE.E.Garcia&B.McLaughlin(Eds.),MeetingtheChallengeofLinguisticandCulturalDiversityinEarlyChildhoodEducation(pp.49-63).NewYork:TeachersCollegePress.
Guenther,J.(2013).ArewemakingeducationcountinremoteAustraliancommunitiesorjustcountingeducation?TheAustralianJournalofIndigenousEducation,42(02),157-170.
Gutierrez-Clellen,V.(1996).Languagediversity:Implicationsforassessment.InK.Cole,P.Dale,&D.Thal(Eds.),AssessmentofCommunicationandLanguage(pp.29-56).Baltimore:PaulH.Brookes.
Hardy,I.(2015).Alogicofenumeration:ThenatureandeffectsofnationalliteracyandnumeracytestinginAustralia.JournalofEducationPolicy,30(3),335-362.
Hoffer,T.B.(2000).Accountabilityineducation.InM.T.Hallinan(Ed.),HandbookoftheSociologyofEducation(pp.529-543):Springer.
JointCommitteeonTestingPractices.(2004).Codeoffairtestingpracticesineducation.Washington,DC:AmericanPsychologicalAssociation.http://www.apa.org/science/programs/testing/fair-testing.pdf(Accessed17January,2017).
Kane,M.T.(2013).Validatingtheinterpretationsandusesoftestscores.JournalofEducationalMeasurement,50(1),1-73.
Kral,I.(2009).TheliteracyquestioninremoteIndigenousAustralia.Canberra:CentreforAboriginalEconomicPolicyResearch,AustralianNationalUniversity.
Kral,I.(2012).Talk,textandtechnology:LiteracyandsocialpracticeinaremoteIndigenouscommunity.Bristol:MultilingualMatters.
Kunnan,A.J.(2000).Fairnessandjusticeforall.InA.J.Kunnan(Ed.),Fairnessandvalidationinlanguageassessment:Selectedpapersfromthe19thLanguageTestingResearchColloquium,Orlando,Florida(Vol.9,pp.1-14).Cambridge:UniversityofCambridge,LocalExamsSyndicate.
Kunnan,A.J.(2004).Testfairness.InM.Milanovic&C.J.Weir(Eds.),EuropeanLanguageTestinginaGlobalContext:ProceedingsoftheALTEBarcelonaConference,July2001(Vol.18,pp.27-48).Cambridge:CambridgeUniversityPress.
Lewis,S.,&Hardy,I.(2017).Trackingthetopological:Theeffectsofstandardiseddatauponteachers’practice.BritishJournalofEducationalStudies,65(2),219-238.
Lingard,B.,etal.(2013).Testingregimes,accountabilitiesandeducationpolicy:commensurateglobalandnationaldevelopments.JournalofEducationPolicy,28(5),539-556.doi:10.1080/02680939.2013.820042
Lingard,B.,&Sellar,S.(2013).‘Catalystdata’:PerversesystemiceffectsofauditandaccountabilityinAustralianschooling.JournalofEducationPolicy,28(5),634-656.
Lingard,B.,etal.(2017).Accountabilitiesinschoolsandschoolsystems.InG.Noblit(Ed.),OxfordResearchEncyclopediaofEducation.Oxford:OxfordUniversityPress.
Lynch,B.K.(2001).Rethinkingassessmentfromacriticalperspective.Languagetesting,18(4),351-372.Mahuteau,S.,etal.(2015).EducationalOutcomesofYoungIndigenousAustralians.Adelaide:Report
submittedtotheNationalCentreforStudentEquityinHigherEducation(NCSEHE),Australia.NationalInstituteofLabourStudies(NILS),FlindersUniversity.
Mann,S.(2011).Acriticalreviewofqualitativeinterviewsinappliedlinguistics.AppliedLinguistics,32(1),6-24.
Marmion,D.,etal.(2014).Community,Identity,Wellbeing:TheReportoftheSecondNationalIndigenousLanguagesSurvey.RetrievedfromCanberra:http://aiatsis.gov.au/publications/products/community-identity-wellbeing-report-second-national-indigenous-languages-survey
19
Martino,W.,&Rezai-Rashti,G.(2013).‘Gaptalk’andtheglobalrescalingofeducationalaccountabilityinCanada.JournalofEducationPolicy,28(5),589-611.
Mathison,S.(2003).Theaccumulationofdisadvantage:Theroleofeducationaltestingintheschoolcareerofminoritychildren.Workplace,10,32-43.
McNamara,T.(2008).Thesocio-politicalandpowerdimensionsoftests.InE.Shohamy&N.H.Hornberger(Eds.),Encyclopediaoflanguageandeducation(2nded.,Vol.7:Languagetestingandassessment,pp.415-427).Dordrecht:Springer.
Meakins,F.(2012).Whichmix—code-switchingoramixedlanguage?—GurindjiKriol.JournalofPidginandCreoleLanguages,27(1),105-140.
Meakins,F.(2014).Languagecontactvarieties.InH.Koch&R.Nordlinger(Eds.),TheLanguagesandLinguisticsofAustralia:Acomprehensiveguide(pp.365-416).Berlin:DeGruyterMouton.
Menken,K.(2006).Teachingtothetest:HowNoChildLeftBehindimpactslanguagepolicy,curriculum,andinstructionforEnglishlanguagelearners.BilingualResearchJournal,30(2),521-546.
Menken,K.(2008).Englishlearnersleftbehind:Standardizedtestingaslanguagepolicy.Clevedon:MultilingualMatters.
Messick,S.(1980).Testvalidityandtheethicsofassessment.AmericanPsychologist,35(11),1012-1027.Messick,S.(1989).Validity.InR.L.Linn(Ed.),EducationalMeasurement(3ed.,pp.13-103).NewYork:
AmericanCouncilonEducationandMacmillan.Messick,S.(1996).Validityandwashbackinlanguagetesting.Languagetesting,13(3),241-256.Miles,M.B.,etal.(2014).Qualitativedataanalysis:Amethodssourcebook(3rded.).ThousandOaks:Sage.Milroy,J.(2001).Languageideologiesandtheconsequencesofstandardization.JournalofSociolinguistics,
5(4),530-555.Milroy,J.(2012).Sociolinguisticsandideologiesinlanguagehistory.Thehandbookofhistorical
sociolinguistics,569-584.MinisterialCouncilonEducation,EmploymentTrainingandYouthAffairs.(2008).Melbournedeclaration
oneducationalgoalsforyoungAustralians.RetrievedfromMelbourne:O'Shannessy,C.(2005).LightWarlpiri:Anewlanguage.AustralianJournalofLinguistics,25(1),31-57.Reay,D.,&Wiliam,D.(1999).’I’llbeanothing’:structure,agencyandtheconstructionofidentitythrough
assessment.BritishEducationalResearchJournal,25(3),343-354.Sahlberg,P.(2015).Finnishlessons2.0:WhatcantheworldlearnfromeducationalchangeinFinland?New
York:TeachersCollegePress.Sellwood,J.,&Angelo,D.(2013).Everywhereandnowhere:InvisibilityofAboriginalandTorresStrait
IslandercontactlanguagesineducationandIndigenouslanguagecontexts.AustralianReviewofAppliedLinguistics,36(3),250-266.
Shohamy,E.(2001).Thepoweroftests:Acriticalperspectiveontheusesoflanguagetests.NewYork:Longman.
Shohamy,E.(2004).Assessmentinmulticulturalsocieties:Applyingdemocraticprinciplesandpracticestolanguagetesting.InB.Norton&K.Toohey(Eds.),Criticalpedagogiesandlanguagelearning(pp.72-92).Cambridge:CambridgeUniversityPress.
Shohamy,E.(2011).Assessingmultilingualcompetencies:Adoptingconstructvalidassessmentpolicies.TheModernLanguageJournal,95(3),418-429.
Simpson,J.H.,etal.(2009).GapsinAustralia'sIndigenouslanguagepolicy:DismantlingbilingualeducationintheNorthernTerritory(0855756713).Retrievedfrom
Solano-Flores,G.(2009).ThetestingofEnglishlanguagelearnersasastochasticprocess:Populationmisspecification,measurementerror,andovergeneralization.InK.Ercikan&W.-M.Roth(Eds.),Generalizingfromeducationalresearch(pp.33-45).NewYork:Routledge.
Star,S.L.,&Lampland,M.(2009).Reckoningwithstandards.InM.Lampland&S.L.Star(Eds.),Standardsandtheirstories:Howquantifying,classifyingandformalizingpracticesshapeeverydaylife(pp.3-24).Ithaca:CornellUniversityPress.
Stevens,M.L.(2008).Cultureandeducation.TheANNALSoftheAmericanAcademyofPoliticalandSocialScience,619(1),97-113.
Talmy,S.(2010).Qualitativeinterviewsinappliedlinguistics:Fromresearchinstrumenttosocialpractice.AnnualReviewofAppliedLinguistics,30,128-148.
Talmy,S.(2011).Theinterviewascollaborativeachievement:Interaction,identity,andideologyinaspeechevent.AppliedLinguistics,32(1),25-42.
Thompson,G.(2014).NAPLAN,MySchoolandaccountability:Teacherperceptionsoftheeffectsoftesting.Internationaleducationjournal:comparativeperspectives,12(2).
Unsworth,P.(2013).StudyingtheeffectsofNAPLANonIndigenouseducationinremoteschools,Australia.(PhDThesis),SchoolofEducation,UniversityofSouthAustralia.
20
Wall,D.(1997).Impactandwashbackinlanguagetesting.InC.Clapham&D.Corson(Eds.),Encyclopediaoflanguageandeducation(Vol.7,pp.291-302).Dordrecht:KluwerAcademicPublishers.
Wigglesworth,G.,etal.(2011).NAPLANlanguageassessmentsforIndigenouschildreninremotecommunities:Issuesandproblems.AustralianReviewofAppliedLinguistics,34(3),320-343.
21
AppendixAInterviewquestions A.INTERVIEWSFORPRINCIPALSThankyoufortakingthetimetoparticipateinthisstudy.AsyouknowwearelookingatNAPLANandwhateffectsithasonschools–thestudents,theteachersandthecommunitymoregenerally.CouldyougivemeageneralpictureofhowNAPLANaffectsyourschoolandthestudents?CouldyoudescribetheNAPLANtestingperiodhere?DoesNAPLANtestinghaveanyeffectonteachingpractices?(Ifso,what…)WhatisyouropinionoftheNAPLANtestpapersandtasks(inrelationtothestudentshere)?WouldyourecommendanychangestoNAPLAN?Doyoufeelpressureaboutyourschool’sperformanceonNAPLAN?HowdoyouthinkyourstudentsandteachersviewNAPLAN?B.FOCUSGROUPSFORTEACHERSCouldyougivemeageneralpictureofhowNAPLANaffectsyourschoolandthestudents?CouldyoudescribetheNAPLANtestingperiodhere?HowdoyouthinkyourstudentsfeelaboutNAPLAN?Doyoufeelpressureaboutyourstudents’performancesonNAPLAN?DoesNAPLANtestinghaveanyeffectonwhatyoudoinclass?(Ifso,what…)WhatisyouropinionoftheNAPLANtestpapersandtasks(inrelationtothestudentshere)?WouldyourecommendanychangestoNAPLAN?
22
Minerva Access is the Institutional Repository of The University of Melbourne
Author/s:
Macqueen, S; Knoch, U; Wigglesworth, G; Nordlinger, R; Singer, R; McNamara, T; Brickle, R
Title:
The impact of national standardized literacy and numeracy testing on children and teaching
staff in remote Australian Indigenous communities
Date:
2019
Citation:
Macqueen, S., Knoch, U., Wigglesworth, G., Nordlinger, R., Singer, R., McNamara, T. &
Brickle, R. (2019). The impact of national standardized literacy and numeracy testing on
children and teaching staff in remote Australian Indigenous communities. Language Testing,
36 (2), pp.265-287. https://doi.org/10.1177/0265532218775758.
Persistent Link:
http://hdl.handle.net/11343/217031
File Description:
Accepted version
top related