educational measurement and school accountability directorate the small schools dilemma – reliable...

Educational Measurement and School Accountability Directorate

The small schools dilemma – reliable analysis and target setting

from NAPLAN data

Educational Measurement and School Accountability Directorate (EMSAD)


Small school data analysis

Medium and large school analysis relies on comparison of outcomes with those of previous years and with outcomes of similar schools.

Small schools analysis needs to focus on the progress of cohorts rather than comparison of outcomes for different cohorts.


Data instability

• Aggregated data for small schools are inherently unstable.

• Confidence intervals around means and band percentages are much greater than for larger schools.

• Cohort differences are likely to be much greater.


Percentage in bands – large school


Means trend – large school


Percentage in bands – small school


Means trend – small school


Small school confidence intervalsConfidence interval

of +/- 52


Confidence interval of +/- 13

Large school confidence intervals


What is a ‘small school’ for analysis and reporting purposes

• Reporting limits are usually set out of privacy considerations. ACARA uses a minimum of 5 students. NSW DEC uses a minimum of 10 students.

• Numbers required for comparison of cohort performance are greater than for reporting purposes.

• Overall school performance measures used in school performance graphs require results for a minimum of 15 students to construct a ‘reliable’ estimate. (Performance measures based on 10 to 14 students are provided but should be interpreted with caution.)

• The decisions you make regarding analysis and target setting need to be based on the size of the group and the nature of the data (e.g. means or band percentages.)


School size dataThe average enrolment of NSW DEC primary schools is 260 students (approximately 37 per year).

Ten per cent of NSW DEC primary schools have less than 23 students (3 per year).

Twenty per cent of NSW DEC primary schools have less than 50 students (7 per year)

Thirty per cent of NSW DEC primary schools have less than 90 students (13 per year)

About a third of NSW DEC primary schools could described as ‘Small schools’ for analysis and reporting purposes.


Percentage of government primary schools with less than 50 students

State PercentageACT 7NSW 31NT UnavailableQLD 37SA 27TAS 21VIC 28WA 22


Useful data sources for small schools

• Expected growth data

• Item analyses by syllabus outcomes for tracked cohorts and individual students

• Relative performance graphs

• Data aggregated over several years


Y3 to Y5 average growth in reading by Y3 reading score

Y3 reading score

MeanY3 reading

scoreMean

Y3 reading score

Mean

5 362 293 109 406 81

87 310 302 105 416 79

128 252 311 105 426 78

157 231 320 103 437 76

179 192 328 101 448 75

198 180 337 96 461 73

214 160 345 96 474 73

228 147 354 91 489 70

241 141 362 90 506 68

252 131 370 88 527 63

263 125 379 87 553 52

274 114 388 82 591 31

284 111 397 80 669 -25


Expected growth

• Expected growth differs according to the starting score.

• Approximately 57% of Year 5 students

achieved expected growth in reading


Year 5 student growth chart


Year 7 student growth chart


Cohort item analysis – Year 3 reading 2008

RS 2.5: Makes some inferences about ideas implicit in texts


Cohort item analysis – Year 5 reading 2010



Individual item analysis – Year 3 reading 2008



Relative performance – numeracy v reading


Relative performance – writing v reading


Using rolling 3-Year average data

2008 2009 2010 3 year total 3 year %age

Band 1 2 1 2 5 26

Band 2 3 2 1 6 32

Band 3 2 2 1 5 26

Band 4 0 0 1 1 5

Band 5 1 0 1 2 11

Band 6 0 0 0 0 0

Total 8 5 6 19 100

Average 410.8 423.7 422.6 417.9


Target setting for small schools

• Targets should be achievable (realistic) and (preferably) measurable.

• It is preferable if small school targets don’t rely on comparison with raw performance of previous cohorts. Comparison with 3-year averages may be appropriate.

• Small school targets are best set in relation to the prior performance of the students to whom the targets relate.


Target setting options

• Using the school target setting tool (Replace Year 5 data with Year 3 data and express targets in terms of percentage proficient, for example. )

• Expected growth targets

• Three year rolling average targets

• Improvement targets in terms of syllabus outcomes (data extracted from individual student learning plans).


Using the school target setting tool

2373

Step 2: Enter the anticipated number of students in Year 5 in each year

2011 16 2012 16 2013 16

2008 2009 2010 Average* Unadjusted Adjusted % Unadjusted Adjusted % Unadjusted Adjusted %

Band 8 0.0 0.0 14.3 6.3 6.30 0 1 1 6.3 0 0 1 1 6.3 0 0 1 1 6.3

Band 7 0.0 0.0 0.0 0.0 0.00 0 0 0 0.0 0 0 0 0 0.0 0 0 0 0 0.0

Band 6 0.0 0.0 14.3 6.3 6.30 0 1 1 6.3 0 0 1 1 6.3 0 0 1 1 6.3

Band 5 0.0 80.0 57.1 50.0 50.00 0 8 8 50.0 0 0 8 8 50.0 0 0 8 8 50.0

Band 4 50.0 0.0 0.0 12.5 12.50 0 2 2 12.5 0 0 2 2 12.5 0 0 2 2 12.5

Band 3 50.0 20.0 14.3 25.0 25.00 0 4 4 25.0 0 0 4 4 25.0 0 0 4 4 25.0

100.0 100.0 100.0 100.0 ### 0.0 16 16 100.0 16 16 100.0 16 16 100.0

Step 3: Use the arrows to select the desired band movements. This moves students up from one band to the next.

Note: the band distribution chart reflects the percentages of the adjusted numbers above Step 4: Select Chart - Minimum Standard or Proficient

Target Setting Spreadsheet: Year 5 ReadingSample Small School

Step 1: Select starting point for analysis - 2010 results or 3-year average

3 Year Average

Band Percentages 2008 - 2010 2011 Targets 2012 Targets 2013 Targets

Minimum Standard

Number of students Number of students Number of students

100.0

20.0

14.3

37.5

37.5

37.5

37.5

0

20

40

60

80

100

120

2008 2009 2010 3 Yr Ave 2011 2012 2013

Pe

rce

nta

ge

Percentage of students at or below minimum standard

0

10

20

30

40

50

60

Band 3 Band 4 Band 5 Band 6 Band 7 Band 8

Pe

rce

nta

ge

Band Distribution

State 3 Year Average 2011 2012 2013 School Actual School Target Target

zone StateRegion

Ambitious target line Moderate target line

Pro

ficie

nt

Min

imum

Min

imum

Min

imum

Pro

ficie

nt

Pro

ficie

ntReplace Year 5 data

with 2009 Year 3 band percentages


Possible types of targets • Increase percentage of all students with individual learning

plans (K-6) achieving 80% of their learning goals for reading (from 2011 achievement average of 66%)

• Increase the percentage of all students achieving greater than or equal to expected growth to 70% in 2011 (from 59% in 2010)

• Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN bands from X% to Y% in Year 5 2011

• Increase the three year aggregated percentage of Year 5 students achieving in the top three bands to 81% (from the 2008-2010 average of 87.5%).

educational measurement and school accountability directorate the small schools dilemma – reliable...

Documents

bands small school slide

trend small school slide

large school analysis

bands large school slide

trend large school slide

school performance graphs

small schools analysis

larger schools