port phillip/bayside network: using student achievement data to inform improved teaching programs...

Port Phillip/Bayside Port Phillip/Bayside Network:Network:

Using student achievement data to Using student achievement data to inform improved teaching programs inform improved teaching programs

Presented byPresented by

Philip Holmes-SmithPhilip Holmes-Smith(School Research Evaluation and Measurement Services)(School Research Evaluation and Measurement Services)

www.sreams.com.auwww.sreams.com.auS R E A M S

Research Evaluation And Measurement Services

Overview of the workshopOverview of the workshop1.1. Some definitions and some facts about testsSome definitions and some facts about tests

a)a) Diagnostic vs. Summative TestingDiagnostic vs. Summative Testing

b)b) The reliability of summative (standardised) testsThe reliability of summative (standardised) tests

c)c) Using other summative tests to monitor student progressUsing other summative tests to monitor student progress

1.1. Using and interpreting NAPLAN DataUsing and interpreting NAPLAN Data

a)a) Assessment of Learning Assessment of Learning

- Using SPA- Using SPA

b)b) Assessment for Learning (Using tests diagnostically to improve learning)Assessment for Learning (Using tests diagnostically to improve learning)

- Writing Criteria Report and the Student Response Report- Writing Criteria Report and the Student Response Report

- Item Analysis Report- Item Analysis Report

1(a).1(a).

DiagnosticDiagnostic

vs.vs.

Summative TestsSummative Tests

Diagnostic TestingDiagnostic Testing

Assessment tools such as the Assessment tools such as the Maths Online InterviewMaths Online Interview, , Marie Marie Clay InventoryClay Inventory and and ProbeProbe together with teacher assessments together with teacher assessments such as such as teacher questioningteacher questioning in class, in class, teacher observationsteacher observations and and student workstudent work (including portfolios) are all examples of (including portfolios) are all examples of ““DiagnosticDiagnostic” data” data

Research shows that our most effective teachers, in terms of Research shows that our most effective teachers, in terms of improving the learning outcomes of students, constantly use improving the learning outcomes of students, constantly use diagnostic information to inform their teaching. diagnostic information to inform their teaching.

IFIF a teacher users diagnostic information about what each a teacher users diagnostic information about what each student can and can’t do to inform their teaching for each student can and can’t do to inform their teaching for each student, Hattie (2003) shows that this has the single biggest student, Hattie (2003) shows that this has the single biggest impact on improving student learning outcomesimpact on improving student learning outcomes

Summative (Standardised) TestingSummative (Standardised) Testing

AIM/NAPLANAIM/NAPLAN and other tests like and other tests like TORCHTORCH, , PAT-RPAT-R, , PAT-MathPAT-Math, , SA SpellingSA Spelling and and On-Demand Adaptive TestsOn-Demand Adaptive Tests are referred to as are referred to as ““SummativeSummative” tests” tests

Summative testing is essential to monitor the effectiveness of Summative testing is essential to monitor the effectiveness of your teaching. (We will look at ways of doing this later.)your teaching. (We will look at ways of doing this later.)

Research shows that summative tests Research shows that summative tests do notdo not lead to improved lead to improved learning outcomes. As the saying goes:learning outcomes. As the saying goes:

““You don’t fatten a pig by weighing it”You don’t fatten a pig by weighing it”

So, although it is essential, keep summative testing to a So, although it is essential, keep summative testing to a minimum.minimum.

1(b).1(b).

The Reliability of The Reliability of

Summative TestsSummative Tests

Three QuestionsThree Questions

1.1. Do you believe that your students’ NAPLAN results accurately Do you believe that your students’ NAPLAN results accurately reflect their level of performance?reflect their level of performance?

Three QuestionsThree Questions

1.1. Do you believe that your students’ NAPLAN results accurately Do you believe that your students’ NAPLAN results accurately reflect their level of performance?reflect their level of performance?

2.2. If we acknowledge that the odd student will have a lucky If we acknowledge that the odd student will have a lucky guessing day or a horror day, what about the majority?guessing day or a horror day, what about the majority?

Have your weakest students received a low score?Have your weakest students received a low score?

Have your average students received a score at about expected level?Have your average students received a score at about expected level?

Have your best students received a high score?Have your best students received a high score?

Three QuestionsThree Questions

1.1. Do you believe that your students’ NAPLAN results accurately Do you believe that your students’ NAPLAN results accurately reflect their level of performance?reflect their level of performance?

2.2. If we acknowledge that the odd student will have a lucky If we acknowledge that the odd student will have a lucky guessing day or a horror day, what about the majority?guessing day or a horror day, what about the majority?

Have your weakest students received a low score?Have your weakest students received a low score?

Have your average students received a score at about expected level?Have your average students received a score at about expected level?

Have your best students received a high score?Have your best students received a high score?

3.3. Think about your students who received high and low scores: Think about your students who received high and low scores: Are your low scores too low? Are your low scores too low?

Are your high scores too high?Are your high scores too high?

Is this reading score reliable?

Is this reading score reliable?

Summary Statements about ScoresSummary Statements about Scores

Low scores (i.e. more than 0.5 VELS levels below Low scores (i.e. more than 0.5 VELS levels below expected) indicate poor performance but the actual expected) indicate poor performance but the actual values should be considered as indicative only. values should be considered as indicative only.

High scores (i.e. more than 0.5 VELS levels above High scores (i.e. more than 0.5 VELS levels above expected) indicate good performance but the expected) indicate good performance but the actual values should be considered as indicative actual values should be considered as indicative only.only.

Average scores indicate roughly expected levels of Average scores indicate roughly expected levels of performance but the actual values should be performance but the actual values should be considered as indicative only.considered as indicative only.

Item Difficulties forItem Difficulties forShe’s CryingShe’s Crying

on the TORCHon the TORCHscale score scalescale score scale

Converting Raw Converting Raw test Scores test Scores

((She’s CryingShe’s Crying))to TORCH scale to TORCH scale

scorescore

Test difficultiesTest difficultiesof theof the

TORCH TestsTORCH Testson the TORCH on the TORCH

score scale score scale together with together with Year Level Year Level mean scoresmean scores

Different norm tables for different testsDifferent norm tables for different tests

Source:Source:ACER, 2006ACER, 2006

Test difficultiesTest difficultiesof theof the

PAT-Maths TestsPAT-Maths Testson the PATMon the PATM

scale score scale scale score scale together with Year together with Year

Level mean Level mean scoresscores

Year 1

Year 2

Year 3

Year 4

Year 5

Year 6&7

Year 8&9

Year 10

Summative Testing and TriangulationSummative Testing and Triangulation Even if you give the right test to the right student, sometimes, Even if you give the right test to the right student, sometimes,

the test score does not reflect the true ability of the student – the test score does not reflect the true ability of the student – every measurement is associated with some errorevery measurement is associated with some error

To overcome this we should aim to get at least three To overcome this we should aim to get at least three independent measures – what researchers call independent measures – what researchers call TRIANGULATION.TRIANGULATION.

This may include:This may include: Teacher judgmentTeacher judgment NAPLAN resultsNAPLAN results Other pen & paper summative tests (e.g. TORCH, PAT-R, PAT-Maths)Other pen & paper summative tests (e.g. TORCH, PAT-R, PAT-Maths) On-line summative tests (e.g. On-Demand ‘Adaptive’ testing, On-line summative tests (e.g. On-Demand ‘Adaptive’ testing,

Assessment of English)Assessment of English)

BUT remember, more summative testing does not lead to BUT remember, more summative testing does not lead to improved learning outcomes – so keep the summative testing improved learning outcomes – so keep the summative testing to a minimumto a minimum

Things to look for in a summative testThings to look for in a summative test

Needs to have a single developmental scale that Needs to have a single developmental scale that shows increasing levels of achievement over all the shows increasing levels of achievement over all the year levels at your schoolyear levels at your school

Needs to have “norms” or expected levels for each Needs to have “norms” or expected levels for each year level (e.g. The National “norm” for Yr 3 students year level (e.g. The National “norm” for Yr 3 students on TORCH is an average of 34.7).on TORCH is an average of 34.7).

Needs to be able to demonstrate growth from one Needs to be able to demonstrate growth from one year to the next (e.g. during Yr 4, the average student year to the next (e.g. during Yr 4, the average student grows from a score of 34.7 in Yr 3 to an expected grows from a score of 34.7 in Yr 3 to an expected score of 41.4 in Yr 4score of 41.4 in Yr 4

As a bonus, the test could also provides diagnostic As a bonus, the test could also provides diagnostic information information

Norms for Norms for Year 3 to Year 3 to Year 10Year 10On the On the

TORCH TORCH scalescale

My Recommended Summative TestsMy Recommended Summative Tests(Pen & Paper)(Pen & Paper)

Reading ComprehensionReading Comprehension TORCHTORCH and and TORCH plusTORCH plus Progressive Achievement Test - ReadingProgressive Achievement Test - Reading

(PAT-R, 4(PAT-R, 4thth Edition) Edition) MathematicsMathematics

Progressive Achievement Test - MathematicsProgressive Achievement Test - Mathematics (PAT-Maths, 3(PAT-Maths, 3rdrd Edition) combined with the Edition) combined with the I Can Do MathsI Can Do Maths

SpellingSpelling South Australian SpellingSouth Australian Spelling

My Recommended Summative TestsMy Recommended Summative Tests(On-Line)(On-Line)

On-Demand - Reading ComprehensionOn-Demand - Reading Comprehension The 30-item “On-Demand”The 30-item “On-Demand” Adaptive Reading Adaptive Reading test test

On-Demand - SpellingOn-Demand - Spelling The 30-itemThe 30-item “On-Demand”“On-Demand” Adaptive Spelling Adaptive Spelling testtest

On-Demand - Writing ConventionsOn-Demand - Writing Conventions The 30-itemThe 30-item “On-Demand”“On-Demand” Adaptive Writing Adaptive Writing testtest

Assessment of English in the Early Years (2010). Includes:Assessment of English in the Early Years (2010). Includes: Comprehension, Comprehension, Spelling, Spelling, WritingWriting Speaking & ListeningSpeaking & Listening

On-Demand - Mathematics On-Demand - Mathematics (Number, Measurement, Chance & Data and Space)(Number, Measurement, Chance & Data and Space) The 60-itemThe 60-item “On-Demand”“On-Demand” Adaptive General Mathematics Adaptive General Mathematics testtest

2(b). Assessment 2(b). Assessment ofof Learning Learning

Using SPAUsing SPA

The NAPLAN Data Service Main MenuThe NAPLAN Data Service Main Menu

Student Achievement Level Report

The Student Achievement Level Report MenuThe Student Achievement Level Report Menu

Graphical Format

The Student Achievement Level ReportThe Student Achievement Level Report

The Student Achievement Level Report MenuThe Student Achievement Level Report Menu

Tabular Format

The Student Achievement Level ReportThe Student Achievement Level ReportThese are the estimated VELS equivalent scores

Tabular form of the National Report is not yet available

Extracting Outcome Level Data for Further Analysis

Cut-points for colour codingCut-points for colour coding

Yr 3 Yr 5 Yr 7 Yr 9

<= 1.675 <= 2.675 <= 3.675 <= 4.675

1.676 - 2.674 2.676 - 3.674 3.676 - 4.674 4.676 - 5.674

>= 2.675 >= 3.675 >= 4.675 >= 5.675

Working with the extracted dataWorking with the extracted data

See SPASee SPA

2(b) Assessment 2(b) Assessment forfor Learning Learning

Using the tests diagnostically to Using the tests diagnostically to

improve student learningimprove student learning

The NAPLAN Data Service Main MenuThe NAPLAN Data Service Main Menu

Writing Criteria Report

The Writing Criteria Report MenuThe Writing Criteria Report Menu

Writing – The marking rubricWriting – The marking rubric

NAPLANNAPLAN Marking rubric is comprised of 10 criteria. Namely:Marking rubric is comprised of 10 criteria. Namely:

- Audience - Cohesion- Text structure - Paragraphing- Ideas - Sentence structure- Character and setting - Punctuation- Vocabulary - Spelling

• The marking rubric can be downloaded from: The marking rubric can be downloaded from: http://www.naplan.edu.au/verve/_resources/napmarkguide08.pdf or or www.sreams.com.au . .

• This document also contains some annotated marked examples.This document also contains some annotated marked examples.

Writing – The marking rubric Writing – The marking rubric (cont)(cont)

Writing – Annotated marked example Writing – Annotated marked example

The Writing The Writing Criteria ReportCriteria Report

Across the State less than 20% Across the State less than 20% received a score of “1” and received a score of “1” and about 40% received a score of about 40% received a score of “3”“3”

However, in this school, nearly However, in this school, nearly 40% received a score of “1” 40% received a score of “1” and under 20% received a and under 20% received a score of “3”score of “3”

This school could benefit from This school could benefit from some lessons on how to some lessons on how to develop ideas in a Narrativedevelop ideas in a Narrative

The Student Response Report MenuThe Student Response Report Menu

Writing responses (by criteria)

The Student Response Report – Writing (by criteria)The Student Response Report – Writing (by criteria)

The NAPLAN Data Service Main MenuThe NAPLAN Data Service Main Menu

Item Analysis Report

The Item Analysis Report MenuThe Item Analysis Report Menu

Reading Report in Item Order

The Item Analysis ReportThe Item Analysis Report

Year 5 Reading

Correct = ?Most common incorrect = ?

Year 5 Reading

Correct = D (93%)

Most commonincorrect = A, B, C (2-3% each)

Year 5 Reading

Correct = ?

Most commonincorrect = ?

Year 5 Reading

Correct = B (56%)

Most commonincorrect = C (34%)