port phillip/bayside network: using student achievement data to inform improved teaching programs...
TRANSCRIPT
Port Phillip/Bayside Port Phillip/Bayside Network:Network:
Using student achievement data to Using student achievement data to inform improved teaching programs inform improved teaching programs
Presented byPresented by
Philip Holmes-SmithPhilip Holmes-Smith(School Research Evaluation and Measurement Services)(School Research Evaluation and Measurement Services)
www.sreams.com.auwww.sreams.com.auS R E A M S
Research Evaluation And Measurement Services
Overview of the workshopOverview of the workshop1.1. Some definitions and some facts about testsSome definitions and some facts about tests
a)a) Diagnostic vs. Summative TestingDiagnostic vs. Summative Testing
b)b) The reliability of summative (standardised) testsThe reliability of summative (standardised) tests
c)c) Using other summative tests to monitor student progressUsing other summative tests to monitor student progress
1.1. Using and interpreting NAPLAN DataUsing and interpreting NAPLAN Data
a)a) Assessment of Learning Assessment of Learning
- Using SPA- Using SPA
b)b) Assessment for Learning (Using tests diagnostically to improve learning)Assessment for Learning (Using tests diagnostically to improve learning)
- Writing Criteria Report and the Student Response Report- Writing Criteria Report and the Student Response Report
- Item Analysis Report- Item Analysis Report
1(a).1(a).
DiagnosticDiagnostic
vs.vs.
Summative TestsSummative Tests
Diagnostic TestingDiagnostic Testing
Assessment tools such as the Assessment tools such as the Maths Online InterviewMaths Online Interview, , Marie Marie Clay InventoryClay Inventory and and ProbeProbe together with teacher assessments together with teacher assessments such as such as teacher questioningteacher questioning in class, in class, teacher observationsteacher observations and and student workstudent work (including portfolios) are all examples of (including portfolios) are all examples of ““DiagnosticDiagnostic” data” data
Research shows that our most effective teachers, in terms of Research shows that our most effective teachers, in terms of improving the learning outcomes of students, constantly use improving the learning outcomes of students, constantly use diagnostic information to inform their teaching. diagnostic information to inform their teaching.
IFIF a teacher users diagnostic information about what each a teacher users diagnostic information about what each student can and can’t do to inform their teaching for each student can and can’t do to inform their teaching for each student, Hattie (2003) shows that this has the single biggest student, Hattie (2003) shows that this has the single biggest impact on improving student learning outcomesimpact on improving student learning outcomes
Summative (Standardised) TestingSummative (Standardised) Testing
AIM/NAPLANAIM/NAPLAN and other tests like and other tests like TORCHTORCH, , PAT-RPAT-R, , PAT-MathPAT-Math, , SA SpellingSA Spelling and and On-Demand Adaptive TestsOn-Demand Adaptive Tests are referred to as are referred to as ““SummativeSummative” tests” tests
Summative testing is essential to monitor the effectiveness of Summative testing is essential to monitor the effectiveness of your teaching. (We will look at ways of doing this later.)your teaching. (We will look at ways of doing this later.)
Research shows that summative tests Research shows that summative tests do notdo not lead to improved lead to improved learning outcomes. As the saying goes:learning outcomes. As the saying goes:
““You don’t fatten a pig by weighing it”You don’t fatten a pig by weighing it”
So, although it is essential, keep summative testing to a So, although it is essential, keep summative testing to a minimum.minimum.
1(b).1(b).
The Reliability of The Reliability of
Summative TestsSummative Tests
Three QuestionsThree Questions
1.1. Do you believe that your students’ NAPLAN results accurately Do you believe that your students’ NAPLAN results accurately reflect their level of performance?reflect their level of performance?
Three QuestionsThree Questions
1.1. Do you believe that your students’ NAPLAN results accurately Do you believe that your students’ NAPLAN results accurately reflect their level of performance?reflect their level of performance?
2.2. If we acknowledge that the odd student will have a lucky If we acknowledge that the odd student will have a lucky guessing day or a horror day, what about the majority?guessing day or a horror day, what about the majority?
Have your weakest students received a low score?Have your weakest students received a low score?
Have your average students received a score at about expected level?Have your average students received a score at about expected level?
Have your best students received a high score?Have your best students received a high score?
Three QuestionsThree Questions
1.1. Do you believe that your students’ NAPLAN results accurately Do you believe that your students’ NAPLAN results accurately reflect their level of performance?reflect their level of performance?
2.2. If we acknowledge that the odd student will have a lucky If we acknowledge that the odd student will have a lucky guessing day or a horror day, what about the majority?guessing day or a horror day, what about the majority?
Have your weakest students received a low score?Have your weakest students received a low score?
Have your average students received a score at about expected level?Have your average students received a score at about expected level?
Have your best students received a high score?Have your best students received a high score?
3.3. Think about your students who received high and low scores: Think about your students who received high and low scores: Are your low scores too low? Are your low scores too low?
Are your high scores too high?Are your high scores too high?
Is this reading score reliable?
Is this reading score reliable?
Summary Statements about ScoresSummary Statements about Scores
Low scores (i.e. more than 0.5 VELS levels below Low scores (i.e. more than 0.5 VELS levels below expected) indicate poor performance but the actual expected) indicate poor performance but the actual values should be considered as indicative only. values should be considered as indicative only.
High scores (i.e. more than 0.5 VELS levels above High scores (i.e. more than 0.5 VELS levels above expected) indicate good performance but the expected) indicate good performance but the actual values should be considered as indicative actual values should be considered as indicative only.only.
Average scores indicate roughly expected levels of Average scores indicate roughly expected levels of performance but the actual values should be performance but the actual values should be considered as indicative only.considered as indicative only.
Item Difficulties forItem Difficulties forShe’s CryingShe’s Crying
on the TORCHon the TORCHscale score scalescale score scale
Converting Raw Converting Raw test Scores test Scores
((She’s CryingShe’s Crying))to TORCH scale to TORCH scale
scorescore
Test difficultiesTest difficultiesof theof the
TORCH TestsTORCH Testson the TORCH on the TORCH
score scale score scale together with together with Year Level Year Level mean scoresmean scores
Different norm tables for different testsDifferent norm tables for different tests
Source:Source:ACER, 2006ACER, 2006
Test difficultiesTest difficultiesof theof the
PAT-Maths TestsPAT-Maths Testson the PATMon the PATM
scale score scale scale score scale together with Year together with Year
Level mean Level mean scoresscores
Year 1
Year 2
Year 3
Year 4
Year 5
Year 6&7
Year 8&9
Year 10
Summative Testing and TriangulationSummative Testing and Triangulation Even if you give the right test to the right student, sometimes, Even if you give the right test to the right student, sometimes,
the test score does not reflect the true ability of the student – the test score does not reflect the true ability of the student – every measurement is associated with some errorevery measurement is associated with some error
To overcome this we should aim to get at least three To overcome this we should aim to get at least three independent measures – what researchers call independent measures – what researchers call TRIANGULATION.TRIANGULATION.
This may include:This may include: Teacher judgmentTeacher judgment NAPLAN resultsNAPLAN results Other pen & paper summative tests (e.g. TORCH, PAT-R, PAT-Maths)Other pen & paper summative tests (e.g. TORCH, PAT-R, PAT-Maths) On-line summative tests (e.g. On-Demand ‘Adaptive’ testing, On-line summative tests (e.g. On-Demand ‘Adaptive’ testing,
Assessment of English)Assessment of English)
BUT remember, more summative testing does not lead to BUT remember, more summative testing does not lead to improved learning outcomes – so keep the summative testing improved learning outcomes – so keep the summative testing to a minimumto a minimum
Things to look for in a summative testThings to look for in a summative test
Needs to have a single developmental scale that Needs to have a single developmental scale that shows increasing levels of achievement over all the shows increasing levels of achievement over all the year levels at your schoolyear levels at your school
Needs to have “norms” or expected levels for each Needs to have “norms” or expected levels for each year level (e.g. The National “norm” for Yr 3 students year level (e.g. The National “norm” for Yr 3 students on TORCH is an average of 34.7).on TORCH is an average of 34.7).
Needs to be able to demonstrate growth from one Needs to be able to demonstrate growth from one year to the next (e.g. during Yr 4, the average student year to the next (e.g. during Yr 4, the average student grows from a score of 34.7 in Yr 3 to an expected grows from a score of 34.7 in Yr 3 to an expected score of 41.4 in Yr 4score of 41.4 in Yr 4
As a bonus, the test could also provides diagnostic As a bonus, the test could also provides diagnostic information information
Norms for Norms for Year 3 to Year 3 to Year 10Year 10On the On the
TORCH TORCH scalescale
My Recommended Summative TestsMy Recommended Summative Tests(Pen & Paper)(Pen & Paper)
Reading ComprehensionReading Comprehension TORCHTORCH and and TORCH plusTORCH plus Progressive Achievement Test - ReadingProgressive Achievement Test - Reading
(PAT-R, 4(PAT-R, 4thth Edition) Edition) MathematicsMathematics
Progressive Achievement Test - MathematicsProgressive Achievement Test - Mathematics (PAT-Maths, 3(PAT-Maths, 3rdrd Edition) combined with the Edition) combined with the I Can Do MathsI Can Do Maths
SpellingSpelling South Australian SpellingSouth Australian Spelling
My Recommended Summative TestsMy Recommended Summative Tests(On-Line)(On-Line)
On-Demand - Reading ComprehensionOn-Demand - Reading Comprehension The 30-item “On-Demand”The 30-item “On-Demand” Adaptive Reading Adaptive Reading test test
On-Demand - SpellingOn-Demand - Spelling The 30-itemThe 30-item “On-Demand”“On-Demand” Adaptive Spelling Adaptive Spelling testtest
On-Demand - Writing ConventionsOn-Demand - Writing Conventions The 30-itemThe 30-item “On-Demand”“On-Demand” Adaptive Writing Adaptive Writing testtest
Assessment of English in the Early Years (2010). Includes:Assessment of English in the Early Years (2010). Includes: Comprehension, Comprehension, Spelling, Spelling, WritingWriting Speaking & ListeningSpeaking & Listening
On-Demand - Mathematics On-Demand - Mathematics (Number, Measurement, Chance & Data and Space)(Number, Measurement, Chance & Data and Space) The 60-itemThe 60-item “On-Demand”“On-Demand” Adaptive General Mathematics Adaptive General Mathematics testtest
2(b). Assessment 2(b). Assessment ofof Learning Learning
Using SPAUsing SPA
The NAPLAN Data Service Main MenuThe NAPLAN Data Service Main Menu
Student Achievement Level Report
The Student Achievement Level Report MenuThe Student Achievement Level Report Menu
Graphical Format
The Student Achievement Level ReportThe Student Achievement Level Report
The Student Achievement Level Report MenuThe Student Achievement Level Report Menu
Tabular Format
The Student Achievement Level ReportThe Student Achievement Level ReportThese are the estimated VELS equivalent scores
Tabular form of the National Report is not yet available
Extracting Outcome Level Data for Further Analysis
Cut-points for colour codingCut-points for colour coding
Yr 3 Yr 5 Yr 7 Yr 9
<= 1.675 <= 2.675 <= 3.675 <= 4.675
1.676 - 2.674 2.676 - 3.674 3.676 - 4.674 4.676 - 5.674
>= 2.675 >= 3.675 >= 4.675 >= 5.675
Working with the extracted dataWorking with the extracted data
See SPASee SPA
2(b) Assessment 2(b) Assessment forfor Learning Learning
Using the tests diagnostically to Using the tests diagnostically to
improve student learningimprove student learning
The NAPLAN Data Service Main MenuThe NAPLAN Data Service Main Menu
Writing Criteria Report
The Writing Criteria Report MenuThe Writing Criteria Report Menu
Writing – The marking rubricWriting – The marking rubric
NAPLANNAPLAN Marking rubric is comprised of 10 criteria. Namely:Marking rubric is comprised of 10 criteria. Namely:
- Audience - Cohesion- Text structure - Paragraphing- Ideas - Sentence structure- Character and setting - Punctuation- Vocabulary - Spelling
• The marking rubric can be downloaded from: The marking rubric can be downloaded from: http://www.naplan.edu.au/verve/_resources/napmarkguide08.pdf or or www.sreams.com.au . .
• This document also contains some annotated marked examples.This document also contains some annotated marked examples.
Writing – The marking rubric Writing – The marking rubric (cont)(cont)
Writing – Annotated marked example Writing – Annotated marked example
The Writing The Writing Criteria ReportCriteria Report
Across the State less than 20% Across the State less than 20% received a score of “1” and received a score of “1” and about 40% received a score of about 40% received a score of “3”“3”
However, in this school, nearly However, in this school, nearly 40% received a score of “1” 40% received a score of “1” and under 20% received a and under 20% received a score of “3”score of “3”
This school could benefit from This school could benefit from some lessons on how to some lessons on how to develop ideas in a Narrativedevelop ideas in a Narrative
The Student Response Report MenuThe Student Response Report Menu
Writing responses (by criteria)
The Student Response Report – Writing (by criteria)The Student Response Report – Writing (by criteria)
The NAPLAN Data Service Main MenuThe NAPLAN Data Service Main Menu
Item Analysis Report
The Item Analysis Report MenuThe Item Analysis Report Menu
Reading Report in Item Order
The Item Analysis ReportThe Item Analysis Report
Year 5 Reading
Correct = ?Most common incorrect = ?
Year 5 Reading
Correct = D (93%)
Most commonincorrect = A, B, C (2-3% each)
Year 5 Reading
Correct = ?
Most commonincorrect = ?
Year 5 Reading
Correct = B (56%)
Most commonincorrect = C (34%)