paul stemmer stemmerp@mi naep state coordinator michigan department of education
DESCRIPTION
Describing NAEP Background Question Data: A Cautionary Tale. Paul Stemmer [email protected] NAEP State Coordinator Michigan Department of Education National Conference on Student Assessment, Minneapolis, Minnesota June 2012. NAEP Grade 4 Reading Trends Comparison. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/1.jpg)
Paul [email protected] State CoordinatorMichigan Department of Education
National Conference on Student Assessment, Minneapolis, MinnesotaJune 2012
Paul [email protected] State CoordinatorMichigan Department of Education
National Conference on Student Assessment, Minneapolis, MinnesotaJune 2012
Describing NAEP Background Question Data: A Cautionary Tale
Describing NAEP Background Question Data: A Cautionary Tale
![Page 2: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/2.jpg)
NAEP Background Question Data Analysis
NAEP Grade 4 Reading Trends ComparisonNAEP Grade 4 Reading Trends Comparison
2
![Page 3: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/3.jpg)
NAEP Background Question Data Analysis
NAEP Grade 4 Reading Trends Comparison (Enlarged)NAEP Grade 4 Reading Trends Comparison (Enlarged)
3
![Page 4: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/4.jpg)
NAEP Background Question Data Analysis
What is different about these comparisons?What is different about these comparisons?
4
![Page 5: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/5.jpg)
NAEP Background Question Data Analysis
What is different about these comparisons? (enlarged)What is different about these comparisons? (enlarged)
5
Avg SS 202 214 219 222 * 229 237 238 * 217 219 221 * 215 225 224 Nat. Pub. MA MI OHAvg SS 220 237 219 224
![Page 6: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/6.jpg)
NAEP Background Question Data Analysis
Life is Good!Life is Good!
6
![Page 7: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/7.jpg)
NAEP Background Question Data Analysis
NAEP Trends in Grade 4 Reading Average Scaled ScoresNAEP Trends in Grade 4 Reading Average Scaled Scores
![Page 8: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/8.jpg)
NAEP Background Question Data Analysis
Wait a Minute, What Happened?Wait a Minute, What Happened?
8
![Page 9: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/9.jpg)
NAEP Background Question Data Analysis
Wait a Minute, What Happened? (enlarged)Wait a Minute, What Happened? (enlarged)
9
Avg. SS * 217 219 222 * * 191 190
![Page 10: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/10.jpg)
NAEP Background Question Data Analysis
What explains this? What about other TUDAs?What explains this? What about other TUDAs?
10
![Page 11: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/11.jpg)
NAEP Background Question Data Analysis
What about OHIO and Cleveland?What about OHIO and Cleveland?
11
![Page 12: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/12.jpg)
NAEP Background Question Data Analysis
What about MA and Boston?What about MA and Boston?
12
![Page 13: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/13.jpg)
NAEP Background Question Data Analysis
Why the discrepancies?Why the discrepancies?
Two Possibilities
A. If the data is in error? Differences in respondent question interpretation –
definitions? Need to improve the question
Social Desirability
B. If data is “true,” perhaps more interesting? Internal School Variables External School Factors
13
![Page 14: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/14.jpg)
NAEP Background Question Data Analysis
Checking Social Desirability 09 to 11 ComparisonChecking Social Desirability 09 to 11 Comparison
14
![Page 15: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/15.jpg)
NAEP Background Question Data Analysis
Why the discrepancies?Why the discrepancies?
If data is “true,” perhaps more interesting?What are the possible intervening factors?
Internal School Variables Professional development Rigor Instructional Methods/Impact Professional knowhow vs. execution Administration, management and follow through Resources and resource utilization
External School Factors Chronic Absenteeism Concentration of Poverty Health Factors Cultural differences
15
![Page 16: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/16.jpg)
NAEP Background Question Data Analysis
Conclusions?Conclusions?
Descriptive data can be helpful and informative
Have to keep reminding ourselves it is not causative/predictive
Are we jumping to solutions without fully understanding how best to solve these problems of large urban school districts?
Do we infer too much from snapshot assessment results about the quality of teaching and administration?
Does arguing otherwise sound like excuses?
Aren’t we all guilty of trying to connect the dots? Trying to simplify the story Making inferences beyond what the data is telling us.
16
![Page 17: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/17.jpg)
NAEP Background Question Data Analysis
FinallyFinally
Prediction is very hard, especially about the future. - Yogi Berra
17
![Page 18: Paul Stemmer stemmerp@mi NAEP State Coordinator Michigan Department of Education](https://reader036.vdocuments.mx/reader036/viewer/2022070415/56814fea550346895dbdb8fa/html5/thumbnails/18.jpg)
NAEP Background Question Data Analysis
Contact:
Paul Stemmer, Ph.D., NAEP State Coordinator
Michigan Department of EducationOffice of Educational Assessment and
AccountabilityPO Box 30008
Lansing, MI 48909
(517)[email protected]
For More Information: