finding what works in learning: simple ways to analyze education research studies
TRANSCRIPT
Finding What Works in Learning:
Simple Ways to Analyze Education Research Studies
Dr. Tim Hudson Vice President of Learning
DreamBox Learning@DocHudsonMath
Dr. Gina BurkhardtEducation & Organizational
Consultant, Former EVP at American Institutes for Research (AIR)
• Easily analyze research and ask probing questions to meaningfully and successfully review educational research about curricular programs
• Eight key questions to guide analysis of the “fine print” of a research study
• Using a simple rubric to quickly analyze the quality of any research study
• Ideas for conducting research in your own classroom, school, or district
“But where I’m flummoxed is how we are supposed to provide practicing educators with the tools to evaluate
these kinds of findings.”
-Justin Reich http://blogs.edweek.org/edweek/edtechresearcher/2013/08/the_headlines_versus_the_study_in_blended_algebra_program.html
EdWeek in 2013: The Headlines Versus The Study in Blended Algebra Program
Research-Based PLUS
Real-World Studies
What is the basis of the program in terms of the existing research and literature?
What is the research conducted in the real world that validates the program’s impact?
Visible Learning by John Hattie
http://visible-learning.org/glossary/#2_Piagetian_programs
Piagetian Programs“Focus on the thinking processes rather than the outcomes and do
not impose the adult thinking process on to children.”
Headlines and Studies
DreamBox Learning Drives Compelling Achievement Gains in Elementary Mathematics, Suggests Harvard University Personalized Learning Study
Study of 3,000 Students in Grades 3-5 Showed DreamBox Improved Math Achievement on NWEA MAP and PARCC Assessments
Key Findings
TechnicalAppendix
(2016). DreamBox Learning Achievement Growth in the Howard County Public School System and Rocketship Education . Center for Education Policy at Harvard University. http://cepr.harvard.edu/publications/dreambox-learning-achievement-growth
1
Conducted by the Center for Education Policy Research (CEPR) at Harvard University
3,000 Students in Grades 3-5
2 School Systems
100 Classrooms
2013-14 & 2014-15
NWEA MAP, PARCC, and State Assessments
Quasi-Experimental Analysis
SRI Study: Rocketship Students Experience Stronger Math Gains with More Online Instruction
DreamBox Intelligent Adaptive Learning™ Platform Significantly Improves Students’ Math Scores
SRI International Study of Rocketship Education’s Use of DreamBox Learning Validates Learning Gains
2
Conducted by SRI International
583 Students in Grades K-1
1 School System
3 Schools
2010-11
NWEA MAP Assessments
Randomized Controlled Trial
http://cepr.harvard.edu/publications/dreambox-learning-achievement-growth, Technical Appendix, page 13
Acme’s Algebra 2 ProgramQuadruples Growth for 11th Graders
Researchers from the prestigious University of Antarctica examined high schools where at least half of the students in Algebra 2 classes successfully completed 75 percent of the Acme Algebra 2 curriculum during the 2014–15 school year. Researchers found 12 schools in the region that fit these criteria, for a total of 984 eleventh-grade Algebra 2 students. For use as a comparison group, researchers also identified 12 demographically comparable schools in the area with 1,126 Algebra 2 students. For the baseline scores, researchers used the percentage of eleventh-grade students who scored proficient or advanced on the 2014 end-of-course Algebra 2 state exam at each of the 24 schools. In schools using Acme’s Algebra 2 online math program, the percentage of eleventh graders scoring at these top two levels on the state Algebra 2 exam in the spring of 2015 was 4 percent higher than the eleventh-grade spring 2014 baseline percentage. In the control-group schools, the percentage only increased by 1 percent (p < .01).
FAKE
Acme’s Algebra 2 ProgramQuadruples Growth for 11th Graders
(continued) Acme’s Algebra 2 online math program is a curriculum that allows students to move at their own pace as they view online videos and complete related practice problems. As reported in the study, students worked on the Acme program at school for 2 hours each week during the school year. They were also able to access the program at home. The Algebra 2 curriculum for the 12 schools was rewritten to align with the Acme Algebra 2 program. Therefore teachers received 5 days of training for the Acme online program before and during the school year. Teachers also used Acme’s reports throughout the school year to inform their instruction. Acme’s Algebra 2 online program costs $100 per student each year. Funding for this study was provided by Acme Education, Inc.
FAKE
Eight Key Questions to Ask of Every Study
The Rubric
1. Who conducted the study?
1. Who conducted the study?
2. What was the research methodology?
2. What was the research methodology?
3. Whose pre-test and post-test scores were compared?
3. Whose pre-test and post-test scores were compared?
4. Which students were included in (or excluded from) the treatment group?
Possible Bias in Selecting the Treatment Group1. Treatment group only includes students who complete a certain performance
milestone during the study time period. (Quasi-Experimental or Case Study)
Possible Bias in Selecting the Treatment Group1. Treatment group only includes students who complete a certain performance
milestone during the study time period. (Quasi-Experimental or Case Study)
Athlete Monday Tuesday Wednesday Thursday Friday Final (2 meters)
1 1.66 1.77 1.88 1.99 2.012 1.99 2.01 2.02 2.01 2.023 1.30 1.35 1.40 1.45 1.454 1.00 1.02 1.01 1.02 1.01
Possible Bias in Selecting the Treatment Group1. Treatment group only includes students who complete a certain performance
milestone during the study time period. (Quasi-Experimental or Case Study)
Athlete Monday Tuesday Wednesday Thursday Friday Final (2 meters)
1 1.66 1.77 1.88 1.99 2.012 1.99 2.01 2.02 2.01 2.023 1.30 1.35 1.40 1.45 1.454 1.00 1.02 1.01 1.02 1.01
Treatment Group
Possible Bias in Selecting the Treatment Group1. Treatment group only includes students who complete a certain performance
milestone during the study time period. (Quasi-Experimental or Case Study)2. School faculty or administration in the treatment group volunteered to use the
program, indicating there are other significant motivators & accountability systems.
Possible Bias in Selecting the Treatment Group1. Treatment group only includes students who complete a certain performance
milestone during the study time period. (Quasi-Experimental or Case Study)2. School faculty or administration in the treatment group volunteered to use the
program, indicating there are other significant motivators & accountability systems.3. Students assigned to the treatment group experienced significant classroom
curricular changes that coincided with the implementation of the program, which increases the difficulty of isolating the program’s impact
Possible Bias in Selecting the Treatment Group1. Treatment group only includes students who complete a certain performance
milestone during the study time period. (Quasi-Experimental or Case Study)2. School faculty or administration in the treatment group volunteered to use the
program, indicating there are other significant motivators & accountability systems.3. Students assigned to the treatment group experienced significant classroom
curricular changes that coincided with the implementation of the program, which increases the difficulty of isolating the program’s impact
4. Teachers in the treatment group participated in significant professional development (PD) that accompanied the implementation of the program that was different than the PD for teachers in the control group
4. Which students were included in (or excluded from) the treatment group?
5. What were the school support requirements during the study?
5. What were the school support requirements during the study?
6. How large and representative was the researched population?
6. How large and representative was the researched population?
7. How significant were the findings?
http://fivethirtyeight.com/features/not-even-scientists-can-easily-explain-p-values/
7. How significant were the findings?
7. How significant were the findings?
8. To what extent did learning and achievement improve?
8. To what extent did learning and achievement improve?
Acme Algebra 2
8. To what extent did learning and achievement improve?
Spring 2014 % Proficient Spring 2015 % Proficient
TreatmentControl
+ 4%
+ 1%
4x Growth
Acme Algebra 2
What was the baseline?
Spring 2014 % Proficient Spring 2015 % Proficient
TreatmentControl
+ 4%
+ 1%
4x Growth
???
Acme Algebra 2
What was the baseline?
Spring 2014 % Proficient Spring 2015 % Proficient
TreatmentControl
94%
91%
4x Growth
90%
Acme Algebra 2
What was the baseline?
Spring 2014 % Proficient Spring 2015 % Proficient
TreatmentControl
54%
51%
4x Growth
50%
Acme Algebra 2
What was the baseline?
Spring 2014 % Proficient Spring 2015 % Proficient
TreatmentControl
24%
21%
4x Growth
20%
Acme Algebra 2
8. To what extent did learning and achievement improve?
8. To what extent did learning and achievement improve?
The Rubric
Considerations for Designing & Conducting Your Own Studies• Randomized Controlled Trials are difficult and often inadvisable
in many education contexts• Quasi-Experimental Studies need valid comparison groups
(quasi-control and quasi-treatment groups)• Case Studies often have no comparison group, which must be
considered when interpreting any potentially causal impact• Create unbiased treatment and control groups (i.e., no minimum
restrictions on the amount of curriculum completed or only considering students who are “on grade level” in the program)
• Use same-student comparisons, not averages of a classroom, grade level, or school, nor different cohort comparisons
• Consider growth from each student’s baseline
Q & A
Dr. Tim Hudson Vice President of Learning
DreamBox Learning@DocHudsonMath
Dr. Gina BurkhardtEducation & Organizational
Consultant, Former EVP at American Institutes for Research (AIR)
Thank You!
DreamBox Learning® K–8 MathTransformative Learning Experiences available in English & Spanish
DreamBox Lessons & Virtual ManipulativesIntelligently adapt & individualize to:• Students’ own intuitive strategies• Kinds of mistakes• Efficiency of strategy• Scaffolding needed• Response time
“I’ve tried teaching this topic with pencil, paper, and plastic manipulatives for twenty years and students have never really gotten it. I can immediately see that DreamBox has invented new, digital manipulatives and games that are far more powerful for student learning.”– Urban District Math Leader, CA
AssignFocus™
Differentiated assignments for every student through your Insight Dashboard
To accelerate learning, offer remediation, and adjust classroom instruction
Prepare for the Next Unit Homework
Assignment Review for Assessment
Empowered Teaching with Data.Inspired Learning with Technology.
iNACOL is right around the corner… Catch us at @DreamBox_Learn
Learn more and see how it works:www.DreamBox.com/request-a-demo
Efficacy: Independent Validation from SRI and from CEPR at Harvard University
We value your feedback!Let us know how we’re doing:
www.surveymonkey.com/r/GC6ZCM7