steven h. shaha, phd, dba professor, center for public policy and administration
DESCRIPTION
Program Evaluation and Impact Assessment: Internet-Delivered, On-Demand Professional Development Participating Schools versus Their Respective Districts. Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator July 2013. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/1.jpg)
1
Program Evaluation and Impact Assessment: Internet-Delivered, On-Demand
Professional DevelopmentParticipating Schools
versus Their Respective Districts
Steven H. Shaha, PhD, DBAProfessor, Center for Public Policy and Administration
Independent EvaluatorJuly 2013
NOTE: Findings that follow are published as: Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013 (in press)
![Page 2: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/2.jpg)
2
Overarching Research Question:
Does teacher engagement in PD 360 and Observation 360,
tools within the Educator Effectiveness System,
significantly affect student success?
![Page 3: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/3.jpg)
3
Methods
• Design: Quasi-experimental, retrospective, pre-post, normalized treatment-control / participation vs. non-participation (2009-10, 2010-11)
• Goal: Multi-State, large n with comparable student populations (matched, controlled)
• Student Change:* Metric was percent students classified as Proficient or Advanced in respective States.
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 4: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/4.jpg)
4
SampleParticipation
– Systematic sample of 169 elementary schools, in 73 districts, in 19 States• N determined by a priori Power analysis
– Schools eligible for inclusion in the sample as participating Schools met the following criteria:• More than 10 teachers total• 80% or more of teachers viewed materials• Minimum average of 90.0 minutes of viewing per teacher for the school
– Districts included were only those for which eligible schools were included• Normalizing for difference in socio-economic and demographic factors between participating Schools and their
Districts cumulatively as the statistical comparison group
Data– Participation data were extracted from the Internet-based professional development
application as surveilled– Student performance data were captured from publically available, Internet-accessed
sources (school as unit of measure, percent Proficient or Advanced as metric)
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 5: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/5.jpg)
5
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 6: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/6.jpg)
6
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 7: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/7.jpg)
7
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
Improvement is percent change: [(Year2-Year1)/Year1]Comparative change: [School change/District change]7
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 8: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/8.jpg)
8
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
That’s 4.2% better than baseline.
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 9: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/9.jpg)
9
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
That’s 4.2% better than baseline.
Participating Schools improved by 11.1 net percentage points (p<.001).
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 10: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/10.jpg)
10
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
That’s 4.2% better than baseline.
Participating Schools improved by 11.1 net percentage points (p<.001).
That’s 18.9% better than baseline.
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 11: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/11.jpg)
11
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
That’s 4.2% better than baseline.
Participating Schools improved by 11.1 net percentage points (p<.001).
That’s 18.9% better than baseline.
Participating Schools improved by 8.4 points more than Districts (p<.001).
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 12: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/12.jpg)
12
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
That’s 4.2% better than baseline.
Participating Schools improved by 11.1 net percentage points (p<.001).
That’s 18.9% better than baseline.
Participating Schools improved by 8.4 points more than Districts (p<.001).
That’s 14.7% more versus baselines.
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 13: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/13.jpg)
13
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
That’s 4.2% better than baseline.
Participating Schools improved by 11.1 net percentage points (p<.001).
That’s 18.9% better than baseline.
Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001).
Participating Schools improved by 8.4 points more than Districts (p<.001).
That’s 14.7% more versus baselines.
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 14: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/14.jpg)
14
Impacts on Math
58.4
69.5
62.8
65.5
52.054.056.058.060.062.064.066.068.070.072.0
Year 1 Year 2
Comparative Change in Math Performance
Schools
Districts
Math: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Signifi cance
Pct. Change from Yr 1
Statistical Signifi cance
Schools Mean 58.4 69.5 11.1 p<.001 4.2 18.9% p<.001 4.5StDev 21.9 21.1 Effect Size Effect Size
Districts Mean 62.8 65.5 2.6 p<.01 4.2% p<.01
StDev 20.4 20.2
Net Difference 8.4 p<.001 14.7% p<.001Level of Signifi cance p=0.065 p=0.085 p<.001 p<.001
Districts improved by 2.6 net percentage points (p<.01).
That’s 4.2% better than baseline.
Participating Schools improved by 11.1 net percentage points (p<.001).
That’s 18.9% better than baseline.
Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001).
That’s a 4.5 Effect Size versus baselines (p<.001)
Participating Schools improved by 8.4 points more than Districts (p<.001).
That’s 14.7% more versus baselines.
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 15: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/15.jpg)
15
Impacts on MathComparative Growth for Participating SchoolsIn percentage of students Proficient or Advanced:
• 11.1 net improvement for Schools (p<.001)
– 8.4 more than their respective Districts (p<.001)
– 18.9% better than their Yr. 1 baseline (p<.001)
– 14.7% better than Districts vs. baselines (p<.001)
• Effect Sizes:– 4.2 times greater improvement for net growth vs.
Districts (p<.001)
– 4.5 times greater improvement for growth from baselines vs. Districts (p<.001)
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 16: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/16.jpg)
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 17: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/17.jpg)
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
![Page 18: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/18.jpg)
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 19: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/19.jpg)
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
That’s 2.5% better than baseline.
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 20: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/20.jpg)
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
That’s 2.5% better than baseline.
Participating Schools improved by 10.3 net percentage points (p<.001).
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 21: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/21.jpg)
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
That’s 2.5% better than baseline.
Participating Schools improved by 10.3 net percentage points (p<.001).
That’s 18.1% better than baseline.
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 22: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/22.jpg)
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
That’s 2.5% better than baseline.
Participating Schools improved by 10.3 net percentage points (p<.001).
That’s 18.1% better than baseline.
Participating Schools improved by 8.7 points more than Districts (p<.001).
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 23: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/23.jpg)
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
That’s 2.5% better than baseline.
Participating Schools improved by 10.3 net percentage points (p<.001).
That’s 18.1% better than baseline.
Participating Schools improved by 8.7 points more than Districts (p<.001).
That’s 15.6% more versus baselines.
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 24: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/24.jpg)
24
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
That’s 2.5% better than baseline.
Participating Schools improved by 10.3 net percentage points (p<.001).
That’s 18.1% better than baseline.
Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001).
Participating Schools improved by 8.7 points more than Districts (p<.001).
That’s 15.6% more versus baselines.
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 25: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/25.jpg)
25
Reading: Pct. of Students Proficient OR Advanced
Year 1 Year 2Net
ChangeStatistical
Significance
Pct. Change from Yr 1
Statistical Significance
Schools Mean 56.9 67.2 10.3 p<.001 6.5 18.1% p<.001 7.2StDev 22.4 22.1 Effect Size Effect Size
Districts Mean 61.2 62.7 1.6 p<.01 2.5% p<.01
StDev 21.1 20.7
Net Difference 8.7 p<.001 15.6% p<.001Level of Significance p=0.079 p=0.063 p<.001 p<.001
56.9
67.2
61.262.7
50.052.054.056.058.060.062.064.066.068.0
Year 1 Year 2
Comparative Change in Reading Performance
Schools
Districts
Districts improved by 1.6 net percentage points (p<.01).
That’s 2.5% better than baseline.
Participating Schools improved by 10.3 net percentage points (p<.001).
That’s 18.1% better than baseline.
Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001).
That’s a 7.2 Effect Size versus baselines (p<.001)
Participating Schools improved by 8.7 points more than Districts (p<.001).
That’s 15.6% more versus baselines.
Impacts on Reading
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
![Page 26: Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration](https://reader035.vdocuments.mx/reader035/viewer/2022062408/56813669550346895d9df657/html5/thumbnails/26.jpg)
26
Impacts on ReadingComparative Growth for Participating SchoolsIn percentage of students Proficient or Advanced:
• 10.3 net improvement for Schools (p<.001)
– 8.7 more than their respective Districts (p<.001)
– 18.1% better than their Yr. 1 baseline (p<.001)
– 15.6% better than Districts vs. baselines (p<.001)
• Effect Sizes:– 6.5 times greater improvement for net growth vs.
Districts (p<.001)
– 7.2 times greater improvement for growth from baselines vs. Districts (p<.001)
Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.
*The original study of 15.3% cited has an errata underway to become 18.1%.