Damian BebellJames BurrastonCenter for the Study of Testing, Evaluation, & Educational PolicyLynch School of EducationBoston College Dec 15, 2014

Natick Public Schools 21st Century Teaching and Learning

Study: Evaluation Results

Background and perspective• Empirical educational research, measurement, and evaluation

• Background in psychometrics, measurement, and evaluation

• Leveraging technology to advance the state of the art in

educational research and measurement…

Data for the sake of datavs

Ownership of “success”and ongoing formative use of information

Quantifying computer access

US Average Student : Computer Ratios1983 = 125 : 1 1998 = 6 : 1 2007= 4 : 1 2010=3 : 1

Source: Market Data Retrieval, Public School Technology Survey 1998-2006; Education Week, 2007; Gray, Thomas, & Lewis, 2010

Shared access Sporadic use

Limited impact

1:1 Computing

The rise of 1:1 Computing

• 1:1 computing programs continuously evolving• Widespread proliferation across world• Few programs have more than 5 years experience• Fewer still have invested in empirical research and evaluation

efforts• So, lots of interest but limited research and evidence

1:1 student computing research

• 1:1 defines critical technology access…pedagogical inferences

• Emphasis on selection and mechanics of program, more limited understanding of how teaching and learning change

• Building level leadership

• Pedagogical and content support for teachers

• Systematic reflection, asking questions, and willingness to adapt/evolve

Many different definitions of success

Natick/Boston College Research PartnershipSeptember 2012- June 2014

Design and conduct external study to document impacts and changes as students and teachers transition to new technology rich computing environments.

Document through a series of reports and presentations potentially evolving teaching and learning practices afforded by the rich learning settings.

Document how Natick teachers and students accessing, using, and relating to technology in support of teaching and learning.

Provide NPS with a rich empirical model (and data) for all future inquiries and investigations concerning the relationship between technology practices and a variety of outcomes. 

www.bc.edu/natick21

Natick/Boston College Research Partnership

AcknowledgmentsSchool Leadership

TeachersStudents

District LeadershipNatick Community

www.bc.edu/natick21

Long Term Goals/OutcomesStudent Outcomes/Program Goals• Decrease behavioral and attendance issues• Decrease special education/ACT referrals • Increase in students’ computer use time in all subject areas• Increase in student use of computers for doing homework in all subject

areas• Increase in homework completion Staff/Teacher Outcomes/Program Goals• Increase in teachers use of technology-based resources to enhance

instruction• Increase in teachers allowing students to use multiple technology-based

resources to address assessment demands• Increase in the frequency that teachers structure learning opportunities

that: are inquiry-based, include a writing component, allow students to engage with and utilize technology-based resources, and allow students to demonstrate mastery

• In Year 1 (2012/2013), revision of at least one unit per trimester where technology is used to transform learning from a traditional teacher-directed curriculum-based activity to a blended learning experience with an inquiry-based mode of facilitating learning

• Decrease the number of teacher complaints concerning the lack of technology resources available for instruction

NHS Research/Evaluation Design

• 7th-12th Student Survey (*Fall 2010, Fall 2012, Spring 2013)*92% of the NPS seventh grade student population participated in a pilot survey in Fall 2010.

• 7th-12th Teacher Survey (Fall 2012, Spring 2013)

• Classroom Observations

• Focus Groups

• Staff Interviews

• Secondary analyses of school records (achievement, attendance, Special Ed status, course patterns)

• Student Video Competition

NHS Student Video Competition

“How have computers and technology changed the way you learn new things and/or how your teachers teach.”

www.bc.edu/natick21

Survey Response Rates/Parent Consent

So, what did we learn?

Selected Results

•Intro to Results •Dramatic increase in student tech use/shift in teacher practices•Exploring trends within the student results•Student and Teacher Beliefs•Exploring Quantitative Impacts and Outcomes

Results: Students’ increased use of technology across grades and subject

areas

Major increase in students’ frequency of technology use across core classes (2010-2012):

Major shifts in students’ classroompractices(2010-2012)

Major shift in teacher’s frequency of technology use in the classroom (2010-

2012):

Shifts in teachers’ classroom computer use according to students (2012-2014)

Where (and how often were) NHS students using technology in 2014?

Digging deeper in the Results: Exploring trends within student practices

(2013-2014)

Exploring gender differences in frequency of NHS boys and girls technology use across their classes

Exploring differences in student and teacher technology use across NHS students enrolled in one or more honors courses

Exploring differences in student practices comparing NHS students enrolled in one or more honors courses

Exploring differences in student practices across socio-economic status for NHS students

Results:Natick Student and Teacher Attitudes and Beliefs

Natick students report using a computer in school improves their

education:

Natick teachers report using a computer in school improves their

teaching:

Results: Empirically examining student outcomes associated

with 1:1 Computing and new facilities

Impacts on NHS student homework completion

Average number of minutes completing home per night and the percent of homework completed using a computer

Impacts on NHS student homework completion

Average homework completion rates self-reported by students

Impacts on NHS Student Attendance

Average student absences before and during first year 1:1 implementation

Impacts on NHS Student AttendanceDistribution of student absences before and during the first year of 1:1 computing

There appears to no relationship between students reported frequency of technology use in school and their absences

Impacts on NHS Student Discipline

Paired-sample t-tests examining change in frequency of detentions from the 2011-2012 school year to 2012-2013.

Impacts on NHS Student Discipline

Independent-samples t-tests comparing students technology-related disciplinary problems from 2012/2013 to 2013/2014 across grade

levels

Impacts on NHS Student Discipline

Average teacher and student perceptions of students' ability to get around the school Internet filter

Impacts on NHS Special Education

Percentage of students who received various levels of special education services during the last year in the old high school building and the first year

in the new building.

Impacts on NHS student achievement

Average ELA and Math MCAS scaled scores over time for the NHS class of 2015 and 2016

Impacts on NHS student achievement

Percentage of students who scored "proficient" or better on the MCAS from 6th to 10th grade for classes of 2015 and 2016

Impacts on NHS student achievement

Percentage of students who scored as “proficient” or higher on the 10th grade MCAS from 2009 to 2014.

Revisiting Long Term Goals/OutcomesStudent Outcomes/Program Goals• Decreasing behavioral and attendance issues• Decreasing special education/ACT referrals • Increase in computer use time in all subject areas• Increase in student use of computers for doing homework in all subject areas• Increase in homework return rates  Staff/Teacher Outcomes/Program Goals• Increase in the frequency that teachers use technology-based resources to

enhance instruction• Increase in the frequency that teachers allow students to use multiple

technology-based resources to address assessment demands• Increase in the frequency that teachers structure learning opportunities that:

are inquiry-based, include a writing component, allow students to engage with and utilize technology-based resources, and allow students to demonstrate mastery

• In Year 1 (2012/2013), revision of at least one unit per trimester where technology is used to transform learning from a traditional teacher-directed curriculum-based activity to a blended learning experience with an inquiry-based mode of facilitating learning

• Decrease the number of teacher complaints concerning the lack of technology resources available for instruction

Questions/Follow Up

• Damian Bebell, Principal Investigator• bebell@bc.edu

• James Burraston, Research Associate• james.burraston.1@bc.edu

www.bc.edu/natick21