Evaluation Process/Requirements for

CAPP Algebra Project

Presentation Outline

Introduction

Evaluation Goals Process

Outcome Measures

Potential Research Questions

Data Collection

Questions & Discussion

Evaluation

Goals Examine the process of decision-making for how to allocate

available resources associated with the grant (including personnel) Who is involved, what type of constraints are there?

Describe how formative evaluation can be used in the schools Connections between formative assessment results and changes

in pedagogy or curriculum

Process Algebra Program participants will be active in helping to define

the process of evaluation, key questions Local knowledge and priorities will be critical to the evaluation

design Unclear how a single policy or approach can be supported

statewide given the broad differences in skills, priorities, and school contexts

Evaluation Outcome Measures

Process Changes in teacher’s understanding and use of formative

assessment in the classroom Changes in teacher’s self-efficacy and engagement with the

assessment process Student perceptions of the usefulness of formative

assessments in their own learningOutcomes

Measure student outcomes in mathematical knowledge State examinations, district benchmarks, teacher

perceptions Student college-readiness in mathematics

Possibly ACT Explore/Plan These examinations are standardized measures of college

readiness and are scaled with each other and the ACT; they could be used to look for “growth”

Potential Research Questions

Generate findings that will inform the broader research community about the process of integrating formative assessments into schools in meaningful ways. RQ 1: In what ways will teachers use the capabilities

of formative assessments to inform changes in pedagogy or curriculum?

RQ 2: What conditions are critical to successfully support integration of formative assessment into daily/weekly practice?

RQ 3: Can successful integration of mathematics formative assessment into practice improve student outcomes in mathematics?

Research Question 1

In what ways will teachers use the capabilities of formative assessments to inform changes in pedagogy or curriculum? We will create logic models for the functioning of each of

the school programs with respect to the intended outcomes Logic models are explicit statements of how specific aspects

of program design will lead to specific program outcomes – for example, how do grade-level meetings on formative assessment outcomes lead to better outcomes for students in mathematics?

The clear descriptions provided by the program logic models will allow the evaluation to examine not only if outcomes are achieved, but how they are achieved (or not) Will allow for good formative data to assist in program

improvement Additional logic models will be created for school cluster (1

HS and 2 middle schools) plans/objectives.

Research Question 2

What conditions are critical to successfully support integration of formative assessment into daily/weekly practice?

By understanding the conditions that allow for success across various contexts this evaluation can support the likely implementation of future state or district policies that have formative assessment as a critical component

Research Question 3

Can successful integration of mathematics formative assessment into practice improve student outcomes in mathematics?

As much as possible we would like to tie the particular use of formative assessments to improvement in student performance on a wide array of student assessments

Classroom, district, state, and national assessments Student surveys about engagement/self-efficacy in mathematics

Where Algebra Project participants are willing to introduce design features around implementation (which will allow for a connection between the program and student achievement), we will work with schools to get good estimates of the impact of program implementation

Staggered implementation of intervention, construction of strong comparison groups, roll-out across other schools in districts over time

Data Collection Sources/Modes

District Data Responsibilities Student Administrative Data

Administrative data include data that are typically collected by districts at the student level (achievement, grades, attendance, parental education, etc.)

Must have access to these data for comparison

School Data Responsibilities Participants in Project must be willing to participate in

surveys, interviews, and classroom observations by researchers Interviews (1 or 2 per year – 45 minutes per interview) Survey (1 or 2 short surveys – 15 minutes per survey) Observations (1-4 classroom observations per year)

Data Collection Sources/Modes (cont.)

Portfolio Assembly The nature of the portfolio will be discussed with school

teams during the planning year. Options include student-level portfolios, or Project-level portfolios or both. Student-level – track changes in student performance

on clearly identified areas of importance Project-level – track changes and components of the

project as curriculum or pedagogical changes are introduced into the classrooms

Student Responses Student surveys Student focus groups

Potential Spillovers

Training a group of teachers to use formative assessments may raise the possibility of formative assessments being used school-wide

Integration of standards and concentration on particular methods or difficulties that students have in mathematics across grades may ease transition into the new curriculum based on the common core standards

Evaluation Footprint

Our goal is to limit the intrusion of the evaluation in the school Logic models will be created by us through

discussions with Project groups in the school and then shared with the school for confirmation and modification

It is our goal to try to incorporate and support Project participant’s ideas for evaluating particular aspects of their program

Thank You for Your Time

Questions?