msp program evaluation carol l. fletcher, ph.d. trc project director meetings 1/27/09 and 2/5/09...

29
MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09

Upload: madeleine-norton

Post on 19-Jan-2016

218 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

MSP Program EvaluationMSP Program Evaluation

Carol L. Fletcher, Ph.D.TRC Project Director

Meetings1/27/09 and 2/5/09

Carol L. Fletcher, Ph.D.TRC Project Director

Meetings1/27/09 and 2/5/09

Page 2: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

TRC CriteriaTRC Criteria

• How has your Regional Collaborative impacted• Teacher Content Knowledge• Classroom Practice• Student Achievement

• How has your Regional Collaborative impacted• Teacher Content Knowledge• Classroom Practice• Student Achievement

Page 3: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Guskey ModelGuskey Model

• Based on mediators of relationship between professional development and improvements in student learning• i.e. administrator knowledge and

practice, school culture, teacher knowledge and practice, parent knowledge and practice

• Based on mediators of relationship between professional development and improvements in student learning• i.e. administrator knowledge and

practice, school culture, teacher knowledge and practice, parent knowledge and practice

Page 4: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Five Critical Levels of PD Evaluation

Five Critical Levels of PD Evaluation

• Inform planning, formative and summative evaluations

• Helps to identify the “what” and the “why” evaluation

• Hierarchically arranged from simple to complex

• Success at one level is necessary for success at following levels

• Inform planning, formative and summative evaluations

• Helps to identify the “what” and the “why” evaluation

• Hierarchically arranged from simple to complex

• Success at one level is necessary for success at following levels

Page 5: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Level 1: Participants’ ReactionsLevel 1: Participants’ Reactions

• Did they like it?• Will it be useful• Was the leader

knowledgeable and helpful?

• Where the facilities comfortable?

• Did they like it?• Will it be useful• Was the leader

knowledgeable and helpful?

• Where the facilities comfortable?

• Questionnaires• Interviews• Participant

journals

• Questionnaires• Interviews• Participant

journals

What questions are addressed? How will information be gathered?

Page 6: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Level 2: Participants’ LearningLevel 2: Participants’ Learning

• Did participants acquire the targeted knowledge and skills?

• Did participants acquire the targeted knowledge and skills?

• Questionnaires• Interviews• Participant

journals/portfolios• Demonstrations

• Questionnaires• Interviews• Participant

journals/portfolios• Demonstrations

What questions are addressed? How will information be gathered?

Page 7: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Level 3: Organization Support and Change

Level 3: Organization Support and Change

• Did it affect organizational climate, procedures, or policies

• Were sufficient resources made available?

• Were successes recognized & shared?

• Did it affect organizational climate, procedures, or policies

• Were sufficient resources made available?

• Were successes recognized & shared?

• Questionnaires• Structured

Interviews• Participant

journals/portfolios• Focus groups• District/school

records

• Questionnaires• Structured

Interviews• Participant

journals/portfolios• Focus groups• District/school

records

What questions are addressed? How will information be gathered?

Page 8: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Level 4: Participants’ use of new knowledge and skills

Level 4: Participants’ use of new knowledge and skills

• Did participants effectively apply the new knowledge and skills?

• Did participants effectively apply the new knowledge and skills?

• Questionnaires• Structured

Interviews• Participant

journals/portfolios/ reflections

• Direct observation

• Videotapes

• Questionnaires• Structured

Interviews• Participant

journals/portfolios/ reflections

• Direct observation

• Videotapes

What questions are addressed? How will information be gathered?

Page 9: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Level 5: Student learning outcomesLevel 5: Student learning outcomes

• Did it affect student performance or achievement?

• Are students more engaged learners?

• Do students select more rigorous coursework?

• Did it affect student attitudes toward science or math courses/careers?

• Did it affect student performance or achievement?

• Are students more engaged learners?

• Do students select more rigorous coursework?

• Did it affect student attitudes toward science or math courses/careers?

• Questionnaires• School/student

records• Structured

Interviews• Participant

journals/portfolios/ reflections

• Direct observation• Videotapes

• Questionnaires• School/student

records• Structured

Interviews• Participant

journals/portfolios/ reflections

• Direct observation• Videotapes

What questions are addressed? How will information be gathered?

Page 10: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

NSF Developed ToolsNSF Developed Tools

• Horizon Research• Assessing Teacher Learning About

Science Teaching (ATLAST)• www.horizon-research.com/atlast/• Instruments to measure student and

teacher science content knowledge• Force and Motion• Flow of Matter and Energy in Living

Systems• Plate Tectonics

• Horizon Research• Assessing Teacher Learning About

Science Teaching (ATLAST)• www.horizon-research.com/atlast/• Instruments to measure student and

teacher science content knowledge• Force and Motion• Flow of Matter and Energy in Living

Systems• Plate Tectonics

Page 11: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

ATLAST continuedATLAST continued

• Student tests - paper copies only• Teacher tests - paper or online versions• The student and teachers assessments

measure knowledge of the science content, while the teacher assessment also measures a teacher's ability to use science content knowledge to analyze student thinking, and to make instructional decisions. All of the items on the teacher assessment are set in instructional contexts.

• Student tests - paper copies only• Teacher tests - paper or online versions• The student and teachers assessments

measure knowledge of the science content, while the teacher assessment also measures a teacher's ability to use science content knowledge to analyze student thinking, and to make instructional decisions. All of the items on the teacher assessment are set in instructional contexts.

Page 12: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

ATLAST continuedATLAST continued

• HRI provides the following analyses:• Percent of respondents choosing each

answer choice;• Percent of respondents answering each

item correctly;• Group mean score; and• Significance testing and computation of

effect sizes for pre- and post-test group mean scores.

• HRI provides the following analyses:• Percent of respondents choosing each

answer choice;• Percent of respondents answering each

item correctly;• Group mean score; and• Significance testing and computation of

effect sizes for pre- and post-test group mean scores.

Page 13: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings
Page 14: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

MOSARTMOSART

• Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART)

• Harvard; PI - Phillip Sadler• Content instrument for K-12 physical

science and earth science, based on research about student misconceptions

• Completion of online tutorial (~30 min) required to download instruments

• Misconception Oriented Standards-based Assessment Resource for Teachers (MOSART)

• Harvard; PI - Phillip Sadler• Content instrument for K-12 physical

science and earth science, based on research about student misconceptions

• Completion of online tutorial (~30 min) required to download instruments

www.cfa.harvard.edu/smgphp/mosart

Page 15: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings
Page 16: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

MOSART continuedMOSART continued

• K-4 & 5-8 Physical Science, Earth Science, Astronomy/Space Science

• 9-12 Physics, Chemistry, Earth Science, Astronomy/Space Science

• Multiple versions of the test are available for pre/post administration

• K-4 & 5-8 Physical Science, Earth Science, Astronomy/Space Science

• 9-12 Physics, Chemistry, Earth Science, Astronomy/Space Science

• Multiple versions of the test are available for pre/post administration

Page 17: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

MSPnet ToolboxMSPnet Toolbox

• http://hub.mspnet.org/index.cfm/msp_tools

• Clearinghouse for tools used and shared by MSP projects

• Tools may include assessment instruments, evaluation protocols, form letters, etc.

• http://hub.mspnet.org/index.cfm/msp_tools

• Clearinghouse for tools used and shared by MSP projects

• Tools may include assessment instruments, evaluation protocols, form letters, etc.

Page 18: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings
Page 19: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

University of LouisvilleUniversity of Louisville

• Diagnostic Assessments for Middle School Teachers

• Each assessment is composed of 25 items—20 multiple-choice and 5 open-response.

• Six versions of each assessment are available in paper-and-pencil format

• Diagnostic Assessments for Middle School Teachers

• Each assessment is composed of 25 items—20 multiple-choice and 5 open-response.

• Six versions of each assessment are available in paper-and-pencil format

http://louisville.edu/education/research/centers/crmstd/diag_sci_assess_middle_teachers.html

Page 20: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

University of Louisville (cont.)

University of Louisville (cont.)

• Assessments available free of charge• Optional scoring service for $10/teacher• Detailed summary of teachers'

performance that includes scores on individual items, on each science subdomain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge)

• Assessments available free of charge• Optional scoring service for $10/teacher• Detailed summary of teachers'

performance that includes scores on individual items, on each science subdomain in the content area, and on four different knowledge types (memorized, conceptual understanding, higher-order thinking, pedagogical content knowledge)

Page 21: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

University of Louisville (cont.)

University of Louisville (cont.)

Physical Science

Life Science Earth/Space Science

Matter Structure/Function Atmosphere/Hydrosphere

Motion & Forces Internal Regulation Lithosphere

Energy Heredity/Diversity Space

Interdependence

Page 22: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

MSP Knowledge Management and

Dissemination

MSP Knowledge Management and

Dissemination• Knowledge reviews good for citing

research base• Searchable Instrument database

for science and math

• Knowledge reviews good for citing research base

• Searchable Instrument database for science and math

http://www.mspkmd.net/

Page 23: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings
Page 24: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Online Evaluation Resource Library (OERL)

http://oerl.sri.com/

Online Evaluation Resource Library (OERL)

http://oerl.sri.com/

• Teacher and student instruments• Content assessments, interviews,

surveys,classroom observations

• Teacher and student instruments• Content assessments, interviews,

surveys,classroom observations

Page 25: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings
Page 26: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Thinking About Mathematics Instruction

(TMI)

Thinking About Mathematics Instruction

(TMI)• http://www2.edc.org/tmi/

tmi_survey.html

• Survey to investigate elementary and middle school principals leadership content knowledge (LCK) for mathematics

• http://www2.edc.org/tmi/tmi_survey.html

• Survey to investigate elementary and middle school principals leadership content knowledge (LCK) for mathematics

Page 27: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings
Page 28: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings

Distributed LeadershipDistributed Leadership

• http://www.sesp.northwestern.edu/dls/instruments/

• Instructional Leadership Daily Practices Log

• Principal Experience Sampling Method (ESM) Log

• School Staff Network Survey

• http://www.sesp.northwestern.edu/dls/instruments/

• Instructional Leadership Daily Practices Log

• Principal Experience Sampling Method (ESM) Log

• School Staff Network Survey

Page 29: MSP Program Evaluation Carol L. Fletcher, Ph.D. TRC Project Director Meetings 1/27/09 and 2/5/09 Carol L. Fletcher, Ph.D. TRC Project Director Meetings