Slide 1Slide 1

Educator Preparation Advisory Council (EPAC)

Data SubcommitteeFebruary 25, 2014

Slide 2

Welcome and Introductions

Welcome

Dr. Sarah Barzee

Introductions

Future meetings: schedule 1-2 hour virtual meetings/conf calls with pre-work

EPAC workplan for 2013-14

Underlying Assumptions of EPAC Charge

The quality of instruction plays a central role in student learning (academic, behavioral and social).

Educator preparation programs ensure baseline knowledge, skills and dispositions are demonstrated (CCT, SLS, CCSSO Learner Ready Definition, etc.) and contribute to the quality of instruction.

Overall goal is to improve programs, not ensure compliance, and provide useful information for improvement of preparation policy and practice.

Adapted from Evaluation of Teacher Preparation ProgramsNational Academy of Education, 2013

Slide 3

Purposes of Educator Preparation Program

EvaluationEnsuring accountability and monitoring program quality

and providing reliable information to the general public and policy makers

Providing information to consumers to help them make choices about preparation programs and providing future employers information to support hiring decisions

Supporting continuous program improvement with relevant performance data and measures that can identify strengths and weaknesses of existing programs

Adapted from Evaluation of Teacher Preparation ProgramsNational Academy of Education, 2013

Slide 4

Validity of Program Approval Decisions

Must be based on multiple measures (quantitative and qualitative data)

Construct validity

Content validity

Predictive validity

Consequential validity

Evaluation system must be adaptable to changing educational standards, curricula, assessment and modes of instruction

Slide 5

Slide 6

Program ApprovalProcess

Review of Programs based on EPAC Principles 1-5, multiple measures and qualitative criteria as well as statutory requirements

PROGRAM APPROVAL PROCESS and DECISION BY the STATE BOARD OF

EDUCATION(at the individual program, not institutional, level)

Data and Accountability System

Performance Categories:•Recruitment and Completion Rates•Employment and Retention Rates•Pre-Service Performance Rates•Educator Effectiveness (surveys, eval data)•District Partnership Quality

+

Will Determine

EDUCATOR PREPARATION PROGRAM APPROVAL Based on EPAC Principles

Assessment Subcommittee To review and make recommendations on new assessments to be developed as part of accountability system

Slide 7

Inter-related Work of EPAC Subcommittees

Program Approval: Develop a new, more rigorous program approval process and regulations to guide approval decisions by the State Board of Education (SBE) based on review of efficacy of curriculum, as well as accountability data on a program’s measures of quality.

Data Collection, Analysis and Reporting: Develop a new data collection, analysis and reporting system for institutional reporting and an accountability system for program approval, as well as provide biennial research data on supply and demand, to inform continuous improvement. Data from accountability system will be linked with program approval decisions

Assessment: Guide development of new assessment options including performance assessments, clinical experience evaluations, feedback surveys. Data from new and existing assessments will be used in the data and accountability system.

Slide 8

Data Subcommittee Outcomes EPAC Principles

1. Program Entry Standards2. Staffing & Support of Clinical Experiences3. Clinical Experience Requirements4. District-Program Partnerships & Shared Responsibility5. Program Completion & Candidate Assessment Standards6. Program Effectiveness & Accountability

Develop: An institutional reporting system An accountability system to be used as part of

program approval Biennial data report on supply and demand

Use supports from CCSSO/NTEP cross-state collaborative, 2013 – 2015

Creating a Culture of EvidenceUse quantitative and qualitative data to make valid

inferences (interpretations and findings) that inform program improvement

Shift focus from compliance to inquiry and program improvement

Data must inform collaboration and shared responsibility for IHE faculty and staff to review and make changes in program structure, practices, policies and teaching

School based faculty and administration also have a shared responsibility to collaborate with IHE partners

Adapted from Evaluation of Teacher Preparation ProgramsNational Academy of Education, 2013

Slide 9

Attributes and Measures Related to Program

QualityTwo tables from Evaluation of Teacher

Preparation Programs, National Academy of Education, 2013

Table 2.1 Attributes Related to TPP Quality and

Evidence Used to Measure Them (page 27)

Table 2.2 Main Types of Evidence Used by

Different TPP Evaluation Systems (page 60)

Slide 10

Slide 11

Slide 12

6 EPAC Principles & Alignment to Performance Indicators

1. Program Entry Standards

2. Staffing & Support of Clinical Experiences

3. Clinical Experience Requirements

4. District-Program Partnerships & Shared Responsibility

5. Program Completion & Candidate Assessment Standards

6. Program Effectiveness & Accountability

Slide 13

Title II HEA Mandate for Accountability of Teacher

Preparation ProgramsExpectation of identifying At-Risk or Low-

Performing preparation programs has been in place since 1999-2000 Title II HEA mandate; this work will redefine criteria for effective, at-risk and low-performing programs

See current definition of At-Risk or Low-Performing institutions that has been in place since 2000 and the basis of reporting low-performing IHEs in the annual Title II state report

Slide 14

Designing the Data and Accountability System

Identify accountability categories

Are the previously identified accountability categories sufficient? Recruitment and Completion Rates Employment and Retention Rates Pre-Service Performance Rates Educator Effectiveness (surveys, eval data) District Partnership Quality

How will we weight each category? Are they all equally weighted?

How do we measure these? Which data points from existing assessments or new assessments do we use?

Is there a “trigger” of any specific data point(s), categories that would require immediate program review?

Slide 15

Designing the Data and Accountability System

Data currently available: Title II Report (refer to 2013 report

https://title2.ed.gov/Public/Home.aspx) Employment/Staff File Data http://www.sde.ct.gov/sde/cwp/view.asp?

a=2613&q=322570 Completer Rates (in Title II report and ETS Title II system) Pass Rates (Praxis II, Foundations of Reading, CAT, ACTFL)

http://www.sde.ct.gov/sde/cwp/view.asp?a=2613&Q=333728

Assessments yet to be designed or implemented to provide necessary data points Feedback surveys from teachers Feedback surveys from employers Pre-service assessments Statewide student teaching/clinical experience evaluation

instrument Measure of IHE/District Partnership quality

Slide 16

Slide 17

Cautions about Data Use and Reporting

Consider the source (self-report, district, IHE, feds, etc.)

Consider its “completeness” or missing data

Consider the quality of the data

Comparisons of different N size

Limitations to certain data points as part of the annual data reporting or as part of the accountability system

Evolution and adaptability of data system and accountability system over time

Slide 18

Case Study: LouisianaReview Louisiana’s accountability system for

educator preparation program

First adopted in 2003

Revised version adopted in 2013

Review 2011-2012 Annual Report for Teacher Preparation

Slide 19

Designing IHE Data and Performance Reports: Next

StepsDiscuss how to annually or biennially report educator

preparation data and performance profiles

Build off of existing data systems such as Title II, Certification, CSDE Staff File, etc.

Review and consider a “dashboard” system for displaying annual profile data on each institution and individual program (to the extent that we have program level data and meets the suppression test)

Identify key design features desired for this on-line reporting system

Slide 20

Next Meeting (virtual) Survey subcommittee for recommendations about:

Categories to be used in accountability system

Weighting of categories

Underlying data points within each category

Overall rating system: Identify levels (e.g., low-performing, at-risk, effective, etc.)

Identify whether system measures are calculated annually, biennially or other

Trigger for off-cycle review: which data point or category can trigger a program approval review outside of established cycle if overall rating system?

April meeting: debrief and finalize the above recommendations

Set dates for: April, May and JuneSlide 21

Next MeetingsMay: Presentation from Ed Klonoski, President

of COSC, on Dashboard System Design for IHE Profiles and Performance Reports Design similar to what we have for school district profiles?

June: Supply and Demand Study preview and summary of data compiled and analyzed

Slide 22