northern california pbis symposium november 18, 2013
TRANSCRIPT
Using PBIS Assessment to Create School and District
PBIS Evaluation Plans
Northern California PBIS SymposiumNovember 18, 2013
Are we doing what we said we would do?
Agenda
Overview of available surveys/tools on the PBIS Assessment website
Developing an Evaluation Action PlanDecisions on which tools to chooseDecisions on timelines
Assessment Example Plans
Developing Evaluation Reports
Questions/Comments
Acknowledgments
School-wide Positive Behavior Support: Implementers' Blueprint and Self-Assessment
Sugai, G., Horner, R.H., Algozzine, R., et.al. (2010).
Evaluation Blueprint for School-Wide Positive Behavior Interventions & Support
Algozzine, R., Horner, R.H, Sugai, G., et.al. (2010).
Fidelity Measures to Improve Implementation of Positive Behavioral Support
Tobin, T. J., Vincent, C. G., Horner, R. H., Rossetto Dickey, C., & May, S. A. (2012).
PBIS Evaluation Blueprint
“A hallmark of School-wide Positive Behavior Support (SWPBS) is a commitment to formal evaluation.”
“Evaluation is the process of collecting and using information for decision making.
Foundations
Evaluation is the process of gathering data toAnswer questionsMake decisions
Define the core questions/decisions, before selecting measures.
Provide an adequate context for interpreting the data.
Effective evaluation is cyclical (repeated)Use the evaluation process for improvement
Evaluation Cycle
Core Questions
Evaluation Plan
Data Collection
Evaluation Report
Core Evaluation QuestionsContext
What are/were the goals and objectives for SWPBIS implementation? (state/district capacity; school adoption; student outcomes)
Who delivered the training and technical assistance for SWPBIS implementation?
InputWhat professional development events were part of SWPBIS implementation support?
Was projected level of TA capacity provided (training/coaching)?
Who received the professional development (schools/ cohorts)?
Core Evaluation QuestionsFidelity
To what extent was Tier I SWPBIS implemented with fidelity ?
To what extent were Tier II and Tier III SWPBIS implemented with fidelity?
To what extent is the Leadership Team establishing core functions?
Leadership Team
Active Coordination
FundingVisibility Political
Support
Training Coaching Evaluation
Local School/District Teams/Demonstrations
BehavioralExpertise
Policy
Sugai et al., www.pbis.org
Core Evaluation QuestionsImpact
To what extent is SWPBIS associated with changes in student behavioral outcomes?
To what extent is SWPBIS associated with changes in academic performance, and dropout rates?
To what extent is district/state capacity established? (local training, coaching, evaluation, behavioral expertise)
To what extent is leadership and policy structure established?
Core Evaluation Questions
Replication, Sustainability, and ImprovementTo what extent do Fidelity and Impact outcomes sustain across time?
To what extent does initial SWPBIS implementation affect Implementation with later cohorts?
To what extent did SWPBIS implementation change educational/behavioral capacity/policy?
To what extent did SWPBIS implementation affect systemic educational practice?
Evaluation Questions
FidelityTo what extent was SWPBS implemented as designed?
To what extent was SWPBS implemented with fidelity?
“Are we doing what we said we would do?”
To What Extent Can
PBIS Applications.org
PBIS Assessment
PBIS Evaluation
Help us answer the question:“Are we doing what we said we would do?”
Current Tools in PBIS Assessment
Research Tool
Annual Assessment
Tool
Progress Monitoring
Tool
UniversalSystem
School-wide Evaluation Tool
(SET)
Benchmarks of Quality (BoQ)
Early Childhood BoQ (ECBoQ)
PBIS Self-Assessment Survey (SAS)
Team Implementation
Checklist (TIC)
Secondary &TertiarySystems
Benchmarks of Advanced Tiers (BAT)
PBIS Self-Assessment Survey (SAS)
Monitoring Advanced Tiers
Tool (MATT)
Outcome Measures: School Safety Survey
Current Tools in PBIS Assessment
Research Tool
Annual Assessment
Tool
Progress Monitoring
Tool
UniversalSystem
School-wide Evaluation Tool
(SET)
Benchmarks of Quality (BoQ)
Early Childhood BoQ (ECBoQ)
PBIS Self-Assessment Survey (SAS)
Team Implementation
Checklist (TIC)
Secondary &TertiarySystems
Individual Student School-wide Evaluation
Tool(ISSET)
Benchmarks of Advanced Tiers (BAT)
PBIS Self-Assessment Survey (SAS)
Monitoring Advanced Tiers
Tool (MATT)
Outcome Measures: School Safety Survey
www.pbisapps.org
A Tour of
at
Evaluation Action PlanningDecisions:
Evaluation ToolsWhich tools answer your district/schools’ evaluation questions?
Evaluation TimelineWhen/how often will you administer the tools?
Evaluation ReportingHow will you report outcomes to schools and to the District?How often will you report outcomes to schools and to the District?
Evaluation Action Planning
Decisions about Evaluation ToolsDoes your District/School need a research tool?
(SET/ISSET)
Which annual assessment tool(s) will you use?BoQ, SAS, SSS, BAT, ECBoQ
Which progressing monitoring tool(s) will you use?TIC, MATT
Does your District/School need to evaluate Tier I or Tier II & III implementation or all three?
Evaluation Action Planning
It is about…
CONTEXTYour settings => Your needs
Which Surveys Meet Your
District’s Needs?
Example District A
Evaluation Questions
What is our baseline, and what has been our progress through the year?
Are we improving on our implementation of PBIS?
How do our individual staff members rate our implementation?
Are our self-assessments consistent with data from outside evaluators?
Newly implementing, grant needing specific research data as well as fidelity of implementation data
District A: Evaluation Plan
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
SET X X X X
TIC X X X X X X X X X
Safety Survey X X X X
SAS X X X X
Example District B
Evaluation Questions
In what specific PBIS areas are we strong, and in which areas do we need more training?
Are we improving on our implementation of PBIS?
How do our staff members actually rate our PBIS implementation?
How is our implementation of Tiers II and III?
Implementing PBIS for three years, have been using the TIC/SET, and have been consistently over 80%.
District B: Evaluation Plan
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
BoQ X X X
SAS X X X
Safety Survey X X X
ATT X X X X X X X X X
Example District C
Evaluation Questions
Which schools are not meeting the SET benchmark?
In what specific PBIS areas are we strong, and in which areas do we need more training?
How can we tailor our training and evaluation to meet the needs of various schools?
Are we improving on our implementation of PBIS?
How do our individual staff members rate our implementation?
Implementing for five years, some schools are doing very well, but others are not making progress.
District C: Evaluation Plan: Cohort 1
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
SASSurvey X
SET X X X
Safety Survey X X X
TIC X X X X X X X X X
Schools still not performing well
District C: Evaluation Plan: Cohort 2
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
SASSurvey X
BoQ X X X
Safety Survey X X X
MATT X X X X X X X X X
Schools performing well
Example District D
Evaluation Questions
In what specific PBIS areas are we strong, and in which areas do we need more training?
In what academic areas are we strong, and in which areas do we need more improvement?
How can we tailor our training/evaluation to meet the needs of various schools?
How can we schedule our evaluations so that we are not overwhelming our schools?
The district wants to bring RtI Behavior and Academics together and create one evaluation plan.
Assessment Instrument
Fall (August/Sept/Oct)
Winter (Dec/Jan)
Spring(April/May) Purpose
Team Implementation Checklist (TIC)
X X X Fidelity of Implementation
Self-Assessment Survey
X Fidelity of implementation
School Safety Survey
X Outcomes related to Protective and Risk Factors
SET X Fidelity of Implementation
AIMsWeb X X XScreening
Progress Monitor
SAT10 X Summative Evaluation
DIBELS X X X Screening
Progress monitoring
Nieves Flores, 2012
Task: Developing Your Plan
Questions to Consider:Which surveys are your schools/districts currently using?Do they answer your evaluation questions:
Are we implementing SWPBIS with fidelity?Is there more information that we need?
Pbisapps.org > PBIS Assessment > Find Out More about Surveys
Do your surveys assess the Tiers you are implementing?Do you need a research tool?
Task: Developing Your Plan
Are you “topping out” of the surveys you are currently taking?
Do you need to customize your evaluation plan for different schools, or will one evaluation plan fit for all schools?
Do you want to assess all tiers of PBIS implementation?
Is your Evaluation Plan manageable for schools?
District Evaluation Planning Template
Measure Year One Year Two Year Three
Fall Winter Spring Fall Winter Spring Fall Winter Spring
SWPBIS School or District Evaluation Reports
Basic Contents of SWPBIS Evaluation Reports
From the PBIS Evaluation BlueprintContext
Input
Impact on SWPBIS Fidelity
Impact on Student Outcomes
Implications for the Future
Context
Define SWPBIS
Define goals of the specific project
Define stakeholders and evaluation questions
Input
Who received what support and from whom?Leadership team
Local Capacity Building Training, Coaching, Behavioral Expertise, and Evaluation
School teams
Impact on SWPBIS Fidelity
Leadership TeamSWPBIS District Leadership Team Self-Assessment
Capacity DevelopmentNumber of trainers/coaches available to support school teams/districtsBehavioral expertise available to support Tier II and Tier III implementationEvaluation capacity
School TeamsTier I Implementation (TIC, BoQ, SET, SAS)
Collectively and/or by training cohortTier II / Tier III Implementation (MATT, BAT, ISSET)
Collectively and/or by training cohort
Impact on Student Outcomes
Behavioral OutcomesOffice Discipline ReferralsSuspension/ExpulsionAttendanceDrop Out/GraduationDisproportionality in ODR and suspension/expulsion
Academic outcomesCurriculum Based Measures
Oral Reading Fluency, Dibels, AimsWebStandardized Measures
State and NationalMath, reading, writing assessments
Impact and Future Directions
Advances to datePlans for next three yearsAreas in need of adaptation
Narrowing of evaluation questions and concerns
Sharing Evaluation Reports
State and District examples at PBIS.orgSend in your district examples, so we can all learn!
Support and Resources
Portal Support SectionSearch FunctionCommunity ForumChat with us online
Account Questions:[email protected]
Technical Assistance Questions:
New phone number1-855-455-8194
Questions & Answers
When you leave the webinar, please complete the survey.
THANK YOU!