Central Oregon PBIS ConferenceBend, OregonApril 22, 2019

Practical Problem Solving:Drilling Down Into School Data for

Improved Decision-Making

Alan Cook – University of OregonJessica Daily– University of Oregon

Presenters

Alan Cook

Training@pbisapps.orgPBIS Applications Training Team

Jessica Daily

Training@pbisapps.orgTraining Team Lead

Objectives

A. Provide language to describe the rationale for building comprehensive data collection and decision systems for monitoring data related to SWPBIS implementation

B. Describe the role of a data analyst in gathering and summarizing data prior to a team meeting to maximize meeting minutes

C. Describe the core features of data analysis or “drilling down” into data sources (e.g., fidelity surveys, referral data)

D. Practice conducting a drill down using referral data using the School-Wide Information System (SWIS)

3

Rationale for Data-Based Decision Making in SWPBIS

Research on Sustainability of SWPBIS

What single factor is most related to high sustainability of SWPBIS?

5

The frequency with which DATA are presented to all school staff

Pinkelman, S., McIntosh K., Rasplica, C. K., Berg, T., & Strickland-Cohen, M. K. (2015).

Outcome/Goal

SWPBIS and Outcomes

6

Desired Outcome: Enhanced social competence& academic achievement for all students.

How do we get to our desiredoutcome?

SystemsStaff Behavior Supports

PracticesStudent Behavior Supports

DataDecision Making Supports

SWPBIS Continuum of Supports (Tiers)

7

All specialized interventions are more effective and more durable with universal, school-wide behavioral expectations as a foundation.

Tertiary—intensive, individualized5% of student population

Secondary—targeted, small group15% of student population

Universal—primary prevention provided to all students, effective for approximately 80%

Types of Information (Data)

Fidelity Data

Outcome Data

Did we implement the systems and strategies we agreed upon?

Is the plan resulting in progress toward our goals? 8

Connecting Fidelity & Outcome Data

Lucky SustainingPositive outcomes, low understanding of how they were achieved

Replication of success is unlikely

Positive outcomes, high understanding of how they were achieved

Replication of success likely

Losing Ground LearningUndesired outcomes, low understanding of how they were achieved

Replication of failure likely

Undesired outcomes, high understanding of how they were achieved

Replication of mistakes unlikely

Out

com

es

Fidelity9

Data-Based Decision Making

Decisions are more likely to be effective and

efficient when they are based on....

DATA

The quality of data-based problem solving depends most on the first stepDefine the problem to be solved with:

• Precision(who, what, where, when, how often, why)

• Clarity(general agreement on priorities)

10

Why Use Data for Decision Making?

Data help place the “problem” in the local context rather than on the students.Data help us ask the right questions…they do not provide the

solutions. Use data to:

o Identify problemsoRefine problems and understand the contextoDefine the questions that lead to a solution

11

Problem Solving with Precision

The statement(s) of a problem is important for team-based problem solving.o Everyone must be working on the same problem with the same assumptions.

Problems are often framed in the “primary” form. o Raises awarenesso Not useful for problem solving

Precise problem statements result from a detailed data review and are solvable.

Data Team Roles

Data Collectors

All staff are part of PBIS data collection at some level and should receive training, coaching, and materials on procedures!

Survey coordinators and responders (e.g., Tiered Fidelity Inventory, School Climate Survey)

Universal Screening measures

Behavioral incident referrals

Targeted intervention data (e.g., CICO points)

Individualized intervention data (e.g., behavior support plan data)

14

Team Facilitators/Leaders

Before meetingso Schedule and communicate expectationso Coordinate tasks with minute taker, data analyst, or others

During meetingso Keep meeting focused on agenda and scheduleo Encourage active participation across members

After/Between meetingso Ensure that information (meeting minutes and tasks) are sent outo Regularly organize updates at all-staff, grade-level, and other meetings

15

Data Analyst

Standing Meetings (PBIS team, grade-level teams, data teams)o Provide standard set of reports with updated progresso Provide special-request reports on request or when potential problems are

identified o Summarize findings in person or at least in writing to support interpretation

Special Meetings (e.g., parent-teacher conferences)o Provide special-request reports on requesto Provide support in interpreting data

16

Data Integrity and Confidentiality

Inaccurate Data

False Assumptions

Poor Decisions

Accurate Data

Better Assumptions

Good Decisions

What is Data Integrity?

Data Integrity• Maintaining the accuracy and

consistency of data over its entire life cycle

• Critical aspect to the design, implementation, and usage of any system that stores, processes, or retrieves data

For data to be useful, it should reliably match overall perceptions across staff, students, and families.

18

Garbage in, garbage out

19

Garbage Data

Garbage Results

Incredible System

What is Confidentiality?

Data ConfidentialityProtection of potentially identifiable information

(In SWIS): o Person-identifiable

o Site/Building-identifiable

o Account access

Data about staff and students are highly confidential

Regular review of district policies related to student, staff, and school-level data

Keep passwords confidential

Log out every time you step away

20

Drilling Down into Your Data

Drilling Down and The Game of Clue

22

When you drill down into your data, you are like a detectivewho follows clues to isolate a problem context and function until it is precise but still socially important to address.

“Drill Down” Approach

The Drill Down approach starts with the big picture and looks for potential “red flags” that might indicate a problem that might need to be addressed. Then we identify and address the problem context and finally the function

1.What is our current reality?

2.Is there a problem (red flag)?

3.What is the precise context & function of the problem?

What, where, when, who, how often, why? Be efficient, don’t “drown” by getting too specific (e.g., red heads with blue shoes)

23

Primary

System Update/Big Picture

Red Flag

Identify Problemwith

Precision

Use DATA to define…

a PRIMARYsummary statement

a PRECISEproblem

statementthen

24

Creating a Precise Problem Statement

Drilling Down with Different Tools

Step 1: Look at overall picture What are the levels, trends,

peaks, and/or valleys?

How does this year compare with last year?

How do our data compare with national/regional norms?

How do our data compare with our preferred/expected status?

Step 2: Filter to a more precise problem statement by adding in additional data/“clues”

25

Drilling Down with the TFI (Steps 1 & 2)

Step 1. Overall (Total Score) Step 2. Breakdown by Tier

26Levels, trends, peaks?

Drilling Down with the TFI (Step 3)

Step 3. Breakdown by Tier & Subscale

27

Drilling Down with the TFI (Step 4)Step 4. Review Specific Items Use the items to build an action plan

for areas that can be improved

28

Our TFI Total Score was below 80%o Which tiers?o Which subscale(s)?o Which items?o Compared to other/previous fidelity data

“We’ve had a decreasing trend in our Tier 1 Scale for the last three months. Looks like the subscale of Implementation is the area we need to focus on, specifically with defining and teaching expectations, discipline policies, as well as feedback/ acknowledgement. Additionally, our faculty and student/family/community involvement have decreased over the same time period…….The staff are reporting similar areas as ‘not in place’ and as ‘priorities for improvement’ on the SAS.”

Asking the Right Questions: Fidelity

Primary Statement

Precise Problem

Statement

Outcomes/Referrals at Tier I (Step 1)

Step 1. Summary of overall referral level patterns, trends, & peaks

30

Outcomes/Referrals at Tier 1 (Step 2 & 3)

Step 2: Review Core Data to get an overall picture of what is happening

31

Step 3: Identify “Red Flag” to investigate further

Outcomes/Referrals at Tier I (Step 4)

Step 4. Isolate red flag and search for contextual factors (one at a time)

32

Outcomes/Referrals at Tier I (Step 5)

Step 5. Based on the context of the behavior (what, where, when, who) identify the perceived functionor motivation

Filters: Date Range: 9/1-10/31 Behavior: Defiance Time: Across DaysLocation: Mostly Classroom Grades: Mostly 6th – 8th

33

Prioritizing Multiple Problems

Prioritizing problems will likely be based upon criteria such as:

oSeverity/intensity – How dangerous is the behavior(e.g., aggression vs. inappropriate language)?

oFrequency – How often is it currently occurring?

oTrend – Is it decreasing, increasing, or consistent?

TIPS II Training Manual (2013)

Drill Down Worksheet

Drill Down Worksheet

This Drill Down Worksheet allows you to track the “steps” of the drill down • Record-keeping• Efficient

“back-tracking” • Replication

36

SWIS Drill Down Activity:

Log into: www.pbisapps.orgSWIS Demo

account“We do” & “You Do”

37

SWIS Drill Down Activity:

As we go through the Drill Down process record our steps on the SWIS Drill Down worksheet

38

565 studentsGrades 6, 7 & 8

Harbor Haven MS: Is there a problem?

Lately, the faculty buzz around Harbor Haven is that student behavior is getting worse. When we started the school year, everything seemed fine. But several months later, the teachers are complaining that they are constantly dealing with students messing around and not getting to class in a timely manner after lunch and after other breaks and transitions.

What kind of statement is this….primary or precise?

What additional information do I need to get more precise?

15

Harbor Haven MS: Is there a problem?

Is there a problem?

If so, what is it?

16SWIS Core Reports

Scho

ol-w

ide

Data

Harbor Haven MS: SWIS Core Reports A

18

What Do I Know?

43

• I know pieces of information.

• But I do not know if any of this information is connected.

• I need to drill down to look for connections.

Choosing a Red Flag

What? Inappropriate language, defiance, disruption

Where? Hallway, playground, commons, cafeteria

Who? Majority of referrals are from 8th graders

When? 8:30am-10:30am, 11am-11:30am, & 1:30pm-2:15pm

44

Data Drill Down

23

Add Hallway to our “Include in Dataset” field.

Data Drill Down

24

Change the graph type to change the lens of analysis.

Data Drill Down

25

Data Drill Down

25

Use the summary to analyze problem size.

Data Drill Down

28

Add the clues you learn to the dataset to analyze with more precision and clarity.

Change the graph type to change the lens of analysis.

Use the summary to analyze problem size.

Data Drill Down

26

Data Drill Down

26

Data Drill Down

26

Data Drill Down for Connections

Many 6th graders are having issues with being tardy in the hallway around 1:00, and the behavior might be maintained by peer attention.

30

Precise Problem Statement & Solution Development

Hallway 6th Graders Tardies at 1:00

Possibly Attention Maintained

Drill Down

54

2018-2019X

X

X

X

X

X

Hallways

6th graders

1:00

Tardy

Obtain Peer Attention

Many 6th graders are having issues with being tardy in the hallway around 1:00, and the behavior might be maintained by peer attention.

1015

SWPBIS Basic Decision Guidelines

What are indicators that a student needs additional supports for externalizing social behavior?

o Tier I 0-1 major ODRFewer than X minor/total

o Tier II 2-5 major ODRsX-X minor/total

o Tier III 6+ major ODRsX+ minor/total

55

Is the problem a system-level or student-level issue (rule of 10)?

o System Issue10+ students within a similar context (e.g., location, time of day, grade) engaging in similar behavior(s)

o Individual/Small GroupFewer than 10 students within similar context engaging in similar behaviors

Solution Development

Target Area(s): Problem behaviors in the hallways

Goal: Reduce referrals for tardies in the hallways by 50%

Solution Component Action Step(s)

Prevention-Increase active supervision. Vice-principal available at that time to monitor hallway. Grade levels establish a supervision rotation between classes.

Teaching -Reteach behavioral expectations in the hallway areas.- Retrain staff on identifying possible motivation.

Recognition-Increase recognition for appropriate behavior.-Provide feedback tickets that can be collected and used for spirit wear.-Dance party for the classroom with the fewest referrals.

Extinction-Post weekly grade-level SWIS data.-Encourage all students to work for the dance party making peer attention for problem behavior less likely.

Correct. Consq. -Active supervision and continued early consequence (ODR)

Data collection-Survey staff and students to determine whether strategies (e.g., active supervision, teaching, and recognition) are implemented consistently-Use weekly SWIS data to evaluate change

Precise Problem Statement: Many 6th graders are having issues with being tardy in the hallway around 1:00, and the behavior might be maintained by peer attention.

Solution Components

What are the Action Steps?Who is

Responsible?By When?

How will Fidelity be Measured?

Notes/Updates

PreventionIncrease active supervision Vice Principals

All staffOngoing Staff visible in the hallways,

especially during the 1:00 passing period

Teaching

Teach behavioral expectations for the commons and hallways

Teachers will teach hallway expectations to their homeroom classes

January 15 Staff sign-off sheet next to the PBIS bulletin board in the Faculty Lounge to indicate completion

Incentives for the class that can have 10 students tell the principal the expectations first

RecognitionDance PartyRecognition tickets for school store

PBIS Team will coordinate Party by January 31Tickets by January 10

Dance Party earnedtickets used at the Spirit Store

ExtinctionPost weekly dataEncourage all students to work for the Dance Party

All staff Ongoing Weekly data posted in the commons and hallway

Correct. Consq.Active supervision and continued early consequence (minor/major ODR)

All staff Ongoing Staff visible in the hallways

What data?Who is responsible for

gathering the data?When/How often will

data be gathered?Where will data be

shared?Who will see the

data?

Data CollectionODR record SWIS Data Entry person and principal

share report with teachers.Weekly Emailed to staff and posted in the

hallways and commons for students

All staff and students

Goal: Reduce referrals for 6th grade tardiness in the hallways by 50%

Big Idea About Sharing Data

58

Know your audience• What does the team need to know

to make good decisions?

• What do staff need to know? Is there anything they need to do?

• Who else would benefit from seeing an update on student behavior (e.g., students, families, parent group)?

Tell the storyHelp paint the picture with data: What, where, when, who, how often, why?

Big Idea About Sharing Data

59

Drill DownExample #2

Drill Down #2

Let’s see what happens when we start our drill down procedure with a problem behavior instead of a location.

Staff are reporting that there is a rise of inappropriate language in the K-3 wing of the school.

I’ll open up the Drill Down Tool

For this demo, I’m going to change the dates to last school year. You can choose to limit or expand the time frame depending on your search.

Don’t forget, I’m going to record my steps carefully using the SWIS Drill Down Worksheet. I can always use this sheet to backtrack if I need to.

61

Drill Down #2

71

Tools to Get Organized with

SWIS

Drill Down Video Tutorial

73https://www.pbisapps.org/Resources/Pages/Data-Drill-Down-How-To.aspx

Drill Down for Disproportionality

74PBISApps.org | Ethnicity: Data Drill Down

Drill Down Worksheet

75

SWIS Schedule

76

You can also check out the back cover of your SWIS manual for a reporting schedule. This will come in handy for delegating which reports will be run and by whom.

Connect With Your SWIS Facilitator

77

Your SWIS facilitator has been specifically trained on how to effectively use the Drill Down tool. If you have any questions or need any pointers, you can always get in touch with them.

Not using Drill Down?

78

Not using Drill Down is like throwing darts blindfolded. Good luck hitting the target!

~Tia Rupe, SWIS Facilitator

Review

Questions? Comments? Thanks for attending our Drill Down session today. Please don’t hesitate to reach out to us either at support@pbisapps.org

or training@pbisapps.org if you need any assistance or have further questions!

79