selecting tier 1 fidelity instruments: perspectives from ... · pbis was developed initially in the...

PhillipWhitefield,UniversityofSydney

CarlaDeAngelis,NSWDepartmentofEducation

Yasmin Ibrahim,NSWDepartmentofEducation

SelectingTier1FidelityInstruments:Perspectives

fromEvaluators

Introduction

• FocusonimplementationissuesforSchool-WidePositiveBehaviour Support/PositiveBehaviour forLearning

• Reportingonpreliminaryaction-basedpilotstudyofusabilityfactorsoffidelityinstrumentsbyexternaladvisorsinAustralianstates,mainlyNSW.

• Theinformationisintendedtocontributetoimprovementgoals,byfocusingonthetechnicalassistanceprovidedtoschools.

2

Acknowledgements

3

Wewouldliketoacknowledgethetechnicalassistancereceivedoverthe

pasttenyearsfromProfessorTimLewis,Dr LoriNewcomer,

ProfessorRobHorner&ProfessorTerryScott.

APBS:PBISAustralia

4

• AssociationofPositiveBehaviour SupportnetworkinAustralia

• Australia-widenetworkacrossschoolsectors,universities,agencies

• Advocacy,Research,Facilitation,Sustainability

PBIS Australia is a group of educationalists from universities, school and other agency sectors across Australia. It was established in 2013 in response to increasing numbers of Australian schools and systems implementing PBIS .The purpose of PBIS Australia is to improve the access to and scope of PBIS (Positive Behaviour Interventions and Support) implementation across Australia. This cooperative approach encourages innovation and sharing of information and resources across and between university and school sectors and promotes wellbeing in schools. PBIS Australia networks nationally and internationally to provide advocacy for research and implementation.

What is Positive Behaviour Interventions and Support (PBIS)? Positive Behaviour Interventions and Support (PBIS) is a problem solving process for creating safer and more effective schools. The process is based upon a three-tiered continuum from universal prevention to individualised intensive supports and emphasises analysing data to inform decision making, identifying systems that support staff, and identifying, implementing and evaluating evidenced-based practices that improve the social-emotional and learning outcomes of all students. PBIS is a proactive instructional approach to discipline aiming to increase the likelihood that students and staff are responsible, safe, respectful and engaged learners.

Implementation of PBIS involves building the capability of teachers to embed the teaching and monitoring of social-emotional skills into the curriculum. Some students, approximately 10-15%, will respond to Tier 1 universal prevention supports but will still exhibit some specific difficulties. As with Tier 1 systems, Tier 2 targeted support is a team driven process designed to enhance and build upon what has been taught to students at the universal level. A smaller group of students, approximately 1-5%, may require more individualised and intensive supports, in conjunction with Tier 1 and Tier 2 supports. Interventions focus on creating and implementing individualised behaviour support plans that are linked to the universal system.

PBIS supports a cultural shift through the use of teacher performance feedback to increase the likelihood that effective, evidence-based strategies will be applied. The use of these strategies increases the self-efficacy of teachers and reduces the level of unproductive classroom behaviours. Student capabilities are aligned with the goal of developing self-discipline, connectedness to school and positive student-teacher relationships. These factors are strongly associated with an increased protection from risks such as bullying, violence and substance abuse.

PBIS was developed initially in the USA over 30 years ago. It is solidly grounded in evidence based practices and systems and is currently being implemented in over 20,000 USA schools and in many countries worldwide.

Implementation of PBIS in Australia Over 2,000 Australian schools in all states and territories are implementing PBIS systems. PBIS

Australia aims to determine the scope of implementation in Australian schools. PBIS in Australia is referred to variously as Positive Behaviour for Learning (PBL), Positive Behaviour Supports (PBS) and School Wide Positive Behaviour Supports (SWPBS).

PBIS Australia Strategic Directions

Sustaining and supporting the scale up of PBIS implementation in Australia

Facilitating collaboration and advocacy for PBIS framework/systems in Australia

Promoting research on PBIS implementation fidelity and efficacy in Australia

PBIS Australia

NewSouthWalesPublicSchools

Approximately2200PublicSchoolsinNSWeducatingover790,000studentsin:PreschoolsInfantsschools

(Kindergarten- Year2)

Primaryschools(Kindergarten- Year6)

HighSchools(Year7-Year12)

SchoolsforSpecificPurposes(includinghospitalschoolsandSpecialEducationsettings)

JuvenileJusticeSchools

DistanceEducation

5

QualityofLifeObjectives(Carr,2007)

“Ourchiefconcernisnotwithproblembehaviour,andcertainlynotwithproblempeople,butratherwithproblemcontexts”

Outcomes to be Sustained

• Environments that encourage pro-social behaviour

• Environments where positive behaviour is more effective than problem behaviour

• Systems that support teaching, learning and leading

• Adults and students knowing what is expected of them.

• Student academic and social & emotional learning is supported by a continuum of

practices.

ThePBLProcess

WellbeingOutcomesandAcademicAchievement

SupportingDecisionMakingSupportingStaff

SupportingStudentOutcomes

Planned processes & procedures that support adults to use evidence-based educational

programs as intended

Systems in schools

SignificanceofImplementationBlueprints

SignificanceofImplementationBlueprints

11

Improving Student Performance – 7 Themes

Whatworksbest:Evidence-basedpracticestohelpimproveNSWstudentperformanceCentreforEducationStatisticsandEvaluation

1. High expectations

2. Explicit teaching

3. Effective feedback

4. Use of data to inform practice

5. Classroom management

6. Wellbeing

7. Collaboration

WellbeingFrameworkforschoolsAContextofWellbeing&Prevention

Discourse

PositiveBehaviourForLearninginNSW

ApplicationsofMulti-TieredSystemsofSupport

ImplementationFidelityMatters!

• Implementingasintended

• Positivelypredictsstudentoutcomes

• Validationpurposes• Hard-wiredintoPBL/SWPBSprocess

• Actionplanningpurpose(notascoringpurpose)• Researchtopracticebridging• Improvedoutcomesforourteachers&students

• Datahasashortlife-span

17

TypesofFidelity(Harn,Paris &Stoolmiller,2013)

Structural/surface

• Objectiveevaluationofthedeliveryofcomponents

• Measuredbyobservationorself-report.

Process

• Examiningthequalityofthedelivery(rating,subjectivequalities).

Multi-dimensional

• Combinationoftheabove

NSWPBL/SWPBSImplementation

• Statelevelcoordinationpersonnel=2(FTE)

• 2016-2017AdvisorsacrossfourDirectoratesteams=36

(FTE)

• Coachingnetworksofinternal/external:non-dedicated

personnel

19

Contextualmatters:TechnicalAssistance

• Availabilityofinstruments(SET,BOQ,BAT,ISSET,TFI,SAS,TIC,IPI)

• Variationsbetweenstates(training,technicalassistance,fidelityinstrumentuse)

• Variationswithinstates(training,technicalassistance,fidelityinstrumentuse)

• Variationwithfidelityinstrumentselectionandusage(knowledge,timing,selectionofinstruments)

FidelityInstrumentUsabilityRationale

1.Startingpoint:

• Advisorteamsplayacriticalrolewithevaluationpractices

• Self-assessmentpracticesareencouraged

• Instrumentsevaluatefidelityindifferentways

2.Problems:

• Selectionprocessforinstrumentsisnotdefined

• Potentialfornon-strategicuseofinstruments,particularlyinsettingsover2-3+yearsofimplementation

• Comparativeknowledgeofinstrumentrangeispossiblylimited

• Traininginuseofinstrumentsispotentiallyinadequate

3.Goalofmakingdataaccurate,notinstrumentadvocacy

Expectations

• Basicquestion:“Whatusabilityfactorsinfluencethe

selectionofafidelityinstrumentforusewithaschool”?• Usabilitypreferencesmorelikelytobebasedonfamiliarityof

aninstrument.

• Usabilitypreferencesmaytranslateintoconservative

practices

• Knowledgeofmulti-levelinstrumentsislikelytobelimited.

22

Instrumentguide(sample)

Purpose

Usage

Sources

School-WideEvaluation

Tool

BenchmarksofQuality TieredFidelity

Inventory(Tier1)

Formative:Universals

• Animprovementmeasurethatsupportstheschooltodeterminethedegreeandqualityofuniversalsupports.

• Contributetoamonitoringprocesstoannuallyreviewachievement.

Formative:Universals

• Animprovementmeasurethatsupportstheschooltodeterminethedegreeandqualityofuniversalsupports.

• Contributetoamonitoringprocesstoannuallyreviewachievement.

• Supportsiterativeself- evaluationandreviewpractice

Formative:Continuum• Aninitialassessmentto

determineiftheyareusing,orneedPBL

• AguideforimplementationofUniversal,TargetedandIntensivesupports

• Anindexofsustainedimplementationateachofthethreetiers.

• Supportsiterativeself-evaluationandreviewpractice

Atleastevery12months Atleastevery12months Multipletimesduringayear

• ExternalInterviews,• External

Observations• ExternalDocument

review• Datasourcesclearly

alignedtoevaluationquestions

• Structuredinternalrating• Structuredexternalreview

(discrepancyprocesswithrubric)

• ExternalObservations(Walk-throughtool)

• Externalinterviews• StructuredInternal

rating• Evaluationsub-

scalesclearlyidentifydatasources

InstrumentGuide(sample)

24

Productreview

Metric

Key

Publications

School-WideEvaluationTool(SET) BenchmarksofQuality TieredFidelityInventory(Tier1)

Formallyreviewed:

• Schoolbehaviourmanagementdocument

• SchoolStrategicDirections• PBLAnnualActionPlanfor

meetingschool-widebehavioursupportgoals

• Socialskillsinstructionalmaterials/implementationtimeline

• Behaviouralincidentsummariesorreports(e.g.,officereferrals,suspensions,expulsions)

• Disciplinereferralform(s)

Referencedinrubric:

• Teamminutes• Missionstatement• ActionPlanacrossthe10subscales• Disciplineproceduredocument,

includingminorvsmajordefinitions

• Scheduleoflessondelivery• Disciplinedatabase• Examplesofdisciplinesummaries• Rewardsystemguideline

Formallyreviewed:• Tier1handbook• Teammeetingroledescriptions• Schoolwebsite• Staffhandbook• Datadecisionrules• Professionaldevelopmentcalendar• Teamminutes• Surveydata• Teamactionplan• Fidelitytools• Studentoutcomesummaries• Disciplinepolicies&flowcharts• Lessonplans• Schoolnewsletters• Annualschoolreports• Staff,student,schoolcommunitysurveys

0-2subscalecode&scoringguideAscoreof≥80%isthecurrentcriterion

WeightedcodeswithadetailedscoringrubricAscoreof≥80%isthecurrentcriterion

0-2subscalecode&scoringguideAscoreof≥70%isthecurrentcriterion.

Sugai,G.,Lewis-Palmer,T.L.,Todd,A.W.,&Horner,R.H.(2001).School-wideEvaluationTool(SET).Eugene,OR:EducationalandCommunitySupports.Availablefromhttp://www.pbis.org

Cohen,R.,Kincaid,D.,&Childs,K.E.(2007).Measuringschool-widepositivebehavioursupportimplementation:Developmentandvalidationofthebenchmarksofquality.JournalofPositiveBehaviorInterventions,9,203-213.

Mcintosh,K.,Massar,M.M.,Algozzine,R.F.,George,H.P.,Horner,R.H.,Lewis,T.J.,Swain-Bradway,J.(2017)TechnicalAdequacyofSWPBISTieredFidelityInventory.JournalofPositiveBehaviourInterventions,19(1)

InstrumentGuide(Sample)

25

EvaluationContentAreas

Italics

indicates

uniqueconcept/

Wording

School-WideEvaluationTool(SET) BenchmarksofQuality TieredFidelityInventory(Tier1)

System for responding: A documented system formanaging/responding to “behavioural violations”; Definition of“office” versus classroom managed problems; A documented crisisplan for dangerous situations and displayed; Staff agreement onhandling a crisis.

Documented Discipline systems: Processdocumented (narrative/graphic); disciplineprocedures documented; Referral formincludes “useful” information; Problembehaviour defined; Major/Minor defined;“Suggested array” of responses.

Problem behaviour definitions: Clear definition ofbehaviour that “interferes with academic & socialsuccess”; clear documentation of executive vs teachermanaged problems: sharing with community.Discipline Policies: Policies & procedures describe &emphasise “proactive, instructive or restorative”approaches with consistent implementation.

Monitoring and decision-making: Documented referral process usingthe big 5; Documented process for summarising discipline data;Discipline summaries shared with staff at least every 4 months; Data isused by team to designing, implementing and revising school-widesystem.

Data management: System is used to collect &analyse “ODR” by big 5.Additional: attendance, academic.Usage: Analysed by team monthly; shared withstaff monthly.

Discipline data: “Instantaneous access” to graphedsummaries of big 5.Data-based decision making: team uses discipline &academic outcome data at least monthly- linked toaction plan.

Expectations defined: Staff agreement on 3-5 positively stated schoolrules; Displayed In 8-10 designated locations (including classrooms).

Expectations: 3-5 positively stated &displayed; Apply to staff & students;Developed & displayed in specific settings;Rules are linked to expectations; Staff involvedin development & review.

Behavioural expectations: 5 or fewer positivelystated; Defined by location for students & staff(matrix).

System for rewarding expectations: A documented system forrewarding.

Rewards: System enables consistent usage;Range of rewarding methods; Linked torules/expectations; Variation for effectiveness;Ratio standard; Students involved; Staffrewards.

Feedback&acknowledgment:documentedsystem-linkedtoschool-wideexpectations,usedacrosssettings&withinclassrooms.

PilotStudy

• DistributionofGoogle-basedsurveytoTAteamsin

Australianstates

• Requirementofminimumuseofaspecificinstrumentof

threetimes

• InvestigatedparticipantviewsonT1&T2+oversixteenfactors

• Investigatedparticipantratings• SharedcomparisontablefortheTFI,SET&BOQ

• Plannedfocus-groupfollow-upinsemester1,2017

Respondents (n=21)

Lessthan1

year 24%

1year,9%

2years,

33%

3years,5%

5years,5%

5+years,

24%

TAEXPERIENCE

76.19%

4.76%

19.05%

NSW QLD ACT

LOCATION

RespondentsarePBL/SWPBSadvisors/trainerswithtrainingandexperienceintheuseoffidelityinstruments

RespondentExperience(n=21)

28

57.14%

9.52%

33.33%

BOQ

0-3 3-6 7+

85.71%

9.52%4.76%

TFI

0-3 3-6 7+

14.29%

9.52%

76.19%

SET

0-3 3-6 7+

SelectionofTier2&3InstrumentISSET,TFI,BAT

29

BAT TFI ISSET0

2

4

6

8

10

12

14

Tier2/3Preferences

1 2 3 unsure

Preferredinstrumentforadministrationpurposes(Timerequirements,simplicity,resourceorganistion,demandonschool)

30

66.67%

20.00%

13.33%

Administration

SET TFI BOQ

Preferredinstrumentforaccuracypurposes(Validityovertime)

31

61.54%15.38%

23.08%

Accuracy

SET TFI BOQ

Preferredinstrumentforschoolutilitypurposes(Howwellthedataisusedbytheschool)

32

61.54%15.38%

23.08%

Accuracy

SET TFI BOQ

UsabilityfactorsInvestigated

SET,TFI,BOQ(Universal) overT1&T+2

Personalexperience Externalassessment properties

Expectationsoftheschool Graphicsummary

Preferenceforareasevaluated Schoolself-assessment

Preferenceforterminology Teamcapacitybuilding

Evaluationdepth(detailsofthedata) ConnectionwithTier2/3

Comprehensiveness(numberof features) Maintaining accuracyofmeasurement(validity)

Simplicityofuse Maintainingreliability(consistent,stable)

Usabilityforteamactionplanning Maintainingactionplanning

Administration time

AdministrationTime

34

Stronglyagree Agree Unsure Disagree Stronglydisagree

1 28.57% 33.33% 4.76% 23.81% 9.52%

2+ 28.57% 28.57% 9.52% 28.57% 4.76%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

TimetoAdminister

1 2+

PersonalExperiencewithAdministering

35


1 52.38% 33.33% 0.00% 14.29% 0.00%

2+ 33.33% 57.14% 0.00% 9.52% 0.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

PersonalExperiencewithadministering

1 2+

Terminology

36


1 23.81% 19.05% 9.52% 38.10% 9.52%

2+ 23.81% 19.05% 9.52% 38.10% 9.52%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

Preferenceforterminologyused

1 2+

Simplicity(Easytouse&understand)

37


1 42.86% 28.57% 9.52% 14.29% 4.76%

2+ 33.33% 38.10% 14.29% 9.52% 4.76%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Simplicity

1 2+

GraphicSummaryCapacity

38


1 52.38% 38.10% 4.76% 0.00% 4.76%

2+ 42.86% 42.86% 9.52% 0.00% 4.76%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Capacityforgraphicsummaries

1 2+

ExternalProperties(Interviews,Observations)

39


1 33.33% 47.62% 4.76% 14.29% 0.00%

2+ 28.57% 52.38% 4.76% 14.29% 0.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Externalassessmentproperties

1 2+

Tier2/3Connections

40


1 23.81% 33.33% 23.81% 19.05% 0.00%

2+ 33.33% 47.62% 19.05% 0.00% 0.00%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Connectionwithadvancedtiersevaluation

1 2+

EvaluationAreas

41


1 47.62% 42.86% 0.00% 4.76% 4.76%

2+ 47.62% 47.62% 0.00% 4.76% 0.00%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Preferenceforareasevaluated

1 2+

Depth(Detailofthedatacollected)

42


1 61.90% 33.33% 0.00% 4.76% 0.00%

2+ 71.43% 28.57% 0.00% 0.00% 0.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Depthofdatathattheevaluationyields

1 2+

Comprehensiveness(Numberofuniversalfeaturescovered)

43


1 42.86% 38.10% 0.00% 19.05% 0.00%

2+ 47.62% 38.10% 0.00% 14.29% 0.00%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%

Comprehensivenessoftheevaluation

1 2+

Self-AssessmentCapabilities

44


1 33.33% 47.62% 0.00% 14.29% 4.76%

2+ 33.33% 52.38% 4.76% 9.52% 0.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

Capacityforschoolself-assessment

1 2+

SchoolTeams

Stronglyagree

Agree Unsure DisagreeStronglydisagree

1 61.90% 33.33% 4.76% 0.00% 0.00%

2+ 66.67% 23.81% 9.52% 0.00% 0.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Contributiontoteamcapacitybuilding

1 2+

Stronglyagree

Agree Unsure DisagreeStronglydisagree

1 85.71% 14.29% 0.00% 0.00% 0.00%

2+ 85.71% 9.52% 4.76% 0.00% 0.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Usabilityforteamactionplanning

1 2+

ExpectationsoftheSchool

46


1 38.10% 38.10% 9.52% 9.52% 4.76%

2+ 38.10% 42.86% 9.52% 9.52% 0.00%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Expectationsoftheschool

1 2+

Maintenance+2:Reliability(Stability&Consistency)

47

71.43%

19.05%

4.76%

Reliabilityofinstrument


Maintenance(+2):Validity(Accuracyofmeasurement)

48

57.14%

28.57%

4.76%

4.76%

Validityofinstrument


Maintenance(+2):ActionPlanning

49

76.19%

19.05%

4.76%

0.00%

Maintainingactionplanningbytheschoolteam


CommentsonSchoolWideEvaluationTool(SET)

50

Strengths AreasforImprovement SpecificReasonsforUsage SpecificReasonsforNot

Using

Implications

• Quickandeasy

• Triangulationofdata

• Comparativedata

generated

• Evaluatescritical

featuresschool-wide

• Externalmeasure

• Randomcollection

• Usedforbaselinedata

• Includeresponsesfromparents

• Questionsshouldhaveequalvalue.Somequestionstooheavilyweightede.g.regionalsupport

• Clearerscoringcriteria• Timeconsuming–

wholeprocess• Contextualisationof

language• Timeconsuming

processtocollectdocumentation

• Baselinedata• Comparativedata• Annualtracking• Actionplanning• Easeofinterpretation• Externalmeasureof

implementation

• LinkstoSchoolExcellenceFramework

• Reviewcurrentstatus

• Datanotdetailedenough

• Onlymeasures

Tier1

• Moreconsistenttrainingtousethetoolclearlyandconsistentlytogathermoreaccuratedatae.g.questiontoT–“doyouteachbehaviour…”

BenchmarksofQuality(BOQ)

51


Using

Implications

• Comprehensiveassessment

• Teambasedreflectionprocess

• Supportsactionplanning• Clearinformationon

whatexemplarypracticelookslike

• Coversalluniversalareas• Rubricstosupport

decisiononscoring• Depthofanalysis• Externalmeasure

• BoQ walkthroughincludedaspartofscore

• Providesupporttodealwithdiscrepancies

• AccuracyofimplementationreliesondepthofknowledgebyPBLadministrator

• Abilitytouseelectronically

• Annualplanninguseful• Gaugeforreadiness

foradvancedtiers• Specificallylooksat

classrooms• Datausefulfor

planning• Comprehensive

• Notsuitableforschoolsimplementinglessthanayear

• DoesnotindicatehowwellateamimplementsTier2

• Schoolsneedanexternalcoachtoadminister

• Timetoadminister

• Provideopportunitiesfortraininganduseoftoolasmisconceptionsexist

TieredFidelityInventory

52


Using

Implications

• Feedbackisexplicit

• Self-assessment

• Addressesalltiersusingonetool

• Walkthrough

• Feedbackandactionplanningneeddevelopment

• Encouragesschoolstoself-monitor

• Scaffoldforteamself-reflection

• Noclassroomsystems

• Notvalidatedbyresearch

• Nottrainedtouseit

• Provideopportunitiesfortraininganduseoftoolasmisconceptionsexist

GeneralComments

53

BoQ SET TFI Other

• Providesmoredetaileddata

• Mostexperiencedtoolused

• Idealforschoolsinstart-upphase

• Lackofexperienceadministeringtool

• Feedbackandactionplanning

• Choiceoftooldependsoncontext

Implications&FutureWork

1.Focusgroupstoinvestigatespecificfactors:

• Administrationtime

• Externalversusinternalproperties• Self-assessmentversusexternalvalidation

• Language/terminologyoftheinstruments

• Integrationoftierswithinaninstrument

2.PlantrainingforAdvisorswithafocuson:

• Comparativefeaturesoftheinstruments

• Tier2/Tier3instrumentuse54

Contacts

[email protected]@PhillipSteed

[email protected]@carla_d_a

[email protected]@yasmin_e_ibby

selecting tier 1 fidelity instruments: perspectives from ... · pbis was developed initially in the...

Documents