A Randomized Controlled Trial of Compass for Transition Youth with ASD

Lisa Ruble, PhD &

Medina Adams MS

University of Kentucky

John McGrew, PHD

IUPUI

NASP 2018

The problem

The seamless transition from school to post-school

activities is failing.

Employment rates – 4-12% (Tayler & Seltzer, 2011).

Lowest rate of employment and highest rate of no

activities compared to other adults with disabilities

(Shattuck, et al., 2012).

Higher rate of noncompetitive future jobs (39%) compared to other

students with disabilities SLI (2.3%); LD (1%); ID (20%)

(Cameto, Levine, & Wagner, 2004).

Lowest percentage of friendships; second lowest for

living independently, and more likely to live with

parents even after controlling for level of functioning

(Anderson, et al., 2013).

Need Better transition planning

Congressional report concluded only 2% of ASD research focused on transition and adult issues (US Dept of Health and Human Services, 2017).

Autism Interagency Coordinating Committee called for empirically-supported approaches for improving transition planning and outcomes for students with ASD (2012).

No empirical studies of a transition planning intervention (Wehman, et al. 2014).

Collaborative Model for

Promoting Competence and

Success Tested in 2 RCTs for young children

Both showed improved IEP goal

attainment outcomes (1.4; 1.5 effect

size)

The second included a third group that

compared web-based teacher

coaching and results in a 1.1 effect size

(not different from the face-to-face

group)

Used an implementation science

framework to adapt COMPASS for

transition age youth and test it in a

small RCT.

Collaborative Model for

Promoting Competence and

Success Builds from clinical decision making

based on collaboration, mutual

respect, and support

• Authentic assessment of

parent/teacher priorities and

concerns

• Teacher and Family/student-

centered approach

• Support for parent-teacher alliance

• Links to important quality of life

outcomes

Outcomes based monitoring

Performance feedback & problem

solving

• Teaching plan changes if child is

not responding

• Ongoing and proactive

IEP outcomes

Interaction

Teacher

Child

Evidence Based Practice or Best

Available Research Evidence

Student / Family

Characteristics, Needs, Values, & Preferences

Teacher Expertise and

Resources

Decision Making

McGrew, J., Ruble, L., & Smith, I. (2016). Autism Spectrum Disorder and Evidence‐Based Practice in

Psychology. Clinical Psychology: Science and Practice, 23(3), 239-255.

Consultation as Intervention

Consultation has a

“multiplier effect”

Consultation as Implementation Strategy

Implementation Practice

Trainer/ consultant

COMPASS

Intervention Practice

Teacher/ clinician/ caregiver

Instructional Quality

Practice Outcome

Child

Child Educational Outcomes

Dunst, C. J., Trivette, C. M., & Raab, M. (2013). An implementation science framework for conceptualizing

and operationalizing fidelity in early childhood intervention studies. Journal of Early Intervention, 35, 85-101

COMPASS for Young Children

Initial Consultation

a. Review evidence based programs and COMPASS philosophy

b. Identify personal / environmental challenges & supports

c. Identify goals/ make measureable

d. Develop teaching plans

Coaching Sessions

a. Review videotape of implementation of teaching plans

b. Record progress

c. Adapt teaching plans if necessary

2.5 – 3 hrs

1 hr X 4

< than 10 hrs

Update IEP

Research Questions

(a) Can we adapt COMPASS for transition planning and implementation to improve IEP goal attainment outcomes for students with ASD?

(b) What degree of fidelity did consultants achieve in delivering the modified version of COMPASS?

(c) How well did teachers adhere to evidence-based instruction in COMPASS?

(d) How did teachers, parents, and students perceive the consultation? special attention to teacher acceptability, usability, feasibility, burden

(e) How did student self-determination, participation in jobs, and transition planning quality compare?

What vocational activities, 3 months following school?

COMPASS Adaptation – Mixed Methods

Focus groups (n = 10)

40 participants (individuals with ASD; parents; policymakers; school providers; adult service providers)

General questions:

1. What are the critical elements of good IEP transition planning and what are the barriers?

2. What are the critical elements of a good transition intervention and the barriers?

3. How will we know if transition planning has been successful (what intervention outcomes should we expect)?

Snell-Rood, C., Ruble, L., Kleinert, H., McGrew, J., Adams, M., Rodgers, A., Yu, Y. Wong, W. &

Odom, J. (2017). Stakeholder Perspectives on Transition Planning, Implementation, and

Outcomes for Students with Autism Spectrum Disorder. Manuscript revised and resubmitted.

Themes

Adaptations

1. Altered Assessment

2. Altered Handouts

3. Altered Consultation

4. Redefined Key Players

Snell-Rood, C., Ruble, L., Kleinert, H., McGrew, J., Adams, M., Rodgers, A., Yu, Y. Wong, W. & Odom,

J. (2017). Stakeholder Perspectives on Transition Planning, Implementation, and Outcomes for

Students with Autism Spectrum Disorder. Manuscript revised and resubmitted.

Adapting COMPASS for

Transition Planning and Implementation

Consolidated Framework for Implementation Research (CFIR)

(Damschroder, et al., 2009)

Characteristics of Intervention*

Settings:

Outer & Inner

Characteristics of Individuals*

Implementation Process*

Characteristics of the intervention

Theme Description Implication for COMPASS Product

Navigator/

Coordinator

A person (or layperson) who

navigates and educates the

professional, parent, or individual with

ASD through different services and

options.

Facilitate navigation through

expanded consultant role;

provide information on adult

services

Altered handouts:

Top 10 List

Transition Process Resource

Guide

Ruble, L., Adams, M., McGrew, J., Snell-Rood, C., & Kleinert, H. (2017). Adapting COMPASS for Youth

with ASD to Improve Post-School Outcomes Using the Consolidated Framework for Implementation

Science. Manuscript revised and resubmitted.

COMPASS for Transition Youth

Initial Consultation

a. Review evidence based programs and COMPASS philosophy

b. Identify personal / environmental challenges & supports

c. Identify goals/ make measureable

d. Develop teaching plans

Coaching Sessions

a. Review videotape of implementation of teaching plans

b. Record progress

c. Adapt teaching plans if necessary

Update IEP

Discuss post-secondary goals

Complete adapted COMPASS profile

Assessed need for information on services

Provided information

IEP goals linked to post-secondary goals

Videotape, work samples, data

Student involvement as much as possible

Review Parent Progress toward Post-Secondary Goals

Parent goals for post-secondary outcomes

RCT Consort Diagram

Participant

Description

•Certified teachers and 3 were male; 85% MA; 10% BA; 5% Doctorate

•Case manager (oversaw IEP)

•Number of years teaching = 12.3 years

•Number of students with autism taught = 35

Teachers

•Had IEP with Autism as eligibility category

•Diagnosis confirmed by ADOS

•Mean age = 18.2 years; 90% male; 70% White; 15% Black; 10% multi-racial; 5% Asian

•Number of experimental group students who attended consultation: 7

•Classroom Placement

•>80% in General Ed: 8

•40-80% in General Ed: 4

•<40% in General Ed: 8

Students

Student Comparisons –

Baseline Assessment

Comparison

IQ

Adaptive behavior

BASC-2

CARS

Child age

Family income

Teacher experience

Group

Assignment

Control COMPASS

Any

Difference

?

Measures

•Student IEP outcome

•Parent, student, teacher satisfaction

•Teacher report of acceptability, feasibility, usability, burden

•Secondary outcomes – self determination, transition planning quality, community participation jobs

•3-month report of vocational status

Outcome

•Consultant fidelity

•Teacher adherenceProcess

Process Measures: Fidelity &

Adherence

Consultant Fidelity

Reliable 35 item

yes/no checklist

completed by parents/teachers & independent rater

Reliable 17-item

yes/no checklist

completed by teachers and independent rater

Teacher Adherence

Reliable 5-point Likert-

type scale

1 2 3 4 5

0-20% 21-40% 41-60% 61-80% 80-100%

Primary Student Outcome:

PET-GAS

-2 -1 0 +1 +2

Present

levels

Progress Goal

met

Goal

exceede

d

Goal

very

much

exceede

d

Student outcome measured using psychometric-equivalence tested

goal attainment scaling (Ruble, McGrew, & Toland 2013).

Reliability of .94 for baseline and .86 for final

PET comes from assessment of three features of GAS

Goal measurability

Goal intervention equidistance

Goal difficulty

Transition Planning Quality

30 item parent and self report rating scale

(alpha - .98)

Self Determination12-item parent and self report (alpha = .96)

Community

Participation

Parent and self

report

Satisfaction Initial Consultation – 14 items (alpha = .68 - .97)

1 “strongly disagree” to 4 “strongly agree”

I felt involved during the consultation and able to express my vies

The Consultant's’ communication skills were effective

The consultant listened to what I had to say

The consultant was knowledgeable bout ASD

Coaching Sessions – Session Rating Scale (Duncan & Miller, 2007) – measured using 10cm scale

Teacher Acceptability & Usability of COMPASS

•1 ‘not at all likely’ to 4 ‘very likely’

•How likely would you be to use COMPASS for all your students?

•How likely would you be to recommend use of COMPASS for all teachers of special education students?

Acceptability – 5 items (alpha .99)

•1’strongly disagree’ to 4 ‘strongly agree’

•The COMPASS assessment forms used for the initial planning were very useful

•Teaching plans were understandable

Usability – 11 items (alpha .94)

Teacher Feasibility and

Burden of COMPASS

Feasibility – 5 items (alpha .39)

1 ‘not at all hard’ to 4 ‘very hard’

How hard was it to make time for the initial consultation?

How hard was it to implement the teaching plans?

Burden – 4 items (alpha .33)

1 ‘not at all hard’ to 4 ‘very hard’

How burdensome in terms of time and effort was the initial consultation, the coaching session, data collection activities, and videotaping?

Due to low alpha feasibility and burden was measured at the item level

Results

No Differences Between Groups

Construct Measure

Severity CARS (HF)

Adaptive Skills Vineland (TR)

IQ KBIT-2 IQ

Age Child age (years)

Number of Services Services received

Hours of Services Hours of services

Teacher Years Teaching Years teaching

Number Students Taught Students taught

SES Family income

Does COMPASS Improve

Transition IEP Goal Attainment

Outcomes?**

Control Treatment

(n = 9) (n = 11)

Variable M SD M SD t(df) p d*

GAS change

score

0.82 0.67 2.61 0.81 5.30(18) < .001 2.41

GAS final score 1.94 0.77 3.61 0.81 4.69(18) < .001 2.11

Ruble, L., McGrew, J., Toland, M., Dalrymple,N., Adams, M., & Snell-Rood, C. (2017). Randomized Control

Trial of COMPASS for Transition Outcomes of Youth with ASD. Manuscript submitted.

**determined by observer unaware of group assignment

What is the consultant fidelity of the modified version of COMPASS?

• 90-94% of components implemented based on parent and teacher judgements and review of independent researcher.

• No differences across informants.

Consultation Fidelity

• 98-100%% of elements implemented based on teacher and independent research ratings.

Coaching Fidelity

How well did teachers adhere to

evidence-based instruction in COMPASS?

Coaching session adherence:

1 2 3 4 5

0-20% 21-40% 41-60% 61-80% 80-100%

0

0.5

1

1.5

2

2.5

3

3.5

4

1 2 3 4

Mean Adherence by coaching session

Adherence

Transition Planning Quality

Time Group t-test

(2-tailed)

Effect size

Comparison

Total (SD)

COMPASS

Total (SD)

Baseline 55.2 (49) 46.1 (45) -.44, p = .67 0.19

Final 37.9 (47) 77.4 (45) 1.9, p = .07 0.86

Other Student Outcomes

Self Determination (parent report)

Mean 30.4 for control; 33.7 for COMPASS, t = -.56, p = .58,

effect size = .33

Other Student Outcomes

Community Participation

No significant differences across areas

For jobs, possible trend for frequency of participation:

Mean 1.7 control;

3.1 for COMPASS,

t = -1.6, p = .14

Effect size = 0.80

Parent and Teacher

Satisfaction

Initial Consultation

Parent - 3.8 (.18)

Teacher - 3.8 (.26)

Student - 3.2 (.8) n=2

Coaching Sessions

2 3 4

Teacher (n = 10) 9.1 9.8 9.6

Parent (n=4-6) 9.8 9.9 9.9

Student (n=6-8) 6.6 7.9 8.3

Teacher Acceptability, Usability,

Feasibility, and Burden of

COMPASS

Construct Mean (SD)

Acceptability 3.0 (.84)

Usability 3.3 (.47)

Range

Feasibility* 1.3 - 1.9

Burden* 1.4 -1.7

1 ‘not at all likely’ to 4 ‘very likely’

1 ‘strongly disagree’ to 4 ‘strongly agree’

1 ‘not at all hard’ to 4 ‘very hard’

3-month

outcomes

Vocational Index (Taylor &

Seltzer, 2013)

t = -.80, p = .44,

effect size = 0.41

Control 4.0

COMPASS 5.6

Conclusions

The adapted version of COMPASS for transition age students was effective for improving IEP outcomes.

The consultants could implement COMPASS with high fidelity.

Teacher adherence to teaching plans increased over time.

Overall, teachers, parents, and students were satisfied.

Teachers reported that COMPASS was acceptable and usable and that the activities were not overly burdensome and feasible to implement.

Transition planning quality improved for COMPASS.

Community participation in jobs and self-determination, while having higher mean scores, were not significantly different between groups.

Other Key

Observations

1. Teacher of record (case manager) may not actually teach the student (or know the student)

2. Student involvement as much as possible

They are key when attending general education for most classes

Students in general education rarely had time with the special education teacher

3. Parent involvement critical

4. Personalized teaching plans essential

5. Clearly articulated plans for post-school outcomes

Barrier of getting vocational rehabilitation at the table

Parent / caregivers are necessary in achievement of post-secondary goals

6. Teacher and Student: Performance-based feedback

Measurable goals

Clear teaching plans

7. Teacher and Student: Outcomes-based feedback

Data collection (video, work samples, self-monitoring data)

Goal attainment scaling

Acknowledgements:

Co-investigators:

John McGrew, IUPUI

Nancy Dalrymple

Claire Snell-Rood, UC

Harold Kleinert, UK

Research Team

Medina Adams, UK

Jorgina Arballo, UK

Jordan Findley, UK

Teri Krakovich, IUPUI

Abbey Love, UK

Alexis Rodgers, UK

Venus Wong, UK

Yue Yu, IUPUI

Parents and Caregivers

Students

School Administrators and Teachers

Funder: NIMH

R34MH104208

www.ukautism.org

Thank you!