statistical models for the analysis of single-case intervention data introduction to: regression...

35
Statistical Models for the Analysis of Single-Case Intervention Data Introduction to: Regression Models Multilevel Models

Upload: emory-ball

Post on 04-Jan-2016

239 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Statistical Models for the Analysis of Single-Case Intervention Data

Introduction to:

Regression Models

Multilevel Models

Page 2: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Why consider statistical models?

Can provide effect size estimates and confidence intervals for those estimates.

e.g., We are 95% confident the immediate shift in level for Jenny was an increase of between 11 and 14 minutes of time spent reading, or between 1.5 and 2.0 standard deviations

Page 3: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Regression

Imagine a scatter plot showing the relationship between motivation and achievement.

Motivation

Achievement

Page 4: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Regression allows us to summarize the relationship between the variables.

0

20

40

60

80

0 1 2 3 4 5 6

Motivation

Achievement

eXY 10

Page 5: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Often when we think of regression we think of each data point coming from a different individual, but all the observations could come from the same individual.

010203040506070

0 1 2 3 4 5

Time

Behavior

Page 6: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

010203040506070

0 1 2 3 4 5

Time

Behavior

eTimeY 10

Rise

Run

0

1Run

Rise

What is the rate of change for Jody?

Page 7: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

For single-case studies we expect a discontinuity

0

10

20

30

40

50

60

70

0 1 2 3 4 5 6

Baseline

Intervention

Page 8: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

0

10

20

30

40

50

60

70

0 1 2 3 4 5 6

What is the shift in level for Jody?

Effect =

0Phase 1Phase

ePhaseY 10

10

Page 9: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

What is the immediate shift in level and the shift in slope

for Jody?

0Phase 1Phase

eTimePhaseTimePhaseY *3210

1

0

0

10

20

30

40

50

60

70

80

90

100

Aslope2

-5 -4 -3 -2 -1 0 1 2 3 4 5

slope3

Page 10: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

0

10

20

30

40

50

60

70

80

90

100

Issues to Keep in Mind

You may have to choose at what point in time you should calculate the effect size

Page 11: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

0

10

20

30

40

50

60

70

80

90

100

You may want to standardize the effect size

If so, what SD should be use?

1

SDES 1

Page 12: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

There needs to be a match between the trajectory specified in the model and what is seen in the data

Effect = b1?

ijijjjij ePhaseY 10

This seems incorrect

Page 13: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

What is seen may require specification of a complex growth trajectory

Do you think specification tends to be easier when there are more or less observations in a phase?

Page 14: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Correct model specification requires more than just correctly specifying the growth trajectory

iii ePhaseY 10

Should you assume the errors (ei):are independent?have common variance? are normally distributed?

Page 15: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Cindy

Lucy

George

John

Imagine we have multiple cases

ijijjjij ePhaseY 10

0

10

20

30

40

50

60

70

0 1 2 3 4 5 6 0 1 2 3 4 5 60

102030405060708090

0 1 2 3 4 5 60

102030405060708090

0 1 2 3 4 5 60

102030405060708090

Page 16: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

A separate regression could be obtained for each case

ePhaseYGeorge 10

ePhaseYLucy 10

ePhaseYCindy 10

ePhaseYJohn 10

Or a multilevel analysis could be run

Page 17: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Multilevel Model

Multilevel models allow us to answer additional questions:

• What is the average treatment effect?• Does the size of the effect vary across

participants?• What factors relate to effect size?

ijijjjij ePhaseY 10

jj r0000

jj r1101

Page 18: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

What is the average effect for the participants?

Average Effect = γ10

ijijjjij ePhaseY 10

jj r0000

jj r1101

Average Baseline Level = γ00

Page 19: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Does the size of the effect vary across participants?

?)( 1 jrVAR

ijijjjij ePhaseY 10

jj r0000

jj r1101

Page 20: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

What factors relate to effect size?

ADD

Non-ADD

ijijjjij ePhaseY 10

jjj rADD 001000

jjj rADD 111101

Page 21: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Issues to Keep in Mind

There still needs to be a match between the trajectory specified in the model and what is seen in the data

Effect = b1?

ijijjjij ePhaseY 10

This seems incorrect

Page 22: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Correct model specification requires assumptions about multiple error terms

ijijjjij ePhaseY 10

jj r0000

jj r1101

Should you assume the errors (eij, r0j, r1j) are independent? Normally distributed?

Page 23: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

If one standardizes the effect size, what SD should be used for standardization

ijijjjij ePhaseY 10

jj r0000

jj r1101

Within case variance? Between case variance?

Page 24: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Imagine we have multiple studies

24

ijkijkjkjkijk eDY 10 ),0(~ 2eijk Ne

),0(~1

0

ujk

jkN

u

u

jkkjk u0000 jkkjk u1101

kk v0000000

kk v1010010 ),0(~10

00v

k

k Nv

v

Page 25: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Multilevel models were developed for large sample size conditions, but single-case applications tend to have a very small number of cases.

Given small sample sizes the variances (e.g. variance in the treatment effect across participants) will generally be more poorly estimated than the averages (e.g. the average treatment effect).

Page 26: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Example Analysis

Summarize results from 5 studies that examined the effect of intervention on autistic children’s speech

DV: Percent intervals with child speech

IV: Intervention based on increased parent verbalizations

Design: Multiple baseline across participants

Page 27: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Laski, K. E., Charlop, M. H., & Schreibman, L. (1988). Training parents to use the natural language paradigm to increase their autistic children’s speech. Journal of Applied Behavior Analysis, 21, 391-400.

Page 28: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

One child from Laski et al.

28

iii eDY 10

21.58ˆ0

95.241̂

Page 29: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

2-Level Model for Laski et al.

29

ijijjjij eDY 10

jj u1101 jj u0000

Parameter Estimated Estimate

SE p

Fixed Effects

Average Baseline Level (θ00)

34.17 8.16 .0030

Average Treatment Effect (θ10)

33.57 5.00 .0001

Variance Components

Variance in Baseline Level ( )

577.8 299.1 .0267

Variance in Treatment Effects ( )

185.4 110.5 .0468

Covariance u0 & u1 ( ) -294.1 110.5 .0838

Variance Within Person ( )

147.3 18.7 <.0001

2

0u

2

1u

01u2e

Page 30: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Software Code: SAS, R

30

SAS:proc mixed covtest;class Case;model Y= D / solution ddfm=sat;random intercept D / sub=Case type=un;

R:twolevel <- lmer(Y ~ D + (1 + D | Case), data2)summary(twolevel)

Page 31: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

3-Level Model for All Five Studies

31

ijkijkjkjkijk eDY 10 ),0(~ 2eijk Ne

),0(~1

0

ujk

jkN

u

u

jkkjk u0000 jkkjk u1101

kk v0000000

kk v1010010 ),0(~10

00v

k

k Nv

v

Page 32: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

3-Level Model Results

32

Parameter Estimated Estimate

SE p

Fixed Effects

Average Baseline Level (γ000) 18.71 6.31 .0345

Average Treatment Effect (γ100) 31.72 9.37 .0309

Variance Components

Between Study Variance in Baseline Level ( )

128.2 128.3 .1588

Between Study Variance in Treatment Effects ( )

377.2 320.7 .1198

Covariance v0 & v1 ( ) 165.5 169.8 .3296

Within Study Variance in Baseline Level ( )

316.9 103.6 .0011

Within Study Variance in Treatment Effects ( )

222.2 81.8 .0033

Covariance u0 & u1 ( ) -47.8 70.1 .4956

Variance Within Person ( ) 328.7 15.7 <.0001

2

0u

2

1u

01u2e

2

0v

2

1v

01v

Page 33: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Software Code: SAS, R

33

SAS:proc mixed covtest;class Study Case;model Y= D / solution ddfm=sat;random intercept D / sub=Study type=un;random intercept D / sub=Case(Study) type=un;

R:threelevel <- lmer(Y ~ D + (1 + D | Study:Case) + (1 + D | Study), data3)summary(threelevel)

Page 34: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Statistical models (regression and multilevel) provide a flexible approach for estimating treatment effects from single-case data, but care must be taken to ensure the model being used is consistent with the data being analyzed.

Conclusion

Page 35: Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

Applications and IllustrationsBaek, E., & Ferron, J. M. (2013). Multilevel models for multiple-baseline data: Modeling across

participant variation in autocorrelation and residual variance. Behavior Research Methods, 45, 65-74.

Baek, E. K., Moeyaert, M., Petit-Bois, M., Beretvas, S. N., Van den Noortgate, W., & Ferron, J. M. (2014). The use of multilevel analysis for integrating single-case experimental design results within a study and across studies. Neuropsychological Rehabilitation, 24, 590-606.

Ferron, J. M., Moeyaert, M., Van den Noortgate, W., & Beretvas, S. N. (in press). Estimating casual effects from multiple-baseline studies: Implications for design and analysis. Psychological Methods.

Moeyaert, M., Ferron, J., Beretvas, S. N., & Van den Noortgate, W. (2014). From a single-level analysis to a multilevel analysis of single-case experimental designs. Journal of School Psychology, 52, 191-211.

Moeyaert, M., Ugille, M., Ferron, J., Onghena, P., Heyvaert, M., Beretvas, S. N., & Van den Noortgate, W. (in press). Estimating intervention effects across different types of single-subject experimental designs: Empirical illustration. School Psychology Quarterly.

Rindskopf, D., & Ferron, J. (2014). Using multilevel models to analyze single-case design data. In T. R. Kratochwill & J. R. Levin (Eds.), Single-Case Intervention Research: Statistical and Methodological Advances (pp. 221-246). American Psychological Association.

Shadish, W.R., Kyse, E.N., & Rindskopf, D.M. (2013). Analyzing data from single-case designs using multilevel models: new applications and some agenda items for future research. Psychological Methods, 18, 385-405.

Van den Noortgate, W., & Onghena, P. (2003). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325-346.

Van den Noortgate, W., Onghena, P. (2007). The aggregation of single-case results using hierarchical linear models. The Behavior Analyst Today, 8(2), 196-209.

Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental designs. Evidence-Based Communication Assessment and Intervention, 2, 142-151.