Running head: BEHAVIORAL CODING 1

Behavioral Coding of Siblings of Children with Autism

Melanie C. Hankins

University of Florida

BEHAVIORAL CODING 2

Introduction

The purpose of this pilot study was to determine if the siblings of children with Autism

Spectrum Disorder (ASD) could learn and appropriately utilize interventions that might improve

the social behavior of the children with ASD. The ultimate goal of teaching these interventions

to the siblings is to cause improvement of social interactions in the still-developing child with

Autism Spectrum Disorder. Ten minute video-recordings of the interactions between the sibling

and the child with ASD were made before and after the sibling was taught specific interventions

that were coded according to the frequency in which the specific intervention occurred. This

pilot study, “In-Home Training for Siblings of Children with Autism”, was based on the results

of a larger, sentinel study, “In-Home Training for Fathers of Children with Autism: A Follow up

Study and Evaluation of Four Individual Training Components”, which incorporated similar

father-child in-home training. The principle investigator of this study was Dr. Jennifer Elder

(See Appendix A). If the full study is approved by the NIH, it should receive $2 million in

funding and would last a total of five years (Elder, 2010).

Autism Spectrum Disorder (ASD) is a developmental disorder that contains three general

domains of abnormal behaviors as explained in the Diagnostic and Statistical Manual of Mental

Disorders, 5th Edition (DSM V) criteria: social interaction, communication, and

restricted/repetitive behaviors or interests. ASD includes: Asperger’s Syndrome, Autism,

Pervasive Developmental Disorder Not Otherwise Specified (PDDNOS), and Childhood

Disintegrative Disorder (American Psychiatric Association, 2013). Autism affects 1 in 50

children, and its prevalence appears to have been increasing, although the cause of this increase

remains unclear. School-aged boys are more commonly affected than girls, with boys being four

times more likely to have ASD (Blumburg et al., 2013). Helping children with ASD requires a

BEHAVIORAL CODING 3

high level of interdisciplinary care, particularly among specialties associated with pediatrics. It

is also important to find ways to improve the quality of life of affected families, as autism affects

entire families, not just the autistic children. Providing family members with education,

encouragement, and behavioral skills to allow them to adequately manage care is especially

important. According to Autism Speaks (2010), early detection of any abnormalities in

development is critical to improve such children’s quality of life. These families often must rely

on the skills of a nurse who is educated and informed about Autism Spectrum Disorder.

Description

This study utilized specific interventions taught to the siblings of the autistic children. Some

of these interventions consisted of behaviors that sometimes occurred spontaneously, even

without specific training. During each session, the children were first recorded during a two

minute “settling-in” period, and those two minutes were not coded for the interventions. Prior to

the training of each specific intervention to the siblings, a baseline video was recorded of the

sibling and autistic child’s interaction during play, during which the video was coded by

counting the frequency of each intervention. The siblings were then taught one of the

interventions. After the sibling had received the training for each intervention, a ten minute

video was recorded that was coded for the interventions. The siblings carried out the

interventions they were taught during the recordings, and the interventions were coded, even if

these interventions observed had not yet been taught to the sibling of the autistic child. The

interventions were introduced one at a time and are described below:

Imitation/Animation (IA): A movement cycle that begins within 5 seconds of an

Autistic Disorder (ASD) child initiation (e.g. child utters a vocalization, jumps up and

BEHAVIORAL CODING 4

down), wherein the sibling imitates the child’s behaviors in an animated manner (e.g.

with exaggerated, lively movement).

Following the Child’s Lead (FL): A movement cycle that begins when the sibling

joins the ASD child and plays the game or plays with the toy the ASD child has

selected (e.g. child picks up a book and starts to look at it, sibling gets next to the

child, looks too and comments on the pictures; ASD child stacks one block on

another and the sibling joins in and stacks a third block on top) within 5 seconds of

the ASD child’s behavior and ends with the completion of the sibling’s action.

Commenting on the Child (C): A movement cycle that begins with an ASD child’s

action or vocalization and ends with a related and immediate sibling comment (e.g.

“So, you found a red car” rather than “Is that car red?”).

Methodology

These interventions were coded by students with conferred inter-rater agreement. According

to the Center for Educator Compensation Reform (2011), inter-rater agreement is the degree to

which two or more raters give the same rating to the same specific observable situation using the

same rating scale. Before coding the videos, the raters were trained by a research assistant with

the aid of the principal investigator. After coding 27 videos, the conferred inter-rater agreement

was calculated, and the definitions of the training components were further refined. After the

definitions were refined, the coders calibrated using eight videos that were not previously coded,

the two minute settling-in videos. There was then independent coding of eight randomly-

selected established videos, two from each family. Independent inter-rater agreement was then

calculated and analyzed. The inter-rater agreement was calculated by the number of agreements,

BEHAVIORAL CODING 5

divided by the number of agreements plus the number of disagreements (Fleiss, Levin, Paik,

2003).

The hypothesis of this study was that after teaching the interventions to the sibling of the

autistic child, there would be an improvement in social interaction in the still-developing

children with autism spectrum disorder. The sample contained four ethnically diverse autistic

spectrum disorder child-sibling pairs.

After the conferred inter-rater agreement and independent inter-rater agreement were

calculated, the results were analyzed. The results were different than expected. Our conferred

inter-rater agreement had a high level of reliability, whereas our independent inter-rater

agreement had a very lower level of reliability. Additional work is needed to improve inter-rater

agreement prior to conducting the larger study and further suggestions to increase the quality of

the research gathered are included below.

Findings

The calculated conferred inter-rater agreement was 98.3 for Family A, 93.1 for Family B,

97 for Family C, and 96 for Family D overall. Imitation/Animation was 95 for Family A, 96.4

for Family B, 95.4 for Family C, and 95.6 for Family D. Following the Child’s Lead had an

agreement of 100 for Family A, 90.1 for Family B, 95.7 for Family C, and 94.5 for Family D.

Commenting on the Child had an agreement of 100 for Family A, 92.8 for Family B, 100 for

Family C, and 97.9 for Family D (See Figure 1).

BEHAVIORAL CODING 6

Figure 1. Conferred Inter-rater Agreement

The calculated independent inter-rater agreement was significantly lower, Family A was

74.6, Family B was 72.2, Family C was 56.8, and Family D was 39.7 overall.

Imitation/Animation had an agreement of 67.4 for Family A, 84 for Family B, 91.5 for Family C,

and 79.5 for Family D. Following the Child’s Lead had an agreement of 70.8 for Family A, 70.2

for Family B, 78.8 for Family C, and 27.2 for Family D. Commenting on the Child had an

agreement of 85.5 for Family A, 62.5 for Family B, 0 for Family C, and 12.5 for Family D (See

Figure 2).

Figure 2. Independent Inter-rater Agreement

BEHAVIORAL CODING 7

Figure 3. Inter-rater Agreement

Commenting on the Child had the lowest inter-rater agreement (See Figure 3). This

could be partially due to the fact that this intervention had the lowest frequency during the

videotaping sessions. This intervention appeared difficult for still-developing children to word

comments to meet this definition; however, according to the Child Development Institute (2013),

children aged 8-10 should be able to form complex sentences at the adult level, so it is unclear as

to why the siblings had difficulty with this.

The findings indicated that the siblings of the ASD children were able to learn and use

these interventions. There was an increase in the frequency of the interventions after they were

taught the interventions, and the frequency counts remained higher with each subsequent

intervention that was taught. Commenting on the Child had the lowest frequency counts out of

the three interventions, but showed an increase in frequency after the intervention was taught.

Suggestions

There are multiple suggestions that, if implemented in the larger study, could improve the

reliability and accuracy of the data gathering process, and also improve the quality of the

research. It would be useful to replicate the study with more subjects. This pilot study was very

BEHAVIORAL CODING 8

limited in the sample size, as there were only four sibling/autistic child pairs. There could also

be stricter inclusion criteria. One of the subjects was non-verbal, which could have altered the

results. Improving the quality of the video recording might be helpful in improving coding

accuracy. One of the challenges of coding the videos related to the quality of the videos. At

times it was difficult to understand what the subjects were saying, so it was unclear if the siblings

were commenting on the autistic child, or imitating the autistic child, potentially altering the

results. Another suggestion is to use the two minute “settling-in” videos as reinforcing/teaching

time. Often, the sibling may not have remembered the prior intervention that was taught, and

this could be utilized to further instruct the subjects. Additional reminder sessions and

encouraging daily practice with families could be very beneficial. As stated previously, the

subjects sometimes appeared to have difficulty utilizing the previously taught interventions. The

current definitions used in the coding leave considerable room for subjective interpretation by

the rater, even after the further improvements noted above. These definitions could be further

improved, which could positively impact the inter-rater agreement. More formal training of

coding with the principal investigator, incorporating several practice videos with feedback on

coding accuracy would be very helpful. More formal coding training could improve the

reliability of the study. Incorporating a short training video that clearly defined the interventions

with examples of Imitation/Animation, Following the Child’s Lead, and Commenting on the

Child that could be played before each coding session would remind the coders of the correct

techniques, which would benefit the study. Also, recalibrating if there is a long interval between

coding sessions, perhaps utilizing the “settling-in” two minute videos, would remind the coders

of the interventions and correct techniques to use while coding.

BEHAVIORAL CODING 9

Because Commenting had the lowest inter-rater agreement, it may be beneficial to drop this

intervention from the larger study or code for another element, such as coding for a general

response or praise. It may be beneficial to follow-up with families after the interventions and

video-recording are finished to analyze if the siblings are still utilizing the interventions. This

could be analyzed at three and six months. Social interaction could also be analyzed at these

times to discover if there was any improvement in social interaction of the autistic child. It

would also be beneficial to study the way in which the siblings were impacted by these

interventions by analyzing their quality of life and exploring whether or not they felt empowered

by utilizing these interventions. The refined definitions are included below:

I/A: Movement cycle that begins within 5 seconds of an ASD child’s initiation, (e.g.,

child utters a vocalization, jumps up and down, changes body position) where the

sibling imitates/matches the child’s behavior and/or vocalization in an animated

manner (e.g., with exaggerated affect, lively movement). Each movement and

vocalization count as an individual event.

FL: Movement cycle that begins when the sibling joins the ASD child and plays the

game or plays with the toy the ASD child has selected (e.g., child picks up a book and

starts to look at it, sibling gets next to the child, looks too and reads book; ASD child

stacks one block on another and sibling joins in and stacks a third block on top)

within 5 seconds of the ASD child’s behavior and ends with the completion of the

sibling’s action, and is not to be confused with imitating ASD child’s direct action.

Sibling goes along with ASD child’s initiation of an event and contributes to play.

Commenting: Movement cycle that begins with an ASD child’s action or vocalization

and ends with a related and immediate sibling comment (e.g., “So, you found a red

BEHAVIORAL CODING 10

car” rather than “Is that a red car?”). Sibling making a verbal remark directly related

to the ASD child’s behavior and/or actions. “W” questions (what, when, why, how,

where) are not included in commenting; additionally, asking “yes or no” questions are

also not included in commenting.

Conclusion

The raters viewed one of the in-home filming sessions with the preceptor, and were able to

see the research process from start to finish. One of the observers coded 27 videos of sibling-

autistic disorder child interaction. With the assistance of the other coder, they calculated

conferred inter-rater agreement and independent inter-rater agreement. The coders then

determined how to solidify data gathering and coding techniques in order to ensure standardized

methods for future replication and to prevent bias. They then refined the definitions of the

training components to be more applicable to school-aged children. Finally, they offered

suggestions for the future NIH-funded study.

In order to carry out the functions of the study, it was important that to have a good

background on ASD and knowledge about characteristics that are prevalent in this group. The

coders had to learn how to code data by observing behavior carefully and counting the

occurrences of specifically defined behaviors, and calculating conferred inter-rater agreement

and independent inter-rater agreement. Microsoft Excel was used to formulate graphs of the

collected data. Additionally, the coders attempted to remain unbiased during the data-gathering

process.

During the research process, there were some difficulties with the study itself. There were a

few times where it was difficult for the coders to meet with their preceptor due to their very busy

nursing school schedules. There were also a few lapses between coding sessions, as schedules

BEHAVIORAL CODING 11

did not always warrant having a set time to code every week. Also, the inter-rater accuracy in

the study was lower than expected; however, this provided an opportunity to develop invaluable

insight as to how to improve the study.

There was a great deal to learn during this study, from the acquisition of information about

Autism Spectrum Disorder, to how to conduct research. It was invaluable to see research being

conducted and learn how to reduce bias. Improving the data-gathering process during the coding

of the videos to increase accuracy of the information was important, as was the ability to work

effectively with a partner. The importance of family involvement in these types studies is clear,

not just the importance of subject participation. The families encouraged the subject to

participate and set the schedule for the video recording sessions.

In order to improve the quality of the research, several techniques were used to keep the

researchers focused on the study. The raters communicated frequently, attempting to understand

the needs of all persons involved, and displayed patience throughout the entire process. The

coders met frequently (at least every week) to collaborate on coding sessions, and there were

other meetings to communicate with the preceptor to discuss progress in the research and to

discuss other logistical issues. These meetings were very helpful because they ensured the

coders were moving at an appropriate pace, and the project remained on target. If there had been

any other adversity during this project, it would have been important to communicate these

issues and come up with a plan to address them.

Recommendations

The honors program requires a great deal of time, patience, and dedication, so it is important

to ensure enough time to finish the project. It is also important to start working on the honors

project as early as possible. Starting over the summer is helpful, as it made time management

BEHAVIORAL CODING 12

considerably easier during the fall and spring semesters. Also, it is very important to pick a topic

for the project that is interesting. It is much easier to do the project when you are passionate

about it. It is very important to meet with your preceptor regularly. The preceptor is a great

resource and will provide significant assistance and support during the project. Finally, it is

important for the student to be as objective as possible, it is easy to be biased in your research,

however, and this can significantly alter your results.

Summary

This study by Dr. Jennifer Elder, “In-Home Training for Siblings of Children with Autism”,

was a pilot study based on a sentinel study, “In-Home Training for Fathers of Children with

Autism: A Follow up Study and Evaluation of Four Individual Training Components” (Elder,

2010). The current study incorporated several training components that were taught to the

siblings of the autistic children. These siblings performed the interventions while being

videotaped during interactions with the autistic child. These videos were then coded for the

different interventions taught, Imitation/Animation, Following the Child’s Lead, and

Commenting on the Child. After the coding of the videos, the data collected was analyzed using

conferred inter-rater agreement, and later twenty randomly-selected videos were also coded

using independent inter-rater agreement. The results indicate that the children were able to learn

and utilize specifically taught interventions with their siblings with Autistic Spectrum Disorder.

After this data was collected, suggestions were made for further improvement of the study.

BEHAVIORAL CODING 13

References

American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental

Disorders (5th ed.) Washington DC: Author.

Autism Speaks. (2010). How is autism treated? Retrieved from:

http://www.autismspeaks.org/family-services/tool-kits/100-day-kit/treatments-therapies.

Blumburg, S.J., Bramlett, M. D. Kogan, M. D., Schieve, L.A., Jones, J. R., Lu, M.C. (2013).

Changes in prevalence of parent-reported autism spectrum disorder in school-aged US

children. Centers for Disease Control and Prevention, Retrieved from:

http://www.cdc.gov/nchs/data/nhsr/nhsr065.pdf

Center for Educator Compensation Reform. (2011) Research synthesis: D. measurement.

Retrieved from:

http://www.cecr.ed.gov/researchSyntheses/34008_CECR_RS_Inter_Rater_measurement

_508.pdf

Child Development Institute. (2013). Language development in children. Orange, CA. Retrieved

from: http://childdevelopmentinfo.com/child- development/language_development /

Elder, J.H., Donaldson, S.O., Kairalla, J., Valcante, G., Bendixen, R., Ferdis, R., Self, E.,

Walker, J., Palau, C., & Serrano, M. (2010). In-home training for fathers of children

with autism: A follow up study and evaluation of four individual training components.

Journal of Child and Family Studies, 20(3), 263-271.

Elder, J.H., Valcante, G., Yarandi, H., White, D., & Elder, T.H. (2005). Evaluating in-home

training for fathers of children with autism using single-subject experimentation and

group analysis methods. Nursing Research, 54(1), 22-32.

Fleiss, J.L., Levin, B., Paik, M.C. (2003). The measurement of inter-rater agreement. Statistical

BEHAVIORAL CODING 14

Methods for Rates and Proportions (18, 598-625). Retrieved from:

http://hpm.fk.ugm.ac.id/hpmlama/images/Biostatistik/Tutorial_4_AS/2-chapter18.pdf

National Institute of Mental Health. (2007). Autism spectrum disorders (pervasive

developmental disorders). Retrieved from:

http://www.nimh.nih.gov/health/topics/autism-spectrum-disorders-pervasive-

developmental-disorders/index.shtml

BEHAVIORAL CODING 15

Appendix

Appendix A

In-Home Training for Fathers of Children with Autism: A Follow up Study and Evaluation of

Four Individual Training Components

Literature regarding fathers of children with autism remains sparse, and because mothers

are the more common intervening parent, few training methods have focused on fathers. Thus,

we sought to evaluate effects of in-home training directed at fathers and their ability to train

mothers in the same manner in which they were trained. Fathers were taught four skills

commonly associated with in-home training interventions for parents of children with autism:

following the child's lead, imitation with animation, commenting on the child, and expectant

waiting. Father skills were evaluated twice a week for 12 weeks during videotaped in-home

father-child play sessions. Analyses included visual inspection of graphed data and statistical

analyses of father skill acquisition, mother skill acquisition, and child behaviors with both

parents. A multivariate repeated measures analysis of 18 dyads revealed significant increases in

frequencies of fathers' imitation with animation, expectant waiting, and commenting on the child.

Child initiating rates increased significantly as did frequencies of child non-speech vocalizations.

Analysis of mothers revealed significant increases in frequencies of imitation with animation,

expectant waiting, and following the child's lead. Child behaviors had similar results for father

and mother sessions. Findings are consistent with those from the sentinel study indicating that

fathers can effectively implement skills that promote father-child social interactions and that

children respond positively to this approach (Elder, 2010).