Data Extraction and Study Coding

36

The Review Protocol

! A priori statement of aims and methods of the review

! Research question(s), aims, methods are considered in advance of identifying the relevant literature – Conduct review with minimal bias – Greater efficiency in review process

(Torgerson, 2003)

37

Review Protocol Advantages

! Clear research question before the review ð Avoid retrieving irrelevant papers ! A priori inclusion and exclusion criteria ð avoid

changing criteria as review progresses or studies may be included on basis of their results

! If decisions are explicit it enables them to be justified

! Develop protocol as independently as possible from the literature ð avoid influence by one or two major studies, no bias (Torgerson, 2003)

38

Inclusion and Exclusion Criteria

! In high-quality systematic review inclusion and exclusion criteria need to be rigorously and transparently reported – E.g., time span of publications, type of

research to be reviewed (study design), relevance to research question (Torgerson, 2003)

39

Inclusion and Exclusion Criteria (cont.)

! Key features of inclusion and exclusion criteria: – They are established a priori – They are explicit – They are applied stringently – All retrieved studies are listed in the review

either under included or excluded together with a justification (Torgerson, 2003)

40

Inclusion and Exclusion Criteria - Example

! 1. Types of studies included – Randomized Controlled Trials – Non-randomized quasi-experimental designs – Single-subject experimental studies

! 2. Types of participants included – Children and youth between CA of 12 mo (the

earliest the diagnosis can be made reliably) to 21 yrs (when children stop receiving school services mandated under IDEA in the US)

41

Inclusion and Exclusion Criteria - Example

! 2. Types of participants included – Diagnosis of Autism Spectrum Disorders

(Autism Prototype, Childhood Disintegrative Disorder, Rett Syndrome, PDD-NOS).

– Children with Asperger’s will not be included

– Children with additional diagnoses qualify as long as they are diagnosed with ASD.

– Children described as “autistic-like” will be excluded. 42

Inclusion and Exclusion Criteria - Example

! 3. Types of interventions (and comparisons) included: – Picture Exchange Communication System

(PECS) (Frost & Bondy, 2002). –  Innovations/adaptations of the PECS protocol

will be included if it is clearly specified how this innovation/modification deviates

– Comparison: any communication intervention that is appropriate for beginning communicators 43

Inclusion and Exclusion Criteria - Example

! 4. Types of outcomes included: – pre-linguistic behaviors such as eye contact

(duration) or joint attention (frequency),

– speech production (% intelligible vocalizations), – expressive

! social regulation functions (e.g., frequency of initiating), and

! communicative functions (e.g., requesting, commenting, frequency/%).

44

C2 Review Protocol

45

What is Data Extraction? ! Process of extracting the relevant information

from each study, either by copying into a table, or directly entering it into a database –  Reviewer may use a data extraction sheet, also called

a study coding form

! This process of reading through a study and filling out the data extraction sheet with the appropriate information taken from that study is often referred to as study coding

46

Coding Manual ! Essential for coding process ! Specifies and describes what study features

should be extracted ! Helps to minimize error and bias in the

judgments of the coding process ! Facilitate the consistency of coding ! Coding manual pilot-tested with several articles

and revised accordingly before actual use

47

The Coding Process

! Two different parts of study coding: 1. Encoding information about study

characteristics (study descriptors) 2. Encoding information about the empirical

findings of the study (effect sizes) (Lipsey & Wilson, 2001)

48

The Coding Process (cont.)

! Study characteristics can be divided into – Those that represent the phenomenon

under study, e.g., kind of treatment (type of technology), instructional format, population being studied

– Those that represent the research methods that are used, e.g., particular designs, measures, procedures

ð Ideally neutral, but often times flaws can distort results, therefore, need to code

49

Study characteristics - Example

! I. Research Design Characteristics – Single-subject experimental designs

! (Adapted) Alternating Treatments Design ([A]ATD)  

! Parallel Treatments Design (PTD)  ! Multiple Baseline Design (MBD)  ! Multiple Probe Design (MPD)  ! Nonconcurrent MBD/MPD  ! MBD/MPD combined with ATD/AATD  ! Withdrawal ! Changing Criterion Design 50

Study characteristics - Example

! II. Participant Characteristics – N of subjects enrolled  

– N of subjects that completed the study and entered analyses

– Disability  – ASD diagnostic tests  – Age at ASD Diagnosis  – Severity of autism  – Degree of mental retardation 51

Study characteristics - Example

! II. Participant Characteristics (cont.) – Chronological age – Gender/Race/Ethnicity  – Receptive language (standardized tests)  – Expressive language (standardized tests)  – Overall developmental functioning  – Speech before intervention  – Speech imitation before intervention – AAC imitation before intervention 52

Study characteristics - Example

! III. Intervention Characteristics – Adherence to PECS protocol  

–  Innovation to the PECS protocol  – PECS Phases implemented in intervention  – Length of the intervention  – Density of the intervention schedule  –  Interventionist preparation – Treatment integrity  –  Type of treatment integrity 53

Effect Size Extraction

! Involves converting descriptive or other statistical information contained in studies into a standard metric by which studies can be compared

! Research finding of interest often are relationships in a particular form between two specified constructs –  E.g., differences between experimental conditions

measured on a dependent variable representing a certain outcome construct

54

Developing Coding Manual

! Coding manual and form are generally developed into two different modules – One for study characteristics that apply to

the entire study – The second for coding effect size

information ! Treatment studies can have multiple outcome

measures, and therefore, multiple, distinct effect sizes (Lipsey & Wilson, 2001)

55

Developing Coding Manual (cont.)

! Use close-ended items as much as possible (end goal may be to build a database)

! Design for easy use – Cluster items that deal with similar themes

and format so that they are easy to complete (Lipsey & Wilson, 2001)

56

Developing Coding Manual (cont.)

! Have a full definition of each item and guidelines for coding – Assists the coder in handling ambiguous,

unusual cases ! Bottom line: spell everything out in detail,

give full definitions of the various response options, and provide guidelines for dealing with borderline cases

57

Potential Bias ! “Data extraction bias”: reviewer introduces bias

into the review –  Studies in accord with own view –  Different judgments of quality or methodological adequacy

to different studies –  Reviewer’s awareness of the study authors, the journal,

the results ð blinding the reviewers –  Using multiple reviewers –  Validating the data extraction

! Checked by another person ! Two people working independently (much less common but

likely to be more reliable) (Petticrew & Roberts, 2006)

58

Reliability on Coding ! Coder reliability has two dimensions: 1.  Consistency of a single coder 2.  Consistency between different coders ! To check ð draw subsample of coded studies, have

coders code them again and compare the results ! For single coder reliability: this person needs to code

studies again after sufficient time has passed so that no memory exists of original coding

! For between coder reliability: different coders code the same sample without reference to what the other has done (Lipsey & Wilson, 2001)

59

Reliability on Coding

! Once the appropriate double-coding has been done on a sample, the two sets of results are compared item by item

! Calculate reliability coefficients: –  For continuous variables use Pearson’s r (e.g.,

participant’s age, effect size values) –  For categorical coding variables use percentage

agreement procedure or Cohen’s kappa (e.g., type of intervention, outcome measure, design)

! Report actual reliability, then resolve discrepancies (Lipsey & Wilson, 2001)

60

Training of Coders (Lipsey & Wilson, 2001)

! Coding studies is one of the most technically demanding aspects of a systematic review

! Effective coder must –  Understand coding protocol in detail and depth –  Have the knowledge and skills to properly read

and interpret research report –  Must have background in methodology –  Be familiar with specific research domain to

perform coding task well

61