data extraction and study coding - ktdrr...
Post on 19-Mar-2018
219 Views
Preview:
TRANSCRIPT
Data Extraction and Study Coding
36
The Review Protocol
! A priori statement of aims and methods of the review
! Research question(s), aims, methods are considered in advance of identifying the relevant literature – Conduct review with minimal bias – Greater efficiency in review process
(Torgerson, 2003)
37
Review Protocol Advantages
! Clear research question before the review ð Avoid retrieving irrelevant papers ! A priori inclusion and exclusion criteria ð avoid
changing criteria as review progresses or studies may be included on basis of their results
! If decisions are explicit it enables them to be justified
! Develop protocol as independently as possible from the literature ð avoid influence by one or two major studies, no bias (Torgerson, 2003)
38
Inclusion and Exclusion Criteria
! In high-quality systematic review inclusion and exclusion criteria need to be rigorously and transparently reported – E.g., time span of publications, type of
research to be reviewed (study design), relevance to research question (Torgerson, 2003)
39
Inclusion and Exclusion Criteria (cont.)
! Key features of inclusion and exclusion criteria: – They are established a priori – They are explicit – They are applied stringently – All retrieved studies are listed in the review
either under included or excluded together with a justification (Torgerson, 2003)
40
Inclusion and Exclusion Criteria - Example
! 1. Types of studies included – Randomized Controlled Trials – Non-randomized quasi-experimental designs – Single-subject experimental studies
! 2. Types of participants included – Children and youth between CA of 12 mo (the
earliest the diagnosis can be made reliably) to 21 yrs (when children stop receiving school services mandated under IDEA in the US)
41
Inclusion and Exclusion Criteria - Example
! 2. Types of participants included – Diagnosis of Autism Spectrum Disorders
(Autism Prototype, Childhood Disintegrative Disorder, Rett Syndrome, PDD-NOS).
– Children with Asperger’s will not be included
– Children with additional diagnoses qualify as long as they are diagnosed with ASD.
– Children described as “autistic-like” will be excluded. 42
Inclusion and Exclusion Criteria - Example
! 3. Types of interventions (and comparisons) included: – Picture Exchange Communication System
(PECS) (Frost & Bondy, 2002). – Innovations/adaptations of the PECS protocol
will be included if it is clearly specified how this innovation/modification deviates
– Comparison: any communication intervention that is appropriate for beginning communicators 43
Inclusion and Exclusion Criteria - Example
! 4. Types of outcomes included: – pre-linguistic behaviors such as eye contact
(duration) or joint attention (frequency),
– speech production (% intelligible vocalizations), – expressive
! social regulation functions (e.g., frequency of initiating), and
! communicative functions (e.g., requesting, commenting, frequency/%).
44
C2 Review Protocol
45
What is Data Extraction? ! Process of extracting the relevant information
from each study, either by copying into a table, or directly entering it into a database – Reviewer may use a data extraction sheet, also called
a study coding form
! This process of reading through a study and filling out the data extraction sheet with the appropriate information taken from that study is often referred to as study coding
46
Coding Manual ! Essential for coding process ! Specifies and describes what study features
should be extracted ! Helps to minimize error and bias in the
judgments of the coding process ! Facilitate the consistency of coding ! Coding manual pilot-tested with several articles
and revised accordingly before actual use
47
The Coding Process
! Two different parts of study coding: 1. Encoding information about study
characteristics (study descriptors) 2. Encoding information about the empirical
findings of the study (effect sizes) (Lipsey & Wilson, 2001)
48
The Coding Process (cont.)
! Study characteristics can be divided into – Those that represent the phenomenon
under study, e.g., kind of treatment (type of technology), instructional format, population being studied
– Those that represent the research methods that are used, e.g., particular designs, measures, procedures
ð Ideally neutral, but often times flaws can distort results, therefore, need to code
49
Study characteristics - Example
! I. Research Design Characteristics – Single-subject experimental designs
! (Adapted) Alternating Treatments Design ([A]ATD)
! Parallel Treatments Design (PTD) ! Multiple Baseline Design (MBD) ! Multiple Probe Design (MPD) ! Nonconcurrent MBD/MPD ! MBD/MPD combined with ATD/AATD ! Withdrawal ! Changing Criterion Design 50
Study characteristics - Example
! II. Participant Characteristics – N of subjects enrolled
– N of subjects that completed the study and entered analyses
– Disability – ASD diagnostic tests – Age at ASD Diagnosis – Severity of autism – Degree of mental retardation 51
Study characteristics - Example
! II. Participant Characteristics (cont.) – Chronological age – Gender/Race/Ethnicity – Receptive language (standardized tests) – Expressive language (standardized tests) – Overall developmental functioning – Speech before intervention – Speech imitation before intervention – AAC imitation before intervention 52
Study characteristics - Example
! III. Intervention Characteristics – Adherence to PECS protocol
– Innovation to the PECS protocol – PECS Phases implemented in intervention – Length of the intervention – Density of the intervention schedule – Interventionist preparation – Treatment integrity – Type of treatment integrity 53
Effect Size Extraction
! Involves converting descriptive or other statistical information contained in studies into a standard metric by which studies can be compared
! Research finding of interest often are relationships in a particular form between two specified constructs – E.g., differences between experimental conditions
measured on a dependent variable representing a certain outcome construct
54
Developing Coding Manual
! Coding manual and form are generally developed into two different modules – One for study characteristics that apply to
the entire study – The second for coding effect size
information ! Treatment studies can have multiple outcome
measures, and therefore, multiple, distinct effect sizes (Lipsey & Wilson, 2001)
55
Developing Coding Manual (cont.)
! Use close-ended items as much as possible (end goal may be to build a database)
! Design for easy use – Cluster items that deal with similar themes
and format so that they are easy to complete (Lipsey & Wilson, 2001)
56
Developing Coding Manual (cont.)
! Have a full definition of each item and guidelines for coding – Assists the coder in handling ambiguous,
unusual cases ! Bottom line: spell everything out in detail,
give full definitions of the various response options, and provide guidelines for dealing with borderline cases
57
Potential Bias ! “Data extraction bias”: reviewer introduces bias
into the review – Studies in accord with own view – Different judgments of quality or methodological adequacy
to different studies – Reviewer’s awareness of the study authors, the journal,
the results ð blinding the reviewers – Using multiple reviewers – Validating the data extraction
! Checked by another person ! Two people working independently (much less common but
likely to be more reliable) (Petticrew & Roberts, 2006)
58
Reliability on Coding ! Coder reliability has two dimensions: 1. Consistency of a single coder 2. Consistency between different coders ! To check ð draw subsample of coded studies, have
coders code them again and compare the results ! For single coder reliability: this person needs to code
studies again after sufficient time has passed so that no memory exists of original coding
! For between coder reliability: different coders code the same sample without reference to what the other has done (Lipsey & Wilson, 2001)
59
Reliability on Coding
! Once the appropriate double-coding has been done on a sample, the two sets of results are compared item by item
! Calculate reliability coefficients: – For continuous variables use Pearson’s r (e.g.,
participant’s age, effect size values) – For categorical coding variables use percentage
agreement procedure or Cohen’s kappa (e.g., type of intervention, outcome measure, design)
! Report actual reliability, then resolve discrepancies (Lipsey & Wilson, 2001)
60
Training of Coders (Lipsey & Wilson, 2001)
! Coding studies is one of the most technically demanding aspects of a systematic review
! Effective coder must – Understand coding protocol in detail and depth – Have the knowledge and skills to properly read
and interpret research report – Must have background in methodology – Be familiar with specific research domain to
perform coding task well
61
top related