wilson coding protocol page 1 development of coding protocol coding protocol: essential feature of...
Post on 23-Dec-2015
234 Views
Preview:
TRANSCRIPT
Wilson Coding Protocol Page 1
Development of Coding Protocol
Coding protocol: essential feature of meta-analysis Goal: transparent and replicable
description of studies extraction of findings
Wilson Coding Protocol Page 2
Topics for Coding
Eligibility criteria and screening form Development of coding protocol Hierarchical nature of data Assessing reliability of coding Training of coders Common mistakes
Wilson Coding Protocol Page 3
Study Eligibility Criteria
Flow from research question Identify specifics of:
Defining features of the program/policy/intervention Eligible designs; required methods Key sample features Required outcomes Required statistical data Geographical/linguistic restrictions, if any Time frame, if any
Also explicitly states what is excluded
Wilson Coding Protocol Page 4
Study Eligibility Screening Form
Develop a screening form with criteria Complete form for all studies retrieved as potentially
eligible Modify criteria after examining sample of studies
(controversial) Double-code eligibility Maintain database on results for each study screened Example
Wilson Coding Protocol Page 5
Development of Coding Protocol
Goal of protocol Describe studies Differentiate studies Extract findings (effect sizes if possible)
Coding forms and manual Both important
Wilson Coding Protocol Page 6
Development of Coding Protocol
Iterative nature of development Structuring data
Data hierarchical (findings within studies) Coding protocol needs to allow for this complexity Analysis of effect sizes needs to respect this structure Flat-file (example) Relational hierarchical file (example)
Wilson Coding Protocol Page 7
Example of a Flat File
ID Paradigm ES1 DV1 ES2 DV2 ES3 DV3 ES4 DV422 2 0.77 323 2 0.77 331 1 -0.1 5 -0.05 5 -0.2 1136 2 0.94 340 1 0.96 1182 1 0.29 11
185 1 0.65 5 0.58 5 0.48 5 0.068 5186 1 0.83 5204 2 0.88 3229 2 0.97 3246 2 0.91 3274 2 0.86 3 -0.31 3 0.79 3 1.17 3295 2 7.03 3 6.46 3 . 3 0.57 .626 1 0.87 3 -0.04 3 0.1 3 0.9 3
1366 2 0.5 3
Note that there is only one record (row) per study
Multiple ESs handled by having multiplevariables, one for each potential ES.
Wilson Coding Protocol Page 8
Example of a Hierarchical Structure
ID PubYear MeanAge TxStyle100 92 15.5 2
7049 82 14.5 1
OutcomeID ESNum Type TxN CgN ES
100 1 1 24 24 -0.39100 2 1 24 24 0100 3 1 24 24 0.09100 4 1 24 24 -1.05100 5 1 24 24 -0.44
7049 1 2 30 30 0.347049 2 4 30 30 0.787049 3 1 30 30 0
Note that a single record in the file above is “related” to five records in the file to the right
Study Level Data File
Effect Size Level Data File
Wilson Coding Protocol Page 9
Example of a More Complex MultipleFile Data Structure
ID PubYear MeanAge TxStyle100 92 15.5 2
7049 82 14.5 1
Study Level Data File Outcome Level Data FileID OutNum Constrct Scale
100 1 2 1100 2 6 1100 3 4 2
7049 1 2 47049 2 6 3
ID OutNum ESNum Months TxN CgN ES100 1 1 0 24 24 -0.39100 1 2 6 22 22 0100 2 3 0 24 24 0.09100 2 4 6 22 22 -1.05100 3 5 0 24 24 -0.44100 3 6 6 22 21 0.34
7049 1 2 0 30 30 0.787049 1 6 12 29 28 0.787049 2 2 0 30 30 0
Effect Size Level Data FileNote that study 100 has 2 records in the outcomes data file and 6 outcomes in the effect size data file, 2 for each outcome measured at different points in time (Months)
Wilson Coding Protocol Page 10
Advantages & Disadvantages of Multiple Flat Files Data Structure Advantages
Can “grow” to any number of ESs Reduces coding task (faster coding) Simplifies data cleanup Smaller data files to manipulate
Disadvantages Complex to implement Data must be manipulated prior to analysis Must be able to select a single ES per study for any analysis
When to use Large number of ESs per study are possible
Wilson Coding Protocol Page 11
Concept of “Working” Analysis Files
Study Data File
Outcome Data File
ES Data File
Composite Data File
createcompositedata file
select subset of ESs of interest to current analysis,e.g., a specific outcome atposttest
verify that there is only asingle ES per study
yes
Working Analysis File
Permanent Data Files
Average ESs, further selectbased explicit criteria, orselect randomly
no
Wilson Coding Protocol Page 12
Example: SPSS ES Data File
Wilson Coding Protocol Page 13
Example: SPSS ES+Outcome Data File
Wilson Coding Protocol Page 14
Example: SPSS ES+Outcome+Study Data File
Wilson Coding Protocol Page 15
Example: Creating Subset for Analysis
Wilson Coding Protocol Page 16
Example: Final Working File fora Single Analysis
Wilson Coding Protocol Page 17
Concept of “Working” Analysis Files
Study Data File
Outcome Data File
ES Data File
Composite Data File
createcompositedata file
select subset of ESs of interest to current analysis,e.g., a specific outcome atposttest
verify that there is only asingle ES per study
yes
Working Analysis File
Permanent Data Files
Average ESs, further selectbased on explicit criteria, orselect randomly
no
Wilson Coding Protocol Page 18
What about Sub-Samples?
What if you are interested in coding ESs separately for different sub-samples, such as, boys and girls, or high-risk and low-risk youth, etc? Just say “no”!
Often not enough of such data for meaningful analysis Complicates coding and data structure
Well, if you must, plan your data structure carefully Include a full sample effect size for each dependent measure
of interest Place sub-sample in a separate data file or use some other
method to reliable determine ESs that are statistically dependent
Wilson Coding Protocol Page 19
Coding Mechanics
Paper Coding (see Appendix E) include data file variable names on coding form all data along left or right margin eases data entry
Coding into a spreadsheet Coding directly into a computer database
Wilson Coding Protocol Page 20
Coding Directly into a Computer Database Advantages
Avoids additional step of transferring data from paper to computer
Easy access to data for data cleanup Data base can perform calculations during coding process (e.g.,
calculation of effect sizes) Faster coding
Disadvantages Can be time consuming to set up
the bigger the meta-analysis the bigger the payoff Requires a higher level of computer skill
Wilson Coding Protocol Page 21
Example of Database with Forms
Wilson Coding Protocol Page 22
Assessing Reliability of Coding
Inter-rater reliability and double coding Intra-rater reliability
Wilson Coding Protocol Page 23
Training Coders
Regular meetings (develops normative understandings) Annotate coding manual “Specialist” coders
Wilson Coding Protocol Page 24
Common Mistakes
Not understanding or planning the analysis prior to coding
Underestimating time, effort, and technical/statistical demands
Using a spreadsheet for managing a large review Variable names not on coding forms Not breaking apart difficult judgments
Wilson Coding Protocol Page 25
Common Mistakes
Over-coding—Trying to extract more detail than routinely reported
Wilson Coding Protocol Page 26
Comments on Managing the Bibliography Major activity Information you need to track
source of reference (e.g., PsychLit, Dissertation Abs.) retrieval status
retrieved, requested from ILL, etc. eligibility status
eligible not eligible relevant review article
coded status
Word processor not up to the task Spreadsheets are cumbersome Use a database of some form
top related