Systematic Review Module 7:Systematic Review Module 7:Rating the Quality of Individual Rating the Quality of Individual
StudiesStudies
Meera Viswanathan, PhD Meera Viswanathan, PhD RTI-UNC EPCRTI-UNC EPC
Learning ObjectivesLearning Objectives
Define the concept of quality Define the concept of quality assessmentassessment
What are the reasons for quality What are the reasons for quality assessment?assessment?
What are the stages in quality What are the stages in quality assessment?assessment?
How do we report quality assessment?How do we report quality assessment?
CER Process OverviewCER Process Overview
3
Prepare topic:
· Refine key questions
· Develop analytic frameworks
Search for and select
studies:
· Identify eligibility criteria
· Search for relevant studies
· Select evidence for inclusion
Abstract data:
· Extract evidence from studies
· Construct evidence tables
Analyze and synthesize data:
· Assess quality of studies
· Assess applicability of studies
· Apply qualitative methods
· Apply quantitative methods (meta-analyses)
· Rate the strength of a body of evidence
Present findings
Reasons for Quality Reasons for Quality AssessmentAssessment
Quality assessment is required for– Interpreting results
– Grading the body of evidence
Quality assessment may also be used for– Selecting studies for inclusion
– Selecting studies for pooling
4
What is Quality Assessment?What is Quality Assessment?
Quality can be defined as “the extent to which all aspects of a study’s design and conduct can be shown to protect against systematic bias, nonsystematic bias, and inferential error” (Lohr, 2004)
Considered synonymous with internal validity
Relevant for individual studies Distinct from assessment of risk of bias for
a body of evidence
5
Lohr KN. Rating the Strength of Scientific Evidence: Relevance for Quality Improvement Programs. International Journal for Quality in Health Care 2004; 16(1):9-18.
What are Components of What are Components of Quality Assessment?Quality Assessment?
Systematic errors include selection bias and confounding, in which values tend to be inaccurate in a particular direction
Nonsystematic errors are attributable to chance
Inferential errors result from problems in data analysis and interpretation, such as choice of the wrong statistical measure or wrongly rejecting the null hypothesis
6
Lohr KN, Carey TS. Assessing 'best evidence': issues in grading the quality of studies for systematic reviews. Joint Commission . Journal On Quality Improvement 1999, Sep; 25(9), 470-9.
Consider the Contribution of an Consider the Contribution of an Individual Study to a Body of EvidenceIndividual Study to a Body of Evidence
7
Internal validity Internal validity of resultsof results
Size of studySize of study(random error)(random error)
Direction and degree of resultsDirection and degree of results
Relevance of Relevance of resultsresults
(applicability)(applicability)
Type of studyType of study
Limitations in Limitations in study design and conductstudy design and conduct
Risk of BiasRisk of Bias
PrecisionPrecision
DirectnessDirectness
ConsistencyConsistency
ApplicabilityApplicability
What Are the Stages in What Are the Stages in Quality Assessment?Quality Assessment?
Classify the study design Apply predefined criteria for quality and
critical appraisal Arrive at a summary judgment of the
study’s quality
8
Questions to Consider When Questions to Consider When Classifying Study DesignClassifying Study Design
Is a control group present? Is there concurrent assessment of intervention or
exposure status? Do investigators have control over allocation and
timing? Do investigators randomly allocate interventions? Is there more than one group? Is there concurrent assessment of outcomes?
9
Apply Predefined CriteriaApply Predefined Criteria
Apply one of several available tools that consider Similarity of groups at baseline in terms of baseline
characteristics and prognostic factors Extent to which valid primary outcomes were
described Blinding of subjects and providers Blinded assessment of the outcome Intention-to-treat analysis Differential loss to followup between the compared
groups or overall high loss to followup Conflict of interest
10
Additional Criteria for TrialsAdditional Criteria for Trials
Methods used for randomization Allocation concealment
11
Additional Criteria for Additional Criteria for Observational StudiesObservational Studies
Sample size Methods for selecting participants
(inception cohort, methods to avoid selection bias)
Methods for measuring exposure variables Methods to deal with any design-specific
issues such as recall bias and interviewer bias
Analytical methods to control confounding
12
Arrive at a Universal Arrive at a Universal Judgment of QualityJudgment of Quality
Assign ratings of good, fair, or poor Ratings may vary across outcomes for
an individual study Ratings should be based on the
assessment of the impact of individual criteria on overall internal validity rather than on summary scores
13
Attributes of Good StudiesAttributes of Good Studies
A formal randomized controlled study Clear description of the population, setting,
interventions, and comparison groups Appropriate measurement of outcomes Appropriate statistical and analytic methods
and reporting No reporting errors Low dropout rate Clear reporting of dropouts
14
Attributes of Fair StudiesAttributes of Fair Studies
Fair studies do not meet all the criteria required for a rating of good quality, because they have some deficiencies
No flaw is likely to cause major bias Missing information often drives rating
15
Attributes of Poor StudiesAttributes of Poor Studies
Significant biases, including– Errors in design, analysis, or reporting
– Large amounts of missing information
– Discrepancies in reporting
16
Reporting Quality Reporting Quality AssessmentAssessment
Overall assessments of quality must be accompanied by a statement of– Flaws in design or execution of a study
– Assessment of the potential consequences of that flaw
Poor studies may be excluded or included– Decisions should be guided by gaps in
current evidence
– Selective inclusion of poor studies for subgroups should be justified
17
Key MessagesKey Messages
Transparency of process– Full reporting on all elements of quality for
each individual study
– Clear instructions on how abstractors scored quality
– Description of reconciliation process
Transparency of judgment– Explanation of final score
18
Key SourceKey Source
Draft AHRQ Methods Guide, Chapter 6, AHRQ, 2007 http://www.effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf.
1919