Using Criteria to Appraise a Meta-analyses

Paul CroninB.A., M.B. B.Ch. B.A.O., M.S.,

M.R.C.P.I., .F.R.C.R.

Department of Radiology,

Division of Cardiothoracic Radiology,

University of Michigan, Ann Arbor, Michigan

AUR 59th Annual MeetingWestin Boston Waterfront,

Boston, Massachusetts

April 12-15, 2011March 23-26 , 2010

Critical Appraisal of Secondary

Research

• Reviews of the scientific literature

–Selective summaries

• Highly subjective

–Objective summaries

• Quantitative information syntheses of the

best research evidence

• The challenge is where along this

spectrum the review lies

Principles of Critical Appraisal –

Secondary Literature

• Systematic review and Meta-analysis

of diagnostic test accuracy

– (1) Finding the evidence –

• Framing objectives of the review

• Identifying the relevant literature

– (2) Assessment of study quality and

applicability to the clinical problem at

hand

Principles of Critical Appraisal –

Secondary Literature

• Systematic review and Meta-analysis

of diagnostic test accuracy

– (3) Summarizing the evidence

• Qualitatively and if appropriate,

quantitatively (meta-analysis)

– (4) Clinical Interpretation

• Application of findings and development of

recommendations

Questions

• Question 1:

–What is the PICO of the study and is it

close enough to your PICO?

• Question 2:

–How well was the study done?

• Question 3:

–What do the results mean and could they

have been due to chance?

Question 1

• What is the PICO of the study and is it

close enough to your PICO?

Finding the Evidence

• Defining the research question

–P – Population / problem

– I – Intervention / index test

–C – Control / reference test

–O – Outcome

Identifying the Evidence

• Literature search strategy– 1. The target condition of interest (all

subheadings)

– 2. The name of the test (all subheadings) and text

– 3. Combine the results of these searches with Boolean operator “AND”

– 4. Further search terms to restrict the search should only be introduced if this approach produces unmanageably large and non-specific results

Question 2

• How well was the study done?

Biases

• Studies that are chosen for inclusion

–Selection bias

• Published in the first place

–Publication bias

• Level of importance attributed to the

study results by the secondary

researcher

• Results are summarized

QFASSteps EBP Secondary

research

Critical appraisal issue

Q Formulate a

question

(PICO)

Formulate a

question

(PICO)

Does the research ask a

clearly focused

question (PICO) and

use it to direct the

search?

F Find the best

evidence

Find the best

evidence

Did the search find all the

best evidence?

A Appraise the

included

studies

Appraise the

included

studies

Have the studies been

critically appraised?

S Synthesis the

results

(summary tables

and plots)

Have the results been

synthesized with

appropriate summary

tables and plots?

Question

• Does the research ask a clearly

focused question (PICO) and use it to

direct the search?

–Excluded studies should be recorded

with reasons for the exclusions

• Helps to eliminate selection bias

Find

• Search strategy

–Did the search find all the best

evidence?

• Use PubMed for MEDLlNE, OLDMEDLINE,

EMBASE, OVID, CANCERLIT, and the

Cochrane Library databases, etc

Find

• Search strategy

–Did the search find all the best

evidence?

• Searching of relevant journals

• Conference proceedings

• Reference lists of articles in the searches

• Non-English as well as English papers

–Overcoming selection bias

Find

• Overcoming publication bias

–Much primary research missed because

of “publication bias”

• Reluctant to submit/publish null or negative

results

–Good-quality analysis takes account of

unpublished studies

Find

• Overcoming publication bias

–Discussed in the methods section

–Check

• Clinical trials registries

–http://www.controlled-trials.com

• Contacting experts

• Conference proceedings

• Internet

• Other sources of unpublished literature

Appraise

• Have the studies been critically

appraised (using appropriate quality

criteria)?

–Searching has been well done

–Validity of the results and conclusions

depends on the quality of the individual

studies included

–Good-quality analysis should include a

critical appraisal for each of the studies

Appraise

• Have the studies been critically

appraised (using appropriate quality

criteria)?

–This ideally should be appraised by two

assessors working independently

–Points of disagreement discussed and a

consensus reached

–Critical assessment can be subjective -

avoided a single reviewer

Quality Appraisal of Evidence

–1. Methods and criteria used for the

selection of the study population

–2. Selection of the reference standard

–3. Execution of index test strategy,

comparator and reference standard

–4. Interpretation of index test,

comparator and reference standard

–5. Analysis of the results

Quality Appraisal of Evidence

• QUADAS

– QUality

– Assessment

– Diagnostic

– Accuracy

– Studies

The QUADAS tool

Yes No Unclear

1. Was the spectrum of patients representative of the patients who will receive the test in practice? ( ) ( ) ( )

2. Were selection criteria clearly described? ( ) ( ) ( )

3. Is the reference standard likely to correctly classify the target condition? ( ) ( ) ( )

4.Is the time period between reference standard and index test short enough to be reasonably sure that

the target condition did not change between the two tests?( ) ( ) ( )

5.Did the whole sample or a random selection of the sample, receive verification using a reference

standard of diagnosis?( ) ( ) ( )

6. Did patients receive the same reference standard regardless of the index test result? ( ) ( ) ( )

7.Was the reference standard independent of the index test (i.e. the index test did not form part of the

reference standard)?( ) ( ) ( )

8. Was the execution of the index test described in sufficient detail to permit replication of the test? ( ) ( ) ( )

9. Was the execution of the reference standard described in sufficient detail to permit its replication? ( ) ( ) ( )

10. Were the index test results interpreted without knowledge of the results of the reference standard? ( ) ( ) ( )

11. Were the reference standard results interpreted without knowledge of the results of the index test? ( ) ( ) ( )

12.Were the same clinical data available when test results were interpreted as would be available when

the test is used in practice?( ) ( ) ( )

13. Were uninterpretable/ intermediate test results reported? ( ) ( ) ( )

14. Were withdrawals from the study explained? ( ) ( ) ( )

Synthesis

• Have the results been synthesized

with appropriate summary tables and

plots?

–Summary of the included studies (table)

–Show the types of studies

• Tests

• Number of subjects

• Results (95% confidence intervals)

–Graphic presentations – “forest plots”

Synthesis

• Have the results been synthesized

with appropriate summary tables and

plots?

–Heterogeneity should be assessed and

discussed

Question 3

• What do the results mean?

Results

• Weighted values to each individual

study according to their size

• Expressed in a standard way

–Sensitivity, specificity, PPV, NPV,

AUROC curve

–RR, OR, etc

• Traditionally displayed in a figure

called a forest plot

Forest Plot

• Individual studies are represented by

a black square and a horizontal line

–Corresponds to point estimate and 95%

CI

• The size (area) of the black square

reflects the weight of the study

–Weight is proportional to the inverse of

the study variance

Forest Plot

• The result is not significant at

conventional levels (P > 0.05)

• The diamond at the bottom

represents the combined or pooled

result of all studies with its 95% CI

Statistical Methods for Pooling Data

• SROC

• Hierarchical

summary ROC

(HSROC)

• Bivariate random-

effects model

Summary ROC Curve

Exploring Heterogeneity

• Heterogeneity can be assessed using

the 'eyeball' test

• Formally with statistical tests, such

as I2 and the Cochran chi-square (Q)

test

• The amount of heterogeneity is

calculated as the I2 value

–0 if none near 1 if a lot

Exploring Heterogeneity

• The statistical significance is

assessed using the Cochran Q test

– If Cochran Q is statistically significant -

there is definite heterogeneity

– If Cochran Q is not statistically

significant but the ratio of Cochran Q

and the degrees of freedom (Q/df) is

greater than 1 - there is possible

heterogeneity

Exploring Heterogeneity

• The statistical significance is

assessed using the Cochran Q test

– If Cochran Q is not statistically

significant and Q/df is less then 1 - then

heterogeneity is very unlikely

Exploring Publication Bias

• Funnel plot

• +/- linear

regression

Assessment of Publication Bias

• Funnel plot

• Non-parametric

linear regression

• Parametric linear

regression

– formally test for

funnel plot

asymmetry

Clinical Interpretation

of the Evidence

• Fagan plot (Bayes

Nomogram)

Clinical Interpretation

of the Evidence• Likelihood ratio scattergraph

– 1. Left Upper Quadrant, Likelihood Ratio Positive > 10, Likelihood Ratio Negative <0.1: Exclusion & Confirmation

– 2. Right Upper Quadrant, Likelihood Ratio Positive >10, Likelihood Ratio Negative >0.1: Confirmation Only

– 3. Left Lower Quadrant, Likelihood Ratio Positive <10, Likelihood Ratio Negative <0.1: Exclusion Only

– 4. Right Lower Quadrant, Likelihood Ratio Positive <10, Likelihood Ratio Negative >0.1: No Exclusion or Confirmation

Clinical Interpretation

of the Evidence

• Predictive Values

and Probability

Modifying Plot

Probability Modifying Plot

Section/topic #Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Checklist item

Reported on page #

TITLE

Title 1 Identify the report as a systematic review, meta-analysis, or both.

ABSTRACT

Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study

eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results;

limitations; conclusions and implications of key findings; systematic review registration number.

INTRODUCTION

Rationale 3 Describe the rationale for the review in the context of what is already known.

Objectives 4 Provide an explicit statement of questions being addressed with reference to participants,

interventions, comparisons, outcomes, and study design (PICOS).

METHODS

Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if

available, provide registration information including registration number.

Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years

considered, language, publication status) used as criteria for eligibility, giving rationale.

Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors

to identify additional studies) in the search and date last searched.

Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it

could be repeated.

Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis).

Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and

any processes for obtaining and confirming data from investigators.

Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any

assumptions and simplifications made.

Risk of bias in individual

studies

12 Describe methods used for assessing risk of bias of individual studies (including specification of

whether this was done at the study or outcome level), and how this information is to be used in any

data synthesis.

Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means).

Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures

of consistency (e.g., I2) for each meta-analysis.

Section/topic # Checklist item Reported on page #

Risk of bias across

studies

15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias,

selective reporting within studies).

Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.

RESULTS

Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons

for exclusions at each stage, ideally with a flow diagram.

Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-

up period) and provide the citations.

Risk of bias within

studies

19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item

12).

Results of individual studies

20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.

Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of

consistency.

Risk of bias across

studies

22 Present results of any assessment of risk of bias across studies (see Item 15).

Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression

[see Item 16]).

DISCUSSION

Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their

relevance to key groups (e.g., healthcare providers, users, and policy makers).

Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review-level (e.g., incomplete

retrieval of identified research, reporting bias).

Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for

future research.

FUNDING

Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of

funders for the systematic review.

AMSTAR 2

Summary

• Question

–The review is a systematic review based

on a clear clinical question and the PICO

is similar to ours

Summary

• Find

–The inclusion criteria and search

methods are stated in the methods

section

– Inclusion criteria were based on the

clinical question

–A comprehensive search of the literature

was conducted, including MEDLINE

(PubMed) and EMBASE

Summary

• Find

–Contacted the authors of included

papers directly and checked the

reference lists for further relevant

papers

–They also searched the Cochrane

controlled trial register for unpublished

clinical trials

Summary

• Appraise

–The studies were critically appraised

using the QUODAS, DELPHI, Jadad

quality scores

–Scores are shown in the paper

Summary

• Synthesise

–The paper includes a clear summary

table of the included studies

–Forest plots

–Heterogeneity analysis for the outcomes

–Publication bias

–Clinical interpretation of the evidence

Using Criteria to Appraise a Meta-analyses

Paul CroninB.A., M.B. B.Ch. B.A.O., M.S.,

M.R.C.P.I., .F.R.C.R.

Department of Radiology,

Division of Cardiothoracic Radiology,

University of Michigan, Ann Arbor, Michigan