MAKING SENSE OF DIAGNOSTIC META-ANALYSIS A VISUAL-GRAPHICAL FRAMEWORK

Ben A. Dwamena, MD. Department of Radiology, University of Michigan Medical School/ Nuclear Medicine Service (115), VA Ann Arbor Health Care System, Ann Arbor.

Francesca C. Dwamena, MD. Department of Medicine, Division of General Internal Medicine, Michigan State University College of Human Medicine, East Lansing.

EDUCATIONAL GOALS AND OBJECTIVES

PROVIDE A VISUAL/GRAPHIC FRAMEWORK FOR DIAGNOSTIC META-ANALYSIS THROUGH THE USE OF – Flow Charts – Contingency Tables – Forest Plots– Funnel Graphs– Scatter Diagrams– Summary Receiver Operator Characteristic Curves– Etc

DIAGNOSTIC META-ANALYSES

Improve quality of future primary studies

by identifying methodological deficiencies

Identify reasons for variation in reported

results

Generate valid summary estimates of

diagnostic performance

VISUAL DISPLAYS/GRAPHS

Provide more user-friendly summaries of large quantitative data sets

Preliminary data exploration before definite data synthesis

Clarify difficult statistical concepts and interpretation

GENERAL ARCHITECTURE

Formulate Question

Develop Search Strategy and Retrieve

Articles

Select Eligible Studies and Assess Quality

Extract Data and Calculate Individual

Summary Measures

Choose Model for Pooling

Investigate Heterogeneity and Biases

DATA SOURCES

VISUAL/GRAPHIC DISPLAYS BASED ON

Dwamena BA, Sonnad SS, et al.

Metastases From Non-small Cell Lung

Cancer: Mediastinal Staging In The

1990s- A Meta-analytic Comparison Of

PET and CT.Radiology 1999; 213:530-

36.

Published Work of Other Investigators in

the Field Based on Either Original or

Simulated Data

RESEARCH QUESTION

GENERAL EXAMPLE: How Accurate Is a Sign, Symptom, or Diagnostic Test in Predicting the True Diagnostic Category of a Patient?

RELEVANT QUESTION: Addresses Population or Patient Group, Diagnostic Intervention, Disease of Interest

FINDING RELEVANT STUDIES

SEARCH FOR EXISTING REVIEWS

FIND PUBLISHED PRIMARY STUDIES

Break down research question into components

Use appropriate synonyms

Use electronic databases, hand searching,etc

LOOK FOR UNPUBLISHED PRIMARY

STUDIES

(write to experts, search registries for

completed/ongoing trials)

BREAKING QUESTION DOWN INTO

COMPONENTS “What is the accuracy of fecal occult blood test for detection of colorectal

cancer? “ may be represented by a VENN DIAGRAM:

RECOMMENDED SEARCH STRATEGY REGARDING TEST

PERFORMANCE

Deville WL et. al. BMC Medical Research Methodology 2002, 2:9-22

QUALITY CITERIA

PATIENT SELECTION: Consecutive vs. Non-consecutive or convenience

sample

SPECTRUM: Clinically relevant population versus case-control

REFERENCE STANDARD: Full vs. Partial reporting of cut-off value

DIAGNOSTIC TEST: Full vs. partial reporting of cut-off value

DATA COLLECTION: Prospective versus Retrospective versus unknown

DETAILS OF POPULATION: Sufficient versus Insufficient

VERIFICATION: Complete versus different reference tests versus

incomplete

INTERPRETATION OF RESULTS: Blinded versus Unblinded

METHODOLOGICAL STANDARDS

Quality of Each Selected Paper Should Be Assessed

Independently by at Least Two Reviewers.

Chance-adjusted Agreement Should Be Reported

and Disagreements Solved by Consensus or

Arbitration.

To Improve Agreement, Reviewers Should Pilot

Their Quality Assessment Tools in a Subset of

Included Studies or Studies Evaluating a Different

Diagnostic Test

METHODOLOGICAL STANDARDS

FLOW CHART OF STUDY RETRIEVAL AND

SELECTION

DATA EXTRACTION

Info About the Participants Included in the Study, Time of Data Collection and the Testing Procedures.

The Cut-off Point Used in Dichotomous Testing Reasons and the Number of Participants Excluded Because of Indeterminate Results or Unfeasibility.

Extracted Information May Be Used in Subgroup Analyses and Statistical Pooling.

DATA EXTRACTION

Multiple Reviewers Should Independently Extract the Required Information.

Obtain Data Construct the Diagnostic 2 × 2 Table: Absolute Numbers in the Four Cells Are Needed.

Obtain Totals 'Diseased' and 'Non-diseased' to Calculate Prior Probability (Pre-test Probability) From Recalculated Sensitivity, Specificity, Likelihood Ratios, Predictive Values

CONTINGENCY TABLE FOR EXTRACTION OF TEST

DATA

DIAGNOSTIC VS. TREATMENT TRIAL

True Positives =Experimental Group With the Monitored Outcome Present (a).

False Positives = Control Group With the Outcome Present (b).

False Negatives=experimental Group With the Outcome Absent (c).

True Negatives Are the Control Group With the Outcome Absent (d).

DIAGNOSTIC VS. TREATMENT TRIAL

Relative risk in experimental group {[a/(a +

c)]/[b/(b+ d)]} =Likelihood Ratio for a

Positive Test.

Relative Risk in Control Group = Likelihood

Ratio for a Negative Test.

The Expression for the Odds Ratio (OR) Is (a

x d)/(b x c).

CONTINGENCY DATA FOR NSCLC PET STUDY

CHOICE OF MODEL AND INDEX FOR

POOLING OF TEST PERFORMANCE

SEARCHING FOR THRESHOLD EFFECT

Test for the Presence of Threshold Effect

Between Studies by Calculating a Spearman

Correlation Coefficient Between Sensitivity

and Specificity of All Included Studies

A Spearman Correlation of < -0.6, Suggests

Evidence of Interdependence of Sensitivity

and Specificity , and SROC Curves Should Be

Constructed or ROC Curves Can Be Pooled

SEARCHING FOR HOMOGENEITY

Perform Chi-square or Fisher’s Test for Small Number

Studies.

If Sensitivity and Specificity Are Homogeneous, and Show

No Threshold Effect, They Can Be Pooled by Fixed Effect

Model.

If Heterogeneity Is Present, Restrict the Analysis to a

Qualitative Overview; Pool Data From Homogeneous Sub-

groups; Use Random Effect Model.

SROC PROCEDURE

1 SCATTER PLOT TPR VS. FPR

Visualization of range of results from primary studies

2 REGRESSION OF D ON S

Straight lines fitted to estimate (a) best fit to the data (b) remove effect of possible relationship between results and positivity threshold

3 BACKTRANSFORMATION OF REGRESSION TO CONVENTIONAL AXES

Presentation of combined results into a single ROC curve

LINEAR REGRESSION ANALYSIS

Logit transformations of the TP rate

(sensitivity) and FP rate (1 - specificity).

D=ln(DOR) =logit(TPR) – logit(FPR)

Differences in logit transformations, D,

regressed on sums of logit transformations, S.

S=logit(TPR)+logit(FPR)

Logit(TPR)=natural log odds of a TP result and

logit(FPR) =natural log of the odds of a FP test

result.

LINEAR REGRESSION MODELS

ORDINARY LEAST SQUARES METHODStudies are weighted equally

WEIGHTED LEAST SQUARES METHODWeighted by the inverse variance weights of the diagnostic odds ratio, or simply the sample size

ROBUST-RESISTANT METHODMinimizes the influence of outliers

LINEAR REGRESSION PLOT

SUMMARY ROC CURVE

Back transformation of logistic regression to conventional axes of sensitivity [TPR] vs. (1 – specificity) [FPR]) with the equation

TPR = 1/{1 + exp[- a/(1 - b )]} [(1 - FPR)/(FPR)](1 + b )/(1 - b ).

Slope (b) and intercept (a) are obtained from the linear regression analyses

SUMMARY ROC CURVE

FOREST PLOT OF STUDY-SPECIFIC AND SUMMARY SENSITIVITY AND

SPECIFICITY

FIXED EFFECTS META-ANALYSIS

Assumes Same Diagnostic Accuracy in All Studies

Variation in Sensitivity and Specificity From Published Reports Due to

Random Error/chanceThreshold Variation

FIXED EFFECTS META-ANALYSIS

RANDOM EFFECTS META-ANALYSIS

Assumes Diagnostic Accuracy Varies From Study to Study

Variation in of Reported Accuracy Estimates Are Randomly Distributed About Some Central Value Represented by SROC.

Variation Due to Stage of Disease, Clinical Presentation, Prevalence of Disease, Study Design Etc.

RANDOM EFFECTS META-ANALYSIS

WEIGHTED HISTOGRAM BREAST CANCER DATA

DEALING WITH HETEROGENEITY

Repeat Analysis Analysis After Excluding Outliers

Conduct Analysis With Predefined Subgroups.

Use Analysis of Variance With the Lndor As

Dependent Variable and Categorical Variables for

Subgroups As Factors to Look for Differences Among

Subgroups;

Construct Multivariate Models to Search for the

Independent Effect of Study Characteristics

GALBRAITH PLOT

Standardized Log-odds Ratio Plotted Against the Reciprocal of the Standard Error.

Small Studies/less Precise Results Appear on the Left Side and the Largest Trials on the Right End .

A Regression Line , Through the Origin, Represents the Overall Log-odds Ratio.

Lines +/- 2 Above Regression Line, Represent the 95 Per Cent Boundaries of the Overall Log-odds Ratio.

The Majority of Results Should Lie in This Area in the Absence of Heterogeneity.

GALBRAITH PLOT

FUNNEL DIAGRAM

A Funnel Diagram (A.K.A. Funnel Plot, Funnel Graph)

Is a Special Type of Scatter Plot With an Estimate

Sample Size on One Axis and Effect-size Estimate

on the Other Axis.

Based on the Well Known Statistical Principle That

Sampling Error Decreases As Sample Size Increases.

Used to Search for Publication Bias and to Test

Whether All Studies Come From a Single Population.

SIMULATED FUNNEL PLOTS FOR EXPLORING

PUBLICATION BIAS

FUNNEL PLOT REGRESSION

EGGER’S REGRESSION METHOD FOR DETECTING

PUBLICATION BIAS

FUNNEL PLOT OF NSCLC PET DATA

NORMAL QUANTILE PLOTS

NORMAL QUANTILE PLOTS

NORMAL QUANTILE PLOT OF

NSCLC PET DATA

NORMAL QUANTILE PLOT OF AXILLARY BREAST CANCER PET DATA

ODDS RATIO SUBGROUP ANALYSIS

SROC SUBGROUP ANALYSIS

SOFTWARE

METATEST (Dr Lau, NEMC, Boston)

SROC Curve Analysis

METAWIN (Sinauer Associates, Sunderland, MA)

Scatter, Funnel, Normal Quantile, Forest, Weight

Histogram, Radial (Galbraith) and Cummulative Meta-

analysis Plots

STATSDIRECT(StatsDirect Ltd, Herts, UK)

Forest, Funnel and L’abbe Plots