GRADING QUALITY OF EVIDENCETHE GRADE APPROACH

Society of General International Medicine

32nd Annual Meeting, May 14th 2009

Elie A. Akl, MD, MPH, PhD David Atkins, MD, MPHEric Bass, MD, MPH Yngve Falck-Ytter, MD Stephanie Chang, MD, MPH

1

Session outline

Introductions, objectives (5 min)

Overview of the GRADE approach (25 min)

Applying the GRADE approach (45 min)

Wrap-up (10 min) Session evaluation (5

min)

Disclosure

Presenters are members of the GRADE working group and have received honoraria related to this work that were deposited into research accounts

No conflict of interest related to pharmaceutical industry

Objectives

Learning objectives

To enumerate GRADE categories for quality of evidence

To list the GRADE factors that affect the quality of evidence

To apply the GRADE approach to a specific body of evidence

To discuss the strengths and limitations of the GRADE approach

Overview of the GRADE approach

GRADE WORKING GROUP

Grades of Recommendation Assessment,

Development and Evaluation

CMAJ 2003, BMJ 2004, BMC 2004, BMC 2005, AJRCCM 2006, Chest 2006, BMJ 2008

“Extent to which confidence in estimate of effect adequate to support decision”

GRADE definition of Quality of Evidence

GRADE rating of outcomes

GRADE rates the quality of evidence for each outcome separately The type of evidence may be different

for different outcomes Different audiences are likely to have

varying perspective on the importance of outcomes

GRADE considers desirable and undesirable outcomes and rates their relative importance

9

Desirable outcomes lower mortality reduced hospital stay reduced duration of disease reduced resource expenditure

Undesirable outcomes adverse reactions the development of resistance costs of treatment

GRADE rating of outcomes

2

Critical for decision making

Important, but not critical for decision making

Of lowimportance

5

6

7

8

9

3

4

1

GRADE rating of outcomes

Ranking outcomes by their relative importance can help to focus attention on those outcomes that are considered most important

Outcome choice should be based on what is important, and not what was measured

12

GRADE rating of outcomes

GRADE uses a comprehensive and transparent conceptual framework for rating the quality of evidence

13

High:

Moderate:

Low:

Very low:

GRADE levels of Evidence

High: considerable confidence in estimate of effect

Moderate: further research likely to have

impact on confidence in estimate, may change estimate

Low: further research is very likely to impact on confidence, likely to change the estimate

Very low: any estimate of effect is very uncertain

GRADE levels of Evidence

Quality starts high for evidence from RCTs

Quality starts low for evidence from observational studies

5 factors lower the quality of evidence

3 factors can increase the quality of evidence

Determinants of quality

Factors that lower quality1. Study limitations (in design and

execution)2. Inconsistency3. Indirectness4. Reporting bias5. Imprecision

17

1. Study limitations (in design and execution)

Inappropriate randomization Lack of concealment Intention to treat principle violated Inadequate blinding Loss to follow-up Early stopping for benefit

Factors that lower quality

From Cates , CDSR 2008

CDSR 2008

Factors that lower quality

Overall judgment required

Factors that lower quality

2. Inconsistency Assess for inconsistency (Heterogeneity)

variation in size of effect overlap in confidence intervals statistical significance of heterogeneity I2

If inconsistency look for explanation patients, intervention, outcome, methods

If unexplained inconsistency downgrade quality

Factors that lower quality

Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.

2. Inconsistency

Heparin or vitamin K antagonists for survival in patients with cancer:

Factors that lower quality

Capurso G, Schünemann HJ, Terrenato I, Moretti A, Koch M, Muti P, Capurso L, Delle Fave G. Meta-analysis: the use of non-steroidal anti-inflammatory drugs and pancreatic cancer risk for different exposure categories.

Aliment Pharmacol Ther. 2007 Oct 15;26(8):1089-99.

2. Inconsistency

Non-steroidal drug use and risk of pancreatic cancer:

Factors that lower quality

3. Indirectness of Evidence Differences in populations/patients

mild versus severe COPD

Differences in interventions all inhaled steroids, new vs. old

Differences in outcomes important vs. surrogate;

Factors that lower quality

Alendronate

Risedronate

Placebo

3. Indirectness of Evidence indirect comparisons

interested in A versus B have A versus C and B versus C

Factors that lower quality

4. Publication bias

Number of small studies

Faster and multiple publication of “positive” trials

Fewer and slower publication of “negative” trials

Factors that lower quality

Egger M, Smith DS. BMJ 1995;310:752-54 27

I.V. Mg in acute myocardial infarctionPublication bias

Meta-analysisYusuf S.Circulation 1993

ISIS-4Lancet 1995

Egger M, Cochrane Colloquium Lyon 2001 28

Funnel plotS

tand

ard

Err

or

Odds ratio0.1 0.3 1 3

3

2

1

0

100.6

Symmetrical:No publication bias

Egger M, Cochrane Colloquium Lyon 2001 29

Funnel plotS

tand

ard

Err

or

Odds ratio0.1 0.3 1 3

3

2

1

0

100.6

Asymmetrical:Publication bias?

Egger M, Smith DS. BMJ 1995;310:752-54 30

I.V. Mg in acute myocardial infarctionPublication bias

Meta-analysisYusuf S.Circulation 1993

ISIS-4Lancet 1995

Egger M, Smith DS. BMJ 1995;310:752-54 31

Meta-analysis confirmed by mega-trials

5. Imprecision small sample size

small number of events

wide confidence intervals uncertainty about magnitude of effect

how to decide if CI too wide? grade down one level? grade down two levels?

Factors that lower quality

Factors that raise quality1. Large magnitude of effect

2. Dose response relation

3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

33

1. Large magnitude of effect large (RRR 50%) can raise by one level very large (RRR 80%) can raise by two

levels common criteria

everyone used to do badly almost everyone does well

Examples oral anticoagulation for mechanical heart

valves insulin for diabetic ketoacidosis hip replacement for severe osteoarthritis

Factors that raise quality

2. Dose response relation

higher INR – increased bleeding

childhood lymphoblastic leukemia risk for CNS malignancies 15 years after

cranial irradiation no radiation: 1% (95% CI 0% to 2.1%) 12 Gy: 1.6% (95% CI 0% to 3.4%) 18 Gy: 3.3% (95% CI 0.9% to 5.6%)

Factors that raise quality

3. All plausible confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

Factors that raise quality

Example 1: higher death rates in private for-profit versus private not-for-profit hospitals

patients in the not-for-profit hospitals likely sicker than those in the for-profit hospitals

for-profit hospitals are likely to admit a larger proportion of well-insured patients than not-for-profit hospitals (and thus have more resources with a spill over effect)

Factors that raise quality

Example 2: hypoglycaemic drug phenformin causes lactic acidosis

The related agent metformin is under suspicion for the same toxicity.

Large observational studies have failed to demonstrate an association

Clinicians would be more alert to lactic acidosis in the presence of the agent

Factors that raise quality

Summary of GRADE framework for rating the quality of evidence

39

Quality of evidence

Study design Lower if Higher if

High Randomised trial Study quality: Serious limitations Very serious limitations I mportant inconsistency Directness: Some uncertainty Major uncertainty Sparse or imprecise data High probability of reporting bias

Strong association: Strong, no plausible confounders Very strong, no major threats to validity Evidence of a Dose response gradient All plausible confounders would have reduced the eff ect

Moderate

Low Observational study

Very low

Evidence Profiles and Summary of Findings (SoF) Tables summarize the rating of the quality of evidence across selected outcomes

41

42

43

Applying the GRADE approach

Exercise: parenteral anticoagulation for prolonging the survival of patients with cancer

Wrap-up

46

Advantages of GRADE

Developed by a widely representative group of international guideline developers

Clear separation between quality of evidence and strength of recommendations

Explicit evaluation of the importance of outcomes

Explicit, comprehensive criteria for downgrading and upgrading quality of evidence ratings

47

Advantages of GRADE

Transparent process of moving from evidence to recommendations

Explicit acknowledgment of values and preferences

Clear, pragmatic interpretation of strong versus weak recommendations for clinicians, patients, and policy makers

Useful for systematic reviews and health technology assessments, as well as guidelines

48

Disadvantages of GRADE

Involves a number of judgments that might affect its reliability

Requires expertise/training

49

Session evaluation

Thank you!