critical appraisal systematic r eview
DESCRIPTION
Critical appraisal Systematic R eview. กิตติพันธุ์ ฤกษ์ เกษม ภาควิชาศัลยศาสตร์ มหาวิทยาลัยเชียงใหม่. Systematic review vs Meta-analysis. Systematic review: a systematic approach to minimising bias and error - PowerPoint PPT PresentationTRANSCRIPT
Critical appraisal Systematic Review
กิ�ตต�พั�นธุ์� ฤกิษ์เกิษ์มภาควิ�ชาศั�ลยศัาสตร์
มหาวิ�ทยาล�ยเช�ยงใหม�
Systematic review vs Meta-analysis
• Systematic review: a systematic approach to minimising bias and error
• Meta-analysis: a statistical analysis, which aim to produce a single estimate of a treatment effect
• Systematic review may or may not include Meta-analysis
Why do we need systematic review?
• The early 1980s uncomplicated MI
“Should pt receive a beta-blocker for secondary prevention before discharge?”
Library: 4 randomised controlled trials (RCT)
Beta-blocker vs placebo
RCT 1.Mortality and hospital readmission is not different
RCT 2. Not conclusiveRCT 3. Beta-blocker not shown benefitRCT 4. Long term beta-blocker reduces the
mortality and rate of re-infarction
A review in BMJ 1981
• There is no clear evidence that beta-blocker improves long tem survival after MI despite almost 20 yrs of clinical trials
• Good enough!!
Another review in European Heart Journal 1981
“it seems perfectly reasonable to treat patients who have survived an infarction with beta-blocker”
Limitation of a single study
• Too small sample size false negative
Problem of Conventional review
• Prone to bias and error
– Select only evidence support the author’s view– Not specify methodological quality of studies– Finally choose most vote ignore sample size,
and design
Meta-analysis = combining all available data
• Attractive alternative to such large, expensive and problematic study
• Weight average of the result
large > small trial
Meta-analysis Beta-blocker trials - MI
Beta-blocker better Placebo better
Cumulative meta-analysis of beta-blocker trials
Cumulative meta-analysis
Significant effect from 1980 onwards (OR not across 1
Meta-analysis Beta-blocker trials - MI
Beta-blocker better Placebo better
Maybe Unnecessary trials
Benefit
• Estimate the overall effect
• Examine different result between studies (heterogeneity)
• Identified insufficient data
Cochrane collaboration• International organisation of
health care profession
• Promoting accessibility of systematic review
• Foster development of systematic review
• 50 collaborative review groups
www.cochrane.org/cochrane/ccweb.htm
Potentials of systematic review
• Good
• Bad
Systematic review Basic structure and types
กิ�ตต�พั�นธุ์� ฤกิษ์เกิษ์มภาควิ�ชาศั�ลยศัาสตร์
มหาวิ�ทยาล�ยเช�ยงใหม�
1. Basic structure
Like primary research
• Why- Introduction, background
• How-method
• What we found-result
• What it mean-discussion
Basic structure• Abstract • Introduction
– Background– Objectives
• Method “treat a paper like a patient in 1 reseach” – Type of studies– Inclusion criteria type of participants– Exclusion criteria– Type of intervention– Type of outcome measures– Search strategy for identification of study– Method of analysis
Basic structure
• Result (special diagram)• Conclusion• Reference
Odds ratio.05 .5 1 2
% Weigh t Odd s ra tio (95% CI)
0.32 (0.01,8.06) Foresell et al. 1989 23.2
0.13 (0.01,2.53) Sbarigia et al.1999 55.1
0.37 (0.01,9.17) Kasprzak et al. 1999 21.8
Pluskwa et al. 1989 0.0 Not estimable
Prough et al. 1989 0.0 Not estimable
Binder et al. 1999 0.0 Not estimable
0.22 (0.04,1.33) Overall (95% CI )
Black square = OR, horizontal line = 95% confidence interval
Area of black square = weight, diamond = combined OR with 95% CI
LA better GA better
Forest plot Stroke rate ‘LA vs GA in carotid sx trial’ 1966-2001
2. types
Types
• Systematic review of primary research– Observational studies– Diagnostic screening– RCT
3. Method “treat a paper like a patient in 1 reseach”
The process (1)
• Research question• Writing protocol• Searching• Article retrieval• Literature review
The process (2)
• Inclusion/ exclusion criteria• Validity and quality of articles• Data extraction/ synthesis• Interpretation
The question
• Is local anesthesia is better than general anesthesia during carotid endarterectomy?
Writing the protocol• Background• Objectives• Type of studies• Inclusion criteria• Type of participants• Exclusion criteria• Type of intervention• Type of outcome measures• Search strategy for identification of study• Method of analysis• Reference
Searching
• Medline• Other database• Hand searching the literature• Writing to people
Getting the article
• Which ones to get?• It takes time• Libraries• Inter-library Loans
Literature review
• You don’t have to read the whole paper yet!
• Translation
Validity and quality of articles
• Do read the paper and see what the author thought was wrong
• Unequal intervention/control size
• Hidden loss to follow up
Data extraction
• Read method carefully
• Design a form
Synthesis/ Interpretation
• Estimates and confidence intervals
pool effect make by statistic method e.g. Peto method (fix method) give more weight effect for large study than small study (P value)
• Difference between studies (Heterogeneity) Chi-squared test (P value)
Small RCTs show LA is marginal lower mortality than GA
Study or Subgroup
Binder 1999Forssell 1989Kasprzak 2006McCarthy 2001aPluskwa 1989Prough 1989Sbarigia 1999
Total (95% CI)
Total eventsHeterogeneity: Chi² = 1.44, df = 3 (P = 0.70); I² = 0%Test for overall effect: Z = 1.94 (P = 0.05)
Events
0001000
1
Total
27569134101355
286
Events
0111003
6
Total
19559533101052
274
Weight
14.5%14.5%28.5%
42.6%
100.0%
Peto, Fixed, 95% CI
Not estimable0.13 [0.00, 6.70]0.14 [0.00, 7.12]
0.97 [0.06, 15.85]Not estimableNot estimable
0.12 [0.01, 1.21]
0.23 [0.05, 1.01]
Local General Peto Odds Ratio Peto Odds RatioPeto, Fixed, 95% CI
0.10.2 0.5 1 2 5 10Local better General better
Critical Appraisal
1. Are the result valid?
2. What are the results?
1. Are the result valid?
• Did this review address a sensible clinical question
• Was the search for relavant studies detailed and exhaustive?
• Were selection and assessment of studies reproducible?
• Were the primary studies of high methodological quality?
Publication bias
“A (significant) beneficial treatment effect are published, but an equal result remain unpublished”
• In general medical journal and public heath journal reported statistically significant 85.4%
• In psychological journal 95.6%
Time lag bias
• “Positive result will dominate the literature for several year until the negative will report later”
• HIV trial in USA, median time to publish of positive result 4.2 years, but negative result 6.4 years
Duplicate publication bias• “ one study presents and reports
several times” “ include this lead to overestimation of treatment effect”
• Ondersetron to prevent postoperative nausea vomitting 16 studies
3 duplicated papers
• Sometimes difficult to say, since not share single common authors !!!!!!
Language bias
• “Authors tend to report positive result in international papers, English language journal, but if negative result are published in local journal”
Outcome reporting bias
• In trials many outcome is recorded but only favorable finding will be reported
• Clinical trials by drug companies, unpublished trials gave information on adverse effect > published trials
Selection Bias
• Tend to happen in non Randomised controlled trials (non RCT)
• For example select low risk group to new treatment group
2. What are the result?
• Were the results similar from study to study? If yes, the credit of single estimates is OK.– Point estimates similar?– Overlapping confidence interval– Test for heterogeneity? (Chi square test)– Percentage of variability (I2 ) good < 20%,
concern 20-50%,serious concern > 50%
Look overlapping confidence
interval
Study or Subgroup
Agrifoglio 1987Allen 1994Andersen 1980Bartoloni 1991Becquemin 1991Bowyer 2000Brown 1999Calligaro 2001Corson 1987ECST 1998Gabelman 1983Ghali 1997Godin 1989Gurer 2003Harbaugh 2000Hartsell 1999Imparato 1998Jopling 1983Kalko 2007Kraiss 1995Kucey 1998Love 2000Marrocco 2004McCarthy 2001bMcCleary 1996Mertens 2000Mofidi 2006Muskett 1986NASCET 1998Ombrellaro 1996Palmer 1989Quigley 2000Reina 1998Rignano 1999Santamaria 2004Schwartz 1988Shah 1994Sternbach 2002Stone 2000Stoughton 1998Syrek 1999Taylor 1999
Total (95% CI)
Total eventsHeterogeneity: Chi² = 33.76, df = 30 (P = 0.29); I² = 11%Test for overall effect: Z = 2.65 (P = 0.008)
Events
0260120310000122
470200002101000203000520012
88
Total
2927417969
13227215718514558483350
200632116
338250
30618
18520050
1003222
19230
10414018412611676
20314965422667
15083
140
9564
Events
241062043
17100201
12031
233000030
14001312231000
16
131
Total
2831015275
220228132216223
1681405750
16517111859330
127178
109624363
1403372
17945
13101263778
18816956
162219324615826
1152
10631
Weight
1.5%4.4%5.1%
4.8%3.0%
5.1%2.8%1.6%0.7%
2.2%1.0%2.2%
22.2%
3.1%0.3%8.4%2.2%
1.5%0.8%
3.0%
2.8%
0.8%0.7%4.2%0.6%1.0%1.5%4.5%2.2%
0.5%5.2%
100.0%
Peto, Fixed, 95% CI
0.13 [0.01, 2.06]0.58 [0.12, 2.89]
3.66 [0.82, 16.41]Not estimable
0.37 [0.08, 1.71]0.84 [0.12, 6.02]
Not estimable0.87 [0.20, 3.91]0.54 [0.07, 4.08]0.35 [0.02, 5.03]0.11 [0.00, 5.68]
Not estimableNot estimable
0.42 [0.04, 4.09]3.57 [0.12, 105.59]1.99 [0.21, 19.35]0.65 [0.32, 1.33]
Not estimable0.22 [0.03, 1.55]
0.33 [0.00, 294.58]0.30 [0.09, 0.98]0.16 [0.02, 1.56]
Not estimable11.13 [0.67, 186.25]7.62 [0.15, 384.38]
Not estimable0.34 [0.05, 2.43]
Not estimable0.34 [0.04, 2.53]
Not estimable3.34 [0.08, 137.98]
0.07 [0.00, 4.13]1.67 [0.32, 8.77]
0.23 [0.00, 16.24]0.01 [0.00, 0.28]0.15 [0.01, 2.35]0.51 [0.10, 2.56]
2.89 [0.29, 28.92]Not estimableNot estimable
3.72 [0.04, 369.48]1.03 [0.23, 4.60]
0.63 [0.45, 0.89]
Local General Peto Odds Ratio Peto Odds RatioPeto, Fixed, 95% CI
0.10.2 0.5 1 2 5 10Local better General better
Rerkasem Cochrane Database Syst Rev 2008; (4):CD000126.
Small RCTs show LA is marginal lower mortality than GA
Study or Subgroup
Binder 1999Forssell 1989Kasprzak 2006McCarthy 2001aPluskwa 1989Prough 1989Sbarigia 1999
Total (95% CI)
Total eventsHeterogeneity: Chi² = 1.44, df = 3 (P = 0.70); I² = 0%Test for overall effect: Z = 1.94 (P = 0.05)
Events
0001000
1
Total
27569134101355
286
Events
0111003
6
Total
19559533101052
274
Weight
14.5%14.5%28.5%
42.6%
100.0%
Peto, Fixed, 95% CI
Not estimable0.13 [0.00, 6.70]0.14 [0.00, 7.12]
0.97 [0.06, 15.85]Not estimableNot estimable
0.12 [0.01, 1.21]
0.23 [0.05, 1.01]
Local General Peto Odds Ratio Peto Odds RatioPeto, Fixed, 95% CI
0.10.2 0.5 1 2 5 10Local better General better
Rerkasem Cochrane Database Syst Rev 2008 (4):CD000126.
2. What are the result?
• Were the results similar from study to study?
• What are the overall results of the reviews?
• How precise were the results?