HOW TO APPRAISE A PAPER

CRITICALLY

The science of ‘trashing’ a paper

Unimportant issue

UnoriginalHypothesis not tested

Different type of study required

Compromised original protocol

Sample size too smallPoor statistics

Unjustified conclusion

Conflict of interest

Badly written

Why should critically Appraise a paper?

1/25/2010 3 So, critically Appraise What You Read

What is “ best evidence”?• You need to use evidence from research studies

to support your arguments and decisions in caring for your patients

• One tool to assist you in this is critical appraisal• Using critical appraisal skills you can understand

the methods and results of a study and then be able to assess the quality of the study.

1/25/2010 4

The Hierarchy of Evidence1. Systematic reviews & meta-analyses

2. Randomised controlled trials

3. Cohort studies

4. Case-control studies

5. Cross sectional surveys

6. Case reports

7. Expert opinion

8. Anecdote

The quality of a study depends on

Extrinsic factors: Journal Author Institution Quoted by other

authors Quoted in

newspapers

Intrinsic factors: Study design Number of participants Confounders Author bias

1/25/2010 6

Critical appraisal

Critical appraisal looks at the intrinsic factors

1/25/2010 7

“Critical appraisal is - the process of systematically examining

research evidence to assess its validity, reliability and relevance before using it to inform a decision.”( Hill H & Spittlehouse C, EBM 2001; 3:2)

- the assessment of evidence by systematically reviewing its relevance, validity and results to specific situations. - Chambers, R. (1998).

Key Steps to Effective Critical Appraisal

1. Are the results valid? ( Validity)2. What are the results? ( Important)3. How will these results be relevant to the patient? ( Applicability)

V.I.A 1/25/2010 8

Validity, reliability, relevance

Validity: To what extent is the study a close representation of the “truth”?

Reliability: Are the results credible and (important) repeatable?

Relevance: Will the results help me in my own(applicability) study or practice?1/25/2010 9

Validity and Reliability

• A test is valid when it measures what it’s supposed to.

• If a test is reliable, it yields consistent results. • A test can be both reliable and valid, one or

the other, or neither. • Reliability is a prerequisite for measurement

validity.

1/25/2010 10

Reliable, but Not Valid!

1/25/2010 11

Not Reliable, Not Valid!

1/25/2010 12

Reliable and Valid

1/25/2010 13

Mark Newman - Middlesex University updated 04/20011/25/2010 14

Steps to EBM1. Formulate an answerable focused clinical question - “ PICO model “2. Search the literature for “the best external evidence”3. Critically appraise the evidence for its validity and usefulness/important4. Implement the useful evidence in clinical practice5. Evaluate the results

1/25/2010 15

Critical appraisal questions

• What is the paper about? • Why was the study done? • What type of study was done? • Was it primary research (experiment, RCT, cohort, case-control, cross-

sectional, longitudinal, case report /series)?

1/25/2010 16

Critical appraisal questions• Is it of interest?

– Title, abstract, source.• Why was it done?

– Introduction.• Useful references?• Important or novel ideas?• Should end with a clear statement of the purpose of the

study.• The absence of such a statement can imply that the

authors had no clear idea of what they were trying to find out.

• Or they didn’t find anything but wanted to publish!

1/25/2010 17

Critical appraisal questions• How was it done?

– METHODS.• Brief but should include enough detail to enable one to

judge quality.• Must include who was studied and how they were

recruited.• Basic demographics must be there.• What data was collected?• Have areas of potential bias been adequately addressed?• Is sample size calculation provided?• Are assumptions reasonable / referenced?• Is study design appropriate to address hypotheses?• Was the study ethical?

1/25/2010 18

                                                            

Specific types of study

Randomised Control Trials (RCT’s)Studies in which there are two groups (one treatment – one control). Patients are randomly assigned to both groups to reduce the risk of bias and increase the probability that differences between the groups can be attributed to the treatment. Factors that may lead to bias are called ‘confounders’.

Cohort StudiesStudies in which patients (e.g. a group of smokers) are followed over time and compared with a control group (non-smokers) to see what health problems they develop.

Critical appraisal questionsIs study design appropriate to address hypotheses?

Systematic reviewsComprehensive surveys of a topic – locating, appraising and synthesising evidence from as many relevant scientific studies as possible. They summarise conclusions about effectiveness and provide a unique collation of the known evidence on a given topic. Meta AnalysesQuantitative and statistical studies in which the results are similar enough statistically to be combined and analyzed as if it was one study.Case Control StudiesStudies in which patients who already have a certain condition (e.g.lung cancer could be asked how much they smoked in the past), and the answers are compared with a sample of the general population

Critical appraisal questionsIs study design appropriate to address hypotheses?

1/25/2010 22

Critical appraisal questionsIs study design appropriate to address hypotheses?

Observational StudiesCollective name given to studies (usually cohort or case-control) in which nature is allowed to take its’ course, seeking to find an association between one factor (e.g. being overweight) and another (e.g. dying from heart disease).

Case SeriesCollections of reports on the treatment of individual patients, or of reports on a single patient – no statistical validity because they do not use a control group with which to compare the outcomes

• What has it found? – RESULTS.

• The data should be there – not just statistics.• Are the aims in the introduction addressed in the results?• Are the results clearly presented?• Are ALL the results presented?• How have statistical results been presented?• Are p-values or confidence intervals most appropriate?• Are the results correct?• Can the reader draw their own conclusions from the

results?

1/25/2010 23

Critical appraisal questions

• DISCUSSION– What does it mean?– Effect of methodological limitations– Are the results put into context of previous literature?

• Is there any previous literature?– Are similarities / differences with previous findings discussed?– Is the impact of methodological limitations discussed?– What does it mean for health care?– Is it relevant to my patients?

1/25/2010 24

Critical appraisal questions

What Makes a study “Good”• Good research

– Appropriate study design– Well executed procedures– Appropriate analysis– Adequate sample size– . . . and more

• Discussion of failings– Methodological imperfections– Potential Bias / Confounding / etc– Effect that these may have on results and conclusions

1/25/2010 25

Clinical Scenario

Can furosemide be used for rate control of atrial fibrillation (AF) in a patient with congestive heart failure (CHF)?

1/25/2010 26

Getting Started1. Formulate research (PICO) question

Population: Patients with atrial fibrillation and CHF Intervention: furosemide Comparison: placebo Outcome: Mortality, effectiveness of rate control

2. Search appropriate sources3. Find paper (s)4. Read to suit you!5. Appraise the paper (s)

1/25/2010 27

Patient with Atrial Fibrillation in CHF ( P)

Comparison ( C )

PICO Schematic Model Treatment

furosemide

Effectiveness of rate control

( O )

Effectiveness of rate control

( O )

Placebo Placebo

1/25/2010 28

Clinical problem

Define the search question

Choose a resource/database

Create a search strategy

Summarize the evidence

Apply the evidence

Try another relevant resource

Create a search strategy

Summarize the evidence

Try another relevant resource

General Search Strategy

Adapted from: Sackett, D. et al. 2000. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd Edition. Toronto: Churchill Livingstone.

Pooryield

1/25/2010 29

Study Types for Question TypesDiagnosis Prospective cohort study with

good quality validation against “Gold Standard”

Therapy Randomized controlled clinical trial (RCT)

Etiology/Harm

RCT, cohort or case-control study (probably retrospective)

Risk factors/ Prognosis

Prospective cohort study

1/25/2010 30

How are the importance of the study ?

IMPORTANCE of the study Results How large was the treatment effect

- Relative risk (RR) - Absolute risk reduction (ARR)- Relative risk reduction (RRR)- An odds ratio- Needed number to treat (NNT)

How precise was the estimate of the treatment effect (relates to statistical significance)

1/25/2010 31

Absolute Risk Reduction

• Consider a study in which – 20% of a control group died

– but only 15% of the treatment group did

• The absolute risk reduction would be:

20% - 15% = 5%

• NOTE: An absolute risk reduction of 0% would indicate that there is no difference between the groups.

1/25/2010 32

Risk Ratios• The relative risk is the risk of events among

intervention patients as a proportion of the risk among patients in the control group

• For the same study the relative risk would be:15% / 20% = 75% or 0.75

• Therefore the relative risk reduction would be:100% - 75% = 25% or 0.25

• NOTE: A relative risk of 100% or 1 would indicate that there is no difference between the groups

1/25/2010 33

Odds Ratios Odds of an event = no. of events / no. of non-events e.g. 51 boys/100 births Odds of boy = 51/49 = 1.04 Odds >1 means event more likely to happen than notOdds of an impossibility are zeroOdds of a certainty are infinity

Odds ratio = odds in intervention group / odds in control group

04/11/23 34

Odds Ratios

When events are rare Odds (OR) and Risk (RR) are similar

As prevalence (control event rate/CER) and OR increase, the error in using OR as an approximation for RR becomes unacceptable.

04/11/23 35

Problem with relative risk (A)Died Survived Total Risk of Dying

Treatment 20 30 50 20 / 50 = 0.4

Control 30 20 50 30 / 50 = 0.6

Total 50 50 100

1/25/2010 36

• Absolute Risk Reduction = 0.6 - 0.4 = 0.2 = 20%• Relative Risk = 0.4 / 0.6 = 0.67 = 67%• Relative Risk Reduction = 1 – 0.67 = 0.33 = 33%

Problem with relative risk (B)

Died Survived Total Risk of Dying

Treatment 20 4980 5000 20 / 4980 = 0.004

Control 30 4970 5000 30 / 4970 = 0.006

Total 50 9950 10000

1/25/2010 37

• Absolute Risk Reduction = 0.006 - 0.004 = 0.002• Relative Risk = 0.004 / 0.006 = 0.67 = 67%• Relative Risk Reduction = 1 – 0.67 = 0.33 = 33%• RRR is the same even though this drug is rubbish!

Number Needed to Treat

• Is based on absolute figures rather than relative

• Can be easier to understand

• “The number of patients who must receive the intervention to produce one additional positive outcome”

• NNT = 1 / Absolute Risk Reduction

• For study A, NNT = 1 / 0.2 = 5

• For study B, NNT = 1 / 0.002 = 5001/25/2010 38

Applying the results to your patient• Estimate your patient’s risk relative to the average

participant of the study, as a fraction ‘f’• You may think that your patient is 50% more likely

to die without therapy than the people in the study so f = 1.5

• The NNT for your patient will beNNT / f

• For study A your patient’s NNT is:5 / 1.5 = 3.3

1/25/2010 39

P value and Confidence Intervals

Two methods of assessing role of chance P values (testing the null hypothesis)

If P < 0.05 then the difference between the 2 groups is statistically significant

Confidence Intervals (CI) Estimate the range of values (of RR, ARR, RRR)

and likely to include the true value If the value corresponding to no effect I.e. RR of 1

or ARR or RRR of 0, falls outside the 95% CI then the result is statistically significant.

1/25/2010 40

How the precise of the result?

• p-values aims at measuring the precision of the results

• The p-value is calculated using statistical tests that take into account the study design, the variation within the data, and the sample size.

• a p-value of less than 0.05 is sufficient to say that the observed difference indicates an actual difference or statistical significant.

1/25/2010 41

How much confidence do we have?• CONFIDENCE INTERVALS

• telling how close your observed result is to the actual result.

• CI can tell whether a result is significant ( statistical and clinical significant) but p-value only statistical significant)

• A 95% Confidence Interval of 0.1 – 2.0 tells that if you ran the trial again, there is a 95% chance getting a result somewhere between 0.1 and 2.0

• Or if you conducted the trial 100 times you would get a result between 0.1 and 2.0 in 95 of those trials.

1/25/2010 42

Four possible outcomes from any study

1. Difference is clinically important and statistically significant i.e. important and real.

2. Of clinical importance but not statistically significant. sample size too small.

3. Statistically significant but not clinically important i.e. not clinically meaningful.

4. Neither clinically important nor statistically significant.

APPLICABILITY Applicability of the study Discussion my patient so different from those in the

study that the results cannot apply?- Inclusion and exclusion criteria

Is the treatment feasible in my setting- Cost - Availability

Will the potential benefits of treatment be outweighed by harms

1/25/2010 44

Case scenario

• 76 year old male admitted to your clinics due to acute diarrhea for one week.

• Medication such as kaopectine, imodium had been given but in vain

• After consultation with a internist, the probiotic was prescribed.

• Diarrhea improved after 2-3 days.

1/25/2010 45

1/25/2010 46

1/25/2010 47

EBM

1/25/2010 48

How can we do Critical Appraisal?

• Use common sense• Use simple checklists• Use different checklists depending on the

different types of studies (i.e., RCTS, systematic reviews etc)

• Checklists help you focus on the important parts of the article

1/25/2010 49

How to Read a Research Article? • Purpose of reading the paper.• Do not read the abstract.• Read the title, find out who the authors are and where they

work, look for sources of funding and conflicts of interest• Look at the tables and figures• Is there a diagram to show the flow of participants through

each stage of the study • Why was the study done and what hypothesis were the authors

testing?• What is broadly the topic of research?• Therapy, Diagnosis, Screening, Prognosis, Causation?• What type of study was done?• Is this a primary (experimental, clinical trial, survey) or a

secondary paper (review, meta-analysis, guideline, economic analysis)?

• Was the study design appropriate?• Now proceed to a critical appraisal of the paper

1/25/2010 50

TELAAH KRITIS JURNAL

04/11/23 51

No HAL YANG

DINILAI CHECK LIST PENILAIAN YA TIDAK

1 Judul Makalah

a. Apakah judul tidak terlalu panjang atau terlalu pendek?

b. Apakah judul menggambarkan isi utama penilaian?

c. Apakah judul cukup menarik?d. Apakah judul menggunakan

singkatan selain yang baku?

2 Abstrak a. Apakah merupakan abstrak satu paragraf, atau abstrak terstruktur?

b. Apakah sudah tercakup komponen IMRAC (Introduction, methods, Results, Conclusion?)

c. Apakah secara keseluruhan abstrak informatif?

d. Apakah abstrak lebih dari 250 kata?

Terstruktur

ü

3 Pendahuluan a. Apakah mengemukakan alasan dilakukannya penelitian?

b. Apakah menyatakan hipotesis atau tujuan penelitian?

c. Apakah pendahuluan didukung oleh pustaka yang kuat & relevan?

(

04/11/23 52

4 Metode a. Apakah disebutkan desain, tempat & waktu penelitian?

b. Apakah disebutkan populasi sumber (populasi terjangkau)?

c. Apakah kriteria pemilihan (inklusi & eksklusi) dijelaskan?

d. Apakah cara pemilihan subjek (teknik sampling) disebutkan?

e. Apakah perkiraan besar sampel disebutkan & disebut pula alasannya?

f. Apakah perkiraan besar sampel dihitung dengant rumus yang sesuai?

g. Apakah observasi, pengukuran, serta intervensi dirinci sehingga orang lain dapat mengulanginya?

h. Bila teknik pengukuran tidak dirinci, apakah disebutkan rujukannya?

i. Apakah definisi istilah & variabel penting dikemukakan?

j. Apakah ethical clearance diperoleh?

k. Apakah disebutkan rencana analisis, batas kemaknaan & power penelitian?

04/11/23 53

5 Hasil a. Apakah disertakan tabel deskripsi subjek penelitian?

b. Apakah karakteristik subjek yang penting (data awal) dibandingkan kesetaraannya?

c. Apakah dilakukan uji hipotesis untuk kesetaraan ini?

d. Apakah disebutkan jumlah subjek yang diteliti?

e. Apakah dijelaskan subyek yang drop out dengan alasannya?

f. Apakah semua hasil di dalam tabel disebutkan dalam naskah?

g. Apakah semua outcome yang penting disebutkan dalam hasil?

h. Apakah subyek yang drop out diikutkan dalam analisis?

i. Apakah disertakan hasil uji statistik (x2,t) derajat kebebasan (degree of freedom), dan nilai p?

j. Apakah dalam hasil disertakan komentar & pendapat?

04/11/23 54

6 Diskusi a. Apakah semua hal yang relevan dibahas?b. Apakah dibahas keterbatasan penelitian, dan

kemungkinan dampaknya terhadap hasil?

c. Apakah disebutkan kesulitan penelitian, penyimpangan dari protokol, dan kemungkinan dampaknya terhadap hasil?

d. Apakah pembahasan dilakukan dengan meghubungkannya dengan teori dan hasil penelitian terdahulu?

e. Apakah dibahas hubungan hasil dengan praktek klinis?

f. Apakah disertakan kesimpulan utama penelitian?

g. Apakah kesimpulan didasarkan pada data penelitian?

h. Apakah efek samping dikemukakan dan dibahas?

i. Apakah disebutkan hasil tambahan selama diobservasi?

j. Apakah disebutkan generalisasi hasil penelitian?

k. Apakah disertakan saran penelitian selanjutnya, dengan anjuran metodologis yang tpat?

04/11/23 55

04/11/23 56

KHUSUS - EBMMenilai VIA (Validity, Important, Aplicability)

Validitas 1. Apakah awal penelitian didefinisikan dengan jelas?

Ya, awal penelitian dan tujuan penelitian dijelaskan secara rinci.

2. Apakah dinyatakan desain penelitian dengan jelas ?

3. Apakah ada pembanding yang jelas ?

04/11/23 57

4. Apakah pemantauan pasien dilakukan cukup panjang dan lengkap ? 5. Apakah ada identifikasi jelas kelompok dengan prognostik yang berbeda ?

6. Apakah outcome dinilai dengan kriteria objektif?

Important

1.Apakah outcome/hasil dipaparkan secara jelas (hasil uji statistic dengan hasil nilai P)?

2. Seberapa besarkah ketepatan estimasi outcome yang didapat dengan nilai OR,RR,PR dengan nilai korelasi 95% CI ?

04/11/23 58

1. Apakah pasien kita mirip dengan subjek yang diteliti?

2. Apakah bukti ini akan mempunyai pengaruh yang penting secara klinis terhadap kesembuhan pasien kita tentang apa yang telah ditawarkan/diberikan kepada pasien kita?

•  

Aplicability

04/11/23 59