Introduction to Critical Appraisal :

Quantitative Research

South East London

Outreach Librarians

January 2008

Learning objectives• Understand the principles of critical

appraisal and its role in evidence based practice

• Be able to appraise quantitative research and judge its validity

• Be able to assess the relevance of published research to your own work

Daily Mail exercise• Would you treat a patient based on this

article? Why?

• Validity

• Reliability

• Transferable to practice

What is evidence based practice?

Evidence-based practice is the integration of • individual clinical expertise

with the • best available external clinical evidence

from systematic researchand

• patient’s values and expectations

The evidence-based practice process.

• Decision or question arising from a patient’s care.

• Formulate a focused question.

• Search for the best evidence.

• Appraise the evidence.

• Apply the evidence.

Why does evidence from research fail to get into practice?

• 75% cannot understand the statistics

• 70% cannot critically appraise a research paper

• Using research for Practice: a UK experience of the barriers scale.

Dunn, V. et al.

What is critical appraisal?

• Weighing up evidence to see how useful it is in decision making

• Balanced assessment of benefits and strengths of research against its flaws and weaknesses

• Assess research process and results• Skill that needs to be practiced by all

health professionals as part of their work

What critical appraisal is NOT• Negative dismissal of any piece of

research

• Assessment of results alone

• Based entirely on statistical analysis

• Only to be undertaken by researchers/ statisticians

Why do we need to critically appraise?

• “It usually comes as a surprise to students to learn that some (the purists would say 99% of) published articles belong in the bin and should not be used to inform practice” (Greenhalgh 2001)

• Find that 1% - save time and avoid information overload

How do I appraise?• Mostly common sense.• You don’t have to be a statistical expert!• Checklists help you focus on the most

important aspects of the article.• Different checklists for different types of

research.• Will help you decide if research is valid

and relevant.

Research methodsQuantitative • Uses numbers to

describe and analyse• Useful for finding

precise answers to defined questions

Qualitative• Uses words to

describe and analyse• Useful for finding

detailed information about people’s perceptions and attitudes

Levels of quantitative evidence.

(In order of decreasing scientific validity.)

• Systematic reviews.• Randomized controlled trials.• Prospective studies (cohort studies).• Retrospective studies (case control).• Case series and reports• Opinions of respected authorities.

Systematic Reviews.• Thorough search of literature carried

out.• All RCTs (or other studies) on a

similar subject synthesised and summarised.

• Meta-analysis to combine statistical findings of similar studies.

Randomised Controlled Trials (RCTs)

• Normal treatment/placebo versus new treatment.

• Participants are randomised.

• If possible should be double-blinded.

• Intention to treat analysis

Cohort studies

• prospective• groups (cohorts) • exposure to a risk factor • followed over a period of time• compare rates of development of an

outcome of interest • Confounding factors and bias

Case control studies

• Retrospective• Subjects confirmed with a disease

(cases) are compared with non-diseased subjects (controls) in relation to possible past exposure to a risk factor.

• Confounding factors and bias

Appraising original research

Are the results valid?• Is the research question focused?• Was the method appropriate?• How was it conducted, e.g. randomisation,

blinding, recruitment and follow up?What are the results?• How was data collected and analysed?• Are they significant?Will the results help my work with patients?

Appraising systematic reviews.

In addition to the above:• Was a thorough literature search

carried out ?• Publication bias - papers with more

‘interesting’ results are more likely to be:– Submitted for publication– Accepted for publication– Published in a major journal– Published in the English language

Reviews in general medical journals

• 50 reviews in 4 major journals 1985-6• No statement of methods• Summary inappropriate

• “Current systematic reviews do not routinely use scientific methods to identify, assess and synthesise information” (Mulrow, 1987)

Is the research question focused?

• Patient (e.g. child) • Intervention (e.g. MMR vaccine)

• Comparison (e.g. single vaccines)

• Outcome (e.g. autism)

Are results significant?

• How was data collected?

• Which statistical analyses were used?

• How precise are the results?

• How are the results presented?

Intention to treat analyses

• Analysing people, at the end of the trial, in the groups to which they were randomised, even if they did not receive the intended intervention

Statistical analyses

Odds ratios, absolute and relative risks/benefits

• The likelihood of something happening vs the likelihood of something not happening

Numbers needed to treat (NNT)• The number of people you would need to

treat to see one additional occurrence of a specific beneficial outcome

Odds Ratio Diagrams. (Blobbograms or Forest Plots.)

Odds Ratio Diagrams

• Line of no effect – no difference between treatment and control group

• Result (blob) to the Left of the line of no effect = Less of the outcome in the treatment group.

• Result to the Right of the line = More of the outcome.

• BUT - Is the outcome good or bad?

Cardiac deaths – Less = good

Smoking cessation – More = good

Confidence Intervals.• Longer confidence interval = less

confident of results – wider range.

• Shorter confidence interval = more confident – narrower range.

• Crosses line of no effect/no significance = Inconclusive results.

Confidence intervals

P Values.• P stands for probability - how likely is the

result to have occurred by chance?• P value of less than 0.05 means likelihood

of results being due to chance is less than 1 in 20 = “statistically significant”.

• P values and confidence intervals should be consistent

Number Needed to Treat

• The number of people you would need to treat to see one additional occurrence of a specific beneficial outcome.

• The number of patients that need to be treated to prevent one bad outcome.

• The NNT can be calculated by finding the Absolute Risk Reduction (ARR)

Events or outcomes are used for reporting results. The event rate is the proportion of patients in a group in whom the event is observed

Outcome event Total

Yes No

Experimental group a b a + b

Control group c d c + d

Total a + c b + d a + b + c + d

CER and EER

• Control Event Rate (CER) is the proportion of patients in the control group in whom an event is observed.

CER = c/(c+d)

• Experimental Event Rate (EER) is the proportion of patients in the experimental in whom an event is observed.

EER = a/(a+b)

AAR & NNT

• Absolute Risk Reduction is the difference between the Control Event Rate (CER) and the Experimental Event Rate (EER).

ARR = CER – EER

• Number needed to treat (NNT)

NNT = 1/ARR

 

Outcome event Total

Yes No

Experimentalgroup

3 7 10

Control group 5 5 10

Total 8 12 20

Answers

• What is the event ? • Lack of concentration and sleeping• What is the control event rate (CER)? • 5/10 = 0.50• What is the experimental event rate (EER)?• 3/10 = 0.30• Calculate the absolute risk reduction (ARR) • 0.50 – 0.30 = 0.20 • What is the number needed to treat (NNT)? • 1.00/0.20 = 5

Are results relevant?• Can I apply these results to my own

practice?• Is my local setting significantly different?• Are these findings applicable to my

patients?• Are findings specific/detailed enough to be

applied?• Were all outcomes considered?

The good news!

• Some resources have already been critically appraised for you.

• An increasing number of guidelines and summaries of appraised evidence are available on the internet.

Summary.

• Search for resources that have already been appraised first, e.g. Guidelines, Cochrane systematic reviews.

• Search down through levels of evidence, e.g. systematic reviews, RCTs.

• Use checklists to appraise research.• How can these results be put into practice?