critical appraisal overview
DESCRIPTION
An overview of critical appraisal in medicine. Talk by Elspeth Hill and James Durrand as part of the Fastbleep Academic Masterclasses 2011.TRANSCRIPT
Tues 10th May
Critical Appraisal
Master class
Critical Appraisal: WTF?
(well that’s fab)
Introductions
Elspeth Hill• MRes Medical Sciences• PhD Education Student• Journal Reviewer• Cochrane Author• Nervous wreck
James Durrand• BSc Pathology• MB ChB (nearly!)• Academic Foundation Post• RSM Rep• Massive geek
You?
Outline
1. Evidence-based medicine
2. Principles of critical appraisal
3. Jargon busting
4. Clinical studies
5. Lab stuff
6. Qualitative work
Evidence-Based Medicine
What is it?
Structure of EBM
Question
•What do I need to know?
Evidence
•What is known about this topic?
•What is in the literature?
Critical Ap
praisal
•Evaluation of the evidence
•What do we actually know?
Applicatio
n
•Try your suggestion based on evidence
Monitor
•How did that work out?
Critical Appraisal
Should I believe this paper?
ClinicalShould I treat my patients differently based on this paper?
EducationShould we include this in the medical school curriculum?
Lab ScienceShould we divert resources to investigate this further?
3 Keys of Appraisal
1. Validity
2. Applicability
3. Impact
ValidityValidity:How far has this paper actually answered the question they set out to answer?
Question: Does endovascular emergency AAA repair reduce mortality?
Answer:Endovascular emergency AAA repair is associated with reduced blood loss.
ApplicabilityApplicabilityDoes the evidence from this paper apply to my patients?
Trial TitleCleverstatin reduces cardiac events in male patients with type II diabetes. A randomized controlled trial
Population1000 patients in Helsinki, Finland
Should I change my Wigan patients to the more expensive cleverstatin?
ImpactImpactHow much of a difference would this make?
Trial TitleHigh flow oxygen increases mortality in acute myocardial infarction. A randomised controlled trial.
RecommendationPatients should no longer be given maximal oxygen therapy in acute MI.
I am the SHO in A&E: a man presents with crushing chest pain – am I really going to withhold oxygen?
How to approach a paper!
Case Study
First impressions
What journal?
•BMJ vs The Antarctic Journal of Homeopathic Resuscitation
•Impact factors
•Conflict of Interests
•Track Record
•Abstract?
Research Question
•What is the research question at the centre of this paper?
•Is it an important question?
•Did I learn anything from reading this paper?
Introduction
. Outlines and references preceding work and ‘the story so far’.
. Points out weaknesses and gaps in preceding work.
. Justifies need and usefulness of the study.
. Puts the paper in context.
Methods
What kind of a study is it?
Meta-analysis or Systematic
Review
Multiple RCTs or 1 really good one
Single RCTs
Cohort studies
Case-control studies/case series
Expert opinion
Expert opinion
Thoughts of one person who has seen many cases
Case Reports/Series
What happened with one patient?What happened with a few patients?
Case-control studies
Risk Factors
Time
Cohort studies
Time
Chewing Gum Lung Cancer
Smoking
A B
C
Confounding factors
Randomized controlled trials (RCTs)...Randomised…– minimise bias by randomly assigning subjects to groups
...Controlled… – compare your intervention to a control, where the only difference is
the intervention
…Trial… – you don’t know what’s going to happen
The Risk of Bias
Selection bias
Treatment bias
Observer bias
‘Systematic differences between the two groups’
Minimising bias
•Blinding
•Double blinding
•Triple blinding
•Losing will to live?
Results
•Groups at baseline
•Intention to treat
•Selective outcome reporting
•Subgroup analyses
•Side effects
DiscussionWhat does this mean?How was the data interpreted?What does this mean for practice?Does it fit with other studies?
Have they acknowledged/justified the flaws?
Conclusion
Is this a reasonable conclusion?Does it answer the research question?
Lab ResearchWhat is the research question?
Description – observationClarification – mechanism
Very similar to clinical studies, but watch out for relevance
cell biology vs animal studies vs human tissue vs human trials
• Does it follow from previous research – theoretical justification
• n numbers, selective reporting• are the results meaningful – does it matter
if the CRP is high if all the animals died?• good modeling – young animals do not
represent aged obese smoking Salford residents
• adequacy of controls – is that really the only difference
• outcome assessment
• are the controls normal• are the interventions different, how was
this assessed• is there another explanation
• how does it fit with existing research• do their conclusions support extrapolations
Qualitative/Education
Scientific methods - leaves on trees
Did they need to use words?•Well-problematized topic•Clear research questions – relevance to the real world?•Appropriate methodology - RCT in education•Clear recipe methods•Limitations and conclusions
Big problemSo many qualitative methods, but 90% is description, very little is justification, even less clarification
Take home messages
1. take nothing at face value
2. healthy skepticism
3. anyone can do it!
Further ReadingBooks:
How to read a paper. Greenhalgh T. BMJ books. Blackwell publishing
The doctors guide to critical appraisal. Gosall N, Gosall G. Pastest books
Bad Science. Ben Goldacre
Papers:
Why Most Published Research Findings Are False. John Ioannidis
Courses:
Research methods and critical appraisal- www.rsm.ac.uk (events)