Critical Reading

Critical Appraisal

Definition: assessment of methodological quality

If you are deciding whether a paper is worth reading – do so on the design of the methods

Need to know two things:

• What type of study was done?• Was the study design appropriate?

What type of study?

Primary – these report research first hand.

• Experimental i.e. humans, animals;

artificial and controlled surroundings.

• Clinical trials – intervention offered.

• Survey – something is measured in a

group.

What type of study?

Secondary – summarise and draw conclusions from primary studies.

• Overview– Non systematic (summary)– Systematic (rigorous and pre-defined methodology)– Meta-analyses (integration of numerical data from

more than one study)• Guidelines (leads to advice on behaviour)• Decision analyses (to help make choices for

doctor or patient)• Economic analyses (i.e. is this a good use of

resources?)

Specific Types of StudyRandomised Controlled Trial (RCT)• Population is randomly allocated to two groups• One group is given a specific treatment or

intervention • On average the groups are identical because

they are randomised and therefore any difference in the measured outcome is due to the intervention

• Specified follow up period and specified outcomes

• e.g. drug better than placebo; surgical procedure compared with sham

Randomised Controlled Trial (RCT)

Advantages

• Allows rigorous evaluation of a single variable in a previously defined population e.g. a new drug.

• Prospective i.e. collect the information after you decide to do the study.

• Tries to disprove the null hypothesis

• Tries to eradicate bias because the two groups are identical.

• Allows for meta-analysis later.

Randomised Controlled Trial (RCT)

Disadvantages• Expensive and time consuming which can lead to

problems including:• Too few subjects• Too short a time• Who controls the study?• End point not clinical• Possibility of hidden bias:• Imperfect randomisation• Failure to randomise all eligible patients – who is

included/excluded.• Assessors not blinded.

Cohort study

• Two (or more) groups of people are selected on

a basis of a difference in exposure to a particular

agent i.e. vaccine, environmental toxin, medicine.

• Group followed up (usually for years) to see how

many in each group develop a particular

disease/outcome.

• e.g. Peto– 40,000 UK doctors.

• e.g. COCP causes breast cancer?

Case Control Study

• Patients with a particular disease are identified and “matched” with controls.

• Data is collected retrospectively either from medical records or from memory, looking for a causal agent.

• Looks for associations but not necessarily the same as cause.

• e.g. SIDS and sleeping position.

• Does whooping cough vaccine cause brain damage?

• Do overhead cables cause leukaemia?

Cross Sectional Survey

• A representative sample of subjects or patients

are studied (interviewed, questionaired,

examined) to answer a specific clinical question

at a specific time.

• e.g. normal height of three year olds

• what do most GP’s think about the use of

Viagra?

Case Reports

• Medical history of a single patient in a story form.

• Lots of information given which may not be seen in a trial or a survey.

• Often written and published fast compared to studies

• e.g. Thalidomide

Importance of ethics

Hierarchy of Evidence

(Systematic Review and Meta-analysis)

Randomised Controlled Trial

Cohort Studies

Case Control Studies

Cross Sectional Surveys

Case Reports

Was design appropriate?

In general:

• Therapy – i.e. effect of intervention – RCT• Diagnosis – ? test valid (can we trust it) or

reliable (? same result if repeated) – cross sectional survey with both gold standard and new test

• Screening – large population, pre-symptomatic – cross sectional survey

• Prognosis – i.e. what happens to someone if a disease is picked up at an early stage – longitude cohort study

• Causation – e.g. ? possible harmful agent leads to cause – cohort or case control study

- ? case report.

Assessing Methodological Quality

Questions to Ask• general framework• specifics dependant on type of paper

Logical ProgressionIntroduction - Title

- Abstract- Introduction

MethodsResults (Statistics!)Discussion

Seven essential questions:

Introduction

1. Why was the study done?

• Is the study original or does it add to the

literature in any way? e.g. bigger, better,

larger, more rigorous

• Is it interesting?

• Is there a clear research question?

Why was the study done?

i.e. what was the key research question/

what hypotheses were the author testing?

Hypothesis is usually presented in the

negative – the

“null hypothesis”

Studies try to disprove this lack of difference

or null hypothesis.

Seven essential questions:

Methods

2. What type of study? Who is it about?

• How recruited?

• Who included?

• Who excluded?

• Studied in “real life circumstances”?

• Applicable?

Seven essential questions:

3. Was it well designed?• i.e. does the study make sense?• What specific intervention or manoeuvre

was being considered and what was it being compared to?

• Is what happened what the author said happened?

• What outcome was measured and how?• i.e. length of life, quality of life, reduction

in pain• need to be objective.

Seven essential questions:

4. Was systematic bias avoided?

• i.e. was it adequately controlled for?

[ Bias = anything that erroneously influences the conclusions about groups and distorts comparisons

e.g. RCT – method of randomisation, assessment ? truly blind.

Cohorts – population differencesCase control – true diagnosis, recall (and

influences) ]

Seven essential questions:

5. Was it large enough and long enough to make results credible?

• Size is important!

• Power!

Seven essential questions:

Results

6. What was found?

• Should be logical – simplecomplex

Seven essential questions:

Discussion

7. What are the implications?

• For:

- you

- practice

- patients

- further work

• and do you agree?

Four possible outcomes from any study

1. Difference is clinically and statistically significant i.e. important and real.

2. Of clinical significance but not statistically so. ?sample size too small.

3. Statistically significance but not clinically i.e. not clinically meaningful.

4. Neither clinically nor statistically significant.

Recommended Reading

Ian Crombie : The Pocket Guide to Critical Appraisal

Trish Greenhalgh : How to read a paper; the basis of evidence based medicine