CRITICAL APPRAISAL

AP DR JEMAIMA CHE HAMZAHMD (UKM) MS (OFTAL) UKM PHD (UK)

DEPARTMENT OF OPHTHALMOLOGYUKM MEDICAL CENTRE

 MINGGU PENYELIDIKAN PERUBATAN & KESIHATAN

Lecture content

Introduction

What is critical appraisal?

Steps of critical appraisal

Introduction

Continuous medical education is essential part in a doctor’s life Acquiring up-to-date knowledge

and skills for clinical practice Providing patients with the best

possible standard of care Revalidation and appraisal of

competency

However………

Information overload Number of new research articles published

each year are continually increasing > than 12,000 new articles per year + 300 RCTs are added to the MEDLINE database

each week

Evidence-based medicine

Findings of scientific research should be apply to individual patients as part of a doctor’s clinical decision-making process

Therefore, doctors must be able to: Select and appraise scientific literature that

is relevant to their field Understand the implications of research

findings for individual patients Elicit patients' own preferences Develop an appropriate management plan

based on the combination of this information

Bad science

In June 2008, the Sunday Express published an article about the link between suicides and phone masts

The increase in deaths among young people in Britain’s suicide capital could be linked to radio waves from dozens of mobile phone transmitter masts near the victims’ homes.

Dr Roger Coghill, who sits on a Government advisory committee on mobile radiation, has discovered that all youngsters who have killed themselves in Bridgend, South Wales, over the past 18 months lived far closer than average to a mast (Johnston 2008)

Ben Goldacre, a medical doctor and author of the weekly Bad Science column in the Guardian, investigated the claim made by the Sunday Express article and found out the following:

I contacted Dr Coghill, since his work is now a matter of great public concern, and it is vital his evidence can be properly assessed. He was unable to give me the data. No paper has been published. He himself would not describe the work as a “study”. There are no statistics presented on it, and I cannot see the raw figures. In fact Dr Coghill tells me he has lost the figures. Despite its potentially massive public health importance, Dr Coghill is sadly unable to make his material assessable. (Goldacre 2008)

Behind the headlines

Some news are genuinely based on valid studies, but jump to wrong conclusions by failing to consider some important aspects, such as the study design and the level of evidence of the original research.

In July 2008, an article was published on the Daily Mail claiming that there is a link between vegetarian diet and infertility (Daily Mail Reporter 2008).

The article was based on a cross-sectional study on soy food intake and semen quality published in the medical journal Human Reproduction (Chavarro et al. 2008)

Behind the Headlines, A study reported that eating tofu can significantly lower your sperm count

Limitation of the study: it was small, and mainly looked at overweight or obese men who had presented to a fertility clinic. It focused only on soy (soya) intake,

Daily Mail’s claim that there is a causal link between eating a ‘vegetarian diet’ and reduced fertility is misleading. (NHS Knowledge Service 2008)

Bias in the location and selection of studies

If a study had not obtained positive results, would it have been published - and quoted in the news?

When reviewing the literature published in scientific/medical journals, we should consider that papers with significant positive results are more likely to submitted and accepted for publication (publication bias); published in a major journal written in English (Tower of Babel bias); published in a journal indexed in a literature database, especially in less

developed countries (database bias); cited by other authors (citation bias); published repeatedly (multiple publication bias); … and quoted by newspapers!

(Egger & Smith 1998; Gregoire, Derderian, & Le Lorier 1995)

Hierarchy of evidence

Lecture content

Introduction

What is critical appraisal?

Steps of critical appraisal

Critical appraisal

Defined as "...application of rules of evidence to a study to assess the validity of the data, completeness of reporting, methods and procedures, conclusions, compliance with ethical standards, etc. The rules of evidence vary with circumstances.“

Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University Press

Con’t

A systematic process used to identify the strengths and weaknesses of a research article whether to use the results of a study in clinical practice

No 'gold-standard' instrument for critical appraisal

Criteria are not static Different study designs are prone to various sources of

systematic bias Increased awareness of the potential influence of other

non methodological factors, such as conflicts of interest

Lecture content

Introduction

What is critical appraisal?

Steps of critical appraisal

Ten steps of critical appraisal

Is the study question relevant? Does the study add anything new? What type of research question is being asked? Was the study design appropriate for the

research question? Did the study methods address the most

important potential sources of bias? Was the study performed according to the

original protocol? Does the study test a stated hypothesis? Were the statistical analyses performed

correctly? Do the data justify the conclusions? Are there any conflicts of interest?

Is the study’s research Q relevant?

Relevant to the researcher’s own field of work Addresses an important topic and adds to what

is already known about that subject

Subjective It might be crucial to some, but irrelevant to

others

Does the study add anything new?

New ideas and knowledge are developed on the basis of previous work ‘Standing on the shoulders of giants’

Two types of work: Study that make a substantive new contribution to

knowledge - rare Study that makes an incremental advance can also be of

value E.g. a study might increase confidence in the validity of

previous research by replicating its findings, or might enhance the ability to generalize a study by extending the original research findings to a new population of patients or clinical context

What type of research Q do the study pose?

The most fundamental task Identifying the specific research question that

an article addresses - this will determine the optimal study design and have a major bearing on the importance and relevance of the findings

Well-developed research question usually identifies 4 components: Population Intervention Comparator Outcomes

Con’t

Clinical research questions fall into 2 categories:1. Questions about the effectiveness of treatment

Relate to whether one treatment is better than another in terms of clinical effectiveness (benefit and harm) or cost-effectiveness

2. Questions about the frequency of eventsRefer to the incidence or prevalence of disease or other clinical phenomena, risk factors, diagnosis, prognosis or prediction of specific clinical outcomes and investigations on the quality of health care

Was the study design appropriate for the research question?

Study design Study design

Qualitative

Document

Passive observati

onParticipan

t observati

on

In depth interview

Focus group

Quantitative

Clinical trial

Randomised

Non randomis

ed

Observational

Cohort

Case-control

Cross sectional

Cross sectional

Measures prevalence of health outcomes/determinants of health/both in a population at a point in time

Strengths Allows comparison of

many different variables at the same time

Limitation Unable to provide definite

information about cause-and-effect relationships

Case control

Strengths Relatively economical and quick. Ideal for studying rare diseases or

diseases that can be defined specifically

Able to examine the association between multiple exposures and the disease of interest

Limitation Unable to examine multiple

diseases that might be associated with an exposure

Selection of appropriate controls can be challenging

Recall bias Cannot be used absolute risk for

disease after an exposure (uses OR)

Cohort

A longitudinal study (prospective or retrospective)

Strengths For examining disease after rare and

highly specific exposures Able to monitor the occurrence

of multiple diseases potentially caused by an exposure

Allows direct measurement of the absolute risk of developing a disease after an exposure

Limitation Selection bias and confounding factor Can only examine a single exposure or

group of related exposures Cannot be used if the population at risk

is not known 

Randomised controlled trial (RCT) Gold standard of study design

Most rigorous way of: determining whether a cause-

effect relation exists between treatment and outcome

for assessing the cost effectiveness of a treatment

Limitations: Ethical and practical concerns More costly and time

consuming than other studies     

Sequence generation - a rule to allocate

interventions to participants

Allocation concealment – secure implementation of the

schedule of random assignment

Blin

din

g

Did the study methods address the key potential sources of bias?

Bias – deviation from the truth Can be attributed to:

chance (e.g. a random error) affect the precision of the study

does not influence the results in any particular direction

to the study methods (systematic bias) e.g. how the study is conducted, be it how study participants

were selected, how data was collected, or through the

researchers' analysis or interpretation.

results in the overestimation or underestimation of the 'truth’

has a direction

Type of bias

Type of bias Description

Selection bias Systematic differences between baseline characteristics of the groups that are compared.

Performance bias Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest.

Detection bias Systematic differences between groups in how outcomes are determined.

Attrition bias Systematic differences between groups in withdrawals from a study.

Reporting bias Systematic differences between reported and unreported findings.

**Different study designs are prone to varying sources of systematic bias

Eg. In RCT:

Was the study performed in line with the original protocol?

Deviations from the planned protocol can affect the validity or relevance of a study

Failure to recruit the planned number of participants Sampling bias - those who actually were recruited might

be different from those who weren't for some reason Reduces the power of the study

changes to the inclusion and exclusion criteria Variation in the provided treatments or interventions changes to the employed techniques or technologies Changes to the duration of follow-up

Does the study test has a stated hypothesis?

Study hypotheses must be identified a priori  Developed from theory or previous experience

Post-hoc analysis produces high false positive findings 1 in 20 associations tested will be significant

(positive) by chance alone at significance level of 5% (P = 0.05)

Con’t

Sample size depends on hypothesis Whether it is a equivalence or noninferiority

trial vs. a superiority trial

Check all data relevant to the stated study objectives have been reported, and that selected outcomes have not been omitted

Were the statistical analyses performed correctly?

Tools used in the statistical analysis and the rationale for this approach should be included in the 'Method' section Approach to dealing with missing data and

the statistical techniques applied

In the 'Results' section, Defaulted or lost in follow-up and missing

data should be clearly identified

Do the data justify the conclusions?

Whether the conclusions are based on the accumulated data Overemphasis - Statistically significant findings are too small to be of

clinical value

Under emphasis - Dismissing large and potentially important

differences between groups that are not statistically significant,

because of small sample size

Generalisation of findings to broader groups of patients or

contexts than was reasonable given in their study sample

Whether statistically significant associations have been

misinterpreted to imply a cause and effect

Are there any conflicts of interest?

Conflicts of interest occur when personal factors have the potential to influence professional roles or responsibilities Decision such as which studies will be conducted in their unit, which

patients will be invited to participate in a study and whether certain clinical occurrences should be reported as adverse events affects validity of study

These decisions require researchers to act with integrity and not for personal or institutional gain

Check for declaration about the source of funding for the study and, if a potential conflict of interest had been identified for a statement about how this conflict was managed. 

Judge whether the declared factors are important and might have influenced the validity of the study's findings

Evidence based medicine

Formulate your questions into a format according to PICO tool (Population, Intervention, Comparison, Outcome) to easily do literature review

Literature review - if you can find a pre-appraised resource (e.g. systematic review), you can miss out the next step

Critical appraisal of your results

Decide what action to take from your findings

Evaluate your new or amended practice

Conclusion

Critical appraisal is a systematic process through which the

strengths and weaknesses of a research study can be identified

This process enables us to assess the study's usefulness and

whether its findings are trustworthy

The most important component of critical appraisal is careful

assessment of the study design (Don’t forget evaluation of the

statistical methods used, interpretation of the findings and

potential conflicts of interest)

Consideration of the importance of the research to our own

patients will help us identify the most relevant, high-quality

studies available to guide in our clinical practice

Any Questions? Contact jemaima@ppukm.ukm.edu.my