quality education for a healthier scotland pharmacy critical appraisal skills basic i nhs education...

Quality Education for a Healthier Scotland

Pharmacy

Critical Appraisal SkillsCritical Appraisal SkillsBasic IBasic I

NHS Education ScotlandProduced in collaboration with the

Association of Scottish Medicines Information

Pharmacists Group


Pharmacy

What is critical appraisal?

• This is the term given to describe the skills used when reading a paper to enable one to assess the validity (i.e. how close to truth) and usefulness (i.e. can the results be applied to your practice) of the results.

• Forms an integral part of evidence based medicine (EBM).


PharmacyWhat is EBM?

• EBM is the judicious use of current best evidence, combined with clinical experience, to make decisions about patient care.


Pharmacy

5 key steps that underpin EBM

• Define the specific question to be answered.• Find the best evidence to answer the question.• Critically evaluate the evidence to assess it validity

and usefulness.• Apply the results of the critical evaluation to practice.• Evaluate the performance of the intervention.


PharmacyWhy do we need evidence?

• Resources should only be allocated to interventions that are effective.

• The only way of judging effectiveness is EVIDENCE!


PharmacyWhat are “good” sources of evidence?

Less reliable sources

Glossy literature from pharmaceutical companies

Press releases from pharmaceutical companies

Magazines such as Pulse

Advertisements in medical journals

Conference abstracts about clinical trials

Trusted sources

Scottish Medicines Consortium

SIGN

NICE

Knowledge Network

Peer reviewed clinical journals

Summaries of information published by NHS bodies (e.g. UKMi Q&As)


PharmacyHierarchy of evidence

• Evidence comes in different forms and can be ranked in terms of importance.

• Quite often there may not be any high-levels of evidence to support a clinical intervention. In these cases it may be necessary to use evidence from the lower end of the hierarchy scale.


Pharmacy

Expert opinion/ clinical experience from respected sources

Case series then Case reports

Cross sectional surveys

Case control studies

Cohort studies

RCTs

Systematic reviewsMeta-analyses

Hierarchy of evidence


PharmacyFormat of clinical trials

Clinical trials are usually written in a standard format. This normally consists of:

» Title» Authors» Abstract – contains a brief summary of the trial.» Introduction» Methods» Results» Discussion» Conclusion


PharmacyKey questions to ask when assessing how valid

and useful a clinical trial is

• Patient or population – does the trial assess a commonly seen clinical condition and were the patients studied similar to your patient?

• Intervention – what was the medicine being tested? Was it used at the normal dose?

• Comparison – What was the experimental medicine compared to? Trials that compare a new medicine to a placebo are not useful when trying to decide where the medicine fits into current practice.

• Outcome – Was the end-point relevant to the patient (i.e. was it patient-oriented like a reduction in risk for a having a heart attack)?


PharmacyOutcomes

Patient orientated

Outcomes that directly improve the outcome for patients.

e.g. Reduction in hip fracture; improved cardiovascular mortality; prevention of a stroke

Disease orientated

Outcomes that are the result of a change to a disease characteristic

e.g. Change in bone mineral density; lowering of blood pressure; a reduction in cholesterol


PharmacySurrogate outcomes

• Outcomes that are not patient-orientated are surrogate outcomes.

• Often physiological or biochemical markers

• Cannot always assume that they are always a good indication of disease progression or improved survival.


PharmacyThree key factors affecting the

results of a trial

The intervention

Bias – researchers take steps to minimise by use of a control group, randomisation and blinding but some biases can still exist.

Chance – statistical tests are used to assess this.


PharmacyTypes of Statistics

Descriptive

Summarises or describes the sample

[Please note that for purposes of critical appraisal this type will not be discussed in this training package].

Inferential

Concerned with generalising from the sample to make inferences and estimates about a wider population.


Pharmacy

Inferences and Estimates

Inferences

Can conclusions be drawn from the sample be generalised to the population?

e.g. If a better response is seen with a medicine in a sample will it hold true in the general population

Help answer whether results may have occurred by chance in the trial.

Estimates

Given an observed size of effect in the sample, what is the likely value (or range of values) you will see in the population?

Help assess usefulness of a trial.


Pharmacy

Inferential Statistics


Pharmacy

• There are 1 of 2 assumptions made for interventions in clinical trials:– Null hypothesis (i.e. no difference between the control group

and the experimental group).– Alternative hypothesis (i.e. there is a difference between the

control group and the experimental group).

• Generally it is the null hypothesis that is assumed however.

Hypothesis Testing


Pharmacy

• The probability that a difference will be seen between 2 interventions in a clinical trial.

• Measured on a scale of 0 (impossible for event to happen) to 1 (the event will certainly happen)

• i.e. P = 1 would always happen• P = 0.05 would happen 1 time in 20• P = 0.02 would happen 1 time in 50• P = 0.01 would happen 1 time in 100• P = 0.001 would happen 1 time in 1000

Probability (P)


Pharmacy

• If p-value is less than 1 in 20 (p<0.05) then the result is regarded as being statistically significant; and the possibility of the difference observed arising by chance is low

=> If this is the case then one can reject

the null Hypothesis

P values


Pharmacy

• The probability that a test will detect a real difference in treatment outcomes in a sample if it is present in the population

• Usually expressed as a percentage and often set at 80-90%

Power


Pharmacy

Power and Sample Size

Sample size determinants:

Size of the difference being investigated

Level of significance


Pharmacy

Errors that can arise when drawing conclusions from data

• Type 1 error (alpha)• The data suggests a difference between the

groups when there is really no difference = False positive

• Often called significance level• A level of significance of p<0.05 represents

a 5% probability of making a type 1 error


PharmacyErrors that can arise when drawing conclusions from data

Type 2 error (beta)• The results fail to pick up a real difference that exists between the groups, and a conclusion is made that no difference exists = False negative• 100-(power)% is the probability of making a type 2 error


Pharmacy

Population Estimates


Pharmacy

• A tool for inferring the characteristics / parameters of a whole population from the measurements in one sample

• One of the most widely misused terms in statistics• In 95% of cases the ‘True Population Mean’ will lie within +/-2 SEM of the sample mean

SEM = SD ------ n

• Should NEVER be used instead of SD to indicate dispersion of measurements

Standard Error of the Mean (SEM)


Pharmacy

• Represents the range of values within which the true population mean lies.

• Indicate the precision (or imprecision) with which a study sample estimates the true population value for the whole population.

• Important role whenever we wish to apply results of a clinical study to the general population

• Narrower the range the more reliable the results

Confidence Intervals (CI)


PharmacyConfidence Intervals

Calculated by adding and subtracting multiples of the standard error of the

mean to and from the sample mean

95% confidence interval normally used (i.e. can be 95% confident that the

population value lies within this interval); or alternatively stated that there is a

1 in 20 chance (5%) that the true value lies outside the range quoted.

The narrower the CI, the more confident you can be the sample represents the

population


Pharmacy

Comparing means:No difference if CI overlap (i.e. even though 2 mean values maydiffer, extensive overlap of their respective CIs may suggest that thedifference is not statistically significant)

When comparing differences between means:No difference if CI includes 0

For proportions (e.g. RR):No differences if CI includes 1

For further information on this topic:-Statistics in divided doses: Number 8 (July 2005).Produced by the

North West Medicines Information Service. Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose8.pdf

How to Interpret?


Pharmacy

Estimation Statistics

Help assess “usefulness of the trial” by determining clinical importance and magnitude of the benefit by using data to estimate a range of probable values for the population.


Pharmacy

Example study

Group Total Number of patients in each group

Pain free within 2 hours

Intervention Group (Received Drug X)

2073 845

Control Group (Received Placebo)

1128 96


Pharmacy

Definition:

The proportion of patients in whom an event is observed

Control Event Rate (CER)

Vs

Experimental Event Rate (EER)

Event Rates


Pharmacy

Control Event Rate (CER) =

Event Rate in control group

Total patients in control group

Example

CER = 96/1128 = 0.085 (9%)

Control Event Rate


Pharmacy

Experimental Event Rate (EER) =

Event Rate in experimental group

Total patients in experimental group

Example

EER = 845/2073 = 0.41 (41%)

Experimental Event Rate


Pharmacy

Absolute Risk Reduction (ARR) is way of expressing differences between groups.

It is the difference in the event rate between the control event rate (CER) and the experimental event rate (EER).

ARR = CER-EER

Example

ARR = 9-41 = -32

Absolute Risk Reduction


Pharmacy

• Is an alternative means of expressing the difference between groups as a percentage• The Relative Risk Reduction (RRR) is the percent reduction in events in the experimental event rate (EER) and the control

event rate (CER).

RRR = (CER-EER) X 100

CER

Example

RRR = (9-41/9) X 100 = 356%

Relative Risk Reduction


Pharmacy

Why calculate?

Sometimes the trial may just state “the treatment reduced the risk” but does not state whether this is relative risk reduction or absolute risk reduction. Obviously the relative risk reduction looks more impressive since a larger number. Be aware of this and use the figures given to calculate both.

Neither RRR or ARR are intuitive ways to look at data. Numbers needed to Treat (NNT) is the more relevant way to look at the figures.


Pharmacy

Definition:

The number of people who needed to be treated to produce one particular clinical outcome

(e.g. How many patients need to receive Drug X instead of placebo to allow one patient to be pain free at 2 hours?)

NNT = ____1____ or ____1____

CER-EER ARR

Example

NNT = 1/32 = 3

Numbers Needed to Treat (NNT)


Pharmacy

Numbers Needed to Harm (NNH)

• This value can be similarly calculated when looking at adverse effects in a clinical trial.

• It is the number of patients you would need to treat with the experimental medicine (rather than the placebo or control)

for one additional patient to suffer an adverse effect.» NNH = 1/(EER-CER)


PharmacyExample NNH

Medicine Y is given to patients for treatment of hypertension. 12 of the 4000 patients given medicine Y experience a rash compared with 2 out of 3000 given placebo.

CER = 2/3000 = 0.00066

EER = 12/4000 = 0.003

NNH = 1/ (0.003-0.00066) = 428

Therefore 428 patients must be treated with medicine Y

rather than placebo for an additional 1 patient to have an

adverse effect.


PharmacyRelative Risk

The relative risk (RR) is the size of the effect in the experimental group

relative to the size of the effect in the control group. The relative risk is

often quoted in a clinical trial paper.

RR = CER/EER

Example

RR= 9/41 = 0.21

A relative risk of 1.0 means that there is no difference between the

experimental and control groups. This result shows a RR < 1.0

indicating that the patients on the medication are more likely to be

pain free at 2 hours than those receiving placebo.


Pharmacy

The ratio of patients in the treatment group succumbing to a particular end point compared to the control group

Compares the probability of the event occurring with the probability that it will not occur.

If >1 = event more likely to happen

If <1 = event less likely to happen

Odds Ratio (OR)


Pharmacy

The odds ratio must be calculated first for control and treatment group:

In isolation it is difficult to imagine what this figure means but an OR of 1

means that the two groups were equally likely to be pain free within 2 hours.

An OR higher than 1 means that the treatment group was more likely to

experience the event (pain free within 2 hours) than the control group.

An OR of less than 1 would mean that the treatment group was less likely to

be pain free at 2 hours. In this case the OR was 8 so the treatment group was

8x more likely to be pain free at 2 hours.

Odds Ratio (II)


PharmacySummary

• Don’t always believe everything you read!

• Choose your source of evidence wisely and systematically to answer your question.

• Use estimation statistics to help evaluate usefulness and clinical importance of a trial.

• Utilise population estimates to determine how reflective of the true population the trial results are likely to be.

• Statistical significance does not always equate to clinical significance.

• There is a lot of information available but you have to choose the best evidence available. Remember that all evidence is not equal!


PharmacyReferences

Brignell J. How do Relative Risk and Odds Ratio compare? (April 2006). Available at http://www.numberwatch.co.uk/rr&or.htm

Burls A. What is Critical Appraisal in Evidenced Based Medicine 2nd ed. Oxford: University of Oxford. Available at www.whatisseries.co.uk

DeCaro, S. A. A student's guide to the conceptual side of inferential statistics (2003). Available from http://psychology.sdecnet.com/stathelp.htm.

Easton V, McColl JH. Confidence Intervals in Statistics Glossary V1.1. Available from http://www.stats.gla.ac.uk/steps/glossary/confidence_intervals.html#confinterval

Greenhalgh T. How to read a paper: The basics of evidence based medicine 2nd edition. London: BNJ Books 2001.

Swinscow TDV, Campbell MJ. Statistics at Square One 10th edition. London: British Medical Association 2002. Statistics in Divided Doses, Assessing the reliability of a sample (Number 3). North West Medicines Information

Service (September 2001). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose3.pdf Statistics in Divided Doses, Variability, probability and power (Number 4). North West Medicines Information

Service (May 2002). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose4.pdf Statistics in Divided Doses, First steps in analysis - comparing the means of large samples (Number 5). North

West Medicines Information Service (November 2002). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose5.pdfStatistics in Divided Doses, Confidence intervals (Number 8). North West Medicines Information Service (July

2005). Available at http://www.ukmi.nhs.uk/filestore/misc/StatsinDivDose8.pdf Wills S et al. Critical Appraisal of Clinical Trials E-learning Module via NHS Education South Central. Available

for free registration at http://www.learning.nesc.nhs.uk/

quality education for a healthier scotland pharmacy critical appraisal skills basic i nhs education...

Documents