Experimental Design

Dr. Anne MolloyTrinity College Dublin

Ethical Approach to Animal Experimentation

• Replace• Reduce• Refine

Reduce

• Good Experimental Design• Appropriate Statistical Analysis

Good Experimental Design is Essential in Experiments using Animals

• The systems under study are complex with many interacting factors

• High variability can obscure important differences between treatment groups:– Biological variability (15-30% CV in animal responses)– Experimental imprecision (up to 10% CV).

• Confounding variables between treatment groups can affect your ability to interpret effects.– Is a difference due to the treatment or a secondary effect of

the treatment? (e.g. weight loss, lack of appetite)

Do a Pilot Study and Generate Preliminary Data for Power Calculations.

Observational study -not an experiment; an experience (RA Fisher 1890-1962)

• Observational; generates data to give you the average magnitude and variability of the measurements of interest

• Gives background information on the general feasibility of the project (essentially validates the hypothesis)

• Allows you to get used to the system you will be working with and get information that might improve the design of the main study

Dealing with Subject Variation

• Choose genetically uniform animals where possible• Avoid clinical and sub-clinical disease• Standardize the diet and environment - house under

optimal conditions • Uniform weight and age (else choose a randomized

block design)• Replicate a sufficient number of times.

– Increases the confidence of a genuine result– Allows outliers to be detected.

Some issues to think about before you set out to test your hypothesis

• What is the best treatment structure to answer the question? – Scientifically – Economically

• What type of data are being collected?– Categorical, numerical (discrete or continuous), ranks,

scores or ratios. This will determine the statistical analysis to be used

• How many replicates will be needed per group?– Too many: wasteful; diminishing additional information– Too few: Important effects can be rejected as non-

significant

Choosing the Correct Design • How many treatments (independent variables)?

– e.g. Dose Response over Time • How many outcome measurements (dependent variables)

– Aim for the maximum amount of informative data from each experiment – (but power for one)

• Are there important confounding factors that should be considered?– Gender, age– Dose Response over Time x Gender x Age

• Complex experiments with more treatment groups generally allow reduction in the number of animals per group.

• Continuous numerical type data generally require smaller sample sizes than categorical data

Types of Study Design

• Completely randomized study (basic type)– Random not haphazard sampling

• Randomized block design: e.g. stratify by weight or age. (removes systematic sources of variation)

• Factorial Design: e.g examine two or more independent variables in one study

• Crossover, sequential, repeated measures, split plot, latin square designs

• Can greatly reduce the number of animals required– ANOVA type analysis is essential

Example: You want to examine the effect of two well known drugs on tissue bio- markers

Control Drug 1

Experiment 1

Control Drug 2

Experiment 2

Control Drug 2Drug 1

Reduces animals by the number of controls in one experiment

Identify the Experimental Unit

Saline Drug Saline Drug

Control Diet Experimental Diet

Defines the independent unit of replicationCage; animal; tissue

Sometimes pseudoreplication is unavoidable – so be aware of effect limitations

Power and Sample Size Calculation in the Design of Experiments

What is the likelihood that the statistical analysis of your data will detect a significant effect given that the experimental treatment truly has an effect? POWER How many replicates will be needed to allow a reliable statistical judgement to be made? SAMPLE SIZE

The Information You Need

• What is the variability of the parameter being measured?

• What effect size do you want to see?• What significance level to you want to detect

(commonly use minimum of p=0.05)?• What power do you want to have (commonly

use 0.80)

This information is used to calculate the sample size

Variability of the Parameter

• An estimate of central location: “About how much?”(e.g. the mean value)

• An estimate of variation: “How spread out?”

(e.g. the standard deviation)

An experiment: Testing the difference between two means

• In an experiment we often want to test the difference between two means where the means are sample estimates based on small numbers.

• It is easier to detect a difference if:– The means are far apart– There is a low level of variability between

measurements– There is good confidence in the estimate of the mean

Plasma Cysteine (µmol/L)

160 180 200 220 240 260 280

20 Results

0

1

2

3F

req

ue

nc

y

Mean 235: SD 22.9: Mean 236: SD 22.3

150 180 210 240 270 300

50 Results

0

1

2

3

4

5

Fre

qu

en

cy

Mean 236: SD 23.8

150.0 200.0 250.0 300.0 350.0

2,500 Results

0

50

100

150

200

250

Fre

qu

en

cy

SEM= SD/N

SEM=22.9/20 =5.12 SEM=22.3/50 =3.15

SEM=23.8/2500 =0.48

Mean 235: SD 21.4

150 200 250 300500 Results

0

10

20

30

40

50

60

Fre

qu

en

cy

SEM=21.4/500 =0.96

Means and SDs are about

the same!

Coefficient of Variation (CV)

(SD/Mean)% = 10.1%

Mean = 236SD = 23.82SD = 48 (approx)3SD = 71 (approx)

About 95% of results are between 236 ± 48i.e. 188 and 284

About 99.7% of results are between 236 ± 71i.e. 165 and 307150.0 200.0 250.0 300.0 350.0

Plasma Cysteine (umol/L)

0

50

100

150

200

250

Fre

qu

ency

Fitting a 'Normal' or 'Gaussian' Distribution

We can make the same predictions for a sample mean using SEM instead of SD

Having confidence in the estimate of the mean value

160 180 200 220 240 260 280

20 Results

0

1

2

3F

req

ue

nc

y

Mean 235: SEM=5.12

2 SEMs = 10.24

We can be 95% confident that true mean of the population will fall between 224.8 and 245.2

This is a sample.We don’t know the ‘true’ mean of the population

The sample mean is our best guess of the true population mean (µ) but with a small sample there is much uncertainty and we need a wide margin of error

The effect of increasing numbers

150 200 250 300500 Results

0

10

20

30

40

50

60

Fre

qu

en

cy

160 180 200 220 240 260 280

20 Results

0

1

2

3F

req

ue

nc

y

Mean 235: SD=22.9 Mean 235: SD=21.4

20 = 4.47SEM=5.12

500 = 22.36SEM=0.96

Number of samples is increased 25 timesStandard error is decreased by 25 = 5 times95% CI of the mean is 5 times narrower

Sample size considerations: Viewpoint 1Fix the sample size at six replicates per group and CV at 10%

The significance depends on the effect size

Effect you want in treated group

Student’s t-test

50% difference P<0.0001

25% P=0.0009

15% P=0.015

12% P=0.048

10% P=0.09

2 groups – control and treated6 replicates per group; CV of the assay 10%Cut-off for a significant result P=0.05 (Mean of treated outside the 95% CI of the controls) P=0.05

Sample size considerations: Viewpoint 2Fix the effect size at 25% difference and CV at 10%. The significance

depends on the number of replicates

Number of replicates per group

Student’s t-test

6 P=0.0009

5 P=0.0029

4 P=0.009

3 P=0.03

2 P=0.12

25% difference expected; CV of the assay 10%Cut-off for a significant result P=0.05

P=0.05

Sample size considerations: Viewpoint 3 Fix the effect size at 25% and number of replicates at 6. The significance depends on the

variability of the data (CV)

CV of the Assay Student’s t-test

10% P=0.0009

15% P=0.009

20% P=0.037

25% P=0.08

30% P=0.14

25% difference expected; 6 replicates per groupCut-off for a significant result P=0.05 P=0.05

Summary: The underlying issues in demonstrating a significant effect

• The size of the effect you are interested in seeing

– Big – e.g. 50% difference will be seen with very few data points

– Small - major considerations

• The precision of the measurement

– Low CVs - few replicates needed– High CVs – multiple replicates

How do we interpret a non-significant result?

A. There is no difference between the groupsB. There is a difference but we didn’t see it

(because of low numbers, SD too wide, etc.)

The decision to reject or not reject the Null Hypothesis can lead to two types of error.

Interpreting a Statistical Test

Correct Decision

α (Type 1) Error

(p value)

Reject H(o)

Declare that the treatment has an effect

Significant

β (Type II) Error

Correct Decision

Do not reject H(o)

Declare that the treatment has

no effect

Not Significant

The Null is false

The treatment has an effect

The Null is true

The treatment has no effect

DECISIONRESULTS

RealityEvidence from the experiment

β-Errors and the overlap between two sample distributions

Mean sample A

Continuous data range

95% CI of mean A 95% CI of mean B

Mean sample B

Miss an effect: β-errorSee an effect: POWER

Some Power Calculators

• http://www.dssresearch.com/toolkit/spcalc/power.asp

• http://statpages.org/ • leads to Java applets for power and sample size calculations.

• http://www.stat.uiowa.edu/%7Erlenth/Power/index.html• Direct into Java applet site

General Formula

r = 16 (CV/d)2

r= No of replicatesCV= coefficient of variation (SD/mean) (as a percent)d=difference required (as a percent)

Valid for a Type I error =5% and Type II error =80%.

Some General Comments on Statistics

• All statistical tests make assumptions– They assume independent data points –ignore this

at your peril!– They assume that the data are a good

representation of the wider experimental series under study

– Some assumptions are very specific to the test being carried out

Final Thoughts• Ideally, to minimise the sample number, use equal numbers of

control and treated animals.• Ethically, if an experiment is particularly stressful, lower numbers

may be desired in the treated group. This requires use of more animals overall to gain equivalent power – but can be justified.

• Remember - Statistical tests assume that the experiment has been done on a random sample from the complete population of similar items and that each result is an independent event. This is often not the case in laboratory research.

• Statistical logic is only part of the data interpretation. Scientific judgement and common sense are essential.

Dealing with Experimental Variation

• Randomization – Essential!– Ensures that the remaining inescapable differences are

spread among all the treatment groups

– Minimises potential bias

– Provides a reliable estimate of the true variability

– “Control treatment” must be one of the randomized arms of the experiment

Power Considerations

• You know the variability of the parameter being measured

• What effect size do you want to see?• You need a minimum significance level of

p=0.05• What power do you want to have (commonly

use 0.80)