What does it take to perform a

Systematic Review of animal studies?

Carlijn Hooijmans

C.Hooijmans@cdl.umcn.nl

Central Animal Laboratory, SYstematic Review Centre for Laboratory animal Experimentation; SYRCLE

Radboud University Medical Centre

www.umcn.nl/SYRCLE

1

Systematic steps:

1. Phrase the research question

2. Define in- and exclusion criteria

3. Search systematically for ALL original papers

4. Select relevant papers

5. Assess study quality and validity

6. Extract data

7. Analyze results (when possible perform MA)

8. Interpret and present data

2

Focus of this presentation:

1. Phrase the research question

2. Define in- and exclusion criteria

3. Search systematically for ALL original papers

4. Select relevant papers

5. Assess study quality and validity

6. Extract data

7. Analyze results (when possible perform MA)

8. Interpret and present data

3

Focus of this presentation:

1. Phrase the research question

2. Define in- and exclusion criteria

3. Search systematically for ALL original papers

4. Select relevant papers

5. Assess study quality and validity

6. Extract data

7. Analyze results (when possible perform MA)

8. Interpret and present data

4

Goal of Searching systematically

• Detect the maximum amount of available information

Advantages of Searching systematically

• Prevent false or imprecise conclusions

• Prevent unnecessary experimentation

• Obtain new insights that may arise from aggregating

earlier work

5

How to search systematically?

• Formulate adequate and specific research question

Define: - Disease of interest/ health problem

- Population

- Intervention/exposure

- Outcome measures

Example: „What is the effect of [intervention/exposure] on [outcome

measures] in [population studied] for [disease of

interest/health problem]?

• Identify appropriate databases

• Create and run a comprehensive search strategy

6

How to search systematically?

• Transform research question into search strategy for PubMed

Critical search components (SC).

• Identify relevant search terms for each SC.

o Identify Medical Subject Headings (MeSH) terms

"fatty acids, omega-3"[MeSH Terms]

o Identify free-text terms (synonims)

Fish oil [tiab], Omega-3 [tiab] , PUFAs [tiab], DHA [tiab], EPA [tiab], omega 3 fatty acids

[tiab], omega 3 fatty acid [tiab], polyunsaturated fatty acids [tiab] etc…..

o Combine MeSH terms and free-text terms

"fatty acids, omega-3"[MeSH Terms] OR Fish oil [tiab] OR

Omega-3 [tiab] OR, PUFAs [tiab] OR etc

• Repeat step 1-3 for every SC

• Combine SCs

7

Example: Systematic Review about supplementation of omega-3

fatty acids in animal models for Alzheimer’s Disease

Quick search in PubMed:

8

Comphrensive search strategy in PubMed:

Omega-3

Animal models

Alzheimer‟s

Example: Systematic Review about supplementation of omega-3

fatty acids in animal models for Alzheimer’s Disease

9

More information about searching

systematically and finding all animal studies?

10

Focus of this presentation:

1. Phrase the research question

2. Define in- and exclusion criteria

3. Search systematically for ALL original papers

4. Select relevant papers

5. Assess study quality and validity

6. Extract data

7. Analyze results (when possible perform MA)

8. Interpret and present data

11

Why should we critically appraise the included papers?

• Low methodological quality often causes bias in the study

results

overestimation or underestimation

Macleod et al. Stroke, 2008

12

Why should we critically appraise the

included papers?

• The conclusions resulting from a SR/meta-analysis are

dependent of the quality of the original included studies

Transparancy >>>> improves interpretation/reliability

Invalid studies may produce misleading results

Garbage in = garbage out

13

How do we critically appraise the included

papers?

• 2 dimensions:

External validity

Internal validity

• External validity:

Generalizability of the study results

• Internal validity:

The extent to which the results of a study are correct for

the circumstances being studied (methodological quality)

Threatened by bias (systematic errors)

14

Type of bias Description Solution

Selection bias:

Systematic differences between baseline

characteristics of the groups that are

compared

Randomization

Performance

bias:

Systematic differences between groups

in the care or in exposure to factors other

than the intervention of interest

Allocation concealment

Detection bias:

Systematic differences between groups in

how outcomes are determined

Blinding

Attrition bias:

Systematic differences between groups in

“drop outs” from a study

Reporting drop outs

(reason and nr)

How do we critically appraise the included

papers?

15

0% 20% 40% 60% 80% 100%

1) Was it stated in the method section that the experiment was randomized?

2) In case the answer at 1) was yes: Was the method of randomization adequate?

2) In case the answer at 1) was no: Were the groups similar at baseline?

3) Was the allocation to the different groups during the randomization process concealed?

4) Were the caregivers blinded for the allocation of the animals to the specific groups

5) Was the outcome assessment blinded?

6) Methods for outcome assessment the same in both groups?

7) Is the timing of the intervention during the day similar in both groups?

8) Was the outcome assessment randomized across the groups?

9) Number of excluded animals specified per experimental group for each outcome measure?

10) Reason for exclusion mentioned for each excluded animal?

yes no unclear na

Yes=Low risk of bias. No= High risk of bias. ?=Unclear risk of bias. N.a.=not applicable

How do we critically appraise the included

papers?

16

17

Focus of this presentation:

1. Phrase the research question

2. Define in- and exclusion criteria

3. Search systematically for ALL original papers

4. Select relevant papers

5. Assess study quality and validity

6. Extract data

7. Analyze results (when possible perform MA)

8. Interpret and present data

18

Why conducting meta-analyses?

• To increase power

• To increase the precision of estimates of treatment

effects

• To obtain new information about safety and efficacy of

treatments that is not directly visible in the individual

studies

• To generate new hypothesis

19

0 -5 +5

Cell death Adapted data for MA

measure mean sd n mean sd n effect effect size (confidence interval)

Study A % 1.71 -0.05 6 1,9 0,07 5 significant -2.91 [-4.83, -0.98]

Study B ng/ml 104 -6.29 6 114 6,28 6 significant -1.51 [-2.86, -0.16]

Study C % 1 -0.17 12 1,3 0,35 12 significant -1.05 [-1.92, -0.19]

Study D nr of cells -241 14,7 8 -233 17,8 7 not significant -0.50 [-1.53, 0.54]

Study E nr of cells -190 15,1 6 -168 17,6 7 significant -1.24 [-2.47, -0.01]

Study F nr of cells -161 14,6 7 -181 10,3 7 not significant -0.19 [-1.24, 0.86]

Overall -1.02 [-1.61, -0.43]

Data from individual studies

Control groupExperimental group

How to conduct a meta-analyses?

20

Subgroup analyses (fictive dataset)

Overall -1.47 [2.33, -0.61]

21

Subgroup analyses (fictive dataset)

Overall -1.47 [2.33, -0.61]

Males -0.79 [-1.51, -0.06]

Females -3.64 [-4.78, -2.50]

22

Subgroup analyses (fictive dataset)

Overall -1.47 [2.33, -0.61]

Males -0.79 [-1.51, -0.06]

Females -3.64 [-4.78, -2.50]

Hippocampus -1.92 [-2.65, -1.19]

Cortex 0.37 [-0.24, 0.98]

23

Subgroup analyses (fictive dataset)

Overall -1.47 [2.33, -0.61]

Males -0.79 [-1.51, -0.06]

Females -3.64 [-4.78, -2.50]

Hippocampus -1.92 [-2.65, -1.19]

Cortex 0.37 [-0.24, 0.98]

24

So we discussed:

1. Phrase the research question

2. Define in- and exclusion criteria

3. Search systematically for ALL original papers

4. Select relevant papers

5. Assess study quality and validity

6. Extract data

7. Analyze results (when possible perform MA)

8. Interpret and present data

25

Narrative review versus systematic review

Feature Narrative review Systematic review

Research

question

Often unclear or broad Specified and specific

Sources and

search

Not usually specified Comprehensive and explicit

search strategy.

More than 1 database.

Study

selection

Not usually specified Explicit selection criteria

Risk of bias

assessment

Not usually present or only

implicit

Critical appraisal on the basis

of explicit criteria

Data

synthesis

Often a qualitative summary Often also a quantitative

summary (meta-analysis)

26

27