Systematic reviews and meta-analyses in infectious diseases: topics that

merit special attention

L. Leibovici1,2, D. Yahav2,3 and M. Paul2,4

1) Department of Medicine E, Rabin Medical Centre, Beilinson Hospital, Petah-Tiqva, 2) Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv, 3) Unit of

Infectious Diseases, Rabin Medical Centre, Beilinson Hospital, Petah-Tiqva and 4) Unit of Infectious Diseases, Rambam Health Care Campus, Haifa, Israel

Abstract

As in other disciplines in medicine, systematic reviews (SR) and meta-analyses (MA) in infectious diseases are an important aid to clinical

decision-making. In the present article we review features important to SRs in infectious diseases that should be addressed in most SRs and

MAs. We stress the need to include in the SR analysis all patients that were randomized; and all studies that were performed. Authors of

SRs should choose one main outcome that matters to patients, and base their conclusions mainly on this outcome. Resistance as an

outcome is a topic that should be addressed in all SRs of antibiotic treatment. Ethical aspects and especially patients’ safety should be

addressed in SRs.

Keywords: Ethics, meta-analysis, outcomes, resistance, systematic review

Article published online: 12 December 2013

Clin Microbiol Infect 2014; 20: 101–104

Corresponding author: L. Leibovici, Department of Medicine E,

Beilinson Hospital, 49100 Petah-Tiqva, Israel

E-mail: leibovic@post.tau.ac.il

As in other disciplines in medicine systematic reviews (SR) and

meta-analyses (MA) in infectious diseases are an important aid

to clinical decision-making. In the present article we review

unique features of infectious diseases that should be addressed

in most SRs and MA. Our recommendations are based on

several assumptions that are common to all SRs and MA.

Indeed, they are the basis of evidence-based medicine. First,

rigorous methodology is needed to remove bias and provide

patients and physicians with unprejudiced recommendations.

Second, an SR should start with a protocol that defines

population, intervention, comparisons and outcome; and then

the pre-defined data should be extracted from the published

studies. We should not start by asking which populations or

outcomes were addressed in the original trials; but define

them beforehand and seek for them in the results of the trials.

Third, our definitions of the population, intervention and

outcomes should stem directly from the question, does it

matter to patients?

We distinguish between SRs: the systematic collection of all

the research that was performed and that addressed a given

question, according to a defined protocol; and MA: the

mathematical combination of the results of the individual

studies (usually randomized controlled trials (RCTs)).

Population

Randomized controlled trials of antibiotic agents sport a

multitude of patient groups—intent-to-treat (ITT): including all

patients that were randomized; modified ITT: all randomized

patients that were given at least one dose of the study drug;

clinical modified ITT: patients that match the disease criteria;

clinically evaluable (CE): patients that could be evaluated at the

test-of-cure day as either cure or failure; microbiologically

modified ITT: patients who were randomized and have a

microbiologically documented infection (usually susceptibility

to the test drug and the comparator is also a request); ITT

microbiologically evaluable; and the same groups according to

the treatment as actually given (per-protocol). Which should

be chosen for extraction in an SR and MA? For methodological

reasons the ITT population should be preferred [1]. It is

important to understand that groups are comparable only at

ª2013 The Authors

Clinical Microbiology and Infection ª2013 European Society of Clinical Microbiology and Infectious Diseases

REVIEW 10.1111/1469-0691.12497

the point of randomization, and any selection of patients after

randomization carrires the risk of nullifying the advantage of

randomization. Hence, ITT analysis is vital to avoid the

introduction of bias.

However, in an ITT analysis, patients that dropped out of

follow up are usually ‘carried forward’ in the same state as last

seen. Most of the time the last state is ‘still infected’. If the trial

aims to prove non-inferiority or equivalence rather than

superiority (as is the case for the majority of antibiotics trials) a

large number of drop-outs in both arms favours non-inferiority

or equivalence in ITT analysis, and per-protocol analysis is no

less important.

Even in infections in which microbiological documentation

seems sine qua non for diagnosis (e.g. cystitis) we should collect

data for an SR and MA on all randomized patients. For example,

in an SR and MA addressing antibiotic treatment for cystitis [2]

some studies included patients on the basis of urinary symptoms

and positive urinalysis, before the results of urine culture were

available. In some patients urine cultures were negative. We

would like to know whether one treatment is better than

another in these patients (with a negative culture) as well. The

data on all patients answer the question that matters to patients:

in a woman with urinary symptoms and leucocyturia, managed

according to clinical practice (i.e. given antibiotics before the

results of the culture are known), is one drug better than

another? In terms of efficacy and mechanisms of action, the

women with positive culture are of interest. Hence, we should

collect and report data on all the randomized patients; on those

with negative cultures and on those with a positive culture (and

preferably also separate those with susceptible isolates from

those with resistant isolates).

Intervention and Comparison

Systematic reviews can ask general questions about interven-

tions; e.g. are b-lactam/aminoglycoside combinations better

than a single b-lactam drug [3,4]? In cancer patients on

chemotherapy, is prophylaxis with anti-fungal drugs that have

anti-mould activity better than with drugs that do not have

such activity [5]? These general questions are helpful for

managing patients: their results can be translated into practice

recommendations at specific locations, according to local

pathogens, susceptibilities and customs.

We should address interventions that are relevant to clinical

practice. Probably we would not consider treatment with

fluoroquinolones for patients with skin and soft-tissue infections

who are at low risk for gram-negative pathogens. Yet because of

regulatory requirements such studies are carried out [6–8].

Similarly, guidelines do not recommend vancomycin plus

aztreonam [9] for treatment of skin and soft-tissue infections,

nor is it a combination in daily use, yet it is used as a comparator

in RCTs of skin and soft-tissue infections. We doubt whether

the inclusion of such comparisons in SRs is of use.

Outcomes

The choice of outcomes in SRs was addressed in depth in

another review [10]. In short, the outcomes should matter to

patients and should be chosen specifically for the SR. Many times

outcomes that are chosen in RCTs are problematic. They are

chosen to limit the sample size, andwemight suspect sometimes

that they are chosen because they are easier to manipulate. For

example, a change in antibiotic treatment is less important to

patients if the outcome is otherwise benign. Choosing such an

outcome might bias in favour of newer, trendier antibiotics. (In

an unblinded study, when in doubt whether the patient is

responding, a physician might be more likely to stop an old

medication rather than a new ‘wonder’ drug.)

For SRs of infections that might end in death, fatality rate

should be the main outcome. Then we should count major

complications (e.g. the need for tracheal intubation, haemodial-

ysis, operation, severe adverse events, or admission to an

intensive care unit). A timely return to the former functional

trajectory and a short hospital stay are important to most

patients. Severe infections have long-term consequences [11].

Randomized controlled trials do not plan for long-term follow up;

however, the decision to look for long-term consequences in SRs

might make a difference for the planning of future trials as well.

The question of the main outcome might be more compli-

cated in less severe infections. What matters to people with

simple cystitis? Certainly, clinical improvement or cure rather

than bacteriological eradication. But how to quantify clinical

cure? Is it the duration of symptoms that matters, or the

duration of severe symptoms?Which urinary symptoms matter

most? We doubt that ‘cure at 7 days’ or ‘cure at 30 days’ really

capture the outcomes that are important to patients. Probably

we should ask the patients about their perceptions.

The choice of one main outcome that matters to patients

should be emphasized. The conclusions of the review should

be based mainly on this outcome. This is to prevent the

situation in which conclusions in favour of an intervention are

based on one of many secondary outcomes.

Induction of Resistance

Antibiotics are a non-renewable resource. Use of antibiotics

induces resistance and limits our anti-infective arsenal for the

ª2013 The Authors

Clinical Microbiology and Infection ª2013 European Society of Clinical Microbiology and Infectious Diseases, CMI, 20, 101–104

102 Clinical Microbiology and Infection, Volume 20 Number 2, February 2014 CMI

future. If an intervention could influence resistance it is

important to document this influence. Most SRs include only

RCTs, and the time-frames of RCTs are not compatible with

the time-frame of development of resistance in a unit or

hospital. Nevertheless, data on acquisition of resistance by

pathogens, super-infections caused by resistant pathogens, or

colonization with resistant bacteria often exist and should be

collected [12].

Inclusion of Observational Studies

Most SRs and MA include only RCTs. In MA of RCTs we

combine an effect size (e.g. relative risk). We do this because

the randomization allows us to assume that any differences

between the arms of a study, but for the effect of the

intervention, are due to chance.This assumption does not hold

true in observational studies, and combination of observational

studies in MA is problematic. However, there are instances in

which we want to look at observational studies, and even

combine their results using MA; e.g. comparisons in which an

RCT would be unethical; or questions for which few or no

RCTs were performed and the chances of obtaining results

from an RCT in the near future are non-existent. For example,

the question of whether appropriate, early, empirical antibiotic

treatment has a benefit cannot be addressed directly through

an RCT but many observational studies were performed with

this question in mind and their results are important when

deciding on the broadness of empirical antibiotic treatment

[13].

Completeness of the Database

A problem common to all SRs is obtaining the complete results

of all the trials that were performed. Mandatory registration of

trials might solve this problem; but we are not there yet.

Infectious diseases offer quite a few disturbing examples: a

Cochrane review could not reach strong conclusions on the

effectiveness of oseltamivir because the authors were denied

access to the full data on unpublished trials [14]. We were

surprised to find that 27 trials comparing cefepime with

another antibiotic were not published; and could not ascertain

the design and full data of these trials [15,16]. Withholding data

raises a strong suspicion of bias and manipulation. We should

remember that the main investment in RCTs is the goodwill

and efforts of the participants and researchers, and not the

money of the sponsor. This creates an ethical commitment to

publish every trial in full.

Ethics

We have proposed a framework for including in SRs informa-

tion on the ethical aspects of studies [17]. We had a clear

impression that reports with possible ethical flaws also had

weaker methodology.

Safety of included patients is probably the most important

ethical aspect. For example, we have shown that in some

clinical trials of antibiotic treatment the in vitro susceptibility of

pathogens to the study drugs was not disclosed to the

attending physicians in real time, even for septic patients [18].

Summary

In conclusion, we stress the need to include in the SRs and MA

all patients that were randomized; and all studies that were

performed. Authors of SRs should choose one main outcome

that matters to patients, and base their conclusions mainly on

this outcome. Resistance as an outcome is a topic that should

be addressed in all SRs of antibiotic treatment. Ethical aspects

and especially patient safety should be addressed in SRs.

Transparency Declaration

All authors declared that there are no conflicts of interest.

References

1. Montori VM, Guyatt GH. Intention-to-treat principle. CMAJ 2001; 165:

1339–1341.

2. Zalmanovici Trestioreanu A, Green H, Paul M, Yaphe J, Leibovici L.

Antimicrobial agents for treating uncomplicated urinary tract infection

in women. Cochrane Database Syst Rev 2010; (10): CD007182.

3. Paul M, Soares-Weiser K, Leibovici L. b-Lactam monotherapy

versus b-lactam-aminoglycoside combination therapy for fever with

neutropenia: systematic review and meta-analysis. BMJ 2003; 326:

1111–1115.

4. Paul M, Benuri-Silbiger I, Soares-Weiser K, Leibovici L. b-Lactammonotherapy versus b-lactam-aminoglycoside combination therapy for

sepsis in immunocompetent patients: systematic review and

meta-analysis of randomized trials. BMJ 2004; 328: 668–672.

5. Ethier MC, Science M, Beyene J, Briel M, Lehrnbecher T, Sung L.

Mould-active compared with fluconazole prophylaxis to prevent

invasive fungal diseases in cancer patients receiving chemotherapy or

haematopoietic stem-cell transplantation: a systematic review and

meta-analysis of randomised controlled trials. Br J Cancer 2012; 106:

1626–1637.

6. Gyssens IC, Dryden M, Kujath P et al. A randomized trial of the efficacy

and safety of sequential intravenous/oral moxifloxacin monotherapy

versus intravenous piperacillin/tazobactam followed by oral amoxicillin/

ª2013 The Authors

Clinical Microbiology and Infection ª2013 European Society of Clinical Microbiology and Infectious Diseases, CMI, 20, 101–104

CMI Leibovici et al. Systematic reviews in infectious diseases 103

clavulanate for complicated skin and skin structure infections.

J Antimicrob Chemother 2011; 66: 2632–2642.

7. Vick-Fragoso R, Hern�andez-Oliva G, Cruz-Alc�azar J et al. Efficacy and

safety of sequential intravenous/oral moxifloxacin vs intravenous/oral

amoxicillin/clavulanate for complicated skin and skin structure infec-

tions. Infection 2009; 37: 407–417.

8. Graham DR, Talan DA, Nichols RL et al. Once-daily, high-dose

levofloxacin versus ticarcillin-clavulanate alone or followed by

amoxicillin-clavulanate for complicated skin and skin-structure

infections: a randomized, open-label trial. Clin Infect Dis 2002; 35:

381–389.

9. Corey GR, Wilcox MH, Talbot GH et al.CANVAS 1: the first Phase III,

randomized, double-blind study evaluating ceftaroline fosamil for the

treatment of patients with complicated skin and skin structure

infections. J Antimicrob Chemother 2010; 65 (suppl 4): iv41–iv51.

10. Leibovici L, Falagas ME. Systematic reviews and meta-analyses in

infectious diseases: how are they done and what are their strengths

and limitations? Infect Dis Clin North Am 2009; 23: 181–194.

11. Leibovici L. Long-term consequences of severe infections. Clin Microbiol

Infect 2013; 19: 510–512.

12. Leibovici L, Soares-Weiser K, Paul M et al. Considering resistance in

systematic reviews of antibiotic treatment. J Antimicrob Chemother

2003; 52: 564–571.

13. Paul M, Shani V, Muchtar E, Kariv G, Robenshtok E, Leibovici L.

Systematic review and meta-analysis of the efficacy of appropriate

empiric antibiotic therapy for sepsis. Antimicrob Agents Chemother 2010;

54: 4851–4863.

14. Jefferson T, Jones M, Doshi P, Del Mar C, Dooley L, Foxlee R.

Neuraminidase inhibitors for preventing and treating influenza in

healthy adults. Cochrane Database Syst Rev 2010; (2): CD001265.

15. Leibovici L, Yahav D, Paul M. Excess mortality related to cefepime.

Lancet Infect Dis 2010; 10: 293–294.

16. Leibovici L, Yahav D, Paul M. Meta-analysis of a possible signal of

increased mortality associated with cefepime use. Clin Infect Dis 2010;

51: 1350–1351.

17. Weingarten MA, Paul M, Leibovici L. Assessing ethics of trials in

systematic reviews. BMJ 2004; 328: 1013–1014.

18. Leibovici L, Paul M. A breach in patients’ safety in randomized

controlled trials of antibiotic drugs. J Antimicrob Chemother 2013; 68:

2179–2180.

ª2013 The Authors

Clinical Microbiology and Infection ª2013 European Society of Clinical Microbiology and Infectious Diseases, CMI, 20, 101–104

104 Clinical Microbiology and Infection, Volume 20 Number 2, February 2014 CMI