Developing systematic reviews

Karin Hannes

Featuring Mieke HeyvaertCentre for Methodology of Educational Research

Program

• 14.00: Introduction

• 14.30: Searching the literature

• 15.00: Critical appraisal

• 15.20: Break

• 15.30: Critical appraisal exercise

• 16.00: Data-analysis: a worked example (M. Heyvaert)

• 16.45: Anatomy of a systematic review

Introducing Cochrane,

Campbell and systematic

reviews

Outline

• Systematic reviews

– What is it and why do we need them?

– How do they look like?

– Who is producing them and where can we locate them?

– What is their (potential) impact?

– How can we start producing one ourselves?

What is it?

A systematic review is a „systematic identification, evaluation and synthesis of all relevant studies regarding a specific topic, based on an explicit and pre-defined methodology‟

If high quality research

studies exist...

Summarize them!

Half of what you learn in school will beshown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you whichhalf! (Dr. Sydney Burnwell)

Why do we need them?

• Evidence of „effectiveness‟: the extent to which an intervention, when used appropriately, achieves the intended effect.

• Evidence of „feasibility‟: the extent to which an intervention is practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context.

• Evidence of „appropriateness‟ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given.

• Evidence of „meaningfulness‟: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population.

Why do we need them?

“Over two million articles are published

annually in the biomedical literature in over

20,000 journals – literally a small mountain of

information … a stack … would rise 500

metres”

Cynthia Mulrow, in Systematic Reviews (BMJ

Publishing Group, 1995)

And within that stack, we would be

looking for a needle…

What is it?

• Lack rigor

• Methodology not

transparent

• Different reviewers reach

different

conclusions

• Become out of date

• Scientific rigor to minimise bias

• Explicit and reproducible

methodology

• Regularly updated

(Cochrane/Campbell)

Traditional

reviewSystematic

Review

What is it?

Predefined format:

To help review authors be systematic

To help people reading the reviews to find

information quickly

Transparancy

Reliability

Key characteristics of a SR

1. A clearly stated set of objectives with pre-defined

eligibility criteria for studies

2. An explicit, reproducible methodology

3. A well defined, systematic search that attempts to

identify all studies that would meet the eligibility criteria

4. An assessment of the methodological quality of the

findings of the included studies (assessment of risk of

bias )

5. A systematic extraction, synthesis, and presentation

of the characteristics and findings of the included

studies

PART 2: SYSTEMATIC REVIEWS OF

EFFECTIVENESS

HOW DO THEY LOOK LIKE?

What is a meta-analysis?

Optional part of a systematic review

Systematic reviews

Meta-analyses

• To identify a common

effect among a set of

studies

• To improve precision of

an estimate

• To investigate whether

the effect is constant

• To answer controversies

arising from conflicting

studies or to generate

new hypotheses

Meta-analysis

Gives a more precise estimate of effect than when

derived from the individual studies included within a

review

Intervention Comparisonno effect

Skills training versus usual curricula in the

prevention of drug use in school kids.

1. Summary statistic

+ CI for each study

individually (RR,

OR, RD…)

2. Pooled intervention

effect + CI is

calculated as a

“weighted average”

Meta-analysis

Facilitates investigations of the consistency of

evidence across studies, and the exploration of

differences across studies.

Intervention Standardno effect

Skills training versus usual curricula in the

prevention of drug use in school kids.

Heterogeneity!

CC & C2: meta-analyse

„Skill-based interventions‟ prove to be an effective

strategy to prevent from marihuana use in schoolkids.

Narrative summary

PART 3: SYSTEMATIC REVIEWS

WHO IS PRODUCING THEM?

“Combinations of data might

be better than attempts to

choose amongst them”

Astronomy, 17th century.

Karl Pearson, 1904.“Many of the groups are far too

small to allow of any definite

opinion being formed at all, having

regard to the probable error

involved”

Glass, 1976.

Archie Cochrane‟s challenge

“It is surely a great

criticism of our profession

that we have not organised a

critical summary, by

specialty or subspecialty,

adapted periodically, of all

relevant randomised

controlled trials.”

(1979)

Cochrane Collaboration: 1993

an international not-for-profit

organisation which aims to help

people make well-informed

decisions about healthcare by

preparing, maintaining and

promoting the accessibility of

systematic reviews of the effects of

health care interventions

Cochrane Collaboration

Collaborative

Review

Groups

Fields

The

Consumer

Network

Centres

Steering

Group Methods

Groups

Steering Group: Policy making body of the Cochrane

Collaboration.

Elected representatives of each of the Collaboration

entities

Fields: Represent a

population, group, or

type of care that

overlaps multiple

Review Group area.

Examples: Primary

Health Care, Health

Care of Older Adults,

Complementary

Medicine

Methods Groups:

Develop methods

and products integral

to internal functioning

of the Collaboration

Develop state of the

art methods for

systematic review

Examples: Statistical,

Economics, Placebo

Effects, Informatics,

Qualitative

Centres: Help organise and register

review groups

Facilitate collaboration among

reviewers

Provide training and consultation

Establish liaisons

Promote the Cochrane Collaboration

Provide unique contributionReview Groups: Focus on Treatment of disease or health problemsTo prepare and maintain systematic reviews

To develop a specialised trials register

Consumer Network:

Provides consumer input

Helps set priorities

Helps with dissemination

Campbell Collaboration (C2): 2000

an international organisation which

aims to help people make well-

informed decisions about public

policy (crime & justice, education,

social welfare) by preparing,

maintaining and promoting the

accessibility of systematic reviews of

the effects of social and

behavioural interventions

Campbell Collaboration

Coordinating

Groups

Users

Group

Partners

Steering

Group Methods

Groups

Users group:

Knowledge translation

to increase the impact of Campbell

reviews in policy and practice arenas,

and to make the information more

accessible

Methods Groups:

Improving the

methodology of

research synthesis,

and disseminating

guidelines for state-

of-the-art review

methods

Coordinating Groups:

Provide editorial

services and support to

authors of Campbell

reviews and build links

with users of systematic

reviews:

Crime and Justice

Social Welfare

Education

International Development

Steering group: strategic and policy

making body

Partners:

Institutes who have a

formal agreement with

Campbell and support the

same goals.

The Belgian Campbell Group

http://www.campbell-

collaboration.be/Belgian_Campb

ell_collaboration/Welcome.html

Examples of C2 Review Protocols and Reviews

are available on:

http://www.campbellcollaboration.org/library.php

CC & C2 Collaboration goals

• To ensure high quality, up-to-date systematic reviews

are available across a broad range of topics

• To promote access to systematic reviews

• To develop an efficient, transparent organisational

structure and management system for the Collaboration

• To achieve sustainability of the CollaborationPrinciples

Collaboration

Building on the enthusiasm of individuals

Avoiding duplication

Minimising bias

Keeping up to date

Striving for relevance

Promoting access

Ensuring quality

Continuity

PART 4: SYSTEMATIC REVIEWS

WHAT IS THEIR (POTENTIAL)

IMPACT?

Evaluate a solution

• Situation: 1 mj. people die each year due to traffic accidents

• Solution: Educate the drivers

• Systematic Review: 24 studies on education, no evidence that it reduces traffic accidents

• Recommendations: Stimulate teleconferences and the use of safe, public transport

“Driver education is big business – our results show that it is also a big con” (Prof. Roberts)

Identify gaps in „evidence‟

• Situation: anti-psychotics are subscribed for pregnant women with a psychotic disease

• Systematic Review: No literature on adverse effects of anti-psychotics on the mother or the development of the (unborn) baby

– RCT‟s with pregnant women are considered unethical

– Medicals‟ referals are based on habits and opinions

• Recommendation: Research on the effects of anti-psychotics in pregnant women is necessary

“The continued use of antipsychotic drugs in women during pregnancy and lactation without sound evidence raises serious clinical and ethical concerns.” (Webb)

Identify solutions that cause „harm‟

• Situation: Illinois law: Chicago Public Schools mandated to identify childrenat-risk for future criminal behaviour.

• Solution: Scaired straigth programs: take them on tours of adult prison facilities

• Systematic Review: Not only does a scared straight program fail to detercrime, it actually leads to more offending behaviour.

“Governments need to adopt rigorous evaluation to ensure that they are not causing more harm to the citizens they intend to protect” (Dr. Petrosino).

PART 5: SYSTEMATIC REVIEWS

HOW CAN WE CONTRIBUTE TO

THEM?

1. Select a topic

2. Make a review team

3. Develop/Register your title

4. Write a protocol

1. Select a topic

Systematic reviews:

• Commissioned

• Invited

• Unsolicited

Motivation to undertake a review:

• Resolve conflicting evidence

• Address questions when practice is uncertain

• Explore variations in practice

• Confirm appropriateness of current practice

• Highlight need for future research

• Learn to ask questions

(Treatment or Intervention):

• P population

• I intervention

• C comparison

• O outcome

1. Select a topic

Does providing information (I) at school have a preventive effect on the use of drugs (O) in high school students (P) compared to sanctions whensomeone is caught using drugs (C) ?

1. Select a topic

“Information campagnes in the

prevention of drug use”

“Does information provision at school

decreases the risk of drug use in high

school students (compared to a

standard approach)?” Scope of the question?

BROAD (LUMP) NARROW (SPLIT)

Advan

-tages

Comprehensive

Generalizibility

Effect modifiers (why different effect

between apples and oranges?)

Manageable for review team

Easy to read

Dis

advan

-tages

More resources

“mixing apples and oranges”

Evidence sparse?

Lack of generalizibility

Choice of scope “biased” by authors

Excercise 1: developing a question

2. The review team

Why more than 1 person?

• Detection of errors (selection of eligible studies, data extraction)

• Reduces risk of bias

• When more than one person / team is interested in topic

Who should be in the team?

• Expertise in topic area

• Expertise in systematic review methodology

• Incorporate view of “users” (teachers, consumers, therapists)

• Address questions that are important to people

• Take account of outcomes that are important to those affected

• Make it accessible to people making decisions

• Reflects variability in populations, settings…

3. Registering a title

Contact a Cochrane Review Group (CRG) or Campbell

Coordinating Group (CCG) and fill in a title form

Cochrane Review Group:

• > 52

• Specific condition (eg

incontinence, learning & developmental

disorder group)

• Group of diseases (eg ear nose

and throat disorders group)

• Organ (eg renal group, heart group)

• Patient (eg neonatal group, pregnancy

and childbirth group, pain and palliative

care)

Campbell Coordinating Group:

• 4

• Crime & Justice CG

• Education CG

• Social welfare CG

• International Development

There is a potential to

co-register reviews!

4. Write a protocol

• A priori statement of aims and methods of the review

• Research question(s), aims, methods are considered in

advance of identifying the relevant literature

– Conduct review with minimal bias

– Access to peer review

– Greater efficiency in review process

(Torgerson, 2003)

ADVANTAGES:

• Clear research question before the review

Avoid retrieving irrelevant papers

• A priori inclusion and exclusion criteria avoid changing criteria as review progresses or studies may be included on basis of their results

• If decisions are explicit it enables them to be justified

• Develop protocol as independently as possible from the literature avoid influence by one or two major studies, less bias (Torgerson, 2003)

Review Protocol Anatomy

• Title and authors

• Background

• Objectives

• Methods

– Searching for studies

– Selecting studies

– Data extraction (including assessment study quality)

– Data analysis

• Acknowledgements / conflicts of interests

• References

• Tables / Figures

Review Protocol Contents

BACKGROUND AND OBJECTIVES

• Describes the context of the review: why is it important?– Description of the problem

– Description of intervention

– How the intervention might work

– Description potential subgroups

• Clarify conceptual issues central to the review

• Precise statement of objective and research question, using PICO(S)– Which population?

– Which intervention(s)?

– Which comparison(s)?

– Which outcome(s)?

– (Which study design?)

A well formulated objective focuses the review

To assess the effects of

home-based programmes aimed at

improving developmental outcomes for

pre-school children from disadvantaged

families

Review Protocol Contents

METHODS

• How to select studies?

• How to search for studies?

• Assessment quality of included studies

• How to extract data?

• How to analyse results?

Review Protocol Contents

SELECTING STUDIES• Define inclusion and exclusion criteria

• Key features:

– Established a priori

– Explicit (explain all terms)

– Based on PICO

– Include study design, if applicable

• All retrieved studies are listed in the review either under included or

excluded together with a justification

Review Protocol Contents

Types of studies

Randomised controlled trials. The control group will either receive no intervention or standard

care. Studies comparing two different types of home based programme without a control group

will be excluded.

Types of participants:

Home based child development interventions for pre-school children from socially

disadvantaged families & Parents with children up to the age of school entry and who are

socially disadvantaged in respect of poverty, lone parenthood or ethnic minority status. Age of

school entry can vary between countries (4 to 7 years) and so the upper age range for this

review will be the school entry age for the country in which the trial took place.

Studies will be excluded if they aim to recruit particular clinical subgroups of parents.

Types of outcome measures

Primary outcomes: Cognitive development (including language development and attention) &

Socio-emotional development (including self regulation and behavioural development) +

Adverse outcomes: Parents feel disempowered.

Types of interventions

Home based interventions designed to improve child intellectual and socio-emotional

development through the provision of relevant knowledge and skills to the parent.

The intervention is delivered by trained lay or professional family visitors.

Review Protocol Contents

SEARCH FOR STUDIES

• Process needs to be reproducible!

• Describe search terms + search strings (BOOLEAN

LOGIC)

• Describe sources (electronic databases, other sources,

etc)

Review Protocol Contents

ASSESSMENT OF STUDY QUALITY

• Study quality may affect conclusions

• A systematic review always includes an assessment of

individual study quality

• Different tools available

• Describe how to use this information

– Description methodological quality

– Exclude studies of poor quality?

– To include in conclusions (strong versus weak evidence)

Review Protocol Contents

DATA EXTRACTION

• Process of reading through a study and extracting the

relevant information from each study, preferably by two

independent reviewers

• The reviewer fills out an data extraction sheet with the

appropriate information taken from that study

• Data extraction sheet may be on paper or electronically

Extraction form consists of two different parts

1. characteristics that apply to the entire study

2. results: studies typically have multiple outcomes

and therefore different effect sizes

Descriptive

Statistical

part

Summary

• Review protocol important first step when

undertaking systematic review

• Helps to focus and structure the review

• Limits the scope for bias

• In addition to how you plan to deal with the

different steps in a review, include information

on:– How many reviewers?

– How are reviewers trained?

– How to resolve disagreements?

Check the resource centre at

the Campbell Website for

specific guidance from

methods groups and

coordinating groups

http://www.campbellcollaboration.or

g/resources/research.php

References / more reading

• Egger M & Davey-Smith G. (2001). Principles of and procedures for systematic reviews. In M Egger G Smith & D Altman (Eds.), Systematic Reviews in Health Care: Meta-analysis in Context (2nd ed., pp. 23-42 ). London, UK: BMJ Books.

• Lipsey MW & Wilson DB. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.

• Miller S & Eakin A. Home based child development interventions for pre-school children from socially disadvantaged families (Protocol). The Campbell Library, 2011.

• Petticrew M & Roberts H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA: Blackwell Publishing.

• Torgerson C. (2003). Systematic Reviews. London, UK: Continuum.

• Resources Campbell Collaboration: http://www.campbellcollaboration.org/

• education_articles/ECG_Resources_for_Reviewers.php

• Resources Cochrane Collaboration: http://www.cochrane.org/training/cochrane-handbook

Searching the literature

Outline

• The Information Retrieval Process– Search strategies: Decisions and challenges

• Searching the Main Databases– Selection and Types of databases

– Preparing a search strategy

– Implementing a search

– Saving & Managing the results

• Additional Databases & Retrieval Methods

• Wrap Up

• Resources & Additional Readings

Information Retrieval:

A Continuous Process

– Preliminary (scoping) Searches• Supports beginning steps: Definition of key concepts &

research question

• Use of standard reference tools and broad searches for review articles and key primary studies

– Main Searches• Identification of primary studies through searches of

databases, Web, branching, manual searches

• Most difficult given a number of challenges

Main Searches: Decisions

• Selection of Information Retrieval Tools

– Scope of search: Which disciplines or subject fields should be searched (including all related fields)?

– Availability of indexing tools & expertise: Which tools do we have access to at our institution? Are there others who can perform searches for us?

– Format of indexing tools: What format are they in (e.g. online, print, web-based)?

– Dates: How far back does the indexing go for each tool?

– Language: What is the language of the material that is indexed? How can we locate non-English material?

– Unpublished work: How can we access dissertations, reports, and other grey literature?

Exclusion criteria:

Selection of Databases

• Consult your academic library‟s website to learn what

databases are accessible from your institution!

– Education: ERIC, British Education Index, Australian Education Index, CBCA Education, Education: A SAGE Full-text Collection; Education Full text, Linguistics and Language Behavior Abstracts, …

– Psychology: PsycINFO, PubMed (Medline), Ageline, Psychology: A SAGE Full-Text Collection,Criminology: A SAGE Full-Text Collection, …

– Sociology: Sociological Abstracts, Contemporary Women‟s Issues. Sociology: A SAGE Full-text Collection, …

– Health Care: Medline, CINAHL, EMBASE, AMED, Cochrane Trial Register,…

– Multidisciplinary: Academic Search Premier, ProQuestDissertations and Theses, FRANCIS, Social Sciences Index, SCOPUS, Web of Science

Main Searches: Decisions– Preparation of Search

Strategies

• What are the key concepts to be searched?

• How are these represented in each discipline?

• What are their related terms?

• How are these key concepts represented in the controlled vocabulary within each database to be searched?

Approaches to Parent Involvement

for

Improving the Academic Performance

of

Elementary School Age Children

Chad Nye, Herb Turner, Jamie Schwartz

The purpose of this review is to determine the

effectiveness of parental involvement in

improving the academic performance of school

age children in grades K-6.

Using a Thesaurus

1. From the research question, determine the main concepts to be searched (usually there are three):

– Intervention: Parental involvement

– Outcome: Academic performance

– Population: Kindergarten or Elementary students

2. Consult the main database to be searched.

3. Look up each concept in the thesaurus for this database.

– A thesaurus is an alphabetical listing of the controlled vocabulary(or descriptors) used within a subject database

– A hierarchical arrangement is used so that Broader, Narrowerand Related headings may be discovered

Check definition of terms.

Keep those from which

the definition matches

your inclusion criteria.

Example: ERIC

• Selecting the ERIC Descriptors

– Descriptors: Parental Involvement See: Parent participation

– Related descriptors: Family involvement, Parent-school relationship, Parent role, Parents as teachers

– Related keywords: parent* involvement, parent* effectiveness, parent* support, family support

– Descriptors: Academic Performance See: Academic achievement

– Related descriptors: Science achievement, Reading achievement, Writing achievement, Achievement gains

– Descriptors: Elementary School Children See: Elementary school students

– Related descriptors: Elementary education, Primary education, Kindergarten

Intervention

Outcome

Population

Main Searches: Decisions

• Construction of the Search Statements

– What terms should be searched as descriptors or as “keywords”?

– What Boolean operators should be used? AND / OR / NOT / NEXT / NEAR?

– Where should truncation characters be used? (e.g. parent* will retrieve parent, parents, parental)

– What limiting features are available to narrow results? (e.g. use of Publication Type codes, time period, language)?

ERIC Example

Boolean Operators

AND: Both terms must be present in order for a record to be

retrieved. Used to combine different concepts.

e.g parent participation AND achievement

OR: Either term may be present in order for a record to be

retrieved. Used to search for related terms or synonyms.

e.g. parent OR family

NOT: Used between two terms to ensure that the second term

will not appear in any of the results.

e.g. literacy NOT adult

(Parental involvement OR parent participation) AND academic

achievement AND (elementary OR primary education)

Example: ERIC, cont‟d

• Combining Keywords/Descriptors using Boolean operators:

1. DE=(Parent participation OR Family involvement OR Parent role OR Parent-school relationship OR Parents as teachers)

2. “Parent* involvement” OR “Parent* effectiveness” OR “Parent* support” OR “family support”

3. #1 OR #2 = 28,958 records

4. DE=(Academic achievement OR Science achievement OR Reading achievement OR Achievement gains) –46,574 records

5. DE=(Elementary school students OR elementary education OR elementary schools OR primary education OR kindergarten) - 291997 recs

6. #3 AND #4 AND #5 = 1,669 records

Intervention

Outcome

Population

Limiting Your Results

Using the Limiting Commands:

• Limiting fields contain information that is common to a large number of records within a database. These include language, document type, publication year and so.

• Some limiting fields will vary across databases (e.g. Classification Code, Age Group)

• Decisions about whether you are going to restrict:

• The language of the document

• The search to a certain time period

• The results to empirical studies only

ERIC record

AUTHOR : Polovina,-Nada; Stanisic,-Jelena

TITLE: A Study on Family-School Cooperation Based on an Analysis of School Documentation

PUBLICATION YEAR: 2007

SOURCE: Online Submission. Journal of Educational Research (Belgrade), v39 n1 p115-133

ERIC DOCUMENT LINK: http://www.eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED499159

DOCUMENT TYPE: Journal-Articles; Reports-Research

LANGUAGE: English

DESCRIPTORS: Student-Behavior; Attendance-Patterns; Child-Development; Parents-; Family-School-Relationship; Parent-Teacher-Cooperation; Foreign-Countries; Elementary-Schools; Parent-Participation; Grades-Scholastic; Academic-Achievement; Parent-Influence; Parent-Child-Relationship

IDENTIFIERS: Serbia-

ABSTRACT: Family-school cooperation is a very complex process that can be studied at different levels in a number of different ways. This study has covered only some aspects of cooperation between parents and teachers, based on school documentation of a Belgrade elementary school. The study covered analyses of 60 Attendance Registers pertaining to 60 classes with 1289 students from Grade 1 through Grade 8 during an academic year. The unit of analysis included: parents attendance at PTA meetings and individual meetings between parents and teachers. In addition to the frequency of parents' visits to school, therelationship between such registered parents' visits and overall academic performance, grades in conduct, excused and unexcused absence from classes were also considered. The research findings indicated interference between development factors (attitude change in parent-child relationship and growing-up) and parents' informal "theory of critical grades" i.e. transitional processes in schooling. The findings confirmed that parents' individual visits to school were mainly meant to offer an excuse for the student's absence from school, while attendance at PTA meetings was linked to poor grades in conduct and missed classes (both excused and unexcused). The findings also showed that parents pursued visiting strategies which were pragmatic, less time-consuming and less emotionally draining ones. The closing part refers to discussions on practical use of the study and possible further research. (Contains 4 graphs.) [This article is the result of the project "Education for Knowledge-Based Society" No. 149001 (2006-2010), financially supported by the Ministry for Science and Environmental Protection of the Republic of Serbia.] (Author)

High quality search strategies deliver relevant results!

Next Steps

Repeat these steps for each database to be searched.

Additional Retrieval Methods

• We haven‟t talked about….

– The web and other resources…to locate grey literature

• Use Advanced Search screens on large engines (eg. Google,

Altavista, AlltheWeb, MSN Live)

• Consult specific sites

• Screen conference abstracts and databasis storing

dissertations

– Manual searches

• Browse the Table of Contents of key journals for current years

– Reference Searching

• Look for relevant references in key-articles retrieved and

conduct a forward reference search in Web of Science.

Managing Your Results

• Export the results– Save as a Text file

• Import into a bibliographic management software:– RefWorks,

– Reference Manager,

– EndNote

• Edit your inhouse database– Add Source code for each database searched (e.g ERIC1,

PsycINFO1…)

– Add notes to the records (e.g.includes vs excludes)

• Compile a Search History document listing the original search strategies

Information Retrieval: Wrap Up

• Importance of information retrieval process– Not a “one-shot”deal

– Requires expertise in the planning and implementation of

searches

– Consulting with the Trials Search Advisor or an Information

Specialist is highly recommended

• Use a bibliographic management software– Store, manage and organize results

• Must have ability to replicate review– Documentation of entire process, including search strategies

used for each database, decisions taken, etc.

Resources

C2. Education Coordinating Group. (2009). Information Retrieval Methods Group Systematic Review Checklist and Database Worksheet. Available: http://www.campbellcollaboration.org/education_articles/ECG_Resources_for_Reviewers.php

Designed to help both reviewers and those reviewing C2 protocols.

– The Checklist itemizes the recommended steps for information retrieval during the protocol and review stages. A good resource for new C2 reviewers as it provides an indication of the criteria that will be used to evaluate the information retrieval component of a protocol and a review;

– The Database worksheet provides a useable, expandable template for the documentation of database searches.

Lefebvre, C., Manheimer, E., & Glanville, J. (2008, Feb.). Chap. 6: Searching for studies. In J.P.T. Higgins & S. Green (Eds.). Cochrane handbook for systematic reviews of interventions. London: Wiley.

Provides a “how to” guide for Cochrane Trial Search Coordinators. Clearly written and easy to understand for those who have no experience with searching. Focus is on the retrieval of information in the health sciences.

Information Retrieval Methods Group (2009). Searching for studies. Campbell Collaboration.

This comprehensive document provides the background on the C2 policy related to information retrieval and will be useful to Trials Search Co-ordinators (or Advisors) who are new to their post, as well those who are conducting reviews. This document outlines some general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design and carry out search strategies, manage references found during the search process and correctly document and report the search process. It is currently being revised and updated.

Readings

• Gomersall, A. (2007). Literature searching: Waste of time of essential skill? Evidence & Policy: A Journal of Research, Debate and Practice, 3(2), 8301-308. Available:

• Hopewell, S., Clarke, M., & Mallett, S. (2006). Grey literature and systematic reviews. In Dr. H.R. Rothstein, A.J. Sutton, & M. Borenstein (Eds.). Publication bias in meta-analysis (pp.49-72). New York: John Wiley & Sons.

• McGowan, J. & Sampson, M. (2005). Systematic reviews need systematic searchers. Journal of the Medical Library Association, 93(1), 74-80.

• Wade, A., Turner, H. M., Rothstein, H. R., & Lavenberg, J. (2006). Information retrieval and the role of the information specialist in producing high-quality systematic reviews in the social, behavioral, and education sciences. Evidence & Policy: A Journal of Research, Debate and Practice, 2(1), 89-108

Critical appraisal

Why is it so important?

• Meta-analysis aims to increase precision

• Meta-analysis of studies with bias in results gives very

precise but wrong results

• Garbage in, garbage out

Methodological rigor relates

directly to the interpretation

and generalization of findings

(Troia, 1999).

Bias versus imprecision

BIAS Ideal study

Bias:

A systematic error in the results or the inferences

Methodological flaw

Overestimation or underestimation

Bias versus imprecision

Ideal study IMPRECISION

Imprecision:

• A random error in the results

• Sample variation

• Direction of error is random

Risk of Bias versus Bias

BIAS + IMPRECISION

•Clear empirical evidence that particular flaws in study

design can lead to bias.

•Usually impossible to know to what extent biases have

affected the results.

•Key consideration = should the results be believed

http://www.sign.ac.

uk/methodology/

checklists.html

http://www.cochrane

.org/contact/method

s-groups

Online guideance:

Chapter Critical Appraisal

Domaine Description Judgement

Sequence generation: Did the study use a randomizedsequence of assignments (centralized)?

QUOTE: “patients were randomly allocated” COMMENT: probablydone, since earlier reports of this study describe use of random sequences.

YES (low risk of bias)

Allocation concealment:Did the study use anymechanism that shields those who enroll patients in a study from knowing the next assignment?

QUOTE: “We used sealed enveloppes to allocate the students to either the intervention or the control group”. COMMENT: Risk unclear since there is no mention of any safeguards used, such as opaque or sequentially numbered enveloppes.

YES (low risk)

NO (high risk)

UNCLEAR (uncertain)

Blinding: Participants, Providers, Outcome assessors

Lack of blinding could affect actual outcomes•Differential drop-out•Differential cross-over to alternative intervention•Differential administration of co-interventions

Lack of blinding could affect the outcome asssessment

YES

NO

UNCLEAR

Incomplete outcome data: Data available, butexcluded from analysis: Withdrawal, Do not attendfollow-up appointment, Failure to complete questionnaire / diaries; Cannot be located (lost to follow-up), Decision by investigator to cease follow-up, Data or records are lost

QUOTE: “Participants found to be ineligible after enrolment”.COMMENT: Justifiable, so not considered as leading to missing outcome data. VERSUS: Difference in proportion of incomplete data across groups and related to outcomes (e.g. adverse effectsin experimental group)

YES

NO

UNCLEAR

Selective outcome reporting: Selection of a subset of the variables recorded for inclusion in publication,

on the basis of the results

Omission of non-significant outcomesChoice of data for an outcome (e.g. competences)Choice of analysis (e.g. scores on a test for skills)Reporting of subsets of data (e.g. no attitudes, little correlation)Under-reporting of data (e.g. only “not significant”)

YES

NO

UNCLEAR

Cochrane Tool

“Risk of bias summary”

Downs and Black

instrument (J Epidemiol

Community Health

1998;52:377-84)

http://www.nccmt.ca/

registry/view/eng/9.html

Tool for Non-

randomized

studies

Focus on

confounding

issues in studies!

In summary

• Risk of bias assessment is a very important part of

your review

• Think about it when you write your protocol, describe

the methodology

• For RCT: use the Cochrane tool

• For NRS:

• Potential biases are likely to be greater (selection

bias & reporting bias)

• Use the appropriate tool to assess risk of bias

• Consider how potential confounders are

addressed

Excercise: critical appraisal

“Schrijven van reviews”:

Een praktijkvoorbeeld

Mieke Heyvaert

Ph. D. Fellow of the Research Foundation Flanders (FWO)

Methodology of Educational Sciences Research Group, Faculty of

Psychology and Educational Sciences, Katholieke Universiteit

Leuven, Belgium

“Schrijven van reviews” -

Een praktijkvoorbeeld:

A meta-analysis of intervention effects on

challenging behaviour (CB) among

persons with intellectual disabilities (ID)

Background

Interventions for CB among persons with ID

Examples of CB: Verbal and physical aggression,

Property damage and destructiveness,

Disruptive and antisocial behaviour,

Overactivity, Temper tantrums, Screaming,

Stereotyped and repetitive behaviour, General

delinquency, and Self-injurious behaviour (e.g.

head punching, self-biting, skin picking, and

hitting against hard objects or other body-parts)

Background

• High prevalence of CB among persons with ID

• Negative impact of CB for persons with ID and

their family/context

Intervention research

Bio–Psycho–Social intervention components

Study aim

• Systematically review biological,

psychotherapeutic, and contextual intervention

components applied to treat CB among persons

with ID, & analyse intervention effects and

moderating variables

• Method: Meta-analysis

Boost the total sample size and effect

precision by combining „sufficiently‟

homogeneous results across studies

Method – Literature search

• Idealiter:

– Systematic search of Electronic databases;

Grey literature; Journals; Reference lists;

Citation indexes; Contact authors & experts

• For this study:

– Systematic search of 4 electronic databases:

ERIC, PsycINFO, Web of Science, Medline

Method – Literature search

Systematic search of 4 electronic databases:

ERIC, PsycINFO, Web of Science, Medline

How would you conduct this systematic

search?

Method – Literature search

• Articles reporting on intervention effects for CB among

persons with ID by combining key terms describing the target

group and outcome variables

• Key terms Target group: intellectual disability, mental

retardation, learning difficulty, complex/high support needs

• Key terms Outcome variables: challenging behaviour,

problem behaviour, behavioural problems, aggression, self-

injury, self-injurious behaviour

• Terms in singular and plural, in US- and UK-terminology

Method – Literature search

• Included: English-language articles

• Included: Empirical studies

• Included: Published in peer-reviewed journals

• Included: Published between January 2000 - November 2008

• Excluded: Articles that only reported on diagnostic

instruments / only presented review material

• Excluded: Articles had to contain data or sufficient statistical

information to make quantitative analysis feasible

Method – Data-analysis

• Coding

• Meta-analysis

• Sensitivity analysis

• Subgroup analysis

• Meta-regression analysis

• Publication bias analysis

Intervention effects

Intervention characteristics

• Intervention type

• Intervention combination

Methodological features

• Study quality

• Study design

• Data collection

• Reliability

Study characteristics

• Research period

• Continent

Participant features

• Gender

• Age

• Degree of ID

Coding

Method – Meta-analysis

Statistical analyses:

Comprehensive Meta-Analysis Version 2.0

(Borenstein et al. 2009, http://www.meta-

analysis.com)

Method – Meta-analysis

Fixed-effect vs. Random-effects meta-analysis:

• Fixed-effect meta-analysis:

• We assume that there is one true effect size („fixed

effect‟) which underlies all the studies in the analysis

• The goal is to compute the common effect size for the

identified population, and NOT to generalize to other

populations

Fixed-effect meta-analysis: Rather rare !

Method – Meta-analysis

Fixed-effect vs. Random-effects meta-analysis:• Random-effects meta-analysis:

• We allow that the true effect could vary from study to study. There may be different effect sizes underlying different studies – E.g. the effect size might be higher in studies where a more intensive variant of an intervention is used, with older participants, with higher educated participants,…

• The goal is to generalize to a range of scenarios

Random-effects meta-analysis: Often done ! When accumulating data from studies written by different independent researchers, it would be unlikely that all the studies are functionally equivalent

Method – Meta-analysis

Random-effects meta-analysis:

+ Between-study variance into account

+ Study weights are more balanced: Less

relative weight to large studies

+ Inferential results referring to a universe of

more diverse studies than fixed-effect

analyses

+ Width of the confidence intervals: Generally

more „conservative‟ and „realistic‟

Method – Meta-analysis

Random-effects meta-analysis:

• Assuming that the studies in the analysis only

represent a random sample of effect sizes that

could have been observed

• The summary effect is the estimate of the mean

of these effects

• Studies weighted by study precision: Studies

that yield more precise estimates of the effect

size are assigned more weight

Method – Meta-analysis

• (1) Calculate effect sizes (standardised mean

difference; SMD) and variances for all included

studies‟ intervention effects

• (2) Calculate the summary effect (effect size

and variance) with a 95% confidence interval

(CI)

Results – Meta-analysis

(1) Calculate effect sizes (SMD) and variances for

all included studies‟ intervention effects:

• Combined effect sizes and their standard

errors computed for all 30 articles are

presented in Table 1

• The applied interventions made an

improvement in each study: All calculated

effect sizes are positive

Method – Meta-analysis

(2) Calculate the summary effect (effect size and

variance) with a 95% confidence interval (CI) –

Random effects model:

• SMD is 0.671 with a 95% CI of 0.570 to 0.771

• Medium (effect size around 0.5) to large (effect

size around 0.8) effect

• The null hypothesis that the mean of these

effects is zero, can be rejected, Z(N = 30) =

13.070, p < 0.001

Results – Meta-analysis

Comparison: For the Fixed-effect model:

• Standardized mean difference is 0.670

• 95% CI of 0.572 to 0.768

• The null hypothesis that the mean of these

effects is zero, can be rejected, Z(N = 30) =

13.452, p < 0.001

The results for the fixed- and random-effects

analysis are analogous

Method – Sensitivity analysis

= Determine the robustness of our results by

examining whether our conclusions might differ

substantially if a study was dropped

• Running the analysis 30 times, each time

removing one study, in order to show each

study‟s impact on the combined effect

Results – Sensitivity analysis

• Calculate for each study the overall

standardized mean difference when that study

was hypothetically removed from the meta-

analysis

• Since the 30 overall effect sizes formed by

omitting each study separately varied between

0.649 and 0.689 (see next Table), our results

look very robust

Method – Subgroup & meta-regression analysis

To assess the impact of moderating variables:

• Subgroup analysis: Compare treatment effects

across groups concerning categorical variables

• Meta-regression analysis: Explores the impact

of continuous moderators

Results – Subgroup analysis

• Analyses of variance to compare treatment

effects across groups for the categorical

variables: intervention type, intervention

combination, quality assessment, study design,

data collection, reliability, length of the research

period, continent, gender, age, degree of ID

• For all these variables the differences between

the groups are not significantly related to the

effect size

Results – Meta-regression analysis

• One continuous moderator: Publication year

• This variable is not significantly related to the

effect size

• Possible explanation of „not significantly

related to the effect size‟: Relatively small

number of studies included in the meta-

analysis

Method – Publication bias analysis

We refer to publication bias when studies

included in an analysis differ systematically

from all studies that could be included:

Particularly, studies with larger effects are more

likely to be published, and this can lead to an

upward bias in the summary effect

Method – Publication bias analysis

• (A) Funnel plot analysis: Plot the included

studies

• (B) Classic fail-safe N analysis: Do we need to

be concerned that the entire observed effect is

an artifact of bias?

• (C) Duval‟s and Tweedie‟s trim and fill-analysis:

How would the effect size shift if the apparent

bias were to be removed?

Results – (A) Funnel plot analysis

Results – (A) Funnel plot analysis

• In the Figure a measure of study size

(precision, the inverse of standard error) is

plotted on the vertical axis as a function of

effect size on the horizontal axis

• In the absence of publication bias, we would

expect the included studies (white circles) to be

distributed symmetrically around the combined

effect size - OK!

Results – (B) Classic fail-safe N analysis

Results – (B) Classic fail-safe N analysis

• Quantify the possible publication bias effect

• The fail-safe N is 1336 We would need to

locate & include 1336 'null' studies in order for

the combined two-tailed p-value to exceed 0.05

• Because this number is large, we can be

relatively confident that the treatment effect,

while possibly inflated by the exclusion of some

studies, is real

Results – (C) Trim and fill-analysis

Trim and fill plot of precision by standardised difference in means:

Plot with observed (white) and imputed (black circles) studies

Results – (C) Trim and fill-analysis

Left side of the mean effect:

• Duval and Tweedie's trim and fill analysis

suggests that four studies are missing

• Random-effects model: SMD & 95% CI for the

combined studies is 0.671 (0.570, 0.771)

• Using trim and fill: Imputed SMD & 95% CI is

0.634 (0.525, 0.744) - Little bit lower than the

original SMD

Results – (C) Trim and fill-analysis

Right side of the mean effect:

• Duval and Tweedie's trim and fill analysis

suggests that no studies are missing

Conclusions

• We found effect sizes ranging from 0.223 to

1.411 The effect sizes vary between a small

and a very large effect, all indicating positive

treatment effects

• The combined effect size over all studies is

0.671, with a 95% CI of 0.570 to 0.771, which is

a medium to large effect

• The implemented sensitivity analysis revealed

that this effect is robust

Conclusions

• Analyses of variance showed no significant different

treatment effects for biological, psychotherapeutic, and

contextual interventions

• Differences between unimodal and multimodal

treatments turned out to be not significant as well

• The calculated means and standard errors for the

moderators intervention type and intervention

combination show that there are only small differences

between the mean effects for biological,

psychotherapeutic, and contextual interventions, and for

unimodal and multimodal treatments

Conclusions

• Some authors claim that there is evidence for the effectiveness of pharmacological treatments, although many authors assert that they lack empirical effectiveness. Additionally, there are many concerns regarding their adverse effects

• Concerning psychotherapeutic and contextual interventions too, there are authors advocating their effectiveness, while some say that the evidence is rather limited

• Our meta-analysis shows that there is evidence for the effectiveness of pharmacological, psychotherapeutic and contextual interventions, used alone or in combination

• Furthermore, we found no indications for the superiority of one of the treatment approaches or combination types

Conclusions

• Treatment effects also did not vary much across groups for the categorical moderators included in our meta-analysis: quality assessment, study design, data collection, reliability, length of the research period, continent, and gender, age, & level of ID of participants

• We also did not find any effect of the continuous moderator publication year on the treatment effect

• The nonsignificance of moderator effects is not uncommon, due to low statistical power for detecting interaction between a moderator variable and the independent variable

• There exist only small differences between the mean effects for the tested categorical moderating variables, so even if we would increase the power, we would most likely not detect clinically important differences

Conclusions

• A funnel plot-, a fail-safe N-, and Duval‟s and

Tweedie‟s trim and fill-analysis demonstrate

that our meta-analysis does not seem to suffer

much from publication bias effects

Conclusions

• In summary, interventions for challenging behaviour among persons with ID described in the 30 included articles were effective, with only small differences between the mean effects for biological, psychotherapeutic, and contextual treatments, and for unimodal and multimodal interventions

• In contrast to claims in the literature that the evidence for one or another intervention is still rather limited, the effects in our meta-analysis were robust and convincing

Bibliography

• Borenstein M., Hedges L. V., Higgins J., & Rothstein H.

(2009). Introduction to meta-analysis. Wiley, Chichester.

• Comprehensive Meta-Analysis Version 2.0 (2009).

http://www.meta-analysis.com

• Heyvaert, M., Maes, B., & Onghena, P. (2010). A meta-

analysis of intervention effects on challenging behaviour

among persons with intellectual disabilities. Journal of

Intellectual Disability Research, 54, 634-649.

Data-analysis: narrative

approaches

Synthesis examples

Preliminary synthesis examples

– Textual summaries

– Tabulation

– Grouping and clustering

• Exploring relationships examples

– Idea webbing

– Synthesis tabulation

– Concept mapping

Preliminary synthesis

• Textualsummaryexample:

Young et al (1999) and Camit (2002) report on the effectiveness and implementation of a smoke alarm preventioncampaign in NSW Australia for Asianforeigners, including info sessions by skilledworkers. Qualitative data onbarriers to purchase were alsocollected in focus groups. The mainobservation is that….

• Tabulation example

Author/year

Location/Setting

Target population Method Mainfindings

Robertset al (2004)

London, UK, urban

58 adults and 41 children in qualstudy, 2145 householdsexposed to intervention

Focus groups and interviewsBefore and After studydesign

Problemswith smokealarmsidentified as major barrier

Camit(1998)

NSW, Australia, mixed

Chinese, vietnamese, arabic-speaking(numbers notgiven)

Focus groups

Implementationsuccesfulusinglanguageappropriateapproach.

Preliminary synthesis

• Grouping example

Grouping according to

Location Focus of report Population

UKDiGiuseppe et al (1999), Roberts et al (2004)

Broad, generalfactors affectingprogram: DiGiuseppe (1999), Camit (2002)

Ethnically mixed:Camit (2002), Young et al (1999), Roberts et al (2004)

AustraliaCamit et al (2002), Young et al (1999)

Individual factors affecting program:Roberts (2004), Young et al (1999)

Ethnically mixed & low income:DiGiuseppe (1999)

Preliminary synthesis: thematic

• Within case

• Cross-case

Study Themes

Study 1 Involvement of fire service

False alarm issues

Fear from installers

Testing of knowledge

Study 2 Lack of awareness of danger

Overestimation of cost

Frequent moving

Lack of knowledge on detectors

Barriers Facilitators

Barriers and facilitators to installation of smoke alarms

General Anxiety of property damage Landlord approval for installation

Specific to alarm

campaign

Lack of knowledge on installation

Unwillingness to install

Anxiety to letting installer in the house

Installation of alarm by skilled

worker from community

Barriers and facilitators to continued use of smoke alarms

Specific to alarm Problems with maintenance Project workers offer to maintain

Exploring relationships: Web of Ideas

Exploring relationships:

Synthesis Tabulation

Exploring relationships:

Concept mapping

http://www.lancs.ac.uk/shm/research/nssr/res

earch/dissemination/publications.php

Systematic Review Summary

• Establish objectives, selection criteria

• Search for eligible studies

• Apply selection criteria

• Assess study quality, extract data

• Analyse results using meta-analysis,

where appropriate and possible

• Perform sensitivity analyses

• Prepare report (keep it up-to-date!)

Acknowledgments:

Staff members of CMPO: Wim Van den Noortgate

Staff members of CEBAM: Bert Aertgeerts, Filip

Cools, Trudy Bekkering

Cochrane and Campbell Colleagues International

Contact:

Karin.hannes@ped.kuleuven.be

Tel: 016/326220

ANNEX: Anatomy of a

systematic review

Anatomy of a Systematic Review

• Background/Introduction– Establish need

– (Distinguish from previous review efforts)

– State objectives and review questions

• Methods– Criteria for inclusion and exclusion

• Type of population

• Type of studies

• Type of intervention (+ comparison)

• Type of outcomes

Anatomy of a Systematic Review (cont.)

• Methods

– Locating studies: Consulted data sources (databases, grey literature, reference searches, expert consulting etc.,

– Search strategy (final)

Anatomy of a Systematic Review (cont.)

• Methods

– Proces of selecting

studies

• Include screening

instrument in annex

1.

Published between 2005 (jan) and 2008

(dec)

Already filtered out.

2.

Conducted within health care or a

health care context

Include:

Syntheses of qualitative with quantitative research) by

synthesis methods other than informal review.

Exclude:

Papers commenting on methodological issues but

without including details of the outcomes of the

synthesis.

Papers that do not explicitly describe or name a method

for synthesis.

Reviews on concepts/definitions used within health care

or research issues

3. Published in English language

4. Published in a peer-reviewed journal

Inclusion criteria review on QES in the literature (update Dixon-Woods)

Possible screening criteria

• Timespan

• Language restrictions

• Discipline / Scientific field

• INCLUSION and EXCLUSION

CRITERIA

Consider a

flowchart of

results from

screening

Anatomy of a Systematic Review (cont.)

• Methods:– Data Extraction

• Introduce coding form

• Describe and define coding categories

• Describe process of data extraction „at least two independent reviewers‟

Do not re-invent the fire!

„We used the EPOC guidance on data-extraction (reference)‟…

Anatomy of a Systematic Review (cont.)

• Results– Descriptive results – Inferential results (if applicable)

• Discussion• Conclusions

– Implications for practice– Implications for research

• References • Appendix: search strings, critical appraisal checklist,

list with excluded studies (usually a flow chart), coding/extraction sheets, outcomes of meta-synthesis exercise etc.

• Consider developing a user sheet (short summary avoiding scientific jargon)