Information retrieval in systematic

reviews:

A case study of the crime prevention literature

American Society of Criminology,

19 November 2014

Lisa Tompson and Jyoti Belur

UCL Department of Security and Crime Science

Overview

• Systematic reviews and systematic searching

• Stages in a search

– Structuring the research question

– Choosing sources

– Creating search strategies

– Review and refine results

• Findings

Systematic reviews

• Systematic review: “a review of research literature using systematic

and explicit, accountable methods” (Gough, Oliver and Thomas, 2012:2)

• A transparent search strategy is a distinctive feature of systematic

reviews

• Searching for studies ~ data collection

• Overarching principle is to MINIMISE BIAS

• Synthesising studies not found systematically risks producing

unreliable or unrepresentative results

Publication bias

• Publication bias ~ selection (sampling) bias

• Strong positive effects are likely to be (Alderson & Green, 2002):

– Published more (publication bias)

– Published more rapidly (time lag bias)

– Cited more often (citation bias)

– Published in multiple outlets (multiple publication bias)

– Published in English (language bias)

Stages in a search

1• Structure the research question

2• Choose databases/sources

3• Create search strategies for the selected sources

4• Review results and revise search strategies if necessary

5• Process references

6• Log and report the search

7• Update the search if necessary

From Hammerstrøm et.

al., 2011

Campbell Collaboration

1• Structure the research question

2• Choose databases/sources

3• Create search strategies for the selected sources

4• Review results and revise search strategies if necessary

5• Process references

6• Log and report the search

7• Update the search if necessary

MISSION:

“to identify the best available evidence

on approaches to reducing crime

(and the potential savings to the police

service, their crime reduction partners

and the public)”

Structuring our research question

Findings from systematic

review or meta-analyses

Broadly defined ‘crime

prevention’

Overall aim was to search for evaluations of interventions in all relevant

fields that might have a crime prevention outcome

Creating a reference data set

• Harvesting references from known lists of systematic reviews in Crime

Prevention and Criminal Justice

– Thus, we performed backwards reference checking at the beginning of our search to

generate known studies

– Three lists; one known at beginning, two found serendipitously

• We used these to help us:

– Scope out what journals/sources would be the most appropriate to search

– To generate search terms (keywords and controlled vocabulary)

– Test the ways in which we refined each search strategy

Choosing sources

• Crime prevention knowledge base very fragmented

– Wide-ranging and multi-disciplinary sources selected

• 15 electronic databases:

– Criminal justice specific

– Social sciences

– Multi-disciplinary

– Grey literature

• Other sources:

– UK National Police Library

– Hand searches

– Reference checking

– Contacting experts

– Grey literature specialist

Create search strategy for databases - keywords

• Research question broken down into

concepts

• Keywords and synonyms generated for

each concept

– diversity of expression important to capture

• OR combines search terms

that are representing the same

concept

• AND combines terms that are

representing different concepts

Create search strategy for databases – controlled

vocabulary

• Controlled vocabulary - manual assignment of descriptors

• Thesauri available in some

databases

– this organises controlled

vocabulary

– Can ‘explode’ a term to search

for subordinate terms

• Thesauri are though

database-specific

Create search strategy for databases – building

the search syntax

• The most powerful types of searches apply a search syntax

– Boolean operators: OR, AND (not AND NOT)

1) Natural language terms for crime types

2) Controlled vocabulary terms for crime types

4) Natural language terms for prevention outcome

5) Controlled vocabulary terms for prevention outcome

8) Natural language terms for evidence synthesis research design

9) Controlled vocabulary terms for evidence synthesis research design

3) #1 OR #2

6) #4 OR #5

7) #3 AND #6

10) #8 OR #9

11) #7

AND #10

Review and revise results

• Prudent to test (pilot) search terms before finalising the search syntax

• ‘Review’ search term generated over 3 million hits in SCOPUS

– Other imprecise terms were ‘drugs’ and ‘disorder

• To increase the precision of our keywords we incorporated speech

marks, wildcards and proximity operators into our search terms

Speech marks Wildcards Proximity operators

“calls for service”

“bodily harm”

Crim*

offen?e

“firearm? NEAR/5 offender?”

systematic* PRE/2 review

Review and revise results

• We performed sensitivity analysis on each term

• Also restricted searches by:

– Date (post-1975)

– Language (English)

– Document type

– Subject headings

– Keywords (in multi-disciplinary databases – used two researchers)

• We empirically tested the results against the ‘reference data set’ of

known studies

Database Unique N relevant Hit rate (%) Database records

Applied Social Sciences Index and Abstracts (ASSIA) 416 22 5.3

Criminal Justice Abstracts (CJA) 1,319 77 5.8 Almost 500,000

Criminal Justice Periodicals (CJP) 412 28 6.8

CINCH 117 0 0

Education Resources Information Centre (ERIC) 458 10 2.1 Over 1,000,000

International Bibliography of the Social Sciences (IBSS) 190 12 6.3 Over 2,628,800

National Criminal Justice Reference Service (NCJRS) 2,389 54 2.3 Over 200,000

ProQuest theses and dissertations 598 3 0.5 Almost 3,000,000

PsycEXTRA 347 7 2.0 Over 315,000

PsycINFO 457 84 18.4 Over 3.5 million

SCOPUS 2,635 91 3.5 Over 53,000,000

Social Policy and Practice 334 30 8.9 Over 300,000

Social Sciences Full Texts 78 3 3.8

Sociological Abstracts 927 18 1.9 Over 955,030

Web of Knowledge 1,408 32 2.3 Over 60,000,000

Total 12,085

Results of database searches – individual results

studies found in individual databases

Results of database searches – database overlap

Database TOTAL Uniquewith

ASSIA

with

CJA

with

CJP

with

ERIC

with

IBSS

with

NCJRS

with

PsycEXTRA

with

PsycINFO

with

SCOPUS

with

SP&P

with

SA

with

T & d

with

WoK

ASSIA 15 0 N/A 9 4 0 4 4 0 6 11 2 0 0 2

CJA 27 15 0 N/A 2 1 0 0 0 6 9 1 1 0 0

CJP 11 1 1 9 N/A 1 1 2 0 3 8 1 2 0 0

ERIC 5 2 1 1 0 N/A 1 1 0 1 0 0 0 0 0

IBSS 3 0 1 3 0 0 N/A 0 0 1 3 1 0 0 2

NCJRS 30 17 1 9 2 2 0 N/A 1 5 9 5 1 0 2

PsycEXTRA 3 2 0 0 0 1 0 0 N/A N/A 0 0 0 0 0

PsycINFO 16 14 0 2 1 0 0 1 N/A N/A 0 0 0 0 1

SCOPUS 32 15 1 4 0 0 0 1 1 3 N/A 2 1 0 7

SP&P 12 12 0 0 0 0 0 0 0 0 0 N/A 0 0 0

SA 7 1 0 5 4 0 3 4 0 3 4 1 N/A 0 1

T & D 3 0 0 0 2 0 0 0 0 0 0 0 3 N/A 0

WoK 10 5 0 0 0 0 0 0 0 1 3 2 0 0 N/A

84 unique studies out of a total of 174 studies

• ‘Unique’ column refers to studies not retrieved from other databases

– Just under half (84 of 174 studies) unique

– Three databases have no unique records – suggesting redundancy

Identified studies by source and type

Type of publication N %

Book chapter 18 5.5

Book 12 3.6

Dissertation 14 4.3

Journal article 197 60.1

Report 87 26.5

Total 328

Summary

• High quality search will involve a substantial time involvement

• Minimising bias = wide range of sources and search tactics

– Search terms need to capture the diversity of expression

• Not a linear process

– Piloting search terms is important

– Empirically testing against a reference dataset useful for verification

• Database specific conventions – tailor search strategy to each one

• Grey literature vital in crime prevention

Thank you

Bowers, K., Tompson, L. & Johnson, S. (in press) Implementing information science in

policing: mapping the evidence base. Policing: A Journal of Policy and Practice.

Tompson, L. and Belur, J. (due to be submitted) Information retrieval in systematic reviews;

A case study of the crime prevention literature.

Lisa Tompson

l.tompson@ucl.ac.uk

Final number of included studies = 325

Our systematic search flowchart