ORIGINAL ARTICLES

No consensus exists on search reporting methods for systematic reviews

Margaret Sampsona,b,*,1, Jessie McGowana,c,d,e, Jennifer Tetzlaffb,Elise Cogob,c, David Moherb,f,g

aDepartment of Information Studies, University of Wales, Aberystwyth, Wales, UKbChalmers Research Group, Children’s Hospital of Eastern Ontario Research Institute, 401 Smyth Road,

Ottawa, Ontario K1H 8L1, CanadacInstitute of Population Health, University of Ottawa, Ottawa, Ontario, Canada

dDepartment of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, CanadaeDepartment of Family Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada

fDepartment of Pediatrics, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, CanadagDepartment of Epidemiology and Community Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada

Accepted 3 October 2007

Abstract

Objectives: The reporting of the search methods used in systematic reviews has implications for how systematic reviews are crit-ically appraised, their reproducibility and how easily they may be updated. The objective of this paper was to identify validated or eval-uated search reporting instruments used in reporting systematic review searches and to compare reported and recommended searchingpractices.

Study Design and Setting: This cohort study was a systematic review. The search strategy to identify instruments addressing thereporting of search strategies was developed first in the Library & Information Science Abstracts (LISAs) database and then adaptedfor MEDLINE and five additional databases. Additional instruments were identified through experts. Current reporting practice data wereanalyzed from a cohort of 297 recent systematic reviews.

Results: Of the 11 instruments examined, 7 cited supporting evidence but only 4 were validated. Eighteen different reporting itemswere identified but only one item, ‘‘databases used,’’ appeared in all instruments. There was a trend toward including more items in morerecent instruments (r 5 0.41). Current search reporting practices ranged from a high of 98.7% for databases used to a low of 11.4% forqualifications of the searcher.

Conclusions: There is no clear consensus regarding optimum reporting of systematic review search methods and commonly recom-mended items show suboptimal reporting. � 2008 Elsevier Inc. All rights reserved.

Keywords: Information storage and retrieval; Systematic reviews; Reproducibility of results; Publishing standards; Epidemiologic methods; Search strategies

Journal of Clinical Epidemiology 61 (2008) 748e754

1. Introduction

1.1. Background

The reporting of the search methods used to identify po-tentially relevant articles is an important aspect of any sys-tematic review because the search identifies the evidencebase of the review. Clear and transparent reporting of all

Some of the material in this manuscript has been presented as a poster

presentation at the Cochrane Colloquium (Dublin, Ireland October, 2006).1 Margaret Sampson had full access to all the data in the study and

takes responsibility for the integrity of the data and the accuracy of the data

analysis.

* Corresponding author. Children’s Hospital of Eastern Ontario

Research Institute, Chalmers Research Group, 401 Smyth Road, Ottawa,

Ontario K1H 8L1, Canada. Tel.: 613-737-7600.

E-mail address: msampson@cheo.on.ca (M. Sampson).

0895-4356/08/$ e see front matter � 2008 Elsevier Inc. All rights reserved.

doi: 10.1016/j.jclinepi.2007.10.009

aspects of the search enables readers to critically appraisethe search and forms the basis for replicating the searchfor any subsequent updating of a systematic review. Thereis no clear consensus regarding what constitutes an ade-quate search or what components should be reported toachieve transparency for readers. Evidence suggests thatthe quality of reporting of systematic reviews, including as-pects of the search, is less than optimal both historically [1]and more recently [2,3].

One possible explanation could be that systematic re-viewers and librarians (information specialists) may havefew comprehensive search reporting instruments availableto inform them as to the conduct and/or reporting of search-ing. The search reporting instruments that are available maycomplicate the process due to contradictions between theinstruments.

749M. Sampson et al. / Journal of Clinical Epidemiology 61 (2008) 748e754

1.2. Objectives

We set out to identify and compare any instruments thatinform the reporting of searches used in conducting a sys-tematic review and compare these standards to currentpractice.

2. Methods

2.1. Reporting instruments

2.1.1. Data sourcesWe followed the Quality of Reporting of Meta-analyses

(QUOROM) published guidelines for meta-analyses [4].Search reporting instruments (i.e., scales, checklists, orcomponents) were identified as part of a systematic reviewof evidence related to quality issues and errors in complexsearch strategies used in systematic reviews and healthtechnology assessments [5]. Instruments were sought thatevaluated or validated the quality of literature searches orprovided guidance in the conduct or reporting of literaturesearches. Instruments published or produced in any disci-pline after 1979 were sought. This date was chosen as it co-incides with the rise in the creation of electronicbibliographic databases and thus the widespread use ofelectronic searching.

The search strategy was developed initially in theLibrary & Information Science Abstracts (LISA, CSAinterface) database with the objective of high recall (i.e.,trying to retrieve all relevant studies). The search strategywas adapted for each of the following databases: LISA1969eMay 2005; Cochrane Methodology Register andCochrane Database of Methodology Reviews (completedreviews only) (The Cochrane Library 2005, Issue 2, Wileyinterface); MEDLINE (Ovid interface) 1966eJune Week 1,2005; PsycINFO (Ovid interface) 1806eJune Week 2,2005; CINAHL (Ovid interface) 1982eJune Week 2,2005; HealthSTAR (Ovid interface) 1987eMay 2005; andHealth and Psychosocial Instruments (HAPIdOvid inter-face) 1985eMarch 2005. Search strategies for all databasesare presented in the technical report of that project [6], theMEDLINE search strategy is presented in Appendix.

Additional efforts to identify instruments includedsearching in-house databases of reprints and correspon-dence with information specialists and other experts inthe field (June 2005eAugust 2006).

2.2. Study selection

Eligible studies were those that (1) reported an instru-ment that either evaluated or validated the quality of liter-ature searches, (2) provided guidance in the conduct orreporting of literature searches, or (3) reported empiricalevidence that informed the conduct of one or more aspectsof the electronic search for systematic reviews.

A screening calibration exercise was performed withthree members of the research team who rated 60 records

to establish consistency in ratings. After this exercise, titlesand abstracts were screened by one of the three reviewers.Articles appearing potentially relevant were retrieved andtwo reviewers assessed each of the full reports, arriving atconsensus regarding eligibility for any questions of thereview. Subsequent classification of eligible items wasconducted by one team member and verified by a secondteam member. The instruments that appeared suitable forguidance or assessment of the report of the search wereconsidered for this report.

2.3. Data extraction

Two members of the team identified the number andpool of items included in each instrument and their com-monality across instruments. We considered an instrumentto be evidence based if it cited research findings in supportof the included items, and to be validated if the authorsreported some form of validity or reliability research orsimply described the instrument as being validated.

2.4. Data analysis

Data were summarized descriptively. A correlation be-tween date of the instrument’s development and the numberof items included on it was undertaken post hoc to quantifya trend that was apparent in these data. No other correla-tions were undertaken.

2.5. Current practice

Recent systematic reviews were identified througha search of MEDLINE for systematic reviews indexed dur-ing November 2004 and written in English. We selected themost recent month before study commencement that in-cluded newly indexed Cochrane reviews after ascertainingthat this month was otherwise typical in terms of thenumber of new entries matching our search criteria. Weincluded Cochrane reviews because they are known to beof high quality, methodologically sound, and are peerreviewed against methodological criteria [7]. Searchinga month of MEDLINE that did not include a quarterly issuewould result in no Cochrane reviews being included. Thesearch to identify systematic reviews was conducted inMEDLINE (Ovid interface) 1966e2005 February Week2. We used Montori et al.’s balanced five-term search strat-egy (specificity O sensitivity) [8] and supplemented it withhis balanced three-term strategy (sensitivity O specificity),modified slightly to require that the string ‘‘methods’’ ap-pear in the abstract. We further limited the search to sys-tematic reviews reported in English. The search strategyis presented in Appendix.

Titles and abstracts of the studies were screened by onemember of the research team to determine whether the re-cord represented a systematic review, with a second re-viewer confirming ineligibility for any records excludedby the first reviewer. Any conflicts were resolved in favor

750 M. Sampson et al. / Journal of Clinical Epidemiology 61 (2008) 748e754

of advancement to the next stage, along with any recordsthat appeared to meet our definition and those where eligi-bility remained unclear. The full articles were then obtainedand final selection was made, based on the consensus deci-sion of two reviewers.

When it was clear that the intent of the authors was toconduct a narrative literature review (i.e., authors identifiedthe review as an overview with no specific review ques-tion), as opposed to a systematic review, articles wereexcluded. All other articles were included.

Data collection was completed using a form thatincluded 51 questions focusing on epidemiological anddescriptive characteristics, including those with a potentialfor bias (full results are reported elsewhere [5]). Two re-viewers independently pilot tested the form on a separateset of five articles. Each article was screened independentlyby one of three team members with a 5% sample beingscreened in duplicate by two reviewers. Any uncertaintieswere discussed and conflicts were resolved by consensus.

The initial data collection included data on all aspects ofsearch reporting addressed by five or more of the scales.Additional data were extracted on each aspect of searchreporting addressed by three or four instruments.

3. Results

3.1. Reporting scales

We identified 9,155 records for broad screening, ofwhich 256 full text articles were obtained for further assess-ment. Of these, 113 were eligible for some aspect of thesystematic review, with 28 of these records containing as-pects that could be used as search assessment instruments.Eleven of these records appeared useful for guiding orassessing a published report of the search for a systematicreview and were included in this review of instruments[4,9e18]. The 17 instruments that were not included fo-cused on searches in contexts other than reporting searchmethodsdoften assessing the electronic search strategy aspart of an evaluation of training effectiveness. The QUOR-OM flow diagram is presented in Fig. 1.

Three instruments limited their scope to aspects of thesearch; the remaining eight encompassed the entire reportwith the search as one aspect to be addressed (we labeledthese as global instruments). Seven (63.6%) of the instru-ments were evidence based, whereas four reported some formof validation in their development. Six (54.5%) instrumentswere intended to guide the conduct or reporting and five(45.5%) as appraisal instruments. Eight (72.7%) instrumentswere tailored for systematic reviews (SR), meta-analyses(MA), or health technology assessments (HTA), and three(27.3%) for clinical practice guidelines (CPG) (Table 1).

Most of the instruments published in 2004 or later wereevidence based, but few of them were validated. There doesnot appear to be as strong an emphasis on validation before

publication as in disciplines such as psychology. Validationefforts for one instrument have been described, includingface and content validity [18], and preliminary results forconstruct validity and reproducibility [19], thus demonstrat-ing that validation is possible.

3.2. Items

We identified 18 distinct search-related items addressedby the instruments (Table 1), as well as two minimum stan-dards (Table 2). Only one, databases searched, was com-mon to all 11 instruments. Check it out! [11] had themost items (n 5 11); Patrick et al.’s standard [12] had thefewest (n 5 4). Both addressed only the search. The othersearch-specific instrument included nine items [17]. Allitems that appeared on only one instrument appeared ona search-specific instrument, and each search-specific in-strument had one unique item. Across all instruments, bothglobal and search specific, the median number of includeditems was 7.5. There was a modest correlation (0.41) be-tween year of publication or production and the numberof items on the instrument, with the trend being towardmore items in more recently published instruments.

3.3. Current reporting practice

For the cohort of systematic reviews indexed in Novem-ber 2004, 1,064 records matched the search criteria. A totalof 755 records were obtained for further assessment and297 records were eligible. The QUOROM flow diagramand additional characteristics of these systematic reviews

11 Eligible for the review

9155 Records screened

256 Articles retrieved for further assessment

8858 Excluded - no apparent relevance

240 Articles excluded133 No apparent relevance85 Research evidence or theoretical

rationale pertaining to one or more elements of the electronic search, or prevalence of one or more types of search errors in the electronic search.

17 Inform the assessment or peer review of the electronic search strategy only

5 Not available, unable to assess further

Fig. 1. Systematic review study flowchartdsearch reporting instruments.

Table 1

Characteristics of the scales

Name Year Type of instrument Type of report Scope

No of search-related

items

Evidence

based Validated

AMSTAR [18] 2007 Appraisal instrument SR Global 8 Y Y

PRISMA [16] 2006 Reporting checklist SR or MA of RCTs Global 9 Y

Check it out! [11] 2005 Appraisal instrument HTA & CPG Search only 11

Cochrane Handbook [15] 2005 Conduct & reporting

standard

SR Global 8 Y

STARLITE [17] 2004 Reporting standard Qualitative SR Search only 9

Patrick [12] 2004 Appraisal instrument MA Search only 4 Ya

AGREE [13] 2001 Appraisal instrument CPG Global 6 Y Y

INAHTA [14] 2001 Reporting guideline HTA Global 7

Oxman & Guyatt [10] 2000 Appraisal instrument MA Global 5 Y

MOOSE [9] 2000 Reporting standard MAb Global 9 Y

QUOROM [4] 1999 Reporting checklist MA of RCTs Global 7 Y Y

a Most evidence cited is practice data.b MA of observational studies.

751M. Sampson et al. / Journal of Clinical Epidemiology 61 (2008) 748e754

are reported elsewhere [5]. Reporting of items present onthree or more of the instruments in the review is presentedin Table 3.

Almost all reviews examined here stated the databasesthat were searched. Vendor or platform was less frequently

Table 2

Items from the 18 instruments

Item

N of instruments

with item

Instruments including

this item

Databases used 11 (4; 9e18)

Dates covered by the search 10 (4; 9e16; 18)

Statement of the search terms

used

8 (9e14; 17; 18)

Statement of any language

restrictions

8 (4; 9; 11; 13e17)

Statement of nondatabase

methods used

7 (4; 9; 10; 13e15; 17)

Additional inclusion/

exclusion criteria

7 (4; 9e11; 13; 16; 17)

Presentation of the full

electronic search strategy

6 (11; 14e18)

Statement of any publication

status restrictions

5 (4; 9; 16e18)

Platform or vendor for

electronic databases

4 (9; 11; 15; 17)

End date of search 3 (11; 15; 16)

List of excluded references 3 (11; 14; 18)

Qualifications of the searcher 3 (9; 11; 15)

Is the reported strategy

repeatable?

2 (9; 15)

Number of references

identified

2 (11; 16)

CONSORT-style flow diagram

or other accounting for all

references

2 (4; 16)

Evidence of effectiveness of

the search strategy

1 (12)

Statement of any

methodological filters used

1 (11)

Description of the sampling

strategy

1 (17)

reported. In many cases where the platform was known, itwas because only PubMed or the Cochrane Central Regis-ter of Controlled Trials was searched. In most cases, thesearch platform could be determined when the full elec-tronic search strategy was presented. Thus, for an addi-tional 63 systematic reviews (21.2%), the knowledgeablereader could determine the platform used, however, it is im-portant for transparency that all readers understand whichplatform is being used.

Dates covered by the search were reported moderatelywell, and an additional 48 (16.2%) (Non-Cochrane: 32[19.0%], Cochrane: 16 [12.4%]) provided partial informa-tion on the dates covered by the search, such as reportingdates for only some databases, only start dates (such as‘‘1966 to present’’) or only end dates.

Most reports (85.6%) provided some description of theelectronic search but here Cochrane reviews and journalpublished reviews differed considerably. In Cochrane re-views, search strategies using a Boolean-based searchstrategy were the standard. In most instances, only theMEDLINE search was presented with an indication if otherdatabases were also searched. Journal published reviewsmost typically stated only a few key terms, sometimes assubject headings, sometimes as free text terms, a level ofreporting that is inadequate for critical appraisal or replica-tion in all but the simplest of review questions. Forty re-views (13.5%) gave no information on search terms used.Three reviews did not report searching a database.

Reporting of exclusion criteria, whether as search re-strictions or otherwise, exceeded 50% for all aspects exam-ined, although reporting of actual excluded studies wasrarely seen.

Finally, qualifications of the searcher were infrequentlyreported directly. This information could sometimes be de-termined in one of three ways: acknowledgment of a librar-ian’s assistance, reporting of which author undertook thesearch and listing of their academic credentials, usuallyin the byline, or in Cochrane reviews, by an explicit

752 M. Sampson et al. / Journal of Clinical Epidemiology 61 (2008) 748e754

statement that a trial search coordinator had undertaken thesearch.

4. Discussion

Most of the instruments published in 2004 or more re-cently are evidence based, but few are validated. Therewas not as strong an emphasis on validation before publica-tion as in other disciplines such as psychology. This may inpart be due to difficulty in obtaining funding to develop andvalidate instrumentsdboth aspects are essential.

Fewer instruments focusing on the search strategy ratherthan global instruments are validated or evidence based.The aspects of search-only instruments may be as goodas the global instruments but they have not been tested.This indicates the need for additional primary research inthe characteristics that result in valid unbiased searches. Itis not necessarily an indictment of search-specific instru-ments as we rated a search as valid or evidence based whenthere was any supporting research cited. The issue of searchreproducibility was not an item that was investigated in thisstudy. However, in a previous study, we reported that errorsin electronic search strategies of systematic reviews werecommon [20].

None of the instruments were exhaustive. The largest,Check it out! contained only 11 of the 18 items identifiedin this survey. Some item, such as reporting of language

Table 3

Reporting of items in recent systematic reviews

Item

Overall,

n (%)

Cochrane,

n (%)

Non-Cochrane,

n (%)

N 5 297 N 5 129 N 5 168

Databases used 293 (98.7) 129 (100.0) 165 (98.2)

Platform or vendor for

electronic databases

107 (36.0) 55 (42.6) 52 (31.0)

Dates covered by the search 207 (69.7) 107 (82.9) 100 (59.5)

End date of search 230 (77.4) 117 (90.7) 113 (67.3)

Statement of the search

terms useda,b254 (85.6) 127 (98.4) 127 (75.6)

Presentation of the full

electronic search strategy

127 (42.8) 101 (78.3) 26 (15.5)

Key terms stated 127 (42.8) 26 (20.2) 101 (60.1)

Statement of nondatabase

search methods used

271 (91.2) 129 (100.0) 142 (84.5)

Statement of any publication

status restrictionsa191 (64.3) 96 (74.4) 95 (56.4)

Statement of any language

restrictionsa166 (55.9) 80 (62.0) 86 (51.2)

Additional inclusion/

exclusion criteria stated

255 (85.9) 129 (100.0) 126 (75.0)

List of excluded references 168 (56.6) 125 (96.9) 43 (25.6)

Qualifications of the

searcher(s)

34 (11.4) 24 (18.6) 10 (6.0)

a Some of these items are reported elsewhere [5].b This item was divided in more or less complete reporting, either a full

electronic search strategy that could typically be replicated, or a more lim-

ited presentation of key terms, which is generally not sufficient to permit

replication of the search.

restrictions or filters, would seem redundant when the entiresearch strategy is reported, yet instruments listing the com-plete electronic search as an item also included reporting oflimits.

Two items are of particular interest as they extend be-yond the mechanical aspects of the search. Interestingly,each of these appeared on only one instrument. The firstwas sampling strategy, meaning whether the search wasdesigned to be comprehensive, selective, or purposive.Whereas, the approach used in quantitative systematic re-viewing of health care interventions is almost always com-prehensive in the sense of attempting to include every studythat meets the inclusion criteria, rather than purposive sam-pling of informative cases, it is reasonable for authors tostate that explicitly in the methods of the review. This itemmay become more important in the future if searches forrapid reviews or updating existing reviews adopt other ap-proaches, and would provide consistency between quantita-tive and qualitative reviews.

The other item of interest is Patrick’s ‘‘evidence of theeffectiveness of the search strategy.’’ In the original paper,Patrick considered three types of evidence of effectivenessof search strategies for meta-analyses: a previously vali-dated search, a published strategy, or strategy based on ex-pert opinion. While we contest that a published strategy isnecessarily evidence of effectiveness [20] and expert opin-ion is not strong evidence [21] we would encourage addi-tional research into methods to test the effectiveness ofsearch strategies, and reporting of evidence of effectivenessis an ideal worth pursuing.

Current reporting is below 80% for 9 of the 12 items thatappeared on three or more scales. Our methods were quiteliberal heredthis represents simply the presence of item inthe report, with no assessment of its adequacy. For instance,for ‘‘databases used’’ we recorded this as being reportedeven when there were clear errors, such as reporting a searchplatform such as Ovid as a database. For reporting of ex-cluded studies, Cochrane reviews almost always providedthe heading ‘‘References to studies excluded from this re-view;’’ however, this is not a full report of those retrievedby the search and excluded, as explained in the reviewer’shandbook sections 3.6.2 and 5.2.3 [15]. We find the headingmisleading and would prefer to see a more declarative head-ing such as ‘‘Partial list of studies excluded from thisreview.’’ We counted the qualifications of searchers as re-ported when their academic qualifications were stateddhowever, unless the individual was described as a trialsearch coordinator, librarian, or was credentialed as MLISor similar, we could not ascertain whether they had any train-ing or preparation to search, or were ‘‘talented amateurs.’’

We have not critically appraised the degree of evidentialsupport for all of the items put forth in these instruments.Two major reasons for including an item in a reportingstandard are that the item is important for replication, in-cluding in an update, or that evidence exists that the itemcan impact on review quality. Rigorous reporting of

753M. Sampson et al. / Journal of Clinical Epidemiology 61 (2008) 748e754

methods is essential for reproducibility. Reproducibilityhelps ensure the integrity of the results and permits updat-ing using the same methods to synthesize emerging evi-dence into the review. Almost all of the items presentedin these instruments support replication and updating, andthus may be justified without further evidence.

Some items, such as search dates, limits applied, and da-tabase selection, allow some assessment of the scope of theeffort to build the evidence, and thus its likely complete-ness. Other items, like qualifications of the searcher and ev-idence of effectiveness of the search strategy, address howcompetently the sources may have been searched. Evidenceshould guide the standards for such items. However, wenote that historically incomplete reporting of searchmethods is a barrier to assessing any such association be-tween the conduct of the search and the subsequent qualityof the systematic review. For example, we recently attemp-ted to determine if searches conducted by a librarian weremore or less likely to include a consequential error com-pared to searches conducted by those without such special-ization. Due to limits in reporting, we were only able tostudy the association between the presence of consequentialerrors and whether the person doing the search was identi-fied through authorship, acknowledgment, or a statement ofcontributorship; we were unable to ascertain the credentialsof searchers [22]. Adherence to the reporting standardcalled for by Check it out! [11] and MOOSE [9] wouldfacilitate such research.

The trend seen in these instruments toward requiring re-porting of more aspects of the conduct of the search is to beapplauded. Transparency allows the reader to assess theadequacy of the method based on the most comprehensiveand up-to-date evidencedcomplete reporting of reviewmethods today may enable those reviews to meet the stan-dards of future best evidence on the conduct of the search.

The main argument against more complete reporting isthat the increase in word count is not feasible or not accept-able to journal editors. The credibility of this argument isattenuating with the rise of electronic publishing and thepotential for providing web-only supplemental material,even for print-based journals. Editors of major journalshave overwhelmingly endorsed the CONSORT reportingstandard for randomized controlled trials, with associatedextra words.

The number of identified instruments now stands at 11,and there may be additional instruments we failed to iden-tify or currently in development. Producers and consumersof evidence-based summaries way benefit from fewerinstruments, and we urge the community to refine andcombine current instruments. A consensus-based processwould allow for an inclusive approach. Another benefit ofclear reporting of searching is as a learning tool for noviceresearchers and systematic reviewers. They can use thistool when they read systematic reviews and it will raisetheir awareness of the importance of the elements of theapproach to searching.

5. Conclusions

There is no clear consensus regarding optimum report-ing of systematic review search methods, although thereis a trend toward mandating more complete reporting. Suchtransparency facilitates replication, updating, critical ap-praisal, and methodological research toward optimal con-duct and reporting of searches used to form the evidencebase of systematic reviews.

This systematic review may be informative for those de-veloping and refining reporting guidelines and instruments.Meanwhile, these findings may aid individual authors oreditors seeking to optimize their own reporting, as well asthose undertaking critical appraisal of systematic reviews.We hope that this review will promote convergence ofcurrent efforts in defining optimal reporting of systematicreview searches.

Acknowledgments

We thank Andrea Tricco for assistance with screeningand data abstraction and Raymond Daniel for assistancewith document acquisition.

This research was funded in part by CADTH underCCOHTA’s 2005 Health Technology Assessment CapacityBuilding Grants Program and in part by the NationalLibrary of Medicine, National Institutes of Health(MZ402054). Funding body had a role in designing thestudy, in the collection, analysis, and interpretation of data;in the writing of the manuscript; or in the decision to submitthe manuscript for publication.

Competing interests: Several of the authors have been orare currently involved in the development of one or more ofthe instruments discussed in this report. Several of theauthors are also authors or contributors for one or moreof the systematic reviews included in the 2004 cohort.

Appendix

MEDLINE (Ovid) search strategy for the identificationof validated instruments to assess search strategies

1. search$.tw.2. exp ‘‘information storage and retrieval’’/ or Medical

Informatics/3. Data Compression/4. 2 not 35. or/1,46. quality control/ or Evaluation Studies/ or ‘‘reproduc-

ibility of results’’/7. (quality adj2 assess$).tw.8. or/6-79. (quality or evaluat$).tw.

10. (precision or recall or sensitivity or relevance orspecificity).tw.

11. (performance adj2 (measur$ or indicat$ orassess$)).tw.

754 M. Sampson et al. / Journal of Clinical Epidemiology 61 (2008) 748e754

12. ‘‘Sensitivity and Specificity’’/13. or/10-1214. (error$ or mistake$ or failure$ or inaccura$ or

misspell$).tw.15. (MeSH or (subject adj2 heading$) or (controlled

adj2 vocabulary) or redundan$ or explod$ or ex-plos$ or spell$ or Medline).tw.

16. truncat$.tw.17. (truncat$ adj6 (protein$ or gene$)).mp.18. 16 not 1719. exp Documentation/20. exp Molecular Sequence Data/21. 19 not 2022. or/15,18,2123. EVIDENCE-BASED MEDICINE/24. META-ANALYSIS/25. Review.pt.26. ‘‘Review Literature’’/27. or/23-2628. ‘‘INFORMATION STORAGE AND RETRIEVAL’’/

st [Standards]29. and/27-2830. and/5,8,1331. and/5,13-1432. and/9,13,2233. or/29-3234. ‘‘cochrane database of systematic reviews’’.jn.35. 33 not 3436. limit 35 to yr51980 e 2005

MEDLINE search to identify systematic reviews firstindexed in MEDLINE in November 2004, adapted fromMontori et al. [8].

1. 200411$.ed.2. limit 1 to english3. 2 and (cochrane database of systematic reviews.jn. or

search.tw. or meta-analysis.pt. or medline.tw. or system-atic review.tw. or ((meta-analysis.mp,pt. or review.pt.or search$.tw.) and methods.ab.))

References

[1] Brazier H. Poorly executed and inadequately documented? An anal-

ysis of the literature searches on which systematic reviews are based.

2nd Symposium on Systematic Reviews: Beyond the Basic, 1999.

[2] Golder S, Loke Y, McIntosh HM. Room for improvement? A survey

of the methods used in systematic reviews of adverse effects. BMC

Med Res Methodol 2006;6:3.

[3] Sampson M, McGowan J. Reporting of the role of the expert searcher

in Cochrane reviews. Melbourne, Australia: XIII Cochrane Collo-

quium; 2005.

[4] Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF,

et al. Improving the quality of reports of meta-analyses of random-

ized controlled trials: the QUOROM statement. Lancet 1999;354:

1896e900.

[5] Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemi-

ology and reporting characteristics of systematic reviews. PLoS Med

2007;4(3):e78.

[6] Sampson M, McGowan J, Lefebvre C, Grimshaw JM, Moher D.

PRESS: (Peer Review Electronic Search Strategy). CADTH Tech-

nical Report 2007. Available at www.cadth.ca.

[7] Ruddy R, House A. Meta-review of high-quality systematic reviews

of interventions in key areas of liaison psychiatry. Br J Psychiatry

2005;187:109e20.

[8] Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal

search strategies for retrieving systematic reviews from Medline: an-

alytical survey. BMJ 2005;330(7482):68.

[9] Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD,

Rennie D, et al. Meta-analysis of observational studies in epidemiol-

ogy: a proposal for reporting. Meta-analysis of observational studies

in epidemiology (MOOSE) group. JAMA 2000;283(15):2008e12.

[10] Shea B, Dube C, Moher D. Assessing the quality of reports of sys-

tematic review: the QUOROM statement compared to other tools.

In: Egger M, Smith GD, Altman DG, editors. Systematic reviews

in health care: meta-analysis in context. London: BMJ Publishing

Group; 2001. p. 122e39.

[11] Jensen MF, Ket H. Check it out! A checklist for evaluating the report-

ing of literature search methodology in HTAs and CPGs [Poster pre-

sentation]. Second Annual Meeting of the Health Technology

Assessment International (HTAi) Rome, Italy, 2005.

[12] Patrick TB, Demiris G, Folk LC, Moxley DE, Mitchell JA, Tao D.

Evidence-based retrieval in evidence-based medicine. J Med Libr

Assoc 2004;92(2):196e9.

[13] The AGREE Collaboration. Appraisal of guidelines for research &

evaluation (AGREE) instrument. London: St. George’s Hospital

Medical School; 2001.

[14] International Network of Agencies for Health Technology Assess-

ment. INAHTA HTA Checklist, 2001.

[15] Higgins JPT, Green S. Cochrane handbook for systematic reviews of

interventions 4.2.5. Updated 2005. Available at http://www.cochrane.

org/resources/handbook/hbook.htm. Accessed July 8, 2008.

[16] Moher D, Tetzlaff J, Liberati A, Altman DG, for the PRISMA group.

Recommendations for reporting systematic reviews of randomized

trials: the PRISMA Statement. In preparation, 2008.

[17] Booth A. ‘‘Brimful of STARLITE’’: toward standards for reporting

literature searches. J Med Libr Assoc 2006;94:421.

[18] Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C,

et al. Development of AMSTAR: a measurement tool to assess the

methodological quality of systematic reviews. BMC Med Res

Methodol 2007;7:10.

[19] Shea B, Grimshaw JM, Wells G, Boers M, Hamel C, Porter A, et al.

Advancement of the methodological quality through a measurement

tool to assess reviews. Dublin, Ireland: AMSTAR. XIV Cochane

Colloquium; 2006.

[20] Sampson M, McGowan J. Errors in search strategies were identified

by type and frequency. J Clin Epidemiol 2006;59:1057e63.

[21] Phillips B, Ball C, Sackett D, Badenoch D, Straus S, Haynes RB,

et al. Oxford centre for evidence-based medicine levels of evi-

dence. Updated 2001. Available at http://www.cebm.net/levels_of_

evidence.asp. Accessed November 5, 2007.

[22] Zhang L, Sampson M, McGowan J. Reporting of the role of the

expert searcher in Cochrane reviews. Evid Based Libr Info Pract

2006;1(4):3e16.