![Page 1: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/1.jpg)
Reproducibility of Search Strategies in Systematic ReviewsPreliminary Results
Jonathan Koffel, MSIBio-Medical Library, University of Minnesota
Melissa Rethlefsen, MLIS AHIPEccles Health Sciences Library, University of Utah
![Page 2: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/2.jpg)
Rise of the Systematic Review
2003 2004 2005 2006 2007 2008 2009 2010 2011 20120
1000
2000
3000
4000
5000
6000
7000
8000
9000Meta-Analysis[pt]
DARE Citations
Reporting Guidelines:PRISMAMOOSECochraneInstitute of Medicine
![Page 3: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/3.jpg)
Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.
Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.
Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 2009
![Page 4: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/4.jpg)
“The Scopus, EBSCOhost, Ovid, and Web of Science platforms were searched.”
Pediatrics. 2012 Mar;129(3):548-56
![Page 5: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/5.jpg)
Previous Research• Good reporting of some search elements2
– Database names, search terms
• Poor reporting of other elements2
– Date search executed, database platform
• Reproducibility is heterogeneously defined, but generally low3,4
2 Sampson et al. J Clin Epidemiol. 2008 Aug;61(8):748-54.3 Yoshii et al. J Med Libr Assoc. 2009 Jan;97(1):21-9.4 Golder et al. J Clin Epidemiol. 2013 Mar;66(3):253-60
![Page 6: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/6.jpg)
Purpose
1. Examine search reporting and reproducibility in systematic reviews published in high-impact medicine journals in 2012
2. Investigate the impact of librarian involvement and reporting guidelines on reporting/reproducibility rates
![Page 7: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/7.jpg)
Hypotheses
1. Reporting/reproducibility rates will be higher when a librarian was involved.
2. Reporting/reproducibility rates will be higher when a guideline was mentioned or required by the journal.
![Page 8: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/8.jpg)
Journals Included
Cardiology• Circ Cardiovasc Genet• Circ Cardiovasc Interv• Circ Heart Fail• Circ Res• Circulation• Eur Heart J• J Am Coll Cardiol• JACC Cardiovasc Imaging• JACC Cardiovasc Interv• Nat Rev Cardiol
Surgery• Am J Surg Pathol• Am J Transplant• Ann Surg• Ann Surg Oncol• Arch Surg• Br J Surg• Endoscopy• J Am Coll Surg• J Neurol Neurosurg
Psychiatry• Surg Obes Relat Dis
Pediatrics
• Arch Dis Child• Arch Dis Child Fetal
Neonatal Ed• Arch Pediatr Adolesc Med• Eur Child Adolesc
Psychiatry• J Am Acad Child Adolesc
Psychiatry• J Pediatr• Pediatr Allergy Immunol• Pediatr Infect Dis J• Pediatrics• Semin Fetal Neonatal Med
![Page 9: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/9.jpg)
Identification of Studies
• PubMed search on 9/2/2013 using modified Montori strategy5
(search*[Title/Abstract] OR meta-analysis[Publication Type] OR MEDLINE[Title/Abstract] OR EMBASE[Title/Abstract] OR Meta-analysis[Title/Abstract] OR (systematic[Title/Abstract] AND review[Title/Abstract]) AND (("2012/01/01"[PDat] : "2012/12/31"[PDat])
• PubMed SR subset filter also used for comparison– <4% of included articles found only by Montori strategy
5 Montori et al. BMJ. 2005 Jan 8;330(7482):68.
![Page 10: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/10.jpg)
Identification of Studies
• All retrieved studies screened by both authors to identify systematic reviews– Searches at least one published literature database– Inclusion/exclusion criteria are prespecified – Does not limit to a certain number of journals or subset of
journals
• Disagreements resolved by review of original articles and consensus
![Page 11: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/11.jpg)
Cardiology Surgery Pediatrics
216 317 201
51 142 77
Retrieved
Included
270
![Page 12: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/12.jpg)
Items Identified
• Items covered:1. Search elements (eg., terms, strategy, limits)
2. Librarian involvement (author and role)
3. Guideline mentioned
![Page 13: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/13.jpg)
Data Extraction
• Data extraction form created in Qualtrics• Included articles split between authors• Initial entry completed in March, 2014
![Page 14: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/14.jpg)
Analyses
• SPSS used for all analyses– Logistic regression (odd ratio)– Chi Square/Fisher (p-value)– Mann-Whitney/T-Test (means)– α =.01
![Page 15: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/15.jpg)
Search Updated
Search Execution Date
Full Strategy Named
Booleans
Start Year
Providers Named
Start/Last Year
Limits Indicated
Last Year
Specific Terms Named
Databases Searched
0% 25% 50% 75% 100%
3%
22%
34%
44%
60%
68%
73%
75%
81%
89%
100%
No significant differences based on discipline
![Page 16: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/16.jpg)
Search Updated
Search Execution Date
Full Strategy Named
Booleans
Start Year
Providers Named
Start/Last Year
Limits Indicated
Last Year
Specific Terms Named
Databases Searched
0% 25% 50% 75% 100%
3%
22%
34%
44%
60%
68%
73%
75%
81%
89%
100%
0.16
0.31
0.6
0.67
0.76
0.98
Current StudySampson 2008*
*
*
*
*
* p<.01 for comparison
Sampson et al. J Clin Epidemiol. 2008 Aug;61(8):748-54.
![Page 17: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/17.jpg)
Librarian Involvement
4%
12%
83%
1%Librarian Author
Librarian in Text/Acknowledgement
Librarian Not Men-tioned
Unknown
Cardiology Surgery Pediatrics0%
5%
10%
15%
20%
25%
30%
35%
40%
0%
13%
34%
P<.001P<.01
P<.001
![Page 18: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/18.jpg)
49%51%
Reporting Guideline Mentioned
Guideline Men-tionedGuideline Not Men-tioned
84%
16%
PRISMA Flowchart Provided
FlowchartNo Flowchart
62%
38%
Journal Requires PRISMA
PRISMA RequiredPRISMA Not Required
![Page 19: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/19.jpg)
Conservative Reproducibility
• Includes full search strategy for at least one database– Database searched– Provider– Date range– Search date– Terms– Booleans– Indication of limits
![Page 20: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/20.jpg)
Liberal Reproducibility
• Includes core search elements– Database– Date range– Specific terms– Indication of limits
![Page 21: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/21.jpg)
Reproducibility Rates
• Reproducibility rates <8% in prior studies– Previous criteria mirrored conservative criteria
Conservative Liberal
Surgery 11% 56%
Pediatrics 12% 57%
Cardiology 12% 39%
Total 11% 53%
![Page 22: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/22.jpg)
Librarian Guideline Journal
Database Named ns ns ns
Databases Searched MD 1.34 (.78-1.89)*** ns ns
Provider Named ns ns ns
Search Years Indicated ns ns ns
Search Date ns ns ns
Search Updated ns ns ns
Search Terms OR .2 (.08-.49)*** ns ns
Full Strategy Presented ns ns ns
Boolean ns ns ns
Limits ns ns ns
Conservative Reproducibility
ns ns ns
Liberal Reproducibility ns ns ns
**p<.01 ***p<.001Discipline included in regression
![Page 23: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/23.jpg)
Librarian Author/None Librarian Text/None
Database Named ns ns
Databases Searched MD 2.8 (1.8-3.9)*** MD 1.3 (.7-2.0)***
Provider Named n/a ns
Search Years Indicated ns ns
Search Date ns ns
Search Updated OR .08 (.01-.52)** ns
Search Terms OR .16 (.04-.71)** OR .18 (.07-.48)***
Full Strategy Presented ns ns
Boolean ns ns
Limits ns ns
Conservative Reproducibility
ns ns
Liberal Reproducibility ns ns
**p<.01 ***p<.001Discipline, Journal Requirements and Guideline Mentioned included in regression
![Page 24: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/24.jpg)
Results
• Reproducibility is not predicted by:– Librarian involvement– Journal requirements– Guidelines mentioned in the paper
![Page 25: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/25.jpg)
Results
• Librarian Involvement– Increases number of databases searched– Decreases updating of search results
when librarian is an author– Decreases reporting of search terms
![Page 26: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/26.jpg)
Limitations
• Published information– Librarian could be involved, but not
mentioned• Ambiguous search elements
– Booleans? Limits? Full strategy?• Limited to three medical disciplines and
a single year
![Page 27: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/27.jpg)
Next Steps
• Duplicate entry of data• Examination of reporting in greater
depth– Publication lag– Presentation of elements for more than
one database
![Page 28: Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa](https://reader030.vdocuments.mx/reader030/viewer/2022032518/56649ccb5503460f9499404f/html5/thumbnails/28.jpg)
Conclusions
1. Librarian involvement does not intrinsically mean better reporting
2. We must work with authors and publishers to improve reporting