beyond medline for literature searches

Journal of Nursing Scholarship Second Quarter 2003 177

Clinical Scholarship

Beyond MEDLINE for Literature SearchesVicki S. Conn, Sang-arun Isaramalai, Sabyasachi Rath, Peeranuch Jantarakupt,Rohini Wadhawan, Yashodhara Dash

Purpose: To describe strategies for a comprehensive literature search.Organizing Construct: MEDLINE searches result in limited numbers of studies that are often

biased toward statistically significant findings. Diversified search strategies are needed.Methods: Empirical evidence about the recall and precision of diverse search strategies is

presented. Challenges and strengths of each search strategy are identified.Findings: Search strategies vary in recall and precision. Often sensitivity and specificity are

inversely related. Valuable search strategies include examination of multiple diversecomputerized databases, ancestry searches, citation index searches, examination of researchregistries, journal hand searching, contact with the “invisible college,” examination of abstracts,Internet searches, and contact with sources of synthesized information.

Conclusions: Extending searches beyond MEDLINE enables researchers to conduct moresystematic comprehensive searches.

JOURNAL OF NURSING SCHOLARSHIP, 2003; 35:2, 177-182. ©2003 SIGMA THETA TAU INTERNATIONAL.

[Key words: literature searching, publication bias, research synthesis]

* * *

Athorough review of existing scientific literature isessential for determining the evidence for practice andfor designing research that will fulfill the potential of

cumulative knowledge. Many nurses conducted literaturesearches as students, but usually those searches were of limitedscope. Often more comprehensive searches are necessary tofully assess the state of existing knowledge. This paper isfocused on diverse strategies to conduct an extensive literaturesearch.

Rationale for Comprehensive Literature Searching

The volume of scientific literature has grown exponentiallyin the last 50 years. Practicing nurses are expected to basetheir nursing care on scientific evidence. Nurse researchersstrive to conduct studies that will provide the scientificevidence to improve nursing care. Neither can fully meettheir goals if the literature they retrieve is narrow in scope oris not representative of existing science.

Finding the existing science is challenging. Many nursesbegin the search with computerized databases, typicallyMEDLINE, CINAHL, and PsycINFO. These are excellentstarting points. Unfortunately, searches that include only thesesources exclude many valuable studies and the researchreports retrieved by these databases will be a biased sampleof existing studies. If the studies that were easily retrievedthrough the most popular computerized databases accuratelyrepresented all research, searching these databases would be

Vicki S. Conn, RN, PhD, Alpha Iota, Potter-Brinton Professor & AssociateDean for Research, Sinclair School of Nursing, University of Missouri,Columbia, MO; Sang-arun Isaramalai, RN, PhD, Assistant Professor, PrinceSongkla University, Songkla, Thailand; Sabyasachi Rath, MBBS, GraduateStudent, Health Informatics, Peeranuch Jantarakupt, RN, MS, Alpha Iota,Doctoral Candidate, School of Nursing, Rohini Wadhawan, BA, BDS,Graduate Student, Health Services Management, Yashodhara Dash, MBBS,Graduate Student, Health Informatics, all at the University of Missouri,Columbia, MO. Financial support was provided by a grant from the NIHNINR RO1NR07870 to Vicki Conn, principal investigator. Correspondenceto Dr. Conn, S317 School of Nursing-MU, Columbia, MO 65211. E-mail:[email protected]

Accepted for publication January 7, 2003.

adequate. However, extensive evidence shows that these studiesmore often include statistically significant findings than doequally rigorous research found in other locations (Chalmerset al., 1990; Dickersin, Chan, Chalmers, Sacks, & Smith,1987; Dickersin, Min, & Meinert, 1992; Easterbrook, Berlin,Gopalan, & Matthews, 1991; Gotzsche, 1989; Hubbard &Armstrong, 1997; Kleijnen, & Knipschild, 1992; Rosenthal,1979; Simes, 1986, 1987; Serling, 1959; Serling, Rosenbaum,& Weinkam, 1995; Sugita, Kanamori, Izuno, & Miyakawa,1992). Limited searches are also less likely to identify studieswith small samples, which are often intriguing pilot studiesof innovative interventions or studies conducted with difficult-to-recruit samples (Chalmers et al., 1990; Dickersin et al.,1987; Easterbrook et al., 1991; Scherer, Dickersin, &Langenberg, 1994; Thornton & Lee, 2000). Thus reviewingonly the easily located studies leads to a biased view of existing

178 Second Quarter 2003 Journal of Nursing Scholarship

Moving Beyond MEDLINE

research. A balanced review of existing studies requires moreextensive searching to locate a broader scope of research.

Comprehensive search strategies are needed to locate largenumbers of studies for extensive or systematic reviews, suchas would be necessary for a valid meta-analysis. More limitedstrategies might be appropriate for other types of literaturereviews.

Managing a Comprehensive Literature Search

A clearly articulated plan is essential for conducting acomprehensive search. A written search protocol is helpful,based on preliminary perusing of computerized databasesrelated to clear research questions. The objectivity of thereview will be enhanced if a priori inclusion criteria and termsare specified. For example, in a search for studies ofinterventions to increase physical activity, two concepts wouldbe required if the title were: physical activity and intervention.Terms that would indicate physical activity include exercise,exertion, exercise therapy, physical education, physicaltraining, aerobic, endurance, or fitness. Intervention termsinclude behavior, adherence, compliance, patient education,health promotion, health education, behavior therapy,lifestyle, program, evaluation, effect, outcome, intervention,or treatment.

A system to manage the search process and products isnecessary. Bibliographic software, such as EndNote orReference Manager, is useful for both predefined andcustomized fields. For example, a field named “status” couldbe added to indicate whether the article has been obtained,or is needed from a library. Another field labeled “outcomemeasure” might indicate specific outcomes of interest for thereview. Term lists that are linked to fields can be created tostructure the language. Structured language can be helpfulfor later searching within the data file.

The nature of the review will determine whether multiplefields and lists of associated terms are worthwhile. Highlydifferentiated fields and standardized term lists facilitate acomprehensive meta-analysis of a field of research with manypotential eligible studies. Bibliographic software can be usedto record when report-linked search and review activities havebeen completed, such as ancestry searches, coding, or authorcontacts. Overlap between products of search strategies willlikely be extensive. Bibliographic software that can locateduplicate entries is useful to make the process as efficient aspossible.

A staged method of determining that studies are eligiblefor the review can be helpful (Cook et al., 2001). Potentialeligibility would indicate a research report with a title orabstract indicating that the study might meet the criteria forinclusion. This judgment can often be made from abstractswhen studies are available through computerized databases.“Probable” eligibility would be determined after reading theentire research report. “Definite” eligibility would occur wheninformation from the research report is determined to be usefulin the review. For example, in literature reviews for a meta-

analysis, definite eligibility would be declared when data-coding shows that the independent and dependent variablesand effect size can be extracted from the research report.

Search strategies should be completely documented toprevent duplication of effort. Ample time must be allowedfor contacting authors for information about additional studies.Systematic literature reviews are the first step in systematicresearch.

Search Strategies

Diverse search strategies are available to locate research.Recall and precision are accepted measures of the utility ofinformation retrieval (Eysenbach, Tuische, & Diepgen, 2001).Recall—the proportion of relevant studies retrieved by searchstrategies—indicates the sensitivity of the search (Eysenbachet al., 2001). A comprehensive search will have high recall.Precision is the proportion of relevant studies in the retrievedset (Eysenbach et al., 2001; Hersh & Hickam, 1993). Precisionindicates the specificity of the search. Recall and precisionvary dramatically, often inversely, by search strategy. Thepurpose of the literature search will determine the appropriatebalance between recall and precision. The Table showsimportant attributes of some search strategies.

Table. Attributes of Strategies for Literature SearchesStrategy Precision Recall Comments

Computerized High for limited Generally high Experienced librarians databases databases, lower for can assist with complex

general databases structured languages.Ancestry Moderate High Does not generally broadensearches the diversity of studies.Citation index Very low for broad High Most efficient for little-searches topics, high for narrow, studied areas with few

highly specific topics seminal works.Research High High Excellent source that registries avoids publication bias.Journal hand High with well- High Labor intensive but yieldssearches trained searchers higher than computerized

databases. Invisible college High Low to moderate May locate student contacts depending on projects and reports not

response rate yet published or indexed.Presentation High for highly High for Less promising results abstract searches specialized specialized for general conferences.

conferences conferences

Internet searches Low unless Low May locate “grey” literaturespecialized terms and studies in progress.

Computerized Database SearchingComputerized database searching using MEDLINE,

CINAHL, and PsycINFO are familiar to many nurses. Morriseyand DeBourgh (2001) provided an excellent introduction to thisform of searching. This discussion will focus on issues relatedto conducting a comprehensive search. Search skills improvewith practice and most database users achieve satisfactory recallof relevant citations. This situation sometimes leads to whatBooth and O’Rourke (1999) referred to as the “satisfied but



inept searcher.” Many users overestimate the quality of theirsearching performance (Hersh & Hickam, 1993). Experiencedreference librarians typically retrieve about twice as manycitations as do less experienced users (Hersh & Hickam, 1993),because databases are highly structured with complex indexingrules (Dickersin, Scherer, & Lefebvre, 1994). Experiencedsearchers are also best able to generate the independent searchstrategies sometimes required for varied databases because oflack of standardization of databases (Sindhu & Dickson, 1997).A reference librarian can assist with developing free-textstrategies for databases without well-constructed thesauruses(Hek, Langton, & Blunden, 2000). Thus one important strategyfor conducting a comprehensive search is to enlist the assistanceof an experienced health sciences reference librarian. Anotherimportant strategy is to treat searching as an iterative processin which eligible studies are examined for potential keywordsor indexing terms to further develop the search process (Sindhu& Dickson, 1997). Acceptance that a highly sensitive search(high recall) will likely have low specificity (diminished precision)is necessary for comprehensive searching.

Still, even searches conducted by highly skilled librariansresult in only about half of the eligible citations (Avenell,Handoll, & Grant, 2001; Dickersin et al., 1994; Jadad, Moher,& Klassen, 1998; McManus et al., 1998; Poynard & Conn,1985; Sindhu & Dickson, 1997). Indexing problems most likelycause this low yield. Poor reporting, coding systems that donot address some topics, new areas of science without indexterms, inadequate descriptions of research to allow indexersto correctly classify studies, and inevitable lack of consistencyamong coders all contribute to inadequate indexing (Dickersinet al., 1994; Jadad et al., 1998; McManus et al., 1998). Dataconsistently indicate that MEDLINE searches will not retrieveall the citations found by hand-searching journals indexed byMEDLINE. Furthermore, significant numbers of studies arenot indexed in MEDLINE. McDonald and colleagues (2002)found that 30% (n=6,554) of reports of randomized controlledtrials located by extensive hand searching for the CochraneCollaboration were indexed in MEDLINE. Sensitivity andprecision of database searches vary dramatically by scientificarea (Dickersin et al., 1994).

Research on the effectiveness of computerized databasesearching has been almost entirely focused on MEDLINE.CINAHL has been studied infrequently. One search ofnonpharmacologic nursing interventions resulted in 14% ofeligible studies using CINAHL, but all the CINAHL-identifiedstudies were also obtained through other sources (Sindhu &Dickson, 1997). Another nursing search produced no citationsthrough CINAHL, but citations were readily retrieved fromMEDLINE and other databases (Avenell et al., 2001).

Numerous specific databases are available. For example, arecent meta-analysis of interventions to increase physical activityamong aging adults included searches of an exercise-specificdatabase (SPORT Discus) and American Association of RetiredPersons’ Ageline for gerontology research (Conn, Valentine, &Cooper, 2002). Databases that will allow access to internationalstudies are important. For example, EMBASE (Excerpta MedicaDatabase) includes journals from 70 countries (Dickersin et al.,

2002; Hek et al., 2000). The overlap between MEDLINE andEMBASE is only 34% (Dickersin et al., 2002). Databases outsidehealth care might be useful depending on the topic (e.g.Educational Resources Information Center [ERIC]). Databasesnot primarily focused on research might also be useful. Forexample, CHID (Combined Health Information Database)contains information about health promotion programs thatcould be useful in research on health behavior interventions.More general databases will increase the sensitivity of a search,but precision will decline (Dickersin et al., 1994). An experiencedreference librarian can assist with decisions about computerizeddatabases.

Ancestry SearchingAncestry searching refers to the systematic review of citations

from studies included in the review and from review articles.The most thorough approach is to review the text of the researchreport, especially the literature review and discussion, forcitations to other potentially useful studies. Then each referenceis reviewed for other potentially useful studies. These dualstrategies are more effective than simply reviewing the referencelist because titles often misrepresent content (Cooper & Ribble,1989; Sindhu & Dickson, 1997). Ancestry searches expand thenumber of eligible studies. However, reliance on ancestrysearching without adequate attention to other search strategiescan yield a biased set of studies. Studies with statisticallysignificant findings are more likely to be cited in reference liststhan are studies without significant findings (Gotzsche, 1987).For example, Ravnskov (1992) found that studies to testcholesterol-lowering interventions to prevent coronary diseasepublished after 1970 included citations of positive or inconclusivestudies, but they never included contrary evidence. Referencelists of studies generally contain studies similar to the one in theoriginal report. Thus an ancestry search might enlarge thenumber of retrieved studies but it does not always broaden thecharacteristics of studies in the review. Ancestry searching tendsto have high recall and moderate precision.

Citation-Index SearchingCitation-index searching is based on the cumulative nature

of research. Citation indexes include references used inpublished articles. This form of searching is most effective ifa few important works are widely cited by other researchersand are not widely cited by those not conducting similarresearch. Because citation indexes are based on the citingpractices of authors, they generally yield studies similar tothe original citation. Citation-index searching might be helpfulin identifying subject headings for MEDLINE searches. Recalland precision vary dramatically by topic area.

Searching Research RegistriesResearch registries are a high-recall and high-precision

source of information about studies. Research registriesprovide information about studies regardless of the statisticalsignificance of findings because registration occurs beforestudy completion (Egger & Smith, 1998). Specialtyorganizations often manage registries. The Cochrane



CENTRAL register contains information about more than300,000 controlled trials (Lefebvre et al., 2002). Advocatesof research synthesis suggest mandatory research registrationto avoid publication bias. In Australia, all studies approvedby ethics committees are entered into a prospective studyregistry (Stern & Simes, 1997). Registries are a valuablesource of information about studies.

Journal Hand SearchingJournal hand searching requires examining journal article

titles, abstracts, and complete articles. Journals with highpotential recall are selected for hand searching because amajority of eligible studies in a systematic review are from asmall number of journals (Hek et al., 2000). More variedstudies can be retrieved by searching more diverse journals.Given the laborious nature of hand searching, researchershave attempted to document whether hand searching yieldsmore studies than do other methods. Studies consistently haveshown higher recall from hand searching than throughcomputerized databases and other search strategies (Dickersinet al., 2002; Helmer, Savoie, Green, & Kazanjian, 2001;McManus et al., 1998; Poynard & Conn, 1985). Langham,Thompson, and Rowen (1999) reported that they found 84%of the total articles in their review by hand searching, butonly 68% through MEDLINE. The Cochrane collaborationidentified 18,000 controlled trials, not indexed in MEDLINE,through hand searching European journals (Lefebvre et al.,2002). Hand searching has both high precision and strongrecall when well-trained reviewers conduct the searches.Journal hand searching may yield publications that have notyet been indexed in computerized databases (Avenell et al.,2001). Labor intensiveness is the predominant disadvantageof hand searching (Jadad et al., 1998).

Contact with the “Invisible College”The “invisible college” is composed of geographically

dispersed investigators conducting research in a specific areaof science. Experts are easily identifiable from repeatedpublications. Contact with experts might be used to solicitthe names of others who should be contacted (Sindhu &Dickson, 1997). Experts in the field can identify additionalrelevant published and unpublished research (Helmer et al.,2001; Jadad et al., 1998). Experts are often aware of relevantgraduate student work that might not be published but wouldbe appropriate for inclusion. McManus and colleagues (1998)found that 24% of the studies included in their systematicreview would have been missed entirely if experts had notbeen contacted. Contacting experts yields high precision andvariable recall, depending on response rates. The largestdrawback is variable response rates. McManus and colleagues(1998) reported a 53% response rate for queries sent toacademic researchers. Promising copies of the review if expertsprovide citations might increase response rates (Conn et al.,2002).

Searching Conference ProceedingsPresentations at conferences are based on recently completed

research, but often reports of these studies do not appear inthe literature for several years. Many are never published,especially those without statistically significant findings(Callaham, Wears, Weber, Barton, & Young, 1998). Someassociations’ journals include conference abstracts in regularor supplemental issues. Generally, bound copies of proceedingsare available from large organizations. Once studies areidentified, contact with authors might yield unpublished fullreports, prepublication full reports, or copies of presentationmaterials with additional information about the studies. Thissearch strategy produces high recall and precision withspecialized conferences closely linked to the scientific topic(Langham et al., 1999; McManus et al., 1998). Less promisingresults are obtained from more general conferences.

Internet SearchingWorld wide web (WWW) searching is valuable for locating

studies in progress or recently completed that are not yetpublished, presentations at conferences, and researchers whomight be contacted for other studies, electronic journal articles,and other links for information. For example, Eysenbach andcolleagues (2001) found clues concerning unpublished studieson departmental or institutional homepages with descriptionsof research activities, researchers’ personal homepages, onlineconference proceedings or abstracts, announcements aboutgrant awards, press releases by departments or institutionsabout ongoing or recently completed studies, online publishedresearch reports, and Websites for recruiting participants.WWW searching might yield high recall for unpublished andgrey literature (studies not formally published or publishedin difficult to access locations).

Suitable search engines must be powerful enough to handlecomplex queries because Web documents are not indexed withkey words in a controlled vocabulary (Eysenbach et al., 2001).The large number of irrelevant documents on the WWWnecessitates carefully designed and highly specific searchstrategies (Eysenbach et al., 2001). Useful search engines mustbe able to truncate word stems, link synonyms with an “or,”and conduct proximity search with “near.” Searching is aniterative process in which results from one attempt are usedto expand or narrow a subsequent search. “Near” generallylimits the search more than “and.” Spelling variants mightresult in more hits. The number of possible synonyms can beincreased to further expand the search. Similarly, searchesmay be narrowed by reducing the number of synonyms,removing truncated terms or providing longer word stems,and sometimes by using “not” to exclude sites with particularterms (Eysenbach et al., 2001; Robinson & Dickersin, 2002).Eysenbach and colleagues (2001) provided an excellentoverview of these strategies and examples of searches.

Despite rapid proliferation of search engines, Internetsearching remains challenging (Sigouin & Jadad, 2002). Nosingle search engine includes all of the indexable WWW sites(Lawrence & Giles, 1998). Scant work has addressed theprecision and recall of Internet searching. Eysenbach andcolleagues (2001) searched with Alta Vista to find studies forCochrane reviews. They reviewed 429 Web pages to find clues



to 14 studies. The specificity of the topic determines whetherthe “signal-to-noise ratio” makes WWW searching useful.Researchers should be aware that the quality of informationon the Internet varies dramatically (Eysenbach & Diepgen,1998; Hersh, Gorman, & Sacherek, 1998).

Securing International LiteratureNorth American and British studies are most easily retrieved

through standard computerized databases and ancestrysearches. Searching beyond these traditional sources isnecessary to broaden the base of science available for nursingknowledge. For example, an in-progress meta-analysis isfocused on the effects of interventions to increase physicalactivity among adults with chronic illnesses (Conn, 2002-2005). Through extensive literature searching the researchershave retrieved potentially useful research reports in 87 journalspublished outside North America and the United Kingdom.

Depending on the purpose of the literature review, reportspublished in languages other than English are needed.Dickersin and colleagues (1994) found that 20% of the studiesfor which they found citations were not in English. Searchinginternational literature is consistent with the global focus ofnursing. Translators with university affiliations can beespecially helpful, because of their scientific knowledge, inmanaging research reports in other languages.

Journals in some countries have enormous linguistic, financial,and production difficulties (Zielinski, 1995). World HealthOrganization, UNESCO, and the Food and AgricultureOrganization have developed projects such as ExtraMED,ExtraSCI, and AgROM Extra to help address the problem ofaccessing research completed in developing countries (Zielinksi,1995). Grey literature is difficult to locate. SIGLE (System forInformation on Grey Literature in Europe) is a bibliographicdatabase covering European nonconventional (so-called grey)literature in the fields of pure and applied natural sciences andtechnology, economics, social sciences, and humanities. Thepurpose of a literature review will indicate whether extensivesearching for research across the globe is necessary.

Finding the Fugitive LiteratureMany search strategies will reveal clues to other studies that

might be useful. For example, in the area of health promotionresearch, one common hint is a description of a program withoutoutcome data. Or an article might include outcomes other than theones of interest to the reviewers. The clues provide a beginningpoint for a search for the fugitive literature. The first strategy is tocarefully inspect the reference list for a possible relevant citation.Computerized searches for all authors of the reference might leadto additional reports eligible for inclusion in the review. Sometimesunusual words are used that can be searched in text fields ofcomputerized databases (e.g., calling an exercise programWOMENACT). Studies may be located in the National Institutesof Health database of funded studies (Computer Retrieval ofInformation on Scientific Projects [CRISP]) by searching for authornames. Authors may be contacted to seek further information. Acareful system of recording completed strategies will make the searchfor fugitive literature manageable.

References to research reports are often found outsidetraditional peer-reviewed journals. Other possible sourcesinclude company reports and press releases, governmentmonographs and reports, policy documents, publications ofspecial panels or commissions, and regulatory documents.

Other Considerations

Research Studies of Variable QualityThe quality of information retrieved from any source must

be carefully scrutinized. For example, promotional materialon the WWW may be disguised as peer-reviewed research(Eysenbach et al., 2001). Decisions about methodologic qualityof retrieved research are separate from strategies to locatestudies. Easterbrook and colleagues (1991) said that studiespublished in peer-reviewed journals are not necessarily ofhigher methodologic quality than are other studies. Studiesretrieved from any source, including MEDLINE, have to beevaluated in regard to inclusion criteria that may includemethodologic characteristics (de Vet et al., 2001). In meta-analysis methods are coded for empirical analysis as potentialmoderators of effect size. Filters have been proposed as astrategy to retrieve only the most rigorously conducted studies(Booth & O’Rourke, 1999). Unfortunately, filtering studiesmight indicate more about the quality of reporting than thequality of studies. Filtering is an inexact science at best. Evenif filters were effective, research reviewers would not be inuniversal agreement about what should be used. In meta-analysis, inclusion of all relevant studies is consistent withthe aims of science.

Resource Allocation and Extensive SearchingResearchers planning a comprehensive review of the

literature should not underestimate the difficulty or expenseof conducting a well-designed and carefully implementedreview (Dickersin et al., 1994; Jadad et al., 1998). Predictingthe time necessary to conduct searches or to fully considercitations is difficult. Jadad and colleagues (1998) suggestedbibliographic databases are the most cost-effective source,hand searching is an intermediate cost strategy, and directcontact with investigators or organizations is generally quitecostly. Fewer resources are required for very narrow searchesor for searches in areas in which little research has beenconducted. The most common strategies used for systematicreviews are computerized databases, ancestry searches,journal hand searches, and contacting experts (Avenell et al.,2001). Jadad and colleagues (1998) recommended thatresearchers select as many search strategies as their resourcesallow.

Conclusions

A systematic and thorough review of existing research is asimportant as is systematic collection of data for advancingknowledge. Limited and biased searches are inefficient and


inadequate for building the cumulative nature of science.Conclusions about the efficacy of nursing interventions shouldnot be based on inadequate reviews of existing evidence. Abroad array of search strategies are available and they extendfar beyond the customary MEDLINE search. The purpose ofthe literature review will determine the extent of search thatis appropriate for any given project.

References

Avenell, A., Handoll, H.H., & Grant, A.M. (2001). Lessons for searchstrategies from a systematic review, in The Cochrane Library, ofnutritional supplementation trials in patients after hip fracture.American Journal of Clinical Nutrition, 73, 505-510.

Booth, A., & O’Rourke, A.F. (1999). Searching for evidence: Principles andpractice. Evidence Based Medicine, 4, 133-136.

Callaham, M.L., Wears, R.L., Weber, E.J., Barton, C., & Young, G. (1998).Positive-outcome bias and other limitations in the outcome of researchabstracts submitted to a scientific meeting. JAMA, 280, 254-257.

Chalmers, I., Adams, M., Dickersin, K., Hetherington, J., Tarnow-Mordi,W., Meinert, C., et al. (1990). A cohort study of reports of controlledtrials. JAMA, 263, 1401-1405.

Conn, V. (2002-2005). Chronic illness exercise interventions: A meta-analysis. National Institutes of Health—National Institute of NursingResearch, RO1NR07870. Bethesda, MD.

Conn, V., Valentine, J., & Cooper, H. (2002). Interventions to increasephysical activity among aging adults: A meta-analysis. Annals ofBehavioral Medicine, 24, 190-200.

Cook, A.M., Finlay, I.G., Edwards, A.G.K., Hood, K., Higginson, I.J.,Goodwin, D.M., et al. (2001). Efficiency of searching the grey literaturein palliative care. Journal of Pain & Symptom Management, 22, 797-80.

Cooper, H., & Ribble, R. (1989). Influences on the outcomes of literaturesearches for integrative research reviews. Knowledge: Creation,Diffusion, Utilization, 10, 179-201.

de Vet, H.C., van der Weijden, T., Muris, J.W., Heyrman, J., Buntinx, F.,& Knottnerus, J.A. (2001). Systematic reviews of diagnostic research.Considerations about assessment and incorporation of methodologicalquality. European Journal of Epidemiology, 17, 301-306.

Dickersin, K., Chan, S., Chalmers, T.C., Sacks, H.S., & Smith, H., Jr.(1987). Publication bias and clinical trials. Controlled Clinical Trials,8, 343-353.

Dickersin, K., Manheimer, E., Wieland, S., Robinson, K. A., Lefebvre, C.,& McDonald, S. (2002). Development of the Cochrane Collaboration’sCENTRAL Register of controlled clinical trials. Evaluation & theHealth Professions, 25, 38-64.

Dickersin, K., Min, Y.I., & Meinert, C.L. (1992). Factors influencingpublication of research results. JAMA, 267, 374-378.

Dickersin, K., Scherer, R., & Lefebvre, C. (1994). Identifying relevantstudies for systematic reviews. British Medical Journal, 309, 1286-1291.

Easterbrook, P.J., Berlin, J.A., Gopalan, R., & Matthews, D.R. (1991).Publication bias in clinical research. Lancet, 337, 867-872.

Egger, M., & Smith, G.D. (1998). Bias in location and selection of studies.British Medical Journal, 316, 61-66.

Eysenbach, G., & Diepgen, T.L. (1998). Towards quality managementof medical information on the Internet: evaluation, labeling, andfiltering of information. British Medical Journal, 317, 1496-1500.

Eysenbach, G., Tuische, J., & Diepgen, T.L. (2001). Evaluation of theusefulness of Internet searches to identify unpublished clinical trials forsystematic reviews. Medical Informatics & the Internet in Medicine,26, 203-218.

Gotzsche, P.C. (1987). Reference bias in reports of drug trials. BritishMedical Journal Clinical Research, 295, 654-656.

Gotzsche, P.C. (1989). Multiple publication of reports of drug trials.European Journal of Clinical Pharmacology, 36, 429-432.

Hek, G., Langton, H., & Blunden, G. (2000). Systematically searchingand reviewing literature. Nurse Researcher, 7, 40-57.

Helmer, D., Savoie, I., Green, C., & Kazanjian, A. (2001). Extending thesearch to find material for the systematic review. Bulletin of the MedicalLibrary Association, 89, 346-352.

Hersh, W.R., & Hickam, D.H. (1993). A comparison of two methods forindexing and retrieval from a full-text medical database. MedicalDecision Making, 13, 220-226.

Hersh, W.R., Gorman, P.N., & Sacherek, L.S. (1998). Applicability andquality of information for answering clinical questions on the Web.JAMA, 280, 1307-1308.

Hubbard, R., & Armstrong, J.S. (1997). Publication bias against nullresults. Psychological Reports, 80, 337-338.

Jadad, A.R., Moher, D., & Klassen, T.P. (1998). Guides for reading andinterpreting systematic reviews: II. How did the authors find the studiesand assess their quality? Archives of Pediatrics & Adolescent Medicine,152, 812-817.

Kleijnen, J., & Knipschild, P. (1992). Review articles and publicationbias. Arzneimittel-Forschung, 42, 587-591.

Langham, J., Thompson, E., & Rowen, K. (1999). Identification ofrandomized controlled trials from the emergency medicine literature:comparison of hand searching versus MEDLINE searching. Annals ofEmergency Medicine, 34, 25-34.

Lawrence, S., & Giles, C. (1998). Searching the Web. Science, 280, 98-100.

Lefebvre, C., Lusher, A., Dickersin, K., & Manheimer, E. (2002). Literaturesearches. Lancet, 359, 896.

McDonald, S., Lefebvre, C., Antes, G., Galandi, D., Gotzsche, P.,Hammarquist, C., et al. (2002). The contribution of handsearchingEuropean health care journals to the Cochrane Controlled TrialsRegister. Evaluation & the Health Professions, 25, 65-75.

McManus, R.J., Wilson, S., Delaney, B.C., Fitzmaurice, D.A., Hyde, C.J.,Tobias, R.S., et al. (1998). Review of the usefulness of contactingother experts when conducting a literature search for systematicreviews. British Medical Journal, 317, 1562-1563.

Morrisey, L.J., & DeBourgh, G.A. (2001). Refining literature searchingskills for the advanced practice nurse. AACN Clinical Issues: AdvancedPractice in Acute & Critical Care, 12, 560-577.

Poynard, T., & Conn, H.O. (1985). The retrieval of randomized clinicaltrials in liver disease from the medical literature. A comparison ofMEDLARS and manual methods. Controlled Clinical Trials, 6, 271-279.

Ravnskov, U. (1992). Cholesterol lowering trials in coronary heart disease:frequency of citation and outcome. British Medical Journal, 305, 15-19.

Robinson, K.A., & Dickersin, K. (2002). Development of a highly sensitivesearch strategy for the retrieval of reports of controlled trials usingPubMed. International Journal of Epidemiology, 31, 150-153.

Rosenthal, R. (1979). The file drawer problem and tolerance for nullresults. Psychological Bulletin, 86, 638-641.

Scherer, R.W., Dickersin, K., & Langenberg, P. (1994). Full publicationof results initially presented in abstracts. A meta-analysis. JAMA, 272,158-162.

Serling, T. (1959). Publication decisions and their possible effects oninferences drawn from tests of significance. Journal of the AmericanStatistical Association, 54, 30-34.

Serling, T., Rosenbaum, W., & Weinkam, J. (1995). Publication decisionsrevisited: the effect of the outcome of statistical tests on the decision topublish and vice versa. American Statistician, 49, 108-112.

Sigouin, C., & Jadad, A.R. (2002). Awareness of sources of peer-reviewedresearch evidence on the Internet. JAMA, 287, 2867-2869.

Simes, R.J. (1987). Confronting publication bias: a cohort design formeta-analysis. Statistics in Medicine, 6, 11-29.

Simes, R.J. (1986). Publication bias: the case for an international registryof clinical trials. Journal of Clinical Oncology, 4, 1529-1541.

Sindhu, F., & Dickson, R. (1997). The complexity of searching theliterature. International Journal of Nursing Practice, 3, 211-217.

Stern, J.M., & Simes, R.J. (1997). Publication bias: Evidence of delayedpublication in a cohort study of research projects. British MedicalJournal, 315, 640-645.

Sugita, M., Kanamori, M., Izuno, T., & Miyakawa, M. (1992). Estimatinga summarized odds ratio whilst eliminating publication bias in meta-analysis. Japanese Journal of Clinical Oncology, 22, 354-358.

Thornton, A., & Lee, P. (2000). Publication bias in meta-analysis: Itscauses and consequences. Journal of Clinical Epidemiology, 53, 207-216.

Zielinski, C. (1995). New equities of information in an electronic age.British Medical Journal, 310, 1480-1481.


beyond medline for literature searches

Documents