METHODOLOGY Open Access

Reviewing the research methods literature:principles and strategies illustrated by asystematic overview of sampling inqualitative researchStephen J. Gentles1,4*, Cathy Charles1ˆ, David B. Nicholas2, Jenny Ploeg3 and K. Ann McKibbon1

Abstract

Background: Overviews of methods are potentially useful means to increase clarity and enhance collectiveunderstanding of specific methods topics that may be characterized by ambiguity, inconsistency, or a lack ofcomprehensiveness. This type of review represents a distinct literature synthesis method, although to date, itsmethodology remains relatively undeveloped despite several aspects that demand unique review procedures. Thepurpose of this paper is to initiate discussion about what a rigorous systematic approach to reviews of methods,referred to here as systematic methods overviews, might look like by providing tentative suggestions for approachingspecific challenges likely to be encountered. The guidance offered here was derived from experience conducting asystematic methods overview on the topic of sampling in qualitative research.

Results: The guidance is organized into several principles that highlight specific objectives for this type of reviewgiven the common challenges that must be overcome to achieve them. Optional strategies for achieving eachprinciple are also proposed, along with discussion of how they were successfully implemented in the overview onsampling. We describe seven paired principles and strategies that address the following aspects: delimiting theinitial set of publications to consider, searching beyond standard bibliographic databases, searching without theavailability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts andother information to abstract iteratively, accounting for inconsistent terminology used to describe specific methodstopics, and generating rigorous verifiable analytic interpretations. Since a broad aim in systematic methodsoverviews is to describe and interpret the relevant literature in qualitative terms, we suggest that iterative decisionmaking at various stages of the review process, and a rigorous qualitative approach to analysis are necessaryfeatures of this review type.

Conclusions: We believe that the principles and strategies provided here will be useful to anyone choosing toundertake a systematic methods overview. This paper represents an initial effort to promote high quality criticalevaluations of the literature regarding problematic methods topics, which have the potential to promote clearer,shared understandings, and accelerate advances in research methods. Further work is warranted to develop moredefinitive guidance.

Keywords: Systematic review, Literature selection, Research methods, Research methodology, Overview ofmethods, Systematic methods overview, Review methods

* Correspondence: stevegentles@gmail.comˆDeceased1Department of Clinical Epidemiology and Biostatistics, McMaster University,Hamilton, Ontario, Canada4CanChild Centre for Childhood Disability Research, McMaster University,1400 Main Street West, IAHS 408, Hamilton, ON L8S 1C7, CanadaFull list of author information is available at the end of the article

© 2016 The Author(s). Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Gentles et al. Systematic Reviews (2016) 5:172 DOI 10.1186/s13643-016-0343-0

BackgroundWhile reviews of methods are not new, they represent adistinct review type whose methodology remainsrelatively under-addressed in the literature despite theclear implications for unique review procedures. One offew examples to describe it is a chapter containingreflections of two contributing authors in a book of 21reviews on methodological topics compiled for the Brit-ish National Health Service, Health Technology Assess-ment Program [1]. Notable is their observation of howthe differences between the methods reviews and con-ventional quantitative systematic reviews, specifically at-tributable to their varying content and purpose, haveimplications for defining what qualifies as systematic.While the authors describe general aspects of “systemati-city” (including rigorous application of a methodicalsearch, abstraction, and analysis), they also describe ahigh degree of variation within the category of methodsreviews itself and so offer little in the way of concreteguidance. In this paper, we present tentative concreteguidance, in the form of a preliminary set of proposedprinciples and optional strategies, for a rigorous system-atic approach to reviewing and evaluating the literatureon quantitative or qualitative methods topics. For pur-poses of this article, we have used the term systematicmethods overview to emphasize the notion of a system-atic approach to such reviews.The conventional focus of rigorous literature reviews

(i.e., review types for which systematic methods have beencodified, including the various approaches to quantitativesystematic reviews [2–4], and the numerous forms ofqualitative and mixed methods literature synthesis [5–10])is to synthesize empirical research findings from multiplestudies. By contrast, the focus of overviews of methods,including the systematic approach we advocate, is tosynthesize guidance on methods topics. The literatureconsulted for such reviews may include the methods lit-erature, methods-relevant sections of empirical researchreports, or both. Thus, this paper adds to previous workpublished in this journal—namely, recent preliminaryguidance for conducting reviews of theory [11]—that hasextended the application of systematic review methods tonovel review types that are concerned with subject matterother than empirical research findings.Published examples of methods overviews illustrate

the varying objectives they can have. One objective is toestablish methodological standards for appraisal pur-poses. For example, reviews of existing quality appraisalstandards have been used to propose universal standardsfor appraising the quality of primary qualitative research[12] or evaluating qualitative research reports [13]. Asecond objective is to survey the methods-relevant sec-tions of empirical research reports to establish currentpractices on methods use and reporting practices, which

Moher and colleagues [14] recommend as a means for es-tablishing the needs to be addressed in reporting guide-lines (see, for example [15, 16]). A third objective for amethods review is to offer clarity and enhance collectiveunderstanding regarding a specific methods topic thatmay be characterized by ambiguity, inconsistency, or alack of comprehensiveness within the available methodsliterature. An example of this is a overview whose object-ive was to review the inconsistent definitions of intention-to-treat analysis (the methodologically preferred approachto analyze randomized controlled trial data) that havebeen offered in the methods literature and propose a solu-tion for improving conceptual clarity [17]. Such reviewsare warranted because students and researchers who mustlearn or apply research methods typically lack the time tosystematically search, retrieve, review, and compare theavailable literature to develop a thorough and criticalsense of the varied approaches regarding certain contro-versial or ambiguous methods topics.While systematic methods overviews, as a review type,

include both reviews of the methods literature and re-views of methods-relevant sections from empirical studyreports, the guidance provided here is primarily applic-able to reviews of the methods literature since it was de-rived from the experience of conducting such a review[18], described below. To our knowledge, there are nowell-developed proposals on how to rigorously conductsuch reviews. Such guidance would have the potential toimprove the thoroughness and credibility of critical eval-uations of the methods literature, which could increasetheir utility as a tool for generating understandings thatadvance research methods, both qualitative and quanti-tative. Our aim in this paper is thus to initiate discussionabout what might constitute a rigorous approach to sys-tematic methods overviews. While we hope to promoterigor in the conduct of systematic methods overviewswherever possible, we do not wish to suggest that allmethods overviews need be conducted to the samestandard. Rather, we believe that the level of rigor mayneed to be tailored pragmatically to the specific reviewobjectives, which may not always justify the resource re-quirements of an intensive review process.

The example systematic methods overview on samplingin qualitative researchThe principles and strategies we propose in this paper arederived from experience conducting a systematic methodsoverview on the topic of sampling in qualitative research[18]. The main objective of that methods overview was tobring clarity and deeper understanding of the prominentconcepts related to sampling in qualitative research (pur-poseful sampling strategies, saturation, etc.). Specifically,we interpreted the available guidance, commenting onareas lacking clarity, consistency, or comprehensiveness

Gentles et al. Systematic Reviews (2016) 5:172 Page 2 of 11

(without proposing any recommendations on how to dosampling). This was achieved by a comparative and criticalanalysis of publications representing the most influential(i.e., highly cited) guidance across several methodologicaltraditions in qualitative research.The specific methods and procedures for the overview

on sampling [18] from which our proposals are derivedwere developed both after soliciting initial input fromlocal experts in qualitative research and an expert healthlibrarian (KAM) and through ongoing careful deliber-ation throughout the review process. To summarize, inthat review, we employed a transparent and rigorous ap-proach to search the methods literature, selected publi-cations for inclusion according to a purposeful anditerative process, abstracted textual data using structuredabstraction forms, and analyzed (synthesized) the datausing a systematic multi-step approach featuring ab-straction of text, summary of information in matrices,and analytic comparisons.For this article, we reflected on both the problems and

challenges encountered at different stages of the reviewand our means for selecting justifiable procedures to dealwith them. Several principles were then derived by consid-ering the generic nature of these problems, while thegeneralizable aspects of the procedures used to addressthem formed the basis of optional strategies. Further de-tails of the specific methods and procedures used in theoverview on qualitative sampling are provided below to il-lustrate both the types of objectives and challenges thatreviewers will likely need to consider and our approach toimplementing each of the principles and strategies.

Organization of the guidance into principles andstrategiesFor the purposes of this article, principles are general state-ments outlining what we propose are important aims orconsiderations within a particular review process, given theunique objectives or challenges to be overcome with thistype of review. These statements follow the general format,“considering the objective or challenge of X, we propose Yto be an important aim or consideration.” Strategies areoptional and flexible approaches for implementing the pre-vious principle outlined. Thus, generic challenges give riseto principles, which in turn give rise to strategies.We organize the principles and strategies below into

three sections corresponding to processes characteristic ofmost systematic literature synthesis approaches: literatureidentification and selection; data abstraction from the pub-lications selected for inclusion; and analysis, including crit-ical appraisal and synthesis of the abstracted data. Withineach section, we also describe the specific methodologicaldecisions and procedures used in the overview on samplingin qualitative research [18] to illustrate how the principlesand strategies for each review process were applied and

implemented in a specific case. We expect this guidanceand accompanying illustrations will be useful for anyoneconsidering engaging in a methods overview, particularlythose who may be familiar with conventional systematicreview methods but may not yet appreciate some of thechallenges specific to reviewing the methods literature.

Results and discussionLiterature identification and selectionThe identification and selection process includes searchand retrieval of publications and the development and ap-plication of inclusion and exclusion criteria to select thepublications that will be abstracted and analyzed in thefinal review. Literature identification and selection foroverviews of the methods literature is challenging andpotentially more resource-intensive than for most reviewsof empirical research. This is true for several reasons thatwe describe below, alongside discussion of the potentialsolutions. Additionally, we suggest in this section how theselection procedures can be chosen to match the specificanalytic approach used in methods overviews.

Delimiting a manageable set of publicationsOne aspect of methods overviews that can make identifi-cation and selection challenging is the fact that theuniverse of literature containing potentially relevant infor-mation regarding most methods-related topics is expan-sive and often unmanageably so. Reviewers are faced withtwo large categories of literature: the methods literature,where the possible publication types include journal arti-cles, books, and book chapters; and the methods-relevantsections of empirical study reports, where the possible pub-lication types include journal articles, monographs, books,theses, and conference proceedings. In our systematicoverview of sampling in qualitative research, exhaustivelysearching (including retrieval and first-pass screening) allpublication types across both categories of literature forinformation on a single methods-related topic was tooburdensome to be feasible. The following proposedprinciple follows from the need to delimit a manageableset of literature for the review.

Principle #1: Considering the broad universe of poten-tially relevant literature, we propose that an important ob-jective early in the identification and selection stage is todelimit a manageable set of methods-relevant publicationsin accordance with the objectives of the methods overview.

Strategy #1: To limit the set of methods-relevant publi-cations that must be managed in the selection process,reviewers have the option to initially review only themethods literature, and exclude the methods-relevantsections of empirical study reports, provided this alignswith the review’s particular objectives.

Gentles et al. Systematic Reviews (2016) 5:172 Page 3 of 11

We propose that reviewers are justified in choosing toselect only the methods literature when the objective isto map out the range of recognized concepts relevant toa methods topic, to summarize the most authoritative orinfluential definitions or meanings for methods-relatedconcepts, or to demonstrate a problematic lack of clarityregarding a widely established methods-related conceptand potentially make recommendations for a preferredapproach to the methods topic in question. For example,in the case of the methods overview on sampling [18], theprimary aim was to define areas lacking in clarity formultiple widely established sampling-related topics. In thereview on intention-to-treat in the context of missing out-come data [17], the authors identified a lack of claritybased on multiple inconsistent definitions in the literatureand went on to recommend separating the issue of how tohandle missing outcome data from the issue of whetheran intention-to-treat analysis can be claimed.In contrast to strategy #1, it may be appropriate to select

the methods-relevant sections of empirical study reportswhen the objective is to illustrate how a methods conceptis operationalized in research practice or reported byauthors. For example, one could review all the publica-tions in 2 years’ worth of issues of five high-impact field-related journals to answer questions about how re-searchers describe implementing a particular method orapproach, or to quantify how consistently they define orreport using it. Such reviews are often used to highlightgaps in the reporting practices regarding specific methods,which may be used to justify items to address in reportingguidelines (for example, [14–16]).It is worth recognizing that other authors have advo-

cated broader positions regarding the scope of literatureto be considered in a review, expanding on our perspec-tive. Suri [10] (who, like us, emphasizes how differentsampling strategies are suitable for different literaturesynthesis objectives) has, for example, described a two-stage literature sampling procedure (pp. 96–97). First,reviewers use an initial approach to conduct a broad over-view of the field—for reviews of methods topics, thiswould entail an initial review of the research methods lit-erature. This is followed by a second more focused stagein which practical examples are purposefully selected—formethods reviews, this would involve sampling the empir-ical literature to illustrate key themes and variations.While this approach is seductive in its capacity to generatemore in depth and interpretive analytic findings, some re-viewers may consider it too resource-intensive to includethe second step no matter how selective the purposefulsampling. In the overview on sampling where we stoppedafter the first stage [18], we discussed our selective focuson the methods literature as a limitation that left oppor-tunities for further analysis of the literature. We explicitlyrecommended, for example, that theoretical sampling was

a topic for which a future review of the methods sectionsof empirical reports was justified to answer specific ques-tions identified in the primary review.Ultimately, reviewers must make pragmatic decisions

that balance resource considerations, combined with in-formed predictions about the depth and complexity of lit-erature available on their topic, with the stated objectivesof their review. The remaining principles and strategiesapply primarily to overviews that include the methods lit-erature, although some aspects may be relevant to reviewsthat include empirical study reports.

Searching beyond standard bibliographic databasesAn important reality affecting identification and selectionin overviews of the methods literature is the increasedlikelihood for relevant publications to be located insources other than journal articles (which is usually notthe case for overviews of empirical research, where journalarticles generally represent the primary publication type).In the overview on sampling [18], out of 41 full-text publi-cations retrieved and reviewed, only 4 were journal arti-cles, while 37 were books or book chapters. Since manybooks and book chapters did not exist electronically, theirfull text had to be physically retrieved in hardcopy, while11 publications were retrievable only through interlibraryloan or purchase request. The tasks associated with suchretrieval are substantially more time-consuming thanelectronic retrieval. Since a substantial proportion ofmethods-related guidance may be located in publicationtypes that are less comprehensively indexed in standardbibliographic databases, identification and retrieval thusbecome complicated processes.

Principle #2: Considering that important sources ofmethods guidance can be located in non-journal publi-cation types (e.g., books, book chapters) that tend to bepoorly indexed in standard bibliographic databases, it isimportant to consider alternative search methods foridentifying relevant publications to be further screenedfor inclusion.

Strategy #2: To identify books, book chapters, andother non-journal publication types not thoroughlyindexed in standard bibliographic databases, reviewersmay choose to consult one or more of the following lessstandard sources: Google Scholar, publisher web sites, orexpert opinion.In the case of the overview on sampling in qualitative re-

search [18], Google Scholar had two advantages over otherstandard bibliographic databases: it indexes and returns re-cords of books and book chapters likely to contain guid-ance on qualitative research methods topics; and it hasbeen validated as providing higher citation counts than ISIWeb of Science (a producer of numerous bibliographic

Gentles et al. Systematic Reviews (2016) 5:172 Page 4 of 11

databases accessible through institutional subscription) forseveral non-biomedical disciplines including the social sci-ences where qualitative research methods are prominentlyused [19–21]. While we identified numerous useful publica-tions by consulting experts, the author publication lists gen-erated through Google Scholar searches were uniquelyuseful to identify more recent editions of methods booksidentified by experts.

Searching without relevant metadataDetermining what publications to select for inclusion inthe overview on sampling [18] could only rarely be ac-complished by reviewing the publication’s metadata. Thiswas because for the many books and other non-journaltype publications we identified as possibly relevant, thepotential content of interest would be located in only asubsection of the publication. In this common scenariofor reviews of the methods literature (as opposed tomethods overviews that include empirical study reports),reviewers will often be unable to employ standard title,abstract, and keyword database searching or screeningas a means for selecting publications.

Principle #3: Considering that the presence of informa-tion about the topic of interest may not be indicated inthe metadata for books and similar publication types, itis important to consider other means of identifying po-tentially useful publications for further screening.

Strategy #3: One approach to identifying potentially use-ful books and similar publication types is to consider whatclasses of such publications (e.g., all methods manuals fora certain research approach) are likely to contain relevantcontent, then identify, retrieve, and review the full text ofcorresponding publications to determine whether theycontain information on the topic of interest.In the example of the overview on sampling in qualitative

research [18], the topic of interest (sampling) was one ofnumerous topics covered in the general qualitative researchmethods manuals. Consequently, examples from this classof publications first had to be identified for retrieval accord-ing to non-keyword-dependent criteria. Thus, all methodsmanuals within the three research traditions reviewed(grounded theory, phenomenology, and case study) thatmight contain discussion of sampling were sought throughGoogle Scholar and expert opinion, their full text obtained,and hand-searched for relevant content to determine eligi-bility. We used tables of contents and index sections ofbooks to aid this hand searching.

Purposefully selecting literature on conceptual groundsA final consideration in methods overviews relates tothe type of analysis used to generate the review findings.Unlike quantitative systematic reviews where reviewers

aim for accurate or unbiased quantitative estimates—-something that requires identifying and selecting the lit-erature exhaustively to obtain all relevant data available(i.e., a complete sample)—in methods overviews, re-viewers must describe and interpret the relevant litera-ture in qualitative terms to achieve review objectives. Inother words, the aim in methods overviews is to seekcoverage of the qualitative concepts relevant to themethods topic at hand. For example, in the overview ofsampling in qualitative research [18], achieving reviewobjectives entailed providing conceptual coverage ofeight sampling-related topics that emerged as key do-mains. The following principle recognizes that literaturesampling should therefore support generating qualitativeconceptual data as the input to analysis.

Principle #4: Since the analytic findings of a systematicmethods overview are generated through qualitative de-scription and interpretation of the literature on a speci-fied topic, selection of the literature should be guided bya purposeful strategy designed to achieve adequate con-ceptual coverage (i.e., representing an appropriate degreeof variation in relevant ideas) of the topic according toobjectives of the review.

Strategy #4: One strategy for choosing the purposefulapproach to use in selecting the literature according tothe review objectives is to consider whether those objec-tives imply exploring concepts either at a broad overviewlevel, in which case combining maximum variation se-lection with a strategy that limits yield (e.g., critical case,politically important, or sampling for influence—de-scribed below) may be appropriate; or in depth, in whichcase purposeful approaches aimed at revealing innova-tive cases will likely be necessary.In the methods overview on sampling, the implied

scope was broad since we set out to review publicationson sampling across three divergent qualitative researchtraditions—grounded theory, phenomenology, and casestudy—to facilitate making informative conceptual com-parisons. Such an approach would be analogous to max-imum variation sampling.At the same time, the purpose of that review was to critic-

ally interrogate the clarity, consistency, and comprehensive-ness of literature from these traditions that was “most likelyto have widely influenced students’ and researchers’ ideasabout sampling” (p. 1774) [18]. In other words, we explicitlyset out to review and critique the most established and influ-ential (and therefore dominant) literature, since this repre-sents a common basis of knowledge among students andresearchers seeking understanding or practical guidance onsampling in qualitative research. To achieve this objective,we purposefully sampled publications according to the cri-terion of influence, which we operationalized as how often

Gentles et al. Systematic Reviews (2016) 5:172 Page 5 of 11

an author or publication has been referenced in print or in-formal discourse. This second sampling approach alsolimited the literature we needed to consider within ourbroad scope review to a manageable amount.To operationalize this strategy of sampling for influence,

we sought to identify both the most influential authorswithin a qualitative research tradition (all of whose citationswere subsequently screened) and the most influential publi-cations on the topic of interest by non-influential authors.This involved a flexible approach that combined multipleindicators of influence to avoid the dilemma that any singleindicator might provide inadequate coverage. These indica-tors included bibliometric data (h-index for author influ-ence [22]; number of cites for publication influence), expertopinion, and cross-references in the literature (i.e., snowballsampling). As a final selection criterion, a publication wasincluded only if it made an original contribution in terms ofnovel guidance regarding sampling or a related concept;thus, purely secondary sources were excluded. Publish orPerish software (Anne-Wil Harzing; available at http://www.harzing.com/resources/publish-or-perish) was used togenerate bibliometric data via the Google Scholar database.Figure 1 illustrates how identification and selection in themethods overview on sampling was a multi-faceted anditerative process. The authors selected as influential, andthe publications selected for inclusion or exclusion arelisted in Additional file 1 (Matrices 1, 2a, 2b).In summary, the strategies of seeking maximum vari-

ation and sampling for influence were employed in thesampling overview to meet the specific review objectivesdescribed. Reviewers will need to consider the full rangeof purposeful literature sampling approaches at their dis-posal in deciding what best matches the specific aims oftheir own reviews. Suri [10] has recently retooled Pat-ton’s well-known typology of purposeful sampling strat-egies (originally intended for primary research) forapplication to literature synthesis, providing a usefulresource in this respect.

Data abstractionThe purpose of data abstraction in rigorous literature re-views is to locate and record all data relevant to thetopic of interest from the full text of included publica-tions, making them available for subsequent analysis.Conventionally, a data abstraction form—consisting ofnumerous distinct conceptually defined fields to whichcorresponding information from the source publicationis recorded—is developed and employed. There are sev-eral challenges, however, to the processes of developingthe abstraction form and abstracting the data itself whenconducting methods overviews, which we address here.Some of these problems and their solutions may befamiliar to those who have conducted qualitative litera-ture syntheses, which are similarly conceptual.

Iteratively defining conceptual information to abstractIn the overview on sampling [18], while we surveyedmultiple sources beforehand to develop a list of conceptsrelevant for abstraction (e.g., purposeful sampling strat-egies, saturation, sample size), there was no way for usto anticipate some concepts prior to encountering themin the review process. Indeed, in many cases, reviewersare unable to determine the complete set of methods-related concepts that will be the focus of the final reviewa priori without having systematically reviewed the pub-lications to be included. Thus, defining what informationto abstract beforehand may not be feasible.

Principle #5: Considering the potential impracticality ofdefining a complete set of relevant methods-related con-cepts from a body of literature one has not yet systematicallyread, selecting and defining fields for data abstraction mustoften be undertaken iteratively. Thus, concepts to be ab-stracted can be expected to grow and change as data ab-straction proceeds.

Fig. 1 Literature identification and selection process used in the methods overview on sampling [18]

Gentles et al. Systematic Reviews (2016) 5:172 Page 6 of 11

Strategy #5: Reviewers can develop an initial form orset of concepts for abstraction purposes according tostandard methods (e.g., incorporating expert feedback,pilot testing) and remain attentive to the need to itera-tively revise it as concepts are added or modified duringthe review. Reviewers should document revisions and re-turn to re-abstract data from previously abstracted pub-lications as the new data requirements are determined.In the sampling overview [18], we developed and main-

tained the abstraction form in Microsoft Word. We de-rived the initial set of abstraction fields from our ownknowledge of relevant sampling-related concepts, consult-ation with local experts, and reviewing a pilot sample ofpublications. Since the publications in this review includeda large proportion of books, the abstraction process oftenbegan by flagging the broad sections within a publicationcontaining topic-relevant information for detailed reviewto identify text to abstract. When reviewing flagged text,the reviewer occasionally encountered an unanticipatedconcept significant enough to warrant being added as anew field to the abstraction form. For example, a field wasadded to capture how authors described the timing ofsampling decisions, whether before (a priori) or after(ongoing) starting data collection, or whether this was un-clear. In these cases, we systematically documented themodification to the form and returned to previously ab-stracted publications to abstract any information thatmight be relevant to the new field.The logic of this strategy is analogous to the logic used

in a form of research synthesis called best fit frameworksynthesis (BFFS) [23–25]. In that method, reviewers ini-tially code evidence using an a priori framework theyhave selected. When evidence cannot be accommodatedby the selected framework, reviewers then develop newthemes or concepts from which they construct a new ex-panded framework. Both the strategy proposed and theBFFS approach to research synthesis are notable fortheir rigorous and transparent means to adapt a final setof concepts to the content under review.

Accounting for inconsistent terminologyAn important complication affecting the abstractionprocess in methods overviews is that the language usedby authors to describe methods-related concepts caneasily vary across publications. For example, authorsfrom different qualitative research traditions often usedifferent terms for similar methods-related concepts.Furthermore, as we found in the sampling overview [18],there may be cases where no identifiable term, phrase,or label for a methods-related concept is used at all, anda description of it is given instead. This can makesearching the text for relevant concepts based on key-words unreliable.

Principle #6: Since accepted terms may not be usedconsistently to refer to methods concepts, it is necessaryto rely on the definitions for concepts, rather than key-words, to identify relevant information in the publicationto abstract.

Strategy #6: An effective means to systematically identifyrelevant information is to develop and iteratively adjustwritten definitions for key concepts (corresponding to ab-straction fields) that are consistent with and as inclusiveof as much of the literature reviewed as possible.Reviewers then seek information that matches these defi-nitions (rather than keywords) when scanning a publica-tion for relevant data to abstract.In the abstraction process for the sampling overview [18],

we noted the several concepts of interest to the review forwhich abstraction by keyword was particularly problematicdue to inconsistent terminology across publications:sampling, purposeful sampling, sampling strategy, andsaturation (for examples, see Additional file 1, Matrices 3a,3b, 4). We iteratively developed definitions for these con-cepts by abstracting text from publications that either pro-vided an explicit definition or from which an implicitdefinition could be derived, which was recorded in fieldsdedicated to the concept’s definition. Using a method ofconstant comparison, we used text from definition fields toinform and modify a centrally maintained definition of thecorresponding concept to optimize its fit and inclusivenesswith the literature reviewed. Table 1 shows, as an example,the final definition constructed in this way for one of thecentral concepts of the review, qualitative sampling.We applied iteratively developed definitions when

making decisions about what specific text to abstract foran existing field, which allowed us to abstract concept-relevant data even if no recognized keyword was used.For example, this was the case for the sampling-relatedconcept, saturation, where the relevant text available forabstraction in one publication [26]—“to continue to

Table 1 Final definition for qualitative sampling, includingmethodological tradition-specific variations

Term Definition and tradition-specific variations

Sampling The selection of specific data sourcesfrom which data are collected in orderto address the research objectives

In grounded theory What is selected (i.e., the sampling unit)in theoretical sampling is unclearor inconsistent between authors(i.e., it may not simply be data sources)

In phenomenology What is selected is restricted to peopleonly (i.e., a single type of data source)

In case study What is selected includes cases(i.e., in addition to data sources)

Developed after numerous iterations in the methods overview onsampling [18]

Gentles et al. Systematic Reviews (2016) 5:172 Page 7 of 11

collect data until nothing new was being observed or re-corded, no matter how long that takes”—was not accom-panied by any term or label whatsoever.This comparative analytic strategy (and our approach

to analysis more broadly as described in strategy #7,below) is analogous to the process of reciprocal transla-tion—a technique first introduced for meta-ethnographyby Noblit and Hare [27] that has since been recognizedas a common element in a variety of qualitative meta-synthesis approaches [28]. Reciprocal translation, takenbroadly, involves making sense of a study’s findings interms of the findings of the other studies included in thereview. In practice, it has been operationalized in differ-ent ways. Melendez-Torres and colleagues developed atypology from their review of the metasynthesis litera-ture, describing four overlapping categories of specificoperations undertaken in reciprocal translation: visualrepresentation, key paper integration, data reduction andthematic extraction, and line-by-line coding [28]. Theapproaches suggested in both strategies #6 and #7, withtheir emphasis on constant comparison, appear to fallwithin the line-by-line coding category.

AnalysisGenerating credible and verifiable analytic interpretationsThe analysis in a systematic methods overview mustsupport its more general objective, which we suggestedabove is often to offer clarity and enhance collective un-derstanding regarding a chosen methods topic. In ourexperience, this involves describing and interpreting therelevant literature in qualitative terms. Furthermore, anyinterpretative analysis required may entail reaching dif-ferent levels of abstraction, depending on the more spe-cific objectives of the review. For example, in theoverview on sampling [18], we aimed to produce a com-parative analysis of how multiple sampling-related topicswere treated differently within and among differentqualitative research traditions. To promote credibility ofthe review, however, not only should one seek a qualita-tive analytic approach that facilitates reaching varyinglevels of abstraction but that approach must also ensurethat abstract interpretations are supported and justifiedby the source data and not solely the product of the ana-lyst’s speculative thinking.

Principle #7: Considering the qualitative nature of theanalysis required in systematic methods overviews, it isimportant to select an analytic method whose interpreta-tions can be verified as being consistent with the literatureselected, regardless of the level of abstraction reached.

Strategy #7: We suggest employing the constant com-parative method of analysis [29] because it supports de-veloping and verifying analytic links to the source data

throughout progressively interpretive or abstract levels.In applying this approach, we advise a rigorousapproach, documenting how supportive quotes or refer-ences to the original texts are carried forward in the suc-cessive steps of analysis to allow for easy verification.The analytic approach used in the methods overview

on sampling [18] comprised four explicit steps, progres-sing in level of abstraction—data abstraction, matrices,narrative summaries, and final analytic conclusions(Fig. 2). While we have positioned data abstraction asthe second stage of the generic review process (prior toAnalysis), above, we also considered it as an initial stepof analysis in the sampling overview for several reasons.First, it involved a process of constant comparisons anditerative decision-making about the fields to add or de-fine during development and modification of the ab-straction form, through which we established the rangeof concepts to be addressed in the review. At the sametime, abstraction involved continuous analytic decisionsabout what textual quotes (ranging in size from shortphrases to numerous paragraphs) to record in the fieldsthus created. This constant comparative process wasanalogous to open coding in which textual data from pub-lications was compared to conceptual fields (equivalent tocodes) or to other instances of data previously abstractedwhen constructing definitions to optimize their fit withthe overall literature as described in strategy #6. Finally, inthe data abstraction step, we also recorded our first inter-pretive thoughts in dedicated fields, providing initial ma-terial for the more abstract analytic steps.In the second step of the analysis, we constructed topic-

specific matrices, or tables, by copying relevant quotes fromabstraction forms into the appropriate cells of matrices (forthe complete set of analytic matrices developed in the sam-pling review, see Additional file 1 (matrices 3 to 10)). Eachmatrix ranged from one to five pages; row headings, nestedthree-deep, identified the methodological tradition, author,and publication, respectively; and column headings identi-fied the concepts, which corresponded to abstraction fields.Matrices thus allowed us to make further comparisonsacross methodological traditions, and between authors

Fig. 2 Summary of progressive steps of analysis used in the methodsoverview on sampling [18]

Gentles et al. Systematic Reviews (2016) 5:172 Page 8 of 11

within a tradition. In the third step of analysis, we recordedour comparative observations as narrative summaries, inwhich we used illustrative quotes more sparingly. In thefinal step, we developed analytic conclusions based on thenarrative summaries about the sampling-related conceptswithin each methodological tradition for which clarity,consistency, or comprehensiveness of the available guid-ance appeared to be lacking. Higher levels of analysis thusbuilt logically from the lower levels, enabling us to easilyverify analytic conclusions by tracing the support forclaims by comparing the original text of publicationsreviewed.

Integrative versus interpretive methods overviewsThe analytic product of systematic methods overviews iscomparable to qualitative evidence syntheses, since bothinvolve describing and interpreting the relevant literaturein qualitative terms. Most qualitative synthesis approachesstrive to produce new conceptual understandings that varyin level of interpretation. Dixon-Woods and colleagues[30] elaborate on a useful distinction, originating fromNoblit and Hare [27], between integrative and interpretivereviews. Integrative reviews focus on summarizing avail-able primary data and involve using largely secure andwell defined concepts to do so; definitions are used froman early stage to specify categories for abstraction (or cod-ing) of data, which in turn supports their aggregation; theydo not seek as their primary focus to develop or specifynew concepts, although they may achieve some theoreticalor interpretive functions. For interpretive reviews, mean-while, the main focus is to develop new concepts and the-ories that integrate them, with the implication that theconcepts developed become fully defined towards the endof the analysis. These two forms are not completely dis-tinct, and “every integrative synthesis will include ele-ments of interpretation, and every interpretive synthesiswill include elements of aggregation of data” [30].The example methods overview on sampling [18]

could be classified as predominantly integrative becauseits primary goal was to aggregate influential authors’ideas on sampling-related concepts; there were also,however, elements of interpretive synthesis since itaimed to develop new ideas about where clarity in guid-ance on certain sampling-related topics is lacking, anddefinitions for some concepts were flexible and not fixeduntil late in the review. We suggest that most systematicmethods overviews will be classifiable as predominantlyintegrative (aggregative). Nevertheless, more highly in-terpretive methods overviews are also quite possible—forexample, when the review objective is to provide a highlycritical analysis for the purpose of generating new meth-odological guidance. In such cases, reviewers may needto sample more deeply (see strategy #4), specifically byselecting empirical research reports (i.e., to go beyond

dominant or influential ideas in the methods literature)that are likely to feature innovations or instructive les-sons in employing a given method.

ConclusionsIn this paper, we have outlined tentative guidance in theform of seven principles and strategies on how to conductsystematic methods overviews, a review type in whichmethods-relevant literature is systematically analyzed withthe aim of offering clarity and enhancing collective under-standing regarding a specific methods topic. Our proposalsinclude strategies for delimiting the set of publicationsto consider, searching beyond standard bibliographicdatabases, searching without the availability of relevantmetadata, selecting publications on purposeful conceptualgrounds, defining concepts and other information toabstract iteratively, accounting for inconsistent terminology,and generating credible and verifiable analytic interpreta-tions. We hope the suggestions proposed will be useful toothers undertaking reviews on methods topics in future.As far as we are aware, this is the first published source

of concrete guidance for conducting this type of review.It is important to note that our primary objective was toinitiate methodological discussion by stimulating reflec-tion on what rigorous methods for this type of reviewshould look like, leaving the development of morecomplete guidance to future work. While derived fromthe experience of reviewing a single qualitative methodstopic, we believe the principles and strategies providedare generalizable to overviews of both qualitative andquantitative methods topics alike. However, it is ex-pected that additional challenges and insights for con-ducting such reviews have yet to be defined. Thus, wepropose that next steps for developing more definitiveguidance should involve an attempt to collect and inte-grate other reviewers’ perspectives and experiences inconducting systematic methods overviews on a broadrange of qualitative and quantitative methods topics.Formalized guidance and standards would improve thequality of future methods overviews, something we be-lieve has important implications for advancing qualita-tive and quantitative methodology. When undertaken toa high standard, rigorous critical evaluations of the avail-able methods guidance have significant potential tomake implicit controversies explicit, and improve theclarity and precision of our understandings of problem-atic qualitative or quantitative methods issues.A review process central to most types of rigorous re-

views of empirical studies, which we did not explicitly ad-dress in a separate review step above, is quality appraisal.The reason we have not treated this as a separate stepstems from the different objectives of the primarypublications included in overviews of the methods litera-ture (i.e., providing methodological guidance) compared to

Gentles et al. Systematic Reviews (2016) 5:172 Page 9 of 11

the primary publications included in the other establishedreview types (i.e., reporting findings from single empiricalstudies). This is not to say that appraising quality of themethods literature is not an important concern for system-atic methods overviews. Rather, appraisal is much moreintegral to (and difficult to separate from) the analysis step,in which we advocate appraising clarity, consistency, andcomprehensiveness—the quality appraisal criteria that wesuggest are appropriate for the methods literature. As a sec-ond important difference regarding appraisal, we currentlyadvocate appraising the aforementioned aspects at the levelof the literature in aggregate rather than at the level ofindividual publications. One reason for this is that methodsguidance from individual publications generally builds onprevious literature, and thus we feel that ahistorical judg-ments about comprehensiveness of single publications lackrelevance and utility. Additionally, while different methodsauthors may express themselves less clearly than others,their guidance can nonetheless be highly influential anduseful, and should therefore not be downgraded or ignoredbased on considerations of clarity—which raises questionsabout the alternative uses that quality appraisals of individ-ual publications might have. Finally, legitimate variability inthe perspectives that methods authors wish to emphasize,and the levels of generality at which they write aboutmethods, makes critiquing individual publications based onthe criterion of clarity a complex and potentially problem-atic endeavor that is beyond the scope of this paper toaddress. By appraising the current state of the literature at aholistic level, reviewers stand to identify important gaps inunderstanding that represent valuable opportunities forfurther methodological development.To summarize, the principles and strategies provided

here may be useful to those seeking to undertake theirown systematic methods overview. Additional work isneeded, however, to establish guidance that is compre-hensive by comparing the experiences from conductinga variety of methods overviews on a range of methodstopics. Efforts that further advance standards for system-atic methods overviews have the potential to promotehigh-quality critical evaluations that produce conceptu-ally clear and unified understandings of problematicmethods topics, thereby accelerating the advance ofresearch methodology.

Additional file

Additional file 1: Submitted: Analysis_matrices. (DOC 330 kb)

AcknowledgementsNot applicable.

FundingThere was no funding for this work.

Availability of data and materialsThe systematic methods overview used as a worked example in this article(Gentles SJ, Charles C, Ploeg J, McKibbon KA: Sampling in qualitative research:insights from an overview of the methods literature. The Qual Rep 2015,20(11):1772-1789) is available from http://nsuworks.nova.edu/tqr/vol20/iss11/5.

Authors’ contributionsSJG wrote the first draft of this article, with CC contributing to drafting. Allauthors contributed to revising the manuscript. All authors except CC(deceased) approved the final draft. SJG, CC, KAB, and JP were involved indeveloping methods for the systematic methods overview on sampling.

Authors’ informationNot applicable.

Competing interestsThe authors declare that they have no competing interests.

Consent for publicationNot applicable.

Ethics approval and consent to participateNot applicable.

Author details1Department of Clinical Epidemiology and Biostatistics, McMaster University,Hamilton, Ontario, Canada. 2Faculty of Social Work, University of Calgary,Alberta, Canada. 3School of Nursing, McMaster University, Hamilton, Ontario,Canada. 4CanChild Centre for Childhood Disability Research, McMasterUniversity, 1400 Main Street West, IAHS 408, Hamilton, ON L8S 1C7, Canada.

Received: 6 June 2016 Accepted: 14 September 2016

References1. Hutton JL, Ashcroft R. What does “systematic” mean for reviews of

methods? In: Black N, Brazier J, Fitzpatrick R, Reeves B, editors. Healthservices research methods: a guide to best practice. London: BMJPublishing Group; 1998. p. 249–54.

2. Cochrane handbook for systematic reviews of interventions. In. Edited byHiggins JPT, Green S, Version 5.1.0 edn: The Cochrane Collaboration; 2011.

3. Centre for Reviews and Dissemination: Systematic reviews: CRD’s guidance forundertaking reviews in health care. York: Centre for Reviews and Dissemination;2009.

4. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, ClarkeM, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reportingsystematic reviews and meta-analyses of studies that evaluate healthcareinterventions: explanation and elaboration. BMJ. 2009;339:b2700–0.

5. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research:a critical review. BMC Med Res Methodol. 2009;9(1):59.

6. Kastner M, Tricco AC, Soobiah C, Lillie E, Perrier L, Horsley T, Welch V, CogoE, Antony J, Straus SE. What is the most appropriate knowledge synthesismethod to conduct a review? Protocol for a scoping review. BMC Med ResMethodol. 2012;12(1):1–1.

7. Booth A, Noyes J, Flemming K, Gerhardus A. Guidance on choosingqualitative evidence synthesis methods for use in health technologyassessments of complex interventions. In: Integrate-HTA. 2016.

8. Booth A, Sutton A, Papaioannou D. Systematic approaches to successfulliterature review. 2nd ed. London: Sage; 2016.

9. Hannes K, Lockwood C. Synthesizing qualitative research: choosing the rightapproach. Chichester: Wiley-Blackwell; 2012.

10. Suri H. Towards methodologically inclusive research syntheses: expandingpossibilities. New York: Routledge; 2014.

11. Campbell M, Egan M, Lorenc T, Bond L, Popham F, Fenton C, Benzeval M.Considering methodological options for reviews of theory: illustrated by areview of theories linking income and health. Syst Rev. 2014;3(1):1–11.

12. Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in healthcare: controversies and recommendations. Ann Fam Med. 2008;6(4):331–9.

13. Tong A, Sainsbury P, Craig J. Consolidated criteria for reportingqualitativeresearch (COREQ): a 32-item checklist for interviews and focus groups. Int JQual Health Care. 2007;19(6):349–57.

Gentles et al. Systematic Reviews (2016) 5:172 Page 10 of 11

14. Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of healthresearch reporting guidelines. PLoS Med. 2010;7(2):e1000217.

15. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology andreporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.

16. Chan AW, Altman DG. Epidemiology and reporting of randomised trialspublished in PubMed journals. Lancet. 2005;365(9465):1159–62.

17. Alshurafa M, Briel M, Akl EA, Haines T, Moayyedi P, Gentles SJ, Rios L, Tran C,Bhatnagar N, Lamontagne F, et al. Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methodsliterature. PLoS One. 2012;7(11):e49163.

18. Gentles SJ, Charles C, Ploeg J, McKibbon KA. Sampling in qualitativeresearch: insights from an overview of the methods literature. Qual Rep.2015;20(11):1772–89.

19. Harzing A-W, Alakangas S. Google Scholar, Scopus and the Web of Science:a longitudinal and cross-disciplinary comparison. Scientometrics. 2016;106(2):787–804.

20. Harzing A-WK, van der Wal R. Google Scholar as a new source for citationanalysis. Ethics Sci Environ Polit. 2008;8(1):61–73.

21. Kousha K, Thelwall M. Google Scholar citations and Google Web/URLcitations: a multi‐discipline exploratory analysis. J Assoc Inf Sci Technol.2007;58(7):1055–65.

22. Hirsch JE. An index to quantify an individual’s scientific research output.Proc Natl Acad Sci U S A. 2005;102(46):16569–72.

23. Booth A, Carroll C. How to build up the actionable knowledge base: therole of ‘best fit’ framework synthesis for studies of improvement inhealthcare. BMJ Quality Safety. 2015;24(11):700–8.

24. Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refiningthe method. BMC Med Res Methodol. 2013;13(1):37.

25. Carroll C, Booth A, Cooper K. A worked example of “best fit” frameworksynthesis: a systematic review of views concerning the taking of somepotential chemopreventive agents. BMC Med Res Methodol. 2011;11(1):29.

26. Cohen MZ, Kahn DL, Steeves DL. Hermeneutic phenomenological research:a practical guide for nurse researchers. Thousand Oaks: Sage; 2000.

27. Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies.Newbury Park: Sage; 1988.

28. Melendez-Torres GJ, Grant S, Bonell C. A systematic review and criticalappraisal of qualitative metasynthetic practice in public health to develop ataxonomy of operations of reciprocal translation. Res Synthesis Methods.2015;6(4):357–71.

29. Glaser BG, Strauss A. The discovery of grounded theory. Chicago: Aldine; 1967.30. Dixon-Woods M, Agarwal S, Young B, Jones D, Sutton A. Integrative

approaches to qualitative and quantitative evidence. In: UK National HealthService. 2004. p. 1–44.

• We accept pre-submission inquiries

• Our selector tool helps you to find the most relevant journal

• We provide round the clock customer support

• Convenient online submission

• Thorough peer review

• Inclusion in PubMed and all major indexing services

• Maximum visibility for your research

Submit your manuscript atwww.biomedcentral.com/submit

Submit your next manuscript to BioMed Central and we will help you at every step:

Gentles et al. Systematic Reviews (2016) 5:172 Page 11 of 11