application of systematic reviews in speech-and-language therapy

INT J LANG COMMUN DISORD, MAY–JUNE 2011,VOL. 46, NO. 3, 261–272

Research Report

Application of systematic reviews in speech-and-language therapy

Julie Marshall†, Juliet Goldbart†, Caroline Pickstone§ and Susan Roulstone‡†Faculty of Health, Psychology and Social Care, Manchester Metropolitan University, Manchester, UK‡Faculty of Health and Life Sciences, School of Health and Social Care, University of the West of England, Bristol, UK§South Yorkshire Comprehensive Local Research Network and ScHARR, University of Sheffield, Sheffield, UK

(Received 6 July 2009; accepted 26 May 2010)

Background: Systematic reviews are increasingly being carried out in speech-and-language therapy and are usedby practitioners, service commissioners, policy-makers and researchers to inform decision-making, as the bodyof evidence available about speech-and-language therapy grows. Although systematic reviewing is developing toincorporate new methods of review and synthesis, there are currently limitations in the use of some types ofsystematic reviews within speech-and-language therapy.Aims: The purpose of this paper is to provide an overview of the steps involved in the systematic review process andthe range of options available. It highlights some challenges to using this process in speech-and-language therapy,with examples based in part on the authors’ experiences of involvement in two systematic reviews. A number ofdevelopments in systematic review methodology will be outlined and several new approaches to reviewing, bothwithin and outside of speech-and-language therapy, are introduced. These include realist synthesis, evidence-basedpractice briefs, speech BITETM and the journal Evidence-Based Communication Assessment and Intervention.Main contribution: This paper highlights some of the current benefits and limitations of systematic reviews inspeech-and-language therapy. It will facilitate readers to use and carry out systematic reviews in the speech-and-language therapy field.Conclusions: Systematic reviews are useful in speech-and-language therapy, but awareness of their limitations isimportant to practitioners, commissioners, policy-makers and researchers. New developments may further increasethe benefits of systematic reviews.

Keywords: evidence-based practice, speech-and-language therapy, meta-analysis, systematic review, researchsynthesis.

What this paper addsWhat is already knownA number of systematic reviews have been registered or completed that are relevant to speech-and-language therapyand service provision. Much has been written about systematic reviews and their limitations are being recognized.

What this paper addsThe process of and developments in systematic reviewing are described, with particular reference to paediatricspeech-and-language therapy. This perspective permits a greater understanding of the uses and limitations ofsystematic reviews in this field. The paper also provides information about alternative reviewing systems and newdevelopments.

Background

Systematic reviews of research evidence are widelyregarded as one of the most valued and reliable waysof synthesizing research evidence. Evidence needs tobe provided in digestible formats so that practitioners,

Address correspondence to: Julie Marshall, Speech and Language Therapy Clinic, Elizabeth Gaskell Site, Hathersage Road, Manchester M13OJA, UK; e-mail: [email protected]

commissioners and policy-makers, as well as researchers,may use their findings. Practitioners and commission-ers look to systematic reviews as a way of providinga summary of the latest best evidence on which tobase services. Researchers, applying for new funding and

International Journal of Language & Communication DisordersISSN 1368-2822 print/ISSN 1460-6984 online c© 2010 Royal College of Speech & Language Therapists

DOI: 10.3109/13682822.2010.497530

262 Julie Marshall et al.

proposing new studies, are expected to provide evidenceof the need for new research, by identifying gaps andlimitations in the literature in order to demonstrate thatthe new work proposed will be cumulative, relevant andnovel (Schlosser 2007). New research programmes areexpected to build in systematic reviews as the bedrockof understanding the current state of evidence.

The prominence of systematic reviews, one particu-lar type of review method, has arisen partly inresponse to concerns about the feasibility of practition-ers being able to access and interpret research findingsto support their clinical decisions, due principally totime, resource and access constraints. Most practition-ers are likely to find it difficult to maintain an up-to-date knowledge of all relevant literature in their field.This may be particularly challenging in some parts ofspeech-and-language therapy (SLT), such as in develop-mental language impairment, where there are manyresearchers, from a variety of disciplines, investigat-ing many different perspectives on the nature of theimpairment and possible avenues for intervention. Thisbeing the case, an up-to-date review of research evidencewhich summarizes what is known about a particularapproach to intervention becomes an invaluable sourceof information.

Consumers of research have begun to point out moreforcibly that ‘atomized,’ unsynthesized products of theresearch enterprise are of little help to people who wishto use research to inform their decisions. (Chalmerset al. 2002: 25)

Systematic reviews have developed in the context ofthe need for evidence-based practice and form partof the triad of components which Schlosser (2004),amongst others, describes as contributing to evidence-based practice: ‘research evidence, clinical/educationalexpertise, and relevant stakeholder perspectives’. It isto this first component of evidence-based practice—research evidence—that systematic reviews contribute.

The development of systematic reviewing was alsoprompted by concerns that the use of literature reviewsthat are not systematic can provide a distorted picture,since particular studies/papers might be chosen by areviewer to support a particular argument or perspec-tive, rather than the inclusion of studies in a systematicand transparent fashion. As a result, more rigorous andaccountable methods of reviewing literature have beendeveloped, that is, systematic reviews. A history of thesedevelopments is provided by Chalmers et al. (2002).

Within speech-and-language therapy (SLT), earlyresponses to systematic reviewing were mixed. Earlyengagers with the methodology (such as Law et al.1998, Greener et al. 1999) were criticized by otherssuch as Pring (2004), who argued that the research basewithin SLT was not sufficiently extensive, that there

were insufficient good quality randomized controlledtrials (RCTs) and that the process of meta-analysis andthe use of the results of systematic reviews were inappro-priate to SLT research questions.

Explanations for the limited SLT research baseavailable to systematic reviews are multi-factorial andmay include the following points that have been madeby other researchers. The study of speech, language andcommunication difficulties/disorders is relatively youngand poorly funded. Clinical populations tend to besmall and heterogeneous, making large N experimen-tal designs difficult to achieve. Additionally, interna-tional diversity in language, culture, terminology, servicestructure and provision create challenges in the interna-tional comparison/synthesis of research in this field,a point noted by Pennington et al. (2004a, 2004b).Further discussion on this topic is outside of the scopeof the present paper.

The methodology for systematic reviews, however,has continued to develop. When systematic reviewingwas first developed, it emerged as a discrete techniquewhich was applied to the identification and analysis ofRCTs, using meta-analysis as the method of synthesis.Developments in the field have led to the enuncia-tion of the stages of a systematic review and theexpansion of techniques for use at each stage. Someof these developments may facilitate the increased useof systematic reviews in SLT. Increased understand-ing of systematic reviews may encourage the use ofsystematic approaches, even where full systematic reviewsare not carried out and may also influence the design ofresearch and the reporting of findings (Marshall et al.2005).

It is timely to review the use of systematic reviewswithin SLT. Garrett and Thomas (2006) have advocateda broad view of systematic reviews and argue for a phasedapproach to outcomes research, in which systematicreviews may play a part at various stages. In thiscurrent paper, the stages in the process of undertak-ing a systematic review are described and some ofthe limitations and challenges are considered, withreference to examples from SLT. It examines some ofthe recent methodological developments that have takenplace in systematic reviewing and in related reviewingapproaches, again drawing on examples from SLT andrelated fields. It is not in itself a systematic review of thereviews that have been carried out in SLT. It attempts tomake the systematic review process more accessible tothe reader, in order to support the critical appraisal andinterpretation of such reviews. Like any methodologicalapproach, systematic reviewing is open to interpretationand variations in quality. Therefore, an awareness of theprinciples and how they are applied within the field canaid practitioners in their understanding and interpreta-tion of reviews.

Systematic reviews in speech-and-language therapy 263

What are systematic reviews?

Mulrow et al. (1997: 389) have described systematicreviews as: ‘concise summaries of the best availableevidence that address sharply defined clinical questions’.They assist with the management of large bodies ofinformation. Standardized steps and stringent methodsof selection, quality appraisal, synthesis and therecording of those decisions, all help to reduce bias inthe selection and inclusion of studies/papers in reviewsand in using the findings. Transparent and standardizedmethods also permit the replication and updating ofreviews.

The need for a shared and transparent methodologyfor systematic reviews led to the establishment of theCochrane Collaboration, which has become the ‘goldstandard’ for carrying out systematic reviews (see http://www.cochrane.org/). Since then other reviewing systemshave been established, including the Campbell Collabo-ration (see http://www.campbellcollaboration.org/) thatfocuses on reviews related to social welfare, crime, justiceand education, and Evidence for Policy and PracticeInformation (EPPI) (see http://eppi.ioe.ac.uk/cms/)that carries out reviews in a number of areas includingemployment, social care, crime, justice, education andhealth promotion. A systematic review may be registeredwith one of the systematic review organizations (whichwill provide advice and support with the review process,as well as providing publicity and legitimacy to thereview). However, registering with a reviewing bodymay constrain the review in other ways, for exampleby specifying study designs that may be included.

The number of systematic reviews, registered withthe Cochrane or Campbell Collaborations and directlyrelated to SLT, is growing. At the time of writing,there have been approximately 15 completed reviewspublished by the Cochrane Collaboration (with at leastone more registered and with a protocol published).Within the Campbell Collaboration one review has beenpublished and three more are registered. Within EPPInone are directly related to SLT. There are of courseother reviews of related interventions and activities thatwill be of interest to SLTs, For example, one carriedout by Cajkler et al. (2007) and published by EPPI,reviews the impact of training and professional develop-ment activities on the classroom practice of teachingassistants.

The review process

There are a number of protocols available for carryingout a systematic review and the process has beendescribed accessibly in a number of websites, forexample, http://www.cochrane.org/; http://www.cochrane-net.org/openlearning; http://eppi.ioe.ac.uk/

cms/; and http://www.campbellcollaboration.org/artman2/uploads/1/Review Steps.pdf/; and in academicjournals and books. For example, Gough (2007), Khanet al. (2003) and Schlosser (2007) have all describedthe necessary stages of a systematic review, includingframing the question(s), protocol design, identificationof literature, assessment of studies’ quality, summarizingand interpreting findings and finally reporting anddisseminating results. Whilst differing in detail, the keycomponents are similar.

The next section describes and analyses, stage bystage, the process of protocol development and theexecution of a systematic review. It identifies importantissues for reviews within the field of SLT and draws onsome existing reviews as illustrations.

Setting the review question

In the planning phase of a review, identifying the needfor a review and the development of the question are thefirst stages. As in any research, framing the question iscrucial; it sets the parameters for the systematic reviewand drives the remainder of the process. Most systematicreviews that have been reported to date in the fieldof SLT have focused on evidence regarding the effectsof interventions. Exceptions include that of Law et al.(1998), who reviewed studies of the natural historyand prevalence of speech and language impairment inchildren and the use of screening assessments. Morerecently, Nelson et al. (2006) reviewed evidence forthe effectiveness of screening for pre-school children,Desmarais et al. (2008) reviewed the characteristicsof late-talking toddlers and McCormack et al. (2009)reviewed the association between speech impairment inchildren and their participation across the lifespan.

This apparent focus on interventions is perhapsunsurprising given that until recently, the protocolsavailable to guide the systematic review process haveonly focused on reviews of interventions. However,there are now guidelines available for reviewingother kinds of studies. For example, Stroup et al.(2000) report the consensus guidelines developedfor the reporting of meta-analyses of observa-tional studies in epidemiology—‘MOOSE’. Linksto this and other guidelines are available on theCochrane Collaboration website (see http://www.cochrane.org/index authors researchers.htm).

Despite guidelines, reviewing of interventionsremains difficult because, as Pring (2004) andsubsequent systematic reviewers in SLT (for example,Pennington et al. 2004c) point out, reports ofintervention evaluations are frequently poorly described.Additionally, the types of interventions (who, what,where, how, how much etc) in SLT intervention varywidely across the globe. Therefore, in framing the


question for a review of interventions it is necessary,a priori, to consider criteria and to identify as manyfactors as possible that could influence (1) whether astudy meets review criteria, (2) the results of the reviewand (3) its interpretation. The wording of the reviewquestion and whether terms included in it are universallyunderstood, may also be important. For example, asystematic review by Pickstone et al. (2009) considered‘early years’ interventions for children aged up to 6years. Studies that were carried out in Britain, Irelandor Canada, would include children already in school,whereas in countries such as the United States, Japanand Sweden, these children would be preschoolers. As aresult, interventions included in this review includeda range of settings, including schools, nurseries andhomes, raising issues of comparability of interventions.

There are a number of components of studydesign that need to be specified. Some systematicreviews include only randomized controlled trials(RCTs) because they represent a highly graded sourceof evidence, often comparing one intervention withanother. There are a small number of examples that havetaken this approach in SLT reviews (Deane et al. 2001,Law et al. 2004), whereas others have included a widerrange of designs. For example, Pennington et al. (2004c)included studies that had some ‘element of experimentalcontrol’ (p. 154). This approach is especially valuable ifthere are few RCTs or controlled studies to draw on,or if the aim is to answer more complex questions,for example about how an intervention works and inwhat particular contexts. These questions are increasinglyrecognized as important, particularly as SLT interven-tions are frequently complex and provided to heteroge-neous groups.

The selection of outcomes measured in a studymay be important. For example, which components oflanguage and/or speech, how they are measured (forexample, different tests are standardized on differentpopulations and in different languages), when they aremeasured, whether the outcomes are clinical or not,are proxy outcomes used as markers for change? Forexample, are measures of receptive vocabulary assumedto equate to improvements in all aspects of receptivelanguage?

Each of these elements needs to be defined asexplicitly as possible to inform the development of thesearch strategy in the next stage. Early considerationof the ways in which such factors may vary increasesthe likely validity and usefulness of the review, byrecognizing and highlighting features that may be criticalfor successful outcomes. As with designing an originalresearch study, the design of the review is intimatelybound up with the purpose and question of the research.Time spent at the planning stage is therefore crucial toeach subsequent stage.

Sources of data and the search process

Once the review question has been identified, the processof identifying relevant studies can begin. A searchstrategy is developed which builds on the decisions in theprevious stage. The time frame, the range of databases tobe searched and the language(s) to be included should allbe specified within the search strategy. Careful consider-ation of the implications of these decisions is necessary.For example, restricting the time frame for publicationsmay lead to the omission of significant research relevantto the topic. The selection of publications in Englishlanguage only may limit the contributions of someresearchers and the choice of databases will influencethe journals included, the ‘grey literature’, theses etc.These decisions need to be balanced against the time andresources available for sifting the publications identified.A review’s comprehensiveness will reflect the decisionstaken at this stage.

Consideration of inclusion/exclusion criteria basedon variables such as study subjects’ age, gender,education, socio-economic status etc may also influenceincluded research. For example, the term ‘socio-economic status’ may not be universally understood,defined or measured and data on some variables suchas school performance may not be available to healthresearchers (Pennington et al. 2007) and have differentsignificance across the world.

A relatively new development, realist synthesis(Pawson et al. 2004, 2005) may help our understandingof the complex effects of such variables.

They describe realist synthesis as: ‘a model ofresearch synthesis which is designed to work withcomplex social interventions or programmes, and whichis based on the emerging ‘realist’ approach to evaluation’(Pawson et al. 2005: 21). It aims to determine: ‘whatworks for whom, in what circumstances, in what respectsand how’. This approach involves:

• making explicit the theory or assumptions whichunderpin an intervention, that is, how it is thoughtto work and with what effect;

• seeking empirical evidence to support, challengeor adapt the theory; and

• bringing together the programme theory andresearch evidence in order to clarify understand-ing of the programme mechanism, its context andoutcomes.

In an attempt to acknowledge and understand theimpact of inclusion/exclusion criteria Pickstone et al.(2009) drew on the methods of realist synthesis incarrying out their systematic review of environmentalinterventions to improve child language outcomes forchildren with or at risk of primary language impairment.


They ‘mind-mapped’ mechanisms that they hypothe-sized could be occurring in research designs/interventionapproaches. A small pilot search was carried out beforeembarking on the full systematic review. These twoactivities guided the development of appropriate searchterms and thus the identification of relevant literature.The process of synthesis in their review is to include afine grained analysis of studies and their findings.

One of the challenges in the field of SLT researchis the wide number of terms that are used in researchand that many of these terms have multiple connota-tions. For example, the terms ‘speech’, ‘language’ and‘communication’ are all used in lay contexts and‘communication’ is often used in computer sciencecontexts. Additionally there is a plethora of terminologywith similar but not identical meanings (that may not becovered by Medical Subject Headings (MeSH) terms),for example, language, talking, speaking, communicat-ing, chatting, articulation, pronunciation, dysarthria.These differences may be even more evident if searchingis outside of health and educational fields and inlanguages other than English. Searching can thereforegenerate vast quantities of literature that have littlerelevance to the topic; subtle changes in the search termsand how they are organized can alter the sensitivity andspecificity of the search. An example of search termsby Roulstone et al. (2008) in appendix A illustrates thetypical complexity of such terms. The aim is to maximizesensitivity, in order to be as comprehensive and inclusiveas possible, whilst simultaneously being aware of the biasthat can be introduced through exclusivity and acknowl-edging possible resource constraints.

The use of systematic review experts is to berecommended. As an example, Swinkels et al. (2006)describe an iterative process in a physiotherapy relatedsystematic review, with two physiotherapists workingwith a librarian to generate the best possible combina-tion of search terms and databases. It may be appropri-ate, for example, to develop a number of alternativesearch strings to investigate the numbers of papersyielded. A random sample of papers can then be checkedto see what percentage match the inclusion criteria forthe different search strings, thus revealing the sensitivityand specificity of any particular combination.

Inclusion and exclusion criteria defined in theplanning stage guide the subsequent sifting of the studiesin order to identify those that are relevant to theresearch question. Copies of the complete papers arethen obtained for all references that have not so far beenexcluded. Inter-rater reliability checks at this stage aimto reveal any bias in decision-making.

One would expect to see the proposed methodsof the review defined a priori, thus reducing thebias introduced by reactive decision-making. However,Gough (2007) suggests that in some reviews, the process

can be emergent and iterative. So, for example, hesuggests that inclusion and exclusion criteria may not beconfirmed until the later stages of a review, as the featuresof studies which make it possible to evaluate interven-tions emerge from the analysis. Some proponents ofsystematic reviews might argue that such an approachgoes against the systematicity of a review. However, if theprinciples of transparent and explicit decision-makingare observed, then emergent and iterative methods ofdefining the parameters of a review can support aprogressive framing of the review focus. These issuesshould be detailed in the protocol, making it clear fromthe outset that certain decisions will be made or reviewedat specified stages in the review.

Explicit and clear reporting of any such testing,along with the explicit descriptions of the subsequentdecisions, would be the hallmark of a systematicsearch and recently. Booth (2006) has developedexplicit STAndards for Reporting LITErature searches(STARLITE). STARLITE provides the mnemonic forthe standards which are concerned with the Samplingstrategy, the Type of study, Approaches, Range of years,Limits, Inclusions and exclusions, Terms used andElectronic sources.

Quality appraisal

Having identified all the papers that meet the inclusioncriteria, the quality of each paper is assessed. The design,conduct and analysis of each separate study reported onis appraised. This may be carried out either using astandard published appraisal checklist, or one designedby the review team, to meet their particular needs. AsGough (2007) points out, some components of qualityappraisal may be performed at the stage of sifting orinitial searching of papers, where the inclusion criteriaspecify certain types of study, for example only RCTs orstudies with an element of control. Appraisal checklistsor frameworks require the examination of the papers forsources of bias (internal and external validity), selectionbias (participants), performance bias (variations in theways that the intervention has been delivered), measure-ment bias (blinding) and attrition bias (loss to follow-up). Khan et al. (2003) point out that the qualityassessment framework chosen may need to take accountof more than one type of research design, especiallywhere reviews are in areas where little RCT evidence isavailable. Whilst recognizing the apparent pragmatismof such an approach, they highlight potential problemsfor evidence synthesis, particularly if a statistical meta-analysis is proposed.

There is a wide range of quality checklists forthe systematic reviewer to choose from, that have inturn been assessed for their quality and application(for example, West et al. 2002). Katrak et al. (2004)


identified 121 tools for the purposes of critical appraisaland concluded that there was no tool that could beregarded as a ‘gold standard’ for use by allied healthresearchers. Furthermore, Downs and Black (1998) notethat few tools have been tested for their inter-raterreliability in appraising studies. Consequently, mosthigh-quality systematic reviews build in an ‘inter-raterreliability’ check for both selecting studies for inclusionand also for quality appraisal and reporting.

The use of simple checklists has been criticized asthey cover only the more generic design features of astudy and fail to take account of the relevance of thefindings to the specific clinical question under consider-ation. Gough (2007) argues that if research is to beof value in applied settings, then researchers shouldnot just be using what he refers to as ‘abstract genericcriteria’, in quality appraisal but rather should also belooking for the ‘fitness of purpose’ of studies and therelevance of the research for answering conceptual orempirical questions. Thus, he argues that, rather thanmaking an all-or-none decision about including studies,it is more helpful to think about the ‘weighting ofstudies’. He suggests that the quality appraisal of studiesshould therefore include making judgements about thetrustworthiness of a study (that is, is the quality ofthe study good or adequate, irrespective of the actualreview question), the relevance of the study methods tothe specific review question and whether the focus ofthe study is appropriate to the review questions (EPPI2009). Again, it is important that all these decisions aremade explicitly and transparently.

Systematic reviews within SLT have used variousapproaches to the quality appraisal of the research.Pennington et al. (2004c), for example, reported onthe quality of included studies, indicating whether themethodology of the study was described adequately,partially addressed or unclear; features of the studiescovered by their quality appraisal included theassignment of participants, blinding, adequacy ofsample size and reporting of missing data. Desmaraiset al. (2008) reviewed studies which identified the‘characteristics of late talking toddlers’ (not interven-tion studies). They critiqued the studies in terms ofthe definitions of ‘late-talking’, the range of measuresthat were used and included only those studies whichreported the use of psychometrically valid instruments.They did not include an assessment of the overall qualityof the included studies.

Synthesizing/summarizing and interpreting

The stage of synthesis—pulling together the body ofevidence, is arguably the purpose of a systematic review.In this stage, the aim is firstly to summarize the evidenceand provide descriptive information about populations,

interventions, outcomes and designs and then to searchfor consistencies and inconsistencies in the findings.Using standardized ‘data extraction’ forms, data fromeach study are documented and collated in order tomap each study’s findings with respect to the reviewquestion. Depending on the nature of the data beingextracted, the method of synthesis may be described asbeing either integrative or interpretative.

Integrative synthesis

Building on the work of Noblit and Hare (1988), Dixon-Woods et al. (2005) suggest that integrative synthesesare summative in nature and are appropriate where thevariables to be summarized are ‘largely secure and wellspecified’. This kind of synthesis may include meta-analysis, data pooling or descriptive accounts of theliterature. Very few systematic reviews in SLT have usedstatistical meta-analysis. The systematic review by Lawet al. (2004) is one of the few that achieves this. Intheir review they included 25 RCTs evaluating interven-tions in primary speech and language impairment. Theyprovided descriptive data about the studies and thenused meta-analysis to identify a positive effect of theinterventions for particular subgroups of children.

The advantage of meta-analysis is that it provides astandardized, explicit and repeatable method of bringingtogether results from various studies. However, the use ofthis statistical approach within SLT has been criticized.Pring (2004), for example, argues that the variabilityin interventions undermines the value of results fromthe meta-analysis—where results are pooled for meta-analysis, the variation of intervention makes it impossi-ble to determine precisely the effective component.As noted earlier, Pring concluded that the SLT fieldis therefore not ready for systematic reviews since itdoes not yet have a sufficient number of studies ofwell defined and theoretically based interventions tocombine in meta-analyses. Proponents of systematicreviewing have since highlighted the use of the othermethodologies for systematic reviewing which allow theinclusion of studies other than RCTs and studies whichdo not fully meet the quality criteria of the review. Thesemay use methods of synthesis other than meta-analysis(for example, Garrett and Thomas 2006, Beeson andRobey 2006, Robey 1998, Schlosser 2007). For exampleBeeson and Robey (2006) point out that whilst the idealmight be to base conclusions about the evidence base onmeta-analysis of high quality RCTs, in the absence ofsuch studies, systematic reviews can examine whateverstudies constitute the ‘best current evidence’. Schlosser(2007), drawing on Scruggs and Mastropieri (1998),suggests a two stage approach in which firstly all availablestudies are analysed (regardless of quality ratings) and theextent of co-variance between quality scores and study


outcomes are then explored. A subsequent synthesis of‘best evidence’ is carried out if co-variance is found.

Methods of synthesis other than meta-analysisare, however, relatively under-developed and concernremains about their robustness and susceptibility tobias (Dixon-Woods et al. 2005). In response to thisproblem, guidance has been developed to support theuse of narrative synthesis within systematic reviews.Rodgers et al. (2009) suggest that a narrative summaryof the results of studies can be used alongside orinstead of meta-analysis. In a comparison of findingsfrom a narrative synthesis with one using meta-analysis,Rodgers et al. reached broadly similar conclusions forboth methods but concluded that the two made differentcontributions to the review, with meta-analysis offeringmore structured analysis regarding moderators of effects,and narrative synthesis providing more insight into thepotential confounders and moderators that were nottaken into account by meta-analysis. Cirrin and Gillam’s(2008) review of language intervention practices forschool-age students with spoken language disordersand Pickstone et al. (2009) (see above) both draw onnarrative synthesis.

A number of SLT systematic reviews have designatedthe review to include only RCTs but then have beenunable to identify any relevant studies of sufficientquality for inclusion. Since a meta-analysis is clearlyimpossible, authors have taken different approaches tosubsequent analysis. Sellars et al. (2002), for example, intheir review of SLT for dysarthria due to non-progressivebrain damage, identified no RCTs of sufficient qualityand therefore did not progress their analysis of theliterature further. Therapies and management strategies[. . .] cannot presently be supported or refuted by good-quality evidence from RCTs. (p 65). Other reviewershowever, having established the absence of RCTs inan area, have reported on good quality evidence fromother study types. For example, the protocol of Morganand Vogel’s (2008) review of interventions associatedwith acquired brain injury in children and adolescentsincluded only RCTs and quasi-experimental designs.However, the only relevant studies they were able toidentify were case studies and so they provided ananalysis of the quality of these case studies. Although thefindings from the case studies were excluded from thefinal conclusions of the review, the text of the review doestherefore provide information on the ‘best evidence’,available to date.

Apart from the Law et al. (2004) and Robey (1998)reviews, which used meta-analysis, other systematicreviews in SLT have used alternative methods of integra-tive synthesis. Pennington et al. (2004a, 2004b), forexample, reported that the size and diversity of studiesincluded in their systematic review precluded the use ofmeta-analysis; instead they used a descriptive summary

of the papers included under the different types oftherapy identified.

Interpretive synthesis

Interpretive syntheses are concerned with the develop-ment of concepts and theories where the outputis theory, grounded in the data of the originalstudies. Initially interpretive approaches were testedwith primary studies that had used qualitative methods.Synthesis techniques using approaches such as meta-ethnography were developed. In meta-ethnographymodels or higher order theories of human behaviour orexperiences are derived from an interpretive synthesisof the findings of ethnographic research. Meta-ethnography has not been used in SLT to date—notleast because of the dearth of primary qualitative studies.In related fields such as education, however, otherinterpretive approaches have been used to synthesizea range of empirical studies. For example, Cajkleret al. (2007), in a systematic review of the impactof professional training and activity on the classroompractice of teaching assistants, used interpretive methodsto synthesize findings from studies with a wide rangeof designs. Interpretive methods have also been usedin healthcare, for example, in developing a model foradherence to therapy for tuberculosis (Atkins et al.2008). Studies included in Atkins et al.’s review werenot restricted to RCTs; rather the review included anyprimary empirical research of relevance to the topic;this is likely to include qualitative data from ethnogra-phies, Grounded Theory and Critical Theory research.In Cajkler et al.’s (2007) review, following extensivemapping of the studies, their contexts and details ofthe training covered, the studies were then examinedexhaustively for all reports of the impacts of training.These ‘data’ were then coded into themes using theconstant comparison method (a key feature of GroundedTheory methodology).

Realist synthesis (Pawson et al. 2004) (see above)offers a development in interpretive synthesis that maybe applicable to some SLT research. They argue (pp.iv–v) that it is not (intervention) programmes per sethat work, but the underlying resources or rationalesthat are provided for the subjects, that produce thechange. They emphasize the need to consider the theoryunderpinning intervention approaches and argue thatthe power of an intervention lies in its underlyingmechanism. Furthermore, whether or not a mechanismis triggered depends on the context, that is, the character-istics of subjects and the locality. This considerationof theory and mechanisms in turn allows meaningfulcomparisons between studies to be made. Furthermore,Pawson (2006) controversially argues that even ‘badresearch’ can yield ‘good evidence’; thus it is not so


much the generic quality of the studies which he rates asimportant but what they contribute to an understandingof the mechanism by which a programme operates.

To date, although there are no published systematicreviews in SLT that have taken an interpretive approachto synthesis, an awareness of the possibilities that it offersmay facilitate a step change in our understanding of themechanisms which underpin interventions.

Dissemination

The final and vital part of a systematic review is thedissemination of findings in order that they can beused to influence practice, policy-making, commission-ing and further research. Systematic reviews are widelyregarded as the highest level of evidence to supportinterventions. A systematic review that uses meta-analysis should provide confidence that the particularintervention approach reviewed would yield a similareffect size if used with population and contexts similar tothose included in the review. However, this view assumesthat systematic reviews include only high quality RCTsand use meta-analysis to estimate the overall effect sizesof a particular type of intervention. With the evolutionof systematic review methodology, the assumptionsunderpinning some reviews have changed. As indicatedabove, some systematic reviews provide ‘best availableevidence’ and others provide ‘weighted evidence’, whichmust be interpreted with caution. The process that hasbeen used should be made explicit in dissemination andshould be understood by all those who use the findingsof systematic reviews in their work.

SLTs’ own clinical populations and clinical practiceinfrequently match exactly the subjects and interven-tions described in reviews. For example, Raghavendra(2007) notes, regarding those studies in Penning-ton et al.’s (2004c) systematic review (of speech-and-language therapy for children with cerebral palsy) thatincluded single-subject experimental design:

Although the studies showed that speech–languagetherapy (SLT) based on operant and micro teachingmethods was effective in facilitating the outcomesmentioned [. . .] due to the nature of the study design,generalization to other children with CP was difficult.(pp. 103–104)

Similarly, Johnson (2005) comments that theconclusions drawn by Law et al. (2004) about theeffectiveness of SLT, are based mostly on preschoolchildren and cannot therefore be taken as an indicatorof effectiveness across a wider caseload. However, asLaw et al. (2005) note, the very transparency which isa hallmark of the systematic review facilitates this kindof critique, whereby the explicit statements regarding

inclusion criteria of a review allow readers to makeinformed decisions about the relevance of the review.

Researchers and practitioners will find the detailof reviews helpful in identifying the primary researchstudies and in understanding the process of qualityappraisal. For practitioners, reviews may not alwaysprovide the definitive answer to what interventionshould be used with a particular client (group). Clinicaljudgement, in addition to client, family and stakeholderviews are still required to decide on the applicability to anindividual client or service context. Those reviews whichprovide an analysis of the current state of the evidence areperhaps more helpful to practitioners—they fulfil oneof the original aims of systematic reviews, which is toamass and distil a complex literature. Those which showonly that there are no studies meeting the gold standardof high quality RCTs are less helpful for the practitioner,although their value in identifying the gaps in a field,the need for specific research projects, programmes ofresearch and/or profession/clinical group-wide researchstrategies, are recognized.

Developments in reviewing procedures

Systematic reviews have not been the only develop-ment in the reviewing process that have occurred inresponse to the evidence-based practice movement.Recognizing concerns about the resources needed toconduct systematic reviews, a number of other reviewingapproaches have been developed which address (oftenspecific, clinical) questions and they can be less resourceintensive. However when using their findings, it isimportant to be aware of whether a reduction inthe resources used may have been achieved at thecost of compromising on aspects of the process.For example, research design, sampling, and recruit-ment and allocation strategies, blinding, attrition, dataanalysis etc, may not be thoroughly scrutinized.

A small number of these additional reviewingsystems are specific to speech-and-language therapy andthree of them are described briefly below.

• Evidence-Based Practice Briefs is a quarterly, freeelectronic and print-based journal, specific toSLT, that provides topic-based reviews. Based oneveryday questions from clinical practice, each‘brief’ provides a discrete systematic review ofevidence presented in an accessible format, as wellas information about research design. Therapistscan also submit suggestions for reviews. Thesite is produced by the publisher, Pearson (seehttp://www.speechandlanguage.com/ebp/).

• Speech BITETM is an open-access websitedeveloped by the University of Sydney andSpeech Pathology Australia, offering ‘a catalogue


of Best Interventions and Treatment Efficacyacross the scope of Speech Pathology practice’.It includes a range of evidence types, includingsystematic reviews, RCTs, non-RCTs, case seriesand single case experimental designs with aquality rating for some types of study (seehttp://www.speechbite.com).

• The Evidence-Based Communication Assessmentand Intervention journal is edited by Ralf Schlosserand Jeff Sigafoos and provides critical appraisals ofand commentaries on selected research studies andreviews in the broad field of speech-and-languagetherapy. They also include some discussion ofimplications for practice, so that the reviewsare of relevance to practitioners as well asresearchers.

Despite the limited number of systematic reviews in SLTthere are an increasing number of publications address-ing evidence-based practice and encouraging cliniciansto consider and integrate clinical expertise, stakeholderperspectives and evidence, for example, Gillam andGillam (2006), Garrett and Thomas (2006), and Dodd(2008).

Other rapid review sites have been developed outsideof speech-and-language therapy but provide interestingmodels for future developments within the field. Theseare typically in medicine and three examples are givenbelow:

• BestBETs (Best Evidence Topics) forEmergency Medicine topics (see http://www.bestbets.org/home/bets-introduction.php).

• POEMs (Patient Oriented Evidence that Matters).These consist of summary of a valid piece ofresearch that carries information that is importantto patients and their doctors.

• CATs (Critically Appraized Topics). The Centrefor Evidence Based Medicine at Oxford hasdeveloped ‘CAT maker’—a tool for support-ing practitioners to develop a clinical question,and a search strategy (see http://www.cebm.net).

Conclusion

Although the use of systematic reviews within speech-and-language therapy is increasing, their usefulness andappropriacy is not fully agreed upon. The reductivenature of some systematic reviews may be viewed asproblematic. Robust intervention studies of any kind(not just RCTs) are still relatively few in number in SLT,as indicated above and by others. A range of methods ofreviews and interpretation are therefore to be welcomed,at least as an interim measure.

Systematic reviewing should not just be seen asa single method; there are as many ways to conductsystematic reviews as there are questions. There is a needto engage with developments in reviewing approachesin order to develop appropriate methodologies forSLT. There are some basic principles about systematicreviewing regarding transparency, reducing bias and thebasic stages as outlined by authors such as Khan et al.(2003). Methods of reviewing continue to develop tofit the type of research questions that are being askedand the research being reviewed. This widening arrayof methodologies available at each stage of the processallows researchers to set their research question and thento select methods which best suit their question. Theincreasing exploration of this approach within SLT willenable the field to address concerns expressed by Pring(2004) and Johnson (2005).

The methods being explored within SLT reflectthose emerging in the wider field of systematicreviewing. Guidance is being developed to assistreviewers in making their reviews truly systematic.Criticisms of reviewing and the emergence of newsystematic review methods have also had a more generalimpact on how researchers describe other sorts of reviewsthat are not necessarily as comprehensive as a fullsystematic review, with more explicit descriptions ofthe search, sifting and quality appraisal processes. Inorder for practitioners, researchers, policy-makers andcommissioners to maximize the benefit of systematicreviews, it is important that they understand both theprocess and the limitations of systematic reviewing sothat the quality of reviews can be evaluated. Further-more, an awareness of alternative reviewing systems, thatcan be used when full systematic reviews are unavailableor not feasible, is valuable.

Acknowledgements

The authors are grateful to the reviewers for their helpful commentson the first draft of this paper.

References

ATKINS, S., LEWIN, S., SMITH, H., ENGEL, M., FRETHEIM,A. and VOLMINK, J., 2008, Conducting a meta-ethnography of qualitative literature: lessons learnt.BMC Medical Research Methodology, 8, 21 (available at:http://www.biomedcentral.com/1471-2288/8/21) (accessedon 13 February 2009).

BEESON, P. and ROBEY, R., 2006, Evaluating single subject treatmentresearch: lesson learned from the aphasia literature. Neuropsy-chological Review, 16 (4), 161–169.

BOOTH, A., 2006, ‘Brimful of STARLITE’: towards standards forreporting literature searches. Journal of the Medical LibraryAssociation, 94 (4), 421–429.

CAJKLER, W., TENNANT, G., TIKNAZ, Y., SAGE, R., TUCKER,S., TAYLOR, C. and GROSSETESTE, B., 2007, A Systematic


Literature Review on How Training and Professional Develop-ment Activities Impact on Teaching Assistants’ Classroom Practice(1988–2006). EPPI Centre Report Number 1507T (London:University of London, EPPI Centre).

CHALMERS, I., HEDGES, L. and COOPER, H., 2002, A brief historyof research synthesis. Evaluation and the Health Professions, 2(1), 12–37.

CIRRIN, F. and GILLAM, R., 2008, Language intervention practicesfor school-age children with spoken language disorders: asystematic review. Language, Speech and Hearing Services inSchools, 39 (Supple.), 110–137.

DEANE, K., WHURR, R., PLAYFORD, E., BEN-SCHLOMO, Y. andCLARKE, C., 2001, Speech-and-language therapy versusplacebo or no intervention for dysarthria in Parkinson’sdisease. Cochrane Database of Systematic Reviews, issue 2.

DESMARAIS, C., SYLVESTRE, A., MEYER, F., BAIRATI, I. and ROULEAU,N., 2008, Systematic review of the literature on the character-istics of late-talking toddlers. International Journal Languageand Communication Disorders, 43 (4), 361–389.

DIXON-WOODS, M., ARGARWAL, S., JONES, D., YOUNG, B. andSUTTON, A., 2005, Synthesizing qualitative and quantita-tive evidence: a review of possible methods. Journal of Healthservices Research Policy, 10 (1), 45–53.

DODD, B., 2008, Speech–language therapy and evidence-basedpractice In V. Joffe, M. Cruice and S. Chiat (eds), LanguageDisorders in Children and Adults: New Issues in Research andPractice (Chichester: Wiley-Blackwell), pp. 54–67.

DOWNS, S. and BLACK, N., 1998, The feasibility of creating achecklist for the assessment of the methodological qualityboth of randomised and non-randomised studies of healthcare interventions. Journal of Epidemiology and CommunityHealth, 52, 377–384.

EVIDENCE FOR POLICY AND PRACTICE INFORMATION (EPPI),2009, Quality and Relevance Appraisal (availableat: http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=177)(accessed on 1 April 2009).

GARRETT, Z. and THOMAS, J., 2006, Systematic reviews and theirapplication to research in speech-and-language therapy: aresponse to T.R. Pring’s ‘Ask a silly question: two decadesof troublesome trials’. International Journal of Language andCommunication Disorders, 41 (1), 95–105.

GILLAM, S. and GILLAM, R., 2006, Making evidence-based decisionsabout child language intervention in schools. Language,Speech and Hearing Services in Schools, 37, 304–315.

GOUGH, D., 2007, Weight of evidence: a framework for the appraisalof the quality and relevance of evidence. Research Papers inEducation, 22 (2), 213–228.

GREENER, J., ENDERBY, P. and WHURR, R., 1999, Speech-and-language therapy for aphasia following stroke. CochraneDatabase of Systematic Reviews, issue 4.

JOHNSON, J., 2005, Letters to the Editor. Re: Law, Garrett &Nye (2004) The efficacy of treatment for developmentalspeech and language delay/disorder: a meta-analysis. Journalof Speech, Language, and Hearing Research, 48, 1114–1120.

KATRAK, P., BIALOCERKOWSKI, A., MASSEY-WESTROPP, N., SARAVANA

KUMAR, V. and GRIMMER, K., 2004, A systematic review ofthe content of critical appraisal tools. BMC Medical ResearchMethodology, 4, 22. doi: 10.1186/1471-2288-4-22.

KHAN, K., KUNZ, R., KLEIJNEN, J. and ANTES, G., 2003, SystematicReviews to Support Evidence-Based Medicine: How to Reviewand Apply Findings of Healthcare Research (London: RoyalSociety of Medicine).

LAW, J., BOYLE, J., HARRIS, F., HARKNESS, A. and NYE, C., 1998,Screening for speech and language delay: a systematic reviewof the literature. Health Technology Assessment, 9 (2), 1–184.

LAW, J., GARRETT, Z. and NYE, C., 2004, The effectiveness of speech-and-language therapy intervention for children with primaryspeech and language delay or disorder. Cochrane Library,issue 3.

LAW, J., GARRETT, Z. and NYE, C., 2005, The specificity of asystematic review is the key to its value: a response to Johnson2005. Journal of Speech, Language, and Hearing Research, 48,1114–1120.

MARSHALL, J., PENNINGTON, L. and GOLDBART, J., 2005, Describingparticipants, partners and environments in AAC research. InS. von Tetzchner and M. de Jesus Goncalves (eds), Theoreticaland Methodological Issues in Research on Augmentative andAlternative Communication (Toronto, ON: ISAAC), pp. 63–72.

MCCORMACK, J., MCLEOD, S., MCALLISTER, L. and HARRISON, L.,2009, A systematic review of the association of childhoodspeech impairment and participation across the lifespan.International Journal of Speech–Language Pathology, 11 (2),155–170.

MORGAN, A. and VOGEL, A., 2008, Intervention for dysarthriaassociated with acquired brain injury in children andadolescents. Cochrane Database of Systematic Reviews,issue 3.

MULROW, C., COOK, J. and DAVIDOFF, F., 1997, Systematic reviews:critical links in the great chain of evidence. Editorial. Annalsof Internal Medicine, 126 (5), 389–391.

NELSON, H., NYGREN, P., WALKER, M. and PANOSCHA, R., 2006,Screening for speech and language delay in preschool children:systematic evidence review for the US Preventive Services TaskForce. Pediatrics, 117 (2), e298–e319.

NOBLIT, G. and HARE, R., 1988, Meta-Ethnography: SynthesisingQualitative Studies (Newbury Park, CA: Sage).

PAWSON, R., 2006, Digging for nuggets: how ‘bad’ research canyield ‘good’ evidence. International Journal of Social ResearchMethodology, 9 (2), 127–142.

PAWSON, R., GREENHALGH, T., HARVEY, T. and WALSHE, K.,2004, Realist Synthesis: An Introduction. Research MethodsProgramme Working Paper Series (Manchester: ESRC).

PAWSON, R., GREENHALGH, T., HARVEY, T. and WALSHE, K., 2005,Realist review—a new method of systematic review designedfor complex policy interventions. Journal of Health ServicesResearch and Policy, 10 (1), 21–34.

PENNINGTON, L., GOLDBART, J. and MARSHALL, J., 2004a, Directspeech-and-language therapy for children with cerebral palsy:findings from a systematic review. Developmental Medicineand Child Neurology, 47 (1), 57–63.

PENNINGTON, L., GOLDBART, J. and MARSHALL, J., 2004b, Interac-tion training for conversational partners of children withcerebral palsy: a systematic review. International Journal ofLanguage and Communication Disorders, 39 (2), 151–170.

PENNINGTON, L., GOLDBART, J. and MARSHALL, J., 2004c, SpeechTherapy to improve the communication skills of childrenwith cerebral palsy. Cochrane Library, issue 1.

PENNINGTON, L., MARSHALL, J. and GOLDBART, J., 2007, Describ-ing participants in AAC research and their communicativeenvironments: guidelines for research and practice. Disabilityand Rehabilitation, 29, 521–535.

PICKSTONE, C., GOLDBART, J., MARSHALL, J., REES, A. andROULSTONE, S., 2009, A systematic review of environmen-tal interventions to improve child language outcomes forchildren with or at risk of primary language impairment.Journal of Research in Special Educational Needs, 9 (2), 66–79.

PRING, T., 2004, Ask a silly question: two decades of troublesometrials. International Journal of Language and CommunicationDisorders, 39 (3), 285–302.


RAGHAVENDRA, P., 2007, Direct speech-and-language therapy forchildren with cerebral palsy, focusing on communicationand expressive language, appears effective but methodolog-ical flaws advocate for more high quality research. Evidence-Based Communication Assessment and Intervention, 1 (3), 103–104.

ROBEY, R., 1998, A meta-analysis of clinical outcomes in thetreatment of aphasia. Journal of Speech, Language and HearingResearch, 41 (1), 172–187.

RODGERS, M., SOWDEN, A., PETTICREW, M., ARAI, L., ROBERTS, H.,BRITTEN, N. and POPAY, J., 2009, Testing methodologicalguidance on the conduct of narrative synthesis in systematicreviews. effectiveness of interventions to promote smokealarm ownership and function. Evaluation, 15 (1), 49–74.

ROULSTONE, S., PICKSTONE, C., ENDERBY, P., GOLDBART, J. andMARSHALL, J., 2008, Experiences and outcomes of environ-mental influences and interventions for primary languagedelay. Unpublished report (Sheffield: ScHARR Library,University of Sheffield).

SCHLOSSER, R., 2004, Evidence-based practice in AAC: 10 points toconsider. ASHA Leader, 22 June, 6–7 10–11.

SCHLOSSER, R., 2007, Appraising the Quality of Systematic Reviews.Focus: Technical Brief Number 17 (Austin, TX: National

Center for the Dissemination of Disability and RehabilitationResearch), pp. 1–8.

SCRUGGS, T. and MASTROPIERI, M., 1998, Summarizing single-subject research: Issues and applications. Behavior Modifica-tion, 22, 221–242.

SELLARS, C., HUGHES, T. and LANGHORNE, P., 2002, Speech-and-language therapy for dysarthria due to nonprogressive braindamage: a systematic Cochrane review. Clinical Rehabilita-tion, 16 (1), 61–68.

STROUP, D., BERLIN, J., MORTON, S., OLKIN, I., WILLIAMSON, G.,RENNIE, D., MOHER, D., BECKER, B., SIPE, T. and THACKER,S., 2000, Meta-analysis of observational studies in epidemiol-ogy: a proposal for reporting. Journal of the American MedicalAssociation, 283 (15), 2008–2012.

SWINKELS, A., BRIDDON, J. and HALL, J., 2006, Two physiothera-pists, one librarian and a systematic literature review: collabo-ration in action. Health Information and Libraries Journal , 23,248–256.

WEST, S., KING, V., CAREY, T., LOHR, K., MCKOY, N., SUTTON, S.and LUX, L., 2002, Systems to Rate the Strength of ScientificEvidence. Evidence Report, Technology Assessment Number47. AHRQ Publication Number 02-E016 (Rockville, MD:Agency for Healthcare Research and Quality).


Appendix A

Sample Search terms for Embase (via OVID) for a systematic review of environmental interventions to improve child language outcomes forchildren with or at risk of primary language impairment (Roulstone et al. 2008).

∗terms expandedPaediatrics, child, infant, toddler, boy, girl, school, children, preschool

AND

Speech disorder, speech intelligibility, speech therap∗, lang therap∗, speech development, speech delay, language disorder, language developmentdisorder, sign language, child language, language therap∗, language development, language delay,

OR

Nonverbal communication, communication development, communication disorders, speech disorders, speech intelligibility rehabilitation ofspeech and language disorders/language delay/speech therapy disorders, child language

OR

Maternal responsiveness, directiveness, maternal interactive styles, compliance, maternal personality, child temperament.

ANY

RCT, controlled clinical trial, random allocation, double blind, single blind, clinical trial, placebo,

research design, comparative study, evaluation studies, follow-up studies, prospective studies

NOT

Mental retardation, child developmental disorders/pervasive/aspergers, cleft lip or palate, OME, hearing loss, blindness, stuttering, aphonia,pain, crying, analgesia, reading, dyslexia, cerebral palsy, AAC, aged, geriatrics, animal

LIMIT

1996–2007

application of systematic reviews in speech-and-language therapy

Documents