Systematic Reviews 5 8

Download Systematic Reviews 5 8

Post on 07-Apr-2018

218 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • 8/6/2019 Systematic Reviews 5 8

    1/65

    Guidelines for performing Systematic Literature Reviews in

    Software Engineering

    Version 2.3

    EBSE Technical ReportEBSE-2007-01

    Software Engineering GroupSchool of Computer Science and Mathematics

    Keele UniversityKeele, Staffs

    ST5 5BG, UK

    and

    Department of Computer ScienceUniversity of Durham

    Durham,UK

    9 July, 2007

    Kitchenham, 2007

  • 8/6/2019 Systematic Reviews 5 8

    2/65

    0. Document Control Section

    0.1 Contents

    0. Document Control Section......................................................................................i0.1 Contents ..........................................................................................................i0.2 Document Version Control .......................................................................... iii0.3 Document development team ........................................................................v0.4 Executive Summary......................................................................................vi0.5 Glossary ........................................................................................................vi

    1. Introduction............................................................................................................11.1 Source Material used in the Construction of the Guidelines .........................11.2 The Guideline Construction Process..............................................................21.3 The Structure of the Guidelines .....................................................................21.4 How to Use the Guidelines ............................................................................2

    2. Systematic Literature Reviews ..............................................................................32.1 Reasons for Performing Systematic Literature Reviews ...............................32.2 The Importance of Systematic Literature Reviews........................................32.3 Advantages and disadvantages ......................................................................42.4 Features of Systematic Literature Reviews....................................................42.5 Other Types of Review..................................................................................4

    2.5.1 Systematic Mapping Studies..................................................................42.5.2 Tertiary Reviews....................................................................................5

    3. Evidence Based Software Engineering in Context ................................................54. The Review Process...............................................................................................65. Planning .................................................................................................................7

    5.1 The need for a systematic review...................................................................75.2 Commissioning a Review ..............................................................................85.3 The Research Question(s)..............................................................................9

    5.3.1 Question Types ......................................................................................95.3.2 Question Structure ...............................................................................10

    5.4 Developing a Review Protocol ....................................................................125.5 Evaluating a Review Protocol......................................................................135.6 Lessons learned for protocol construction...................................................14

    6. Conducting the review.........................................................................................146.1 Identification of Research............................................................................14

    6.1.1 Generating a search strategy ................................................................146.1.2 Publication Bias ...................................................................................156.1.3 Bibliography Management and Document Retrieval ..........................166.1.4 Documenting the Search......................................................................166.1.5 Lessons learned for Search Procedures................................................17

    6.2 Study Selection ............................................................................................186.2.1 Study selection criteria.........................................................................186.2.2 Study selection process........................................................................196.2.3 Reliability of inclusion decisions.........................................................20

    6.3 Study Quality Assessment ...........................................................................206.3.1 The Hierarchy of Evidence..................................................................21

    6.3.2 Development of Quality Instruments...................................................226.3.3 Using the Quality Instrument...............................................................28

    i

  • 8/6/2019 Systematic Reviews 5 8

    3/65

    6.3.4 Limitations of Quality Assessment......................................................296.4 Data Extraction ............................................................................................29

    6.4.1 Design of Data Extraction Forms ........................................................296.4.2 Contents of Data Collection Forms......................................................30

    Cross-company model .................................................................................31

    Within-company model.................................................................................31What measure was used to check the statistical significance ofprediction accuracy (e.g. absolute residuals, MREs)?...........................32What statistical tests were used to compare the results?.......................32What were the results of the tests? ............................................................32Data Summary ...............................................................................................32

    6.4.3 Data extraction procedures ..................................................................336.4.4 Multiple publications of the same data ................................................336.4.5 Unpublished data, missing data and data requiring manipulation .......346.4.6 Lessons learned about Data Extraction................................................34

    6.5 Data Synthesis..............................................................................................34

    6.5.1 Descriptive (Narrative) synthesis.........................................................346.5.2 Quantitative Synthesis .........................................................................356.5.3 Presentation of Quantitative Results....................................................366.5.4 Qualitative Synthesis ...........................................................................376.5.5 Synthesis of qualitative and quantitative studies .................................386.5.6 Sensitivity analysis...............................................................................386.5.7 Publication bias....................................................................................396.5.8 Lessons Learned about Data Synthesis................................................39

    7. Reporting the review (Dissemination).................................................................397.1 Specifying the Dissemination Strategy........................................................397.2 Formatting the Main Systematic Review Report.........................................407.3 Evaluating Systematic Review Reports .......................................................407.4 Lessons Learned about Reporting Systematic Literature Reviews..............40

    8 Systematic Mapping Studies................................................................................449 Final remarks .......................................................................................................4410 References........................................................................................................45Appendix 1 Steps in a systematic review ................................................................48Appendix 2 Software Engineering Systematic Literature Reviews ........................50Appendix 3 Protocol for a Tertiary study of Systematic Literature Reviews andEvidence-based Guidelines in IT and Software Engineering ......................................53

    ii

  • 8/6/2019 Systematic Reviews 5 8

    4/65

    0.2 Document Version Control

    Document

    status

    Version

    Number

    Date Changes from previous version

    Draft 0.1 1 April 2004 NonePublished 1.0 29 June 2004 Correction of typos

    Additional discussion ofproblems of assessing evidenceSection 7 Final Remarksadded.

    Revision 1.1 17 August2005

    Corrections of typos.

    Major

    Revision

    1.9 25 October

    2005

    Changed title, added SMC as a

    reviser, several new sectionsadded, finalisation of majorrevisions should be version 2.0Changes summarised below:Added Section 2 to but EBSEin ContextExpanded the reporting of thereview processesAdded sections on SystematicMapping and Tertiary reviewsin section 4

    Updated the Reporting theReview SectionAdded two final sections,Systematic Mapping Studiesand Tertiary Reviews

    Further majorrevisions

    2.0 17 March 2007 Revised the section on hierarchyof studies to be consistent withsocial science viewpoints.Removed some general

    discussion that was not well-focused on the construction ofthe guidelines.Revised the section on qualitychecklistsAdded lessons learnt from SEarticles.Removed final section onTertiary reviews (seemedunnecessary)

    Minor

    revisionsafter internal

    2.1 27 March 2007 Correction of Typos

    Inclusion of a GlossaryInclusion of guidelines

    iii

  • 8/6/2019 Systematic Reviews 5 8

    5/65

    review construction processMinor restructuring Mappingreviews and tertiary reviewsmoved into section 3 to avoidinterfering with the flow of the

    guidelines.Further minorrevisions

    2.2 4 April 2007 Typos and grammaticalcorrections.A paragraph on how to read theguidelines included in theIntroduction.

    Revisionsafter externalreview

    2.3 20 July Amendments after externalreview including theintroduction of more examples.

    iv

  • 8/6/2019 Systematic Reviews 5 8

    6/65

    0.3 Document development team

    This document was revised by members of the Evidence-Based SoftwareEngineering(EBSE) Project (EP/CS51839/X) which was funded by the UK

    Economics and Physical Sciences Research Council.

    Name Affiliation RoleBarbara Kitchenham Keele University, UK Lead author

    Stuart Charters Lincoln University, NZ Second authorDavid Budgen University of Durham,

    UKEBSE InternalReviewer

    Pearl Brereton Keele University, UK EBSE InternalReviewer

    Mark Turner Keele University, UK EBSE InternalReviewer

    Steve Linkman Keele University, UK EBSE InternalReviewer

    Magne Jrgensen Simula ResearchLaboratory, Norway

    External reviewer

    Emilia Mendes University ofAuckland, NewZealand

    External reviewer

    Giuseppe Visaggio University of Bari,Italy

    External reviewer

    v

  • 8/6/2019 Systematic Reviews 5 8

    7/65

    0.4 Executive Summary

    The objective of this report is to propose comprehensive guidelines for systematicliterature reviews appropriate for software engineering researchers, including PhD

    students. A systematic literature review is a means of evaluating and interpreting allavailable research relevant to a particular research question, topic area, orphenomenon of interest. Systematic reviews aim to present a fair evaluation of aresearch topic by using a trustworthy, rigorous, and auditable methodology.

    The guidelines presented in this report were derived from three existing guidelinesused by medical researchers, two books produced by researchers with social sciencebackgrounds and discussions with researchers from other disciplines who are involvedin evidence-based practice. The guidelines have been adapted to reflect the specificproblems of software engineering research.

    The guidelines cover three phases of a systematic literature review: planning thereview, conducting the review and reporting the review. They provide a relativelyhigh level description. They do not consider the impact of the research questions onthe review procedures, nor do they specify in detail the mechanisms needed toperform meta-analysis.

    0.5 Glossary

    Meta-analysis. A form of secondary study where research synthesis is based onquantitative statistical methods.

    Primary study. (In the context of evidence) An empirical study investigating aspecific research question.

    Secondary study. A study that reviews all the primary studies relating to a specificresearch question with the aim of integrating/synthesising evidence related to aspecific research question.

    Sensitivity analysis. An analysis procedure aimed at assessing whether the results of asystematic literature review or a meta-analysis are unduly influenced by a smallnumber of studies. Sensitivity analysis methods involve assessing the impact of highleverage studies (e.g. large studies or studies with atypical results), and ensuring that

    overall results of a systematic literature remain the same if low quality studies (orhigh quality) studies are omitted from the analysis, or analysed separately.

    Systematic literature review (also referred to as a systematic review). A form ofsecondary study that uses a well-defined methodology to identify, analyse andinterpret all available evidence related to a specific research question in a way that isunbiased and (to a degree) repeatable.

    Systematic review protocol. A plan that describes the conduct of a proposedsystematic literature review.

    vi

  • 8/6/2019 Systematic Reviews 5 8

    8/65

    Systematic mapping study (also referred to as a scoping study). A broad review ofprimary studies in a specific topic area that aims to identify what evidence is availableon the topic.

    Tertiary study (also called a tertiary review). A review of secondary studies related to

    the same research question.

    vii

  • 8/6/2019 Systematic Reviews 5 8

    9/65

    1. Introduction

    This document presents general guidelines for undertaking systematic reviews. Thegoal of this document is to introduce the methodology for performing rigorous

    reviews of current empirical evidence to the software engineering community. It isaimed primarily at software engineering researchers including PhD students. It doesnot cover details of meta-analysis (a statistical procedure for synthesising quantitativeresults from different studies), nor does it discuss the implications that different typesof systematic review questions have on research procedures.

    The original impetus for employing systematic literature review practice was tosupport evidence-based medicine, and many guidelines reflect this viewpoint. Thisdocument attempts to construct g...

Recommended

View more >