Defect Detection

Download Defect Detection

Post on 25-Oct-2014




2 download

Embed Size (px)


<p>DEFECT PREDICTION MODEL FOR TESTING PHASE</p> <p>MUHAMMAD DHIAUDDIN BIN MOHAMED SUFFIAN</p> <p>UNIVERSITI TEKNOLOGI MALAYSIA</p> <p>ii</p> <p>DEFECT PREDICTION MODEL FOR TESTING PHASE</p> <p>MUHAMMAD DHIAUDDIN BIN MOHAMED SUFFIAN</p> <p>A project report submitted in fulfillment of the requirements for the award of the degree of Masters of Science (Computer Science Real Time Software Engineering)</p> <p>Faculty of Computer Science and Information System Universiti Teknologi Malaysia</p> <p>MAY 2009</p> <p>iv</p> <p>ALHAMDULILLAH. To my beloved parents, my wife, brothers and sisters who have given me courage and strength</p> <p>v</p> <p>ACKNOWLEDGEMENT</p> <p>In completing the project, there are many individuals who have contributed to the success of this research. First and foremost, special thanks to my academic supervisor, Prof. Dr. Shamsul Sahibuddin who has guided me throughout this research work. Appreciation also goes to my industrial supervisor who is also the Senior Manager of Test Centre of Excellence department, Mr. Mohamed Redzuan Abdullah for his support and constructive comment in completing this project. I am very grateful to my parents and parents in-law who always put trust and faith in me to continue working for this research. Special gratitude goes to my wife who continually gives her dedicated encouragement to me throughout the tough period. Not forgotten, thank you to the members of Test COE department for their cooperation and valuable inputs in ensuring the success of this project. Not forgotten, special thanks to my Six Sigma coach for his constant cooperation and technical guidance. Last but not least, great gratitude expressed to the colleagues of Part Time 9 for Real Time Software Engineering programme. My thanks also go to staffs of Centre for Advanced Software Engineering (CASE) who have involved directly or indirectly in the project.</p> <p>vi</p> <p>ABSTRACT</p> <p>The need for predicting defects in testing phase is important nowadays as part of the improvement initiatives for software production process. Being the group that ensuring successful implementation of verification and validation process area, all test engineers in Test Centre of Excellence (Test COE) department are required to play their part to discover software defects as many as possible and contain them within testing phase. This research is aimed to achieve zero-known post release defects of the software delivered to end-user. To achieve the target, the research effort focuses on establishing a defect prediction model for testing phase using Six Sigma methodology. It identifies the customer needs on the requirement for the prediction model as well as how the model can benefits them. It also outlines the possible factors that associated to defect discovery in testing phase. Analysis of the repeatability and capability of test engineers in finding defects are elaborated. This research also describes the process of identifying type of data to be collected and techniques of obtaining them. Relationship of customer needs with the technical requirements is then explained clearly. Finally, the proposed defect prediction model for testing phase is demonstrated via regression analysis. This is achieved by considering faults found in phases prior to testing phase and also the code size of the software. The achievement of the whole research effort is described at the end of this project together with challenges faced and recommendation for next research work.</p> <p>vii</p> <p>ABSTRAK</p> <p>Keperluan terhadap meramalkan kecacatan dalam fasa pengujian adalah penting pada masa kini sebagai sebahagian daripada inisiatif pembaikan untuk proses penghasilan perisian. Menjadi kumpulan yang memastikan kejayaan perlaksanaan bidang proses verifikasi dan validasi, semua jurutera pengujian di jabatan Pusat Kecemerlangan Pengujian adalah diperlukan dalam memainkan peranan mereka untuk menjumpai kecacatan perisian sebanyak yang mungkin dan membendung kecacatan tersebut dalam lingkungan fasa pengujian. Penyelidikan ini menyasarkan untuk mencapai kecacatan sifar diketahui bagi pasca pelepasan untuk perisian yang diserahkan kepada pengguna akhir. Untuk mencapai sasaran tersebut, usaha penyelidikan bertumpu kepada mewujudkan model ramalan kecacatan untuk fasa pengujian dengan menggunakan kaedah Six Sigma. Ia mengenal pasti keperluan pengguna ke atas keperluan model ramalan dan juga bagaimana model tersebut member manfaat kepada mereka. Ia juga menggariskan faktor-faktor yang berpotensi dikaitkan dengan penemuan kecacatan dalam fasa pengujian. Analisa mengenai kebolehulangan dan kemampuan para jurutera pengujian dalam menjumpai kecacatan turut dihuraikan. Penyelidikan ini juga menerangkan proses mengenal pasti jenis data yang perlu dikumpul and teknik untuk memperolehnya. Kaitan keperluan pengguna dengan keperluan teknikal kemudiannya diterangkan dengan jelas. Akhirnya, cadangan model ramalan kecacatan untuk fasa pengujian ditunjukkan melalui analisa regresi. Ini dicapai dengan menimbang kesilapankesilapan yang dijumpai dalam fasa-fasa sebelum fasa pengujian dan juga saiz kod untuk perisian tersebut. Kejayaan untuk keseluruhan usaha penyelidikan dijelaskan di akhir tesis bersama-sama dengan cabaran yang dihadapi dan cadangan untuk kerja penyelidikan seterusnya.</p> <p>viii</p> <p>TABLE OF CONTENTS</p> <p>CHAPTER DECLARATION DEDICATION</p> <p>TITLE</p> <p>PAGE iii iv v vi vii viii x xi xii</p> <p>ACKNOWLEDGEMENT ABSTRACT ABSTRAK TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES LIST OF ABBREVIATIONS</p> <p>1</p> <p>INTRODUCTION 1.1 Introduction 1.2 Introduction to Defect Prediction Model for Software Testing 1.3 Background of Company 1.4 Background of Problem 1.5 Statement of Problem 1.6 Objectives of Study 1.7 Importance of Study 1.8 Scope of Work 1.9 Project Schedule</p> <p>1 1</p> <p>1 2 3 5 6 7 7 7</p> <p>ix 1.10Project Outline 8</p> <p>2</p> <p>LITERATURE REVIEW ON DEFECT PREDICTION MODEL FOR TESTING PHASE 2.1 Introduction 2.2 Defect Prediction across Software Development Life Cycle (SDLC) 2.3 Reviews on the Defect Prediction across SDLC and Testing Phase 2.4 Applications and Issues of Defect Prediction 2.5 Summary of the Proposed Solution</p> <p>10 10</p> <p>10</p> <p>19 20 30</p> <p>3</p> <p>METHODOLOGY 3.1 Introduction 3.2 Six Sigma - DMADV Methodology 3.3 Supporting Tools</p> <p>31 31 31 36</p> <p>4</p> <p>PROJECT DISCUSSION 4.1 Introduction 4.2 Findings of Define Phase 4.3 Findings of Measure Phase 4.4 Findings of Analyze Phase</p> <p>37 37 37 44 50</p> <p>5</p> <p>CONCLUSION 5.1 Achievements 5.2 Constraints and Challenges 5.3 Recommendation</p> <p>53 53 55 56</p> <p>REFERENCES</p> <p>58</p> <p>x</p> <p>LIST OF TABLES</p> <p>TABLE NO.</p> <p>TITLE</p> <p>PAGE</p> <p>1.1 2.1 2.2</p> <p>Project schedule Short-term defect inflow prediction example Strength and weakness of defect prediction techniques</p> <p>8 17</p> <p>27 32 33</p> <p>3.1 3.2</p> <p>Project team Customer identification</p> <p>xi</p> <p>LIST OF FIGURES</p> <p>FIGURE NO.</p> <p>TITLE</p> <p>PAGE</p> <p>2.1 2.2 2.3 2.4</p> <p>Defects detection techniques Defects per life cycle phase Defects based on testing metrics Relationship between CMM levels and delivered defects</p> <p>12 14 15</p> <p>15 16</p> <p>2.5 2.6</p> <p>Short-term defect inflow prediction example Normalized results from the application of CDM Model to test process</p> <p>19 22</p> <p>2.7 2.8</p> <p>Process Performance Model Graphical representation of Rayleigh model parameters</p> <p>24 25 25 26 32 38 39 40 41 42</p> <p>2.9 2.10 2.11 3.1 4.1 4.2 4.3 4.4 4.5</p> <p>Prediction without process metrics Prediction with process metrics High level schematic of whole phase BN DMADV phases MIMOS software production process Schematic diagram Detail schematic Y to X tree diagram Team charter Customer need statement</p> <p>xii 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13 1st level of KJ analysis 2nd level of KJ analysis Kano analysis House of quality for defect prediction model Test case experiment result Assessment agreement Assessment agreement for within appraiser Assessment agreement for each appraiser against standard 4.14 Assessment agreement for all appraisers against standard 4.15 4.16 4.17 4.18 Operational definition Data collection plan Data for regression Regression result 48 49 50 51 51 48 43 43 44 45 46 47 47</p> <p>xiii</p> <p>LIST OF ABBREVIATIONS</p> <p>BN CMM CMMI COE</p> <p>- Bayesian Network - Capability Maturity Model - Capability Maturity Model Integration - Centre of Excellence</p> <p>COQUALMO - Constructive Quality Model CUT DfSS DMADV FMEA FP IPF ISP JARING KJ KLOC LOC MEMS MIMOS MOF MSA NEMS PC PDF - Code and Unit Testing - Design for Six Sigma - Design, Measure, Analyze, Design, Verify - Failure Mode and Effect Analysis - Function Point - In-Process Fault - Internet Service Provider - Joint Advanced Research Integrated Networking - Kawakita Jiro - Kilo Lines of Code - Lines of Code - Micro-Electro-Mechanical Systems - Malaysian Institute for Microelectronic Systems - Ministry of Finance - Measurement System Analysis - Nano-Electro-Mechanical Systems - Personal Computer - Probability Density Function</p> <p>xiv QFD R&amp;D SDLC SEI TER UAT V&amp;V - Quality Function Deployment - Research and Development - Software Development Life Cycle - Software Engineering Institute - Test Effectiveness Ratio - User Acceptance Test - Verification and Validation</p> <p>1</p> <p>CHAPTER 1</p> <p>INTRODUCTION</p> <p>1.1</p> <p>Introduction</p> <p>This chapter describes the introduction of the research effort presented throughout this project. It illustrates the overview of the research that encourages the establishment of defect prediction model for software testing phase. The discussion continues with background of the research, problem statements, research objectives and the importance of the research. The scope of work and project outline is then explained in the last sections of this chapter.</p> <p>1.2</p> <p>Introduction to Defect Prediction Model for Software Testing</p> <p>As an organization that aims to become a premier applied research centre in frontier technologies, MIMOS has always committed to develop, produce and release high quality software to the market. One of the key aspects to ensure it can be achieved is by having effective and efficient software development process throughout entire SDLC. Thus, prediction or estimation of defects for particular software during testing phase is very crucial to enhance the testing process as part of process improvement in SDLC.</p> <p>2 Being the last gate before acknowledging that the particular software is ready to go to the market requires strong and accurate data and metrics. The initiative on having defect prediction model for testing phase helps in determining defects that are likely to occur during test execution and contributes in providing relevant software quality metrics. Defect prediction model for testing contributes to zero-known post release defects of a software product. This is determined by defect containment in testing phase. Predicting total number of defects at the start of testing allows for wider test coverage to be put in place. As more defects contained within testing phase, it helps in improving quality of software product being delivered to end user. By using testing metrics for predicting total defects, it demonstrates the stability of development effort of releasing a software product.</p> <p>1.3</p> <p>Background of Company MIMOS or Malaysian Institute for Microelectronics System was established on 1st</p> <p>January 1985 as a unit of Prime Ministers department following the initiative by group of academicians led by Tengku Dr. Mohd Azzman Shariffadeen. The initial objective is to conduct microelectronics research to support the industries as well as to develop indigenous products. After going through corporatization exercise as a company under Ministry of Finance (MOF), MIMOS has been focusing on three (3) core functions: Research and Development (R&amp;D), National IT Policy Development and Business Development. Since then, MIMOS has embarking on various initiatives and projects including manufacture affordable personal computer (PC), commission industrial-class water fabrications plant, launch first Malaysias first Internet Service Provider (ISP) called JARING, initiate Computer Forensic Services and also launch AgriBazaar. On 1 July 2006, Dato AbdulWahab Abdullah was appointed as new President and Chief Executive Officer of MIMOS replacing Tengku Dr. Mohd Azzman Sharifadden. The appointment of Dato Abdul Wahab has turned MIMOS from the R&amp;D organization in ICT and microelectronics to world class R&amp;D Centre of Excellence.</p> <p>3 With the tagline Innovation for Life, MIMOS is now the premier applied research centre in frontier technologies aimed at growing globally competitive indigenous industries. Through smart partnerships with local and international universities, research institutes, industries and Malaysia Government, MIMOS focuses on frontier technologies by pursuing exploratory and industry-driven applied research. To date, research and technology areas in MIMOS are refined into eight (8) technology clusters: Advanced Informatics, Communication Technology, Cyberspace Security, Encryption Systems, Grid Computing, Knowledge Technology, Micro Energy, and Micro Systems (MEMS/NEMS) cluster.</p> <p>1.4</p> <p>Background of the Problem</p> <p>As defect becomes the main intention of software testing, test engineers of Test Centre of Excellence (Test COE) are expected to find and discover any errors, bugs and faults in the software through various kinds of testing techniques or strategies. Estimation of defects to be found upfront at the start of testing phase is very important to strategize on executing the test for the software. As this research problem concentrates on formulating a defect prediction model for testing, several issues contribute to this research problem.</p> <p>1.4.1</p> <p>Issue on Better Resource Planning for Test Execution Across Projects</p> <p>Currently, number of test engineers allocated to a particular testing project is based on the size of the project including how detail the requirements are and complexity of the software being developed. At the same time, one test engineer can work in more than one testing project. Thus, there is a need to estimate the number of defect to be found in testing phase. Appropriate number of test engineers can be planned and</p> <p>4 allocated across multiple projects. Estimated number of defects is required to support the resource planning activities within the testing department to ensure the resources are optimized and productivity of every test engineer is high.</p> <p>1.4.2</p> <p>Issue on Wider Test Coverage to Find Defect</p> <p>Defects are found while test execution is still in progress. Test engineer applies various testing techniques to find and discover as many defects as possible based on the baseline requirements. There is no specific pre-determined factor that a test engineer can use as a basis to find defects. As a result, test engineer will continue to find defects until end of the execution schedule. To ensure wider and better test coverage, prediction of defects could improve the way test engineers find defects as they are now having a target of defects to be found. This can be in a fo...</p>


View more >