The adoption of machine learning techniques for software defect prediction: An initial industrial validation

Download The adoption of machine learning techniques for software defect prediction: An initial industrial validation

Post on 15-Aug-2015

97 views

Category:

Software

1 download

TRANSCRIPT

  1. 1. Rakesh Rana1, Miroslaw Staron1, Jrgen Hansson1, Martin Nilsson2, Wilhelm Meding3 1Computer Science & Engineering, Chalmers | University of Gothenburg, Sweden 2Volvo Car Group, Gothenburg, Sweden 3Ericsson, Gothenburg, Sweden rakesh.rana@gu.se The adoption of machine learning techniques for software defect prediction: An initial industrial validation
  2. 2. Software Defect Prediction (SDP) methods Image 1: https://www.reliablesoft.net/how-to-become-an-expert-in-your-niche-even-if-you-are-not/ Image 2: Fenton, Norman, et al. "Predicting software defects in varying development lifecycles using Bayesian nets." Information and Software Technology 49.1 (2007): 32-43. Image 3: Kan, Stephen H. Metrics and models in software quality engineering. Addison-Wesley Longman Publishing Co., Inc., 2002. Image 4: http://www.codeodor.com/index.cfm/2009/11/12/Its-Not-Your-Fault-Your-Software-Sucks/3058
  3. 3. SDP: Methods based on Machine Learning Decision Trees (DTs) Support Vector Machines (SVMs) Artificial Neural Networks (ANNs) Bayesian Belief Networks (BNNs) Image 1: http://www.webpages.uidaho.edu/veg_measure/Modules/Lessons/Module%202(Sampling)/2_3_Accuracy_and_bias.htm Image 2: http://www.business2community.com/marketing/3-easy-keyword-research-tips-for-inbound-marketing-success-0215660#!bKz0G0 Image 3: http://dpss.co.riverside.ca.us/childrens-services-division/adoption-information/foster-adoptive-parent Image 4: http://www.haitian-truth.org/treatys-tighter-adoption-rules-kick-in-for-haiti/
  4. 4. Objective What are the factors that are important for companies to make informed decision to adopt (or not adopt) ML algorithms for the purpose of software defect predictions (SDP)?
  5. 5. Research Process
  6. 6. Framework for Adoption of ML Techniques in Industry
  7. 7. ML characteristics that affects its acceptance for SDP
  8. 8. Organizational characteristics that affects its acceptance for defect prediction task
  9. 9. Study Design Unit of analysis (Domain) Software development process Current methods for SDP Current state of adoption of ML for SDP VCG (Automotive) V-shaped software development Focus on status visualization and analogy based prediction Considering evaluation Ericsson (Telecom) Lean and Agile development Various modes of presenting current status and predictions methods Considering evaluation
  10. 10. Study Design The interviewees: VCG, QM VCG, MetricsTL Ericsson, QM Ericsson, MetricsTL Level Need and importance (Table 2) Level of Satisfaction (Table 3) Level of importance (Table 4) Very Low (VL) The information is not needed. Not satisfactory, improvement is needed. The attribute is not needed for analysis. Low (L) The information is desired, but not considered important. Not satisfactory, improvement is desired. The attribute can be considered but not required. Medium (M) The information is desired and is considered of value (if available). Satisfactory, but could be improved. The attribute is useful for making the analysis. High (H) The information is deemed as needed and is considered important. Satisfaction is high. The information on given attribute is needed for making the analysis. Very High (VH) The information is a must and should be provided with high accuracy. Satisfaction is very high, with low scope for further improvement. Cannot make a decision without information about this attribute.
  11. 11. Results: Information need and its importance for SDP Prediction Needs w.r.t software defects VCG (QM) VCG MetricsTL Ericsson (QM) Ericsson MetricsTL Classification of defect prone files/modules L H VH VH Expected number of defects in SW components H H L VH Expected defect inflow for a project/release H H L VH Release readiness/expected latent defects H VH H VH Severity classification of defects VH M H H VCG most OEMs (Original Equipment Manufacturers) in automotive domain, Model Based Development (MBD) Assessing release readiness is important (High) for both case units
  12. 12. Results: Satisfaction with existing systems Factors: Satisfaction with existing systems VCG (QM) VCG MetricsTL Ericsson (QM) Ericsson MetricsTL Status information H H H H Trend visualization H M M H Predictions accuracy M M L H Cost (current costs are low) VH VH - VH Reliability VH H VH M Cost of obtaining results is very important factor and the current systems we use are very cheap to run and maintain QM at VCG.
  13. 13. Results: Familiarity and competence with ML techniques Factors: Familiarity and competence with ML techniques VCG (QM) VCG MetricsTL Ericsson (QM) Ericsson MetricsTL ML tried in previous project L L - M Understanding of the technology L L - M Ability to implement algorithms in-house VL M - M Academic collaboration M H - M Ability to interpret the results H H - M Ability to assess quality of results H M - M Participating companies in the study show medium to high confidence with their ability to interpret the results from such analysis.
  14. 14. Results: Perceived Benefits Factors: Perceived Benefits VCG (QM) VCG MetricsTL Ericsson (QM) Ericsson MetricsTL Accuracy in predicting H H VH VH Automation of pattern discovery M H VH VH Adaptability to different data sets M H VH VH Ability to handle large data H H M VH Ability to generate new insights H M H H When it comes to the benefits, accuracy and automation are the top priorities for us MetricsTL at Ericsson
  15. 15. Results: Tool availability & External factors Factors: Tool availability & External factors VCG (QM) VCG MetricsTL Ericsson (QM) Ericsson MetricsTL Compatibility with existing systems M L H VH Availability of open source tools L H M VH Low cost of obtaining results VH H H M Support/consulting services H M L VL Adoption by other industries L L L M Use by competitors H M L M Even if open source tools are available, we typically need a vendor in between to do tool integration, manage upgrades and do maintenance work we do not have resources for that QM at VCG. We are not afraid of trying new things and being the first one, but if it is used in automotive sector and we have not tried it surely helps the case QM at VCG.
  16. 16. Specific challenges in adopting ML techniques in industry for SDP Lack of information to make a strong business case Uncertainty on applicability of ML when access to source code is not available How to adapt ML techniques for model driven development How to effectively use text base artefacts for SDP Uncertainty over where ML fits in context of compliance to standards
  17. 17. ML Adoption for SDP: Conclusions ML based techniques have high potential to aid companies in SDP efforts We identified a total of nine important factors and twenty seven related attributes ML adoption framework help increase our understanding of factors and attributes relevant for industrial practitioners ML adoption framework will be useful for Companies, Researchers, and Tool vendors What are the factors that are important for companies to make informed decision to adopt (or not adopt) ML algorithms for the purpose of software defect predictions (SDP)?

Recommended

View more >