secure systems research group - fau development process metrics a look at development process...
TRANSCRIPT
Secure Systems Research Group - FAU
Development Process Metrics
A look at development process metrics with view to managing the development of secure software.
Dr. Michael VanHilst
22 March 2007
Secure Systems Research Group - FAU
Development Process Metrics
A look at development process metrics with view to managing the development
of secure software.
Dr. Michael VanHilst
22 March 2007
Secure Systems Research Group - FAU
ISO-15504 (level 2), CMM (level 4)
Measure how well the process is being followed (using GQM).1. Identify goals (i.e. reliable software products)2. Analyze causal chains in the process3. Identify process steps that contribute to the goals4. Define “measurements” that assess if identified steps are followed.5. Measurement can use indicators (presence of documents) or survey
questions (“How committed was management in the review meeting?”)6. Measure and hold assessment meetings every 2 months
W. S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, January 1995.
Jarvinen, J., Hamann, D., and van Solingen, R., “On integrating assessment and measurement: towards continuous assessment of software engineering processes”, IEEE Software Metrics Symposium, 1999.
Birk, A., Derks, P., Hamann, D., Hirvensalo, J., Oivo, M., Rodenbach, E., van Solingen, R., and Taramaa, J., “Applications of measurement in product-focused process improvement: a comparative industrial case study”, IEEE Software Metrics Symposium, 1998.
Secure Systems Research Group - FAU
Personal Software Process
W. S. Humphrey. A Discipline for Software Engineering. Addison-Wesley, January 1995.
Anne M. Disney and Philip M. Johnson, “Investigating Data Quality Problems in the PSP (Experience Paper)”, ACM SIGSOFT 1998
Philip M. Johnson, Hongbing Kou, Joy Agustin, Christopher Chan, Carleton Moore, Jitender Miglani, Shenyan Zhen, and William E.J. Doane, “Beyond the Personal Software Process: Metrics collection and analysis for the differently disciplined.”, IEEE ICSE 2003
• Requires developers to record lots of notes on how they spend time.
• Developed by Watts Humphrey at SEI, promoted as part of Team Software Process within CMMI.
• Experience shows dedication slacks off over time.
• Studies by Johnson, et al., find self reports inaccurate, easily skewed.
Secure Systems Research Group - FAU
Causal Analysis (CMM level 5)
Dumke, R.R.; Blazey, M.; Hegewald, H.; Reitz, D.; Richter, K.: Causalities in Software Process Measurement and Improvement. International Conference on Software Process and Product Measurement (MENSURA 2006), November, 6-8, 2006
Several other papers on process discovery, inference, or compliance.
Secure Systems Research Group - FAU
Low Level Non-Invasive Metrics
Johnson, P.M., Kou, H., Paulding, M., Zhang, Q., Kagawa, A., and Yamashita, T, Improving software development management through software project telemetry, IEEE Software, July 2005
Andrea Janes, Marco Scotto, Alberto Silliti, and Giancarlo Succi, “A Perspectrive on Non Invasive Software Management”, IEEE Instrumentation and Measurement Technology Conference (IMTC), 2006
Instrument IDE, CM, Office Tools, OS
Reports which app/file in foreground
Used to look for indicators of lowered productivity.
Johnson example looked at contributing factors in build failures.
Secure Systems Research Group - FAU
Siemens Measurement System
• Four measurement categories: Time / Size / Quality / Cost. – Size metric undecided (IFPUG 4.2, COSMIC FFP V2.2)
• Four selected indicators are – Schedule / Budget compliance– Cost of Defect Correction– Feature changes– Defect Detection Profile
• An annual “Measurement Excellence Day” helps to keep participants in the SMS loyal to the program.
Secure Systems Research Group - FAU
Bias in the Data
• Measures conformance to plans/predictions– Time spent– Code written (LOC,FP)– Defects found / fixed
• Analysis always blames the developer– Developer productivity (slow, error prone)– Process conformance (followed process?)
• Won’t drive process change or innovation
Secure Systems Research Group - FAU
Shell Information Technology
• Five metrics are collected: (Tom Dekker)– Project Delivery Rate, – Speed of Delivery, – Defect Density, – Time Schedule, – Effort Cost.
• The target is: 90% of the projects should be within schedule, budget, and within planned scope.
• Revised estimation based on industry benchmarks proved much better than the original values based on Shell’s own history-based data. They were less biased than their own.
• (Less biased, closer to business goals)
Secure Systems Research Group - FAU
Typical Effort vs. Calendar Day
alpha release beta releasestart
develop test … fix
Secure Systems Research Group - FAU
Waste & Delay in Typical Effort
alpha release beta releaserequirements freeze
develop … test … fix
production waste
fault caused
fault foundfault fixedlag
Secure Systems Research Group - FAU
Toyota Product Development System
releasestart
test & develop
Design defects out of the process
A product with many defects at first release will still have many defects at final release
Secure Systems Research Group - FAU
IS Security Design Methods
Richard Baskerville, Information systems security design methods: implications for information systems development, ACM Computing Surveys, 25, 4 (December 1993) Pages: 375 – 414
Chung, L., Nixon, B. Dealing with non-functional requirements: three experimental studies of a process-oriented approach. Proceedings of ICSE’95, pp. 25–37
Eloff, MM, von Solms, SH, ‘Information Security Management : An Approach to combine Process Certification, and product Certification’, Computers and Security, Vol 19, pp 698 – 709
T. Tryfonas, E. Kiountouzis, A. Poulymenakou, Embedding security practices in contemporary information systems development approaches, Information Management & Computer Security, Oct 2001 9,4 183-197
• Much literature on requirements for security
• Very little on the development process
Secure Systems Research Group - FAU
A Toyota Security Method?
• A product with many security flaws at first release will still have many flaws at final release
• We get what we measure. If we don’t measure security we won’t get security. So how do we measure security?
• Is there an idiot-proof or test-first equivalent for driving security flaws out in the development process?
• Suggestion from Nelly: use automated code analysis for security anti-patterns, unsafe library calls, …
Mylopolous has paper on process for transforming insecure code to secure cope.
Secure Systems Research Group - FAU