NORTHROP GRUMMAN PROPRIETARY LEVEL I

Value of Software

Quality Assurance

Part 2

13 March 2018

Matt AndersonDirector, Mission Assurance

American Society for Quality

Collaboration on Quality in the Space

and Defense Industries Conference

Copyright ©2018 Northrop Grumman Systems Corporation

“What do you mean by software quality?”

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/182

Quality is…• “(1) The degree to which a system, component, or process meets specified

requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations.” [IEEE Std 610.12-1990]

• “Degree to which a set of inherent characteristics fulfils requirements” [ISO 9000:2005(E) & CMMI-DEV V1.3]

• “Fitness for use” [Jaran], meaning

• Quality consists of those product features that meet the needs of the customers and thereby provide product satisfaction

• Quality consists of freedom from deficiencies

• Various agile quality papers postulate additional variants and attributes with core elements:• Satisfying customer

requirements/specifications• Results-oriented agile testing

mindset

• The degree to which a product satisfies customer-specified requirements and project identified, derived, and decomposed requirements

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/183

Software Quality Approaches

• Defect Management Approach

• Quality Attributes Approach, e.g. fixed

quality models such as ISO/IEC 25010:2011

Quality in Use Characteristics

• Effectiveness• Efficiency• Satisfaction• Freedom from risk• Context coverage

Product Quality Model

• Functional suitability

• Performance effectiveness

• Compatibility• Usability• Reliability• Security • Maintainability• Portability

ISO/IEC 25010:2011

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/184

“Six months from start of acquisition to

fielded capability… we need more of this”

“Get a minimum viable product deployed

and build on it”

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/185

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/186

“The internet is magic”

“malloc()? free()? Ha, ha, ha! You’re old school!”

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/187

Agile

Image by Dr ian Mitchell, posted at: https://commons.wikimedia.org/wiki/File:Scrum_Framework.png

Continuous integrationDevOps

ScrumKanban

SAFeExtreme programming

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/188

• Configuration management

• Defect management

• Code analysis – Cyclomatic complexity

– Technical debt calculation

– Memory leak detection

– Security

– Size calculation

– Etc.

• Documentation/documentation analysis

• Peer review management

• Continuous integration

• Automated testing

• Load testing

• Test management

• Log analysis

• Performance monitoring

• Data analysis & reporting, dashboards

• Project tracking/management

• Etc.Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/189

Assuring software

supplier quality

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/1810

Coming Soon

• AI/machine learning

• Cognitive autonomy

• Program synthesis

• Formal methods

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/1811

“Quality is free”

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/1812

Approved for Public Release #18-0475. Distribution Unlimited. Date Approved: 03/12/18