isa 201 intermediate information systems acquisition€¦ · software quality assurance 3 overall:...

ISA 201Intermediate

Information Systems Acquisition

Lesson 15Software Quality

Learning Objectives

3Software Quality Assurance

Overall: Given a DoD IT/software acquisition scenario with a Program Manager's (PM) quality statement, evaluate program plans to meet PM expectations.

• Identify characteristics unique to software that impact quality.• Define software quality.• Identify characteristics of generic DOD software system domains (e.g.

Platform IT, Command and Control, and Defense Business Systems), that might influence how each system is reviewed in a software quality program.

• Recognize that every IT acquisition program requires a Program Manager approved software quality statement.

• Given several process-focused and product-focused software quality assurance methods, describe how each assures quality in a software acquisition.

• Recognize the benefits of applying Capability Maturity Model Integrated (CMMI) concepts and principles to a DoD SW development project.

• Given a software acquisition scenario, recognize the preferred method for identifying and tracking defects.

• Given an acquisition scenario with multiple software related programmatic issues, analyze how each may impact the program’s ability to meet its quality objectives .

Today we will learn to:

Lesson Overview


• How is Software Different from Hardware?• What is Software Quality?• Software Domain Considerations• Software Quality Assurance Planning and Methods• Lesson Exercise

Lesson Plan


What Makes Software Different from Hardware

Lesson Overview


• How is Software Different from Hardware?

• What is Software Quality?• Software Domain Considerations• Software Quality Assurance Planning and Methods• Lesson Exercise

Lesson Plan Status


The Program Manager must understand and

define software quality for his or her program

THE ESSENCE: Does it do what it is

supposed to?• Definitions for Software Quality

- The degree to which a system, component, or process meets specified requirements [IEEE Std 610.12-1990]

- The degree to which a system, component, or process meets customer or user needs or expectations (no undesirable properties) [ISO/IEC 9001}

• Other Definitions- Lack of bugs- Adherence to a software quality model

(“ilities”)

Software Quality

Quality from an ISO 25010 Point of View

8Software Quality and Testing

ISO 25010 – Software Engineering – Product Quality, defines TWO MODELS:1. A software product quality model composed of eight characteristics, which are further

subdivided into sub characteristics that can be measured internally or externally- Typically internal quality is obtained by reviews of specs, checking models, or by static analysis of

source code- External quality refers to properties of SW interacting with its environment

2. A system quality in use model composed of five characteristics, which are further subdivided into sub characteristics that can be measured when a product is used in a realistic context of use; i.e., quality perceived by an end user who executes a SW product in a specific context

Measurement of Quality Characteristics


Functional suitability; Reliability; Operability; Performance efficiency; Security; Compatibility; Maintainability; Portability;

Quality Characteristics (ISO 25010)

10

SW Product Quality

FunctionalSuitability

Time Behavior

Resource Utilization

Capacity

Performance Efficiency

CompatibilityUsability PortabilityMaintainability Security Reliability

Modularity

Reusability

Analysability

Modifiability

Testability

Appropriateness Recognizability

Learnability

Operability

User error protection

User interface aesthetics

Accessibility

Confidentiality

Integrity

Non -repudiation

Accountability

Authenticity

Functional Completeness

Functional Correctness

Functional Appropriateness

Co-existence

Interoperability

Maturity

Availability

Fault Tolerance

Recoverability

Adaptability

Installability

Replaceability

ISO/IEC 25010:2011

Each quality characteristic has multiple sub-characteristics. For example, Maintainability has 5 sub-characteristics. Each quality sub-characteristic (e.g. Modularity) is further divided into attributes. An attribute is an entity which can be verified or measured in the software product. Attributes are not defined in the ISO standard, as they vary between different software products


Reliability - The degree to which a system or component performs specified functions under specified conditions for a specified period of time. • Maturity • Fault Tolerance • Recoverability • Availability

Functional suitability - The degree to which the product provides functions that meet stated and implied needs when the product is used under specified conditions • Functional completeness• Functional correctness• Functional appropriateness

Software Quality Model Characteristics

ISO 25010 :2011 Software Engineering -- Product quality provides a SW quality model that identifies eight main quality characteristics:

A quality model is a defined set of characteristics, and of relationships between them, which provides a framework for specifying quality requirements and evaluating quality” ISO 25000

1 2


Performance efficiency - The performance relative to the amount of resources used under stated conditions

- Time Behavior - Resource Utilization - Capacity

Usability - The degree to which the product or system can be used by specified users to achieve specified goals with effectiveness, efficiency & satisfaction

- Appropriateness recognizability- Learnability - Operability- User error protection- User interface aesthetics- Accessibility


ISO 25010 :2011 Software engineering -- Product quality provides a SW quality model that identifies eight main quality characteristics:

3 4


Compatibility - The degree to which two or more systems or components can exchange information and/or perform their required functions while sharing the same hardware or software environment

- Co-existence - Interoperability

Security - The degree of protection of information and data so that unauthorized persons or systems cannot read or modify them and authorized persons or systems are not denied access to them

- Confidentiality - Integrity - Non-repudiation - Accountability - Authenticity - Compliance



5 6


Portability - The degree to which a system or component can be effectively and efficiently transferred from one hardware, software or other operational or usage environment to another

- Replacability- Adaptability - Installability

Maintainability - The degree of effectiveness and efficiency with which the product can be modified

- Modularity - Reusability - Analyzability - Modifiability- Testability



87

Lesson Overview


• How is Software Different from Hardware?• What is Software Quality?

• Software Domain Considerations• Software Quality Assurance Planning and Methods• Lesson Exercise

Lesson Plan Status


• Is there a relationship between software quality and software safety?

• Is there a relationship between software quality and cybersecurity?

Software Domain Considerations

SW quality assurance practices should be chosen to meet the program’s quality objectives and informed by the risks inherent in the type of system being built.

MissionSystems

C4ISR

DefenseBusinessSystems

Platform IT (PIT) systems –Safety, Response time

C4ISR systems –Security, interoperability

DBS systems – Privacy, interoperability

Lesson Overview


• How is Software Different from Hardware?• What is Software Quality?• How do I achieve Software Quality?

• Software Quality Assurance Planning and Methods

• Lesson Exercise

Lesson Plan Status


• systematic activities providing evidence that the software product performs as specified- Desk Checking- Walk-Throughs- Formal Inspections- Joint Reviews- Computer-Based Testing- Product Quality Measures

Product Assurance• systematic activities

providing evidence of the ability of the software process to produce a software product fit for use- Process maturity and

compliance - Risk Management- Independent oversight- SQA audits and reporting

Process Assurance

Software Quality Assurance Methods


The PM is ultimately responsible for setting the quality objectives!

• A quality program includes a management process that is capable of ensuring the following key activities:- Formulating a quality management plan- Applying software engineering principles- Conducting formal technical reviews- Applying a multi-tiered testing strategy- Enforcing Process adherence- Controlling change and measuring the impact of change- Performing SQA audits, keeping records and reporting

• The PM should allow contractors to define and use a preferred quality management process that meets required program support capabilities.

• The DOD does not require third party certification or registration of a supplier’s quality system

Quality Program


Software Quality Assurance Plan


• Software Engineering- The application of a systematic, disciplined, quantifiable approach to the

development, operation, and maintenance of software; that is, the application of engineering to software. [IEEE Std 610.12-1990]

• Software Cleanroom Engineering- Combines formal methods of requirements and design with statistical usage

testing to produce software with nearly none or no defects.- Normally used in systems requiring highly reliable software (space shuttle)

• Continuous Process Improvement- A strategic approach for developing a culture of continuous improvement in the

areas of reliability, process cycle times, costs in terms of less total resource consumption, quality, and productivity.

- Six Sigma: A disciplined approach and methodology for reducing variations in system output.

- Lean Six Sigma: A disciplined, data-driven approach and methodology for eliminating defects in any process — from manufacturing to transactional and from product to service.

- Theory of Constraints: a management paradigm that views any manageable system as being limited in achieving more of its goals by a very small number of constraints.

Process Assurance


*CMMI is a spin-off of original SEI activityCMMI Institute http://cmmiinstitute.com/

• Capability Maturity Model® Integration (CMMI) - Was a collaborative effort sponsored by the Office of the Secretary of

Defense/Acquisition and Technology (OSD/A&T) Systems Engineering with participation by government, industry, and the Software Engineering Institute (SEI)*.

- Objective is to develop a product suite that provides industry and government with a set of products to support process and product improvement.

- It can be used to guide process improvement across a project, a division, or an entire organization.

• Benefits of implementing process improvement:- The quality of a system is highly influenced by the quality of the process

used to acquire, develop, and maintain it. - Process improvement increases product and service quality as

organizations apply it to achieve their business objectives. - Process improvement objectives are aligned with business objectives.- CMMI maturity level can often be a good predictor of whether a software

development project will incur cost and schedule overruns.

Process Assurance:Capability Maturity Model—Integration (CMMI)

http://cmmiinstitute.com/


CONTINUOUS

Path enables organization to incrementally improve processes corresponding to an individual process area (PA) (or group of process areas) selected by the organization.

STAGED

Path enables organization to improve a set of related processes by incrementally addressing successive sets of process areas.



• Capability levels enable your organization to focus its process improvement efforts process area by process area from capability level 0 to capability level 3.

• Maturity levels provide a staging of processes for improvement across your organization from maturity level 1 to maturity level 5.

Level Continuous Representation

Capability Levels

Staged Representation

Maturity Levels

Level 0 Incomplete

Level 1 Performed Initial

Level 2 Managed Managed

Level 3 Defined Defined

Level 4 Quantitatively Managed

Level 5 Optimizing



ISO 9000 - Quality management://www.iso.org/iso/iso_9000

ISO 9001 gives the requirements that an organization must do to manage the processes affecting quality of its products and services

• ISO 9000, Quality Management and Quality Assurance Standards - Provides guidelines as to which document to use and how to use it. - Use of ISO 9001, 9002 or 9003 depends on business structure

• ISO 9001, Quality Systems - Model for Quality Assurance in Design/Development, Production

Installation, and Servicing• ISO 9002, Quality Systems

- Model for Quality Assurance in Production and Installation• ISO 9003, Quality Systems

- Model for Quality Assurance in Final Inspection and Test• ISO 9004, Quality Management and Quality Systems Element

- Along with ISO 9000, ISO 9004 is an advisory document.

Process Assurance:More Quality Process Models and Standards

http://www.iso.org/iso/iso_9000


• SPICE (SW Process Improvement and Capability Determination) (ISO/IEC 15504)- An international standard for software process assessment.- Derived from process lifecycle standard ISO 12207 and ideas of maturity models

like Bootstrap, Trillium and the CMM.• Control Objectives for Information and related Technology (COBIT)

- Provides good practices across a domain and process framework. - Practices designed to help optimize IT-enabled investments, ensure service

.delivery and provide a measure against which to judge when things do go wrong..• Information Technology Infrastructure Library (ITIL)

- Provides international best practices for IT service management.• Consists of a series of books giving guidance on the provision of quality IT.

services, and on the accommodation and environmental facilities needed to support IT.

• Practical Software and Systems Measurement (PSM)- Best practices within the software/system acquisition and engineering

communities.- Goal is to provide Project Managers with the information needed to meet cost,

schedule, and technical objectives on programs.

Other Quality Standards and Initiatives


Product Assurance:Human–based Testing


Various internal DoD and GAO reports and studies have indicated that

establishing meaningful measures and using them to manage contributes to

program success

• Low Defect Potentials (< 1 per function point)• High Defect Removal (> 95%)• Unambiguous, Stable Requirements (< 2.5% change)• Explicit Requirements Achieved (> 97.5% achieved)• High User Satisfaction Ratings (> 90% “Excellent”)

- Installation- Ease of Learning- Ease of Use- Functionality- Compatibility

Product Assurance:Measures


IEEE Standard for a Software QualityMetrics Methodology

IEEE Std 1061-1998 (reaffirmed 2009)

4. Analyze Resultsa) Interpret the resultsb) Identify software qualityc) Make software quality

predictionsd) Ensure compliance with

requirements5. Validate Metrics

a) Apply the validation methodology

b) Apply the validity criteriac) Apply the validation

proceduresd) Document results

1. Establish Requirementsa) Identify list of possible quality

requirementsb) Determine list of quality

requirementsc) Assign a direct metric to each

quality requirement2. Identify Metrics

a) Apply the software quality metrics framework

b) Perform a cost-benefit analysisc) Gain commitment to the metrics

3. Implement Metricsa) Define the data collection

proceduresb) Prototype the measurement

processc) Collect the data and compute the

metric values

Product Assurance:Measures

Lesson Overview


• How is Software Different from Hardware?• What is Software Quality?• How do I achieve Software Quality?• Lesson Exercise Part 1• Software Quality Assurance Planning and Methods

• Lesson Exercise

Lesson Plan Status

Quality Exercise Instructions

It is approximately 6 months after the JTAMS Increment 1 CDR. Your team in the JTAMS program office has received a set of slides from Juggernaut depicting the current status of the overall JTAMS development.The PM believes that something is amiss and has asked your IPT to perform an in-depth analysis to assess the program’s status from a quality perspective. The PM had previously identified the program’s specific Quality Attributes and is concerned that the data does not allow for a good quality assessment. The PM wants your IPT to complete the enclosed assessment templates.

Tasks: Within your IPT, review carefully the set of charts to identify problems, assess their impact, project the outcomes, and evaluate alternatives.

1. Does the data provided meet information needs based on the PM’s Quality Attributes (see slide 2)? If not, identify additional data required to support the PM’s Quality Attributes. Include Indicators and Measures recommendations (Use the ICM Table). Each Team will select the attribute that corresponds with their Team number. (i.e. Team 1 = Attribute 1 (Interoperability), etc )

2. Does the data provided meet the information needs based on your teams assigned ISO 25010 Quality Model Characteristic – (see slide 6)? If not, identify additional data required to support your assigned ISO 25010 Quality Model Characteristic and indicate how leveraging these characteristics can lead to improved quality for JTAMS - Include Indicators and Measures recommendations (Use the ICM Table).

3. Identify recommendations to the PM based on the insight gained from your analysis of Tasks 1 and 2

Go to the Exercise Folder

Summary


Today we learned to:Overall: Given a DoD IT/software acquisition scenario with a Program Manager's (PM) quality statement, evaluate program plans to meet PM expectations.

• Identify characteristics unique to software that impact quality.• Define software quality.• Identify characteristics of generic DOD software system domains (e.g.

Platform IT, Command and Control, and Defense Business Systems), that might influence how each system is reviewed in a software quality program.

• Recognize that every IT acquisition program requires a Program Manager approved software quality statement.

• Given several process-focused and product-focused software quality assurance methods, describe how each assures quality in a software acquisition.

• Recognize the benefits of applying Capability Maturity Model Integrated (CMMI) concepts and principles to a DoD SW development project.

• Given a software acquisition scenario, recognize the preferred method for identifying and tracking defects.

• Given an acquisition scenario with multiple software related programmatic issues, analyze how each may impact the program’s ability to meet its quality objectives .

BACKUP SLIDES NOT USED

33Software Quality and Testing

• Integrated analysis combines multiple indicators and focuses on the cause and effect relationships inherent between IT performance parameters

• These IT performance parameters map directly to the measurement information categories

(Back-up) Integrated Analysis(From Measures Lesson)

McGarry, J., Card, D., Jones, C., Layman, B., Clark, E., Dean, J., & Hall, F. (2002). Practical Software Measurement.


35

SW Product Quality


Time Behavior


Capacity



Modularity

Reusability

Analysability

Modifiability

Testability


Learnability

Operability



Accessibility

Confidentiality

Integrity

Non -repudiation

Accountability

Authenticity




Co-existence

Interoperability

Maturity

Availability

Fault Tolerance

Recoverability

Adaptability

Installability

Replaceability

ISO/IEC 25010:2011


36

SW Product Quality


Time Behavior


Capacity



Modularity

Reusability

Analysability

Modifiability

Testability


Learnability

Operability



Accessibility

Confidentiality

Integrity

Non -repudiation

Accountability

Authenticity




Co-existence

Interoperability

Maturity

Availability

Fault Tolerance

Recoverability

Adaptability

Installability

Replaceability

ISO/IEC 25010:2011

isa 201 intermediate information systems acquisition€¦ · software quality assurance 3 overall:...

Documents