UNIT 2

SOFTWARE QUALITY ASSURANCE

Quality tasks – SQA plan – Teams – Characteristics – Implementation – Documentation

– Reviews and Audits

Software Quality Assurance (SQA) consists of a means of monitoring the software engineering processes and

methods used to ensure quality. It does this by means of audits of the quality management system under which

the software system is created. These audits are backed by one or more standards, usually ISO 9000.

� It is distinct from software quality control which includes reviewing requirements documents, and

software testing. SQA encompasses the entire software development process, which includes processes

such as software design, coding, source code control, code reviews, change management, configuration

management, and release management. Whereas software quality control is a control of products,

software quality assurance is a control of processes.

� Software quality assurance is related to the practice of quality assurance in product manufacturing. There

are, however, some notable differences between software and a manufactured product. These differences

stem from the fact that the manufactured product is physical and can be seen whereas the software

product is not visible. Therefore its function, benefit and costs are not as easily measured. What's more,

when a manufactured product rolls off the assembly line, it is essentially a complete, finished product,

whereas software is never finished. Software lives, grows, evolves, and metamorphoses, unlike its

tangible counterparts. Therefore, the processes and methods to manage, monitor, and measure its

ongoing quality are as fluid and sometimes elusive as are the defects that they are meant to keep in

check.

Quality tasks

Tasks

� Quality Engineering the activity consisting of the cohesive collection of all tasks that are primarily

performed to ensure and help continually improve the quality of an endeavor’s process and work

products

Goals

The typical goals of quality engineering are to:

� Ensure that the necessary levels of quality are achieved.

� Make the achievement of quality predictable and repeatable.

� Minimize endeavor, organizational, and personal risks due to poor quality.

Objectives

The typical objectives of quality engineering are to:

� Define what quality means on the endeavor in terms of a quality model defining quality factors and

quality sub factors.

� Plan the quality tasks including helping the requirements team determine and specify the quality

requirements and associated quality factors (attributes) and quality metrics.

� Assure the quality of the process used by the endeavor.

Thus, quality assurance is concerned with fulfilling the quality requirements and achieving the quality

factors of endeavor’s process.

� Control the quality of the work products delivered during the endeavor.

Thus, quality control is concerned with fulfilling the quality requirements and achieving the quality

factors of the endeavor’s work products.

“Are we building the right products?”

Examples of quality engineering based on scope include:

• Application Quality Engineering

• Business Quality Engineering

• Contact Center Quality Engineering

• Data Center Quality Engineering

Preconditions

Quality engineering typically may begin when the following preconditions hold:

� The endeavor is started.

The quality team is initially staffed and trained in quality engineering.

Completion Criteria

Quality engineering is typically complete when the following post conditions hold:

� The endeavor is complete.

Tasks

The following diagram illustrates the relationships between the quality tasks:

SQA plan

� The purpose of this Software Quality Assurance Plan (SQAP) is to define the techniques, procedures,

and methodologies that will be used at the Center for Space Research (CSR) to assure timely delivery of

the software that meets specified requirements within project resources.

� The use of this plan will help assure the following:

(1) That software development, evaluation and acceptance standards are developed, documented and

followed.

(2) That the results of software quality review and audits will be given to appropriate management within

CSR. This provides feedback as to how well the development effort is conforming to various CSR

development standards.

(3) That test results adhere to acceptance standards.

Teams

The SQA team shall check that the quality is maintained during the project and that the proper quality

procedures are being followed discovered problems are reported to the Project Management. The members of the

project team must work according to the part(s) of the SQAP that applies to their specific task.

The tasks of the SQA team

UR phase (User Requirements)

For the first phase of the project (UR), the SQA team must see to it that the following

Documents are properly reviewed internally before they are submitted for an external review.

URD (User Requirement document)

The SQA team checks before the internal review whether the URD:

� Contains a general description of the software that has to be developed.

� Contains requirements on the software to be developed as stated by the customer.

� Contains constraints on the software to be developed.

� Contains a priority list of the requirements

SPMP (Software Project Management Plan)

The SQA team must check whether the goals of the project are clearly described. A life cycle approach

for the project must be defined. The SQA team must ensure that the SPMP is realistic by checking:

� The assumptions made during the planning of the project (by comparing the actual time spent with the

reserved time in the planning).

� Restrictions with respect to planning (e.g. availability of members).

� External problems (e.g. room availability).

SCMP (Software configuration Management Plan)

With respect to the SCMP, the SQA team checks before the internal review whether the document provides

procedures concerning:

� CI identification.

� CI storage.

� CI change control.

� CI status indication.

All documents must have a unique identifier and backups must be made at least twice every week.

SQAP (Software Quality Assurance Plan)

With respect to the SQAP, the SQA team checks before the internal review whether the SQAP contains:

� Project standards.

� Problem reporting procedures.

� Responsibilities of the project members with respect to quality assurance.

SVVP (Software Verification and Validation Plan)

With respect to the SVVP, the SQA team checks before the internal review whether the SVVP contains:

� Reviewing and audits.

� Testing.

� Tracing.

During internal reviews the SQA team checks these documents and in case of problems, the author(s) and the

team leader are informed. After the corrective action has been taken, the SQA team reviews the document again.

SR Phase (Software Requirement Phase)

For the second phase of the project (SR), the SQA team must see to it that the following documents are

properly reviewed internally before they are submitted for an external review. During internal reviews (see the

SVVP the SQA team checks these documents and in case of problems, the author(s) and the team leader are

informed. After the corrective action has been taken, the SQA team reviews the document again.

SRD (Software Requirement Document)

The SQA team must check before internal reviews whether the SRD:

� Contains requirements on the software to be developed, these requirements must be based on the

software requirements stated in the URD.

� Contains constraints on the software to be developed, these constraints must be based on the software

constraints stated in the URD.

� Contains a logical model.

� Contains a priority list of the requirements.

� Contains a traceability table

SPMP/SR

The SQA team checks before internal reviews whether the SPMP is realistic what concerns:

� The assumptions made during the planning.

� Restrictions with respect to the planning (e.g. availability of members).

� External problems (e.g. external software/code).

SQAP/SR

With respect to the SQAP, the SQA team checks before internal reviews whether the SQAP contains:

� The tasks of the SQA team during the SR phase.

SVVP/SR

With respect to the SVVP, the SQA team checks before internal reviews whether the SVVP contains:

� The Acceptance Test Plan (can be a document on its own).

� The System Test Plan (can be a document on its own).

AD Phase (Architectural Design)

For the third phase of the project (AD), the SQA team must see to it that the following documents are

properly reviewed internally before they are submitted for an external review.

The ADD(Architectural Design Document)

The SQA team checks before internal reviews whether the ADD:

� contains an architectural design of the software to be developed, this design must describe a physical

model and the interfaces between the different classes contains pre and post conditions for the methods

defined in the physical model

� contains an architectural design of the software to be developed, this design must describe a physical

model and the interfaces between the different classes

� contains a traceability table

SPMP/AD

The SQA team checks before internal reviews whether the SPMP is realistic what concerns:

� the assumptions made during the planning

� restrictions with respect to the planning (e.g. availability of members)

� external problems (e.g. external software/code)

SQAP/AD

With respect to the SQAP, the SQA team checks before internal reviews whether the SQAP contains:

� coding and commentary standards

SVVP/AD

With respect to the SVVP, the SQA team checks before internal reviews whether the SVVP contains:

� The Integration Test (IT) plan

SCMP/AD

With respect to the SCMP, the SQA team checks before internal reviews whether the SCMP contains:

� A description of the tools used in support of version control, code creation, compilation and debugging

During internal reviews SVVP the SQA team checks these documents and in case of problems, the author(s) and

the team leader is informed. After the corrective action has been taken, the SQA team reviews the document

again.

DD Phase (Detailed Design Phase)

For the fourth phase of the project (DD), the SQA team must see to it that the following documents are

properly reviewed internally before they are submitted for an external review.

DDD (Detailed Design Document)

The SQA team checks before internal reviews whether the ADD:

� Contains the detailed design of the software to be developed, this design must describe the components

and their interfaces to other components.

� Contains a detailed design of the software to be developed.

� contains a traceability table (see the SVVP)

ATP (Acceptable Test Plan)

The SQA team checks before internal reviews whether the ATP:

� contains all user requirements described in URD.

STP (Software Test Plan)

The SQA team checks before internal reviews whether the STP:

� Contains sufficient tests to test the system.

ITP (Integration Test Plan)

The SQA team checks before internal reviews whether the ITP:

� Contains sufficient tests to test the integration of all components.

UTP (Unit Test Plan)

The SQA team checks before internal reviews whether the UTP:

� Contains sufficient tests to test all units of the system.

The Coding and Commentary standards are described in the SCMP.

TR Phase (Transfer Phase)

For the fifth phase of the project (TR), the SQA team must see to it that the following documents are

properly reviewed internally before they are submitted for an external review.

STD (Software Transfer Document)

The SQA team checks before internal reviews whether the STD:

� Contains a list of all deliverables to be transferred.

� Contains a procedure to build the software.

� Contains a procedure to install the software.

SUM (Software User Manual)

The SQA team checks before internal reviews whether the SUM:

� Contains a tutorial on how to use the software.

� Contains references to all options possible in the software.

In SR section: overview of quality assurance activities in AD, DD and TR phases.

Characteristics:

� Understanding of business approach and goals of the organization

� Understanding of entire software development process

� Strong desire for quality

� Establish and enforce SQA methodologies, processes and Testing Strategies

� Judgment skills to assess high-risk areas of application

� Communication with Analysis and Development team

� Report defects with full evidence

� Take preventive actions

� Take actions for Continuous improvement

� Reports to higher management

� Say No when Quality is insufficient

� Work Management

� Meet deadlines

The ISO 9126-1 software quality model identifies 6 main quality characteristics, namely:

� Functionality

� Reliability

� Usability

� Efficiency

� Maintainability

� Portability

These characteristics are broken down into sub characteristics; a high level table is shown below. It is at the

sub characteristic level that measurement for SPI will occur. The main characteristics of the ISO9126-1 The main

characteristics of the ISO9126-1 quality model, can be defined as follows:-

Functionality

Functionality is the essential purpose of any product or service. For certain items this is relatively easy to define,

for example a ship's anchor has the function of holding a ship at a given location. The more functions a product

has, e.g. an ATM machine, then the more complicated it becomes to define it's functionality.

For software a list of functions can be specified, i.e. a sales order processing systems should be able to record

customer information so that it can be used to reference a sales order. A sales order system should also provide

the following functions:

� Record sales order product, price and quantity.

� Calculate total price.

� Calculate appropriate sales tax.

� Calculate date available to ship, based on inventory.

� Generate purchase orders when stock falls below a given threshold

The list goes on and on but the main point to note is that functionality is expressed as a totality of essential

functions that the software product provides. It is also important to note that the presence or absence of these

functions in a software product can be verified as either existing or not, in that it is a Boolean (either a yes or

no answer). The other software characteristics listed (i.e. usability) are only present to some degree, i.e. not a

simple on or off. Many people get confused between overall process functionality (in which software plays a

part) and software functionality. This is partly due to the fact that Data Flow Diagrams (DFDs) and other

modeling tools can depict process functionality (as a set of data in\data out conversions) and software

functionality. Consider a sales order process, that has both manual and software components. A function of

the sales order process could be to record the sales order but we could implement a hard copy filing cabinet

for the actual orders and only use software for calculating the price, tax and ship date. In this way the

functionality of the software is limited to those calculation functions. SPI, or Software Process Improvement

is different from overall Process Improvement or Process Re-engineering, ISO 9126-1 and other software

quality models do not help measure overall Process costs\benefits but only the software component. The

relationship between software functionality within an overall business process is outside the scope of ISO

9126 and it is only the software functionality, or essential purpose of the software component, that is of

interest for ISO 9126.

Reliability

Once a software system is functioning, as specified, and delivered the reliability characteristic defines the

capability of the system to maintain its service provision under defined conditions for defined periods of

time. One aspect of this characteristic is fault tolerance that is the ability of a system to withstand component

failure. For example if the network goes down for 20 seconds then comes back the system should be able to

recover and continue functioning.

Usability

Usability only exists with regard to functionality and refers to the ease of use for a given function. For

example a function of an ATM machine is to dispense cash as requested. Placing common amounts on the

screen for selection, i.e. $20.00, $40.00, $100.00 etc, does not impact the function of the ATM but addresses

the Usability of the function. The ability to learn how to use a system (learnability) is also a major sub

characteristic of usability.

Efficiency

This characteristic is concerned with the system resources used when providing the required functionality.

The amount of disk space, memory, network etc. provides a good indication of this characteristic. As with a

number of these characteristics, there are overlaps. For example the usability of a system is influenced by the

system's Performance, in that if a system takes 3 hours to respond the system would not be easy to use

although the essential issue is a performance or efficiency characteristic.

Maintainability

The ability to identify and fix a fault within a software component is what the maintainability characteristic

addresses. In other software quality models this characteristic is referenced as supportability. Maintainability

is impacted by code readability or complexity as well as modularization. Anything that helps with identifying

the cause of a fault and then fixing the fault is the concern of maintainability. Also the ability to verify (or

test) a system, i.e. testability, is one of the sub characteristics of maintainability.

Portability

This characteristic refers to how well the software can adopt to changes in its environment or with its

requirements. The sub characteristics of this characteristic include adaptability. Object oriented design and

implementation practices can contribute to the extent to which this characteristic is present in a given system.

Characteristics Sub

characteristics Definitions

Suitability This is the essential Functionality characteristic and refers to the

appropriateness (to specification) of the functions of the software.

Accurateness This refers to the correctness of the functions, an ATM may provide a cash

dispensing function but is the amount correct?

Functionality Interoperability

A given software component or system does not typically function in isolation.

This sub characteristic concerns the ability of a software component to interact

with other components or systems.

Compliance

Where appropriate certain industry (or government) laws and guidelines need to

be complied with, i.e. SOX. This sub characteristic addresses the compliant

capability of software.

Security This sub characteristic relates to unauthorized access to the software functions.

Maturity This sub characteristic concerns frequency of failure of the software.

Reliability Fault tolerance The ability of software to withstand (and recover) from component, or

environmental, failure.

Recoverability Ability to bring back a failed system to full operation, including data and

network connections.

Understandability Determines the ease of which the systems functions can be understood, relates

to user mental models in Human Computer Interaction methods.

Usability Learn ability Learning effort for different users, i.e. novice, expert, casual etc.

Operability Ability of the software to be easily operated by a given user in a given

environment.

Efficiency Time behavior Characterizes response times for a given thru put, i.e. transaction rate.

Resource behavior Characterizes resources used, i.e. memory, cpu, disk and network usage.

Analyzability Characterizes the ability to identify the root cause of a failure within the

software.

Maintainability Changeability Characterizes the amount of effort to change a system.

Stability Characterizes the sensitivity to change of a given system that is the negative

impact that may be caused by system changes.

Testability Characterizes the effort needed to verify (test) a system change.

Adaptability Characterizes the ability of the system to change to new specifications or

operating environments.

Portability Installability Characterizes the effort required to install the software.

Conformance

Similar to compliance for functionality, but this characteristic relates to

portability. One example would be Open SQL conformance which relates to

portability of database used.

Replaceability

Characterizes the plug and play aspect of software components, that is how

easy is it to exchange a given software component within a specified

environment.

Implementation

Quality assurance will be implemented through all the software life cycles of the tool’s development process,

until the release of the software product. The following are the quality assurance tasks for each phase of the

software development:

Requirements phase: When the SRS is being developed, the developer has to ensure that it elucidates the

proposed functionality of the product and to keep refining the SRS until the requirements are clearly stated and

understood.

Specification and Design phase: Due to the great importance for accuracy and completeness in these

documents, weekly reviews shall be conducted between the developer and the professor to identify any defects

and rectify them.

Implementation phase: The developer shall do code reviews when the construction phase of the Tool begins.

Software testing phase: The developer shall test each case. The final product shall be verified with the

functionality of the software as specified in the Software Requirements Specification (SRS) for the Tool.

Through all these phases of the software development, the following shall also be conducted to improve the

software quality:

� Develop and generate SQAP: Generate a finalized SQAP plan

� Communication and Feedback: The developer is encouraged to freely express disagreements,

suggestions and opinions about all aspects of the weekly process of software development.

� Internal audits and evaluations: The Major Professor and the committee are expected to do auditions

and evaluations at the end of each phase in the project.

Documentation

The documentation that describes and supports the [[Project Name]] software or the software

development process shall be created and updated periodically throughout the development cycle.

Table 4-1 is a list of [[Project Name]] software deliverable products and the associated standard or guidelines

used to develop and maintain the software products. Any tailoring guidelines are also found in Table 4-1. Table

4-2 is a list of non-deliverable products.

For the project’s software documents to be developed and not yet listed in Tables 4-1 and 4-2, SQA will

assist in identifying the specifications, standards, and Data Item Descriptions (DIDs) to be followed in the

preparation of the required documentation.

List the software products (or reference the document, e.g. CM Plan, that lists the products) that will be

developed/maintained and identify the associated Data Item Description (DID) or standard or guidelines that

are used to develop/ maintain the software product to which this SQA Plan applies in Table 4-1. If there are any

tailoring guidelines, provide that information in Table 4-1. Identify all non-deliverable products in Table 4-2.

TABLE 4-1. DELIVERABLE SOFTWARE PRODUCTS

NOMENCLATURE DELIVERABLE

DOCUMENTATION

DID, STANDARD, GUIDELINE

[[CI Name]] [[DOCUMENT TYPE,

e.g., SSS]]

[[DID, e.g., DI-IPSC-81431 of MIL-

STD-498]]

[[CI Name]] [[DOCUMENT TYPE]] [[DID, STANDARD, GUIDELINE]]

[[CI Name]] [[DOCUMENT TYPE]] [[DID, STANDARD, GUIDELINE]]

TABLE 4-2. NON-DELIVERABLE SOFTWARE PRODUCTS

DOCUMENT TITLE

[[Document title]]

[[Document title]]

[[Document title]]

State how the documents are to be checked for adequacy. The document review process should include

the criteria and the identification of the review or audit by which the adequacy of each document shall be

confirmed.

All documents will undergo a peer review in accordance with the Peer Review Process, reference (n).

Upon completion of a peer review, SQA records and reports Peer Review measurements (the item reviewed, the

number of errors detected, the phase when the Peer Review was conducted, the number of closed error reports,

and the number of open error reports) to SEPO.

Upon completion of a peer review, the software product will be submitted to SCM and placed under CM control.

The software product will then be processed in accordance with the SCM software product approval and release

process as described in reference (e).

Documentation

� The following documentation will be generated and updated throughout the duration

of software life cycles:

� Comparision document

� Architecture / Design Documentation

� Design and Implementation Documentation

� In general, design rules will also be implemented by the source code (e.g., the modularity features and complexity of the design should transfer to the source code). It is important to ensure that the vendor's plans for modularity and complexity are stated in the design documentation and the modularity and complexity of the design are not lost during implementation. This report addresses design and implementation/source code in two separate documents.

Design Documentation

� The following recommendations for a design document are based on [NASA2100], [DOD2167A], [APPGUIDE], P1059, and [ASMENQA2].

SOFTWARE DESIGN DESCRIPTION

� Architectural Design. o The role of the software within the system and the relationships between this software and other

software components. o Assumptions about the environment including operating system, user interface, program, data

management, data interchange, graphics, and network services, especially assumptions on which safety functions and computer security needs may be based.

o Architectural Design Description. The design of the software including: � Logical or functional decomposition. � Description of the modules including mention of the safety and computer security functions. � Design of the modules in terms of execution control and data flow including flagging safety

and security functions. � Design requirements/constraints on the modules. � Relationships and interactions between the modules. � A schema for the allocation of functions to modules. � Logical data design - conceptual schema. � Entity/data identification and relationships. � Timing and sequencing. � Implementation constraints. � Each state and mode the software operates in and the modules that execute in each state and

mode. � Execution control and data flow between modules in the different states and modes. � Memory and processing time allocation to the modules.

o External Interface Design. Allocation of the software's external interface requirements to modules. The design of each interface in terms of: � Information description. � Initiation criteria. � Expected response. � Protocol and conventions. � Error identification, handling and recovery. � Queuing. � Implementation constraints. � Requirements relative to safety and computer security.

o Software requirements allocation to modules. o Database description. Reference should be made to the document describing the database, its

platform, dependencies, etc., or described it here. o Human interaction. Screens, types of alert mechanisms. o Tracking to any other systems in the plant that have safety functions.

� Detailed Design. o Modules Design and Traceability to Architectural Design. The design of the software into its

modules and traceability between the architectural design modules and software modules. The traceability with flags on safety features and computer security requirements. Descriptions of the modules should include: � Inputs and outputs. � Functions. � Data descriptions and relationships. � Diagrams. � Control and signal flow. � Error handling/messages. � Interfaces between modules.

� Packaging details (placement of modules). � Flags on safety, computer security functions.

o Detailed Design of Modules. Design information necessary to code the modules. The information should include: � Detailed design to the lowest level. � Functions or operations. � Algorithms. � Specific data definitions including data conversions. � Local and global data. � Parameters for initiation and adaptation. � Logic flow (control flow; timing variations; priority assignments; interrupt priorities and

handling). � Error detection and handling. � Physical data design (internal schema; query language; access method; key, record, and data

element definition and structure). � Device interface. � Interrupts and signals. � Limitations that restrict performance of modules.

o External Interface Detailed Design. The traceability of the external interface design of the software into its modules. A description of each external interface including: � Type and purpose. � Description of the data transmitted across the interface including purpose, source and

destination, data type, data representation, size, units of measure, limit/range, accuracy, precision/resolution.

� Messages transmitted and the assignment of data elements to each message. � Priority of interfaces and messages transmitted across them.. � Protocol (fragmentation and reassembly of messages; error control and recovery procedures;

synchronization; flow control; data transfer rate and minimum transfer rate; routing, addressing and naming conventions; transmission services; status, identification, notification and other reporting features; security).

� Description of user responses to system/software safety/security violations. o Coding and Implementation Notes. Information such as stubs for incremental development and use

of compiler options. o Information On Planning, Designing, Executing Tests.

� Technical Documentation

� Software Configuration Management (SCM) Documentation

� Software Verification and Validation Document

� For large or long term projects, the SVVP and the detailed test documentation should be separate documents. For small projects, while the initial planning and SVVP should be completed during the requirements phase, additional information regarding the SV&V tasks, test designs, case, and procedures may be added to the SVVP later. There are several types of SV&V documents that should be produced. The following recommendations for an SVVP are based on [IEEE1012].

� SOFTWARE VERIFICATION AND VALIDATION PLAN

� Objectives. The objectives of performing SV&V should be stated. Sometimes thorough SV&V on every software requirement, design feature, and line of code may not be possible and specific objectives must be selected (e.g., to show that critical performance objectives have been met; to examine software interfaces to the system, especially during failures external to the software).

� Management Activities. Review of V&V Plan, review of SV&V results, monitoring of SV&V activities, policies for iteration of tasks, etc.

� Schedule of SV&V Tasks. Includes milestones and completion dates. � Organizations. Organizations involved in the SV&V activities and specific responsibilities. � Communication. Communication procedures between design team members, code team

members, SV&V team members, and others involved in the project. � Summary of Resources. Staffing, tools, facilities, and finances needed to perform SV&V tasks,

etc. � Facilities. Planning of facilities, especially tools for test activities (e.g., simulation, test case

generation) that require acquisition, staff training, and test. Planning and providing for test facilities may take as long as software development itself, so adequate time should be allotted. Delayed facility planning may result in inadequate test strategies and execution.

� Testing. Distinction among the different types of tests, their objectives, and documentation. � Lifecycle SV&V. Detailed descriptions of SV&V tasks throughout lifecycle. Identifies the

following: o SV&V tasks and objectives for each lifecycle phase (e.g., the specific safety objectives

as a focus of each task). o Methods and criteria used in performing the SV&V tasks. o Inputs and outputs required for each task. o Schedule for SV&V tasks. o Resources for performing SV&V tasks. o Assumptions concerning schedule, resources, or approach. o Responsibilities for performing the tasks.

� SV&V Reporting. Description of how results of implementing the SVVP will be documented. � Anomaly Reporting and Resolution. Description of method for reporting and resolving

anomalies, and tracking anomaly resolution. � Policy for Reiterating a SV&V Task. Description of criteria used to determine the extent to

which a SV&V task should be reperformed when input is changed. � Policy for Deviating from the Approved Plan. A good reviewer during final product evaluation

may pick up some differences between planned and actual activities and ask many questions about the deviation. If procedures were in place and were followed, vendor may be able to demonstrate that the deviation did not detract from quality of the software, or its SV&V.

� Configuration Control Procedures. Configuration management procedures on the documentation examined by SV&V tasks; if this information is included only in a SCMP, then reference to the SCMP should appear in the SVVP.

The following recommendations for other SV&V documents are based on [IEEE1012].

� TASK REPORTS. SOFTWARE VERIFICATION AND VALIDATION PHASE SUMMARY REPORTS. Summary of interim or final results of all SV&V tasks performed during a lifecycle phase. Should be issued for each lifecycle phase. If SV&V tasks are performed by different organizations, the phase summary report will help to consolidate the evaluations of the different organizations.

� ANOMALY REPORTS. The location, impact, cause, criticality, and possible recommendations for each anomaly. Should be generated throughout the lifecycle. Procedures for reporting anomalies might include using key words to characterize each anomaly, severity scales to assess its significance, and procedures for reviewing, approving, and dispatching of the reports.

� SOFTWARE VERIFICATION AND VALIDATION FINAL REPORT. A summary of all SV&V tasks, summary of task results, summary of anomalies and resolution, assessment of overall software quality, and recommendations. This summary may be issued at the end of Installation and Checkout Phase or at conclusion of SV&V effort. It can be used as the baseline for maintenance, to help improve the development process for the next project, and to call attention to outstanding unresolved anomalies.

� Marketing Documentation

� User Documentation

USER'S MANUAL

• Overview of Purpose and Functions. The purpose and main capabilities of the software, and its overall operation in terms of functions, options, and restrictions and limitations.

• Installation and Initialization. The procedures for installing, tailoring, and initiating the software including the following:

o Equipment setup. o Power-on and power-off. o Bootstrap and load. o Initiation commands. o Initialization of files, variables, or other data. o Tailoring, reconfiguration, adaptation.

• Restart. Procedures for restarting software after initialization failures, abnormal termination. • Termination. Procedures for terminating operation, and determining whether normal termination has

occurred. • Recovery Steps. Recovery procedures in terms of: check points; collection of failure data; restoring files;

and, restoring devices to operational mode. • Inputs.

o System inputs. Description of system inputs to the software that may occur while the software is in use and may affect the software's interface with the user (e.g., inputs from a remote sensor). Include applicable attributes of the input such as format, frequency, effect on the software state or mode, allowable range, and units of measure.

o User inputs. Description of user inputs including commands, data, and option selection. • Outputs. Description of expected outputs of the software including results of functions, and error

messages. • Potential Hazards. Description of user response to unexpected events or problems that could cause

serious hazards if user does not respond; description of activities that could cause the need for user response or override.

• Functions and their Operations. Description of each function in terms of: purpose of function; step-by-step procedures for execution; and, related functions.

• Error Messages. Error messages output by the software, the meaning of each error message, and the corresponding corrective actions to be taken.

� Design and Implementation Documentation

In general, design rules will also be implemented by the source code (e.g., the modularity features and complexity of the design should transfer to the source code). It is important to ensure that the vendor's plans for modularity and complexity are stated in the design documentation and the modularity and complexity of the design are not lost during implementation. This report addresses design and implementation/source code in two separate documents.

Review and audits

Review:

Management reviews formally evaluate a project plan or project status relative to that plan. Management reviews

have two purposes. The first is to ensure the adequacy and completeness of each planning document (e.g., PMP,

SQAP, SCMP, SVVP, and Test Plans) for meeting project requirements. The second is to ensure that project

activities are progressing according to the planning documents, identify the need for corrective action to the plan

or the project, and ensure proper allocation of resources. All problems are documented. The results of the

management reviews are summarized in management review reports which are auditable and traceable to and

from the appropriate planning documents.

In contrast, the formal technical review examines the product, and the results of any assurance activities already

conducted on the product. The purpose of technical reviews is to evaluate the software elements (e.g., SRS,

software design description (SDD)) to ensure conformity to its specifications, compliance of the development of

the software elements with its plans, and the integrity of changes to the software elements. The results of the

technical reviews are summarized in technical review reports which are auditable and traceable to and from the

appropriate planning documents. Success of a technical review requires that all participants carefully examine the

inputs to the technical review prior to the review meeting. Section B.1.1 presents an outline of the management

review process; section B.1.2 presents an outline of the technical review process. The descriptions of

management and technical reviews are based on [IEEE1028].

In both the management and technical reviews, experts on specific topics (e.g., design experts for design reviews)

should be present to lend their expertise to the review.

Management Reviews

Responsibilities. The review leader performs the administrative functions of the review and issues the

management review report. Review team members are expected to be prepared for the meeting and ensure that

the review objectives are met.

Inputs. Objectives of the review; list of issues to discuss; the specific planning document; current project

schedule and cost data; reports from previously completed reviews; reports on project resources; and, data on

complete or in progress software elements.

Entry Criteria. The need for conducting management reviews is specified in the appropriate project planning

documents. Other management reviews may also be conducted upon request. The management review is

performed when the review leader establishes/confirms the review objectives and determines that all necessary

documents are available.

Procedures. The review leader and project manager plan for the review by identifying the review team,

scheduling a time and place for the review, and distributing all inputs to the review team. An overview of the

project is conducted for the review team by a qualified project member. Each review team member studies the

inputs and prepares presentations for the review team. The management review consists of the review team:

� Assessing adequacy and completeness of the planning documents (initial review).

� Determining whether or not the project status corresponds to the planning document under review, and

recording any deviations.

� Determining whether or not factors not originally considered are constraining the planning document

under review.

� Listing issues and recommendations to be addressed by upper management and/or others who affect the

project.

� Recommending actions to follow the review, and authorization for additional reviews and audits.

� Identifying other issues that need to be addressed.

Exit Criteria. The management review is complete once the review objectives have been addressed and the

management review report has been issued.

Output. The management review report identifies: the project; the review team; review inputs; review

objectives; action items; list of issues and recommendations; and, recommendations for additional reviews and

information necessary to complete them. (The procedures for closing action items should be part of a plan.)

Technical Reviews

Responsibilities. The review leader performs the administrative functions of the review and issues the technical

review report. The recorder documents the findings, decisions, and recommendations of the review team. Review

team members are expected to be prepared for the meeting and ensure that the review objectives are met.

Recommendations made by the review team should be such that management can act on them quickly.

Management is responsible for responding to recommendations promptly.

Inputs. Objectives of the review; software element being reviewed; the software element's specifications; results

of any assurance activities; any other necessary documentation.

Entry Criteria. The need for conducting technical reviews is specified in the project planning documents. Other

technical reviews may also be conducted upon request. The technical review is performed when the review

objectives are established, the individuals responsible at the review for the software element are prepared for the

review, and the review leader determines the software element is sufficiently complete. Preparation may require

considerable time spent in privately examining the inputs.

Procedures. The review leader plans for the review by identifying the review team, schedules a time and place

for the review, and distributes all inputs to the review team. An overview of the project is conducted for the

review team by a qualified project member. Each review team member studies the software element and related

materials. The technical review consists of the review team:

� Determining whether or not the software element corresponds to the specifications and standards to

which it must adhere, and recording any deviations.

� Listing issues, recommendations, and responsible individuals for resolving the issues.

� Identifying other issues that need to be addressed.

� Documenting the meeting, deficiencies found it the software element, and recommendations for

management (the review leader determines whether or not an additional review should be performed on

reworked software elements).

Exit Criteria. The technical review is complete once the review objectives have been addressed and the

technical review report has been issued.

Output. The technical review report identifies: the review team; the software element; review inputs; the

software element's unresolved deficiencies; list of management issues; action items; and, recommendations for

unresolved issues deficiencies.

V&V activities should be performed prior to the formal review of the product of each of the lifecycle phases. The

reviewer during development checks that these activities have been performed, and uses the resulting reports to

answer the questions in the checklists below.

Checklists for Formal Reviews

Formal reviews may include reviewing SQA, SV&V, and SCM results in order to examine the product. This

review may also help detect whether or not these activities were performed in accordance with their respective

plans. The checklists below provide a guideline for reviewing both the product and plans for assurance activities.

These checklists are not necessarily complete; at any time the reviewer during development may need to expand

upon a given topic. In all checklists, negative answers require further examination by the reviewers.

Software Requirements Review

The following checklist contains questions a reviewer during development ask at the software requirements

review based on [IEEE1028], [STARTS], [SOFTENG], [EWICS2], [IEEEP1059], [HHSFDA], [BIRRELL], and

[ANS104].

Compatibility

� Do the interface requirements enable compatibility of external interfaces (hardware and software)?

Completeness

� Does the SRS contain everything listed in the corresponding documentation content? Does it include all

requirements relating to functionality, performance, constraints, safety, etc.?

� Does SRS include all user requirements (as defined in the concept phase)?

� Do the functional requirements cover all abnormal situations?

� Have the temporal aspects of all functions been considered?

� Are the time-critical functions identified and the time criteria for them specified? Do these include the

maximum and minimum times for their execution?

� Does SRS define those requirements for which future changes are anticipated?

� Are all normal environmental variables included?

� Are the environmental conditions specified for all operating modes (e.g., normal, abnormal, disturbed)?

Consistency

� Is there internal consistency between the software requirements?

� Is the SRS free of contradictions?

� Are the specified models, algorithms, and numerical techniques compatible?

� Does SRS use standard terminology and definitions throughout?

� Is SRS compatible with the operational environment of the hardware and software?

� Has the impact of software on the system and environment been specified?

� Has the impact of the environment on the software been specified?

Correctness

� Does the SRS conform to SRS standards?

� Are algorithms and regulations supported by scientific or other appropriate literature?

� What evidence is there that shows vendor has applied the regulations correctly?

� Does the SRS define the required responses to all expected types of errors and failure modes identified

by the hazard analysis?

� Were the functional requirements analyzed to check if all abnormal situations are covered by system

functions?

� Does SRS reference desired development standards?

� Does the SRS identify external interfaces in terms of input and output mathematical variables?

� Are the requirements for the man-machine interface adequate?

� What is the rationale for each requirement? Is it adequate?

� Is there justification for the design/implementation constraints?

Feasibility

� Will the design, operation, and maintenance of software be feasible?

� Are the specified models, numerical techniques, and algorithms appropriate for the problem to be

solved? Are they accepted practice for nuclear power plants? Can they be implemented within the

imposed constraints?

� Is the quality attributes specified achievable individually and as a group?

Modifiability

� Are requirements organized so as to allow for modifications (e.g., with adequate structure and cross

referencing)?

� Is each unique requirement defined more than once? Are there any redundant statements?

� Is there a set of rules for maintaining the SRS for the rest of the software lifecycle?

Robustness

� for fault tolerance and graceful degradation?

Traceability

� Is there traceability from the next higher level spec (e.g., system concept/requirements and user needs as

defined in concept phase and system design)?

� Does the SRS show explicitly the mapping and complete coverage of all relevant requirements and

design constraints defined in the concept phase, by such means as a coverage matrix or cross-reference?

� Is SRS traceable forward through successive development phases (e.g., into the design, code, and test

documentation)?

� Are safety functions or computer security functions flagged?

Understandability

� Does every requirement have only one interpretation?

� Are the functional requirements in modular form with each function explicitly identified?

� Is there a glossary of terms?

� Is formal or semiformal language used?

� Is the language ambiguous?

� Does the SRS contain only necessary implementation details and no unnecessary details? Is it over

specified?

� Are the requirements clear and specific enough to be the basis for detailed design specs and functional

test cases?

� Does the SRS differentiate between program requirements and other information provided?

Verifiability/Testability

� Are the requirements verifiable (i.e., can the software be checked to see whether requirements have been

fulfilled)?

� Are mathematical functions defined using notation with precisely defined syntax and semantics?

� Is there a verification procedure defined for each requirement in the SRS?

Software Design Review

Reviewers should be able to determine whether or not all design features are consistent with the requirements.

Numerical techniques and algorithms should be appropriate for the problem to be solved. The program design

needs to be partitioned in a manner consistent with the problem to be solved. And, the program should meet the

requirements. The following checklist contains questions a reviewer during development may ask at the software

design review based on [SOFTENG], [IEEE1028], [IEEEP1059], and [ANS104].

Completeness

� Are all the items listed in section 4.6.1. addressed in the SDD?

� Are the SRS requirements fulfilled?

� Is there enough data (logic diagrams, algorithms, storage allocation charts, etc.) available to ensure

design integrity?

� Are algorithms and equations adequate, accurate, and complete?

� Are requirements for the support and test software and hardware to be used in the development of the

product included?

� Does the design implement required program behavior with respect to each program interface?

� Are all program inputs, outputs, and database elements identified and described to the extent needed to

code the program?

� Does the SDD describe the operational environment into which the program must fit?

� Are all required processing steps included?

� Are all possible outcomes of each decision point designated?

� Does the design take into account all expected situations and conditions?

� Does the design specify appropriate behavior in the face of unexpected or improper inputs and other

anomalous conditions?

� Does the SDD reference all desired programming standards?

Consistency

� Are standard terminology and definitions used throughout the SDD? Are the style of presentation and the

level of detail consistent throughout the document?

� Does the design configuration ensure integrity of changes?

� Is there compatibility of the interfaces?

� Is the test documentation compatible with the test requirements of the SRS?

� Is the SDD free of internal contradictions?

� Are the models, algorithms, and numerical techniques that are specified mathematically compatible?

� Are inputs and output formats consistent to the extent possible?

� Are the designs for similar or related functions consistent?

� Are the accuracies and units of inputs, database elements, and outputs that are used together in

computations or logical decisions compatible?

Correctness

� Does the SDD conform to design documentation standards?

� Does the design perform only that which is specified in the SRS unless additional functionality is

justified?

� Is the test documentation current and technically accurate?

� Is the design logic sound -- will the program do what is intended?

� Is the design consistent with documented descriptions and know properties of the operational

environment into which the program must fit?

� Do interface designs agree with documented descriptions and known properties of the interfacing

elements?

� Does the design correctly accommodate all inputs, outputs, and database elements whose format, content,

data rate, etc. are not at the discretion of the designer?

Feasibility

� Are the specified models, algorithms, and numerical techniques accepted practices for use within nuclear

power plants?

� Can they be implemented within the constraints imposed on the system and on the development effort?

� Are the functions as designed implementable within the available resources?

Modifiability

� Does the design use information hiding as follows:

o The modules are organized such that changes in the requirements only require changes to a small

number of modules.

o Functions and data structures likely to changes have interfaces insensitive to changes in

individual functions.

o The design partitions data structure access, database access and I/O access from the application

software by the use of access programs (globally accessible data is not used).

o Functionality is partitioned into programs to maximize the internal cohesion of programs and to

minimize program coupling.

� Does each program have a single function?

Modularity

� Is there a schema for modularity, e.g., model-based?

� Is the design structured so that it comprises relatively small, hierarchically related programs or sets of

programs, each performing a particular, unique function?

� Does the design use specific criteria to limit program size?

Predictability

� Does the design contain programs which provide the required response to identified error conditions?

� Does the design schedule computer resources in a manner that is primarily deterministic and predictable

rather than dynamic?

� Does the design contain a minimum number of interrupts and event driven software? Is justification

given for uses of these features?

� Is plausibility checking performed on the execution of programs to uncover errors associated with the

frequency and/or order or program execution and the permissiveness of program execution?

Robustness

� Are all SRS requirements related to fault tolerance and graceful degradation addressed in the design?

Structuredness

� Does the design use a logical hierarchical control structure?

Traceability

� Does the SDD show mapping and complete coverage of all requirements and design constraints in the

SRS?

� Are all functions in the SDD outside the scope of the SRS identified?

� Are all functions identified so they can be uniquely reference by the code?

� Does the SDD contain or reference a revision history which identifies all modifications to the design and

the rationale for these changes?

� Does the SDD reference the design notes which document design decisions relevant to the software

design?

� Have safety and computer security functions been flagged?

Understandability

� Does the SDD avoid unnecessarily complex designs and design representations.

� Is the SDD written to allow unambiguous interpretation?

Verifiability/Testability

� Does the SDD describe each function using well-defined notation so that the SDD can be verified against

the SRS and the code can be verified against the SDD?

� Are conditions, constraints identified quantitatively so that tests may be designed?

Source Code Review

The following checklist contains the kinds of questions a reviewer during development may ask at the source

code review based on [SOFTENG], [ANS104], and [EWICS2].

Completeness

� Is the code a complete and precise implementation of the design as documented in the SDD?

� Was the code integrated and debugged to satisfy the design specified in the SDD?

� Does the code create the required databases, including the appropriate initial data?

� Are there any unreferenced or undefined variables, constants, or data types?

Consistency

� Is the code logically consistent with the SDD?

� Are the same format, invocation convention, and structure used throughout?

Correctness

� Does the code conform to specified standards?

� Are all variables properly specified and used?

� Are all comments accurate?

� Are all programs invoked with the correct number of parameters?

Modifiability

� Does the code refer to constants symbolically to facilitate change?

� Are cross-references or data dictionaries included to show variable and constant access by the program?

� Does code consist of programs with only one entry point and one exit point? (exception is with fatal error

handling)

� Does code reference labels or other symbolic constants rather than addresses?

Predictability

� Is the code written in a language with well-defined syntax and semantics:

� Was the use of self-modifying code avoided?

� Does the code avoid relying on defaults provided by the programming language?

� Is the code free of unintended infinite loops?

� Does the code avoid recursion?

Robustness

� Does the code protect against detectable runtime errors (e.g., range array index values, division by zero,

out of range variable values, and stack overflow)?

Structuredness

� Is each function of the program recognizable as a block of code?

� Do loops only have one entrance?

Traceability

� Does the code identify each program uniquely?

� Is there a cross-reference framework through which the code can be easily and directly traced to the

SDD?

� Does the code contain or reference a revision history of all code modifications and the reason for them?

� Have all safety and computer security functions been flagged?

Understandability

� Do the comment statements adequately describe each routine, using clear English language?

� Were ambiguous or unnecessarily complex coding used? If so, are they clearly commented?

� Were consistent formatting techniques (e.g., indentation, use of white space) used to enhance clarity?

� Was a mnemonic naming convention used? Does the naming reflect the type of variable?

� Is the valid range of each variable defined?

� Does the code use mathematical equations which correspond to the mathematical models

described/derived in the SDD?

Verifiability

� Are implementation practices and techniques that are difficult to test avoided?

Test Readiness Review

The test readiness review is usually conducted following completion of component testing or software

integration testing. The purpose is to ensure readiness to begin formal integration testing or system testing

without complications, and to ensure that test documentation is complete, that errors have been removed, and that

use of the test facilities has been planned properly. The following are general questions that might be asked at

these reviews:

� Have recommended changes been made to the code as result of source code review or, as appropriate,

component test or integration test?

� Is the error rate sufficiently low to warrant beginning the next type of testing?

� Are all test cases/procedures complete?

� Is the test facility ready? Are schedules approved, personnel and physical requirements specified?

� Have all test tools been checked?

� Have all test procedures been checked?

Test Report Review

The purpose of this review is to assure the completeness of test activities. It is usually required at the end of

development (i.e., following system testing). However, it may be conducted after completion of module,

integration, or system testing.

The reviewer should ask questions about whether the test objectives were met (e.g., whether all test cases were

executed, whether test results matched the expected outputs, and whether the results were traceable to the test

cases and software requirements). The reviewer should also inquire about the nature of major anomalies (e.g.,

whether there were similarities, what corrective actions were taken). The reviewer should check that the

anomalies existed only in the product, and were not caused by the test cases and procedures.

Development Documentation Review

[ASMENQA2] states that a development documentation review should be conducted following

completion of the testing phase, to "assure completion and acceptability of the development documentation."

Although [ASMENQA2] does not define development documentation, it is assumed that this consists of all

documentation produced during the development phases of the lifecycle (e.g., requirements, design, coding). The

purpose of the development documentation review seems to be checking for consistency among the development

documents and to ensure that all changes have been made consistently.

Installation and Checkout Review

The following checklist contains the kinds of questions a reviewer during development may ask at the installation

and checkout review based on [ANS104].

Completeness

� Are elements necessary for rebuilding and testing of the installed program available on the installation

medium (e.g., source code, user-supplied library routines, test cases)?

� Has the vendor provided adequate information for installing the program in more than one operating

environment?

Correctness

� Can all test cases be performed?

� Do the test cases produce results identical to the expected outputs?

� Are all results identical to previous results (when the same tests are executed more than once)? If not, are

differences in results clearly understood and justified?

Understandability

� Are the format and content of the medium properly identified for easy reading of the files?

� Are the installation procedures clearly understandable?

Audits:

SQA Program Audits

In order for SQA to evaluate compliance with the CSR/ACIS SDP, SMP, and the STP, SQA will review and

approve the above plans. These plans will specify that the evidence of work generated is adequate to ensure

compliance with project and contract requirements. Audits performed shall include examination of both internal

and external software deliverables.

� Scheduled Audits

SQA will generate and maintain an Audit Schedule. Audits will occur at the end of each development phase as

indicated in the SMP. The results of audits will be discussed with the individual most responsible for the

production of the deliverable.

� Unscheduled Audits

SQA will perform random and unannounced audits to ensure the corrective actions agreed to during the

Scheduled Audits are being followed. The results of audits will be discussed with the individual most responsible

for the production of the deliverable.

� Audits of the SQA Organization

The SQA function will be audited by the PAM prior to any MSFC project review in which software is involved.

The results of the audit will be available to MSFC at the project review. Checklists will be used.

� Audit Reports

Audit reports, and recommended corrective actions generated by SQA will be brought to the attention of the

individual most responsible for producing the software deliverable. Corrective action will be recommended and

reviewed with the individual and SPM.

Part-A (2 Marks)

1. What are Quality tasks?

2. What is SQA Plan?

3. Define Software quality Assurance (SQA).

4. List out the Teams and Responsibilities.

5. What makes the QA work minimal?

6. What are the factors affecting the SQA effort?

7. List out the seven keys to Leadership.

8. What are the ways of killing the Quality Assurance?

9. What are the Characteristics of Software quality assurances?

10. List out the elements of Documentation.

11. What is SRS?

12. What is SDD?

13. What are the steps in Document Distribution?

14. Draw the Documents Documentation plan worksheet.

15. Give the Reviewing description template.

16. What are action Items?

17. List out the special responsibilities of Technical Review Process.

18. What is software inspection?

19. What is walkthrough?

20. What is audit process?

21. Write short notes on Document verification.

Part-B

1. Explain in detail the Quality tasks and responsibilities. (16)

2. Explain in detail about the Documentation. (16)

3. Explain the Procedural description template and Action items (16)

4. Explain in detail the CMM Compatibility of Reviews and audits. (16)

5. Explain the management review process. (16)

6. Explain the technical review process. (16)

7. Explain the types of reviews. (16)

8. Explain the software inspection process. (16)

9. a) Explain walkthrough process. (8)

b) Explain the Audit process. (8)

10. a) Explain the Document verification and validation. (8)

b) Explain the factors affecting the SQA effort. (8)