Introduction to Software

Quality Assurance

December 7, 2008 2

ObjectivesObjectives

�� To introduce the quality management process To introduce the quality management process

and key quality management activitiesand key quality management activities

�� To explain the role of standards in quality To explain the role of standards in quality

managementmanagement

�� To explain the concept of a software metric, To explain the concept of a software metric,

predictor metrics and control metricspredictor metrics and control metrics

�� To explain how measurement may be used in To explain how measurement may be used in

assessing software quality and the limitations of assessing software quality and the limitations of

software measurementsoftware measurement

December 7, 2008 3

Topics coveredTopics covered

�� Process and product qualityProcess and product quality

�� Quality assurance and standardsQuality assurance and standards

�� Quality planningQuality planning

�� Quality controlQuality control

4

The process improvement cycleThe process improvement cycle

5

�� ���������� �������������� ���� ������ ���������������������������������� �������������������������������������������� ������������������ !��"#$� �� ����%�&�������������������� !��"#$� �� ����%�&��..

�� ()�*+,��������� ����%�&��*�-&��.�/�����������%�&��()�*+,��������� ����%�&��*�-&��.�/�����������%�&��..

�� �)�.�� �������������)�.�� ������������,, �� ����*+,�.�������������������� ����*+,�.������������������..

�� �)�.�� ����� �)�.�� ����� ,, (���+1((�"�-&�2(���+1((�"�-&�2 *�$����������3����*�$����������3������� ��� ..

Process and product qualityProcess and product quality

December 7, 2008 6

What is quality?What is quality?

�� Quality, simplistically, means that a product Quality, simplistically, means that a product

should meet its specification.should meet its specification.

�� This is problematical for software systemsThis is problematical for software systems

�� There is a tension between customer quality There is a tension between customer quality

requirements (efficiency, reliability, etc.) and requirements (efficiency, reliability, etc.) and

developer quality requirements (maintainability, developer quality requirements (maintainability,

reusability, etc.);reusability, etc.);

�� Some quality requirements are difficult to specify Some quality requirements are difficult to specify

in an unambiguous way;in an unambiguous way;

�� Software specifications are usually incomplete and Software specifications are usually incomplete and

often inconsistent.often inconsistent.

������4�5%����� (Software quality)

� 4�5%�����%)���/���������� ������������#����

� 4�5%�����/���� ����6��� !�7�"+8� ������9*���������6��

� 4�5%�����%�&��6����(�������������������������������� "������������ ���������%�&/�$�������������" *�$����������3���� )�����:� ���������3����"��"���;�5���� *+,����

What is software 'quality'?

� Quality software is reasonably bug-free,delivered on time and within budget, meetsrequirements and/or expectations, and is

maintainable

McCall’s Quality Factors ISO 9126 software qualities

functionality does it satisfy user needs?

reliability can the software maintain its levelof performance?

usability how easy is it to use?

efficiency relates to the physical resourcesused during execution

maintainability relates to the effort needed tomake changes to the software

portability how easy can it be moved to anew environment

Setting a standard for common

sub-characteristics of functionality

� suitability

� accuracy

� interoperability

� ability of software to interact with other

software components

� compliance

� degree to which software adheres to

application-related standards or legal

requirements e.g audit

� security

� control of access to the system

sub-characteristics of reliability

� maturity

� frequency of failure due to faults - the more

the software has been used, the more

faults will have been removed

� fault-tolerance

� recoverability

� note that this is distinguished from

‘security’ - see above

sub-characteristics of usability

� understandability

� easy to understand?

� learnability

� easy to learn?

� operability

� easy to use?

sub-characteristics of efficiency

� time behaviour

� e.g. response time

� resource behaviour

� e.g. memory usage

sub-characteristics of maintainability

� analysability

� ease with which the cause of a failure can

be found

� changeability

� how easy is software to change?

� stability

� low risk of modification having unexpected

effects

� testability

sub-characteristics of portability

� adaptability

� installability

� conformance

� standards that have bearing on portability

(compare to ‘compliance’) - e.g. use of high-

level language

� replaceability

� factors giving ‘upwards’ compatibility -

‘downwards’ compatibility is excluded

December 7, 2008 17

Software quality managementSoftware quality management

�� Concerned with ensuring that the required Concerned with ensuring that the required

level of quality is achieved in a software level of quality is achieved in a software

product.product.

�� Involves defining appropriate quality Involves defining appropriate quality

standards and procedures and ensuring standards and procedures and ensuring

that these are followed.that these are followed.

�� Should aim to develop a Should aim to develop a ‘‘quality culturequality culture’’

where quality is seen as everyonewhere quality is seen as everyone’’s s

responsibility.responsibility.

��&%�&����)�� 3

� Quality Management

� Process

� Project Management

� Metrics ,Measurement and Analytical Method

� Testing, Verification & Validation

� Audit

� Configuration Management

Quality Management

� �� ��.�������� (Quality Management) .��"3 �� ��.��+��*�%.� & %�&(������%�*�-&�*�-&��.�/������7" �"���*+B�.��"%�&*�&"�� ������

� �� ��.����������.��"�� /���$

� TQM=Total Quality Management

� CMMI('Capability Maturity Model Integration'),developed by the SEI. It's a model of 5 levelsof process 'maturity' that determine

effectiveness in delivering quality software

December 7, 2008 20

Scope of quality managementScope of quality management

�� Quality management is particularly Quality management is particularly

important for large, complex systems. The important for large, complex systems. The

quality documentation is a record of quality documentation is a record of

progress and supports continuity of progress and supports continuity of

development as the development team development as the development team

changes.changes.

�� For smaller systems, quality management For smaller systems, quality management

needs less documentation and should needs less documentation and should

focus on establishing a quality culture.focus on establishing a quality culture.

December 7, 2008 21

Quality management activitiesQuality management activities

�� Quality assuranceQuality assurance�� Establish organisational procedures and standards for Establish organisational procedures and standards for

quality.quality.

�� Quality planningQuality planning�� Select applicable procedures and standards for a Select applicable procedures and standards for a

particular project and modify these as required.particular project and modify these as required.

�� Quality controlQuality control�� Ensure that procedures and standards are followed Ensure that procedures and standards are followed

by the software development team.by the software development team.

�� Quality managementQuality management should be separate from should be separate from project management to ensure independence.project management to ensure independence.

22

Principal product quality factorsPrincipal product quality factors

23

Quality factorsQuality factors

�� �)�.�� �)�.�� 7�����.D$7�����.D$ �� ������ ����(�*+,����������(�*+,����������������������

�� �)�.�� �)�.�� 7����*�;7����*�; ���������3���������3���� developer developer (�*+,����������(�*+,����������..

�� *%�7�7�"�����6��*%�7�7�"�����6�� (��� % �%�)���D��7����*�;(��� % �%�)���D��7����*�;..�� �)�.�� %�����)�.�� %����,, )�.����%�&*�$���)�.����%�&*�$���(������$�������(������$�������..

December 7, 2008 24

�� The quality of a developed product is The quality of a developed product is

influenced by the quality of the production influenced by the quality of the production

process.process.

�� This is important in software development This is important in software development

as some product quality attributes are hard as some product quality attributes are hard

to assess.to assess.

�� However, there is a very complex and However, there is a very complex and

poorly understood relationship between poorly understood relationship between

software processes and product quality.software processes and product quality.

Process and product qualityProcess and product quality

December 7, 2008 25

ProcessProcess--based qualitybased quality

�� There is a straightforward link between process There is a straightforward link between process

and product in manufactured goods.and product in manufactured goods.

�� More complex for software because:More complex for software because:

�� The application of individual skills and experience is The application of individual skills and experience is

particularly important in software development;particularly important in software development;

�� External factors such as the novelty of an application External factors such as the novelty of an application

or the need for an accelerated development schedule or the need for an accelerated development schedule

may impair product quality.may impair product quality.

�� Care must be taken not to impose inappropriate Care must be taken not to impose inappropriate

process standards process standards -- these could reduce rather these could reduce rather

than improve the product quality.than improve the product quality.December 7, 2008 26

ProcessProcess--based qualitybased quality

December 7, 2008 27

�� Define process standards such as how Define process standards such as how reviews should be conducted, configuration reviews should be conducted, configuration management, etc.management, etc.

�� Monitor the development process to ensure Monitor the development process to ensure that standards are being followed.that standards are being followed.

�� Report on the process to project Report on the process to project management and software provider.management and software provider.

�� DonDon’’t use inappropriate practices simply t use inappropriate practices simply because standards have been established.because standards have been established.

Practical process qualityPractical process quality

December 7, 2008 28

�� Standards are the key to effective quality Standards are the key to effective quality management.management.

�� They may be international, national, They may be international, national, organizational or project standards.organizational or project standards.

�� Product standardsProduct standards define characteristics define characteristics that all components should exhibit e.g. a that all components should exhibit e.g. a common programming style.common programming style.

�� Process standardsProcess standards define how the define how the software process should be enacted.software process should be enacted.

Quality assurance and standardsQuality assurance and standards

December 7, 2008 29

�� Encapsulation of best practiceEncapsulation of best practice-- avoids avoids

repetition of past mistakes.repetition of past mistakes.

�� They are a framework for quality They are a framework for quality

assurance processes assurance processes -- they involve they involve

checking compliance to standards.checking compliance to standards.

�� They provide continuity They provide continuity -- new staff can new staff can

understand the organisation by understand the organisation by

understanding the standards that are understanding the standards that are

used.used.

Importance of standardsImportance of standards

December 7, 2008 30

Product and process standardsProduct and process standards

Product standards Process standards

Design review form Design review conduct

Requirements document structure Submission of documents to CM

Method header format Version release process

Java programming style Project plan approval process

Project plan format Change control process

Change request form Test recording process

December 7, 2008 31

Problems with standardsProblems with standards

�� They may not be seen as relevant and upThey may not be seen as relevant and up--

toto--date by software engineers.date by software engineers.

�� They often involve too much bureaucratic They often involve too much bureaucratic

form filling.form filling.

�� If they are unsupported by software tools, If they are unsupported by software tools,

tedious manual work is often involved to tedious manual work is often involved to

maintain the documentation associated maintain the documentation associated

with the standards.with the standards.

December 7, 2008 32

�� Involve practitioners in development. Engineers Involve practitioners in development. Engineers

should understand the rationale underlying a should understand the rationale underlying a

standard.standard.

�� Review standards and their usage regularly. Review standards and their usage regularly.

Standards can quickly become outdated and this Standards can quickly become outdated and this

reduces their credibility amongst practitioners.reduces their credibility amongst practitioners.

�� Detailed standards should have associated tool Detailed standards should have associated tool

support. Excessive clerical work is the most support. Excessive clerical work is the most

significant complaint against standards.significant complaint against standards.

Standards developmentStandards development

December 7, 2008 33

ISO 9000ISO 9000

�� An international set of standards for quality An international set of standards for quality management.management.

�� Applicable to a range of organisations Applicable to a range of organisations from manufacturing to service industries.from manufacturing to service industries.

�� ISO 9001 applicable to organisations ISO 9001 applicable to organisations which design, develop and maintain which design, develop and maintain products.products.

�� ISO 9001 is a generic model of the quality ISO 9001 is a generic model of the quality process that must be instantiated for each process that must be instantiated for each organisation using the standard.organisation using the standard.

December 7, 2008 34

ISO 9001ISO 9001

Management responsibility Quality system

Control of non-conforming products Design control

Handling, storage, packaging and

delivery

Purchasing

Purchaser-supplied products Product identification and traceability

Process control Inspection and testing

Inspection and test equipment Inspection and test status

Contract review Corrective action

Document control Quality records

Internal quality audits Training

Servicing Statistical techniques

December 7, 2008 35

ISO 9000 certificationISO 9000 certification

�� Quality standards and procedures should be Quality standards and procedures should be

documented in an organisational quality documented in an organisational quality

manual.manual.

�� An external body may certify that an An external body may certify that an

organisationorganisation’’s quality manual conforms to ISO s quality manual conforms to ISO

9000 standards.9000 standards.

�� Some customers require suppliers to be ISO Some customers require suppliers to be ISO

9000 certified although the need for flexibility 9000 certified although the need for flexibility

here is increasingly recognised.here is increasingly recognised.

December 7, 2008 36

ISO 9000 and quality managementISO 9000 and quality management

December 7, 2008 37

Documentation standardsDocumentation standards

�� Particularly important Particularly important -- documents are the documents are the

tangible manifestation of the software.tangible manifestation of the software.

�� Documentation process standardsDocumentation process standards

�� Concerned with how documents should be Concerned with how documents should be

developed, validated and maintained.developed, validated and maintained.

�� Document standardsDocument standards

�� Concerned with document contents, structure, and Concerned with document contents, structure, and

appearance.appearance.

�� Document interchange standardsDocument interchange standards

�� Concerned with the compatibility of electronic Concerned with the compatibility of electronic

documents.documents.December 7, 2008 38

Documentation processDocumentation process

December 7, 2008 39

Document standardsDocument standards

�� Document identification standardsDocument identification standards

�� How documents are uniquely identified.How documents are uniquely identified.

�� Document structure standardsDocument structure standards

�� Standard structure for project documents.Standard structure for project documents.

�� Document presentation standardsDocument presentation standards

�� Define fonts and styles, use of logos, etc.Define fonts and styles, use of logos, etc.

�� Document update standardsDocument update standards

�� Define how changes from previous versions Define how changes from previous versions are reflected in a document.are reflected in a document.

December 7, 2008 40

Document interchange standardsDocument interchange standards

�� Interchange standards allow electronic Interchange standards allow electronic documents to be exchanged, mailed, etc. documents to be exchanged, mailed, etc.

�� Documents are produced using different Documents are produced using different systems and on different computers. Even when systems and on different computers. Even when standard tools are used, standards are needed standard tools are used, standards are needed to define conventions for their use e.g. use of to define conventions for their use e.g. use of style sheets and macros.style sheets and macros.

�� Need for archiving. The lifetime of word Need for archiving. The lifetime of word processing systems may be much less than the processing systems may be much less than the lifetime of the software being documented. An lifetime of the software being documented. An archiving standard may be defined to ensure archiving standard may be defined to ensure that the document can be accessed in future.that the document can be accessed in future.

December 7, 2008 41

Quality planningQuality planning

�� A quality plan sets out the desired product A quality plan sets out the desired product

qualities and how these are assessed and qualities and how these are assessed and

defines the most significant quality attributes.defines the most significant quality attributes.

�� The quality plan should define the quality The quality plan should define the quality

assessment process.assessment process.

�� It should set out which organisational It should set out which organisational

standards should be applied and, where standards should be applied and, where

necessary, define new standards to be used.necessary, define new standards to be used.

December 7, 2008 42

Quality plansQuality plans

�� Quality plan structureQuality plan structure

�� Product introduction;Product introduction;

�� Product plans;Product plans;

�� Process descriptions;Process descriptions;

�� Quality goals;Quality goals;

�� Risks and risk management.Risks and risk management.

�� Quality plans should be short, clearly Quality plans should be short, clearly

documentsdocuments

�� If they are too long, noIf they are too long, no--one will read them.one will read them.

December 7, 2008 43

Software quality attributesSoftware quality attributes

Safety Understandability Portability

Security Testability Usability

Reliability Adaptability Reusability

Resilience Modularity Efficiency

Robustness Complexity Learnability

December 7, 2008 44

Quality controlQuality control

�� This involves checking the software This involves checking the software

development process to ensure that development process to ensure that

procedures and standards are being procedures and standards are being

followed.followed.

�� There are two approaches to quality There are two approaches to quality

controlcontrol

�� Quality reviews;Quality reviews;

�� Automated software assessment and Automated software assessment and

software measurement.software measurement.

December 7, 2008 45

Quality reviewsQuality reviews

�� This is the principal method of validating the This is the principal method of validating the

quality of a process or of a product.quality of a process or of a product.

�� A group examines part or all of a process or A group examines part or all of a process or

system and its documentation to find potential system and its documentation to find potential

problems.problems.

�� There are different types of review with different There are different types of review with different

objectivesobjectives

�� Inspections for defect removal (product);Inspections for defect removal (product);

�� Reviews for progress assessment (product and Reviews for progress assessment (product and

process);process);

�� Quality reviews (product and standards).Quality reviews (product and standards).

December 7, 2008 46

Types of reviewTypes of review

Review type Principal purpose

Design or program

inspections

To detect detailed errors in the requirements, design or code. A checklist of

possible errors should drive the review.

Progress reviews To provide information for management about the overall progress of the

project. This is b oth a process and a product review and is concerned with

costs, plans and schedules.

Quality reviews To carry out a t echnical analysis of product components or documentation to

find mismatches between the specification and the component design, code or

documentation and to ensure that defined quality standards have been followed.

December 7, 2008 47

�� A group of people carefully examine part or all A group of people carefully examine part or all

of a software system and its associated of a software system and its associated

documentation.documentation.

�� Code, designs, specifications, test plans, Code, designs, specifications, test plans,

standards, etc. can all be reviewed.standards, etc. can all be reviewed.

�� Software or documents may be 'signed off' at a Software or documents may be 'signed off' at a

review which signifies that progress to the next review which signifies that progress to the next

development stage has been approved by development stage has been approved by

management.management.

Quality reviewsQuality reviews

December 7, 2008 48

Review functionsReview functions

�� Quality function Quality function -- they are part of the they are part of the

general quality management process.general quality management process.

�� Project management function Project management function -- they they

provide information for project managers.provide information for project managers.

�� Training and communication function Training and communication function --

product knowledge is passed between product knowledge is passed between

development team members.development team members.

December 7, 2008 49

Quality reviewsQuality reviews

�� The objective is the discovery of system The objective is the discovery of system

defects and inconsistencies.defects and inconsistencies.

�� Any documents produced in the process Any documents produced in the process

may be reviewed.may be reviewed.

�� Review teams should be relatively small Review teams should be relatively small

and reviews should be fairly short.and reviews should be fairly short.

�� Records should always be maintained of Records should always be maintained of

quality reviews.quality reviews.

December 7, 2008 50

�� Comments made during the review should be Comments made during the review should be classifiedclassified

�� No action. No change to the software or documentation is No action. No change to the software or documentation is required;required;

�� Refer for repair. Designer or programmer should correct Refer for repair. Designer or programmer should correct an identified fault;an identified fault;

�� Reconsider overall design. The problem identified in the Reconsider overall design. The problem identified in the review impacts other parts of the design. Some overall review impacts other parts of the design. Some overall judgement must be made about the most costjudgement must be made about the most cost--effective effective way of solving the problem;way of solving the problem;

�� Requirements and specification errors may Requirements and specification errors may have to be referred to the client.have to be referred to the client.

Review resultsReview results

Measures, Metrics and Indicators

� Measure (N) �$�%�&/��(������

� Measurement (N) �����

� Metrics �$�%�&/��(������*��+����� �3 ���� �������:�� �+��������

� Indicator �-� metric .�-���$��� metric %�& $��!�����:����"���� software process software project .�-� product ��!�2

[[Pressman1997Pressman1997]]

Why Do We Measure?

� To understand what is happening during development and maintenance.

� To control what is happening on our projects.

� To improve our process and products.

A Good Manager Measures

measurementmeasurement

What do weWhat do weuse as ause as abasis?basis?•• size?size?•• function?function?

project metricsproject metrics

process metricsprocess metricsprocessprocess

productproduct

product metricsproduct metrics

Process Metrics and Project Metrics

� Process metrics� ���������(������������"������ ���� +����%�������

�� ����

� Project metrics� �����������- .�����7����

� ���(�� ����*��&"�$�2

� +��*�����������%��

December 7, 2008 55

Predictor and control metricsPredictor and control metrics

December 7, 2008 56

Product measurement processProduct measurement process

Process Metrics and Software Process Improvement

� ��������:��%�&�����

� ���� metric �������:����!�

� �)� metric %�&/����*+,�*��-&� $��!*�-&�%�&(��)�/++�� +����

[[Pressman1997Pressman1997]]

Private Process Data

� ����%����(�#�� ��% ��*%-��3�������)�*���� ������������������������:���$�� ����

� ����#����$�����(�(���.�*+,�����#��$���������$�� ���� *�-&�%�&����!�(�/���)�����#�/+�������+�� +�����������3� ��*��&�+����%��������� 7�"���� !��"#$� ��*��&�+����%�����

�$�� ����������%�����"

���"�%�����)� metrics /+�������+��*���

� ��� common sense *�-&�%)���������.��"����#�

� �.� feedback �$ �����%�&�$���-�����������+��*����"$���&)�*���

� �"$������%�&/��(�����������.�����$�� ��$����

� �"�"����!*+B�.��"

Project Metrics

� *+B�.��"

� �����������"�*���������6��7�"+�� *+��&"� ��"$�*�-&�%�&(��������$���� ���������*��&"

� �������+������������� product ���"�����3���������(����+�� +��.����%� technical *�-&�*��&�������/��

Software Measurement

� direct measures

� indirect measures

December 7, 2008 62

�� Any type of measurement which relates to a software system, Any type of measurement which relates to a software system, process or related documentationprocess or related documentation

�� Lines of code in a program, the Fog index, number of Lines of code in a program, the Fog index, number of personperson--days required to develop a component.days required to develop a component.

�� Allow the software and the software process to Allow the software and the software process to be quantified.be quantified.

�� May be used to predict product attributes or to control the May be used to predict product attributes or to control the software process.software process.

�� Product metrics can be used for general predictions or to Product metrics can be used for general predictions or to identify anomalous components.identify anomalous components.

Software metricSoftware metric

Software Measurement

� Size-Oriented Metrics� �������*��������

� Function-Oriented Metrics� �������*������%�&7+������%)�

Size-Oriented Metrics

� ����%�&(�%)���*+��" *%�" ��.�$�7���� *����"���� ()���� ��%�� (LOC-Line of Code) ����+�� �$�

projectproject LOCLOC EffortEffort $$((000000)) pppp..docdoc ErrorError DefectDefect PeoplePeople

alphaalpha

betabeta

gammagamma

12,10012,100

27,20027,200

20,20020,200

2424

6262

4343

168168

440440

314314

365365

12241224

10501050

134134

321321

256256

2929

8686

6464

33

55

66

errors per KLOCerrors per KLOC

defect per KLOC defect per KLOC

$ per LOC$ per LOC

documents per KLOCdocuments per KLOC

errorerror//personperson--monthmonth

LOCLOC//personperson--monthmonth

$$//page of documentationpage of documentation

Size-Oriented Metrics

� /�$*+,�%�&"���� �� #����$�*+,�����&%�&���������+����%��������� ������6��

� +1D.�������� Line of Code

�Programming language

dependent

� *+,�+1D.�� programmer %�&��� 7+�����������%�����

Function-Oriented Metrics

� ���%�&��%�&7+������%)�

� /�$�����3���/��7�"��

� ������� function point

Parameter Counted

� user input

� user output

� user inquiries

� files

� external interfaces

Computing Function Points

� �)�()����%�&�� /�����.��!)�.�� -> count-total

� FP = (count-total) x (complexity adjustment value)

� complexity adjustment value

� system reliable?

� data communication?

� performance critical?

Metrics for Quality

� Correctness� defect / KLOC

� Maintainability� MTTC - Mean Time To Change

� Spoilage - �$����($�"%�&���������/���� ��$�.��(�/���$7����/+����

� Integrity ���������

� Usability

Defect Removal Efficiency

� DRE = E / (E+D)� E = ()���� error %�&� $���$ software

� D = ()���� defect .�����$ software

� �����3�)�/+����� process /��� ������������3����)�(�� error $��(��$��/+"� phase �-&�

� DREi = Ei / ( Ei + E(i+1) )

���)� metrics ��������� ������6��4�5������

� �������+�� +������������ ����

� �������+��������������

December 7, 2008 72

Key pointsKey points

�� Software quality management is concerned with Software quality management is concerned with ensuring that software meets its required ensuring that software meets its required standards.standards.

�� Quality assurance procedures should be Quality assurance procedures should be documented in an organisational quality manual.documented in an organisational quality manual.

�� Software standards are an encapsulation of best Software standards are an encapsulation of best practice.practice.

�� Reviews are the most widely used approach for Reviews are the most widely used approach for assessing software quality.assessing software quality.

December 7, 2008 73

Key pointsKey points

�� Software measurement gathers information Software measurement gathers information

about both the software process and the about both the software process and the

software product.software product.

�� Product quality metrics should be used to Product quality metrics should be used to

identify potentially problematical components.identify potentially problematical components.

�� There are no standardised and universally There are no standardised and universally

applicable software metrics.applicable software metrics.