Definitions and Concepts ofTesting and Quality

What is

Testing?

What is

Quality ?

How Related?

Introduction to Testing Software

• Testing is a relatively new discipline although programmers always “debugged” their programs. – Testing was conducted to show that the software

works.

• In the 1970’s Glen Myers wrote a book, Art of Software Testing (1978). – He believed that the main purpose of testing is to

find faults.

• Are these (works .vs. faults) opposing views?• Why should we and why do we Test ?

Why Test ?

• We can’t assume that the software “works”

– How do we answer the question: “Does this software work?”

• Is it reliable ?• Is it functional ?

– Complete– Consistent

• Is it responsive ?

• In general, what is the Quality of our software?– How do we answer this?

How would you answer this about the program that you wrote?How would you answer this about the program your friend wrote?

Quality of Software?

• Depends on what we include as software– Just the executable code ----- How about ?

• Include the pre-populated data in the database• Include the help text and error messages• Include the source code• Include the design document• Include the test scenarios and test results• Include the reference manuals• Include the requirements document

• When we talk about quality how much of the above do we include?

• How do we test these different artifacts?

Testing

• Traditionally, testing includes executing the code. (assuming - code is the main software artifact)

• What do we do with the non-executable software artifacts?– Reviews and inspections

• Expensive in terms of human resources

• A lot to maintain and keep updated

• Can we “prove” that the software works or is defect free?– Theoretically, given an arbitrary program we can not show

that it has no bug.– We can use formal proofs to show the behavior of code

An Informal Survey in the 90’s

• Professionals taking a course in testing:– 60% were new to testing– 20% had 1 to 5 years of experience in testing– 20 % expert testers

• Metric used in testing– Most regularly used: Counting bugs found and ranking them by

severity– Small number used : bug find rate or bug fix rate

• Formal Methods used– Almost none in inspection or formal analysis

• Test tool:– 76 % had been exposed to some automated test tool

• Test definitions– Most of the practicing testers could not supply good definitions of

testing terms; they just did the work!

Historical Progression of Software Quality-Large Host & CentralizedSystems-Single Vendor(hdw,sw,serv)-Long term investment(10 yrs)-Single platform-Systems ran by computer professionals

**Product Reliability & Quality was required and expected**

1980’s & Before

-PC and Desktop Computing became ubiquitous-multiple vendors-quick product development & shorter term investment-systems ran by non-computer individuals

**New product was fashionable& “reboot” was acceptable.

-Web availability to many-Business conducted on the Web-Software and systems are not hobbies but a business again

**Product Reliability & Quality is once again important**

1990’s

Late 1990’s & 2000’s

Current State on Testing• Software is written by many with the

“entrepreneurial” spirit:– Speed to market– New & innovation is treasured– Small organization that can’t afford much more

than “coders”

• Embracing “Agile” process and mistaking it as “fast production regardless of what”:– Not much documented (requirements/design)– Hard to test without documented material

• Lack of Trained/Good/Experienced Testers– Testers are not rewarded as equals with designers

• Improvement tools and standards making development easier and less error prone

Why ?

Quality

• What is Quality?• Some common comments:

– “I know it when I see it”– Reliable – Meets requirements– Fit for use– Easy to use– Responsive– Full function / full feature– Classy and luxurious

How would you define quality?

Some U.S. “pioneers” on Quality

• Pioneers:– Joseph M. Juran

• Quality =>Fitness for use

– W. Edward Deming• Quality => Non-faulty system

• More recently:– Philip Crosby

• Quality => conformance with requirements• Achieve quality via “prevention” of error• Target of Quality is “zero defect”• Measure success via “cost of quality”

Quality

• Quality is a characteristic or attribute – Needs to be clearly defined and agreed to

• May have sub-attributes• Needs specific metrics for each of the sub-attributes

– Needs to be measured with the defined metric(s)– Needs to be tracked and analyzed– Needs to be projected– Needs to be controlled

• Testing and Measurement are two key activities that would help us manage quality.

Note: Hutcheson, in your text book states that quality is a metric and that it measures excellence Discuss --- agree / disagree & why? ; What is a Metric?

Some Precept Concerning Quality & QA

• Quality Requirements Does not Dictate Schedule– Market dictates schedule (especially for small companies)– For large and multiple release software, quality is still a

factor and may affect schedule ---- albeit seldom– Software development process today incorporates both the

need for speed and quality (incorporating the notion of service cost versus rewrite for a replacement new product.)

• Quality does not require “zero defect” Reliability– Commercial (non-life critical or mission critical) products are

not developed with this goal in mind. They are much more market driven --- market does not demand zero defects.

• Focus on proper support

• Focus on main functions and heavily used areas (not all defects are the same)

• Focus on customer productivity (e.g. easy to learn and use)

• Zero Defect is very expensive proposition ( time & resource)

Some Precept Concerning Quality & QA(cont.)

• Users Do Not Always Know What They Want– Users may not know all the requirements (especially for

large, complex systems which require professional or legal knowledge.)

– Users may not have the time or interest to “really focus” on the requirements at the time when asked (timing problem). Users have their own fulltime jobs.

– Users may not know how to prioritize needs from wishes– Users may not know how to articulate clearly all the

requirements. (They are non-software development people.)– Developers may not listen well or properly interpret the

users statements. (They are not industry specialists.)

Requirements is a key factor in software development ---- why?How does it affect software quality? ----- think about definitions of Quality

Some Precept Concerning Quality & QA(cont.)

• Requirements are not always Clear, Stable, and Doable– Not all requirements are technically feasible; sometimes the

“desired” new technology needs to be prototyped first.– Sometimes the requirements are changed, causing re-

design or re-code without proper assessment of schedule impact.

– Requirements are not always reviewed and signed off, but sometimes given in verbal form --- especially small changes.

– People mistake iterative development to mean continuous change of requirements.What’s the danger here? – cost, schedule, quality

Some Precept Concerning Quality & QA(cont.)

• Customers often go for “New and Exciting” Product– “If the product has all the needed features it would sell” is

not necessarily true; people often want new features.– Reliability is not always enough; sometimes customers will

sacrifice quality for new and exciting features.• Recall the story of IBM OS/2 operating system and Microsoft’s

operating system (even though both was commissioned by IBM).

– IBM went for Reliability of the old Host Machines for desktop PC’s– Microsoft went for exciting individual user interfaces

Over emphasis of “exciting features” is one reason why we are regressing in software quality in theLast ten years !

Some Precept Concerning Quality & QA(cont.)

• Price and Availability is sometimes more important to customers (especially in commodity level software) than Product Maturity.– At the commodity level software, the customers are

individuals who wants the product now at an competitive price. (much like shopping for a home product such as a T.V.)

– Sophisticated and full feature software needs to be balanced and sometimes traded off for price and speed.

– Customers don’t always want all the functions and product maturity they think they require ---- if the price is right!

Controlling Quality

• As software size and complexity increased in the 1960’s and 1970’s, many software projects started to fail. – Many software did not perform the required functions– Others had performance (speed) problems– Some had large number of defects that prevented users

from completing their work; some just flat out won’t even install !

• Software “Quality” Crisis was recognized and Quality Assurance was born.

Software Quality Assurance (QA)

• Software QA focused on 2 main areas– Software product– Software process

• The focus on the process areas “borrowed” many techniques from the traditional manufacturing area– Concept of reliability (number of defects, mean

time to failure, probability of failure, etc. metrics)– Concept of process control in terms of looking at

“repeatability” of process--- repeatable process produces “similar” product (or controllable results).

QA, Process Control, and Documentation

• A period of heavy emphasis on software development process and excessive documentation dominated QA --- initially this improved the “software crisis.”– Process included multiple steps of reviews– Process included multiple steps of test

preparation, testing, and test result analysis– Process was controlled by many documents and

document flow which also improved project communications

• But ----- the price was speed and innovation.

Software is NOT manufacturing (or is it?)

• Software Development is extremely labor intensive– People are not uniform like machines used in

manufacturing.

• Software Development often requires innovation– Every software seems to be a one of its kind

although more and more are becoming “standardized”

Some ideas of adjusting the QA

• Hutcheson’s Goals of QA– Quality is defined as customer satisfaction– Quality is achieved by constant refinement– Quality is measured by profit– Quality process should produce a hit everytime

• Hutcheson’ Approaches to Quality Product– Be first to market with the product– Price the product correctly– Ensure the product has both required functions plus

wishful features– Ensure the bugs are kept to minimum ---- less than your

competitors’ bugs.

Do you agree? ---- the end product may have many bugs but that may be ok if it is less than competitors and if the customers don’t mind ------ what about minimizing fix cost by having less bugs? ( the author forgot support business)

Areas of Improvements for QA Process and Control

• Automate Record Keeping

• Improve Documentation Techniques

• Use trained testers and improve on their tools and methodologies

Automate Record Keeping

• Many records are kept and should be automated with on-line forms:– Test plan– Test schedule– Test scripts– Test results– Etc.

• The information often should be available to a set of people:– Repository or Configuration Management– Collaborative web-sites

Improve Documentation Techniques

• Improve the creation and maintenance of documents via an on-line repository or configuration manager– Centrally protected data– Sharable data– Managed data (data relationships are managed)

• Improve the “usability” and “productivity” by providing more and better visualization of data– Replace numbers with graphs and figures– Replace words with pictures

Improve Testers’ Tools and Methodology

• Test Methodology Improvements– Test coverage analysis– Test case generation– Test-Fix-Integrate Process– Test results analysis – Test metrics definition and measurements process– Etc.

• Test tools improvements– Test coverage computation– Test trace– Test script generator– Test result records keeping and automated analysis– Build and integration (daily builds)– Etc.