delivering quality software through formal testing

8/6/2019 Delivering Quality Software Through Formal Testing

1/75

Leonard Aye

MICT Park

Yangon

22 January 2010


2/75

@ [email protected]

[email protected]

MSN [email protected]

Leonard Aye

lenaye

2


3/75

Do we need to test software?

Definitions of Good quality software

Categories of Software

Definition of Testing Development models & testing processes Case studies: Waterfall

V-Model

Agile (SCRUM)

Building a test team

Q&A

3


4/75

Date: June 4th 1996

Place: European Space Agency Launch Site

Location: French Guiana, South America

4


5/75

5

Rocket: Ariane 5


6/75

Why did it blow up? There was an error in the software that calculates the

rockets height and direction.

Error occurred when a 64-bit Floating Point Numberwas converted to a Signed 16-bit Integer Number,

causing the resulting Integer number to overflow

There was no error handling for this type of

Overflow error because this condition was neverdiscovered during testing.

6


7/75

Whats the cost of this failure? Development time: 11 years

Total cost: US $11 Billion

Whats the Conclusion? There should have been more testing!

7


8/75

Whatis testing?

What typesof testing are there?

Whatdo we test exactly?

How much testing is enough?

How much time do we need for testing?

How manytesters do you need?

What makes a good tester?

8


9/75

What is BAD quality software? Its a bad piece of software

if it is full of bugs (errors)!

Its a bad piece of software even if its 100% bugs free but

it doesnt do what the user want

9


10/75


if it is full of bugs (errors)!

Its a bad piece of software even if its 100% bugs free, but

it doesnt do what the user want

Its a bad piece of software

even if its 100% bugs free and

it does exactly what the user want, but

not in the way the user wants it

10


11/75


even if its 100% bugs free and

it does exactly what the user want, and it works in the way the user wants it, but

it works sloooowly

11


12/75

It works without serious errors

Easy to install, configure and use

Has features the users want Works in the way the users expect it to work

Responsive and high-performance

Good aesthetics, e.g. clean and uncluttered

interface and ease of navigation Has good and extensive documentation

Good technical support

12


13/75

3 main types Off-the-shelf /Packaged software

Bespoke software

Customised software

13


14/75

Off-the-Shelf/Packaged software Ready-made software which you can pick up off the

shelf in a supermarket. Some Examples

Desktop Applications:

Windows Vista, Windows 7, Mac OS X

MS Office

Photoshop

Chrome browser

Online sites & serives: Google

Facebook

Youtube

14


15/75

Bespoke software Software specifically written for one customer or

one organisation only

2 types of bespoke software New computerised system for a business

Replace existing computerised system with new

(faster/better) system

15


16/75

Customised software A generic application which is configured for

each customer/client

Some commercial off-the-shelf examples:

Accounting packages, e.g. Sage, MS Money

Payroll systems, e.g. Zpay, Intuit

Ecommerce packages, e.g. E-Volved, Actinic, EROL

16


17/75

17

Off-the-Shelf Bespoke/Customised

User Requirements From within your own

organisation

From customer

Project duration No limits, except for external

pressure

Defined by customer

Deployment via websites or shops Defined with customer

Testing process Defined by development

process

Defined by development

process

Customer feedback via email or website Direct from customer

Customer interaction Little or none High


18/75

A definition Testing is the process of comparing the invisible

to the ambiguous, so as to avoid the unthinkable

happening to the anonymous. James Bach

18


19/75

A few well known definitions Testing is a process used to identify the correctness and

completeness of computer software.

Testing is verifying software features against the system

requirements Testing is a process of verifying that a program functions properly.

Preferred definition Testing is a process of finding bugs in the software

Kaner, Falk & Nguyen, Testing Computer Software, 1999

Testing is a process of trying to break the software duringDevelopment so that it can be fixed, thereby reducing companystime and money after the product is released.

19


20/75

Integration Testing

System Testing Functional Testing

Non-Functional Testing

User Acceptance Testing

End-to-End Testing

Regression TestingExploratory Testing

20


21/75

Part 1 Do we need to test software?



Definition of Testing

Development models & test processes Case studies:

Waterfall

V-Model Agile (SCRUM)


Q&A

21


22/75

Software development model defines a set of ruleson how a software is to be built.

There are many and varied development models: Agile

DSDM (Dynamic Systems Development Method)

FDD (Feature Driven Development)

Iterative

RAD (Rapid Application Development)

RUP (Rational Unified Process)

Scrum

TDD (Test Driven Development) V-Model

Waterfall

XP (eXtreme Programming)

22


23/75

Traditional Iterative/Incremental

V-Model Agile

Waterfall FDD (Feature Driven Development)



ScrumRUP (Rational Unified Process)


TDD (Test Driven Development)

23


24/75

Waterfall model is a software

development process in which

progress is seen as flowing steadily

downwards (like a waterfall) through

the phases of:- Requirements

Design

Construction

Testing, and

Maintenance.

24


25/75

25

Requirements

Analysis

Design

Construction

Testing

Maintenance


26/75

Small to medium scale projects, i.e. < 10 people

and/or timescale ~6-9 months

Requirements are clearly defined and unlikely to

change throughout the life of the project

26


27/75

28

msn shopping

Price comparison site

Online Merchants send in their

product catalogue, daily/weekly

Products are categorised and

published on their site Business model

Online Merchants get greater

exposure to their products from

users on msn

User clicks are recorded and

Charge per click-throughs


28/75

29


29/75

Environment

30

Pr 6

SQL

ServerSQL

Server

Pr 1 Pr 2

SQL

Server

Staging

SQL

Server

Dev

SQL

Server

MerchantMerchant Merchant Merchant

Load Balancer

Test

SQL

Server


30/75

Requirements Website is non-business critical

Overnight switchover (big bang)

Website must display consistently on all existing web

browsers

Website must have high availability (24x7)

Website must be able to support high user traffic

Merchant products upload


31/75

How do we test this system? System Testing

Functional Testing


End-to-End testing

What do we need to test this? Test Servers (Application and DB server) with all necessary

apps installed, i.e. IIS & SQL Server User PCs (Win 98, Win ME, Win 2K, Win NT) with IE 3, IE4,

IE4.01 SP1 ,IE 4.02 SP2, etc.

Test Data (manually created dummy data)

32


32/75

System Testing Ensuring the system meets the user/design

requirements

Functional Testing Aim is to find errors (bugs)

Does it work?

Does it do what its supposed to do?

e.g. functional testing a car would be:-

Check that engine starts if you turn on the ignition key

Check that the break lights come on if you press on the breakpedals

Check that radio comes on if you turn on the radio

33


33/75

System Testing UI Test cases

Layout of products

Product description too long

Display of special characters

Missing images (at site and from merchant)

Missing information, e.g. Detailed description

Missing mandatory information, i.e. price

Special offer (normal price, sale price, and sale indicator) Incorrect or dead-links back to merchant site

Functional Test cases

Correct categorisation of products

Product filtering by type, by brand, by store, by price range(pre-defined or manual)

Product sorting

Product searching via Search box

Product listing per page


Web server breakdown

SQL server breakdown

Load/Stress testing

Average page load times: < 10ms

Min concurrent user: 400

34


34/75


Aim is NOT to find faults but tomeasure performance againstthe required targets

How does it work? Fast, slow, etc.

Is it easy to use? (Usability testing)

Can users of all different versions of OSs and browsers access thesite? (Compatibility testing)

Will the system be available in case of server failures (Reliabilityand Availability testing)

Can the system cope with the growth of data volume or uservolume in the future (Load testing)

e.g. non-functional testing of a car would be:-

Check that it accelerates from 060 mph in under 10 secs

Check that the electric door opens/closes 100,000 times withoutfailure

35


35/75

Usability testing

Invited people from outside the office and asked them to

use the website as normal users, i.e.

Search for products

Find a particular product Filter products by price

Filter products by brand, etc.

This is followed by questionnaires for their feedback

Amending the layout and design of the site based on users

feedback

36


36/75

Compatibility testing

Testing that the application/website is compatible to use on

as many supported browers and OSs as possible

Set up a test laboratory with several PCs running different

OSs and web browsers Check how the site displayed on each type of browser

Use VMWare to set up Virtual Machines

37


37/75

Reliability testing

Testing that the system is still available in the event of failure

in any part of the system

Manually turned off each web servers and check that the user

transactions are re-routed correctly Manually turned off each DB server and check the user data

request is served by a different DB server

38

WS 6

SQL

ServerSQL

Server

WS 1 WS 2

SQL

Server

Load Balancer


38/75

Availability testing

Testing that the system is still available in the event of total

system and site failure

Set up a duplicate/back up site at a different location

Manually turned off main site and check that the usertransactions are re-routed to backup site

Check for data synchronisation between the main DBs and

backup DBs

39

W

S

6

SQL

Ser

ver

SQL

Ser

ver

W

S

1

W

S

2

SQL

Ser

ver

Load Balancer

W

S

6

SQL

Ser

ver

SQL

Ser

ver

W

S

1

W

S

2

SQL

Ser

ver

Load Balancer

Sync

London Derby


39/75

40

SQL

Server

SQL

Server

SQL

Server

SQL

Server

SQL

Server

SQL

Server

Load Balancer

Load Injectors Load Controller

0

10

20

30

40

50

60

70

Users 50 100 150 200 250 300 350 400 450

Load testing analysisTime (ms)

Concurrent Users

Load Testing Example


40/75

Summary: Duration: 10 months from concept to production

Written entirely ASP and SQL

Team:

1 PM

2 Merchant Liaisons

4 Dev

2 DBAs

4 Testers (London, Dublin, Outsourced company)

Big-Bang deployment

Old system turned off Friday night and new system brought

up on Monday morning

41


41/75


42/75




Definition of TestingDevelopment models & testprocesses Case studies:

Waterfall (examples of non-functional testing)

V-Model Agile (SCRUM)


Q&A

43


43/75

44

User

Requirements

System Design

Module Design

Construction

Integration

Testing

System

Testing

User Acceptance

Testing

Int. Test

Design

System Test

Design

User Acceptance

Test Design


44/75

Accepts that testing takes roughly 50% of project time

Activities like test planning and test case designs take

place at the beginning of the project well before code

is written, i.e. User Acceptance tests are written once Requirements are approved

System Acceptance tests are written once System Designs are

completed

Integration tests are written once Module Designs are completed

45


45/75

LCH.Clearnet Clearing house for London Stock Exchange

Also acts as counter-party for trade for bonds, swaps, futures, options.

Volume of trades cleared per year: 1.7 billions

Value of transactions per year: 600 trillion (2008) Project: GCS (Generic Clearing System)

To combine separate individual clearing systems into a single generic clearing

system using new technology and tools, i.e.

HP/Unix servers with Oracle 9

BEA WebLogic for business logic and processing

IBM MQ Series for messaging

J2EE, XML for application development

Case tools: Rational SuiteRose for Requirements & UML models, Requisite Pro for

designing use-cases, TestManager for test case management, ClearQuest for defect

management.

46


46/75

Development Process System consists of 4 main system components

Development teams structured to match the system components

Using V-Model, requirements for each component were captured, system

models were designed, interface specification were created and codingstarted.

Total Dev team is around 120 people:

~50 Devs, 15 Testers, 4 Test automation Devs

Plus BAs, System Analysts, DBAs, various managers.

47

System

Interface

Trade

Validation

TradeProcessing

Engine

Risk

Management

Settlement

& Clearing

SystemInterface


47/75

Implication for testing Each system component is to be built with all features fully implemented.

Each component requires its own component testing.

Without having interface to other components, testing each component requires specially

created Test Harnesses

Once each component is completed and tested, they will then be integratedone component at a time.

Integration testing is required between each component.

Once all the components are fully integrated, the entire system will be tested.

System testing is required for the complete system.

Additionally, End-to-End plus performance and stress testing are required.

48

System

Interface

Trade

Validation

TradeProcessing

Engine

Risk

Management

Settlement

& Clearing

SystemInterface


48/75

Test Harnesses

Q: How to test A without B, and vice-versa?

A:Driversto simulate Up-stream interface

Stubsto simulate Down-stream interface

49

Component

A

Component

B


49/75

Testing Component A in isolation 1 Driver: Usually a piece code, or an application, which can

send all possible input data range and data type

2 Stubs: To intercept output data from Module A and, if required,

send confirmation back to Module A.

50

Component

ADriver

Stub

Stub


50/75

Integration Testing To check that when multiple system components are integrated

together, the combined component worked as a single unit.

Understanding the interface specifications between

components and generating test data required to testintegration

Entering data using specially written Drivers and verifying

results via Stubs or other means, e.g. direct SQL calls to DB

tables.

51


51/75

User Acceptance Testing To verify with the users that the system meets their

requirements

Test cases usually prepared by Testers with theinvolvement of End Users

Tests are run to show positive system behaviour as

well as negative system behaviour, e.g. handling of

error conditions

Tests are run by Users with Testers present

52


52/75

Outcome of GCS project Project took too long to implement

After 2.5 years we were still at Integration Testing stage

Requirements constantly changed due togovernment and industry regulations

Changing requirements meant the each system has to be

redesigned (data models, interface specs, functional specs),

recoded and retested

After 4 years, company decided to cancel the projectafter spending US $75 million

53


53/75


54/75

Traditional Iterative/Incremental

V-Model Agile

Waterfall FDD (Feature Driven Development)



Scrum

RUP (Rational Unified Process)


TDD (Test Driven Development)

55


55/75




Definition of TestingDevelopment models & testprocesses Case studies:

Waterfall (functional & non-functional testing)

V-Model (integration and UAT)

Agile/Iterative (SCRUM)


Q&A

56


56/75

Marshall Wace Asset Management Hedge funds investment company

Buys/Sells stocks, currencies and futures

Total assets managed: ~ US $15 Billion Regularly gives a return of > 10% interest

57


57/75

Project summary To build a new trading system to replace

existing one using new technologies C#, .Net 3, WCF, WGF, XOML

New system must be flexible to react tochanging trading laws and regulations

System is critical to the business It must guarantee 100% reliability

Minimum disruption to the business when newsystem is deployed New system must be introduced slowly and co-exist with the

current system.

58


58/75

59

Trade idea ValidationOrder

Execution

RestrictionsSettlementTrade Book

Existing system


59/75

SCRUM

60


60/75

SCRUM Summary

Each iteration is called a Sprint lasting 2-4 weeks.

Requirements are added to Product Backlog

At the start of the sprint, requirements are picked from the

Backlog to be developed. Usually 5-6 but no more than 10 requirements per sprint.

The sprint team is no bigger than 5-6 people, consisting of

developers and 1 or 2 testers.

Daily meetings for 15 mins. What youve done yesterday

What youre going to do today

Any problems thats stopping you from doing your job.

61


61/75

Key principles Deliver working software incrementally

Fast ROI (Return on Investment)

Managing changing requirement is part of theprocess

Each increment is time-boxed

Within each increment, all steps of software

lifecycle takes place, i.e. analysis, design, dev &test (mini-waterfall), severaltimes

62


62/75

Testing in SCRUM As features/functions are added incrementally

over many Sprints, testing has to ensure that:-1. All new features work correctly

2. All existing features do not break.

How? Functional tests: check new features for bugs

Regression tests: check existing features continue to work

Exploratory tests: uncovers unexpected behaviour

63

Sprint 1 Sprint 2 Sprint 3


63/75

How does SCRUM cope with change? New requirements are added

Existing requirements removed or reprioritised

Each new Sprint only deals with the most up todate requirements

Urgent requirements are brought into current

Sprint by removing planned requirement(s)

64


64/75

Testing in SCRUM Within the Sprint

Specifications are usually not written down

Communication with Devs and Users is key Functional testing of features as soon as implemented

User Acceptance testing before delivery to user

65


65/75

Testing in SCRUM Across several Sprints

Regression tests are run at the beginning of new Sprint

In simplest terms, regression means re-running of existing

tests from previous Sprints, to make sure all existing

features continue to work after new code has been added.

But

Sprint 1 = Functional tests 1 (Ft1)

Regression in Sprint 2 = Ft1 Regression in Sprint 3 = Ft2 + Ft1

No of test cases grow very quickly and can become

unmanageable

66


66/75

Regression testing

67

0

20

40

60

80

100

120

140

160

180

200

1 2 3 4 5

Tests

Cumulative


67/75

Managing regression tests Selectively re-run only those tests which are

relevant, i.e. if modules A, D and E are affected in

current sprint, rerun tests from modules A, D andE from previous sprints

Automated testing

Custom tool

Commercial tool (Robot, QTP) Open source tools (Selenium, Ruby, Watir)

68


68/75

Selected low-volume, low-risk trades

Trade is replicated in both new and old system and outputs

compared

69

Trade ideaTrade

ValidationOrder

Execution

RestrictionsSettlementTrade book

Trade validation Order Execution

RestrictionsSettlementTrade book


69/75

Project Summary Project timescale is 2 years but users were able

to start using the system within 3 months

Deployment was seamless to users Able to change the system when Credit Crunch

happened, i.e.

Disabling short-selling of stocks

Removing Bear Sterns related processes Implementing new changes in law in Hedge Funds

trading

70


70/75




Definition of Testing

Development models & test processes Case studies:

Waterfall (functional & non-functional testing)

V-Model (integration and UAT) Agile/Iterative (SCRUM)


Q&A

71


71/75

How many testers do you need?

Difference between Developer & Tester

72


72/75

How many testers do you need? Dev/Test Ratio

Technology consultancies (Accenture, IBM, Logica) =

6-7:1 Global software company, e.g. Microsoft = 1:1

NASA Space shuttle project = 1:3

NASA Space shuttle life support systems = 1:7

Most IT firms = 4:1

73


73/75

74

This code hasn't yet been tested.

I dont know if it has any bugs.

I only changed one line of code.

Anything is possible, given enough

time.

Let me fix this one final bug, and we

can ship tomorrow.

Its an "undocumented feature"

I like to build things.

This code hasn't yet been tested.

I dont know if its going to work.

The whole system has to be retested.

Everything has flaws, and given enough

time I can prove it.

If you fix this bug, youll add 2 new bugs.

Its a bug.

I like to break things.

Developer speaks The way Tester thinks


74/75

75

Is it half full or half empty?

Developer (an optimist):

The glass is half full.

Tester (a pessimist):The glass is half empty.

Really good Tester (a realist):

The glass is twice as big as it needs

to be.


75/75

Books worth reading Testing Computer SoftwareKaner, Falk & Nguyen, 1999, ISBN: 0471358460

Managing the Testing ProcessBlack, 2009, ISBN: 0470404159

How to Break Web Software: Functional and Security Testing of Web Applications and Web Services Andrews & Whitaker,2006, ISBN: 0321369440

Professional Associations BCS SIGIST (British Computer SocietySpecial Interest Group in Software Testing):

http://www.bcs.org/server.php?show=nav.9262

Software Testing Institute: http://www.softwaretestinginstitute.com/

International Institute for Software Testing (IIST): http://www.testinginstitute.com/

Obtaining Professional qualifications BCS offers ISEB 3 types of Certification examinations in Software Testing: Foundation, Intermediate and Practitioner.

http://www.bcs.org/server.php?show=nav.10920

IIST offers 2 types Certifications on software testing: Certified Software Test Professional and Certified Test Manager.http://www.testinginstitute.com/certification.php

Online resources Agile manifesto : http://agilemanifesto.org/

Scrum Alliance: http://www.scrumalliance.org/

James Bach (proponent of Agile testing, Exploratory testing) : http://www.satisfice.com

Open source test tools: http://www.opensourcetesting.org

Most popular test management tool is TestLink: http://www.teamst.org/

Most popular online bug reporting tool is Bugzilla: https://bugzilla.mozilla.org/

Selenium (automated test tool for testing websites on Firefox): http://seleniumhq.org/

Free Tester magazine (available every 3 months) from SIGIST: http://www.bcs.org/server.php?show=nav.9265
http://www.bcs.org/server.php?show=nav.9262http://www.bcs.org/server.php?show=nav.9262http://www.bcs.org/server.php?show=nav.9262http://www.bcs.org/server.php?show=nav.9262http://www.softwaretestinginstitute.com/http://www.testinginstitute.com/http://www.bcs.org/server.php?show=nav.10920http://www.testinginstitute.com/certification.phphttp://www.testinginstitute.com/certification.phphttp://agilemanifesto.org/http://www.scrumalliance.org/http://www.satisfice.com/http://www.opensourcetesting.org/http://www.teamst.org/https://bugzilla.mozilla.org/http://seleniumhq.org/http://seleniumhq.org/https://bugzilla.mozilla.org/http://www.teamst.org/http://www.opensourcetesting.org/http://www.satisfice.com/http://www.scrumalliance.org/http://agilemanifesto.org/http://www.testinginstitute.com/certification.phphttp://www.bcs.org/server.php?show=nav.10920http://www.testinginstitute.com/http://www.softwaretestinginstitute.com/http://www.bcs.org/server.php?show=nav.9262

delivering quality software through formal testing

Documents