Past, present, future …

1

Tim Koomen Eurostar 2008

Journey from different viewpoints

� What to test

� People

� Techniques

� Tools

� Process� Process

� Organisation

What to test …

What to test …

� Past: testing of functionality

� Present: bias of testing more and more shifts toward non-functional aspects like performance, suitability, usability and security

What to test …

� The enormous amount of what can be tested has forced testers to choose very carefully what (not) to test and how thoroughly

� These choices are usually based on perceived risks

Risk techniques: TMap, RRBT, PRISMA, � Risk techniques: TMap, RRBT, PRISMA, SmarTEST, Failure mode and effects analysis (FMEA)

What to test: future?

� New IT developments (architecture, services, integration issues, standard solutions)

� Growing importance of non-functionals

� Art of testing remains: “choosing what to test and how thoroughly”“choosing what to test and how thoroughly”

People

� Past:

� from ad-hoc testers to …

� professional testers to …

� Present:

� SIGIST’s and � SIGIST’s and

� certified testers (like ISEB, ISTQB, TMap)

� non-functional test experts

Discussions on certificationPro:

� Knowledge of terminology

� Proof of test expertise

� Entry-criterion for professional tester

� Part of the tester’s career path

� Recognition of software testing as a professional specialisation within IT

� Commercial necessity when more than 1 customer

Con:

� Testing as a profession is still immature, therefore certification programs serve as an obstacle to innovation and learning

� Test certification is very light compared to other certified professionals like java programmer (18 months), software architect (15 months), this does not help recognition of our profession

� Commercially misused

� Having a test performed by a certified tester does not guarantee quality of work

� Having a test performed by a certified tester does not guarantee quality of work

� A good tester has knowledge / skills in the following areas:� IT in general,

� testing,

� analytical,

� soft skills,

� business knowledge

Only testing knowledge is part of the certification …

� Certification insufficiently covers non-functional testers

� Once certified, always certified

Sources: I. Evans, M. Bolton, K. Hanselman, R. Kochuyt, R. Marselis

People: future?

� Maybe a formal testing discipline?

� Professionals taking personal responsibility for their work

� Certification level(s) with entry-criteria and continuous professional development

� Governing body� Governing body

� Ethical rules and professional standards

� Disciplinary action for not observing standards or bad professional work

(I. Evans)

(however: far away future …)

� Business knowledge vs. technical knowledge?

� Soft skills certification?

Techniques

� Techniques: early!

� Well-known techniques:

� equivalence partitioning,

� boundary value analysis,

� domain testing, domain testing,

� graph testing,

� decision/condition testing,

� …

� Newer techniques?

Pairwise testing

� Buying a train ticket from the vending machine Parameters:

The aim of pairwise testing is to test all the possibilities of any combination of 2 factors

Parameters:� Trip : one-way; return

� Class : 1st , 2nd

� Payment : credit card; debit card; cash

Pairwise testing

Trip Class Payment

1 One-way 1st Credit card

2 One-way 1st Cash

3 One-way 2nd Debit card

4 Return 1st Debit card4 Return 1st Debit card

5 Return 2nd Credit card

6 Return 2nd Cash

Pairwise testing

Trip Class Payment

1 One-way 1st Credit card

2 One-way 1st Cash

3 One-way 2nd Debit card

4 Return 1st Debit card4 Return 1st Debit card

5 Return 2nd Credit card

6 Return 2nd Cash

Pairwise testing

Trip Class Payment

1 One-way 1st Credit card

2 One-way 1st Cash

3 One-way 2nd Debit card

4 Return 1st Debit card4 Return 1st Debit card

5 Return 2nd Credit card

6 Return 2nd Cash

Pairwise testing

Trip Class Payment

1 One-way 1st Credit card

2 One-way 1st Cash

3 One-way 2nd Debit card

4 Return 1st Debit card4 Return 1st Debit card

5 Return 2nd Credit card

6 Return 2nd Cash

Some more parameters …

� Parameters:� Trip: one-way, return

� Class: 1st , 2nd

� Payment: credit card,debit card, cash

� Discount: yes, no

� Place: lower, upper, standing, roof� Place: lower, upper, standing, roof

� Train type: ICE, slow train, dieselloc, intercity, steam

� Age:

Trip Class Payment

Discou

nt Place Train type Age Catering Color

1 one-way 1st debit card yes upper steam 65+ coffee red

2 return 2nd credit card no standing intercity

Exploratory testing

� … a technique?

Exploratory testing is the simultaneous learning, designing and executing of tests, in other words every form of testing in which the tester designs his tests during the test execution and the

information obtained is reused to design new and improved test cases.(James Bach, 2002)

� … a technique?

� keyword: “… the tester designs his tests…”

Model-based testing

Model-based testing is software testing in which test cases are derived inwhole or in part from a model that describes some (usually functional)

aspects of the system under test (Wikipedia)

Modelrequirements

functional

SystemExecutable

tests

Test design

test tool

Keywords:Automated generation and execution of test cases

functional design

technical design

Techniques … �

� But … which ones do we really use?

Classic: triangle case …

� Input: 3 integer values

� Output: message saying the values represent a scalene, isosceles or equilateral triangle

� How many test cases would be adequate…?

Techniques: future …

Room for more?

� … for non-functionals

� … for interfaces, integration

Tools

Tools for planning and controlling the test

� Testware management tool

� Defect management tool

� Planning and progress monitoring tool

� Workflow tool

Tools for designing the test

� Test data tool

� Test design tool

text based

GUI’s� Test design tool

� Model-based testing tool.

Tools for executing the test

� Automated test execution tool

� Performance, load and stress test tool

� Monitoring tool

� Code coverage tool

� Comparator

� Database manipulation tool

� Simulator

� Stubs and drivers

web-based

service-based

Automated test execution tool

• GUI• object recognition• action words / data-driven• framework / testsuite / components

Tools: future …

� Keeping up with IT developments

� Non-functionals

� Test design?

Process

� Standards� IEEE829 (software test documentation)� IEEE1008 (software unit testing)� BS7925-1 (Software Testing – Vocabulary)� BS-7925-2 (Standard for Software Component Testing) � IEEE 1028 (Reviews and Inspections)� ISO 9126 (Software Quality) � ISO 9126 (Software Quality) � ISO/IEC 25000:2005 (SQuaRE), supersedes ISO9126. � + specials (FDA, AECL)

� Books� 80’s: books on test design techniques (Beizer, Myers), � 90’s: several books on the process of testing (Black, Kaner, Kit, Pol)

� 00’s : more emphasis on agile, context-driven, flexible, adaptive aspects

Process

� Methods / process descriptions

� 90’s: structured test methods (STEP, TMap, Testframe/RRBT) emerged

� 00’s: many organisations invented / implemented their own variant of such a methodmethod

� Process Improvement

� 90’/00’s: specific test process improvement models (TPI, TMM, TMMi, V2M2)

� Certification schemes (ISEB, ISTQB, TMap)

� aimed at people, best practice processes

Process: future

Agile

Integrated in Division between

Formal

Integrated indevelopment best practices

Division between IT and Business testing

Organisation

Development Implementation

Development Test Implementation

Development Test ImplementationDevelopment Test Implementation

Development Test Implementation

Test expertise centre

Organisation

Development Test ImplementationDevelopment Test Implementation

Development Test Implementation

Test department(resource pool)

Test factory(result)

or

Development & test

(resource pool) (result)or

Test Policy

Organisation

Development Test ImplementationDevelopment Test Implementation

Development Test Implementation

Test factory(result)

Supplier organisation

Organisation: future

� Certification of services?

� Certification of test services?

� TaaS?

Past, present, future

� Many things

� “nothing’s changed, really…”

� Biggest changes

� people, organisation, non-functionals, tools

� Achievement

testing keeps up with IT developments� testing keeps up with IT developments

� Prediction

� higher variety … and long way to go

Software complexity (and therefore that of bugs) grows to the limits of our ability to manage that complexity

(Boris Beizer’s law of the complexity barrier)

E. info@timkoomen.nl

M. +31 (0)6 34139260

I. www.timkoomen.nlI. www.timkoomen.nl

Copyright Tim Koomen Testmanagement en -advies