situational testing
TRANSCRIPT
Situational TestingIt’s All In The Mix
Jan Jaap CannegieterRoger Wouterse
Bart FesslSven van Galen
SYSQA B.V.
February 2014
Is T
his
Bo
ok R
ight
For
Me?
INTRODUCTORY
Introductory content is for software testing professionals who are relatively new to the subject matter of the ebook. Introductory ebooks typically feature an overview of an aspect of testing or guidance on understanding the fundamentals of the subject at hand.
INTERMEDIATE
Intermediate ebooks are for software testers who are already familiar with the subject matter of the eBook but have limited hands-on experience in this area. Intermediate level usually focuses on working examples of a concept or relaying experiences of applying the approach in a particular context.
ADVANCED
Advanced ebook content is for software testers who are, or intend to become, experts on the subject matter. These ebooks look at more advanced features of an aspect of software testing and help the reader to develop a more complete understanding of the subject.
This eBook isintended for an
INTERMEDIATEaudience level(s)
Every organisation, every system and every project is different, so every test project requires a unique approach. Situational testing offers various forms of testing and allows you to complete your testing projects flexibly, at the lowest possible cost and in the shortest possible time. Situational testing encourages you to approach testing in a flexible, structured and pragmatic fashion. Situational testing always delivers optimal results. Better results than using a single specific approach:‘it’s all in the mix!’
This publication identifies the essential features of situational testing, explains the various forms of testing that can be used in situational testing and explores situational testing in practice.
© 2013 SYSQA B.V.
All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher. For information contact:
SYSQA B.V.Kabelstraat 51322 AD AlmereThe Netherlandswww.sysqa.com
Abs
trac
t
Jan Jaap Cannegieter has 20 years of experience in requirements engineering, quality assurance, testing, process improvement and project management. Jan Jaap is Vice President of SYSQA B.V., a company of 180 employees specialized in requirements, software testing, quality assurance and software process improvement. Within SYSQA Jan Jaap is responsible for knowledge management, coaching and product development. Jan Jaap is the author of several articles and nine well sold books in the Netherlands and he is regularly (keynote) speaker and workshop leader at several international conferences.
Roger Wouterse is Product Manager at SYSQA. Together with his work for clients, Roger is responsible for product development and coaching and training for both SYSQA and her clients. Having his main focus on requirements and testing, Roger teaches several courses on requirements (e.g. IREB), Situational testing and scrum.
Bart Fessl, Product Manager at SYSQA, is responsible for development of new and continuation of current services regarding testing and coaching and training (both internal and for clients) regarding testing. Having his main focus on testing and test maturity & process improvement, Bart teaches several courses on testing, Situational testing and TMMi.
Sven van Galen is Manager ICT Quality Assurance and member of the Management Team at SYSQA B.V. He is responsible for the development, deployment and evaluation of a team up to 30 employees, mostly testprofessionals. Sven advises (potential) clients on QA in ICT development and gives trainings about QA in ICT to (new) SYSQA professionals. With his team he supports client organizations on requirements engineering, software testing, quality management and process improvement. He has been responsible for several test projects and testprocess improvements.
Bio
grap
hies
Tabl
e of
Con
tent
s 1. Foreword 6
1.1 Why Perform Situational Testing? 6
1.2 About This Publication 6
1.3 Acknowledgments 7
2. The Essence Of Situational Testing 8
2.1 Why we test: value-based testing 8
2.2 How we test: scripted and non-scripted testing 10
2.3 Misunderstandings 11
2.4 The applicability of situational testing in Agile. 12
2.5 Situational testing: it’s all in the mix! 13
3. Forms Of Situational Testing 15
3.1 Factory-based testing 15
3.2 Global scripting 16
3.3 Session-based testing 16
3.4 Bug-hunts 17
3.5 The test tour 18
3.6 Freestyle exploratory testing 19
3.7 Situational testing: tailor made 20
4. Situational Testing In Practice 21
4.1 Test strategy 21
4.2 Execution of the test project 23
5. Conclusion 25
1.1 WhY PERfORM SITUATIONAl TESTINg?
Every testing project is different because systems are different, projects are approached differently and organisations differ in their expectations of a new system or release. Each testing project has its own specific objectives meaning that there is no single approach to testing which is suitable for every situation. In order to decide on the best approach each situation must be looked at individually. This is called situational testing.
The basic principle of situational testing is that the test approach is determined by the characteristics of the system, the project and the organisation’s expectations. A combination of test objectives, test formats and test techniques is selected in order to provide a well-substantiated insight into the quality of a system, at the lowest possible cost and in the shortest possible time. In practice, this represents the most common objective of testing.
Situational testing provides sound guidelines and a useful framework for the structure and execution of a test project. It should always be considered however that exceptions are possible. The situation determines the objectives andthe approach!
Situational testing is in keeping with the LEAN philosophy. LEAN is a management philosophy, focussed on achieving the most value possible by eliminating waste. For example, non-essential documentation is not generated and unnecessary activities are not carried out in situational testing. Only the documentation that is really necessary is produced. In practice, we do see that the emphasis in many testing projects is on generating documentation rather than on carrying out tests (see Figure 1.)
Figure 1. The ratio of documentation to testing in many testing projects.
1.2 AbOUT ThIS PUblICATION
This publication briefly and concisely introduces situational testing. It is assumed that the reader has a certain level of expertise in IT, systems developmentand testing.
1.
Fore
wor
d
DevelopingDocumentation
Testing
Section 2 describes the essence of situational testing, Section 3 looks at the forms of testing that can be used for situational testing purposes, and Section 4 explains how situational testing is approached in practice.
Practical Experience
In order to make this publication as practically useful as possible we have included several examples of problems and opportunities encountered by SYSQA staff in the field. You may recognise some of the situations described. In some cases we have also made recommendations.
This booklet is a SYSQA B.V. publication. SYSQA staff contributes towards the success of IT projects. If you recognise any of the problems below, SYSQA can help you make your projects more successful in future.
> Requirements for the project are incomplete, incorrect, not measurable or completely absent.
> There is a lack of insight into quality during the project.
> Systems go into production even though there are still significant defects.
> Test projects take up an unnecessary amount of time or costs are excessive.
> The project objectives are not achieved.
If your projects are unsuccessful and you do not know why, based on its wealth of knowledge and experience SYSQA can help you find the underlying cause and provide advice and recommendations. Please do not hesitate to contact us to set up a noncommittal meeting with one of our account managers.
1.3 ACkNOWlEDgMENTS
SYSQA’s views on testing are reflected by the approach in this publication. These views are based on a number of sources: day-to-day practice among our clients, our own ideas on testing and numerous books, blogs, presentations, coursesand workshops.
As some of our inspiration has come from other people (that does not necessarily mean they support or endorse this publication), we think it only right to mention them. Direct links to work by third parties are provided in the text, but in many cases there are no such links to our sources of inspiration, nevertheless we do wish to give the people concerned the credit they deserve (in alphabetical order): Gojko Adzic, Henrik Andersson, James Bach, Jon Bach, Sigurdur Birgisson, Michael Bolton, Lisa Crispin, Janet Gregory, Elisabeth Hendrickson, Cem Kaner, Michael Kelly, Lynn McKee, Gitte Ottosen, Huib Schoots and James Whittaker.
This publication was also made possible by contributions from the following members of SYSQA staff (in alphabetical order): Jan Jaap Cannegieter, Bart Fessl, Sven van Galen, Mariëlla van Houte, Teun Klos, Robin Wijngaarden and Roger Wouterse.
There are two issues to situational testing: why do we test, and how do we test? We aim to examine these two questions in this section. We will also be exploring a few common misunderstandings about situational testing and its practical application at Agile.
2.1 WhY WE TEST: VAlUE-bASED TESTINg
Of course testing costs time and money, but we should not lose sight of the fact that it can be far more costly not to test. Nevertheless handling scare resources (time and money) very carefully is obviously crucial.
In order to achieve that it is first necessary to establish what system information stakeholders need. What is the testing actually supposed to show? Situational testing for us is always value-based. This means that testing is used to demonstrate the value of the system for the client and the user organisation.
“Value-based testing” may sound a little abstract. However, the adoption of a classification system means it is practical and applicable in projects. The classification system will vary according to the situation, but one tried and proved way of making value-based testing more specific is to distinguish five different levels, with each level adding value for the business1, see Figure 2.
Figure 2. The five levels of value-based testing.
Level 1 Functionality
At level 1 we determine whether functionality matches requirements and whether it operates flawlessly. This level is also referred to as “specification-based testing”, since it is largely based on specifications drawn up in advance.
Level 2 Performance & Security
Level 2 concerns the system’s reliability in operation; is the system fast enough and does it have adequate security? There is no point in answering these questions until the system does what is required from a functional perspective. Performance and security do however largely determine the value of the system.
1. Based on the keynote presentation “Reinventing software quality” by Gojko Adzic, Agile Testing Days 2013, see Figure 2.
Level 1
Functionality
Level 2
Performance& Security
Level 3
User friendliness
Level 4
Usability
Level 5
Successful
2.
Th
e E
ssen
ce O
f Sit
uati
onal
Tes
ting
Level 3 User FriendlyLevel 3 focuses on the system’s appeal to the user and how easy it is to comprehend, learn and operate. This level is not relevant for every system (for example technical systems without a user interface). This level is often decisive in determining the value of the system for systems used by large groups of inexperienced users.
Level 4 Useable
In level 4 we assess the system’s usability within the organisation. Are staff able to carry out their tasks efficiently and effectively using the system? Does the system fit in with the organisation’s processes?
Level 5 Successful
The highest level of value-based testing deals with the issue of whether the system adds value to the business. It is also determined whether the organisation’s objectives for the system have been achieved. At level 5 we investigate whether the system performs its intended function and achieves the planned objective based on the business case. A much broader look at the system and the project is taken. The known risks are examined but active steps are also taken to examine other risks, which might threaten the value of the system.
It is up to the client to determine which of the above levels of value-based testing are important for a specific project. For this purpose the client will consult with the test manager so that the objectives of the test project are clear. It is important to understand that not every level is relevant for every system or every organisation, but that the idea behind value-based testing is nevertheless always applicable. This also means that the basic principle of situational testing where the situation determines the approach and not vice versa is still followed. The classification set out above must therefore always be used pragmatically, so that the emphasis may vary according to the situation, and some levels may be omitted, or new levels may be added. An analysis based on the ISO 25010 standard may be carried out as an alternative to the above system of categorisation.
Practical Experience
The objective of a recently completed major project for a government agency was to replace paper files with digital versions. Testing revealed that the system satisfied the specifications. Following execution however it emerged that the users were continuing to use the paper files. In this case the problem was not resolved and the objective was not achieved, even though a considerable amount of money was spent.
in order to prevent such situations sound specifications and assessing whether the system meets the specifications is not enough: it is also essential to check that the system achieves the purpose for which it was devised, level 5 of value-based testing.
A supplier or subcontractor in a project setting may well carry out level 1 of value-based testing. In order to maintain insight into the effectiveness of the specification-based testing, the client may introduce a testing system, as it is the client who will be responsible for the quality of the system. The client is always responsible for carrying out the remaining levels of value-based testing, because it has the information required to assess the system from a value point of view, information that is sometimes impossible to document.
2.2 hOW WE TEST: SCRIPTED AND NON-SCRIPTED TESTINg
Now we know what we need to test the next question is how to test the system. In broad terms we can distinguish between scripted and non-scripted testing.
The basic principle of scripted testing is that the tests carried out are based on test scripts developed in advance using testing techniques. Scripted testing uses a chronological approach, ranging from planning, preparation, execution and completion phases. There is a strong emphasis on preparation, and an appreciable amount of the available time will be devoted to this. Work is largely carried out on a planned and methodical basis. The lead time for the total testing process may be extended somewhat, partly because of this focus on preparation. Scripted testing can be used provided a number of preconditions are met:
> The specifications must be available at an early stage.
> The specifications must be complete and accurate.
> The requirements must be stable.
> The testers must start their work at an early stage in the development process.
Non-scripted testing places more demands on the skills and expertise of the tester, with the focus on test execution. It is based on exploratory testing, an approach that relies on the individual abilities and responsibility of the tester. With this approach the tester is simultaneously occupied in learning about the item being tested, the design of test scenarios, the execution of tests and the evaluation of test results. In the case of non-scripted testing the tester does not prepare elaborate scripts in advance, rather he will adapt the priorities based on the outcomes of earlier tests. Non-scripted testing may only be executed where the following peripheral requirements are met:
> The organisation has short lines of communication.
> The organisation and the project are flexible.
> The organisation is decisive.
> Professional testers carry out the tests.
The above descriptions of scripted and non-scripted testing are two extremes. In practice there are various intermediate forms, each with a different ratio between scripted and non-scripted testing. Examples of these intermediate forms include factory-based testing, global scripting, session-based testing, bug-hunts, test tours and freestyle exploratory testing. These different forms of testing and the ratio of scripted to non-scripted testing for each type is shown in Figure 3.
Figure 3. Forms of testing and the ratio of scripted to non-scripted testing2 .We will be looking at the different forms of testing in greater detail in Section 3
2.3 MISUNDERSTANDINgS
There are a number of widespread misunderstandings about the forms of testing discussed above. In this section we will discuss two of these misunderstandings. Firstly one form of testing is not superior to the other. However, it is true to say that a particular type of testing may be more suitable for a particular project or a particular component of the system. Sometimes scripted testing is better and other times non-scripted testing is better. A professional tester will select the test type to suit the situation.
Practical Experience
It is common for testers and test organisations to use the same approach to testing in all circumstances, regardless of the situation. This is often a method they have learned in the past, developed for themselves, prescribed by their organisation, or one adopted by their client. The optimal approach to testing is something that has been exhaustively discussed at numerous conferences. Our view at SYSQA is that the situation determines the value of a specific method, approach or technique.
We are convinced that testers should study and be able to apply all the available test methods, approaches and techniques. They must then be able to assess which approach will be most productive in each particular situation. Applying a method, approach or technique is not a goal in itself. In our view it is necessary to think pragmatically about methods. The objective of testing is to provide an insight into the quality of a system. Moreover, in order to achieve this objective it is permissible to use any available method, approach or technique.
2. Based on the presentation “Telling your exploratory story” by Jon Bach, Agile 2010 conference.
FactoryBased
TestingGlobal
Scripting
Session Based
TestingBug
HuntsTest
Tours
FreestyleExploratory
Testing
Scripted Testing
Non Scripted Testing
The second common misunderstanding is that non-scripted testing is not structured. In fact nothing could be further from the truth. Non-scripted testing is has got less value unless it is handled in a structured manner! Many structural elements, including components of scripted testing, are also used in non-scripted testing. Examples of structural elements familiar from scripted testing that are also applied in non-scripted testing include developing a risk-based test strategy and using test techniques. There are also examples of structural elements, which are used only in scripted or only in non-scripted testing, at least in theory. One example of a structural element, which is theoretically applied only in the case of scripted testing, is the creation of a test scenario. In practice however it is found that this element is often usefully applied in non-scripted testing. An example of a structural element theoretically used only in non-scripted testing is the use of test charters, but here again we find that, in practice, test charters can be very useful in scripted testing. The question whether scripted testing is more structured than non-scripted testing is irrelevant. A relevant question however is whether a particular structural element has added value and should therefore be applied in a particular situation.
2.4 ThE APPlICAbIlITY Of SITUATIONAl TESTINg IN AgIlE.
More and more systems are being developed using Agile development methods. One principle of the Agile approach is that functioning software is more important than exhaustive documentation. This does not necessarily mean that no, or a limited amount of, documentation at all is produced, but as a rule no documentation is available before testing commences. In theory that would make the application of scripted testing impossible in an Agile framework. In practice this is rather different: in some cases global scripts can be drawn up for the tests to be completed, but in general some forms of non-scripted testing are better suited to Agilesystem development.
Despite Agile’s rise, many systems are still developed using the Waterfall methodology, where scripted testing fits in very well. That is not to say however that non-scripted testing cannot be used in these circumstances. The question of what works best is determined in part by the system and the organisation using it, i.e., by the situation.
Practical Experience
Non-scripted testing rose to popularity during the same period as the Agile approach. Many testers are familiar with scripted testing, and attempt to apply scripted testing in Agile projects. This has not met with universal success; often there is a mismatch as Agile is based on different assumptions to the Waterfall methodology.
When introducing experienced testing to an Agile project it is recommended that they should also be trained and coached in the application of non-scripted testing. Implementing the Agile approach also entails major changes for the testers and they should be properly prepared.
2.5 SITUATIONAl TESTINg: IT’S All IN ThE MIx!
On the one hand situational testing is a level of value-based testing and on the other hand a selection of forms of scripted and non-scripted testing. Every test project has its own particular mix. In order to illustrate this three different examples of test projects are described below.
Testing levels 1 to 3 are applied in example 1. It is determined whether functionality is up to the mark, whether performance and security are satisfactory, and whether the system is user-friendly. This is done using global scripting, session-based testing and bug-hunts. This can be shown in tabular form as follows:
Value Based Testing Level
Factory Based Testing
Global Scripting
Session Based Testing
Bug Hunts Test ToursFreestyle
Exploratory
Level 5
Level 4
Level 3
Project 1Level 2
Level 1
Example 2 is far more restricted in scope. Here it is solely determined whether the functionality is good, value-based testing level 1. Only factory-based testing and global scripting are used here. Test project 2 is an example of specification-based testing.
Value Based Testing Level
Factory Based Testing
Global Scripting
Session Based Testing
Bug Hunts Test ToursFreestyle
Exploratory
Level 5
Level 4
Level 3
Level 2
Level 1 Project 2
Example 3 involves two test teams, for example a system test team and an acceptance test team. In this example the system test team (STT) is responsible for value-based testing levels 1 and 2. The system test team will use session-based testing for this. This is shown in light grey in the table below.
The acceptance test team (ATT) is responsible for value-based testing levels 3, 4 and 5, using a combination of bug-hunts and freestyle exploratory testing. This is shown in grey in the table below.
Value Based Testing Level
Factory Based Testing
Global Scripting
Session Based Testing
Bug Hunts Test ToursFreestyle
Exploratory
Level 5 Project 3ATT
Project 3ATTLevel 4
Level 3
Level 2 Project 3STTLevel 1
The final example provides a simple demonstration of the allocation of roles between a system test team and an acceptance test team.
The projects above are merely examples. For your own project the objective of the test project must first be determined, and possibly specified in the appropriate level of value-based testing. Then the form of test you will use to do this can bedecided on.
3.
Form
s O
f Sit
uati
onal
Tes
ting
Planning & Strategy Specification Execution Completion
Test Coordination
In this section we briefly explain the different forms of testing which can be used for situational testing
3.1 fACTORY-bASED TESTINg
In this purest form of scripted testing a great deal of time is devoted to the preparations for test execution. Before a start is made on executing the test, the tester produces detailed test scripts based on a strategy and a time schedule. The stages of factory-based testing are set out in Figure 4.
Figure 4. Stages in factory-based testing.
In theory the test scripts used by testers are detailed enough so that someone else could carry out the tests. The tests are carried out by executing the scripts. Once the scripts have been completed and there are no unfixed bugs left, testing is complete. Scripts can be reused, for example following any modifications to the system. The tester will update the test scripts during the completion process and then archive them. In the event of modifications to the system the scripts must be adapted as necessary.
Factory-based testing may be applied where complex calculations are involved or where system processes are always carried out in the same sequence. Factory-based testing is also appropriate when testers lacking knowledge of the system must carry out the tests. Factory-based testing can only be applied where the preconditions for scripted testing set out in Section 2.2 are met.
Practical Experience
One of the SYSQA’s clients showed an interest in situational testing. The testers in this organisation had been using factory-based testing for several years, but now they wanted to know if things could be done differently. After a one day workshop the participants were a little disappointed. A joint analysis revealed that the overwhelming majority of their systems (estimated at over 80%) were so complex and so difficult to test, and the documentation was so good and stable that factory-based testing was the best suitable form of testing.
The organisation then for the most part continued to follow the previous well-trodden path. While the initial reaction was one of disappointment, in the end the testers were pleased with the confirmation that they were using the form of testing most appropriate to their organisation and situation.
3.2 glObAl SCRIPTINg
With global scripting the testers create scripts, which are considerably less detailed than those used with factory-based test execution starts. Here too the assumption is that the testers prepare the scripts before the testing starts. The scripts can be seen as a checklist, which a tester uses when carrying out the tests. Since the scripts are less detailed it takes less time to prepare them and they require less maintenance, but testers familiar with the system and the scripts can only carry out the tests. In the case of global scripting the test is generally prepared and carried out by the same person.
Global scripting is suitable when there is a permanent test team responsible for the entire test project. As with factory-based testing there is no reason to allow the tester to determine what he will test during the testing process. The assumption again, as in factory-based testing, is that the execution of the tests consists of executing the scripts.
Practical Experience
In a major government organisation the majority of testing time was being spent on creating and maintaining test scripts. However during the testing execution, which was always carried out under extreme time pressure, the testers hardly used the scripts except occasionally as an aide memoire.
Since the majority of the testers in this organisation work for an extended period on a single project it was decided to abandon the existing approach and to work with far more global scripts instead. This not only saved a great deal of time, the testers also found it a far more pleasurable way to work.
3.3 SESSION-bASED TESTINg
Session-based testing1 includes elements of both scripted and non-scripted testing. This type of testing is carried out in sessions lasting from 60 to 120 minutes. Each test is based on a test charter. The following matters are included in the test charter:
> The mission for the session (what is to be demonstrated in this session).
> The related requirements.
> The test points for the session.
> Information about the execution of the session, such as bugs and new test points.
A test point is an aspect of the system falling within the mission for the session and which is to be tested during the session. The tester will adopt the usual test techniques during the completion of the tests. However the test scenarios are not written out in detail in the form of test scripts.
1. Based on the article “Session-Based Test Management” by Jonathan Bach, published by Software Testing and Quality Engineer-ing magazine, November 2000 and the tutorial “Session-Based Test Management” by Michael Kelly, Eurostar 2012
The testers create the test charters prior to the session. In that sense there are preparations for the testing process. Test charters are even less detailed than global scripts, which means that the preparation time involved is much shorter. New test points and test charters will also emerge during the testing process. The priorities can be adjusted based on the testing process. With session-based testing the testing process is more flexible than in the case of global scripting or factory-based testing.
Session-based testing can be applied in virtually any situation. A precondition however is that the person executing the test must be familiar with the system or similar systems as well as understanding the charters. However, this does not imply that users should perform session-based testing. A session-based tester must also be very familiar with test techniques and able to apply them.
Practical Experience
Session-based testing has been found to combine very well with the Agile approach, where a limited amount of documentation is generated before test preparation starts. This means that factory-based testing is impossible. However session-based testing is ideal in this situation as establishing the test objectives at the start of the sprint is east. The charter can be created and test points determined during the sprint. At the end of the spring all team members can perform the test session based on the charters. This approach allows structured, flexible and manageable testing in an Agile project.
3.4 bUg-hUNTS
Bug-hunts bring all interested parties in the project together in a positive, energetic workshop-like activity in which they test the system. Factory-based testing assumes that the testers carry out the tests individually. For bug-hunts a test is carried out by a number of staff members in sessions. These are usually longer than session-based tests, and 3 to 4 hour sessions (including evaluation) are not unusual. A bug-hunt brings different members of staff together, not only testers but also programmers, users and project leaders. In short, anyone interested or involved in a project can participate in a bug-hunt.
Pairs are generally formed, preferably consisting of a tester and a non-tester. A frequent combination is an experienced and a less experienced member of staff. Each pair is given a test mission, scope and some test points. Bug-hunts also work well in combination with test charters. The pairs then go off on the hunt for errors (bugs) within their mission and scope. The pairs are able to work extremely flexibly provided they accomplish their mission for the test and complete any test points they have been given.
Bug-hunts can be a solution if there are not enough testers available for a project. In addition, bug-hunts can also be a lot of fun! A competitive element can be introduced (who can find the most bugs) with prizes like candy bars or sweets.
Bug-hunts are less suitable for testing highly complex systems or system components, because there is no certainty that all functional paths will be covered. Bug-hunts can however deliver very good outcomes in practice compared to more scripted forms of testing and systems with high levels of user interaction lend themselves very well to bug-hunts.
Practical Experience
During a relatively small project a SYSQA consultant intensively used bug-hunts. This consultant was a requirements analyst, QA officer and tester. Once a week the sofware house to which the execution was outsourced delivered system components, which then had to be tested as soon as possible. As the SYSQA consultant did not have enough time for this he organised bug-hunts, referred to as ‘test sessions’ by the organisation in question. Various staff members participated regularly. This meant that participants were actively involved in the project, tests were completed rapidly, everything was kept manageable, replicable and measurable thanks to the use of charters, an last but not least, the sessions were actually fun!
3.5 ThE TEST TOUR
A test tour 2 is a form of testing where a professional tester or a pair of testers goes in search of specific errors. The type of tour determines the type of error being sought. Examples of test tours include the feature tour, the claims tour, the data tour, the asocial tour, the previous version tour and the bad parts tour.
In the feature tour all features that the system possesses according to the documentation are tested. The claims tour assesses whether claims made, for example in brochures, are lived up to. Is the system really as user-friendly or as fast as claimed? The data tour examines the life cycle of data passing through the system. In the asocial tour the tester looks at how the system deals with unusual input. Examples could include large number of keystrokes, switching off power, inputting nonsensical data and attempting to relocate elements within a screen. In the previous version tour the tester checks that bugs removed from a previous version do not recur in a later version. The bad parts tour primarily tests those elements of a system, which previously contained lots of bugs.
Test tours provide the testing process with focus, and little preparation is required, at least in theory. Implementing a range of tours allows more unique bugs to be found. Test tours can also be combined with session-based testing or bug-hunts, and they can also be used to supplement factory-based testing. The assumption with test tours is that, as with session-based testing and bug-hunts, future tests will be determined in part by experiences during past tests. If a large numbers of bugs are found in a particular system element using the data tour for example, it is advisable to test further elements of the system using the data tour.
2. Based on the book “Exploratory Software Testing” by James Whittaker, Addison Wesley, 2010
Practical Experience
One of SYSQA’s clients had developed and implemented a desktop virtualisation platform; highly complex, technical software whose functionality could be tested within a couple of hours using a standard test set. This is always done when software is rolled out for a new client but the standard test set is not adequate if a new feature is being added. In that case the testers use test tours. The bad parts tour is always used to perform additional testing on parts of the software that contained a large number of bigs on precious occasions. The ‘previous version tour’ is also frequently used, with checks carried out to see that errors removed from a previous version have not recurred. Time permitting an asocial tour is also carried out: this involves checking whether the system can be brought down by performing unusual actions.
Such use of tours with new versions of an existing program allowed this organisation to identify significant numbers of unforeseen bugs. The tours allowed the testers to work more creatively and to look at the system from a different angle.
3.6 fREESTYlE ExPlORATORY TESTINg
The purest form of non-scripted testing is freestyle exploratory testing. Exploratory testing 3 is an approach to testing based on the personal capabilities and responsibility of the tester. The tester continually drives up the added value by learning to understand the system, by creating a test design, by executing the tests and by evaluating the results, carrying out all these activities in parallel. These would all be separate, successive activities in scripted testing.
Freestyle exploratory testing differs considerably to unstructured testing or error guessing; it is highly structured, with the tester’s priorities based on the product risks identified in a product risk analysis and the results of previous tests. The tester also uses test techniques in order to ensure exhaustiveness, as well as determining and reporting on the coverage and depth of the testing as the process progresses. If particular elements have been insufficiently tested, the tester will carry out deeper and more extensive testing in that area. Another aspect is the detailed recording of the testing process, often with the use of tools to allow accountability in relation to what has been tested. Where desired the tester may also set up a regression test set during testing. Freestyle exploratory testing can be used with any system. In practice it is often used alongside other forms of test.
3. Based on the book “Explore It!” by Elisabeth Hendrickson, Pragmatic Bookshelf, 2013
Practical Experience
An organisation developing software for non-technical users had been using factory-based testing for several years. Despite the extensive test scripts, mainly produced automatically, the users ran into serious problems during production. The organisation then decided to bring in an additional tester using freestyle exploratory testing alongside factory-based testing. On each test round this tester discovered bugs which factory-based testing had missed, clearly demonstrating the added value of freestyle exploratory testing for this organisation.
3.7 SITUATIONAl TESTINg: TAIlOR MADE
We have not covered every possible form of testing in this section, and other variants exist in addition to those described above. If you have successfully been using a different approach within your organisation, then we suggest you continue to do so, and please do tell us about it!
The types of testing explained in this section can be adapted and anyone can create their own variants, they are not set in stone. The objective of situational testing is however firmly established: providing insight into the quality of a system at the lowest possible cost and in the shortest possible time.
Practical Experience
Many testers seek to strike a balance between freedom and structure and some organisations devise their own variants. During a workshop on situational testing one of the participants spoke up during the explanation of test charters to say that she had already being doing that for a long time. She showed one of her test charters and it did indeed appear to be suitable for session-based testing (they were not familiar with this terminology, but broadly speaking that was the form of testing that they were using). The consultant confirmed for her that situational testing is largely pragmatic. It is all about finding out what is most useful for specific organisation or a specific project. Most elements of situational testing have developed out of practice and have been reinvented in different places.
This section looks at how a test project can be carried out using situational testing. We want to emphasise that this is only an example: every test project is unique and requires a unique approach!
4.1 TEST STRATEgY
In principle every situational testing project starts with a test strategy. The assumption with situational testing is that the most risky functional elements and non-functional aspects of the system will be most extensively tested. Developing a test strategy begins with the identification of the functional and non-functional aspects to be tested. Functional aspects could be system components or a set of associated requirements for example. Examples of non-functional aspects are time behaviour, user-friendliness and security. These functional and non-functional aspects will be referred to as “test units”.
Technical Risk
The technical and operational risk of each test unit will be determined. The technical risk is the probability of likelyhood of an error occurring. The more complex the test unit, the greater the probability of its containing errors. Matters beyond the test unit itself, such as the familiarity of the programmer with the development language and the stability of the requirements, may influence the technical risk.
Operational Risk
Next the operational risk is determined: how significant is the test unit for the organisation? In the case of functional test units the important factors include the frequency of use of the function, the number of users and the consequences if the function fails to perform. The combination of the technical risk and the operational risk determines the significance of the function. An example is shown in Figure 5, where T stands for “test unit”.
Figure 5. Select factors analysed as to technical and operational risk.
Tech
nic
al R
isk
Operational Risk
HighLow
High T2
T5
T4T8
T9 T10 T6
T7T1
T3
4.
Situ
atio
nal T
esti
ng In
Pra
ctic
e
Test Unit
The next step is to determine the form of testing to be used to test each test unit. Take for example test unit 2 in Figure 6. From a technical perspective this is a highly complex test unit. If we now assume that closer study will reveal that there is only a single path through this test unit and many dependencies, then it is probably wise to test it with factory-based testing.
Test unit 7 is certainly simple from a technical perspective, but now let us assume that it includes a user process consisting of several steps. This would be a good reason to create detailed scripts in advance.
Test units T1, T3 and T9 are not too problematic and will only require testing once or perhaps twice. It may suffice to test these once using the feature tour and the data tour, without preparation or extensive documentation.
Test units T4, T5, T6 and T10 will require more in depth and probably also more frequent testing. It is also important to carefully record the outcomes, with session-based testing for example. The first version of the test strategy is included inFigure 6.
Figure 6. Sample test strategy based on situational testing.
Practical Experience
People are not always immediately convinced of the added value of testing. ‘Shouldn’t the programmers just do their job properly?’
It may be useful to invite someone like this to a test strategy session. Not everyone will attend, but those who do should be actively encouraged to identify risks. They can then be asked if they are interested in reducing these risks. This often results in the realisation that testing is in fact a risk reduction measure although it may be perfectly legitimate to accept the risk and not test. Nevertheless we find that after such a session participants generally decide not to ignore the risk, but to carry out the tests.
Tech
nic
al R
isk
Operational Risk
HighLow
High T2
T5
T4T8
T9 T10 T6
T7T1
T3
FactoryBased
FactoryBased
Test Tours
Session Based(Based On Test Charters)
Agile Test Strategy
If development is according to the Agile model, the functional elements of the system will not be known in advance. Developing the test strategy is therefore different when Agile is involved. A document setting out the basic principles of the testing can be drawn up prior to the sprints. This is sometimes referred to as a “master test plan”, but it is very different to the master test plan, which may be familiar from the Waterfall methodology. An Agile project master test plan sets out such matters as the mission for the test project, the general quality requirements, how the work will be carried out and by whom, and what will be documented.
With the Agile methodology a system is created in brief sprints, based on what are called “user stories”. The tester determines how the system elements to be delivered will be tested during the detailing and planning of the user story by the team, the process referred to as sprint refinement + planning. All the forms of situational testing described in Section 3 are available for this purpose. We can put it more strongly than that: testing of the system is optimised if the type of tests to be used is carefully considered at this stage. At the end of a sprint the tester will determine which test scenarios are to be included in the automated regression test set. Each subsequent sprint will involve not only the testing of the new functionality, but also execution of the regression test set.
4.2 ExECUTION Of ThE TEST PROjECT
The test project can start once the strategy has been established. A production-like test environment for each test project will be created well in advance. The testers will also make an early start on studying the system, based for example on documentation or by attending design sessions.
Where factory-based testing, global scripting, session-based testing or bug-hunts are selected, the tester will develop the scripts or charters. It should however be noted here that most of the time not all test charters will be created in advance in the case of session-based testing and bug-hunts, as new test charters may emerge during testing using these methods.
A significant amount of information will be collected on the quality of the system during testing, it will emerge where more or less in-depth testing is required and the test strategy will therefore be reconsidered periodically as the initial tests are completed, perhaps once a week. The test strategy is modified and priorities are re-established based on the actual situation. The ultimate in flexibility! The tester or the team will also adapt the scope and depth of the testing based on the latest information, and report regularly on the progress and status of the project. In the case of these reports, the guiding principle should be “less is more”. The lengthier the report, the less it is read.
Structural elements of all kinds can be used to manage the testing process during its execution, including different test techniques, checklists, the structured recording and reporting of bugs and monitoring the progress and status of the testing project.
Anything goes, just as long as it contributes to the acquisition and communication of information about the quality of the system being tested!
5.
Con
clus
ion Testing is a tool, not a means in itself.The objective of testing is to obtain insight
into the quality of a system at the lowest possible cost and within the shortest possible time, allowing a decision to be made about deploying the system. All forms of situational testing and combinations thereof are options, which can be used to achieve that objective. If a certain element of a test is unsuitable or a certain method proves awkward, then simply choose another one! You are in the best position to assess which approach offers the greatest added value in your situation. If your experience with situational testing is limited, SYSQA can assist you.
Situational testing encourages you to approach testing in a flexible, structured and pragmatic way. Anything goes, as long as it contributes to meeting the objective. Situational testing always delivers better results than the application of a single specific approach. The best results generally come from the use of a combination of forms of testing.
Situational testing is not a stand-alone test method in itself; rather it is a way of combining different forms of testing so that you are always testing in the best way at the lowest cost. Each individual test method can be seen as a toolbox, and situational testing delivers many more tools than can be found in any one of these toolboxes.
www.eurostarconferences.com
Join us online at the links below.