E s s e n t i a l f o r s o f t w a r e t e s t e r sTE TERSUBSCRIBE

It’s FREE for testers

February 2012 v2.0 number 13£ 4 ¤ 5/

Including articles by:

developmenttesting

THIS ISSUE OF PROFESSIONAL TESTERIS SPONSORED BY

Chris AdlardCoverity

Les HattonKingston University and Oakwood Computing Associates

Harry M. Sneed and Manfred Baumgartner ANECON

Boguslaw Czwartkowski Parasoft

Geoff Quentin

As the Ancient Greek philosopher Democritus observed, everything is in a state of fluctuation. In today's fast-moving IT world software systems and the people who develop and maintain them must be flexible. Goals, requirements and environments change and to remain consistent so must software, reflecting the changing world of which it is part. It is never finished. This is the main reason for agile development.

Development must be able to accommo-date change in code architecture and even the reasons for a system to be developed in the first place. The direction of an agile development project may change on a monthly, weekly or even daily basis [1].

But change is the enemy of test. To test something requires that it remains stable long enough to test it. A tester wants to

know that what was tested yesterday is still valid today. The proper goal of testing is to confirm that a given state of the software corresponds to a given state of the requirements. For that the software and the requirements must remain in state long enough to demonstrate that their states are equivalent.

Here we will explain how this goal can be reconciled with the goal of agile development which is to be flexible.

Conventional testing approachIn conventional testing, empirical testing follows development. The test is planned in advance based on the requirements. While the developers are producing the code, the testers are specifying test cases and designing test scenarios. When the software components are finished, they are turned over to the testers and remain

by Harry M. Sneed and Manfred Baumgartner

From test-driven to tester-driven agile development

Testing’s role will be to inhibit code proliferation

PT - February 2012 - professionaltester.com 16

The future of software predicted by Harry M. Sneed and Manfred Baumgartner

Development testing

several project advantages. First, it elimi-nates the team's reliance on a single, assigned project tester who would otherwise constitute a major bottleneck... The professional tester adds value not through more testing but by writing some of the developers' tests, thus freeing them to code more new features…”. It is inte-resting to note that in this project the testing went on even when the tester was temporarily removed from the project. This was because by that time everyone was involved in the testing effort. The developers tested themselves, in do-it-yourself mode (figure 2).

Initially, the testers did not work with developers but alone, to make cross-feature tests and automate customer-defined use cases. The test manager, who was not a member of the team, demanded this separation for two reasons. First, that testers should be independent of programmers to prevent implementation details from affecting their test designs. This is referred to as the contamination argument [5]: that a tester cannot write adequate tests if a programmer, who is biased, dictates what the testing requirements are. The second reason is the verification argument [6], that one of the goals of testing is to verify that the product meets its specification, therefore testers must create tests according to approved specifications, not according to what someone has told them informally or unofficially.

These two arguments represent the main reasons many QA professionals object to agile testing. The counter argument is not that the arguments are wrong, but that to ignore them is better than to enforce them. Enforcing them would mean bringing the project to a grinding halt due to lack of testing capacity. Accepting them as a fact of life may weaken the validity of the test, but not enough to endanger the project. Some published reports on testers' experience in agile teams support this counter argument, for example [7].

To ignore the two arguments the tester must change his or her mindset and

in a steady state as long as they are being tested. If a new version of the software is delivered in the meantime, the testers put it on standby status until they reach their testing goals for the current version.

This approach is based on the waterfall model. As long as the interval between versions is long enough it can be justified. If versions are delivered faster than they can be tested, testing becomes a bottle-neck. For that reason such separation between development and test is not possible in an agile project.

Tester in the team approachAgile development has created a new situation which has to be dealt with in another way. It is no longer possible to wait until the developers are finished to begin the integration and system testing. The system has to be tested while it is in development. One solution is to put the tester or testers in the development team. They sit next to the developers, testing components as soon as they have been compiled (see figure 1).

Do-it-yourself test-driven approach Another solution is to have the developers test. The tester is a test adviser helping the developer to set up test environments, design tests and above all validate test results. One could extend the testing responsibility to analysts and designers as well. This is the approach taken by the Israeli Air Force in their agile development projects [4]. They claim that in their project everyone is involved in test: developers, business analysts and customers. The testers are only there to help them and to ensure they do their work properly. The authors state “having everyone test offers

abandon some principles. The tester must see that continuous interaction with the developers is better than withdrawing into a corner and writing tests based on one's own interpretation of the requirements specification: especially if, as is always the case in agile environments, that lacks detail and rather is implied by stories told by the “user in the team” which are subject to continual change and documented informally if at all. The only accurate, up-to-date description of what the system does is the code. Therefore, agile testing as practiced by the Israeli Air Force was not about comparing documents and system behaviour with those documents, but human judgment of what was appropriate for each situation that arose.

According to the Israeli authors the key was communication. Everyone, including the testers, must sit in the same room. Everyone participated in planning, everyone was present when the stories were told and everyone attended daily stand-up meetings. In this way everyone knew the strengths and weaknesses of colleagues and could sense where to look for defects. Test decisions were based more on human intuition than on written documents. When a defect was detected they knew who was responsible for it and got him or her to fix it with mini-mum bureaucratic administration. The goal was to find as many defects, in as little time, as possible and for that intimate collaboration by all team members was necessary. This testing together with the tester as an adviser to others proved very effective. The number of defects remaining in the software after release was four times fewer and completion was reached faster than in comparable projects using a conventional testing approach [8].

Tester-driven approachThere is a large body of literature on test-driven development. It is well summarized in a book by Kent Beck, the father of extreme programming, of whichit is a pillar [9].

TDD is a method developers use when coding. Before beginning to code a

17PT - February 2012 - professionaltester.com

Development testing

Software

Developers

Tester

Requests

Develops

User in the Team

The tester runs after the developers to test what they have produced as soon as it has been developed

method, test cases for it are devised. These can be coded in a separate test driver class or built into the class containing the method to be tested. Now code intended to pass the test cases is written. In this way the developer can test one method at a time and then combi-nations, at first with one object state and then with others. The goal should be to test all relevant combinations of methods with all relevant representative states of the object type under test [10]. In test-driven development the developer tests. There is no role for a tester.

Tester-driven agile development carries that principle to a higher level. The test objects are not classes but components, ie packages and subsystems. One or more professional testers drive the project. Their first task, after the system architecture is designed but before coding begins, is to set up an integration test environment based on that design. An essential element is the specification of the interfaces between subsystems and components and between the system and users, which are implemented as web services, XML batches or mocked-up user interfaces, for example HTML forms. The testers can then specify test cases and generate test data based on these interface definitions.

Only at this point are the developers involved. The testers assign them components to implement. When one is ready it is turned over to the testers who build it into the integration test environ-ment and start testing it. Components are integrated one after another so change to requirements can be accommodated easily. The project is agile but its pace is

determined by testing not by development, in tune with the Kanban method [11].

Many project effort distribution studies assert that testing takes the greatest part of project effort. If that is true then according to the principles of Kanban testing should be started first. The testers should be the leaders and the developers the followers. Instead of running after the developers trying to test what they produce as soon as possible and falling farther and farther behind in an attempt to do so, the testers determine what is to be delivered next, on the basis of testing expediency, and order it from the developers, who act as suppliers to the testing project.

The user in the team works with the testers and system architects but only indirectly with the developers. First the user must get the architect to either take over or construct a suitable system framework. Often one will already be avail-able. Once it is in place the user can begin to formulate requirements. The stories are told to the testers who decide how best to implement them. In many cases it will be possible to implement them with existing software or with a cloud service: then

there would be no need for developers. Only if some function is absolutely unique to the target application will developers be asked to develop code to implement it. The goal of TDAD is to develop as little as possible and reuse as much as possible. It may be that some services have to be wrapped in order to fit into the target environment and would constitute an order to the developers. So the developers may get orders to produce some entirely new software but more often they will get orders to adapt existing software (figure 3).

There is already far too much code in the world. Maintaining and improving it costs a lot and ties down valuable resources leading to a worsening shortage of skilled personnel. The only way to get out of this legacy code trap is to produce less code. Extreme programming and agile non-tester-driven development are certainly not ways to achieve that. Given the freedom that extreme programming grants them developers inevitably produce more code than is necessary. No self-respecting programmer would ever miss the oppor-tunity to prove his competence on some hairy problem for which there are already dozens of other solutions. We could hardly

PT - February 2012 - professionaltester.com 18

Development testing

Figure 1: The tester in the team

Own softwareStandardservices

Standardservices

Developers

Requests

Integrates

develops

User

Tester as integrator

& tests

Software

Developers

Develops

User in the Team

The tester works next to the developers to advise them on testing themselves

Tester as adviser

& tests

19PT - February 2012 - professionaltester.com

expect him to look for them, since his main purpose in life is to solve hairy problems himself.

The goal of a software project should not be solving the user's problem as quickly as possible. It should be solving it with the least new, untested code possible: in other words, to deliver the best and longest-term solution possible. Every additional code unit produced by a user organization is like a mortgage whose interest is its annual maintenance which accumulates and increases over time. Producing more and more new code is as destructive as producing more and more superfluous consumer goods. But will developers be willing to accept this role and will testers have the ability to take on responsibility for assembling a system? This is mainly a question of training and conditioning. In agile projects as they are currently conducted developers are asked to transform the wishes of the user into executing code as quickly as possible.

Testers are trained and conditioned to run after the developers and to clean up whatever mess they create. They are kept eager to test anything developers decide to give them, in any state and of any quality.

TDAD calls for these roles to be reversed. Developers must be convinced that it is in their best interest to produce as little new code as possible because every line they do not produce is a code line that they will not have to maintain. Users must be made to understand that it is in their best interest to depend on as little new code as possible.

Testers must grow into their new role as system integrators: practicing the search for and testing of available services then persuading the user to adjust require-ments to make them acceptable. If the user refuses, testers commission developers to build a wrapper around suitable services with the objective of

using as much of their existing functionality as possible. Only if there is evidently no suitable ready-made solution should the testers consider contracting developers to produce a new one.

The role of the testers involves the same considerations as in any environment:

Development testing

Figure 2: Do-it-yourself test-driven approach

Figure 3: Tester-Driven Agile Development

Requests

Harry Sneed (http://harrysneed.de) is a software development consultant, trainer, published author and conference speaker. Manfred Baumgartner is head of software testing at ANECON (http://www.anecon.com). The authors wish to thank Rudolf van Megen, CEO and co-founder of SQS (http://sqs.com), for his original suggestion of combining cloud services and agile development principles which was their inspiration for this article

Development testing

PT - February 2012 - professionaltester.com 20

functional, non-functional and security-related [12]. When an appropriate service is found, acceptance testing is carried out. What is accepted and new components required because of what is not accepted are integrated iteratively into the required whole and that integration is tested. Should an integration problem occur the testers specify an appropriate solution and have the developers implement it. The target system emerges component-by-component until it has reached a state that the user can begin to use. After it has gone into productive use it can continue to grow by addition of more functionality, as far as possible using ready-made cloud services. The production system will be a hybrid, comprising as little locally-developed code as possible. The cost of developing and maintaining that will be small and will tend to fall. Users and developers will be freed from the burden of legacy code.

TDD has already proven its merit [13]. TDAD is a logical continuation that defines a new supply chain. The user gives re-quirements (tells stories) to the test/inte-gration team (minimum two members) who search for suitable services in service libraries, eg http://uddi.xml.org. If they find a candidate they integrate it then test it. If no candidate passes they commission code necessary to meet requirements then test it just as they would a foreign service. They treat services of both origins equally:

both must be integrated into the system framework then tested.

It goes without saying that the framework must be tested. This may take time but is essential. Once it is done the framework can be filled with application-oriented functions that deliver user requirements very quickly and these can be exchanged and added to very easily. At that point managing application evolution will become cheap

[1] Mugridge, R.: Managing Agile Project Requirements with Storytest-Driven Development, IEEE Software Magazine, Jan. 2008, p. 68

[2] Crispin, L., Gregory, J.: Agile Testing: A practical Guide for Testers and agile Teams, Addison-Wesley, Boston MA, 2008

[3] Sneed, H., Majoros, M.: Testing Programs Against a Formal Specification, Proc. of IEEE COMPSAC-83 Conference, IEEE Computer Society Press, Chicago, 1983, S. 512

[4] Talby, D., Keren, A, Hazzan, O., Dubinsky, Y.: Agile Software Testing in a Large-Scale Project, IEEE Software Magazine, July 2006, p. 30

[5] Stephens, M., Rosenberg, D: Extreme Programming Refactored – The Case Against XP, Apress Publishers, Chicago, 2003

[6] Kuhn, R., Kacker, R.: Combinatorial Software Testing, IEEE Computer Magazine, August, 2009, p. 94

[7] Crispin, L., Extreme Rules of the Road – How an XP Tester Can Steer the Project Toward Success, STQE Magazine, July 2001, p. 24

[8] Dubinsky, Y.: Agile Metrics at the Israeli Air Force, Proceedings of Agile 2005 Conference, IEEE Press, May 2005, p. 12

[9] Beck, K.: Test-Driven Development by Example, Addison-Wesley, Boston, 2003

[10] Zhang, Y., Patel, S.: Agile Model-Driven Development in Practice, IEEE Software Magazine, March, 2011, p. 84

[11] Poppendieck, M., Poppendieck, T.: Implementing Lean Software Development, Addison-Wesley, Boston, 2006

[12] Tsai, W.T., Zhou, X., Chen, Y: On Testing and Evaluating Service-Oriented Software, IEEE Computer Magazine, August, 2008, p. 40

[13] Shull, F. et al.: What Do We Know about Test-Driven Development?, IEEE Software Magazine, Dec. 2010, p. 16