p_1 col template

Borland® Agile Testing Survey Results 2009 -Update

W h i t e Pa p e rJ u ly 2 0 0 9

Borland Agile Testing Survey Results 2009 - Update

2

Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Delegate Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Agile Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

The Challenges of Tooling for Agile Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

The Role of QA in Agile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3


Executive SummaryThis survey contains primary research data collected at the EuroStar Software Testing event held in Den Hague, Holland, on

November 11-13, 2008, and at the STAREAST event held in Orlando, Florida, on May 5-8, 2009. The survey was given to

attendees regardless of whether they used, or were aware of, Agile software delivery methods. This is an update to the 2008

Survey from November 2008 to include data surveyed at the STAREAST event.

In total, 394 respondents completed the survey from which this combined and updated report has been produced. Respondents

represented many countries, including those outside Europe and the United States, and worked in a wide range of industry

sectors and company sizes, from retail, such as eBay; telecommunications, such as T-Mobile; and large scale IT providers,

like Cap Gemini and Microsoft. As you will see, the additional data captured further enhances our findings from the

previous report.

The driver for this study was to understand in greater detail the needs of Quality Assurance (QA) organizations in Agile

software delivery projects. Borland is managing its own Agile transformation and has discovered challenges that appear to be

consistent with other enterprises who have embarked on a company–wide transformation to Agile.

As part of our own transformation, the Borland development team in Austria, who develop our software testing suite known

as Borland® Silk™, found it necessary to adapt the automated testing applications so that they could scale to meet the needs of

testers integrated into the Agile process. As a result, Borland has now produced a unique set of products to fulfil the needs of

Agile and traditional testing, Borland SilkCentral® Test Manager™, Borland SilkTest® with Silk4J® and Borland SilkPerformer®.

The following sections discuss the findings of the surveys and reflect the needs of genuine QA practitioners from all aspects of

the industry, whether current Borland customers or not.We have summarized what we believe to be the key ‘takeaways’ from

the findings of our survey and provided comparisons to the earlier report where differences are highlighted.

If you wish to learn more about Borland Lifecycle Quality Management (LQM) solutions, visit www.borland.com.

Delegate ResponsesDelegates were asked to give the title that closest described their function within the organization. The majority of responses

came from test management positions and quality practitioners.

Agile ExperienceDelegates were then asked about the level of Agile adoption within their organization. Recent trends have indicated an increase

in Agile in the IT industry from 18% to 30% in the last 3 years.

4


Figure 1: Roles of respondents

The survey indicates a much larger growth trend as perceived by this audience of quality practitioners, indeed 61% of

respondents declared some experience of Agile adoption. Furthermore, the largest portion, 42%, was currently in the pilot

phase of Agile Transformation, and now considers it a viable alternative to current development techniques. The survey also

shows that only 23% of respondents had not considered Agile: no knowledge of Agile adoption (7%) or were in an

organization where it was not being considered (16%).

This indicates that the level of Agile adoption within established IT organizations is gaining pace more quickly than previous

predictions. However, it should also be noted that the numbers that have full adoption is low, only 5%, even though this is an

increase in the figure from the November report (2%). This reflects two key findings from Borland;

� Agile adoption is not a destination, it is a journey. It is in the journey where the benefits are realized and amongst theseis finding a toolset to support the Agile organization.

� A pragmatic approach to Agile transformation is required, since Agile and traditional projects are going to have to operatein an integrated, collaborative way

Agile transformation is a continuous and iterative process, including the adoption of true Agile testing into Agile development

to create Agile delivery teams.

5


Figure 2: Agile experience

The Challenges of Tooling for Agile TestingWe asked users what they saw as the biggest challenge for testing in Agile delivery today. The options we gave were based upon

challenges that we saw during our own Agile adoption, and also those of our customers.

The chart shown reflects only those users that have had actual experience of Agile adoption, i.e. those that responded ‘c’, ‘d’ and

‘e’ as seen in Figure 2. Almost one third (32%) responded that they did not see how the testing cycles could possibly cope with

the short timescales of Agile iterations. This is indicative of teams that are reliant upon manual testing, inherently slower than

automated testing, or a dependence upon tools that are incapable of supporting short iterative delivery. In order for QA to

deliver, they require test automation, and they need it to be fast.

This is somewhat validated by another 25% of respondents who expressed dissatisfaction with the capability of their tools. It

follows that over half the respondents with experience in Agile (57%) identified significant hurdles to Agile adoption because of

6


Figure 3: The Challenges to testing in Agile

tooling limitations for automated Agile testing, specifically with the short timescales involved. This indicates a general lack of

flexibility, or capability in automating testing processes in Agile today, and the speed with which tests could be generated and

executed.

When looking at responses from Agile users as to which tools they actually were using, 42% were not using any form of actual

automation for their testing (18% nothing and 24% basic office products). However, the numbers using the traditional

commercial toolset (19%) and Open Source tools (13%) was comparable.

This indicates that the Agile testing community has considerable exposure and knowledge of current automated testing tool

offerings, but still leaves the question as to why still so little adoption of genuine test automation, and a reliance instead on

manual, office products. So when asked further about test automation in Agile (Figure 5), it was interesting to see the results

and why so many felt they were unable to use automated testing tools.

7


Figure 4: Tools used

37% of the Agile testing community felt that the tools they had did not provide a satisfactory capability for their needs.

This is from a community using current testing tools from both the traditional and Open Source community. A further 37%

believed that process and test automation should be done, but could not see a way to do it, potentially indicative of why so

many users have not made a choice of tool; are relying on simple office tools like spread sheets and word processors or have

developed an in-house solution.

Looking at these specific results in isolation, we see a total of 74% of Agile community respondents felt either ‘short-changed’

or were given a lack of viable options when selecting a tool to automate their testing. In fact, only 12% were actually happy

with the tools they had. Again, indicating a widespread dissatisfaction amongst the Agile testing community with the

offerings available.

So in order to understand greater what these issues or short-comings are, Borland asked respondents what it was that they

really needed from a tool as their top priority. In compiling the questionnaire, Borland considered some of the issues they had

themselves been forced to confront, and highlighted the top five as options, with the ability for other answers to be entered.

8


Figure 5: I think Test Automation in Agile…?

The interesting aspect of this chart is that 91% of respondents agreed with Borland on what they consider to be the biggest

issues in tool provision. The biggest issues surrounded regression testing and metrics respectively.

Many of the issues around regression testing are the lack of use of tool automation which takes away the grind of repetitive

testing and allows testing automation to be run overnight without the need for user oversight or operation, in particular, the

need for speed.

The issue of metrics is a long standing one, giving test management the visibility and oversight of all testing and application

quality so that qualitative and quantitative judgements can be made on product quality.

15% of respondents were concerned about the level of effort required to ensure delivery of complete testing within required

timescales and the burdens that this then places on QA staff, the need for long working hours, typical of manual testing

operations or slower tool automation.

The next two issues were about having a tool with the flexibility to actually operate in an Agile way and not wasting valuable

time correcting environments and scripts instead of testing, and finally allowing quality to be seen as a team responsibility,

ensuring that quality is built into the applications from the start with the involvement of developers and testers

working together.

9


Figure 6: What is needed from a tool

The first three issues are typical of any testing environment, whether traditional or Agile in the need to ensure test coverage.

Of particular interest are the final two issues, cumulatively accounting for 30% of the respondents. These issues map

significantly to the unique demands and needs of an Agile development environment, that is, providing an environment to

sustain and support users, rather than hinder their operation in a fast, collaborative team manner.

We also asked users with Agile experience about programming languages used. It is widely considered appropriate, and even

important, that the test scripts for Agile testing be written in the same language as the application under test, so we asked which

languages was your organization using. There has been a long held assumption that Agile and Open Source go hand-in-hand,

principally because both were founded in small development shops wishing to keep costs to a minimum. There is therefore, a

high expectation that the use of Java® with Eclipse™ will be the highest used language and resource amongst these groups.

From the above chart, filtered to show Agile users only, it can be seen that Java is, as expected, the most widely used language

(49%), however, with the introduction of the latest .NET release, there has been a much greater uptake in this area (40%).

The economics of .NET programming resources have helped contribute to an obvious, but slow transition to .NET. In fact,

a large number of respondents (17%) indicated that they were using both languages.

10


Figure 7: Programming Language used

The Role of QA in AgileOur survey not only considered the tools, but also importantly, the people and their roles and responsibilities for quality in

an Agile organization.We asked the respondents if they believed that the concepts of a modern QA organization in IT

development would change and the result was emphatic.

56% considered that the traditional role of QA will change, and significantly so. These respondents were then asked further,

what will be the most important way that the role of QA will change.

11


By far the greatest number (39%) believed that the concept of an isolated QA team will disappear and that instead, QA will

become an integrated part of development.While not surprising, as this is considered a fundamental part of the Agile

manifesto, referring back to figure 3, the challenges to testing in Agile, 16% believe that developers and testers do not mix.

The cultural aspect of this change is a huge challenge to overcome.

Borland has experience in this, with testers becoming part of the Agile development teams, creating Agile engineering teams.

There was a concern that this would weaken the quality position, but the opposite was found. Instead, the perceived value of

quality was raised and became something that was proactively built into the software, rather than treated as an afterthought.

It was also interesting to note that the second largest response was that there would be a change in perspective on test

automation, from ‘nice to have’ to ‘absolutely necessary’ (23%). So despite the earlier reservations from respondents about

current tool offerings, the need for tool automation is still seen as a priority in order to ensure quality software.

An interesting variation, however, from the last report, is the increase in belief that testers are going to have to do more to

champion the cause of integration testing, user experience and sophisticated test scenarios. This is a reaction to the

continued isolation of QA teams from the Agile process and the need to introduce and maintain quality levels in the systems

being delivered.

12


Borland is the leading vendor of Open Application Lifecycle Management (ALM) solutions - open to customers' processes, tools and platforms –providing the flexibility to manage, measure and improve the software delivery process.

Copyright © 2009 Borland Software Corporation. Borland and all other brand and product names are service marks, trademarks,or registered trademarks of Borland Software Corporation or its subsidiaries in the United States and other countries. All other marksare properties of their respective owners. 26736

www.borland.com

ConclusionsIn a review of these study results we can come to three key conclusions;

Agile adoption is increasingly common in the Enterprise, but is still at a relatively immature stage –With 42% of respondents involved in Agile projects that are in pilot, it’s clear that more and more Enterprises are recognizing

the business advantages of shifting to Agile, but have yet to still fully realize all of the potential benefit. Of particular concern is

the issue of integrating true functional and performance testing into an Agile organization.

Automated Agile Testing is recognized as an area of need, but one that is unfulfilled –57% wished to introduce automated testing into their Agile projects but felt that either the tools were not satisfactory or could

not see how testing could be run in such short timescales. 23% believed that test automation was in fact the most important

change that QA will have to face as part of Agile transformation. The need for speed is paramount.

The role of QA will change as a result of Agile adoption –39% see a fundamental change in the way that QA sits in the IT organization and therefore the way that testing will operate as

part of software development.

However, possibly the most important conclusion is that the transformation to Agile is happening everywhere and quality

assurance must adapt. QA must develop Agile testing processes that harmonize with current Agile development methods – this

will change the way we view QA forever.

As a result of Agile testing, QA will become truly about defect prevention, rather than defect cure.


p_1 col template

Documents

needs of agile

agile organization

agile development

agile process

agile experiencethe

level of agile adoption

knowledge of agile adoption

adoption of true agile