Andy Redwood

The Quality Shift

In the beginning..…….and in some quarters continuing today…

Industry Statistics from 2005

Global IT spend expected to rise to $3.3 trillion in 2005Gartner confidently predicts 40% of an IT budget is spent on testing activity. That’s an incredible $1.3 trillion Spending on Test Tools is forecast to be $2.6 billion by 2005Software bugs cost the US. $60 billion per year64% borne by end-users, 36% borne by developers and vendors.Businesses are losing in excess of $200 billion per yeardue to Application disruptions Over 80% of development & migration is conducted independent of infrastructure considerations

© Andy Redwood 1995 - 2015 Published under License

Industry Statistics from 2014-2015

Global IT spend expected to rise to $3.8 trillion in 2015

Gartner observed that most of the organizations are allocating more than 40 percent of their IT budget towards quality assurance and software testing domains $1.5 trillion Spending on Test Tools is forecast to fall to between 4% and 7% a yearCost of IT failures in the 12 months to Dec 2014 cost an unplanned $602,700for each technology-related problem they faced

Global Outsourcing Testing Services Business $80 billion in 2015

© Andy Redwood 1995 - 2015 Published under License

Sources: Garner, Forrester, KPMG, Evoke Technologies, StickyMinds

Global IT spend expected to rise to $3.3 trillion in 2005Gartner confidently predicts 40% of an IT budget is spent on testing activity. That’s an incredible $1.3 trillion Spending on Test Tools is forecast to be $2.6 billion by 2005Software bugs cost the US. $60 billion per year64% borne by end-users, 36% borne by developers and vendors.Businesses are losing in excess of $200 billion per yeardue to Application disruptions Over 80% of development & migration is conducted independent of infrastructure considerations

Global IT spend expected to rise to $3.8 trillion in 2015

Gartner observed that most of the organizations are allocating more than 40 percent of their IT budget towards quality assurance and software testing domains $1.5 trillion Spending on Test Tools is forecast to fall to between 4% and 7% a yearCost of IT failures in the 12 months to Dec 2014 cost an unplanned $602,700for each technology-related problem they faced

Global Outsourcing Testing Services Business $80 billion in 2015

2005

2014/15

Taking a look at ….

• a few of the subjects in testing that we rely on to delivery –

• Viable Quality• Standardised Documentation• Reusable Assets• Measures of Quality• Shared Services• Tested Solutions

Testing Methodologies

What was promised ….White Box

Black Box

Water

Fall

V-model

Risk-Based

RUP

RADW-Model

TRAD

Spiral

Prototyping

Cleanroom

Lean

XP

Agile

Exploratory

TDD

Top-Down

Bottom-up

Typical Reaction in a Waterfall cascade

Requirements

Analysis & Design

Build

Validate

Implement

Requirements are inconsistent and incomplete

Requirements are still changing so Analyst is defining a moving target

Build is confused by changing requirements, descoping to meet timescales - Sometime descopes the testing, but not the build

At the point of no return – so test what we have

Implement what we have

Higher defect rate in production

Theory relating Development and Test only

Lack of integration with Business Models, Project Management, Business and Technical Operations.

Business Unit 1 Business Unit 2 Business Unit n

Home

Country

Country

…

IB

No SourcingStrategy

•Independent High Cost Sourcing across Business Units

Test

Test

Vendor

Vendor Vendor…

TestTest

Test

TestTest

TestTest

TestTest

Test

Vendor

Agencies

Test

Test

Testing sourced from Agencies

Testing done by Vendors

Independent Test in country

Local TestLocal TestLocal Test

Test

Vendor

Test

Local Test

What actually happens….(is chaotic)

23

Inconsistent

Test Practice

No Procurement

controls

No Asset

Sharing

Vendors sell

different

solutions

SaaSSolutions

What is required…

StrategicPartnersMulti-skilled

Resource Pool

Business Technology Support

Home

Country

Country

Process

IB-Tech

Group widePolicies and Strategies

•COEs – Shared Services for all

Physical or VirtualTest COE

CertifiedSuppliers

‘One Client’ for Technology Solutions

Process Process

Shared Service

Dev

elop

men

t

Test

and

QA

Sup

port

Shared Service

Dev

elop

men

t

Test

and

QA

Sup

port

Shared Service

Dev

elop

men

t

Test

and

QA

Sup

port

Test Leads

29

A quick dive into Agile…

Agile

What is emerging…Quality and Agile practice integration are linking up as DOD drives out what constitutes success.

Each piece of work, a story, is shaped to be small, testable, and valuable.

Stories lend themselves to incremental test effort until DOD is achieved.

Practices need to support two of the core Quality Objectives of Testing: Test at the Earliest Possible Moment Avoiding Duplication when shifting testing from earlier levels

Agile

Tes

ting

Pyra

mid

Indi

cate

s th

e “Q

uanti

ty”

NO

T Ch

rono

logy

•Final user acceptance (production readiness) testing in controlled environmentUAT •Final product level NF

Req testing including perf test in Perf env

NF Reqs. •UAT, Perf, UI, E2E,

NF, Regression in QA env

•UAT/Perf in controlled env

User Interface/Perf/End

to end

•SIT, Story acceptance, cross functional,UI, NF Req., QA regression, UAT/perf in QA env

System Integration/ Functional/

NF Reqs. Test•Unit & integration tests, regressions, in CI env

Service Level Integration

•Unit tests in dev env

Unit Test

Dev

QA

Ops

Prod - 1Approximate

FrequencyGo Live

Per Release

Per deployment

(Daily)

Per deployment

(Hourly)

Per deployment(Weekly/Iteratio

n)

Upon Check in

Per Release

Agile Pyramid – Using historic models to build a new way forward

Final – user acceptance (production readiness) testing in controlled environment

Final – product level NF TIncluding performance testing

UAT, perf, UI, E2E, NFT, regressionIn QA env; UAT/perf in a controlled env

SIT story acceptance, cross-functional, UI, NFT, QA regression, UAT/perf in QA env

Unit and Integration tests with automated regression in CI environment

Unit testing in a dev environment

UAT

NFT

User Interface/Perf/End to end

System integration/functional/NFT

Service Level integration

Unit Test

Define Epics/ User stories and acceptance criteria

Define Technical stories and acceptance criteria

Top

dow

n Pr

epar

ation

Bott

om u

p ex

ecuti

on

Reusability of assets

Full traceability

Map tests to stories for Risk-based coverage

Prepare and Schedule NFT

Prepare and Schedule

IT

TDD

Derive positive and negativetests for each user story from the acceptance criteria

Derive positive and negativetests for each technical story from the acceptance criteria

Tests are risk-based, test are mapped to Stories to ensure coverage. Focus is manualTest Design, then automated test design

Schedule NF Test runs into Test Cycles with a focus that highest risk tests will be run first , at the earliest opportunity, with minimal duplication

Schedule Functional Test runs into Test Cycles with a focus that highest risk tests will be run first , at the earliest opportunity, with minimal duplication

Use all of the testware produced and all the schedules in a TDD method to test and build out code

Agile test prep pyramid

Agile test execution pyramid

StoriesPlanning

Backlog

Sprinting in Pods

Deploy

Test driven development

Debugging and unit testing

Integration test

Definition of Done

System Test

OR

OR

UAT

OR

Automation Pod

Non functional test pod

The aspiration..

Tools and Automation

October 1999 Copyright Redwood Associates Ltd 1999

A Tools slide from 19991. The cost of the tool IS NOT the cost of the tool2. Tools come with their own challenges3. Tools generally automate process4. Integrating tools and changing versions of the same tool, will impact you5. Tools will not improve what you do by themselves

Optimised Automation

Building Automated Tests as a Service Function

100% Manual TestsR1

R2 80% MT 20%AT

Rn x% MT y%AT

Optimised automation

Engagement

Automation Service

20% ATR1 Maintain opATRn

‘Continuous Testing’ Enabled**

** Dedicated Test Environments Required

Test Coverage

Time

% Coverage

Automated execution ‘on demand’

Initial Requirements

R2 Requirements

Rn Requirements

Look cautiously at automation if -

there is no fixed outcome (usability testing, “look and feel” testing)

One-off testing (not repeated so no payback on automation)

Very urgent testing (it will always take longer to script it than to run it manually)

If user buy-in is essential (users sometimes don’t trust automated tests, they want to see it with their own eyes)

If the test has to model very complex calculations (you are likely to make an error calculating the expected result within the script)

Deliver automation off the Project’s Critical Path

Automation Ruleset courtesy of SQS-UK 2005

New York

London

Frankfurt

Singapore

New York, London, Frankfurt& Sydney Test Projects

Bangalore

Sydney

Support Centre

Offshore Centre

Follow the Sun model from 2002

October 1999 Copyright Redwood Associates Ltd 1999

1999 Frustrations with tools• The boss changes (his/her mind)• Mergers and takeovers• Maverick departments• Distributed tests• Withdrawn budgets• Project deadlines• Project Managers objectives

Transfer of related business/functional knowledge and

Testing Requirements to MAN

Discussion on Manual Test Scripting Methodology changes w.r.t. the Full/

Component Regression Test Pack on Hand

Creation/Alteration of Requirement/ Test Plan Tree

Structure in TD

Validation of New/Altered

Tree Structures in TD by TM

Discussion on Automation Architecture changes w.r.t. the

Full/Component Regression Test Pack on Hand

Automation Architecture Document for Regression

Test Pack on hand.

Validation of Automation Architecturefor Regression Test Pack

on hand by TM

Existing Full/Component Regression Pack

De-scope and/or Add New Test Cases/Conditions

Client

SME/BA/TA

Transfer of related business/functional knowledge

to Supplier-BA

Analysis of Revised Full/Component Regression Test Pack to get Test Case Count

Supplier-BA

Supplier-BA

Supplier-BA, MAN

Supplier-BA

TL

Supplier-BA, MAN, TL, AUTO

BA/TA, Client

START

Allocation of Test Cases for Manual Scripting/Alteration to Manual Testing Team along

with schedule

TM

Scripting/Alteration of Manual Test Script in TD in Status:

DESIGN

MAN

Taking stock of Manual Test Scripts in Status: REVIWED at

EOD and allocation of baselined Manual Scripts for

test cycle schedules

Supplier-BA

Peer Review of completed/Altered Manual Scripts by

Supplier-BA and MAN

Fix Defective Manual Scripts and re-submit for peer review

MAN

Update Status of successfully reviewed Manual Test Script

to READY

TM

Convert Manual Script in Status READY to Automation ICON and change Status to

DESIGN

TM

Allocation of Test Scripts for Scripting/Alteration to

Automation Team

TM

Creation of folder in Regression Test Pack define

to CM

AUTO

Seek Clarifications/Resolve Issues in consultation with MAN and/or Supplier-BA

AUTO

Test Execution of allocated Manual Test

Script by AUTO

Can Re-use existing Automation Script(s) ?

Check out existing Script/Data File from CM and suitably

amend the script/Data File in HOME AREA. Add relevant

comment in Script.

AUTO

Update Details Screen of Test Case in TD with Automation Script Name and Data File

Name.

AUTO

Move Automation Test Script and Data File to TEST AREA.

AUTO

Write New Automation Script/Create Data File in HOME

AREA. Add relevant comment in Script.

AUTO

Test Execution of Automation Script with

Data File by AUTO

Identify and Fix Defective Automation Script and/or Data

File.

AUTO

Peer Review of Completed Automation

Script and Data File

Taking stock of completed Automation Test Scripts and

allocation of completed Automation Scripts and Data

Files for peer review.

AUTO-LEAD

Re-used existing Automation Script(s) ?

Check in Automation Test Script and Data File to to CM.

AUTO

Request AUTO to Check in Automation Script and Data

File to CM.

AUTO

Check in Automation Test Script and Data File to CM.

AUTO

Batching of Automation Test Scripts and Creation of Test

Cycles in REGRESSION AREA.

TL

Execution of Batch Script through WinRunner in

REGRESSION AREA by TL

Identify cause of failure of batch execution.

TL

Failure of batch execution tracked to individual automation

script and/or data file

Fix the Problem.

TL/DEV

Ready for Regression.

TL

FAIL

PASS

FAIL

PASS

More Test Cases for Manual Scripting/

Alteration ?

YES

Seek Clarifications/Resolve Issues in consultation with

Supplier-BA

MAN

Functionality Check of allocated

Manual Test in Application

by MAN

Execution of Manual Test Script

by MAN

Update Manual Test Script in TD to Status: REVIEWED

MAN

STOP

NO

PASS

FAIL

PASS

FAIL

START

Update Status of Automation Test Case in TD to

REVIEWED

AUTO

STOP

More Test Cases for Automation Scripting/

Alteration ?

YES

NO

FAIL

PASS

YES

NO

FAIL

PASS

PASSFAIL

YES

NO

Update Status of Automation Test Case in TD to READY

TL

More Batch Scripts to create/alter ?

Execution of Batch Script through TD in REGRESSION AREA

by TL

A

A

B

B

C

C

PASS

FAIL

YES

NO

PASS

FAIL

YES

NO

YES

NO

Manual Test Creation/Alteration Workflow Automation Test Creation/Alteration Workflow

Require Integration Check ?

YES

NO

Legend

ClientClient Team

BA/TABusiness/TestAnalyst

Supplier-BASupplier BusinessAnalyst

MANManual Team

AUTOAutomationTeam

TMTest Manager

TLTechnicalLead

AUTO-LEADAutomationTeam Leader

CMConfigurationManagement

Execute AutomatedRegression

A

PASS

FAIL

2015 Experiences1. Tools are open2. You don’t need to be locked into a single vendor3. Tools and process are integrated4. Tools can be a Cloud Service5. Blended toolsets deliver tangible savings in the real world6. Management understand most of the advantages and disadvantages

2015 Reality is a SaaS service

The Cloud

Source Qlick Test 2014

Server

Offshoring

Evaluation of Offshoring options c2008

Alternatives A1 A2 A3 A4 A5

Home Market Local supply only

Add an offshore partner for onshore working

Add an offshore partner for onshore and offshore working

Joint Venture with Partner

Captive

Options Expand Test Centre using existing set up √

Expand in Home Market only Pricing = time and materials Expand into other selected Home market countries within OITR scope

Expand into all HMC’s √ Retender and Rationalise Local Suppliers

X

Include applications on selected platforms

Include all platforms Nearshore Supply offshore resources from a single supplier

Supply offshore resources from multiple suppliers

Single OTC Multiple OTC’s Offshore Farshore Follow the sun Pricing = Managed Services New Legal Entity Share Captive Buy Property Lease Property Own captive

√√√

√√√

√√√√

√√

√√√√

√√√√√√

√√

√√

√√√√

√√√√√√√

√

√√

√√√√

√√√√√√√√√√√

√

√√

√√√√

√√√√√√√√√√√√

Cost Trends c 2008Global Offshore Market by Industry Global demand for Offshore by region

Industry Percentage Offshoring %

Region Percentage Offshoring %

Information Technology 43 USA 59

Financial Services 17 Europe 27

Telecoms 16 Asia 9

Services 15 Rest of the World 5

Others 9

Cost of Test Analyst Employee Resources by region

Employee working in region

Salary (US$ per Year)

USA 67000

Europe 72000

Australia 51000

India 14000

Scaling via Offshoring

Fixed Budget, Scalability by Alternative

0

50

100

150

200

250

A1 A2 A3 A4

Alternatives

SE

K (

miil

ion

s) o

r N

um

ber

s o

f R

eso

urc

es

Cost (Million)

Resource Scaling

Allows for SEK 24M

OTC set up costs

Offshore partnership will increase resource incrementally, for no budget increase

114 Current FTE’s and

Consultants Sourced Locally

Scale 65% over 1A Scale 90% over 1AScale 25% over 1A

1A 1B 2A 2BRetain

54 FTEs

Cost reduction through Offshoring

Fixed Headcount, Cost Reduction by sourcing Alternative

020406080

100120140160180

A1 A2 A3 A4

Alternatives

SE

K (

mill

ion

s) o

r N

um

ber

of

Res

ou

rces

Fixed Resources

Cost (Millions)

15% Saving over 1A

30% Saving over 1A

40% Saving over 1A

1A 1B 2A 2B

Allows for SEK 24M

OTC set up costs

Retain

54 FTEs

Metrics

Metrics that give precise and concise information

Piper PA28 Flight panel

Six Key sources of information

Conclusion

In the last 10 years or so we have moved from ‘hype’ to ‘real solutions’ Methods that work are integrated into the tools –

– However SDLC processes are still treated distinctly and not always integrated with testing or each other

Some tools are very cost effective and with open source can collaborate to replicate a full testing process and integrate seamlessly into development, metrics and support disciplines

Agile and iterative development techniques have gained momentum and have delivered measurable benefits

Offshore is no longer India – but can be anywhere including SaaS services in the Cloud

If investment continues at pace and the trends continue to show testing demonstrating tangible savings, then within the next 10 years we can expect – SaaS tools, SaaS products and services, Big Data-based solutions. Each will self-test and always perform with optimum efficiency.

The next Quality Shift is on the event horizon

References Gerrard, Paul (System Evolutif), (c1992), W-Model Redwood, Andy (1999), ‘Tooling Around with Quality’, Mercury Interactive User conference, October 1999 Gerrard, Paul (System Evolutif), (2000), ‘Risk-based e-business Testing’. Redwood, Andy (2001), ‘3rd Party Time’, BCS Sigist conference, London, September 2001. Redwood, Andy (2004), ‘A follow the sun Test Automation Strategy’, STARWest conference, LA, November

2004. Redwood, Andy (2006), ‘Test Automation – Pain and Gain’, BCS Sigist conference, London, June 2006. Garcia, Pablo (2009) ‘Agile, a religion’ Presentation Slides, Sigist Conference, Tel Aviv, July 2009 Redwood, Andy (2013), ‘Embedding QA into Agile’, presentation slides, Unicom Conference, London, March

2013 Evoke Technologies (2014), http://www.evoketechnologies.com/blog/software-testing-trends-predictions-

2015/ Qlick Test (2014), Testing in the Cloud, http://www.qliktest.com/services/test-environment-service/ Gartner Newsroom (2015), ‘Gartner Says Worldwide IT Spending on Pace to Grow 2.4 Percent in 2015’,

http://www.gartner.com/newsroom/id/2959717 KPMG (2015), True cost of IT Failures,

http://www.kpmg.com/uk/en/issuesandinsights/articlespublications/newsreleases/pages/true-financial-cost-of-it-failures-to-businesses-revealed-in-kpmg-report.aspx

Andy Redwood

The Quality Shift

Andy Redwood is an Independent Consultant

Redwooda @ MSN.com

@redminda

+44 7885797965