7 Deadly Sins of Agile

Software Test Automation

Craig Smith

Welcome… Image: © Gracie Films / 20th Century Fox Television http://www.paleodietrecipes.org.uk/paleo-images/content/homer-confession.jpg

Adrian Smith

@adrianlsmith

Geeks Hate Repetition

It Started

With…

Image: © New Line Cinema http://www.imfdb.org/images/5/5e/SevenCover.jpg

© United Plankton Pictures https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash4/p480x480/429897_565831523456105_395040135_n.jpg

Discussion:

What is

stopping you

from being

great at

automation?

Image: http://toworkandback.com/wp-content/uploads/2012/08/stop-sign.jpg

Image: http://scitascienda.files.wordpress.com/2013/04/barney-stinson-quote-when-i-get-sad-i-stop-being-sad-and-be-awesome-instead-true-story.jpg?w=500&h=353

Image: http://www.cupojoy.com/images/image/Quality%20Inn%20Exterior.jpg

Building Quality In

AUTOMATED TESTING Automated

Testing

Image: http://www.lastwordonnothing.com/wp-content/uploads/2011/12/iStock_000007001634Small.jpg

Envy

Flawed comparison of manual testing & automation Image: http://workingtropes.lmc.gatech.edu/wiki/images/Man-vs-machine.jpg

How Management

See Testing

Image: http://www.craigsfire.com/wp-content/uploads/2010/02/pool.jpg

How Management

Would Like To See

Testing Image: http://us.123rf.com/400wm/400/400/dotshock/dotshock1012/dotshock101200006/8437158-internet-network-server-room-with-computers-racks-and-digital-receiver-for-digital-tv.jpg

Repeat After Me…

Agile

≠

Automated Testing

Software is a Series of Loops Image: http://2.bp.blogspot.com/_4G2ptSlZZn0/R9nRjimkLII/AAAAAAAAAqA/tzYJjRyhLkk/s400/brekky05011049.jpg

End to end automated tests and cover 10 conditions per

test, manual covers 1 condition per test

Automated tests cover all high priority business flows,

manual tests mostly cover lower priority flows

Metrics need context!

% Regression test cases automated

Automated Manual

Discussion:

How Much

Automated

Testing Is

Enough?

Manual vs Automated…

A Flawed Comparison Image: http://fvhs.com/wp-content/uploads/2012/11/Screen-shot-2012-11-15-at-7.49.37-AM.png

Testing is more than a series

of merely repeatable actions Image: http://cdn.themis-media.com/media/global/images/library/deriv/71/71335.jpg

Look

Left…

Look

Right…

Image: http://3.bp.blogspot.com/_RdF7S0TNAHY/TOqn8MLYajI/AAAAAAAAAJI/QDBFdixiq0E/s1600/2006-06-17%252520Falling-piano.gif

Look

Up!

21

Image: © Jerry Bruckheimer Television http://images2.fanpop.com/images/photos/4700000/CSI-Las-Vegas-9x18-Mascara-csi-4772098-2000-1333.jpg

CSI:

Software

Regression testing –

assessing current state

Automation of test support

activities

Ideal Automation Targets

Data generation / sub-setting

Load generation

Non-functional testing -

performance, security, ...)

Deterministic problems

Big data problems

Image: http://media.merchantcircle.com/30134145/mobile%20target_full.png

Common “Envy” Symptoms

Relying on automation as

the basis for all testing

activities

All tests are built by

developers

Absence of code reviews

Absence of exploratory

testing

Absence of user testing

Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG

Approach: Desired Role

Automation:

I work with developers to

automate tests that

provide business value

and identify system risks

Collaboration:

I work with analysts and SMEs to

ensure that testable acceptance

criteria are created for all stories

Strategy & Planning:

I am involved in the project at all

stages to ensure that testing

provides the greatest value and

quality objectives are achieved

Tools & Techniques:

I use both manual and

automated techniques using

the preferred testing tools

based on the situation

Architecture:

I have an understanding of the

system architecture and can create

tests that verify individual

components and the system as a

whole

Development:

I take an interest in

development practices

and monitor code quality

metrics

Estimating:

I am involved in developing

estimates for projects at a story

level, for tools and

infrastructure and also for

deployment/release activities

Reporting:

I provide metrics that give

insight into project health

and system quality

Agile:

I understand Agile project

delivery and the differences

between testing in the different

phases of an phase

Qualifications & Training:

I have recognised qualifications

in testing and continually

update and maintain my skills

Recruitment & Development:

I am attracted to the organisation

because testing is a cool career path

that offers heaps of opportunities

and a way of continually developing

my skills

Community & Teams:

I am part of a community

of testers that are

embedded within teams

but share common values

Approach: Report

on Confidence

“Envy” Lessons Learned

Avoid comparison between

manual and automated

testing - both are needed

Distinguish between the

automation and the process

that is being automated

Use automation to provide a

baseline

Use automation in

conjunction with manual

techniques

Image: http://idahoptv.org/dialogue4kids/images/season10/robots/bookbot.gif

•Over indulging on commercial test tools

Gluttony

Over indulging on commercial test tools… Image: © Universal Pictures http://static.ifood.tv/files/external-images/photo-26589.jpg

Discussion: What Automated

Tools Do You Use & Why?

Image: http://www.newnorfolkmitre10.com.au/pics/66/66718b.jpg

Some vendors

believe they

have the

“secret

sauce”

for

automation

Image: http://topnews.ae/images/HP-Sauce.jpg

Promise of Automation

Image: http://thebsreport.files.wordpress.com/2009/10/traveling-salesmen-medicine-show.jpg?w=500&h=364

Image: http://www.melhotornot.com/wp-content/uploads/Rowboat.jpg

Commercial Restricts Usage

Justifying The Expense

I have spent a lot of

money on this tool, we

better be using it for everything!

Image: © Warner Bros Television Distribution / Screen Gems http://3.bp.blogspot.com/-1L3LVOu0cZg/UPs4gju7DJI/AAAAAAAANx0/mrdx5RYSSW0/s1600/jetsons_L74.jpg

Underlying commercial tools

technology often not compatible

with development tool chain

Incompatible Technology

Special file formats or

databases

Lack of version control for tests

and/or cannot be versioned

within the software

Not easily combined with

Continuous Integration

Image: http://virtualmarketingofficer.com/wp-content/uploads/2009/06/square-peg.jpg

Not easily adapted or extended

by the developers

Common “Gluttony” Symptoms

A commercial tool forms

the basis of a testing

strategy

Only certain teams or

individuals can

access a tool or run tests

Developers have not been

consulted in the selection

of a testing tools

“We always use <insert tool-name> for testing!”

Image: http://resources2.news.com.au/images/2008/09/02/va1237328073953/burger-6230645.gif

Approach: Tools Guide

Approach: Craftsmanship

Image: © Wild Dancer Productions / Touchstone Television http://www.tvacres.com/images/props_tool_time_binford.jpg

“Gluttony” Lessons Learned

Favour open software tools

where ever possible

Use tools that can easily be

supported by the

development team and play

nicely with existing

development tool chain

Ensure any commercial

tools can be executed in a

command-line mode so it

can be automated

Image: https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-prn1/p480x480/9346_506859889386718_1005343687_n.jpg

Educate!

•User interface forms the basis for all testing

Lust

User interface forms the basis for all testing… Image: http://stuficionado.net/wp-content/uploads/2012/11/computer_hug.jpeg

Traditionally Test via the UI Image: http://stovallorganizing.com/wp-content/uploads/2013/04/Turtleing.jpg

Manual Exploratory

Collaboratively built around

system behaviour

Developer built optimised for fast

feedback

Co

nfi

de

nc

e

Sp

ee

d /

Fe

ed

ba

ck

Exercises components and

systems

Investment Profile

Understand The Architecture Image: http://harrison.kyschools.us/hcms/TechProjects2011.htm

Test Design

F Fast

I Isolated

R Repeatable

S Self Verifying

T Timely

FIRST Test Properties

Image: http://www.moto-choice.com/PRImages/EditorImages/148330-bathurst-finish-2.jpg

Common “Lust” Symptoms

Testers cannot draw the

application or system

architecture

Large proportion of tests

are run through the UI

Testers have limited

technical skills

No collaboration with

developers

Intent of tests is unclear

Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG

Discussion: What

technologies do you need to

understand to automate?

Image: © Paramount Pictures http://americanobjective.com/wp-content/uploads/2012/04/original.jpg

Agile Testing Command Line

Interface Continuous Integration

Version Control

Build Tools Capture Replay

(Selenium) HTML

Test Maintenance &

Data

Specification By Example / ATDD

/ BDD

Concordion / Cucumber

SQL Web Services

Approach: Test

Automation Course

Approach: Use

UI Tools for

Right Purpose

Use tools like Selenium sparingly, good for exploratory testing

Only use tools like HTMLUnit or scrape

screens when no other options available

“Lust” Lessons Learned

Limit the investment in

automated tests that are

executed through the user

interface

Collaborate with developers

Focus investment in

automation at lowest

possible level with clear test

intent

Ensure automation gives

fast feedback

Image: http://foodcandy.com/wp-content/uploads/2013/03/flowers-and-chocolate1.jpg

Pride Too proud to

collaborate

when creating

tests

Pride

Too proud to collaborate when creating tests… Image: http://latimesblogs.latimes.com/.a/6a00d8341c630a53ef0120a955baf0970b-600wi

Poor

Collaboration

= Poor Tests

Image: http://2.bp.blogspot.com/-RYkFdryP-Co/Tj7-0q3DyPI/AAAAAAAABJU/6tETrv8zyb4/s1600/OpsSilo.jpg

Poor

Collaboration

or

Dedicated

Roles

=

Duplication

Image: © Universal Pictures http://cdn3.whatculture.com/wp-content/uploads/2013/03/twins-poster.jpg

No

Definition

of

Quality

Image: https://lh3.googleusercontent.com/-46OEX5ffUX4/TtVNoEnFPmI/AAAAAAAADBo/g55cQRy5im0/s800/zimogs_kvalitates_zime.jpg

Discussion:

What is

something

you regard

as quality?

Image: http://farm7.staticflickr.com/6115/6287632596_5cc5a681c8_z.jpg

Analyst /

Customer

Developer Tester Automation

Elaboration /

Specification Acceptance

Criteria

High

Performing

AGILE

Project

Manager

Good

Collaboration

Developer

Tester

Analyst

More Technical

Less

Technical

Co

de

Des

ign

Un

it T

est

Au

tom

ated

Fu

nct

ion

al /

S

pec

ialis

t

Test

s

Man

ual

Te

st

Exp

lora

tory

Te

st

Req

uir

emen

ts

Cu

sto

mer

C

olla

b.

Developers need more testing involvement

Testers need more technical involvement

Analysts need more testing involvement

Skills

Red

Green Refactor

Red

Green Refactor

ATDD

ATDD

TDD

Image: http://concordion.org/image/example/online-shop/AcceptanceTest.png

Specification

by Example

Common “Pride” Symptoms

Automated tests are being

built in isolation from team

Intent of tests is unclear

or not matched to quality

Poor automation design

(abstraction,

encapsulation, ...)

Maintainability or

compatibility issues

Image: http://farm8.staticflickr.com/7120/7683912508_b23c4c9ff0_b.jpg

Approach:

Defining Quality Quality

Advocates

What does quality mean to the different roles in

the team?

Quality Taxonomy

What are quality attributes?

Quality Prioritisation

How do we know what quality attributes to

include?

Quality Tradeoff Risks

What are risks with the quality attributes we

are trading off?

Quality Measurement

How do we test and measure quality?

Success Sliders

How does quality relate to the sliders?

Quality Definition

What does quality mean?

Next Steps How do we apply quality to our work

1

2

3

4

5

6

7

8

Approach:

Disband Testing

Teams Image: http://prince2pm.files.wordpress.com/2012/01/industry-groups-product-groups.jpg

Approach:

Simple

Strategy

“Pride” Lessons Learned

Collaborate to create good

tests and avoid duplication

Limit the investment in UI

based automated tests

Collaborate with developers

to ensure good technical

practices (encapsulation,

abstraction, reuse, ... )

Test code = Production

code

Image: https://c479107.ssl.cf2.rackcdn.com/files/7178/article/width668/rfwyhksh-1327469166-1327469251.jpg

Sloth

Too lazy to properly maintain automated tests… Image: http://photos.pcpro.co.uk/blogs/wp-content/uploads/2011/06/Asleep-at-computer.jpg

New Feature

System Interface

Change

OS Patch

Reference Data

Changes

Time

Many Causes of

Automated Test Failure

Time

Co

st

/ E

ffo

rt Potential

Value of Maintained

Automated Test Suite

Value of Unmaintained

Automated Test Suite

Manual test execution

Maintained automation

Unmaintained automation

Importance of

Maintenance

Discussion:

How often

are you

red?

How do you

know?

Who cares?

Image: http://www.eriding.net/media/photos/environment/traffic/070328_jbean_mp_environment_traffic_0135.jpg

Continuous Integration

Watch Code

Build / Compile

Run Tests /

Analysis

Publish Results

Stop The

Line!

Failure?

Continuous Delivery

Image: http://www.thoughtworks.com/imgs/drawing-cd-header.png

#notesting Image: http://3.bp.blogspot.com/-jy4Sc0m-oTA/UU9qrGJw56I/AAAAAAAAAC8/zSlPJS9Ed7c/s200/No+testing.JPG

Common “Sloth” Symptoms

Test suite has not been recently

run - state is unknown

Continuous Integration history

shows consistent failures

following

development changes / release

Test suite requires manual

intervention

Duplication within automation

code

Image: http://commons.wikimedia.org/wiki/File:Cairns_Australia_Beer_Run.jpg

Small changes trigger a

cascade of failures

Approach:

Visible

Status /

Stop The

Line

“Sloth” Lessons Learned

Ensure automated tests are

executed using a Continuous

Integration environment

Ensure tests are always

runnable - even if the system in

not being actively developed

Make test results visible -

create transparency of system

health

Ensure collaboration between

developers and testers

Image: http://1.bp.blogspot.com/_yEAvQXpDW6I/S-8HgogJWpI/AAAAAAAAAY0/phpZuCfP610/s400/Linea%2520Berocca.jpg

Rage

Frustration with slow, brittle or unreliable tests… Image: © 20th Century Fox http://kei.dizkartes.nl/static/img/_common/header_404.jpg

Fast

Feedback Image: http://www.digitallanding.com/wp-content/uploads/2012/07/Faster_Internet-e1346361972417.jpg

Discussion:

What is a

good…

Time

benchmark

for a build?

Amount of

coverage? Image: http://www.mdr.de/sachsen/bild235170_v-standardBig_zc-3ad1f7a1.jpg?version=49443

Image: http://1.bp.blogspot.com/-pKB3xu6zOc8/TWU1W_4OEqI/AAAAAAAAAzk/ygEOBNumv-Q/s400/nespresso-clooney.jpg

No Waiting…

100%*

Image: http://shirtoid.com/wp-content/uploads/2013/02/100-percent-sci-fi1.jpg

Common “Rage” Symptoms

Slow Automated Tests

Brittle

Unreliable

Large datasets

Unnecessary integrations

Inadequate environments

Too many / UI / manual tests

Time bound data

External / Prod integrations

Reliance on UI / sequence

False positives / confidence

Failures being ignored

Workarounds investigated

HEALTH

PROJECT

DEVELOPMENT

TESTING

USER

0

1

2

3

4

5

0

1

2

3

4

5

6

7

1 2 3 4 5 6 7 8

New

Ris

ks &

Issues R

ais

ed

Num

ber

of

Ris

ks &

Issues

Iteration

TEST COVERAGE

0

5

10

15

20

25

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8

Num

ber

of

Tests

Num

ber

of

Defe

cts

Iteration

MAINTAINABILITY PERFORMANCE 0

200

400

600

800

1000

1200

1400

1600

1800

4700

4800

4900

5000

5100

5200

5300

5400

5500

5600

5700

1 2 3 4 5 6 7 8

Lin

es o

f T

est C

ode

Lin

es o

f C

ode

Iteration

OVERALL

$-

$5

$10

$15

$20

$25

$30

0

1

2

3

4

5

6

7

8

9

1 2 3 4 5 6 7 8

Busin

ess V

alu

e

Num

ber

of

Featu

res

Iteration

Approach: BVC

Approach:

Cloud /

Central /

Open

Source ALM

“Rage” Lessons Learned

Treat automated tests with

the same importance as

production code

Review, refactor, improve ...

Apply a “Stop the line”

approach to test failure

Eliminate (quarantine)

unreliable tests

Image: http://us.123rf.com/400wm/400/400/edma/edma0907/edma090700217/5226661-businessman-doing-yoga-exercises-in-a-city-park.jpg

Ensure collaboration with

developers

Greed

Trying to cut costs through automation… Image: © 20th Century Fox http://blogs.telegraph.co.uk/news/files/2010/09/gordon-gekko.jpg

Lure of saving labour Image: http://www.caradvice.com.au/thumb/640/556/wp-content/uploads/2010/11/Car-salesman.jpg

Automation is not cheap!

Image: http://designbuildsource.com.au/wp-content/uploads/2012/03/australian-money.jpg

Common “Greed” Symptoms

Investment in commercial

tools using a business-case

based on reducing headcount

Using a predicted ROI as a

way of reducing budget for

testing

Consolidating automated

testing within a special group

Image: © Gracie Films / 20th Century Fox Television http://thegreeneconomy.com/wp-content/uploads/2012/05/greedy-246x220.gif

New Software

Test Engineer

Role

New Graduates,

Recruits, Consultants

& Partners

BA

SME

Other

Up

Skill

As

se

ss

me

nt

Current Test Analysts

Capable + Current Skills +

Desire

Capable + Desire

Not Capable + No Desire

Not Capable + Desire

**Assessment = Technical Test + Interview**

Approach:

Assist & Assess

Approach:

Outsource

Teams Not

Testing

Image: https://sphotos-b-ord.xx.fbcdn.net/hphotos-ash3/p480x480/942876_505183632888249_1411782978_n.jpg

Approach: Avoid Test

Automation Projects

Image: http://www.hifi-writer.com/he/progscan/Files/abcestablish.jpg

“Greed” Lessons Learned

Ensure the reasons for

automation are clear

and are NOT based

purely on saving

money/headcount

Ensure business case

for automation

includes costs for

ongoing maintenance

Image: http://resources3.news.com.au/images/2010/12/02/1225964/296651-piggy-bank-and-coins-chart.jpg

Quality

is Key

Image: http://gs-press.com.au/images/news_articles/cache/key_hand_resized-600x0.jpg

Approach:

Quality

Assessment

Tests are NOT executed on a regular basis

State of current defects is unknown

Functional testing occurs regularly

All testers can execute the functional tests

Production verification testing is used to ensure success

A majority of functional tests have been automated

Exploratory testing forms part of test execution

Customers verify implemented features prior to deployment

A majority of non-functional tests (performance, reliability, ...) are

completed prior to deployment, many are automated

Testers pair with developers to automate tests

Customers verify implemented features as they are completed

Functional and non-functional testing occurs continuously within

development iterations

Developers and testers are performing test first practices

-1

1

2

3

4

Discussion: Test Execution

Wrap

Up

Image: http://www.pria.com.au/sb_cache/priablog/id/924/f/7%20deadly%20sins.jpg

Envy Flawed comparison of manual testing & automation

Gluttony Over indulging on commercial test tools

Lust User interface forms the basis for all testing

Pride Too proud to collaborate when creating tests

Sloth Too lazy to maintain automated tests

Rage Frustration with slow, brittle or unreliable tests

Greed Trying to cut costs through automation

7 Deadly Sins

Why Use Automation?

Image: © Fuzzy Door Productions / 20th Century Fox Television

Don’t Lose

Sight of the

Goal

How geeks really work How Geeks Can Work Together

Questions?

Craig Smith

http://www.craigsmith.id.au

craig@craigsmith.id.au

@smithcdau