testaus 2014: paul gerrard - the changing role of testers

© 2012 Gerrard Consulting Limited

Version 2.0

Intelligent Testing and Assurance

@pau

l_ge

rrardPaul [email protected]

gerrardconsulting.com

The Changing Roles of TestersA personal view

Programme Chair 2014

Helping clients transform their testing through

INNOVATION, COACHING and LEADERSHIP

Our CLIENTS

– Want to be agile rather than follow Agile dogma

– Have a pragmatic approach and are focused on delivery

– Want a solution that fits, not a badly fitting suit.


Version 2.0


Theme

• Pressure to Change

• Software Development Patterns

• Three Innovations that Drive Change

• A New Model for Testing

• We are in Transition

• Leading Change – Killing Clichés.

Intelligent Definition and Assurance Slide 3

Pressure to Change

I meet dozens of Test Managers

through the Test Management Forum

http://uktmf.com

http://uktmf.com/


Version 2.0


Every organisation seems to be

evolving (and quickly)

• 'Traditional' companies are evolving hybrid

Structured-Agile approaches

• Some companies offshore/outsource testing

– Others are bringing it in-house again

• Some are 'shifting-left'

– BDD, TDD or doing story-driven development

– Moving to Continuous Delivery and DevOps

• Embrace change and get promoted


Have you been Agiled?

Agile is mainstream.

Why aren't you Agile?


Version 2.0


Structured versus Agile approaches

Waterfall/Structured

Agile

‘Pure’ Agile

High Structure/Formality

• A one-dimensional view of the range of approaches

• Varying formality, but ALL require discipline

• But this one-dimensional view is too simplistic

• Emerging out of the Agile approach are (what I call)

factory approaches

• Lean, Kanban, Continuous Delivery blah blah…

No man’s land?

Process Formality


Wikipedia – “Agile Methods”


A confusing state of affairs


Version 2.0


There are three patterns, not two

• Some ‘new’ approaches are labelled Agile but

they aren’t!

• Continuous-Delivery approach is appropriate

for rapid-delivery into online/mobile systems

• CD advocates often have an Agile background,

so you might think CD isAgile

• Not so.


Software Development

Patterns

A Model


Version 2.0


Three development patterns

Structured

Agile Continuous

AutonomousIntelligent Definition and Assurance Slide 11

Key pattern characteristics

Characteristic Summary

Structure Structure of the project team

Pace/cadence What drives the rate of decisions making

Leadership How is the team managed/directed?

Definition In what format is requirements knowledge captured?

Testing How is testing performed? (Excludes BDD, TDD)

Auto. Test When are automated tests created? Who leads?

Measurement What/how is project measurement performed?

Governance What form does governance take?


All models are WRONG, some are USEFULNeedless to say, I’ll be generalising quite a lot


Version 2.0


Profiles of the three patterns

Characteristic Structured Agile Continuous

Structure Managed Autonomous Production Cell

Pace/cadence Business decision Team decision Feedback

Leadership Project Managed Guided Research Line Managed

Definition Fixed spec Dynamic spec Live Specs

Testing Scripted Exploratory Automated

Auto. Test Retrospective Developer led Pervasive

Measurement Pervasive Avoided Analytics

Governance Bureaucratic Trust-based Electronic

Intelligent Definition and Assurance Slide 133 patterns – probably more. What about testing?

Three Innovations That

Are Driving Change

Being agile

Shift Left

Analytics


Version 2.0


#1

Agile doesn’t work

but being agile might

Agile is no longer innovative.

Challenging Agile and being agile

might be

Is continuous delivery for you?

Agile doesn’t work?

• Well of course, it works for a lot of teams

• But larger/corporate organisations trying to

‘go Agile’ have difficulties

• Mostly a cultural problem

– Requirements take months to elicit and agree

– Complex stakeholder relationships

– Stakeholders with ‘all the knowledge’ can’t be

spared to be the on-site customer/product owner.



Version 2.0


Pragmatism rules (again)

• Pragmatic companies use the things that work and discard those that don’t

• Let's be honest: Most are doing some form of hybrid approach:

– Embedding testers into their dev teams

– Breaking large requirements into stories

– Development of very small work packages

• TDD, BDD, ATDD, SBE today…

• Continuous Delivery, DevOps… tomorrow?


#2

Shift Left

Developers in test

Testing (testers?) everywhere


Version 2.0


A tester job spec. (May 2014)

• Skills & Requirements– Thorough C# knowledge including Linq

– Good .Net experience including charting and winformsapplications.

– Experience building and maintaining automated test systems

– Ability to interact with the developers to anticipate future needs

– Ability to influence to ensure good test procedures are developed and implemented

• Desirable Skills– Some knowledge of Java for mobile testing

– An interest in photography.


TestingBDD/Test-Driven)Trusted

Requirements + Stories

Shift left (redistributed testing)

Functional Requirements

Build (+little testing) but lots of rework

Testing (lots of checking + bugs)

Trusted Requirements +

StoriesTesting

SAVEDTIME

BDD/Test-Driven


Who is going to do all this?BAs, Developers or Testers?


Version 2.0


From traditional delivery…

… to Continuous Delivery


#3

Test Analytics

Analytics from production

Analytics from test

An opportunity for testing


Version 2.0


Analysis Insight DecisionDevelopment and Testing

ProductionTesting

Introducing Test Analytics

“Introducing Test Analytics”, Paul Gerrard:

http://gerrardconsulting.com/index.php?q=node/630

Julian Harty, "Symbiotic Relationships between Testing and SMAC":

http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between-

testing-and-social-mobile-analytics-cloud-computing/

“The capture, integration and analysis of test and

production monitoring data to inform business and

software development decision-making”.


Data silos for test analytics(an illustration)

Stakeholder Requirements Assurance (Production)

Application

Monitoring

Production

Environment

Monitoring

Stakeholders

Business Goals

and measures

Stakeholder

Involvement/

Engagement

Risk

Requirements

Stories/Feature

Descriptions

Glossary of terms

and Term Usage

Processes

Process Paths

(Workflows)

Manual Tests

Generated Test

Code (unit,

integration,

system level)

Application

Instrumentation

Automated Test

Execution History

Test Status

Feature Code

Change History

Application

Process Flows

User Accesses/

Activities

Feature Calls,

Response times

Feature use cases

(data usage)

Application

Alerts/Failures

Database Accesses

Database Content

Production

failures

System assets

Resource Usage

Log

Performance data

System Events

System Alerts/

Failures/Incidents

Outages


http://gerrardconsulting.com/index.php?q=node/630

http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between-testing-and-social-mobile-analytics-cloud-computing/


Version 2.0


A simple example(implementing a dictionary)

• A list of sentences, complete with source references that contain important glossary phrases

• How is the same phrase is used in different contexts?– Is its use consistent?

– If you have an agreed definition, are anomalies obvious?

– Identify requirements and features that do not mention the key concept – but should do?

• Reference/coverage for any and every concept you define in your glossary:– Requirements

– Stories

– Test cases

– Test execution status and history.


A New Model for

Testing

This is a first attempt. Don't throw stones. Yet.

I'm working on a refined version (with others).

Want to help/review?


Version 2.0


ALL Testing is

Exploratory

You explore sources of knowledge ...

... to build test models ...

... that inform your testing.

The exploration cycle

ExplorationDefinitions (specs)

Oracles

Sources

DesignModels

TestModels

Enquiry

Challenge

Sources:People, documents,experience, system under test

Heuristic

Test Models:Can be documented

or mental models

Enables Prediction

System under test



Version 2.0


The testing cycle

TestingSystem

Under Test

Refines model

Informs tests Apply tests

Interpretoutcome

TestModels

Refine the System

LogsProblems

Applychanges

More explore


Report

What if our source of knowledge is

mostly the system?

• Exploratory testing is the approach to use

when there is little documentation

• Our ‘source of knowledge’ is also the ‘system

under test’

• We need to merge our source of knowledge

and system under test into one...



Version 2.0


"Exploratory" testing

Slide 32

Testing System Under Test

RefineModel

Apply tests

Interpretoutcome

TestModels

Refine the System

LogProblems

Applychanges

Exploration

Enquiry

Challenge

Heuristic

EnablesPrediction

Informtests

Report

Narrative

• All testing is "explore and test (or check)"

• A human OR tool can ‘apply the tests’ but a human creates the model

• ‘Apply test’ means inputs can be sent and outcomes compared in some way

– (Define ‘apply the tests’ as checking if you want)

– But interpretation is always required

• Humans interpret and act on outcomes

• 'Report' probably includes problem reporting.



Version 2.0


What does this mean for testing?

• What is important?– The thought processes: Explore and Test/Check

– Modelling and test design

– Observation, critical thinking, communication

– Transitions: Exploring Testing/Checking

• Less important – logistics– The stages of testing

– Documentation of testing, incident tracking

– Test design techniques

• Assurance covers both testing and logistics.


We are in Transition


Version 2.0


What is Agile really,

really about?

COLLABORATION

and

RAPID FEEDBACK

The trend(s) of testing

• Testing isn’t a stage at the end, it’s a part of all creative activities

– All artefacts are tested, by someone, in some way

• Front-loaded testing (test early, test often)

– But it’s not always called testing

• Test-Driven approach isn’t about test, its about design+risk – devs beginning to embrace it

• Behaviour-driven approach connects requirements, software design and testing.



Version 2.0


Leading Change

Here are the clichés you need to kill.

Here are the soft spots for change.

Developers write code;

testers test because

developers can’t

No they don’t


Version 2.0


Goal: End Manual

Feature Checking

Let the tools do it and

free people to do testing please!

Testing doesn’t just

apply to existing

software, at the end

We test systems, not software.

We test early, we test often but it

might not be called testing


Version 2.0


Testing is about

measuring achievement,

not quality

Align systems delivery with project

goals and risks and you’ve got a seat

on the board

Testers don’t own

testing anymore

Be a TestMaster

"Assurance through the lifecycle"


Version 2.0


New Model Testing

changes what testers

need to learn

Certification teaches logistics and

bureaucracy.

We need to learn and teach

modelling and critical thinking.

The phase after

development is

REWORK, not Testing

Think about it.


Version 2.0


The phase after development is

REWORK, not Testing

• The "Test Phase" should not exist

• Testing must align with development, not

compete with it or rescue it

• Integrate test with code change and remove

the safety net

• Managers point the finger at and put pressure

on development, not testing

• We need a new model of testing.


@pau

l_ge

rrardPaul [email protected]

gerrardconsulting.com

The Changing Roles of TestersA personal view

Programme Chair 2014

testaus 2014: paul gerrard - the changing role of testers

Software

intelligent testing

assurance slide

agile shift

agile background

agile dogma

assurance profiles

agile doesnt work

test testing testers