a new model for testing

A New Model for Testing@

paul_gerrard

Paul [email protected]

gerrardconsulting.com

http://testaxioms.com/?q=node/25

Helping clients transform their testing through

INNOVATION, COACHING and LEADERSHIP

Our CLIENTS– Want to be agile rather than follow Agile

dogma– Have a pragmatic approach and are focused on

delivery– Want a solution that fits, not a badly fitting

suit.

Agenda• Innovations that Drive Change• 3 Patterns of Software Development?• A New Model for Testing• Some Consequences• Some Research• A New Test Process?• Close

What's happening?• Stampede to mobile computing -

Digital• Big Data• Internet of (Every)Thing, Pervasive

Computing• Continuous Delivery, DevOps• (Test) Analytics, Data-Driven

Development• Shift-left, embedded testers, no test

team• Did you really think Agile was the last

word?

Three Innovations That Are Driving

ChangeBeing agileShift LeftAnalytics

(Digital/IoT/IoE and others too, of course)

#1Agile doesn’t work

but being agile might

Agile is no longer "innovative"Challenging Agile and being

agile might beIs continuous delivery for

you?

Have you been Agiled?

#2Shift Left

Developers in test?Testers in development?

Automation, analytics, collaboration, experimentation

#3Test Analytics

Shift-Right?Analytics from production

Analytics from testAn opportunity for testing

Three Software Development

Patterns?A Model

There are three patterns, not just two• Many ‘new’ approaches are labelled 'Agile'

but– They aren’t Agile, they just work– Aimed at factory workers, not knowledge

workers• Continuous-delivery approach is appropriate

for rapid-delivery into online/mobile systems• CD advocates often have an Agile

background, so you might think CD is Agile• Not so.

Intelligent Definition and Assurance Slide 10

Three development patterns


Structured

Agile Continuous

Goal-Bas

edHi-Process

Autonomous

Key pattern characteristicsCharacteristic SummaryStructure Structure of the project team

Pace/cadence What drives the rate of decisions making?

Leadership How is the team managed/directed?

Definition In what format is requirements knowledge captured?

Testing How is testing (mostly) performed?

Auto. Test When are automated tests created? Who leads?

Measurement What/how is project measurement performed?

Governance What form does governance take?


All models are WRONG, some are USEFULNeedless to say, I’ll be generalising quite a lot

Profiles of the three patternsCharacteristic Structured Agile ContinuousStructure Managed team Autonomous Production Cell

Pace/cadence Business decision Team decision Feedback

Leadership Project Managed Guided Research Line Managed

Definition Fixed spec Dynamic spec Live Specs

Testing Scripted Exploratory Automated

Auto. Test Retrospective Developer led Pervasive

Measurement Pervasive Avoided Analytics

Governance Bureaucratic Trust-based Insight-Driven


There aren't three patterns

There are many

From high process to DIY process• In the past, high process was seen to

be the technical solution to the software problem

• But technical solutions cannot solve human problems

• True agility means a team can adapt or select its process on-the-fly

• We need to strip down our thinking to the essential aspects…

The old ways won't work in the future

We need a New Model of Testing (free from logistics)

Forget Logistics(for the time being)

Document or not?Automated or manual?

Agile v waterfall?This business or that

business?This technology v that

technology?

ALL Testing is Exploratory

We explore sources of knowledge ...

... to build test models ...... that inform our testing.

Examples of models

Models are innate, essential, human

Judgement, exploring and testing

Testing(the system)

Our model(s) are adequate

Our model(s) are not adequate

Exploring (sources) Judgement

Creates testmodels

Uses testmodels

We explore sources of knowledge to build test models that inform our testing

BTW – Do Developers explore the same way? I think so.

Exploration process

ExplorationDefinitions specs/stories

People(& you)

Sources

Require-ments

TestModels

Enquiring

Challenging

Sources:People, documents,experience, system under test

Modelling

Test Models:Can be documented

or mental models

Predicting

System under test

Testing process

TestingSystem

Under Test

Refining

Informing Applying

Interpreting

TestModels

Revise the System

Logging Revis

ing

More exploring Reporting

New Model Testing

My BBC talk: http://www.bbc.co.uk/academy/technology/article/art20150522113029398 29 page paper: http://dev.sp.qa/download/newModel

http://www.bbc.co.uk/academy/technology/article/art20150522113029398

http://www.bbc.co.uk/academy/technology/article/art20150522113029398

http://dev.sp.qa/download/newModel



Some Consequences

There are others

Relation toTDD and BDD?

Is TDD really testing BDD is modelling using stories

Test automation from a different

perspectiveAutomation efforts fail too oftenAutomation uses different test

models

Capabilities

Enquiring, Modelling, Predicting, Challenging

Informing, Applying, Interpreting, Refining

Reporting and Logging

• Analysis, enquiry and elicitation• Modelling• Creation of custom models, using

heuristics, guesses, brainstorming, ideation, creative thinking

• Custom test design techniques• Comparison of models, value,

advantages, disadvantages, compromises

• Identification, validation and use of oracles

• Predicate logic and proof• Hypothesis and inference• Socratic method• Rapid Review and Inspection

techniques• Test case design• Test models and the meaning of

coverage• Testing as controlled experiment• Observation, Note taking,

recording

A very different skillset• Basic data analysis and statistics• Decision-making with

incomplete data• Computer forensics• Fault tree analysis• Failure diagnosis• Bug advocacy, triage processes

and negotiation• Meaningful software and test

metrics• Visual presentation of data• Reporting and presentation skills• Understanding stakeholders• Test analytics• Risk management, risk-based

testing and decision-making• Critical Thinking• Interpersonal skills• Dealing with

uncertainty/fallibility

Testing Career Development(speculative)

Foundations

Technical

Management

StrategicTest Strategy Project

IntelligenceTest Assurance

Exploration Forensics Interpretation

Scripting/Programming

Test Automation Technical (Excel, SQL, OS utils etc)

Stakeholder management

Analytics & visualisation

Managing uncertainty

Critical Thinking

ISTQB, TMMi etc...

Supplier Management

Test Process Management

Methodology

Will Robots Replace Testers?

Some research

There is a paper at: https://tkbase.com/resources/viewResource/14

https://tkbase.com/resources/viewResource/14

https://tkbase.com/resources/viewResource/14

A recent study*…• Over the next two decades, 47% of jobs in the

US may be under threat• It ranks 702 occupations in order of their

probability of computerisation– Telemarketers: 99% likely– Recreational therapists: 0.28% likely– Computer programmers: 48% likely

• Something significant is going on out there• If programmers have a 50/50 chance of being

replaced by robots, we should think seriously on how the same might happen to testers.

* “The future of employment: how susceptible are jobs to computerisation?”http://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf


Some systems-related occupations


Occupation Rank (out of 702)

Probability of Computeris-ation

Computer and Information Research Scientists 69 1.5%

Network and Computer Systems Administrators 109 3.0%

Computer and Information Systems Managers 118 3.5%

Information Security Analysts, Web Developers, and Computer Network Architects

208 21%

Computer Occupations, All Other 212 22%Computer Programmers 293 48%Computer Support Specialists 359 65%Computer Operators 428 78%Inspectors, Testers, Sorters, Samplers and Weighers

670 98%

Test Automation = Mechanical Tools

What we REALLY need are THINKING TOOLS

The term Test Automation misleads• It misleads as a label because the

whole of testing cannot be automated

• The label is bad, but the scope of Test Automation is what I call ‘Applying’ in the New Model


Testers need Thinking Tools• There are ten testing activities in the New Model– Test automation tools only support one: ‘Applying’

• The remaining nine activities (information gathering, analysis, modelling, challenging, test design and so on) are not well supported

• All require some level of thinking and skills• Checking is possible when a system and its

purpose are well understood and trusted• Test automation tools are simple in principle…

… compared to the rest of the test process.


Four quadrant model of intelligent test tools

Ability to Investigate

Abili

ty to

Cap

ture

Kno

wle

dge

• Text editors, Screen Shots

Models, visualisations, relationships, transformations

• Note Takers

• Mind Maps

• UML/Case Tools

Control, imagination, discernm

ent, foresight

• Pencil and paper, sketching tools



TERMINATORTESTERNot Yet!

Cervaya™Tool Architecture

A nine month bot journey(But it’s been a twenty year

testing journey so far)https://huddle.eurostarsoftwaretesting.com/resource/test-management/new-model-testing-new-test-process-tool/

https://huddle.eurostarsoftwaretesting.com/resource/test-management/new-model-testing-new-test-process-tool/



Intelligent Definition and Assurance

The vision thing• I want a bot partner/pair that supports

exploratory testing• You know my view (model)

of testing already• Can we use the explore v test paradigm in a bot

that allows you to:– Explore, take notes and model– Record ideas, risks, tests, outcomes and bugs– Generate reports and documentation as a

consequence, rather than requiring you write them• Codename: Cervaya (cervaya.com is a holding page for now)

Slide 40

Schematic


Schema ServerSchema

Manager(web site)

Schema Repository

(web service)

Cervaya Bot ClientRobot Engine

Cervaya Server

Actions(web services)

Application Web Site

Speech Recognition

Interface

Command Line

Interface

Administer Schemas through the Web

DownloadSchema Perform Robot

Actions through services

Application Reporting, Monitoring, Control,

Management

Many App Servers – one (or many) per Schema

robotschemascervaya

The target user is…• (Not really a freestyle exploratory tester

working in a start-up or small product company)• Target users are testers working in:– Regulated, high-integrity, safety critical– High documentation, high accountability– Environments where testers are constrained by their

processes– Testers struggling to apply ‘trad’ methods in an

Agile, Digital, DevOps environment• Why can’t we dictate a detailed test plan for a

bot to document, analyse, visualise?


The location hierarchy• Explorers create maps (or should do!)• The system model/map is a hierarchy of

locations as follows:• Application– Places

• Features– Forms

» Fields• All other content is located with respect to

some level and location in the system model.


A New Test Process

Exploration support• Frustration:

– testers are unimaginative, working by-rote– constant pressure to cut costs

• Productivity of exploratory testing is proven• Testers want to explore, but the need for control and

documentation constrains them• Testers needs tools that can capture plans and tester

activity in real-time• The next generation will be led by tools that support

the exploration of sources of knowledge• These tools might use a “Surveying” metaphor.


Staged ProcessSystem Testing Re-Test

Big, up-front Test Planning

RegressionTestDevelopment

Automation

From Staged to Continuous

Scoping exploration sessions

Interactive Testing Sessions

Clarifications

Continuous Integration

ChangesRefinements

Continuous Dev, Test and Delivery

Bot Supported

Bot Supported

Speech recognition, text to speech• Right now, I’m using:– Google Speech Recognition (a slow web service)– pyttsx a Python Text to Speech module (depends on

pywin32-extensions for Windows)• I’m looking at Amazon Alexa as an alternative– https://www.youtube.com/watch?v=hPXS7rC1PWo– Alexa is the web service, Echo Dot is the latest

device that captures and synthesises voice– It promises to be a lot faster (and it works better).


https://www.youtube.com/watch?v=hPXS7rC1PWo

https://www.youtube.com/watch?v=hPXS7rC1PWo

Real-Time test management• Once you have a shared system model:– The activity of the team can be visualised in real-

time– Tester activity: visible – testers liaise when

necessary– Test paths through the system map can be traced

• Developers can also see progress and requests for more information in context:

• Cervaya has an integration with Slack:– Model changes, additions, queries, test failures

can be posted to Slack/ChatOps service.


From Staged to Continuous Testing• Waterfall projects create, execute tests in stages• Shifted-left testers can:– Model features as they are created by the developers– Create accurate tests a low level as the functionality

emerges over time– End to end tests use low-level tests as building blocks

• Tests and test run histories are manage against versions (and platforms/environments soon)

• The aim: to export tests as soon as practical to be automated to provide continuous feedback.


Testing Styles, Approaches“Structured/Waterfall/ Staged” Testing• Systematic• Transparent, Documented• Reviewable• Auditable• Repeatable, measurable• Automatable• Inflexible, not responsive• Obsolescent/inaccurate

documentation• Prone to biases, inattention• Outdated process• Expensive, Inefficient• Unimaginative, boring

“Exploratory” Testing• Agile (with a small ‘a’)• Improvised, imaginative• Flexible and responsive to change• Faster, cheaper• More effective• Personally enjoyable• Not repeatable• Not easily automated• Little or no documentation• Hard to manage• Hard to scale• Opaque• Not auditable, measurable.


We need to change the testing metaphor

Surveying and testing – the new process?• The “tester as surveyor” affects the

relationship of testing to development• A new style of testing process emerges:– No more documentation created in a knowledge

vacuum– Iterative, incremental knowledge capture closely

aligned with the emergent delivery of functionality• Could this be an Agile test process at last?• It fits the increasingly popular Continuous

Delivery, DevOps development approaches.


New Model Testing changes what

testers & developers need to

learnRoles of developers/testers become

blurredPerhaps they are not different disciplines

after all?Why have a tester certification?

Do Testers Need to Learn How to

Code?Some, perhaps many will

have to

Testers don’t own testing anymore

Be a TestMaster"Assurance through the

lifecycle"

The phase after development is REWORK, not

TestingTesting must align with

development, not compete with it or rescue it

A New Model for Testing@

paul_gerrard

Paul [email protected]

gerrardconsulting.com

http://testaxioms.com/?q=node/25

a new model for testing

Education