evolving from automated to continous testing for agile and devops

Paraso& Proprietary and Confiden1al 1

5/25/16

Evolving from Automated to Continuous Testing

for Agile and DevOps

[email protected] Solution Architect

Paraso& Proprietary and Confiden1al 2 Paraso& Proprietary and Confiden1al 2

To start with…

§  Created: BC 196

§  Found: 1799

§  London: 1801

§  Bri1sh Museum: 1802

§  Transla1on: 1822 §  Egyp1an hieroglyphs §  Demo1c script §  Ancient Greek


Anno 2016, you cannot keep things for yourself…


Impact of Faulty So&ware

-‐5.70% -‐1.9B


Anno 2016: So&ware Failures = Headlines

-$2.55 Bn -4.08%

The day of the announcement companies lost an average of shareholder value

So&ware failures make headline news— eroding customer confidence, shareholder value and brand equity


Escala1ng Cost of Failure: Public companies


Re-‐Inven1ng so&ware development

The evolu1on of the so#ware development lifecycle (SDLC) over the past 3 years is forcing the transforma1on of so&ware tes1ng

Transforma)on Business Impact •  Business risk associated with

so1ware failure at an all 4me high •  Public companies lose an average of

$2.5 Bn in market cap on the day of the announcement

•  “Digital Industrial Revolu4on” makes all a so1ware company

•  Trend toward Dev/Test insourcing as compe44ve advantage is via so1ware

•  Strong adop4on of “agile” development prac4ces

•  Forces more technical tes4ng, early. Agile is forcing “shi1-‐le1”

•  DevOps is applying lean prac4ces to SDLC

•  DevOps is exposing 20 years of process barriers protected by silos and culture

•  Cloud has disrupted tradi4onal SDLC and is driving down costs

•  Switching costs associated with applica4ons at an all 4me low


How to manage the Disrup1on ?

•  Security •  Safety Cri1cal •  FDA • Automo1ve • Aerospace

Compliance • DevOps • Agile •  Lean • Bi-‐Modal

Ini1a1ves • Microservices • Containers • Cloud •  IoT

Technology • Produc1on Outage • Applica1on Slow • User Experience • Connec1vity

Failure

Does the release candidate have an acceptable level of risk?

Mitigate Business Risk

Paraso& Proprietary and Confiden1al 9

Automa1on expands through whole SDLC

Con1nuous Delivery

Con1nuous Integra1on

Delivery Team

Version Control

Build & Unit Test

Automated Acceptance

Tests

User Acceptance

Tests Release

Check In Trigger

Trigger Trigger

Trigger Trigger Approval Approval

Check In

Check In

Feedback

Feedback

Con1nuous Deployment

Produc1on

Business


Fixed Fixed

Impact of Test in the Evolving SDLC (1)

Plan Develop Build Test Release

Test as a 1me-‐boxed event is the prevailing method

“Are we done tes1ng?”



Plan Develop

Build Test

Release

Build Test

Build Test

Build Test

Build Test

Build Test

Build Test

Build Test

Build Test

Build Test

Build Test

“Does the release candidate have an acceptable level of risk?”



Speeding up the conveyor belt does not yield be>er results…

… you need Quality @ Speed

The new so&ware paradigm requires the business to define and establish automated quality gates that con1nuously mi1gate the risk of business failure


How to achieve this Quality @ Speed ?

Take the top-‐down view: §  Business risks drive quality ac1vi1es §  Team collaborates on the defini1on of risk §  Baseline and threshold criteria measured and

monitored §  Results of quality ac1vi1es are visible and

translated for all levels of business

This approach will allow you to go from “automated” to “con1nuous”…


From Automated to Con1nuous (1)


From Automated to Con1nuous (2)

The steps required for moving from “Automated” to “Con1nuous”

§  Business expecta1ons clearly defined §  Business risks iden1fied §  Per applica1on, team, release

§  Defects automa1cally priori1zed versus business drivers §  No defect le& behind §  Mi1gate business risks

§  Tes1ng in ‘complete’ test environments con1nuously §  Leveraging simula1on – Service Virtualiza1on §  Protec1ng the real-‐user experience business process

§  Feedback loop for defect preven1on §  Learn by capturing in paoerns §  Defect preven1on prac1ces enhanced


Con1nuous Tes1ng: Quality Product & Process

Automated Tes)ng provides the “sensors” (detec)on processes) §  Fixing defects is the expected outcome §  Iden1fies opportuni1es for defect preven1on §  Prevents a release candidate from going to the next stage of the release cycle

Con)nuous Tes)ng addresses the broader challenge §  Fix the product §  Op1mize the process (improve the

effec1veness of the “sensors”)

Con1nuously refine the process in light of business risks as well as so&ware quality

Con)nuous Tes)ng addresses the broader challenge

§  Fix the product §  Op1mize the process (improve the effec1veness of the “sensors”) §  Con1nuously refine the process in light of business risks as well as so&ware

quality


Con1nuous Tes1ng

Con)nuous Tes)ng


Transforming the Iden1fica1on of Risk

From To

Causal Observa1ons Probabilis1c

Defect Documenta1on Most Likely Causes

Only Structured Data Structured and Unstructured

Filters Business Policies

Ad Hoc Automated

Tool Dependent Tool Independent

Business Objectives


Con1nuous Tes1ng mi1gates business risk

Why Con)nuous Tes)ng?

Systema1c, centralized decision making to factor

business risk into SDLC

Safety net to allow developers to bring innova1ons to market faster

A feedback system for beoer trade-‐off decisions between release scope,

1me and quality


Con1nuous Tes1ng builds on automa1on

Con)nuous Tes)ng

Not more automa1on, beoer automa1on

Business and development leaders agree upon and automate the mi1ga1on of risks throughout the SDLC

Both business and technical managers make beoer trade-‐off decisions between

release scope, 1me and quality

Organiza1ons evolve from just measuring product quality to measuring the process effec1veness


Con1nuous Tes1ng monitors the process

Con)nuous Tes)ng

Con)nuous Deployment

Con)nuous Delivery

Con)nuous Release

Con)nuous Integra)on

§  Con1nuous Tes1ng focuses on a comprehensive view of business risk

§  Con1nuous Tes1ng provides a workflow for priori1zing fixes and quality improvements, and preven1ng recurrence, according to business risk and business priori1es

Con)nuous Tes)ng enables con)nuous improvement


Feature Release

Business Stakeholder Developer Customer Support

Con1nuous Tes1ng in ac1on: Use Quality Gates

SoKware Feature

Con1nuous measurements mean con1nuous refinement of the process

Real-‐1me feedback from

Objec1ve assessment and go/no go Defects are eliminated at the point that they are easiest to fix

Business Stakeholder Developer Customer Support

Quality gates: Organiza1ons can automa1cally promote so&ware through quality gates when business expecta1ons have been met


Summary: The elements of Con1nuous Tes1ng

Con)nuous Tes)ng re-‐posi)ons the ques)on from “are you done tes)ng?” to “is the level of risk understood and accepted?”

Con)nuous Tes)ng

Risk Assessment

Policy Analysis

Requirements Traceability

Test Environment

Access

Test Op)miza)on

Advanced Analysis

Ensure access to complete test environments

Automate defect preven1on and Policy measurement

Expand test coverage and measure test effec1veness

Define ac1onable prac1ces

Connect func1onal with non-‐func1onal requirements

Process improvement opportuni1es


How we can help ?

Does the release candidate have an acceptable level of risk?

Mitigate Business Risk

Con1nuous Tes1ng •  Test Configura1on and Simula1on •  Test Environment Management • Defect Detec1on • Change Control

Development Tes1ng • Risk Defini1on • Process Control • Defect Preven1on • Visibility

§ Methodology: §  Research:

§  SW to support development & con1nuous tes1ng:


Let us help you to decipher your SDLC…


Paraso& Con1nuous Tes1ng

Paraso& develops automated so&ware quality solu1ons that prevent and detect risks associated with applica1on failure

Emphasizes business expecta1ons to mi1gate

business risks

Enables development

tes1ng

Promotes early detec1on and remedia1on

Facilitates integra1on tes1ng

Provides ubiquitous test environment access with Service Virtualiza1on

Assists in end-‐to-‐end

system tes1ng

The ParasoK Con)nuous Tes)ng

evolving from automated to continous testing for agile and devops

Software