cognitive adaptive testing (analytics-driven digital quality)

1 © 2017 CA. ALL RIGHTS RESERVED.

Cognitive Adaptive Testing based on analytics driven quality

Jonathon WrightDirector (R&D) – Automation, DevOps & Digital [email protected] | linkedin.com/in/automation | slideshare.com/jonathon_wright

© 2017 CA. ALL RIGHTS RESERVED.


Disclaimer

Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current information and resource allocations as of June 1st and is subject to change or withdrawal by CA at any time without notice. The development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.

Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-available basis. The information in this presentation is not deemed to be incorporated into any contract.

Copyright © 2017 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies

THIS PRESENTATION IS FOR YOUR INFORMATIONAL PURPOSES ONLY. CA assumes no responsibility for the accuracy or completeness of the information. TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENT “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. In no event will CA be liable for any loss or damage, direct or indirect, in connection with this presentation, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised in advance of the possibility of such damages.


Abstract

In the digital economy, everyone’s in the software business, and the landscape is constantly changing. The traditional approach to providing business value through lengthy plan–build–test–deploy cycles no longer delivers value in an increasingly competitive, consumer–centric marketplace. Merely compressing these cycles by adopting iterative or agile methods doesn’t address the innovation, speed–to–market, quality and resilience challenges organisations face.

Shifting to a more dynamic, adaptive and continuous model calls for new tools and practices that accelerate communication, collaboration, integration, measurement and automation. Whether labelled this as digital, shift–left or DevOps, understanding the detail of this journey is an essential part to becoming digital–testing ready.

Join Jonathon as he explores the fundamentals of continuous adaptive testing that will enable the continuous delivery pipeline, and to provide instant, smart, cognitive adaptive technology that will power the next generation of insight and value–driven delivery capable platforms. Revolutionising how you test in the digital future will drive smart business decisions and end costly, reactive rework and errors.

Jonathon Wright

CA

Director (R&D) –DevOps, Automation & Digital Assurance


Testing in the New Digital World

14th March 2017


The success of our technologies largely depend

on the fact that, while we were speculating about the

possibility of ultra-intelligence, we increasingly

enveloped the world in so many devices, sensors,

application and data that it became an IT-friendly

environment, where technologies could replace us

without having any understanding, intentions,

interpretations, emotional states, semantic skills,

consciousness, self-awareness or flexible intelligence.

Memory as in algorithms and immense datasets now

outperforms intelligence. MegaTech, Technology in 2050, The Economist, Daniel Franklin, 2015

“

”

THE NEED FOR SPEED

94%of executives face increased pressure to release appsmore quickly

“

Journey to the Digital Testing

© 2017 CA. ALL RIGHTS RESERVED.5

Based on 2014 Vanson Bourne study commissioned by CA


Digital Testing, Quality & Assurance

DIGITAL TESTING, QUALITY & ASSURANCE

http://bit.ly/digital-assurance http://a.co/jlxMHWp


IdeasBeliefs, Values, Culture

Heuristics

Behaviors

Value

AgileLean

Continuous Testing 101Webcast Series – Value-Driven Delivery – Episode I

Methods and Skills

http://catechnologies.podbean.comRay Arell


QUALITY

2/3of business leaders say the future of their business depends on the quality of their software2

VELOCITY

94%of executives face increased pressure to release apps more quickly1

1. 2014 Vanson Bourne study commissioned by CA2. “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015 3. “DevOps and the Cost of Downtime” – IDC 2014

“Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015


COST

25%of a single application’s development and operations costs is wasteful3

While Ensuring a Superior Digital Experience!

Digital TestingDelivering Quality at Speed

SHIFT RIGHT


Cognitive Adaptive Testing (CAT)



Digital Testing

TESTING IN THE DIGITAL WORLD

COGNITIVE ADAPTIVE ADOPTION (DESIGNOPS)

COGNITIVE ADAPTIVE DELIVERY (VALUE-DRIVEN)

COGNITIVE ADAPTIVE LEARNING (ANALYTICS-DRIVEN)

COGNITIVE ADAPTIVE TECHNOLOGY (DIAGNOSIS ENGINE)

WHERE NEXT?

1

2

3

4

5

6

11 © 2016 CA. ALL RIGHTS RESERVED.11 © 2017 CA. ALL RIGHTS RESERVED.

CA Portfolio Management

Unify long term strategy, investment and portfolio planning.

CA Project Management

Collaboratively plan, prioritize and track work across the enterprise.

Develop & Test

Synchronize efforts to dramatically speed app development & increase quality

Manage & Monitor

Make a great customer experience a competitive advantage

Release & Deploy

Control the release process, to continuously advance application quality, improve the customer experience and reduce costs.

ContinuousDelivery

Minimum Viable Experience

Minimum Viable Quality

Value-Driven Delivery

BUSINESS INITIATIVE

Complexity-Informed Organizational Change

BUSINESS VALUE

MinimumViableProduct

CUSTOMER

THINK

MAKE

Digital TestingComplexity-Informed Organizational Change


DesignOpsCreate and deliver Ideas to Outcomes

Pervasive Security

Redefine how work is planned, managed and

executed

Agile Management

Provide a flawless app experience optimized for

performance

Agile Operations

Accelerate and streamline development, testing and

release

Continuous Delivery

Continuous Development Continuous Adaptive Testing Continuous Release

Digital TestingValue-Driven Delivery (Shift-Left & Shift-Right)


DIGITAL ASSURANCE & TESTING

VALUE-DRIVEN DELIVERY

1Cognitive Adaptive Delivery

2Cognitive Adaptive Learning

4Cognitive Adaptive

Technology

3Cognitive AdaptiveTesting

Digital TestingCognitive Adaptive Adoption (CA-A)


Automate manual steps and scripts—eliminating repetitive, error-prone tasks

Design reusable deployment workflows

Integrate best-in-class tools dev through production

Gain visibility and transparency

Close feedback loops throughout the pipeline

Efficiently release multi-team, cross-app, composite releases

Define and enforce continuous delivery best practices

OPTIMIZED

AUTOMATED

CONTINUOUS

Monitor, measure and analyze release processes

Maximize quality as the velocity and volume grows

Improve business execution

Measure and improve team productivity and maturity

3

21

Digital TestingNavigate to Continuous Adaptive Delivery (CA-D)


The ability to reliably release high-quality solutions at any time

COGNITIVE ADAPTIVE TESTING

Accelerate Deliveries

Reduce Errors

Manage Complexity

Increase Visibility

Drive Collaboration

Continually Improve

Digital TestingContinuous Adaptive Testing (CAT)


Digital TestingWhat is analytics driven quality?

Model ProgressiveRefinement

Analytics-Driven


> 50%large organizations globally will compete using advanced analytics & proprietary algorithms

40%enterprises' net-new analytics investment will go to predictive / prescriptive

75%technology-oriented business intelligence competency centers will evolve to focus on information value generation

Source: Gartner

Digital TestingIntelligent Analytics is the New Battleground


SOI

Visualization

(Big) Data

Analytics

Inference (Machine

Learning & AI)

Intelligent Process

Automation

Digital Technology Integrations

Digital TestingSystems of Intelligence


Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas


Portfolio backlog prioritization and

value analysis

Improved scheduling and

capacity

Validated requirements in

backlog

Improved estimates

Predict failures based on code

commits

Dynamic code promotion

criteria

Predict defects/failures

Optimized Tests

Scenario identification

based on trends

DX Scores

Anomaly detection

PLAN DEFINE DEVELOP TEST RELEASE OPERATEPhase

Examples of Cognitive Adaptive

Intelligence

Examples of Outcomes

Investment decision analytics

Code quality

analytics

TestOptimization

& Defectprediction

A/B Testing

& Canary

Log & DX Analytics

DX-basedrequirements


Digital Testing

Cognitive Adaptive Intelligence (CAi)

Systems of Intelligence


AUTOMATION MAINTENANCE

STATIC RULES BIG (un-structured) DATA

IOT, IOE

CX / DX SECURITY

STRUCTUREDAUTOMATION

Digital TestingKey Challenges Posed by Adaptive Digital Systems



Failure detection and prevention

Scripted automation

Limited process automation based on static rules

Individual metrics based on structured data in lifecycle silo

Limited insight and perspectives

Reactive: deals with “Past”

Failure prediction

Automate the automation

Intelligent process automation based on dynamic inference

Analytics based on large scale structured and unstructured data across the lifecycle

Predictive: from “What did happen?” to “What would happen?

TRADITIONAL AUTOMATION

COGNITIVEAUTOMATION

Digital TestingIntelligent Automation Characteristics



AUTOMATED & SELF-HEALING AUTOMATION

DYNAMIC/ADAPTIVE RULES

BIG DATA ANALYTICS

IOT-Ops

DX-Ops SEC-Ops

INTELIGENT / ADAPTIVE / COGNITIVE AUTOMATION

+ +

+

+

+

+

+

Digital TestingChallenges addressed by Cognitive Adaptive Testing (CAT)



REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO

AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS

CONTINUOUS DELIVERY MAINFRAME SECURITY


API analyticsRelease management dashboard Access incident responseReal-time app performance

Digital TestingContinuous Adaptive Delivery (CA-D)

CA Adaptive Data Store


Best of Breed Open Source Technologies

(Hadoop, Spark, Kafka, Elasticsearch)

Industry Proven Architecture

(Lambda & Micro Services)

Single Code Line, Multiple Deployment Options

(Cloud, On Prem, Mainframe)

Encapsulated Complex Technology through Simple

Consumable Interfaces

(RESTful APIs)

CA Analytics

Digital TestingContinuous Adaptive Intelligence (CA-I)



DATA COLLECTORS

Dat

a A

ggre

gato

rAgents

…

Probes

Machine/Log File Data

DATA-DRIVEN APPS

Existing CA Apps

Custom-build Apps

New CA Apps

Partner Apps

ANALYTICS ENGINE

Dat

a In

ject

ion

AP

I

Dat

a A

cce

ss A

PI

Service Layer

Dat

a Ex

trac

tio

n

Data Processing (Speed)

Data Processing (Batch)

Data Bus

Inspired by “Analytics at CA – What’s Cooking? Project Jarvis” Wednesday, November 16: 4:30 PM - 05:15 PMCA Accelerator Zone

Digital TestingCA Reference Architecture – Project Jarvis


Digital Testing – Diagnosis Analytics Engine



Emergent embedded AIfor neural networks

Digital TechnologyCognitive Adaptive Intelligence (iCA)

Singularity Ultra-Intelligence (2030)

“Let an ultra-intelligent machine be defined as a machine that far surpass all the intellectual activates of any person however clever” 1960s


Mode 0 - Descriptive

Cognitive Adaptive Testing (CAT):

1. “How long will it take to test these new features?” --- How long historically (test cases / execution time)2. “What is the associated risk rating of the new platform?” --- Failure rate (cyclomatic complexity > 13)3. “How much technical debt do we have?” 4. “What is the defect density and prediction for this solution?”5. “What is the current conference rating of the solution?”6. “What is the value of this capability?”

How will Intelligent Analytics help?

Digital TestingExample use cases for Intelligent Analytics


What were the past defect trends?

What were the root cause of the past defects?

What defects are we likely to find?

What preventive measures should we take?

Have we overlooked anything for defect prevention?

Predictive analytics

Descriptive Analytics

Diagnostic Analytics

Prescriptive Analytics

Deep Learning

Diagnosis Analytics EngineExamples of Analytics Use case

Cognitive Adaptive Intelligence (CAi)



How do you pin point the failures within the release

pipeline?

Model-Driven Design of Tests from Use Cases 1

2

3

Execute ‘Automation as Code’ from the Model

Pinpoint Failure and Diagnose Issue from Results

“Easy, soon as you check in the new features ready for test, the automation will run then the results are analysed automatically and if we have any failures then it identified them and raise them as bugs!

Diagnosis Analytics EngineTest in DevOps (TiD)


Mode 2 - Predictive

Cognitive Adaptive Testing (CAT):

1. “Can you create me all the tests I need to fully test this system?”2. “How do I successfully test this new feature?”3. “How much testing has been done already?”4. “Have you detected any anomalies within the system?”5. “How much testing will be enough?”

How will Intelligent Analytics help?

Digital TestingExample use cases for Intelligent Analytics


Use Case A – Diagnosis Analytics Engine

Cognitive Learning

Diagnosis Analytics Engine.

Pinpoint Failure Analysis.

Actionable Insight.

Machine Deep Learning.

Always On Test (SaaS)

Test Automation as a Service.

Root Cause Analysis (RCA).

Support for Azure & AWS..

On Prem

Dynamically generated Virtual Automation Environment Provisioning.

Dynamically generated Tests based on Cognitive Adaptive Testing.

0 1 2


Where next?


VALUE-DRIVEN DELIVERY

DIGITAL ASSURANCE & TESTING

1Cognitive Adaptive Delivery

2Cognitive Adaptive Learning

4Cognitive Adaptive

Technology

3Cognitive AdaptiveTesting

Cognitive Adaptive Technology (CA-t)Cognitive Adaptive Adoption (CA-A)

50 © 2017 CA. ALL RIGHTS RESERVED.© 2016 CA. ALL RIGHTS RESERVED.© 2017 CA. ALL RIGHTS RESERVED.

BRINGS TRUE ARTIFICIAL INTELLIGENCE (AI) TO LIFECYCLE VIRTUALIZATION Virtualize services without requiring any knowledge or

decoding of the service protocols

Applies a genome sequence alignment algorithm, discovers byte-level patterns in message protocols

Now virtualize a much wider range of protocols without requiring a new DPH

HIGH ACCURACY: 99.6 – 100%

Increased speed and accuracy with Entropy Weighting + Message Clustering

The more data a service observes the data, more intelligent it becomes

Perfect for performance testing where we deal with tons of data

Cognitive Adaptive Learning (CAI)Current Generation CA Testing™


Cognitive Adaptive Technology (CA-T)Next Generation of Digital Experiences (DX)

1. Digital Mobile, Chapter 2, Digital Experiences (DX), April 2017, Jonathon Wright


Q & A


RBC Maintenance time reduced from 7.5 hours to 2 minutesCGI-Fiko: Estimated 30% effort saved on managing dataRogers: 320% over-testing reducedAMEX : 16% to 100% coverage; 5 hour test creation time reduced to 50 minutes

Customer Benefit Anecdotes Key PersonasBAs, QA Engineer, Testers, Test Automation Engineers, Site Reliability Engineers, SDET & Test in DevOps

64% of Defect Cost originate in reqs phase

Automate the automationmaintenance of test assets following change

95% reduction in time

to create test assets

- BPM Tools- AC/Jira/ALM/TFS- Testing FrameworksHyderabad Business School GITAM University Quality

Flaws: Issues and Challenges in Software Development 2012

Estimations derived from analysis of benchmark data which is a composite from multiple sources.

CA Agile Requirements Designer

CA Product modules:

CA Agile Requirements Designer

+ Automation Builder+ Data Builder+ Automator (Runtime)

Use Cases Collection:• Generate Model-Based Design

Users Stories & Requirements (ARD)• Automate Test Design / Optimize

Test Cases and Coverage• Generate Automation from Model• Link your tests with test data (TDM)• Quickly react to change

ROI Value Proposition Integration PointsChallenge


• Strategic initiative to drive innovation velocity and accelerate developer productivity.

• CA has earned the position of “trusted advisor”, helping to form their “OSI Model for Continuous Delivery”.

• Leveraging ARD, TDM, SV, and RA in combination.

GENERAL MOTORS Key PersonasTest Data Engineer, QA Manager, Quality Engineer, Testers, Business Analysts, Software Engineers, Release Managers

Real World - Value-Driven Delivery (VDD)

Product Feature Areas• CA Agile Requirements Designer• CA Test Data Manager• CA Release Automation• CA Service Virtualization• CA Application Test• CA Blazemeter

Use Cases Collection• Model-Driven Automation• Environment Provisioning• Functional / Integration• Virtualize Constraints• API Testing• Performance Testing• Digital Experiences

TransformTesting organizations toOptimize efficiency

React To changingRequirements faster thanEver before.

GenerateTest case automationInstead of writing themmanually

Optimize environment and data availability through automated delivery

Why CA: Don’t Take Our Word For it

CA is Positioned as a Leader in the Gartner MQ for ARA … Forrester Wave SV Leader

Faster time to deploy releases*

20x

Reduced testing costs & improved efficiency*

25%

Reduction in production errors*

98%

Differentiation Customer Success

5 of the top 5Telecom Companies

4 of the top 5 Retailers

4 of the top 5 Banks

Market Validation

Nominated in the “Best Use of Technology in a Project” category for

work with the London Metal Exchange

*Based on Global 500*Forrest TEI Research


Blogs #ExcuseFreeTesting – Legacy is your Legacy!

https://dzone.com/articles/legacy-is-our-legacy-1

https://dzone.com/articles/legacy-is-our-legacy-1