lights-out testing for lights-on business

Download Lights-Out Testing for Lights-On Business

Post on 24-Jan-2018

54 views

Category:

Software

1 download

Embed Size (px)

TRANSCRIPT

  • 1

    Lights-Out Testing for Lights-On Business

    Ty Henson

    September 20, 2017

  • 2

    Digital RiskIts not just about security, hackers, and data theft

  • 3

    Agenda

    Effective automation testing

    Creating on-demand automation

    The right tool for the job

    Presented by Ty HensonWorksoft, Senior Solution Architect

  • 4

    Late Defect Discovery is Expensive and Disruptive!

    0

    9599

    22781331

    $18 ($0)

    $180 ($1.7 M)

    $360 ($820K)

    $1,800 ($2.3 M)

    $0

    $200

    $400

    $600

    $800

    $1,000

    $1,200

    $1,400

    $1,600

    $1,800

    $2,000

    0

    2,000

    4,000

    6,000

    8,000

    10,000

    12,000

    Discovered inRequirements/Specs

    Discovered in TestExecution

    Discovered in UAT Discovered inProduction

    Defect Correction Cost (5 Project Sample)

    Defects Cost to correct per defect

  • 5

    Mainframe Client Server Web 1.0 Web 2.0 / Mobile HTML 5 VR & Biometrics

    1980 2000

    1990

    2020

    2010 2030

    Modern Day UI EvolutionAs the way applications are written changes. The way we test must also.

  • 6

    Business Analysts/QA

    Does my Business Work?

    Developers/Test Engineers

    Does my Software Work?

    UI Tests The Bulk of Business Application Testing

    6

    End to End (UI)

    Integration ( API)

    Unit (Code)

    UI Testing Remains the Most Challenging to Automate.

    Last 10% takes 90% of the time!

  • 7

    Digital Business ProcessesComplex

  • 8

    1 DiscoverBusiness Analysts/Super Users

    2 AutomateQuality Assurance/Automation Professionals

    3 RunIT Operations/Project Management

    Capture, visualize, document & analyze end-to-end business processes

    Worksoft Analyze

    Build and maintain a reusable portfolio of business process test automation

    Worksoft Analyze, Worksoft Certify

    Run automation at industrial scale for projects & ongoing operations

    Worksoft Certify,Impact, Execution Manager, Business Process Procedure

    The Worksoft Solution

  • 9

    Demonstration

  • 10

    Agenda

    Effective automation testing

    Cross-Platform End to End

    Creating on-demand automation

    The right tool for the job

  • 11

  • 12

    End to End Process

    Enter Order

    Outbound Delivery

    Post Goods Issue

    Invoicing

    Test Data Concepts

  • 13

    End to End Process

    + Add Inventory

    Enter Order

    Outbound Delivery

    Post Goods Issue

    Invoicing

    Test Data Concepts

  • 14

    End to End Process

    + Find or Generate Materials

    + Add Inventory

    Enter Order

    Outbound Delivery

    Post Goods Issue

    Invoicing

    + Cleanup for Repeatability

    Test Data Concepts

  • 15

    Agenda

    Effective automation testing

    Cross-Platform End to End

    Creating on-demand automation

    Self-sufficient and repeatable

    The right tool for the job

  • 16

    Hundreds of tests under management

    across multiple applications

    Tests must often be scheduled in

    sequence with complex dependencies

    Difficult to manage distributed and

    diverse testing resources

    Central monitoring can be a challenge

    Testing time is strictly limited and

    closely scrutinized

    The Trouble with Automation

  • 17

    UI Automation Challenges

    Access Orchestration Scale

  • 18

    1 DiscoverBusiness Analysts/Super Users

    2 AutomateQuality Assurance/Automation Professionals

    3 RunIT Operations/Project Management

    Capture, visualize, document & analyze end-to-end business processes

    Worksoft Analyze

    Build and maintain a reusable portfolio of business process test automation

    Worksoft Analyze, Worksoft Certify

    Run automation at industrial scale for projects & ongoing operations

    Worksoft Certify,Impact, Execution Manager, Business Process Procedure

    The Worksoft Solution

  • 19

    Industrial Scale Intelligent Scheduling

    Distributable agents to [geographically distributed] physical or virtual workstations

    Mass parallel execution ability

    Automatic load balanced execution

    Dependencies to control intelligent execution

    Lights Out Testing and Continuous TestingIndustrial Scale Intelligent Scheduling and Interoperable Execution

    Interoperable Execution

    Providing applications like those above the ability to trigger automation execution

    Rest API calls

    Command line interface (CLI)

    And more

  • 20

    Running Certify From CI - Jenkins

    Jenkins builds and creates build number

    Build Number

    Call Execution Manager

    Execution Manager

    REST

    Run Certify Tests

    Get Status

    Remote Test Lab 1CI/CD Server

    EMAgent

    EMAgent

    EMAgent

    On premise Test Lab 2

  • 21

    Demonstration

  • 22

    Lights Out Testing (LOT) provides the ability to

    execute a battery of integrated tests at pre-

    determined times. These tests will be re-runable

    and will be based on exception reporting,

    requiring manual intervention only if the test fails.

    Tests can be scheduled at predetermined times

    and leverage:

    Off-hour bandwidth

    Use of infrastructure and personnel for other

    tasks

    The execution of large, data centric integrated

    tests during off hours

    Elimination of manual validation and audit

    What is Lights Out Testing (LOT)?Because were covering all

    of the business processes

    in an automated fashion,

    were finding defects that

    also give us a snapshot of

    the data thats causing a

    script to fail. Because were

    now on a 24-hour testing

    cycle, we can isolate the

    cause more efficiently,

    which also results in cost

    savings.

    Sada Krishnan, IT Project

    Manager, Reliant

  • 23

    Cross Platform End to End Automation

    Bidirectional Communication with Other Applications and capabilities

    And more

    SOA Rest APIMock

    Services

    File

    SystemRelationa

    l Queries

    Microsoft

    OfficeExecutio

    n

    Execution Manager

    Interoperability

  • 24

    Quality and Validation for DevOps and Packaged Application Testing SDLC

    Requirements

    Test Cases Documented

    Architect Develop

    Build Unit tests

    CI &

    Unit Test

    Run Unit Tests

    CD to Staging Integration

    API Functional

    API Load Testing

    Application Functional

    UAT

    End to End

    Requirement validation

    Exploratory Testing

    CD to Prod Smoke

    Production Validation

    Prepare

    Test Cases Documented

    Discover

    Business process changes

    Integration

    Realize

    End to End

    Requirement validation

    Exploratory Testing

    Go Live Run

    Production Validation

    Developer based testing specific to

    development of custom applications

    Application functional testing common to

    both Testing SDLC UI and end user

    focus

    Ide

    a

    Ma

    jor

    Developer

    Testing

  • 25

    Supporting Continuous Integration and Continuous Delivery

    Develop

    Code Check in

    BacklogGroom Stories

    Sprint starts

    CI Continuous Integration CD Continuous Delivery

    Development

    Sprints

    Requirements

    Test Cases Documented

    Architect Develop

    Build Unit tests

    CI &

    Unit Test

    Run Unit Tests

    CD to Staging Integration

    API Functional

    API Load Testing

    Application Functional

    UAT

    End to End

    Requirement validation

    Exploratory Testing

    CD to Prod Smoke

    Production Validation

    Worksoft

    Execution

    Manager

    Application is deployed and connected

    End-to-end is available

    Development testing when

    code is not connected

  • 26

    Digital Business ProcessesComplex

  • 27

    Agenda

    Effective automation testing

    Cross-Platform End to End

    Creating on-demand automation

    Self-sufficient and repeatable

    The right tool for the job

    Presented by Ty HensonWorksoft, Senior Solution Architect

  • 28

    Case Study:

    US Federal Government Program

    10 million+ participants; 400 core processes & 500,000

    business process steps tested nightly

    Virtual

    Machines (150+)

  • 29

    Case Study:

    Global Luxury Goods Manufacturer

    Enterprise wide automation effort for end-to-end business

    process validation with Worksoft Certify.

    Significant reduction in production defects for SAP, Web

    applications and more.

    600 hours of automation

    running daily on 30+ virtual machines

    55,000 hours

    of manual

    labor annually

    replaced

    80% business

    coverage of 500

    core processes.

    SCALE SAVINGS COVERAGE

  • 30