reality shock j aracil

Madrid, 26th to 28th May 2014

Reality Shock:

Performance Testing

Jose Aracil - Operations Manager

[email protected]

@pep0 - @globetesting


A radically different kind of user

System-centric User-centric

Users born after 1980

Mainframe DevicesWebClient/Server

Time


2014

A tipping point for app. delivery

“New application

projects surge in 2014

and beyond.”

“Social, mobile, analytics,

cloud have reached the

tipping point in 2014.”

201536 per app

2020120 per app

2010 4 per app

Number of releases

per app per year

• Apps everywhere

• Accessed through multiple devices

• 30x increase in application releases

• Closed feedback loop with customers

• Focus on time-to-value and ROI

What we expect:


Changes are coming faster

Cloud and composite applications are increasingly complex

Customers demand mobile access services

The pace of change is only increasing

Project and delivery teams are both agile and global

Businesses need applications to connect with:

• Customers

• Partners

• Suppliers…


Dev/test lab

Developer

And the need for speed changes everything

Tester

Agile Development

“The new QA is extensively

automated and executed

by a continuous

integration process”--

Forrester Research

Provision

environments at

“agile speed”

Shift left, test

continuously

Agile

Development,

weekly drops


Continuous Testing for

faster app delivery“Shift-Left”

With Confidence

Now Continuous Testing is more critical than ever

• Improve development

efficiency

• Reduce the test burden

• Accelerate application

delivery

Design Develop

Provision

Env.

Deploy

Test App.

Run test

Test results


Application usage has changed

Performance testing must change

Application development has changed


You must

leave the black

box


Sorry… I changed my mind!


Sorry… I changed my mind!updated


Planning Test Strategy

Reporting Final report

Preparation

Tool setupTest Env. + Data Test case design

Monitor config. Scripting

Smoke Test

Execution

Baseline

Soak Test Load Testing Other tests

Stress

Performance Testing

An

ális

is +

Cha

nge

s+

Rep

ort


Planning

Preparation

Execution

Reporting

Test Strategy

Tool setupTest Env. + Data Test case design

Monitor config. Scripting

Smoke Test

Baseline

Soak Test Load Testing Other tests

Stress

Final report

Performance Testing

An

ális

is +

Cha

nge

s+

Rep

ort

But what get’s in the way?

Dependencies


Good for one project


New set of problems…

Multiple

projects

Continuous

Integration

Auditing

Multiple people

Deployment of

LoadGenerators

Dependencies

Asset sharing

Reporting

SecurityCollaboration

Best

practices

Network

conditions

Cloud

Low application

maturity

Agile (good and bad)


Project based testing challenges

Project AHP LoadRunner 8.1

1500 Vusers

Project BHP LoadRunner 11.0

1500 Vusers

Project COther load test SW

2500 Vusers

Project DNo load test budget

“Hope for the best”

Unrealistic “code freeze”

No sharing of best practices & skills – each project is unique and isolated

Duplicated resources (HW and SW) in separate projects

Some test cases not fully working (missing components)

End-to-end behaviour not fully tester


End to end

Search PayBuy Ship Track

EXAMPLE:

The Online Shopping

Process

Product PaymentCustomer Profiles Shipping

Track packagePublic CloudTraditional Managed CloudComposite

APIs enable integration

Each have SLAs and performance

dependencies

Components anywhere– in-house or cloud


Virtualisation Four-legged table

User

Network

Service

Data

User

• Virtualise user conditions to simulate real world conditions

Network

• Virtualise network conditions to simulate production architecture and real user conditions

Service

• Virtualise services and service interactions to simulate E2E business processes

Data

• Virtualise production data and transaction responses to simulate application request and responses


We are now able to truly apply a

agile approached to load testing

given the reduction in scripting and

build time.

Our scripting time is reduced by

more than 75% and resource

productivity across projects has

improved by more that 50%.

Source: TechValidate. TVID: B40-ED8-557

User


Enable Agile performance DeliveryUser

Test

Build Load Test scripts during Dev

Run Unit Tests in Load Test Scenarios

Add Performance Tests as build steps in continuous integration tools


Mobile is different

Analyze Mobile Performance on Network

Backend

monitoring

Mobile

performance

automation

Mobile Network

characteristics

Network

Identify bottlenecks before end users

Test in both good and BAD network connections

Understand impact of mobile on entire system


Existing Infrastructure

Service Virtualization Concept

Mainframe

JDBC

SOAP

RFC/IDO

C

MQ/CIC

S

Third Party

Application Under Test

Mobile

App

Web

browser

Composite

Application

API

SAP System

Existing database

Web service and

Legacy application

REST

Pay-per-transaction

Under

construction

Service


Existing Infrastructure

Service Virtualization Concept

Virtual services are created to

stand in for the real service when

developers or testers need

access

Mainframe

JDBC

SOAP

RFC/IDO

C

MQ/CIC

S

Third Party

Application Under Test

Mobile

App

Web

browser

Composite

Application

API

SAP System

Existing database

Web service and

Legacy application

REST

Pay-per-transaction

Under

construction

Service

Virtualization

SOAP

REST

JDBC

MQ

RFC

CICS

Simulatio

n

Data

Perf.

Config

Service


Service Virtualization is the answer

Functional test

Performance Test

Regression test

Dependencies


Service Virtualization is the answer

Functional test

Performance Test

Regression test

DependenciesVirtualization

Allow frequent access to app components, APIs and services

Remove constraints with virtual version

No access to a component? Use virtual component

Enable performance test teams to set up simulated services


Bring production data into performance

testing as baseline

• Collect, analyze and compare

performance data from production

• Updated log file analysis of production

data for IIS

• Integrate with Google Analytics

DataPerformance application lifecycle

Value:

Performance testing closer to reality with accurate,

realistic, reliable test: reduce risk and improve test results


Data

Virtualise production data and

transaction responses to simulate

application request and responses

Data


3 Keys to better

performance

• Test early

• Test often

• Monitor and leverage

data from monitoring

as baseline for test

The NEW Reality

User Network Service Data


Q&A


IF YOU LIKE IT

PLEASE RATE MEVisit our blog to get all the slides

WWW.GLOBETESTING.COM

[email protected]

reality shock j aracil

Software