front line survival guide - nordic testing days · context: startup, webshop: boutique de jus...

@alientester

@gmuecke

Front Line Survival GuidePERFORMANCE TESTING WITH GATLING

SIMON BERNER & GERALD MÜCKE

NORDIC TESTING DAYS 2018

1

@alientester

@gmuecke

Workshop mission

Theory

What? Why? How?

Introduction to Gatling

The Juice Shop Story

Practice

Test-Mission:

Record

Edit &

Execute Load Tests with Gatling

Conclusion

2

@alientester

@gmuecke

Performance Testing?

3

@alientester

@gmuecke

This Photo is under CC0 License (pexels.com)

What is Performance?

Calculation time

4

@alientester

@gmuecke

Why do we care? 5

This Photo is under CC0 License (pexels.com)

@alientester

@gmuecke

Studies show, that a second-long delay causes a drop of

around 10% page views

around 16% drop in customer satisfaction

6Why do we care?

@alientester

@gmuecke

Types of Load Test

Batch Online

7

• Process the most events in the

least time

• Event Driven

• Focus on Low Response Times

• Get the most done in the least time

and resources

• Data Driven

• Focus on Maximum Throughput

This Photo by Unknown Author is licensed under CC BY-NC-SA This Photo by Unknown Author is licensed under CC BY-NC-SA

@alientester

@gmuecke

Input to Load Test Scenarios

Performance Requirements

Target Users (Concurrent, per Duration, Total)

Response Time Targets (90%, 95%, 99%)

Throughput

Historical Data

Number of Total Users per Duration

Number of Concurrent Users

Peak Loads (Peak Month/Day/Hour/Minute)

Request Logs

Educated Guess / Gut Feeling

8

@alientester

@gmuecke

Example

Requirements

The system is capable of

Serving 1000 concurrent users with

an average Response Time of 1.5s

Source: The Project Manager

What is the most relevant

information?

This Photo by Unknown Author is licensed under CC BY-NC-SA

9

@alientester

@gmuecke

Numbers need a Context

Concurrent Users (CPU)

vs. Concurrent Sessions (Memory)

vs. Users per Period (Capacity)

Average

vs. perceived Average (90 %)

vs. Percentiles (95%,99%,99.9%)

1000 Conc.Users, avg < 1.5s

vs 1000 Users/h, 90% < 1.5s

This Photo by Unknown Author is licensed under CC BY-NC-SA

10

@alientester

@gmuecke

Load Testing Practices

Soak Testing

Discover Leaks

SLA Regressions

Stress Testing

Testing Stability

Overload / Recovery

Benchmarking

Discover Regressions between different Versions or Configurations

11

@alientester

@gmuecke

Three

elements of

a Load Test

• How groups of usersnavigate throughyour application

User Flows

• How to fetch a typical page and it’sresources

Page Scripts

• How the users arrive at your page

Load Function

12

@alientester

@gmuecke

Three

elements of

a Load Test

User Flows

Page Scripts

Load Function

13

GET /index.html

GET /favicon.ico

GET /someScript.js

GET /img/background.jpg

@alientester

@gmuecke

A typical load test: Constant Load

Constant Load

+ Ramp up / down

Allowing the System to adapt to Load (warm up)

Distributes Load (virtual users)

Good for:

Finding latent bugs, i.e.Memory or Resource Leaks

Precise Measurements

Regressions

Stability Issues

Statistical Response Times

Known Load Distributions

Load

%

t

14

@alientester

@gmuecke

Why do a ramp up?

System Warm-up

Allow JIT to optimize code

Allow Caches to be populated

Fetch or Initiate Resources (i.e. Database

Connections)

Allow Queues and Buffers to fill to a stable level

Distribute Load evenly

System’s Performance Characteristics arenon-linear during ramup

15

This Photo by Unknown Author is licensed under CC BY-NC-ND

@alientester

@gmuecke

What do we test for?

Capacity

Oversubscribed resources

CPU

Memory

Bandwith

Storage

Queues

…

Response times under load

Stability

Resources leaks

Memory

Network connections

Filehandle

System crashes

System recovery

…

16

This Photo by Unknown Author is licensed under CC BY-SA

@alientester

@gmuecke

Any heuristics to sum it up? 17

@alientester

@gmuecke

How do we do it? 18

This Photo is under CC0 License (pexels.com)This Photo is under CC0 License (pexels.com)

@alientester

@gmuecke

GatlingAN INTRODUCTION

19

@alientester

@gmuecke

Gatling Basics

Written in Scala

And so are the Load Tests

Asynchronous, Non-Blocking, Event-Driven Concurrency Model

Virtual Users are «Messages», not threads

Less Resource Consumption & Higher Loads

No Closed Feedback Loop with SUT

Standalone or Integration with Maven / Gradle

Useful in Continuous Integration Pipelines

Visual Reports

20

@alientester

@gmuecke

Pros & Cons

Pro

Testing Framework

Small, lightweight → reduced to

max

No installation:

Maven / Gradle Plugin

Download Standalone

Highly Scalable

Documentation & Community

Free & OpenSource

Cons

No UI

only for Recorder

Requires programming skills

Scala

Limited Protocol Support

HTTP & JMS

Basic reporting

21

@alientester

@gmuecke

Test Development with Gatling 22

Scenarios Load ModelsPages

@alientester

@gmuecke

Test Development with Gatling 23

val page1 = exec(

http("request_0")

.get("/")

.headers(acceptHtml)

.resources(...))

Pages Scenarios

val scn =

scenario("Simple")

.exec(page1)

Load Models

scn.inject(

constantUsersPerSec(500)

during (1 minute)

)

@alientester

@gmuecke

Toolchain setup

24

@alientester

@gmuecke

ExerciseRECORD & ADAPT & REPLAY SCRIPTS WITH GATLING

25

@alientester

@gmuecke

The ‚boutique de jus‘ story

We wanna revolutionize the way people consume healty food

We‘ve planned a set of several services in the future

First one is our ultimate juice shop

Business goal:

Our investors expect profit at peak of 1.5Mio$ / month (30days)

Assuming 5$/order and profit margin of 10%

→ we need 100k orders per day

We expect 70k-100k visits on our site per hour

We assume that all visits occur within 10 hours per day

Conversion rate of 10% (submitted orders)

Response time per page 1.5s – 4s

26

@alientester

@gmuecke

Test Mission (using CCD IS EMARI)

Context: Startup, Webshop: boutique de jus

Criteria: 100k user/h, 10k orders/h, Response times: 1.5s-4s

Design: Develop Test Scenarios

Install: Setup the toolchain on your machine

Script: Record the test scenarios (gatling recorder / HAR file)

Execute & Monitor: execute the recorded scripts (and monitor with VisualVM)

Analyze: Have a look at the gatling reports (and VisualVM)

Report: Retrospective

Iterate: Play with your recorded script (adapt and execute)

27

@alientester

@gmuecke

Pacing & Think Time

Think Time

Time to wait between each

request

Resembles a thinking user

Pacing

Recommended for Equi-

Distribution

Keeps the users and load

distributed

Minimum time for a user

transaction before the next begins

Wait time is added at the end

28

Request

Scenario

Iteration n

Scenario

Iteration n+1

Pacing

Pacing TimeThink Time

@alientester

@gmuecke

Analysis &

Reporting

29

Photo: Calvinius

@alientester

@gmuecke

What do we need for a deeper analysis?

Precise reporting of your observations

Baseline configuration of the system

Model of the system (architecture) to visualize the flow

The right people available

30

@alientester

@gmuecke

External Observations

Response Times

Averages, Min, Max

Percentile (90% is perceived average!)

Outliers

Histogram

Error Responses

Which page fails most often?

Live Uses (Users that have not finished their scenario)

Piling up?

Constant

31

@alientester

@gmuecke

Internal Observations

CPU Consumption

System

I/O Activity

Disc

Network

Context Switching

User

Threads & Contention

GC Activity

Algorithms

Hardware Counters

Garbage Collection Activity

GC Logs

Allocation Rates

GC Generational Behavior

Memory Consumption

Heap

Non-Heap

Thread Stacks

Direct Allocations

Code Cache

Metaspace

Leaks

32

@alientester

@gmuecke

SummarySHAPING YOUR MEMORY

33

@alientester

@gmuecke

WARNING: The next slide may

potentially trigger seizures for

people with photosensitive

epilepsy. Viewer discretion is

advised.

34

@alientester

@gmuecke

35

Forget everything what you have heard so far, now comes the real important stuff!

This Photo by Unknown Author is licensed under CC BY-NC-SA

@alientester

@gmuecke

Test Development with Gatling 36

Scenarios Load ModelsPages

@alientester

@gmuecke

Test Development with Gatling 37

val page1 = exec(

http("request_0")

.get("/")

.headers(acceptHtml)

.resources(...))

Pages Scenarios

val scn =

scenario("Simple")

.exec(page1)

Load Models

scn.inject(

constantUsersPerSec(500)

during (1 minute)

)

@alientester

@gmuecke

Thank you!QUESTIONS & FEEDBACK APPRECIATED!

38