front line survival guide - nordic testing days · context: startup, webshop: boutique de jus...
TRANSCRIPT
@alientester
@gmuecke
Front Line Survival GuidePERFORMANCE TESTING WITH GATLING
SIMON BERNER & GERALD MÜCKE
NORDIC TESTING DAYS 2018
1
@alientester
@gmuecke
Workshop mission
Theory
What? Why? How?
Introduction to Gatling
The Juice Shop Story
Practice
Test-Mission:
Record
Edit &
Execute Load Tests with Gatling
Conclusion
2
@alientester
@gmuecke
Performance Testing?
3
@alientester
@gmuecke
This Photo is under CC0 License (pexels.com)
What is Performance?
Calculation time
4
@alientester
@gmuecke
Why do we care? 5
This Photo is under CC0 License (pexels.com)
@alientester
@gmuecke
Studies show, that a second-long delay causes a drop of
around 10% page views
around 16% drop in customer satisfaction
6Why do we care?
@alientester
@gmuecke
Types of Load Test
Batch Online
7
• Process the most events in the
least time
• Event Driven
• Focus on Low Response Times
• Get the most done in the least time
and resources
• Data Driven
• Focus on Maximum Throughput
This Photo by Unknown Author is licensed under CC BY-NC-SA This Photo by Unknown Author is licensed under CC BY-NC-SA
@alientester
@gmuecke
Input to Load Test Scenarios
Performance Requirements
Target Users (Concurrent, per Duration, Total)
Response Time Targets (90%, 95%, 99%)
Throughput
Historical Data
Number of Total Users per Duration
Number of Concurrent Users
Peak Loads (Peak Month/Day/Hour/Minute)
Request Logs
Educated Guess / Gut Feeling
8
@alientester
@gmuecke
Example
Requirements
The system is capable of
Serving 1000 concurrent users with
an average Response Time of 1.5s
Source: The Project Manager
What is the most relevant
information?
This Photo by Unknown Author is licensed under CC BY-NC-SA
9
@alientester
@gmuecke
Numbers need a Context
Concurrent Users (CPU)
vs. Concurrent Sessions (Memory)
vs. Users per Period (Capacity)
Average
vs. perceived Average (90 %)
vs. Percentiles (95%,99%,99.9%)
1000 Conc.Users, avg < 1.5s
vs 1000 Users/h, 90% < 1.5s
This Photo by Unknown Author is licensed under CC BY-NC-SA
10
@alientester
@gmuecke
Load Testing Practices
Soak Testing
Discover Leaks
SLA Regressions
Stress Testing
Testing Stability
Overload / Recovery
Benchmarking
Discover Regressions between different Versions or Configurations
11
@alientester
@gmuecke
Three
elements of
a Load Test
• How groups of usersnavigate throughyour application
User Flows
• How to fetch a typical page and it’sresources
Page Scripts
• How the users arrive at your page
Load Function
12
@alientester
@gmuecke
Three
elements of
a Load Test
User Flows
Page Scripts
Load Function
13
GET /index.html
GET /favicon.ico
GET /someScript.js
GET /img/background.jpg
@alientester
@gmuecke
A typical load test: Constant Load
Constant Load
+ Ramp up / down
Allowing the System to adapt to Load (warm up)
Distributes Load (virtual users)
Good for:
Finding latent bugs, i.e.Memory or Resource Leaks
Precise Measurements
Regressions
Stability Issues
Statistical Response Times
Known Load Distributions
Load
%
t
14
@alientester
@gmuecke
Why do a ramp up?
System Warm-up
Allow JIT to optimize code
Allow Caches to be populated
Fetch or Initiate Resources (i.e. Database
Connections)
Allow Queues and Buffers to fill to a stable level
Distribute Load evenly
System’s Performance Characteristics arenon-linear during ramup
15
This Photo by Unknown Author is licensed under CC BY-NC-ND
@alientester
@gmuecke
What do we test for?
Capacity
Oversubscribed resources
CPU
Memory
Bandwith
Storage
Queues
…
Response times under load
Stability
Resources leaks
Memory
Network connections
Filehandle
System crashes
System recovery
…
16
This Photo by Unknown Author is licensed under CC BY-SA
@alientester
@gmuecke
Any heuristics to sum it up? 17
@alientester
@gmuecke
How do we do it? 18
This Photo is under CC0 License (pexels.com)This Photo is under CC0 License (pexels.com)
@alientester
@gmuecke
GatlingAN INTRODUCTION
19
@alientester
@gmuecke
Gatling Basics
Written in Scala
And so are the Load Tests
Asynchronous, Non-Blocking, Event-Driven Concurrency Model
Virtual Users are «Messages», not threads
Less Resource Consumption & Higher Loads
No Closed Feedback Loop with SUT
Standalone or Integration with Maven / Gradle
Useful in Continuous Integration Pipelines
Visual Reports
20
@alientester
@gmuecke
Pros & Cons
Pro
Testing Framework
Small, lightweight → reduced to
max
No installation:
Maven / Gradle Plugin
Download Standalone
Highly Scalable
Documentation & Community
Free & OpenSource
Cons
No UI
only for Recorder
Requires programming skills
Scala
Limited Protocol Support
HTTP & JMS
Basic reporting
21
@alientester
@gmuecke
Test Development with Gatling 22
Scenarios Load ModelsPages
@alientester
@gmuecke
Test Development with Gatling 23
val page1 = exec(
http("request_0")
.get("/")
.headers(acceptHtml)
.resources(...))
Pages Scenarios
val scn =
scenario("Simple")
.exec(page1)
Load Models
scn.inject(
constantUsersPerSec(500)
during (1 minute)
)
@alientester
@gmuecke
Toolchain setup
24
@alientester
@gmuecke
ExerciseRECORD & ADAPT & REPLAY SCRIPTS WITH GATLING
25
@alientester
@gmuecke
The ‚boutique de jus‘ story
We wanna revolutionize the way people consume healty food
We‘ve planned a set of several services in the future
First one is our ultimate juice shop
Business goal:
Our investors expect profit at peak of 1.5Mio$ / month (30days)
Assuming 5$/order and profit margin of 10%
→ we need 100k orders per day
We expect 70k-100k visits on our site per hour
We assume that all visits occur within 10 hours per day
Conversion rate of 10% (submitted orders)
Response time per page 1.5s – 4s
26
@alientester
@gmuecke
Test Mission (using CCD IS EMARI)
Context: Startup, Webshop: boutique de jus
Criteria: 100k user/h, 10k orders/h, Response times: 1.5s-4s
Design: Develop Test Scenarios
Install: Setup the toolchain on your machine
Script: Record the test scenarios (gatling recorder / HAR file)
Execute & Monitor: execute the recorded scripts (and monitor with VisualVM)
Analyze: Have a look at the gatling reports (and VisualVM)
Report: Retrospective
Iterate: Play with your recorded script (adapt and execute)
27
@alientester
@gmuecke
Pacing & Think Time
Think Time
Time to wait between each
request
Resembles a thinking user
Pacing
Recommended for Equi-
Distribution
Keeps the users and load
distributed
Minimum time for a user
transaction before the next begins
Wait time is added at the end
28
Request
Scenario
Iteration n
Scenario
Iteration n+1
Pacing
Pacing TimeThink Time
@alientester
@gmuecke
Analysis &
Reporting
29
Photo: Calvinius
@alientester
@gmuecke
What do we need for a deeper analysis?
Precise reporting of your observations
Baseline configuration of the system
Model of the system (architecture) to visualize the flow
The right people available
30
@alientester
@gmuecke
External Observations
Response Times
Averages, Min, Max
Percentile (90% is perceived average!)
Outliers
Histogram
Error Responses
Which page fails most often?
Live Uses (Users that have not finished their scenario)
Piling up?
Constant
31
@alientester
@gmuecke
Internal Observations
CPU Consumption
System
I/O Activity
Disc
Network
Context Switching
User
Threads & Contention
GC Activity
Algorithms
Hardware Counters
Garbage Collection Activity
GC Logs
Allocation Rates
GC Generational Behavior
Memory Consumption
Heap
Non-Heap
Thread Stacks
Direct Allocations
Code Cache
Metaspace
Leaks
32
@alientester
@gmuecke
SummarySHAPING YOUR MEMORY
33
@alientester
@gmuecke
WARNING: The next slide may
potentially trigger seizures for
people with photosensitive
epilepsy. Viewer discretion is
advised.
34
@alientester
@gmuecke
35
Forget everything what you have heard so far, now comes the real important stuff!
This Photo by Unknown Author is licensed under CC BY-NC-SA
@alientester
@gmuecke
Test Development with Gatling 36
Scenarios Load ModelsPages
@alientester
@gmuecke
Test Development with Gatling 37
val page1 = exec(
http("request_0")
.get("/")
.headers(acceptHtml)
.resources(...))
Pages Scenarios
val scn =
scenario("Simple")
.exec(page1)
Load Models
scn.inject(
constantUsersPerSec(500)
during (1 minute)
)
@alientester
@gmuecke
Thank you!QUESTIONS & FEEDBACK APPRECIATED!
38