is the current model of load testing broken ukcmg - steve thair
DESCRIPTION
Does the current load testing paradigm make sense in a Agile Web 2.0 AJAX World?TRANSCRIPT
Is the current model of load testing broken?
Steve ThairSeriti Consulting@TheOpsMgr
WEB
Some background…. My User Group – Dec 2011 -
http://www.meetup.com/London-Web-Performance-Group/
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.
http://www.slideshare.net/sthair/continuous-integration-a-performance-engineers-journey
Build
Environment
ContinuousLoad
Injection
Production logs
End User Response
Perf Trends
Perf Details
Delivery Team
Usage Profile
For Each Data Point
Alert
Deploy
Service/API response
Measurements
System monitors
Fail Fail to meet
SLA’s?
Deviation from
normal?
Ready for
Test?
08/04/2023
@TheOpsMgr #ukcmg #webperf
The Killer Comment…
“We’ve had to look at separating load injection from performance
measurement”.- Andrew Harding, Betfair
(c) Seriti Consulting, 2011
4
http://www.seriticonsulting.com/blog/2011/12/9/is-the-current-model-of-loadperformance-testing-broken.html
5
@TheOpsMgr #ukcmg #webperf
So why are you paying all that money for that
expensive brand-name load testing tool then?
What are you using to measure
Performance then?
Surely there are open source or
cheap cloud tools if all you want is load?
What are key reasons behind that decision?
What other issues might we
have with “traditional” load testing models in
a Web 2.0 world?
6
@TheOpsMgr #ukcmg #webperf
Defining the “current model”
“Stephen didn’t explicitly define what the current model of load/performance testing is…”
Alex Podelko
http://applicationperformanceengineeringhub.com/is-the-current-model-of-loadperformance-testing-broken/
(1) Waterfall Development Cycle(2) Load Tool compiles the report(3) Reporting at the end of the test(4) Request / Response paradigm(5) Well-defined customer journeys
“Current Model Straw Man”
8
@TheOpsMgr #ukcmg #webperf
Testing in a CI/Agile world Performance testing initiated every time code is committed to
the CM repository (e.g. SVN, GIT etc) Betfair issue was that their system was complex with many
layers of caching etc so it took longer to “warm up” the environment (to achieve a steady performance state) than they had between check-ins…
So they needed “continuous injection” to keep the environment constantly warm…
The test tool never “stopped” to “compile the report”… So any tool that “reported at the end” wasn’t as useful…
9
@TheOpsMgr #ukcmg #webperf
Test tools in a APM & RUM world “traditional test tools” generally have the ability to deploy
agents to gather metrics from the target environment But the depth of analysis and correlation falls well below that
of modern Application Performance Management tools e.g. AppDynamics
APM tools offer a deeper insight and better event correlation across tiers…
And they are getting (much) cheaper…
@TheOpsMgr #ukcmg #webperf
APM Measurement
@TheOpsMgr #ukcmg #webperf
Google Real Time Analytics
12
@TheOpsMgr #ukcmg #webperf
WebSockets HTTP 1.1
Start Timer GET /index.html Response 200 OK
Stop Timer WebSockets
Start Timer Socket “upgrade” N many frames…
Stop??? Bi-directional socket channel
13
@TheOpsMgr #ukcmg #webperf
HTTP 2.0 HTTP 1.1
Sequential ordered
HTTP 2.0 (& SPDY) Multiplexed over a
single connection Responses
returned out of sequence…
Hard to time!
http://stackoverflow.com/questions/10480122/difference-between-http-pipeling-and-http-multiplexing-with-spdy
14
@TheOpsMgr #ukcmg #webperf
Website Complexity 263,000 entry points…
15
@TheOpsMgr #ukcmg #webperf
Visitor Flow How many paths thru a
website? Classic script-driven
approaches can’t hope to address the complexity
Network and log file replay solutions?
16
@TheOpsMgr #ukcmg #webperf
A PCAP Solution to Replay? POC solution based on Cloudmeter
Pion + custom scripts Read a PCAP (network capture) Identify the HTTP traffic Filter it (based on your requirements) Parameterise it (query strings, POST
parameters etc) Randomise inputs from SQL, CSV
etc Replay it against a test environment
i.e. change the base URL Amplify & rate throttle req/sec as
requiredhttp://www.cloudmeter.com/pion/data-processing.php
@TheOpsMgr #ukcmg #webperf
So what’s the Answer? Use the cheapest method to generate load that you can
find… Move away from scripting-based approaches towards using
real-user session replay (if possible!) Generate load continuously Measure continuously
using APM & RUM type tools – FOSS or Commercial Look for changes in histograms, averages, standard dev etc
Protocol and Framework aware instrumentation AFAIK this currently doesn’t exist…
17
Personal Opinion!YMMV…
08/04/2023
(c) Seriti Consulting, 2011
18
@TheOpsMgr #ukcmg #webperf
@LDNWebPerf User Group! Join our London Web Performance Meetup
http://www.meetup.com/London-Web-Performance-Group/ Next Wednesday 17th Oct – 7pm – Central London Follow us on Twitter @LDNWebPerf #LDNWebPerf & #WebPerf
08/04/2023
(c) Seriti Consulting, 2011
19
@TheOpsMgr #ukcmg #webperf
About Me 21yrs IT experience.
Started with www in 1998 (IIS3! Site Server 3!). Web Architect @ BNP Paribas, CSFB etc Web Operations Manager for www.totaljobs.com, www.tes.co.uk Professional Services Manager @ www.siteconfidence.com Seriti Consulting – specialising in web operations, management and
Performance e:[email protected] m:+44 7971 815 940 Twitter: http://twitter.com/TheOpsMgr Blog: http://www.seriticonsulting.com/blog/ LinkedIn: http://uk.linkedin.com/in/stephenthair Skype: seriti-steve
08/04/2023
(c) Seriti Consulting, 2011
20
@TheOpsMgr #ukcmg #webperf
Questions?