performance testing with 100,000 concurrent users in aws
DESCRIPTION
M-Square build an easy scalable performance test solution on AWS, using open source tools & CI servers, to allow cost-effective testing at scale. The solution is suitable for any organisation type, from startup to enterprise. The talk covers VPC, EC2, S3, ELB’s, AWS API scripting, automation and interesting performance issues when running massive workloads on AWS.TRANSCRIPT
![Page 1: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/1.jpg)
Performance Testing@
With 100,000 concurrent users
![Page 2: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/2.jpg)
WHO ARE WE
• AWS for most client projects and our own• Enterprise/Solution Architecture +
Development• Performance Engineering, Profiling,
Optimization• "DevOps"• AWS for most client projects and our own
![Page 3: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/3.jpg)
WHAT WE WANTED
Performance test applications with 100,000 concurrent users
(Websocket + HTTP)
• Easy to use• Traceable history of tests• Good fit with modern DevOps
processes• Low maintenance and cost
![Page 4: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/4.jpg)
99 Problems
Quickly max out a single test node(or even 10 of them)
Performance tests should be available to whole team(not just the guru)
Shouldn't have to write another App(or pay A LOT for one)
![Page 5: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/5.jpg)
REQUIREMNTS
![Page 6: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/6.jpg)
SOLUTION
(one of many)
![Page 7: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/7.jpg)
STEP 1
Build App(Compile, Test, Package)
![Page 8: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/8.jpg)
STEP 2
Build your environment (Puppet, Chef, etc)Deploy new application version
![Page 9: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/9.jpg)
STEP 3
Create load test slave instances
![Page 10: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/10.jpg)
STEP 4
Start running the test plan against new environment
![Page 11: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/11.jpg)
STEP 5
• Test slaves return results to Jenkins.
• Then clean up• Shut down instances• Snapshot new AMI
![Page 12: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/12.jpg)
PIPELINE
Build app -> Create test env + Deploy -> Test new environment
![Page 13: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/13.jpg)
LESSONS LEARNED 1/2
• Performance problems found and fixed Garbage Collection, Serialization and encoding
• Very helpful when sizing infrastructure Webservice tier network was bottleneck Different instance sizes picked
• Scaling the test clients can be hard
![Page 14: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/14.jpg)
LESSONS LEARNED 2/2
• Performance testing via ELB• Ramp up slowly• Make sure client see DNS
changes• ELB's not great for Websocket
• Linux tuning• Ephemeral port range• Buffers everywhere• File handles, flood protection, etc• It does a pretty good job by
default
![Page 15: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/15.jpg)
Results & Deep Insight
Deep Insight with dynaTrace APM beyond simple JMeter results
Clearly understand the difference between:• Load generation• Real User Monitoring • Always-on deep transaction tracing
![Page 16: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/16.jpg)
TEST AUTOMATION / CI
• Unit Tests are fully integrated as part of CI• Each build also measures performance for each testcase
![Page 17: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/17.jpg)
BONUSES
Easily re-used for different types of build/testing
• BYO load test tool• Spot instances are cheaper• Not just for load testing• Test Chef/Puppet scripts• OS updates
• Browser based (Webdriver, PhantomJS, etc)
• Auto-scale testing• Snapshot AMI after successful tests
![Page 18: Performance testing with 100,000 concurrent users in AWS](https://reader033.vdocuments.mx/reader033/viewer/2022061223/54c562e54a79599f748b459b/html5/thumbnails/18.jpg)
Questions ?
Thanks !
Twitter: msquareau
http://m-square.com.au