cloud load testing with visual studio team services

© naked ALM – Martin Hinshelwood 2013

Martin Hinshelwood | @MrHinshConsultant | MVP | Ranger | Trainer | Coachmartin@nkdagility.com | http://nkdagility.com/blog

Load Testing with Visual Studio Team Services

Scrum?

Join the #DevOps conversation #NDCOlso with @MrHinsh

Join the #DevOps conversation #NDCOlso with @MrHinsh

Join the #DevOps conversation #NDCOlso with @MrHinsh

Join the #DevOps conversation #NDCOlso with @MrHinsh

Professional Scrum Teams

Mechanical /

Amateur / Flaccid Scrum

Values and

Principles

Technical Excellen

ce

The Scrum Guide: http://www.scrumguides.org/

FLOW OFCUSTOMER

VALUE

TEAMAUTONOMY

& ENTERPRISEALIGNMENT

BACKLOG refined with LEARNING

EVIDENCEgathered in

PRODUCTION

MANAGED TECHNICAL

DEBT

PRODUCTION FIRST MINDSET

INFRASTRUCTUREis a FLEXIBLE

RESOURCE

Areas for improvement or Goals to achieve

Join the #DevOps conversation #NDCOlso with @MrHinsh

DevOps?

Join the #DevOps conversation #NDCOlso with @MrHinsh

Join the #DevOps conversation #NDCOlso with @MrHinsh

Join the #DevOps conversation #NDCOlso with @MrHinsh

Join the #DevOps conversation #NDCOlso with @MrHinsh

P R A C T I C E SAutomated TestingContinuous IntegrationContinuous DeploymentRelease Management

P R A C T I C E SUsage MonitoringTelemetry CollectionTesting in ProductionStakeholder Feedback

P R A C T I C E STesting in ProductionUsage MonitoringUser TelemetryStakeholder feedbackFeature flags

P R A C T I C E SCode ReviewsAutomated TestingContinuous Measurement

P R A C T I C E SApplication Performance ManagementInfrastructure as CodeContinuous DeliveryRelease Management Configuration Management Automated Recovery

P R A C T I C E SApplication Performance ManagementInfrastructure as CodeContinuous DeploymentRelease Management Configuration Management Automated Recovery

P R A C T I C E SEnterprise AgileContinuous IntegrationContinuous DeploymentRelease Management

FLOW OFCUSTOMER

VALUE

TEAMAUTONOMY

& ENTERPRISEALIGNMENT

BACKLOG refined with LEARNING

EVIDENCEgathered in

PRODUCTION

MANAGED TECHNICAL

DEBT

PRODUCTION FIRST MINDSET

INFRASTRUCTUREis a FLEXIBLE

RESOURCE

DevOps habits and practices

Join the #DevOps conversation #NDCOlso with @MrHinsh

Testing?

What testing do you do?

Functional Tests

• User Story Tests• Prototypes• Simulations

Exploratory Testing

• Usability Testing• User Acceptance

Testing• Alpha/Beta

Developer Tests

• Unit Tests• Component Tests

Performance & Load Testing

Security Testing-ility Testing

Supp

orti

ng t

he T

eam

Technology Facing

Critique Product

Business Facing

Automated

Automated& Manual

Automated & Tools

Manual

Join the #DevOps conversation #NDCOlso with @MrHinsh

Join the #DevOps conversation #NDCOlso with @MrHinsh

Don’t leave load testing until last!

This demonstration will cover:

• Creating a basic cloud load test• Import Archive• Custom load test in Visual Studio

DEMO: URL Cloud Load Test

That must cost a LOT!

• 20k: FREE• 2m: $8000 (64k NOK)• 2m-10m: $16000 (130k NOK)

So for 2 minutes of 200 simultaneous users per build for 10 deployments per day at 4000 VMU’s, it would be FREE

Before it costs any money you can load test 50 deployments at 200 simultaneous users for 2 minutes.

Or you could ramp up to 1000 simultaneous users.

That must be expensive?Examples Cloud-based Load Testing*

* Requires Visual Studio Enterprise- Services purchased via Microsoft Azure based on actual usage- Additional services are accessible to all users on the account and shared

among them- Build time is actual computing time required to run the build, and

does not include queuing time (if any)- Load test runs specify the number of virtual users and duration in

minutesVirtual users x minutes = virtual user minutes (VUM)

FREE: 20,000 virtual user minutes per monthTHEREAFTER:$0.004/virtual user minute for 20,001-2M VUM$0.002/VUM for 2,000,001-10M VUM$0.001/VUM for usage above 10M VUM/month

Professional DevOps ?

Continuous Delivery

Continuous Integration

Release Management

Streamline and automate the workflow between development and IT Ops and deliver higher quality software more frequently with less risk.

Continuous Quality

Enabling continuous value deliveryDeveloper Version Control Build and

Unit TestingAutomated Acceptance

Testing

Automated Performance

TestingRelease

Check-in Trigger

Feedback

Approval

Feedback

Feedback

Trigger

Trigger

Continuous integration (CI)Developer Version Control Build and

Unit TestingAutomated Acceptance

Testing

Automated Performance

TestingRelease

Check-in Trigger

Feedback

Approval

Feedback

Feedback

Trigger

Trigger

Continuous integration (CI)Developer Version Control Build and

Unit TestingAutomated Acceptance

Testing

Automated Performance

TestingRelease

Check-in Trigger

Feedback

Approval

Feedback

Feedback

Trigger

Trigger

Continuous integration (CI)Developer Version Control Build and

Unit TestingAutomated Acceptance

Testing

Automated Performance

TestingRelease

Check-in Trigger

Feedback

Approval

Feedback

Feedback

Trigger

Trigger

This demonstration will cover:

• Integrating into your DevOps pipeline• https://www.visualstudio.com/docs/integrate/ap

i/load-test/overview

DEMO: Cloud Load Test in VSTS

Join the #DevOps conversation #NDCOlso with @MrHinsh

© naked ALM – Martin Hinshelwood 2013

Professional Teams

Professional Scrum Professional DevOps

Join the #DevOps conversation #NDCOlso with @MrHinsh

© naked ALM – Martin Hinshelwood 2013

Take-Away for Cloud Load Testing

• Cloud-load testing: rich reports in the web• Running Apache JMeter based load tests in the cloud – how to• Creating Load Test Plugins for Cloud-based Load Test• Scheduling Load Test Execution• Performance Testing

Join the #DevOps conversation #NDCOlso with @MrHinsh

© naked ALM – Martin Hinshelwood 201341© 1993-2015 Scrum.org, All Rights Reserved

Suggested Reading

Professional ALM with Visual Studio 2013 The Phoenix Project Software in 30 Days

Join the #DevOps conversation #NDCOlso with @MrHinsh

© naked ALM – Martin Hinshelwood 2013

Suggested Reading

Join the #DevOps conversation #NDCOlso with @MrHinsh

Martin Hinshelwood | @MrHinshConsultant | MVP | Ranger | Trainer | Coachmartin@nkdagility.com | http://nkdagility.com/blog

ThanksLoad Testing with Visual Studio Team Services