the benefits of using an apm solution while performance testing

DevOpsGuys

@DevOpsGuys www.devopsguys.com 1

DevOpsGuysThe benefits of using an

APM solution while performance testing

Why Perf testing without APM is like Corona without the Lime…

DevOpsGuys

@DevOpsGuys www.devopsguys.com 2http://www.flickr.com/photos/lendog64/5781379311/

DevOpsGuys

@DevOpsGuys www.devopsguys.com 3

Perf Testing & APM in 30mins

• Why Performance Test? • What is APM anyway? • Benefits of APM when Perf Testing• With some case studies along the

way!• Q&A

DevOpsGuys

@DevOpsGuys www.devopsguys.com 4

Why Performance Test?

1. Ensure that we meet our customer’s expectations of website response time– So that they convert and we make £££

2. Ensure that we have sufficient computing capacity to meet #1 under load (load testing)

3. Understand how the system reacts when things go wrong (stress testing)

DevOpsGuys

@DevOpsGuys www.devopsguys.com 5

DevOpsGuys

@DevOpsGuys www.devopsguys.com 6

APM Defined

“In the fields of information technology and systems management, application performance management (APM), is the monitoring and managing of performance and availability of software applications. APM strives to detect and diagnose application performance problems to maintain an expected level of service. APM is the translation of IT metrics into business meaning (i.e. value).”http://en.wikipedia.org/wiki/Application_Performance_Management

DevOpsGuys

@DevOpsGuys www.devopsguys.com 7

“5 Dimensions of APM*”

1. End user experience monitoring - (Active and passive)

2. Application runtime architecture discovery and modeling

3. User-defined transaction profiling (also called business transaction management)

4. Application component monitoring5. Reporting & Application data analytics

*According to Gartner

DevOpsGuys

@DevOpsGuys www.devopsguys.com

CA

QuestHP

OpTier

Opnet

IBM

Compuware

Oracle

MicrosoftSL

Inetco

NetScout

eG InnovationsProgress

ASG

IderaVisual networks

Info vista

Network Instruments

Knoa

Aternity

Coradient

CorrelsenseNastel

dynaTracePrecise

Arctures Technologies

BMC

Managed Engine

New RelicAppDynamics

2011Riverbed

OpTier

2012

AppNeta

Dell

2013

Gartner APM Magic Quadrant

Legac

y Ven

dors

DevOpsGuys

@DevOpsGuys www.devopsguys.com 9

BENEFITS OF LOAD TESTING WITH APM TOOLS

Never leave home without it…

DevOpsGuys

@DevOpsGuys www.devopsguys.com 10

#1See the Big Picture

(aka Systems Thinking)

DevOpsGuys

@DevOpsGuys www.devopsguys.com 11

Systems Thinking 101

“Systems thinking is a framework for seeing interrelationships rather than things, for seeing patterns rather than static snapshots. It is a set of general principles spanning fields as diverse as physical and social sciences, engineering and management.” - Peter Senge

DevOpsGuys

@DevOpsGuys www.devopsguys.com 12

Command & Control vs Systems Thinking

http://www.systemsthinking.co.uk/2-1.asp

DevOpsGuys

@DevOpsGuys www.devopsguys.com 13

CASE STUDY #12 WEEKS WASTED…

“any improvement not made at the constraint is an illusion”

DevOpsGuys

@DevOpsGuys www.devopsguys.com 14

The Scenario

• E-commerce customer• New website • High data-start times on page load• Poor load testing results• Initial conversation with client• 2 week delay while hosting provider

scaled out the hardware…• Problem still not fixed…

DevOpsGuys

@DevOpsGuys www.devopsguys.com 15

What’s the root cause of the ~1sec Data Start?

Waterfall Diagram of page load times

DevOpsGuys

@DevOpsGuys www.devopsguys.com 16

DevOpsGuys

@DevOpsGuys www.devopsguys.com 17

#2Drill down to the

Detail(Code-level)

DevOpsGuys

@DevOpsGuys www.devopsguys.com 18

Find the slow code!

DevOpsGuys

@DevOpsGuys www.devopsguys.com 19

#3Faster Iteration= Better Value

DevOpsGuys

@DevOpsGuys www.devopsguys.com 20

Maximise Testing Value

Test

AnalyseFix

Deploy

DevOpsGuys

@DevOpsGuys www.devopsguys.com 21

CASE STUDY #156X FASTER IN A DAY…

9am to 9pm. 12hrs. 56x faster!

DevOpsGuys

@DevOpsGuys www.devopsguys.com 22

The Scenario

• New website pre-launch testing• Customer Portal (SaaS solution)• Start Load testing 9am• 4 “Test & Fix” cycles in the day • Final testing completed at 9pm

DevOpsGuys

@DevOpsGuys www.devopsguys.com 23

The End Results = 56x  Average

ResponseThroughput

(Req/sec)Initial Testing 3024ms 16/minFinal Result 54ms 59/min

56x faster 3.5x throughput

Before 3024ms After

54ms

DevOpsGuys

@DevOpsGuys www.devopsguys.com 24

Nice flat line!

Before

After

DevOpsGuys

@DevOpsGuys www.devopsguys.com 25

#4Team Work“The Enemy is Performance,

not each other…”

DevOpsGuys

@DevOpsGuys www.devopsguys.com 26

The Blame Game…

Ops DevQA

DevOpsGuys

@DevOpsGuys www.devopsguys.com 27

The Fix-It Game…

Ops

DevQA

DevOpsGuys

@DevOpsGuys www.devopsguys.com 28

DevOpsGuys

@DevOpsGuys www.devopsguys.com 29

Summary

1. See the Big Picture (systems thinking)

2. Drill down to the details3. Faster Iteration = Better Value4. Stop the “Blame Game”– aka “There’s your Problem!”

DevOpsGuys

@DevOpsGuys www.devopsguys.com 30

DevOpsGuys

Thank YouWorkshop Session @

1345End-to-end testing & APM demooperations | availability | performance | scalability | migration | strategy

DevOpsGuys

@DevOpsGuys www.devopsguys.com 31

DevOpsGuys provide development & operations support (DevOps) services

for your online applications.

You run your business. We run your website.

It’s as simple as that.

What We Do

DevOpsGuys

@DevOpsGuys www.devopsguys.com 32

Our service portfolio

• 24 x 7 Application-Centric web operations

• DevOps Transformation & Training

• Continuous Delivery implementation

• Infrastructure Automation & “Infrastructure as Code”

• Application Performance Management

• AppDynamics Implementation & Consultancy

• Web Performance consulting

• Web monitoring• Load Testing• Test Automation and

scripting• JetBrains Product

Implementation & Consultancy

• Octopus Deploy implementation

DevOpsGuys

@DevOpsGuys www.devopsguys.com 33

James Smith (@TheDevMgr)

• 16 years of software delivery experience.

• Proven track record– Pizza Hut– KFC– Dell– RAC– Aviva– Fujitsu Siemens– Whitbread

Steve Thair (@TheOpsMgr)

• 25 years of IT Operations experience.

• Proven track record– Vodafone– Credit Suisse Group– IBM– BNP Paribas– Zurich Financial Services– Totaljobs Group– TSL Education

The Leadership Team

DevOpsGuys

@DevOpsGuys www.devopsguys.com 34

Contact UsVisit us at www.devopsguys.com

Call our team on +44 (0)20 8798 3079

Email our engineers at team@devopsguys.com