performance metrics for your build pipeline - presented at vienna webperf october meetup

Post on 22-Apr-2015

142 Views

Category:

Software

2 Downloads

Preview:

Click to see full reader

DESCRIPTION

Software Performance Metrics that you should look at throughout your Build Pipeline and not just when your app crashes in productiong. Find performance and scalability problems as soon as executing your first Unit Test. Simply focus on metrics such as #SQLs, #LogMessages, #Objects on Heap, ...

TRANSCRIPT

1

Checking Performance Along your Build Pipeline

@grabnerandihttp://apmblog.compuware.com

http://bit.ly/dttrial

2

Today we are here for …

Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers

Automated Acceptance

Testing

Automated Capacity Testing

Manual testing• Key showcases• Exploratory testing Release

Cont. Delivery / Deploy

3

Who is doing it? How many successful deployments can they do?

300 Deployments / Year

50-60 Deployments / Day

10+ Deployments / Day

Every 11.6 seconds

4

More on Amazons Story

75% fewer outages since 2006

90% fewer outage minutes

~0.001% of deployments cause a problem

Instantaneous automatic rollback

Deploying every 11.6s

5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Recommended BookRecommended Book

https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/

66 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Status Quo: Unreliable Software Impacts BusinessStatus Quo: Unreliable Software Impacts Business

7

The “War Room” – back then

'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970

The “War Room” – back then

'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970

8

The “War Room” – NOW

Facebook – December 2012

The “War Room” – NOW

Facebook – December 2012

9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

Status Quo: Bugfixing in Production happensStatus Quo: Bugfixing in Production happens

~80% of problems

caused by ~20% patterns

YES we know this

80% Dev Time in Bug Fixing

$60B Defect Costs

BUT

10

4 Situations on

WHY this happens,

HOW to avoid it

4 Situations on

WHY this happens,

HOW to avoid it

11

12

“Blindly” (Re)use Existing

Components

13

Requirement: We need a report

14

Using Hibernate results in 4k+ SQL Statements to display 3 items!

Hibernate Executes 4k+ Statements

Individual Execution VERY

FAST

But Total SUM takes 6s

15

Requirement: We need a fancy UI

16

Using Telerik Controls Results in 9s for Data-Binding of UI Controls

#1: Slow Stored ProcedureDepending on Request

execution time of this SP varies between 1 and 7.5s

#2: 240! Similar SQL StatementsMost of these 240! Statements are

not prepared and just differ in things like Column Names

17

Metrics: # Total SQLs# SQLs / Web Request# Same SQLs / Request

Transferred Rows

Test: With realistic Data

Dev: “Learn” Frameworks

18

19

Implementation Flaws

20

Business Impact requires Action!

21

Solution: Cache to the RESCUE!!

22

Implementation and Rollout

Implemented InMemory Cache

Worked well in Load Testing

23

Result: Out of Memory Crashes!!

Still crashes

Problem fixed!Fixed Version Deployed

24

Metrics: Heap Size, # Objects Allocated,# Objects in Cache

Cache Hit Ratio

Test: With realistic Data

25

12 000 000 $

26

#No “Agile” Deployment

27

Ad on air

Availability dropped to 0%

Load Spike resulted in Unavailability

28

Alternative: “GoDaddy goes DevOps”

Response time improved 4x

1h before SuperBowl KickOff

1h after Game ended

29

Behind the Scenes

30

Metrics: AvailabilityPage Size, # Objects

# Hosts, # Connections

DevOps: “Feature” Switches

31

32

#Push without a Plan

33

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

34

m.store.com redirects to www.store.com

ALL CSS and JS files are redirected to the www domain

This is a lot of time “wasted” especially on high latency mobile

connections

35

Metrics: Load Time, # Resources (Images, …),

# HTTP 3xx, 4xx, 5xx

Dev: Build for Mobile

Test: Test on Mobile

Ops: Monitor Mobile

36

37

#1: Define Measures# of Requests / User

# of Log Messages

# of Exceptions

# Objects Allocated

# Objects In Cache

Cache Hit Ratio

# of Images

# of SQLs

# SQLs per RequestAvailability

# HTTP 3xx, 4xx

Page Size

38

#2: Automate

Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers

Automated Acceptance

Testing

Automated Capacity Testing

Manual testing• Key showcases• Exploratory testing Release

Unit & Integration Tests

Functional Tests

Performance TestsProductionMonitoring

Functional Tests

How? Performance Focus in Test AutomationAnalyzing All Unit / Performance Tests

Analyze Perf Metrics

Identify Regressions

How? Performance Focus in Test Automation

How? Performance Focus in Test Automation

Cross Impact of KPIs

Identify Regressions

4242

#3: Share Results

43

#4: Integrate

44

If we do all that

CANCELLED

45

Want MORE of these and more details?

http://apmblog.compuware.com

46

FREE Products & More Info

dynaTrace Free Trial30 Days Free Trial on http://bit.ly/dttrial After that totally free for local usage

Our Blog: http://apmblog.compuware.com

Follow Me: @grabnerandi

top related