building a metrics-optimized pipeline...2. in order to use these metrics to improve your software...

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Andrew Phillips, VP of ProductsAndreas Grabner, Performance Advocate

Building a Metrics-Optimized Pipeline


1 The “Unicorns” • Recap from Velocity 2015• Why we don’t have to be like Facebook

2 What Unicorns do well • Speed through Automation• Quality through Metrics

3 Which Metrics and How • Service, Business & User Metrics• Pre-Prod: From Integration- & Load Tests• Prod: From Real Users

4 Dynatrace & XL Release • Building a Metric-Driven Pipeline

agenda


U N I C O R N S T A T U S


700 deployments / year

10 + deployments / day

50 – 60 deployments / day

Every 11.6 seconds


• Waterfall Agile: 3 years

• 220 Apps - 1 deployment per month

“EVERYONE can do Continuous Delivery”

“Every manual tester does AUTOMATION”

“WE DON’T LOG BUGS – WE FIX THEM!”

Measures Built-In, Visible to Everyone

Promote your Wins, Educate your Peers


CHALLENGE


Fail Faster!?


It‘s not about blind automation of pushing more

bad code through a shiny pipeline


One goal: deliver better features to customers faster

Two fundamental components: speed + quality


OK, OK, …

But how?

13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE13

#1 Speedend-to-end automation of the software delivery process


Automate, automate and automate


• Continuous Integration

• On-demand environment provisioning

• Automated deployments

• Automated test data management

• Continuous Delivery pipeline orchestration

• ...

End-to-end automation of the software delivery process


• „It‘s just work“ - not easy, but a known problem

• We‘re not all unicorns – goal is to deploy as often as necessary, not as often as theoretically possible

• Metrics-driven improvement, also for your process: define goals, measure current state, address bottlenecks

End-to-end automation of the software delivery process

The data tells you:start here


#2 QualityMetrics-based GO or No-GO decisions


Insufficient Focus on Quality


Making Quality a first-class citizen

„Too hard“

„we‘ll get round to this later“

„not cool enough“


Landing Page Example: Content Delivery Metrics 282! Objects

on that page9.68MB Page Size

8.8s Page Load Time

Most objects are images delivered from your main

domain

Very long Connect time (1.8s) to your CDN


Search Example: Architectural Metrics

26.7sExecution Time 33! Calls to the

same Web Service

171! SQL Queries through LINQ by this Web Service – request

similar data for each call

Architecture Violation: Direct access to DB instead from frontend logic


Marketing Campaign: Business Metrics


1. Not just in production!

2. In order to use these metrics to improve your software delivery process, you need to measure as early as possible in your development and delivery process

3. Fast feedback is both more effective and cheaper at identifying and fixing problems

4. Just reuse what you already have: Automated “Functional” Tests – but convert them into Architectural, Performance and Scalability Validation Tests

When and How shall we measure?


Measures from your Tests in Action

12 0 120ms

3 1 68ms

Build 20 testPurchase OK

testSearch OK


testSearch OK

Build 18 testPurchase FAILED

testSearch OK


testSearch OK

Build # Test Case Status # SQL # Excep CPU

12 0 120ms

3 1 68ms

12 5 60ms

3 1 68ms

75 0 230ms

3 1 68ms

Test & Monitoring Framework Results Architectural Data

We identified a regresesion

Problem solved

Exceptions probably reason for

failed testsProblem fixed but now we have an

architectural regression

Problem fixed but now we have an

architectural regressionNow we have the functional and

architectural confidence

Let’s look behind the

scenes


When shall we measure?

Build/CI Dev Integration

Go-live Analysis

Service-level Service-level

Business application

Service-level

Business application

User experience

Feedback loop!


Metric-Driven Continuous DeliveryWith Dynatrace & XL Release


Capturing and Analyzing Test Executions with Dynatrace


Unit/Integration

Tests for Integration and Performance Pipeline Phase

Performance

Web API

Browser-based

Load


You already capture Metrics for ALL these tests#1: Test captured as PurePath

#2: API Call to Hibernate

#3: SQL Calls


#1: Analyzing every Unit & Integration test

#2: Metrics for each test

#3: Detecting regression based on measure

Unit/Integration Tests can also be baselined!


Unit/Integration Tests: Build-by-Build Quality View

Build Quality Overview in Dynatrace or Jenkins

Build Quality Overview in Dynatrace or Jenkins


Load Tests: We can find hotspots per test easily

#1: Analyze Load Testing Results by Timer Name, Script Name, …

#2: Which TIERS have a problem?

#3: Is it the DATABASE?#4: How HEALTHY is the

JVM and the Host?

#5: Do we have any ERRORS?

#6: LOAD vs. RESPONSE Time over time?


Load Tests: We can compare two runs

Did we get better or worse? DB, Web Requests, API, …

Did we get better or worse? DB, Web Requests, API, …


Load Tests: Pull Results into your Pipeline / Build Server

Embed your Load Testing Results in Jenkins


Production Data: Real User & Application Monitoring


XebiaLabs XL Release allows you to integrate Dynatracequality metrics into your overall Continuous Delivery pipeline

• Automatically verify architectural quality during your integration testing

• Trigger and review performance monitoring results

• Automatically register releases in Dynatrace to allow for “before/after” comparisons of user behaviour

• Deliver better software and close the Continuous Delivery feedback loop!

Continuous Delivery Pipeline Integration


Orchestrating your Delivery Pipeline with XL Release

Insight, visibility & control into

people and tools



ArchitecturalQuality Metrics from

Unit & Integration Tests

Performance Metrics from Load Tests

Deployment Markerto support

Production Monitoring

Review user behaviorThrough UEM Data



Use quality metrics toautomatically make

decisions



Integrate qualityawareness into your

delivery pipeline


Download XL Release Dynatrace Plugin from GitHub


1. Continuous Delivery = Speed + Quality

2. 3 levels of metrics: service, business and user

3. Dynatrace can already collect these metrics for you

4. Get started today: incorporate quality metrics into your release pipeline straight away using tools like XL Release

Summary


• #1: Quality via Metrics

• Define Metrics for Build Process, for Production and for your End Users

• http://apmblog.dynatrace.com/2014/03/13/software-quality-metrics-for-your-continuous-delivery-pipeline-part-i/

• #2: Speed via Automation

• Automated Deployment, Metrics Collection & Regression Detection

• https://community.dynatrace.com/community/display/DOCDT62/Test+Automation+Explained

• http://apmblog.dynatrace.com/2015/09/28/kick-start-continuous-monitoring-with-chef-ansible-and-puppet/

• More Links

• https://xebialabs.com/resources/whitepapers/the-it-managers-guide-to-continuous-delivery/

• Blog on Metrics-Driven Pipeline – AP TODO

• https://github.com/xebialabs-community/xlr-dynatrace-plugin

Action Items for You

http://apmblog.dynatrace.com/2014/03/13/software-quality-metrics-for-your-continuous-delivery-pipeline-part-i/

https://community.dynatrace.com/community/display/DOCDT62/Test+Automation+Explained

http://apmblog.dynatrace.com/2015/09/28/kick-start-continuous-monitoring-with-chef-ansible-and-puppet/

https://xebialabs.com/resources/whitepapers/the-it-managers-guide-to-continuous-delivery/

https://github.com/xebialabs-community/xlr-dynatrace-plugin


• Thursday, 10:30 – Top Performance Problem Patterns

• Learn more metrics

• Thursday, 1:45 – There‘s money in those metrics

• TUI UK on Business Metrics

Sessions of Interest at PERFORM

49 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

building a metrics-optimized pipeline...2. in order to use these metrics to improve your software...

Documents