Andreas GrabnerDevOps Activist@grabnerndi

Accelerate User Driven InnovationFebruary 2017

Mark TomlinsonPerformance Sherpa and CEO @mark_on_task

James PulleyPerformance Evangelist and CTONewCOE LLC, @perfbytes

Drag picture to placeholder or click icon to add

Drag picture to placeholder or click icon to add

confidential

User Driven Innovation?

confidential

24 “Features in a Box” Ship the whole box!

Very late feedback

F r u s t r a t i o n !

confidential

Accelerated User Driven Innovation1 “Feature at a Time”

Optimize Before DeployImmediate Customer Feedback

confidential

DevOps: Faster & Efficient Innovation

Smaller Apps, Micro-Services More Deployments

App-, Service- & End-User Feedback LoopsHappy Users

Lower Costs

+

-

Total Number of Users per User Experience

Conversion Rate

Use Case #1: User Feedback Driven Decisions

New Features + Day # 1 of Mkt Push

Overall increase of Users!

Jump in Conversion Rate!

Use Case #1: User Feedback Driven Decisions

Users keep growing

Increase # of “tolerating” users!

Lower Conversion as Day #1

Day #2 of Marketing Campaign

Use Case #1: User Feedback Driven Decisions

Drop in Conversion Rate

Spikes in FRUSTRATED Users!

Hotfix Deployment was rolled out

Use Case #1: User Feedback Driven Decisions

User Experience Back to Normal

Jump in Conversion Rate!

Fix of the Hotfix was rolled out

Use Case #1: User Feedback Driven Decisions

4x $$$ to IaaS

Use Case #2: Resource / Cost Driven Decisions

Use Case #3: User Behavior Driven Decisions

confidential

Use Case: Accelerated User Driven Innovation

in Finance

confidential

Defining Performance Engineering You must model the system for

performance, before any code is written

You must test the model or idea for performance risk

You must understand the big picture to write small components of code

You must collaborate with others: developers and system architects

You must adapt your thinking and your relationships

“If you can impact the [performance of the] code before it is written, then and only then, can you consider your work to be performance engineering. Otherwise, you are just doing testing.”

- Jim Duggan, Gartner

confidential

Purpose-BuiltJobs

Capture-Playback

LoadRunner

LoadRunner

+ others

Open Sourc

e

Integrated COE Suites

Open Sourc

e

Cloud Tools

Open Source

confidential

DEV:DIY Scala Tool

Yourkit, JProfilerLocal log parsing

TEST:SmartBear/SoapUIDIY Selenium FX

Stand-alone JMeter on Laptop

LT01:Manual Setup and

ExecutionDIY Scala Tool

Curl & PerlShared NFS log parsing

PROD:Solaris tools - prstat

Dynatrace 4.2TraverseNagios

AWR & Ignite DB Monitoring

Tools, Teams and Environments: 2012

confidential

DEV:Dynatrace 4.2

Local log parsingJProfiler

Some JMeter + Plugins

TEST:Functional Test

Automation

LT01:Dynatrace 4.2

Local log parsing JMeter + Plugins in the labAutomated prstat captureAutomated Loadosophia

UploadIgnite DB MonitoringIntegrated Scenarios

PROD:Dynatrace 4.2

Solaris tools - prstatTraverseNagiosIgnite

Tools, Teams and Environments: 2013-14

confidential

DEV:Dynatrace 6.3

SplunkJProfiler

Rundeck + JMeter + PluginsReal-time InfluxDB &

GrafanaJMeter + Plugins

TEST:Functional Test

AutomationExploratory Testing

LT01:Dynatrace 6.3

One-off Logs to Splunk Rundeck + JMeter + Plugins

Real-time InfluxDB & Grafana

Ignite DB MonitoringIntegrated Scenarios

Notification to Wiki/Jira/Slack

PROD:Dynatrace 6.3

SplunkSolaris tools - prstat

TraverseNagiosIgnite

Tools, Teams and Environments: 2015-16

confidential

DEV:Self-Service

MeasurementsDynatrace 7

SplunkPTaaS & ECD

K8 & Prometheus App Dynamics

TEST:Functional Test

AutomationExploratory Testing

LT01:Continuous Execution

Dynatrace 7Splunk

PTaaS & ECDK8 & Prometheus

Ignite DB Monitoring

PROD:Dynatrace 7

SplunkSolaris tools - prstat

TraverseNagiosIgnite

Tools, Teams and Environments: 2017+

confidential

Performance Practice(production-like)

Production Performance

(“the Real World”)

Development(very unreal world)

Fast Performance FlowsPROMOTIONAL FLOW

FEEDBACK FLOW

operateinvent

confidential

Performance Practice(production-like)

Production Performance

(“the Real World”)

Development(very unreal world)

Fast Performance Feedback

6

5

4

1

2

3

CI CD

PROMOTIONAL FLOW

FEEDBACK FLOW

unit perf

essential perf

new releasestrategic tests

triage

operateinvent

design

confidential

Make Performance ContinuousIN DEVELOPMENT

Enable Self-Service Measurements – on demand execution of a small testEvery engineer on the team can build, submit and execute a testTest environments may be built on-the-flyConnected to CI system for unattended executions after nightly builds on on-demand pushes

IN LOADTEST Enable Non-stop schedule for

performance test executions Trending test results on measurement

thresholds, compared to baseline Use a split-schedule for high-priority

test execution - nightly continuous execution, self-service during the day

Connected to CD system to automate release feedback

In all environments use common tooling for monitoring, profiling and logging – on by default, automated and accessible to all.

confidential

Use Case: Accelerated User Driven Innovation

at Dynatrace

confidential

2011: APM about to be disrupted!

Migrate from On-Prem to VM, Cloud, Containers and PaaSArchitectures include micro-services, on-demand scaling, self-healing”Cloud Natives“ demand SaaS based solutionsDigital Transformers demand Analytics for Biz, Dev, Ops & SecMany new players on the market

confidential

Challenges to master!

Bridging the gap between ”New Stack“ and “Enterprise Stack“Deploying the same way our customers do: Continuously!Not disrupting current operations and slower moving customersAligning 300+ engineers across 3 different geosSolution: Innovation through Incubation!

2 major releases/yearcustomers deploy & operate on-prem

26 major releases/year170 prod deployments/dayself-service online sales SaaS & Managed

2011 2016

29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

believe in the mission impossible

6 monthsmajor/minor release+ intermediate fix-packs+ weeks to months rollout delay

sprint releases (continuous-delivery)1h : code to production

NOC lessons learnt

Shift-Left QualityQuality/Performance matters in Dev/Staging as well!Make Dev/CSA/PM dependent from Quality in trunk!

DevOps = start thinking like an Ops before Commit

Shift-Right Metricsenable DEVs defining quality metrics make DEVs to the primary consumers of their metrics

acting asEngineers

Role of Dynatrace DevOps Team

Dynatrace Managed/SaaSOrchestration Layer

Dynatrace Pipeline Visualization

Deployment Timeline

Log Overview using Dynatrace Log APIJIRA Integrations

&Product Managers

confidentialhttps://github.com/Dynatrace/ufo

Raising Awareness of Pipeline Quality

ruxit theme 2014.05.15

Be proud of your feature!DevOps NoOps

Dynatrace Transformation by the numbers

26

170

Releases / Year

Deployments / Day

31000 60hUnit & Int Tests / hour

UI Tests per Build

More Quality

~200 340Code commits / day

Stories per sprint

More Agile

93%Production bugs found by Dev

More Stability 450 99.998%Global EC2

InstancesGlobal Availability

confidential

Use Case: Accelerated User Driven Observations at

eCommerce

confidential

Trends across eCommerce

▪Development taking ownership of all quality issues, except Security

▪ Orchestration and deployment is moving to the public cloud (Jenkins, AWS, CloudAzure)

▪ CI is the norm▪ Serverless computing is becoming second nature▪ Rules based analytics are replacing first tier diagnostics,

moving human performance analysis to tougher issues

confidential

Cautionary Notes

▪Developer Architecture Myopia ▪ Loose Gates on Quality and Performance▪ Developer ownership on quality, but the need for

performance training▪ Broken links between the business and Dev/Ops -

particularly marketing▪ Pattern feedback loop to the code analysis level remains

immature ▪ With rules based analysis, the feeder tier for new

performance engineers is compromised.

confidential

Final Thoughts …

confidential

80%20%organization & culture technology

Confidential, Dynatrace, LLC

Develop

Ship

Deploy

Run

Scale

Compute

nodejs mongo db netty cassandra redis

ansible jenkins puppet chef

docker cloudfoundry rh openshift rh atomic rocket

core os rancher kvm busybox

mesos marathon kubernetes swarm

amazon azure openstack mesosphere calico weave

eureka/hystrix

A whole new technology stack & polyglot development

confidential

Ecosystem …

confidential

Dev/Test: Bad Code Changes

Ops/Biz: Unhappy End Users

Performance: Overloaded COEs

confidential

Dev/Test: Check-In Better Code

Ops/Biz: Use Real-Time End User Feedback

CI/CD/Performance: Stop Bad Builds Faster

confidential

confidential

AI-Driven Digital Operation Ecosystem

DAVIS / REST

Andreas GrabnerDevOps Activist@grabnerndi

Accelerate User Driven InnovationFebruary 2017

Mark TomlinsonPerformance Sherpa and CEO @mark_on_task

James PulleyPerformance Evangelist and CTONewCOE LLC, @perfbytes

Drag picture to placeholder or click icon to add

Drag picture to placeholder or click icon to add