open source application performance monitoring for the ... · note: the configuration applies to...
TRANSCRIPT
![Page 1: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/1.jpg)
OPEN SOURCE APPLICATION PERFORMANCE
MONITORING FOR THE CROWD: INSPECTIT
JUNE 17TH, 2016,
IVAN SENIĆ, CHRISTOPH HEGER, MARIO MANN
WORKSHOP KARLSRUHER ENTWICKLERTAG
![Page 2: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/2.jpg)
AGENDA
1. Introduction
2. Setup of inspectIT
3. Configuration of the Instrumentation
4. -- Break --
5. Analysis of Performance Problems
6. Collaboration of Dev and Ops
7. Questions and Answers
![Page 3: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/3.jpg)
Who are we?
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 3
Ivan Senić Mario Mann Christoph Heger
ivansenic mariomann hegerchr
… from the inspectIT APM Open Source Project
18 contributors and growing
![Page 4: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/4.jpg)
Why is performance important?
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 4
Response Time Limit Perception
0.1 second Users feel their actions are directly causing something to happen
1.0 second Users feel they are navigating freely and stay focused on their current train of
thought
10.0 seconds Users feel a break of the flow, get impatient while waiting and leave the site
Jacob Nielsen, Power of 10: Time Scales in User Experience, 2009
Business Impact
Brand perception, conversions, revenue, shopping card abandonment, page views,
and search engine rankings
User Experience
Reports
49% of online shoppers expect load times of <=2 seconds
50% of online shoppers abandon page if load time is > 2 seconds
42% of men and 35% of women don‘t give a company a second chance after experiencing slow load times
![Page 5: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/5.jpg)
The HealthCare.gov Fiasco
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 5
Summary [Kevin Surace, fastcompany.com; Byron Wolf, CNN]
Among the worst software launches in history
Hundreds of millions of dollars were spent
Outages and slow response times were the norm
U.S. government had to apologize
Companies like Google and Red Hat rushed
to the rescue
Never actually tested for scalability and performance before its launch
[Image: healthcare.gov]
125k plus concurrent users peak
Load times between 6 and 22 seconds
depending on business transaction
![Page 6: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/6.jpg)
Ellen DeGeneres‘ Oscars selfie crashes Twitter
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 6
[Image: Ellen DeGeneres]
Summary
2.6 million retweets in about 2 hours
20 minutes service disruption
Be aware of the unexpected
[Image: Twitter.com]
![Page 7: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/7.jpg)
Metrics
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 7
Good Metrics
Latency (response time of requests, methods, third-party services, etc.)
Utilization (CPU, memory, etc.)
Platforms (devices, browsers, apps, resolutions, geolocation, etc.)
„[…] what you measure you improve […]“ – Larry Dragich
Good Practices
Complete coverage of the entire stack for business transactions (no siloed approaches)
Transaction-level detail
Code level
Correlated across all component tiers
Any device (browser, smartphone, tablet, etc.)
Selective transaction-level business context capture
e.g., user, shopping card details like product, number of items, revenue
Synthetic probing for availability and general health
Recommendation: Gil Tene – How NOT to Measure Latency, https://youtu.be/lJ8ydIuPFeU
![Page 8: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/8.jpg)
Take Away
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 8
[Image source: http://content.hollywire.com/sites/default/files/Success-Kid.jpg]
![Page 9: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/9.jpg)
INSPECTIT
BEHIND THE SCENES
![Page 10: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/10.jpg)
Overview
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 10
System Node 1
JVM
App 1
App 2
System Node n
Target
System
Central
Measurement
Repository
inspectIT UI
Agent
Agent
inspectIT-docker inspectit
Agent-based Approach
![Page 11: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/11.jpg)
Hooking into class loading
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 11
Loading Linking Initialising
class HelloWorld {
public void sayHallo() {
Sysout.println(„Hello World :)“);
}
}
Before method hook
After method hook
start = System.nanoTime();
end = System.nanoTime();
ExecTime = end-start;
Agent Hook Code
(very) simplified
Control flow
redirection
Application Code
![Page 12: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/12.jpg)
Sensors
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 12
Sensor Type Information
Timer Method execution time
Invocation Sequence Method execution tree (trace)
SQL SQL query, parameter bindings
Logging Log message
Exception Java exception
HTTP Data Servlet methods, HTTP parameters
System data e.g., CPU utilization, memory utilization, number
of loaded classes
JMX e.g., thread pool size
![Page 13: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/13.jpg)
SETUP
![Page 14: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/14.jpg)
USB Sticks
Content of USB Stick
• inspectIT
• Agent
• Central Measurement Repository
• UI
• DVDStore Sample Application
• Workshop Tutorial
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 14
![Page 15: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/15.jpg)
Workshop Description
https://github.com/inspectit-labs/workshop
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 15
![Page 16: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/16.jpg)
Practice Time
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 16
https://github.com/inspectit-labs/workshop/blob/master/SETUP.md
![Page 17: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/17.jpg)
INSTRUMENTATION
CONFIGURATION
![Page 18: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/18.jpg)
Instrumentation Configuration
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 18
Why is instrumentation important?
Defines kind/amount of data you will see in inspectIT
Defines how much overhead you will add to the monitored application
![Page 19: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/19.jpg)
Instrumentation Configuration
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 19
What is meaningful instrumentation?
Rule: get as much as possible information with as less as possible instrumentation points
Note: the configuration applies to the complete JVM
Not a good idea:
Methods like toString(), equals() & hashCode()
Getters and setters
Any method that is executed often and it’s expected to be executed fast
Better:
Services, DAOs, database calls, remote calls, etc.
From experience: method that perform some work (execute, perform, calculate, transform, load, etc..)
![Page 20: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/20.jpg)
Register
Look up
configuration
Process
Configuration
Initialize
Agent Registration
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 20
1
Agent CMR
How instrumentation works in inspectIT?
Name, IP
Configuration
JVM
Start
![Page 21: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/21.jpg)
Analyze
Class
Parse Class
Method Instrumentation
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 21
1
Agent CMR
How instrumentation works in inspectIT?
Configuration
Processing
Class structure
Method X, Y
JVM
Load
Class
Modify
Bytecode
![Page 22: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/22.jpg)
Instrumentation Configuration
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 22
Configuration concepts
Complete instrumentation configuration is located on the CMR and managed by the UI
3 basic structures:
Profile
defines on which classes/methods should specific sensors be applied
defines exclude rules
Environment
holds basic settings for the agent (sending strategy, sensor options, etc.)
defines which profiles will be used in the given environment
Mappings
maps single (or multiple) agents to one environment
Mappings
Environments
Profiles
n:1
n:m
![Page 23: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/23.jpg)
Tool Time
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 23
![Page 24: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/24.jpg)
Practice Time
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 24
https://github.com/inspectit-labs/workshop/blob/master/INSTRUMENTATION.md
![Page 25: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/25.jpg)
PERFORMANCE ANALYSIS
![Page 26: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/26.jpg)
Performance Analysis
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 26
What are the goals of the performance analysis?
Monitoring is about getting concrete Numbers continuously
Diagnosing is about finding the root cause when it is slow
Find the root cause(s) of the performance
problem(s)
Propose changes (optional)
Understand application behavior
(where and when is our application slow)
Detect anomalies deviating from the baseline
![Page 27: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/27.jpg)
Reactive vs Proactive Performance Problem Analysis
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 27
• Performance Tests
• Regression Testing
• Performance Prediction
Development
• Application Monitoring
• Real User Experience Monitoring
• Synthetic Monitoring
Operation
Reactive
Proactive
e.g., Search is slow!
e.g., Is search slow?
Application
Deployment
![Page 28: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/28.jpg)
Performance Data
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 28
Proactive
Load test
Provides most accurate timing data
simulate numerous virtual users running numerous use cases
can be executed on the production-like environment
Sometimes time consuming to create and maintain
Alternatively: one-user testing
Does not provide accurate timing as there is only one user using the application
Can show some obvious performance bottlenecks
Can help in reaching the optimal instrumentation configuration
No time needed for setup
Reactive
Production monitoring data
Provides real data of the application
Real load scenario
Real user interactions
Real user experience
![Page 29: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/29.jpg)
Tool Time
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 29
![Page 30: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/30.jpg)
Timer Data
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 30
Method Instrumented Duration Exclusive
duration
foo() Yes 100 ms 70ms
bar() No 30 ms 30 ms
baz() Yes 30 ms 30 ms
Method Instrumented Duration Exclusive
duration
foo() Yes 100 ms 40ms
bar() Yes 30 ms 30 ms
baz() Yes 30 ms 30 ms
Duration vs Exclusive Duration
Case A
Case B
![Page 31: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/31.jpg)
Practice Time
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 31
https://github.com/inspectit-labs/workshop/blob/master/ANALYSIS.md
![Page 32: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/32.jpg)
COLLABORATION
![Page 33: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/33.jpg)
Why is Collaboration important?
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 33
Dev
Ops Biz
Problem Analysis
Performance Problem
Dev Ops
Monitoring
Data
Application
Problem
Future Past DevOpsBiz
Dev Ops
![Page 34: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/34.jpg)
Storages
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 34
Storage
Trace C
Trace B
Trace A
Export Data from the CMR to local Storage
Save
Any collected data
Traces
Timer
Exceptions
etc.
Storages can be stored locally
Offline analysis of data
Export/Import functionality in for data exchange
![Page 35: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/35.jpg)
Practice Time
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 35
https://github.com/inspectit-labs/workshop/blob/master/COLLABORATION.md
![Page 36: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/36.jpg)
What‘s coming next?
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 36
Instrumentation changes w/o application restart
Traces covering distributed JVMs
Monitoring Dashboards
End User Experience Monitoring
Recognition of new and existing problems
2017 2016
![Page 37: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/37.jpg)
Contact
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 37
Gitter
https://gitter.im/inspectIT/chat
Web
http://www.inspectit.rocks
![Page 38: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/38.jpg)
FREE Private APM Workshop
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 38
http://www.novatec-gmbh.de/dienstleistungen/apm-geduld/
APM Strategy
APM Tools
Performance Analysis
Performance Prediction
Monitoring Performance
Testing
![Page 39: Open Source Application Performance Monitoring for the ... · Note: the configuration applies to the complete JVM Not a good idea: Methods like toString(), equals() & hashCode() Getters](https://reader034.vdocuments.mx/reader034/viewer/2022050509/5f99f2f2efad7e73b1473ae3/html5/thumbnails/39.jpg)
Take Away
17.06.2016 Open Source Application Performance Monitoring for the Crowd: inspectIT 39
[Image source: http://content.hollywire.com/sites/default/files/Success-Kid.jpg]