key lessons learned from running 900+ agents with

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Derek Abing | Sentry Insurance

Thorsten Roth| Dynatrace

Key Lessons learned from running 900+ Agents with Application Monitoring

Delete &Insert

Headshothere

2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Who am I

• Performance | Availability | Capacity | Management

• PACMan

• Change Management

3 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

I got a

Promotion

VP

4 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Our Environments – Dev Edition

40 x

40 x

1 x

5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Our Environments – Test Center Edition

100 x

370 x

43 x Servicing Agents, Plugins

and 25,000 Monitors

6 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Our Environments – Production Edition

350 x

100 x

43 x

50M

Servicing Agents, Plugins

and 20,000 Monitors

7 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Keeping up with the latest dynaTrace versions will help improve your dynaTrace performance.

Utilize the deployment guide offered by dynaTrace. This planning process consists of the following steps:

* Collect Sizing Data* Use the Deployment Sizing Calculator* Assess Results* Verify Results

dynaTrace Upgrades

8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Do More with Less

• Connect more Agents/Collector

• Reduce infrastructure costs

• Reduce complexity

9 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Use external Collector. Make sure you always run an external collector in test center or production scenarios!

Run multiple Collector Instances on a single machine where appropriate.Very useful if you have a lot of agent-less monitoring being done via custom plugins.

Separate PurePath and Monitoring CollectorsA PurePath Collector serves Agents.A Monitoring Collector runs Monitors and other plugins.

Use Collector Groups for Agent failover and load balancing

Collector Best Practices

Monitoring

Collector

PurePath

Collector

10 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

dtcollector -service install -instance collector2 -listen localhost:9997 -server MyDynaTraceServerdtcollector -service install -instance collector3 -listen localhost:9999 -server MyDynaTraceServerdtcollector -service install -instance collector4 -listen localhost:9996 -server MyDynaTraceServer

Multiple Collector Instances

11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Collector Groups

12 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Silent Installs

Each new major release of dynaTrace requires new versions of the components to be installed. Make new installs or updates go smoother by utilizing silent installs that are part of each installer.

Example on installing just the Java Agent into the default directory:

msiexec /passive /i C:\dynatrace-<install_version>.msi ADDLOCAL="JavaAgent,JavaAgentx64,DiagnosticsAgent“

Example on installing just the Java Agent into a different directory:

msiexec /passive /i C:\dynatrace-<install_version>.msi APPDIR="D:\dynaTrace\dynaTrace Agent 6.2.0“ ADDLOCAL="JavaAgent,JavaAgentx64,DiagnosticsAgent“

Example on installing just the .NET Agent into the default directory:

msiexec /passive / C:\dynatrace-<install_version>.msiADDLOCAL="DotNetAgent,DotNetAgent20,DotNetAgent11,DiagnosticsAgent"

13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Bootstrap Agent

The bootstrap agent allows the Agent to receive updates automatically without the need to upgrade the Agent itself every time you upgrade to a new dynaTrace version. Updates flow from Server > Collector > Agent.

The bootstrap agent will also allow you to deploy or rollback fixpacks/updates easily.

14 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Utilize the Controlled Agent Rollout feature

15 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Using symbolic links will allow you to upgrade the agents without having to update the JVM options. Works with Windows and Unix/Linux/AIX.

Example Windows 2008 command:

mklink /d C:dT “<dynaTrace_Installation_Directory"

Symbolic Links

16 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Instead of pointing directly at a Collector inside the agent mapping, set up a DNS alias for that Collector and point the agent configuration to the DNS alias. This will allow you to point a bunch of agents all at once to a different Collector if needed instead of having to modify every SUD.

DNS Aliases

17 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Instant Insights. Agents Everywhere.

https://github.com/dynaTrace

18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Database Partitioning Out-of-the-Box

https://community.dynatrace.com/community/display/DL/Performance+Warehouse+Partitioning+for+dynaTrace

19 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Configure storage settings for captured session data

20 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

System Profile Best Practices

• Don’t capture too much data

– Only capture context data you need for BTs and Root Cause Analysis

– Avoid using “*” (asterisk) in Servlet/ASP.NET Sensor

– Reduce length of SQL Capturing and Bind Values in ADO.NET/JDBC

– Exclude Exceptions and Log Messages you don’t need

– Exclude URLS that you don’t need

– Only instrument methods you need

– Avoid “Shotgun” instrumentation -> don’t instrument EVERYHING FROM EVERY CLASS

• Good rule of thumb is to get as specific as possible

• Auto-sensor is your friend

• Avoid Measure Explosions

– BT Splitting that results in many split values (i.e. username, IP, etc)

21 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

22 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

23 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Diffing and Merging of System Profiles

https://community.dynatrace.com/community/display/DTFORUM/Diffing+and+Merging+dynaTrace+System+Profiles+-+Prototype

LIVE DEMO

24 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Reserve licenses in the Dynatrace Server Settings

Can reserve by System Profile and Agent Group

License Reservations

25 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

A monitoring tool is only useful when it is available . Use other tools or custom scripts to monitor critical pieces of dynaTrace (i.e. the windows services associated with each component, scrape the log files for errors, etc).

Take advantage of the self-monitoring measures inside dynaTrace and utilize the Deployment Health Dashboards.

Work with a DBA to ensure optimal health on your dynaTrace database.

Monitor the Monitoring

26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Focus on….

* Skipped Events, Skipped PurePaths

* PurePath Length and PurePath Buffer

* CPU and Memory on both Server and Collector

* Measure Explosion

* Performance Warehouse Performance

Deployment Health Dashboards

27 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

28 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

29 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

30 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

31 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Hands on Training• Architecture

• Installation

• Instrumentation

• General Concepts

• System Profile

• Sensors

• Terms

• UEM

• Dashboarding

• Practice what you learn

32 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Implementation and Management Tips

• Keep up to date with latest dynaTrace versions

• Utilize silent installs for fast deployment

• Take advantage of symbolic links

• Use DNS aliases for your agent configurations

• Manage your License allocation

• Use Database Partitioning if applicable

• Use Collector Groups

• System Profiles and Collector best practices

• Monitor the Monitoring

• Create/Modify/Share custom plugins to extend the functionality

• Train your users

33 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

34 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

35 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

What’s Next?

37 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

What’s Next?

• Enable UEM on more applications

• Leverage Dynatrace Web to onboard more people

• Investigate new Agent technologies

• Mainframe Agent

• Database Agent

• Mobile Agent

• Invest in more Agent licenses

• Integrate dynaTrace with other internal tools and processes

• Develop plugins where applicable

38 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

39 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

40 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE