key lessons learned from running 900+ agents with

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

Derek Abing | Sentry Insurance

Thorsten Roth| Dynatrace

Key Lessons learned from running 900+ Agents with Application Monitoring

Delete &Insert

Headshothere


Who am I

• Performance | Availability | Capacity | Management

• PACMan

• Change Management


I got a

Promotion

VP


Our Environments – Dev Edition

40 x

40 x

1 x


Our Environments – Test Center Edition

100 x

370 x

43 x Servicing Agents, Plugins

and 25,000 Monitors


Our Environments – Production Edition

350 x

100 x

43 x

50M

Servicing Agents, Plugins

and 20,000 Monitors


Keeping up with the latest dynaTrace versions will help improve your dynaTrace performance.

Utilize the deployment guide offered by dynaTrace. This planning process consists of the following steps:

* Collect Sizing Data* Use the Deployment Sizing Calculator* Assess Results* Verify Results

dynaTrace Upgrades

https://community.dynatrace.com/community/display/DOCDT62/Collect+Sizing+Data

https://community.dynatrace.com/community/display/DOCDT62/Use+the+Deployment+Sizing+Calculator

https://community.dynatrace.com/community/display/DOCDT62/Assess+Results

https://community.dynatrace.com/community/display/DOCDT62/Verify+Results


Do More with Less

• Connect more Agents/Collector

• Reduce infrastructure costs

• Reduce complexity


Use external Collector. Make sure you always run an external collector in test center or production scenarios!

Run multiple Collector Instances on a single machine where appropriate.Very useful if you have a lot of agent-less monitoring being done via custom plugins.

Separate PurePath and Monitoring CollectorsA PurePath Collector serves Agents.A Monitoring Collector runs Monitors and other plugins.

Use Collector Groups for Agent failover and load balancing

Collector Best Practices

Monitoring

Collector

PurePath

Collector


dtcollector -service install -instance collector2 -listen localhost:9997 -server MyDynaTraceServerdtcollector -service install -instance collector3 -listen localhost:9999 -server MyDynaTraceServerdtcollector -service install -instance collector4 -listen localhost:9996 -server MyDynaTraceServer

Multiple Collector Instances


Collector Groups


Silent Installs

Each new major release of dynaTrace requires new versions of the components to be installed. Make new installs or updates go smoother by utilizing silent installs that are part of each installer.

Example on installing just the Java Agent into the default directory:

msiexec /passive /i C:\dynatrace-<install_version>.msi ADDLOCAL="JavaAgent,JavaAgentx64,DiagnosticsAgent“

Example on installing just the Java Agent into a different directory:

msiexec /passive /i C:\dynatrace-<install_version>.msi APPDIR="D:\dynaTrace\dynaTrace Agent 6.2.0“ ADDLOCAL="JavaAgent,JavaAgentx64,DiagnosticsAgent“

Example on installing just the .NET Agent into the default directory:

msiexec /passive / C:\dynatrace-<install_version>.msiADDLOCAL="DotNetAgent,DotNetAgent20,DotNetAgent11,DiagnosticsAgent"


Bootstrap Agent

The bootstrap agent allows the Agent to receive updates automatically without the need to upgrade the Agent itself every time you upgrade to a new dynaTrace version. Updates flow from Server > Collector > Agent.

The bootstrap agent will also allow you to deploy or rollback fixpacks/updates easily.


Utilize the Controlled Agent Rollout feature


Using symbolic links will allow you to upgrade the agents without having to update the JVM options. Works with Windows and Unix/Linux/AIX.

Example Windows 2008 command:

mklink /d C:dT “<dynaTrace_Installation_Directory"

Symbolic Links


Instead of pointing directly at a Collector inside the agent mapping, set up a DNS alias for that Collector and point the agent configuration to the DNS alias. This will allow you to point a bunch of agents all at once to a different Collector if needed instead of having to modify every SUD.

DNS Aliases


Instant Insights. Agents Everywhere.

https://github.com/dynaTrace

https://github.com/dynaTrace


Database Partitioning Out-of-the-Box

https://community.dynatrace.com/community/display/DL/Performance+Warehouse+Partitioning+for+dynaTrace

https://community.dynatrace.com/community/display/DL/Performance+Warehouse+Partitioning+for+dynaTrace


Configure storage settings for captured session data


System Profile Best Practices

• Don’t capture too much data

– Only capture context data you need for BTs and Root Cause Analysis

– Avoid using “*” (asterisk) in Servlet/ASP.NET Sensor

– Reduce length of SQL Capturing and Bind Values in ADO.NET/JDBC

– Exclude Exceptions and Log Messages you don’t need

– Exclude URLS that you don’t need

– Only instrument methods you need

– Avoid “Shotgun” instrumentation -> don’t instrument EVERYHING FROM EVERY CLASS

• Good rule of thumb is to get as specific as possible

• Auto-sensor is your friend

• Avoid Measure Explosions

– BT Splitting that results in many split values (i.e. username, IP, etc)


Diffing and Merging of System Profiles

https://community.dynatrace.com/community/display/DTFORUM/Diffing+and+Merging+dynaTrace+System+Profiles+-+Prototype

LIVE DEMO

https://community.dynatrace.com/community/display/DTFORUM/Diffing+and+Merging+dynaTrace+System+Profiles+-+Prototype


Reserve licenses in the Dynatrace Server Settings

Can reserve by System Profile and Agent Group

License Reservations


A monitoring tool is only useful when it is available . Use other tools or custom scripts to monitor critical pieces of dynaTrace (i.e. the windows services associated with each component, scrape the log files for errors, etc).

Take advantage of the self-monitoring measures inside dynaTrace and utilize the Deployment Health Dashboards.

Work with a DBA to ensure optimal health on your dynaTrace database.

Monitor the Monitoring


Focus on….

* Skipped Events, Skipped PurePaths

* PurePath Length and PurePath Buffer

* CPU and Memory on both Server and Collector

* Measure Explosion

* Performance Warehouse Performance

Deployment Health Dashboards


Hands on Training• Architecture

• Installation

• Instrumentation

• General Concepts

• System Profile

• Sensors

• Terms

• UEM

• Dashboarding

• Practice what you learn


Implementation and Management Tips

• Keep up to date with latest dynaTrace versions

• Utilize silent installs for fast deployment

• Take advantage of symbolic links

• Use DNS aliases for your agent configurations

• Manage your License allocation

• Use Database Partitioning if applicable

• Use Collector Groups

• System Profiles and Collector best practices

• Monitor the Monitoring

• Create/Modify/Share custom plugins to extend the functionality

• Train your users


What’s Next?


What’s Next?

• Enable UEM on more applications

• Leverage Dynatrace Web to onboard more people

• Investigate new Agent technologies

• Mainframe Agent

• Database Agent

• Mobile Agent

• Invest in more Agent licenses

• Integrate dynaTrace with other internal tools and processes

• Develop plugins where applicable

40 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

key lessons learned from running 900+ agents with

Documents