real world problem solving using application performance management 10

Pre-Con Education: Real World Problem Solving Using Application Performance Management 10

Haroon Ahmed, Andreas Reiss

DevOps: Agile Ops

CA Technologies

CA APM SWAT

DO5X114E

@CAWorld

#CAWorld

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD

© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.

The content provided in this CA World 2015 presentation is intended for informational purposes only and does not form any type of

warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.

For Informational Purposes Only

Terms of this Presentation


Abstract

CA Application Performance Management 10 dramatically reduces the time needed to find and solve app problems. In this session you will learn about common problem-solving techniques used by experts to solve real-world app problems. You will get a chance to put these techniques to the test in a hands-on lab that mimics an interesting application performance problem.

HaroonAhmed

CA Technologies

Senior Director Engineering Services


Agenda

TRIAGE AND DIAGNOSIS STRATEGY

USE CASE 1 – PIG IN A PYTHON

USE CASE 2 – DEATH BY 1000 PAPER CUTS

USE CASE 3 – OUT OF GAS

1

2

3

4


Typical Application Flow

OS/HWOS/HW

RUNTIMEJVM, CLR, Java Script, PHP


APPLICATION SERVERJ2EE, .Net, Node.js, PHP


SynchronousAsynchronousSynchronous

Asynchronous

MainframeMainframe

MessagingMessaging

ERP, CRMERP, CRM

Web ServicesWeb Services

DatabasesDatabases

Micro ServicesMicro Services

CICS / IMS / TuxedoCICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET

SOAPRESTJMS

JDBC, ADO.NET

OS/HW



SynchronousAsynchronous

Mainframe

Messaging

ERP, CRM

Web Services

Databases

Micro Services

CICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET


What to Monitor in Production

OS/HWOS/HW





SynchronousAsynchronousSynchronous

Asynchronous

MainframeMainframe

MessagingMessaging

ERP, CRMERP, CRM

Web ServicesWeb Services

DatabasesDatabases

Micro ServicesMicro Services

CICS / IMS / TuxedoCICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET

SOAPRESTJMS

JDBC, ADO.NET

OS/HW



SynchronousAsynchronous

Mainframe

Messaging

ERP, CRM

Web Services

Databases

Micro Services

CICS / IMS / Tuxedo

Response

Request

SOAPRESTJMS

JDBC, ADO.NET

FRONTEND BACKEND

MUCK

RESOURCES


Key Focus Areas

• Business Transaction, URL, Web Service

• Servlet, ASP.Net, JMSFrontend

• Web Service, EJB, Enterprise Service

• POJOMuck

• Database, Mainframe, LDAP, Web Services

• JDBC, ADO.Net, CTG etc…Backend

• Thread Pools, JDBC Pools, Memory, CPU

• GC MonitorResources

Metrics

Events

Relationships


Stall & Concurrency Count Brewing problems, Hot spots

Response Time and Throughput Lagging indicator

Resources GC, GC per Interval/time

Thread Utilization

Error Count

Instance Counts

Metrics

10 Minute

ART

STALL


Interpreting Performance Metrics (Production)D

etai

l

Low

High

Ability to RespondReactive Proactive

Heap DumpsThread Dumps

Log Files

PingersProcess

BrewingLaggingHosed

Mean Time To ResolutionHigh Low

Response TimeThroughput

GC HeapCPU Utilization

Stall CountConcurrency Error Count

Instance Counts

GC Monitor

Thread PoolsJDBC Pools

Med


Traces

Error

Stall

Events

Status

Topology

Attribute


Relationships

Dynamic

Correlated View

Decorated

Timeline

Historical

Trending


Use Cases

1. PIG IN A PYTHON

2. DEATH BY A THOUSAND PAPER CUTS

3. OUT OF GAS


USE CASE 1

PIG IN A PYTHON

13


Symptoms: – Slower and slower over time or under load

– Consistent slowness

Root Cause:– External bottleneck

– Over-usage of external system

– Bad Coding: CPU bound component

– Network Saturation

– Threading: Chokepoint

Common Symptoms & Root Cause


In Team Center - relationship view, timeline and spark graphs

Look at frontend metrics, enable min/max to see outliers

Search tab and sort on average response time, enable min/max to see outliers

Run Transaction Tracer and look for long running components (to get single user transaction)

Where to Start…


Backend Bottleneck


Muck – Business Component Bottleneck

77% of the transaction time is spent waiting on the billing system


Framework Bottleneck – Bad XML


Use Case 2

DEATH BY A THOUSAND PAPER CUTS


Symptoms:– Consistent slowness

– Slower and slower under load

Root Cause:– Over-usage of external system

– Layer-itis

– The unending retry

Common Symptoms and Root Cause


In Team Center - relationship view, timeline and spark graphs

Look at frontend metrics, enable min/max to see outliers

Run Transaction Tracer and look for a bar code

Look at the Summary View and sort on # of calls. Look for high

call counts for one or more components in a single user

transaction

Where to Start…


Over Usage of External SystemNote: 4666 SQL calls in a single user txn - 2 SQL statements account for ~95% of SQL calls and ~50% of the total txn time!


Abuse of J2EE Framework


Misconfigured Resource – JDBC Adapter

Note the barcode of SQL calls. Also note commits after every JDBC Select. COMMIT call taking ~5 seconds!!!


Use Case 3

OUT OF GAS


Symptoms:– Sudden chaos

– Foreseeable lockup

– Slower and slower under load or over time

– Consistent slowness

– Sporadic hangs or aberrant errors

Root Cause:– External bottleneck

– Bad Coding: infinite loop

– The unending retry

– Internal resource bottleneck: over usage or under allocation

– Network Saturation

– Threading: Chokepoint, Deadlock/Livelock

Common Symptoms and Root Cause


APM– Sudden/gradual increase in stall counts and stalled events

– Increasing concurrency counts and decreasing responses per interval

– Investigator/search tab used to find components with stalls or high concurrency

Application Server Resources– Look for sudden or slow thread pool / JDBC pool depletion

OS Resources– Look for changes in CPU and Heap utilization

– Increases in CPU and Full GC time/count could point to bad code

– CPU Utilization decreases and JVM halts – out of application server resources

Where to Start…


External Bottleneck – DB Stored Procedure is the Root Cause

Thread Depletion

Increase Count against a DB stored procedure

Stored Procedure execution time from 25 min and UP


Stalls That Degrade Performance Over Time

2

Surge in Application Activity

Stalled threads degrading application reliability over time


Stall Events With Smart Instrumentation


Hands-on Lab

Challenge: Triage and Diagnose Using APM 10


2. Webview1. EM

3. Start

MathApp

4. Run

JMeter

4. Run Lab5

problem

scenario


Hands-on Exercise

1. Log into the lab VM (Administrator/CAdemo123)

2. Start the Enterprise Manager

3. Start Webview

4. Start MathApp

5. Start Baseline Load on MathApp

6. Wait 5 minutes

7. Start Lab5 problem scenario

8. Triage and Diagnose the problem

– Using ATC, Webview, Investigator tree, events etc.


Recommended Sessions

SESSION # TITLE DATE/TIME

DO5X189SHow to Achieve a Customer-Centric View in an Omni-

Channel World11/18/2015 at 1:00 pm

DO5X194SMonitor Microservices, Containers, Cloud Foundry and

Node with CA Application Performance Management11/18/2015 at 4:30 pm

DO5X193S

Customize CA Application Performance Management

with Tips for Using the CA Application Performance

Management Open APIs

11/19/2015 at 4:30 pm


Must See Demos

Application Performance Management and DevOps, featuring APM use in preproduction scenarios

Application Performance Management

Theater 5

Application Performance Management, Modern Monitoring, featuring the new APM Team Center


Theater 5

Ensuring a “5 star” mobile app experience with CA Mobile App Analytics

Mobile App Analytics

Theater 5

Unified Monitoring: APM Integrations including UIM


Theater 5


Follow On Conversations At…

Smart Bar


Theater 5

Tech Talks


Theater 5


Q & A


For More Information

To learn more, please visit:

http://cainc.to/Nv2VOe

CA World ’15

http://cainc.to/Nv2VOe

http://bit.ly/1wbjjqX

real world problem solving using application performance management 10

Technology