all the leaves aren’t brown - nyougnyoug.org/presentations/2013/winter/ezell-nyoug-2013...all the...

Chuck Ezell

Senior Applications Tuner, datAvail

All The Leaves Aren’t Brown

Many Ways to Profile Your Application Code

12/19/2013 2www.datavail.com

• Value in Profiling: the When, What & Why

• Profiling & Profilers: the right tool for the right job

• Server & Client Profiling Tools

• Summary

Agenda


Value in Profiling

• 80% of unplanned outages are due to ill-planned changes made by operations or developers

• 60% of availability and performance errors are the result of mis-configurations

• Looking Ahead: through 2015 80% of outages impacting mission-critical services will be caused by people and process issues. More that 50% will be caused by change/configuration/release integration and hand-off issues.

• 80% of incidents are caused by changes made to the IT environment including application code

Source: Causes of Failure in Web Applications, Soila Pertet and Priya Narasimhan December 2005, Parallel Data Laboratory Carnegie Mellon University, Pittsburgh, PA 15213-3890


Tools Compared

• How hard is it to Setup and Manage?

• Is it instrumented, a sampler or something else?

• What will it profile? Resources, Threading, Memory, Transactional Views, Single focused or Multi-faceted?

• What is the output? Will it keep history? How long?

• Is it useful as a proactive or reactive solution?

• Does it provide any analysis?

• Is it expensive?

Questions We Have to Ask:


Common Java Problems

1. Frequent or Long Garbage Collection App is spending too much time in GC

Causes periods of extreme slowness across entire app

2. Application runs out of Heap Memory (Memory Leak) App runs out of memory

Causes the application to crash

3. Thread Contention Thread locks on a resource for too long.

Causes app to slow or lock and can impact entire app

4. Resource Contention (Pooled Resources) Not enough instances of a pooled resource (e.g. DB connections)

Causes extreme slowness in app, can be wide spread or specific to certain transactions.


Profiling Tools

Just to name a few…

jstat/jstack/jmap


Server Profiling Tools: jstack, jmap jstat

• Jstat, jstack & jmap– Requires java 1.6 and up

– Only runs in the terminal

– Probably already installed

– Easy to use

– Does not retain any history unless you redirect it to a file

– No analysis provided just raw data

– Provides running threads & memory footprint

– Can show locked threads

– Does not require a kill -9 to the JVM for a dump

– Can be great in a pinch

Setup: None

jstat/jstack/jmap



You will need java 1.6 or higher to utilize these tools fully.

To check your java version just enter “java -version” at the terminal.



Check your running JVMs by doing a quick “ps –ef | grep –Xmx”



Using “jps” is much easier.

Using the -l argument will give you the full package name. With this you can grep out JVMs.



Pass process id to jstackUsing the –l argument will give you locked threads

jstack -l 3439



Running a “ps -eL” and grep for the JVM process id

You get the lightweight process ids

What about long running threads or thread contention?



Convert the binary lightweight thread id to hex. Then grep the output of jstack.

You can find the lightweight process id for the thread in the thread stack after converting it into hex



Passing the same JVM process id to jmap will give you the memory footprintfor this point and time.

jmap -heap 3439

But how would we monitor the Garbage Collection (ongoing memory growth/shrink) ?



Passing the same JVM process id to jstat will give you the garbage collection

jstat –gc 22174

jstat –gcoldcapacity –t 22174



jstat –gcnew -t 22174 1000 10

1. Not a great proactive solution.2. Great tools for reactive fire fighting.3. Doesn’t maintain history or reporting.4. Output is essentially on screen and snapshot in time.5. Will help answer questions and identify real problems now.6. No Application Context & is not correlated with End User.


Server Profiling Tools: Visual VM

• VisualVM / Oracle Mission Control– Both are free.

– Profiler much like the depricated jConsole

– Collects classes, threads, GC, CPU and more…

– Capture Heap Dumps & Browse them

– Also built into NetBeans IDE (my IDE of choice)

– Has a plug-in architecture

Setup: Easy


Server Profiling Tools: VisualVM

You’ll need JMX enabled

JMX = Java Management Extensions

• Open a terminal and do a quick “ps –ef | grep jmxremote”

• Look for the JVM Process ID• Then look for the port it’s using for the JVM you want to monitor• Might or might not need authentication



JMX Console & JVM Browser



Flight Recorder



Flight Recorder

1. Not a great proactive solution.2. Great visual tool for reactive fire fighting.3. Maintains history in the form of independent snapshots.4. Output is essentially on screen but can be called back up with

the saved snapshots.5. Will help answer questions and identify real problems now.6. Is not correlated with End User Experience.7. No transactional views. Hard to determine application context.8. Does require JMXREMOTE settings to be added to JVMs. In

addition to security & ports to attach remotely.


Server Profiling Tools: Splunk

• Splunk– There are often multiple JVMs and Apache instances running– Often times we miss issues in the logs because there’s too much to grep,

wouldn’t know what to look for or don’t have the time– Any log or flat file data can be forwarded– Monitor code base for changes, network logs for drops, client machines– The best results come from the best input so collect everything– Create monitoring alerts, dashboards, configure transaction monitoring– Plugin architecture for many different tools (BigIP, Hadoop, MySQL, etc…)– Limited by time & creativity

Splunk Forwarders On Each Server or write directly to server syslog.

Data from the log is collected and forwarded on to Splunk where it’s indexed and acted on.

Setup: Moderate to Easy



• sql_plan_timestamp="09-30-2013 02:51:01", database=OADEV, plan_hash_value=1649080325, sql_id=696c56fyx1rwn, operation="INDEX", options=FULL SCAN, object_name=GL_SETS_OF_BOOKS_U2, object_owner=GL, optimizer=, cost=1, cardinality=1, cpu_cost=7521, io_cost=1, time=1, filter_predicates="HOI2"."ORG_INFORMATION1"=TO_CHAR("GSOB"."SET_OF_BOOKS_ID")"

• sql_plan_timestamp="09-30-2013 02:51:01", database=OADEV, plan_hash_value=2300147657, sql_id=36dx1vwfqmn2j, operation="TABLE ACCESS", options=FULL, object_name=FND_PRODUCT_GROUPS, object_owner=APPLSYS, optimizer=, cost=2, cardinality=1, cpu_cost=7271, io_cost=2, time=1, filter_predicates="sql_plan_timestamp="09-30-2013 02:51:01", database=OADEV, plan_hash_value=171240807, sql_id=7m21y56326kha, operation="INDEX", options=FULL SCAN, object_name=GL_SETS_OF_BOOKS_U2, object_owner=GL, optimizer=, cost=1, cardinality=1, cpu_cost=7521, io_cost=1, time=1, filter_predicates="HOI2"."ORG_INFORMATION1"=TO_CHAR("GSOB"."SET_OF_BOOKS_ID")"

SELECT 'sql_plan_timestamp=""'|| to_char(timestamp,'mm-dd-yyyy hh24:mi:ss') || '", database='||b.name||

', plan_hash_value='||plan_hash_value||', sql_id='||sql_id||', operation="'||operation||

'", options='||options||', object_name='||object_name||', object_owner='||object_owner||

', optimizer='||optimizer||', cost='||cost||', cardinality='||cardinality||', cpu_cost='||cpu_cost||

', io_cost='||io_cost||', time='||time||', filter_predicates="'||filter_predicates||'"' as sql_plan_info

FROM V$SQL_PLAN a, v$database b

where (options like '%SKIP%' OR options like '%FULL%' OR options like '%PUSH%')--OR options like '%RANGE%')

and object_owner != 'SYS'

and timestamp > (sysdate - 1)

order by timestamp asc;

We start with a basic SQL Script. And format the output into key/value pairs.

Then add that SQL to a shell script and set cron to run it and produce the output.



Client Tier

Capture Opportunities

• Web Browser traffic

• Oracle Forms running in client side JVM

• Windows Event Logs

• CPU Load, Memory, Network Stats

• Active Directory Monitoring

• FND Diagnostics from Oracle Forms



Application Tier


• Oracle JVM – OACore

• JVM Threads

• JSP Request/Response

• Apache Requests (Access/Error)

• Forms Processes

• Linux/AIX Server Stats

ApplicationServers

Concurrent processing server



Database Tier


• Oracle Alert Logs

• User Session Data

• Concurrent Requests

• SQL IDs, SQL Profiles & Outlines

• SMON/PMON

• DBWR/LGWR

• Hot Database Objects


Server Profiling Tools: jmap/jstack

jstack & jmap can be setup to forward data to splunk….


Server Profiling Tools: jmap/jstack

1. Not out of the box solution – requires some planning and skill.2. Can be a great proactive solution.3. Great visual tool for reactive AND proactive fire fighting.4. Maintains history in the form of indexed data.5. Output can be anything: alerts, notifications, pdf reports, charts,

dashboards, html pages, etc…6. Will help answer questions and identify real problems now AND

provide data to research patterns over time.7. Does require setup and server to collect and maintain indexes.8. Can provide the greatest flexibility for building what you want if

you have vision, time, skill & creativity.9. It is priced reasonably well – based on the data indexed per day.


Server Profiling Tools: HP Diagnostics

• HP Diagnostics– Integrates will with LoadRunner

– Full Application/Server Diagnostics

– Java Profilers can be configured to save results

– Will capture details for analysis later

– Is a bit confusing to use

– Older architecture utilizing some client tools still

– Profilers Can be server intensive

– Instrumented

Setup: Involved



1. Can be a great proactive solution.2. Great visual tool for reactive and proactive fire fighting.3. Maintains history in the form of snapshots and data.4. Output is saved snapshots and can be recalled from history.5. Will help answer questions and identify real problems now as

well as help identify patterns.6. Each java probe can be intensive (with more probes, etc…)7. You don’t know what you don’t know.8. Is not correlated with End User Experience.9. No transactional views. Hard to determine application context.10. Can be costly to purchase.



• AppDynamics– Captures Full Transaction

– Provides Monitoring History

– Full Profiler down to threads and exception stacks

– Auto Detects other devices and services such as other DBs

– Employs powerful and efficient Java Byte Code Injection

– Full Splunk Cross Integration

– Fast Root Cause Analysis

– Alert monitoring & configurable SLAs

– Scheduled & Automatic Diagnostic Sessions

– Modern architecture (just plain pretty)

– Very low server resource consumption (Netflix runs thousands)

Setup: Easy


Server Profiling Tools: AppDynamicsDashboard – Runtime application architecture discovery, modeling and display

Number 2 on Gartner's 5 Dimensions of APM chart.


Server Profiling Tools: AppDynamics

Not instrumented – Auto detects all connections through Java Byte Code.



User-defined transaction profiling. This provides application context that is intuitive.




Component deep-dive monitoring in application context. - full call stacks and find bottlenecks.




Health Rule Notifications – JVM, SQL, Transaction Slowness, etc…



Monitor JVM Heap Segments and Ratios




End-user experience monitoring



1. Super Easy to configure and get started.2. Great visual tool for reactive and proactive fire fighting.3. Maintains full history in the form of diagnostic snapshots and

data.4. Output can be alerts, notifications, pdf reports, dashboards,

etc.5. Will help answer questions and identify real problems now as

well as help identify patterns.6. Very little overhead (often less than 2%).7. You don’t have to know what you don’t know. Plug and play.8. Can be costly to purchase if you’re running multiple JVMs.9. But there is a free version for single JVM setups.


Client Profiling Tools: Fiddler

• Fiddler (HTTP Request Profiler)

– Easy setup just download and launch

– Create, save & recreate http requests

– Investigate & Debug http header problems

– Capture or Create form submits and re-run them anytime

Setup: Easy



Review http request statistics



Capture and Manipulate http requests for testing purposes



Recreate http POST data requests for testing



Simulate connection speeds, disable cache, measure response time


Tools Compared

Threads MemoryTransaction

ViewsMaintains

HistoryProactive

MonitoringAnalysis Setup

Splunk If Logged If Logged If Logged YES YES No Moderate

HP Diagnostics

YES YES No YES YES Sort Of Involved

AppDynamics YES YES YES YES YES YES Easy

jstack/jmap YES YES NoRedirect

To Flat FileNo No Easy

VisualVM YES YES NoLimitedExports

YES Sort Of Easy

VisualStudio YES YES NoLimitedExports

No YES Easy

Fiddler No No YESLimitedExports

No No Easy

Firebug No YES YESLimitedExports

No No Easy


Questions?

Chuck [email protected]

478-714-1615Senior Applications Tuner, datAvail

• Profiling & Profilers: the right tool for the right job

• Server Profiling Tools

• Client Profiling Tools

• The Heat of the Moment: differentiating profiling tasks

all the leaves aren’t brown - nyougnyoug.org/presentations/2013/winter/ezell-nyoug-2013...all the...

Documents