Cost-Effective Performance Management Investing in Intelligence for Peak Performance and Stability

Automation, if applied strategically to detecting, diagnosing, and resolving performance issues, can have

a dramatic impact on productivity, environment stability, and operational effectiveness. This white paper

explains several cost-cutting strategies and innovative methods to improve the effectiveness of

performance management. Arcturus Technologies provides a performance management solution based

on artificial intelligence that uses embedded expert knowledge to monitor, analyze, and detect

performance problems proactively.

J2EE AND SOA PERFORMANCE MANAGEMENT

Current Challenges

When service thresholds are exceeded, most performance management products only collect, correlate,

and communicate the data pertaining to performance issues. They leave the analysis, interpretation and

decisions about next steps up to the support staff. A “decide and react” approach is the source of much

downtime and revenue loss. To remain competitive, you must have processes in place to identify

unknown problems before they occur.

FACT: End users report 73% of all performance problems. This means that

current monitoring solutions are insufficient and Miss problems.

The reality of static monitor provisioning is that, over time, the monitors become out of sync with the

crucial services keeping the environment stable. It takes much effort to keep monitors up to date in

complex environments. And with most monitoring solutions, once the monitors no longer cover the

entire environment, it takes a comprehensive audit involving architecture, development, and monitoring

staff to identify what is missing and what is no longer applicable. If this effort isn’t done, your end users

or partners soon make you aware of these blind spots at the worst possible time: during production

outages. To stay on top of changes, today’s performance management solutions must be dynamic, fast,

customizable and intelligent.

"Today's applications are an increasingly complex aggregate of many moving parts, and

problems lurk not only in the code, but in architecture, platform configuration, links with

dependent internal or external systems, and infrastructure capacity. Resolving these

issues in production by throwing hardware at them is the most costly and ineffective

approach possible. Yet because hardware is cheap, it is often the remedy of choice."

Jean-Pierre Garbani

VP and Research Director at Forrester Research,

Performance Management and the Application Life Cycle.

Because performance tuning is so complex and time-consuming, many enterprises are running application

servers with configurations that are not fully optimal. Also, a common belief in some IT organizations is

that performance tuning is optional as long as their current response times are sufficient. A lack of early

and proactive performance management leads to expensive corrections later in the project.

FACT: The cost and effort of correcting software defects and environment

issues increases exponentially later in the development lifecycle

It's easy to overlook the fact that tuning increases the stability and longevity of a server. Outages can be

caused by tuning incorrectly or not tuning at all. JVM Tuning is a complex, but vital process in getting the

best performance from J2EE & SOA application. Heap size, the Garbage Collection mechanism, and many

other JVM Tunables impact an enterprise application's performance significantly. If the JVM is not tuned

correctly, performance not only gets impacted, but often leads to outages.

It is much easier to find the root cause of the problem if you see it happening while you are there.

Unfortunately, root-cause analysis often occurs after an incident, when diagnostic data and server state

information is no longer available. This is because the stability of your production environment can’t wait

for a detailed and thorough analysis to occur. So, support teams proceed with steps to restore

performance and functionality. This leads to a loss of root-cause data.

FACT: most organizations require six service desk calls to identify the correct problem resolution group.

Cost-Effective Performance Management

Invest in Knowledge

The degree to which organizations can maintain healthy environments and deliver the consistent

performance depends directly on the capability and availability of their own experts. When subject

matter experts relocate, they take their knowledge and experience with them. The cost required to

replace and train new experts is expensive. A good performance management solution must:

• Reduce the dependence on individual's knowledge

• Foster knowledge sharing

• Reduce the dependence on single subject matter experts, and

• Let non-experts reach sound conclusions based on expert proven and scientific fact.

The solution should also come with pre-defined, industry-accepted knowledge. That way, your staff can

spend time on more important issues, instead of re-inventing the wheel. Organizations with expert

performance management knowledge integrated into the fabric of their operational processes achieve

consistent performance, reduce training time and cut costs.

“An investment in knowledge pays the best interest.”

Benjamin Franklin

Preventive

When done properly, application server configuration and JVM performance tuning reduce the number of

application failures from performance and service failures, errors, and outages. Appropriate tuning also

directly lowers costs for the application infrastructure.

Performance tuning is typically time-consuming and difficult—a task that is often pushed to the end of a

project plan, where it is least beneficial. Yet, tuning is essential to excellent performance.

Automated

The ideal performance management solution should provide a method to automate problem diagnosis,

monitor provisioning, and root-cause detection. This automated knowledge base must be organic,

adjustable, and easily updated with proprietary knowledge, because your applications grow and change

over time.

When issues arise, automation not only provides lightning-fast response time, but also frees your staff to

handle more important work.

Dynamic

Over time, performance monitoring solutions must be able to change as the application changes. The

most efficient way to get comprehensive and dynamic problem detection is to evaluate common systemic

indicators of performance problems holistically, using rule-based analysis.

Combining decision-making intelligence with your application tier’s own internal diagnostic capabilities

gives you a continuous, unobstructed view into every service’s availability, without the maintenance

problems and complexities that come with static monitoring solutions.

PERFORMANCE SIMPLIFIED

Architecture

Server

Applicare uses a proven and trusted infrastructure, built upon core JEE and SOA principles. That way,

Applicare runs equally well in both developmental and production environments. Applicare server hosts

Applicare console and Applicare engine.

Agent

When a server is in a critical state, remote server monitoring can incorrectly label the server as failed. To

access crucial diagnostic data, Applicare deploys a lightweight agent to the managed servers.

Applicare is a JAVA and JEE application, and supports all platforms running on JAVA. Applicare provides

agents for Oracle WebLogic, IBM WebSphere, Red Hat JBoss, Tomcat and generic java app server.

Complete Life-cycle Performance Management

Peak-Performance and Stability in Pre-production Environments

The core goal of Applicare is to provide the diagnostic tools and resources necessary to help staff attain

peak performance. Applicare does this by providing benefits unique to both pre- and post-production

environments.

Saves Time and Money

Enterprises often don’t tune servers frequently or thoroughly enough, simply because the process is so

time consuming. Applicare can analyze the environment, identify areas where there is room for

improvement and can either automatically fix the issue or advises on the fix. So, instead of spending

countless hours mired in the tuning process, your staff can spend their time on more productive tasks.

The financial savings from automating the diagnosis and detection of performance problems add up

quickly when you have fewer outages, better performance, and proper hardware usage.

Automate Troubleshooting and Diagnosis

Whether you are debugging performance problems solo or as a team, it’s useful to see how both code

changes and different tuning variables affect performance. Using Applicare is similar to looking over the

shoulder of a team of performance experts – and your staff will become educated in why certain

recommendations and decisions were made.

Automate Expert Configuration Analysis

Applicare analyzes your entire server environment’s configuration, including your server’s configuration

and deployed applications. It generates a comprehensive, meaningful, and detailed report of

recommendations. Advisor uses embedded knowledge that our performance experts gathered to

illustrate where you are not leveraging your server’s environment correctly. Then, it provides guidance on

how you can resolve those issues and achieve optimal performance.

Applicare also guides by using best practices, and provides specific examples and areas in your

environment where you can benefit most from applying them. This automated guidance is equivalent to

having a team of performance experts evaluate your environment and provide feedback on how well you

are leveraging your infrastructure.

Automate Problem Resolution

Applicare takes performance tuning and problem resolution automation to a new level by providing the

framework for fixing common performance issues with built-in knowledge. Applicare can automatically

make adjustments to your configuration, based on your own custom knowledge or its embedded expert

knowledge. In addition, Applicare provides methods for knowledge input that all can use—from novices

to experts, engineers, and developers. This means you can use your own expertise and experience to

customize the hands-free detection and fixing of your runtime and configuration issues.

Intelligent Detection and Analysis in Production

Automate Runtime Problem Detection

Setting complex thresholds, rules and limits associated with static monitoring solutions are un-necessary

with Applicare. It uses a built-in knowledge base and behavior tracking to determine what conditions can

cause outages or a reduction in performance.

Identify and Recommend Solutions

Applicare examines the run-time environment, detects issues, and generates a report showing what

abnormalities were detected.

Applicare’s recommendations are specific and actionable.

Automate Root-Cause Analysis

Applicare automatically detects server failure conditions in production and auto-analyzes the server state

(using thousands of matrices) to generate a root-cause analysis report instantly. Applicare comes with

embedded knowledge of common failure conditions, and constantly examines the production

environment for any signs of imminent service failure. It identifies server failure conditions with 99.9%

accuracy and advises you about the probable cause, saving precious time. You can also manually execute

the Applicare’s detection process to analyze the run-time state of the servers at any point in time. It is

lightweight and efficient, and provides the least intrusive way of finding problematic patterns that develop

in your environment. It also generates an e-mail that not only tells you that the server failed, but also tells

you why. Because of Applicare’s holistic approach to monitoring, you often get an email notice with the

root-cause information before your users are impacted.

Capture Complete Pre-Failure Conditions

Applicare detects failure conditions and creates a snapshot called a Blackbox. Blackbox lets you analyze

the server state just before system failure.

In this way, you get all the information you need—even if the incident happened at night while you were

sleeping. Applicare can analyze thousands of matrices almost instantly (something that would take you

significant time to do on your own). Applicare drills down to the transaction level in determining the root

cause and identifying the culprit transactions—just one of the many features that makes Applicare a must

have tool.

APPLICARE’S USAGE SCENARIOS

Application Runtime Architecture Discovery

Applicare discovers Application Runtime Architecture and builds Server Interaction Map. Applicare does

that by automatically discovering the transactions flowing through the system and modeling interactions

between various components/services and servers. It provides transactions details like # of calls, Errors,

Average Time/call, Max Time, Min Time & SLA breach. Applicare tracks calls between different types of

servers such as Apache web servers, Java Application servers, Database servers and servers.

Applicare’s Application Runtime Architecture Discovery

End User Experience Monitoring

Applicare provides two ways to monitor End User Experience.

1. Synthetic Transactions

2. Real User Transactions Monitoring

Synthetic Transaction Monitoring

Applicare Synthetic Transaction Monitor helps detect and warn performance issues as soon as they occur

and generally well before users are impacted. When combined with Business Transaction Monitoring

and/or IntelliSense, you can diagnose and pinpoint the issue all the way to the individual method/SQL in

your code.

Applicare Synthetic Transaction Monitor are very easy to configure. Below is a synthetic transaction

monitor that periodically invokes an item display page:

Synthetic Transaction Configuration

This configuration has:

• A page loaded once every minute, and

• An SLA value of 2 seconds set for the page.

Applicare allows creation of fairly complex scenarios using Apache JMeter and running them as synthetic

transactions. Any JMeter script can be used as synthetic transaction/s allowing complex transaction

definitions including validation of content returned from the web servers. Synthetic transactions can be

executed from a local Applicare server or from cloud servers located in different parts of the world

Applicare sends an alert whenever SLA is breached. The Synthetic Transaction Analyzer displays the results

of monitors over time. The image below displays one transaction taking considerably more time than

usual.

Synthetic Transactions Execution Results

Just double-click on the data point to see the actual call graph for that invocation, complete with method

names, parameters and SQLs.

Synthetic Transactions Execution Details

The above image shows the call graph for the transaction in question. The problem? A single SQL is taking

over 97% of the total time spent on the transaction.

Real User Experience Analysis

Applicare Real User Experience Monitor tracks the performance of your production applications for real

user interactions.

Applicare Real user Experience Dashboard

The dashboard provides an overview of your applications performance stats, such as:

• Number of transactions of each type

• Load

• Average response times

• Number of errors

Applicare Real user Transaction Details with time spent on each leg

Double click on any transaction on this view to see the execution graph on the backend application

server/s.

Real user Transaction Execution call graph on the backend server

Transaction view displays stats for every individual request including the entry time, total time spent and

breakup of the time consumed. And, you can filter the transactions by different criteria to detect slow and

failed transactions.

TRANSACTION MONITORING & PROFILING

Applicare provides two ways to monitor transactions.

• Automatic Transaction Profiling with IntelliTrace & IntelliSense

• User Defined Transaction Profiling

Our experience told us that user defined transaction profiling is always reactive in nature. Experts enable

profiling on certain areas to get to root cause after a problem has occurred. This approach works for the

repeatable problems but doesn't work for the problems that are not easily reproducible. It also is very

much people dependent approach. If you have the right people at the place and have all the required

diagnostic data, you get the good results but that isn't the case most of the time. It is rare to have the

right monitoring in place at the time of first occurrence of the problem. That makes root cause analysis an

iterative process and the problem has to occur a few times before the cause is determined.

Our solution to this reactive approach is, let the expert system make the decisions on what to monitor and

when. This way we have adaptive monitoring and problems are automatically detected without requiring

any human intervention. This way whatever part of the system is developing problem Applicare gathers

the data for that part only and adjusts monitoring as needed.

AUTOMATIC TRANSACTION PROFILING

One important aspect of Applicare transaction monitoring, compared with other similar profiling

solutions, is the very low overhead—even at this detailed level of performance data gathering.

The above image shows the performance of a web application running on a Java application server with:

• No instrumentation,

• IntelliTrace & BTM enabled (with no stack capture), and

• IntelliTrace & BTM with full stack capture.

This shows that there is no noticeable difference in throughput (96.9/sec vs 96.5/sec).

Applicare achieves this very low overhead by intelligently adjusting how deep it goes into a call graph

during run-time and what transactions to monitor. It will not gather data beyond methods that take very

little time to execute, compared to the overall transaction. Similarly Applicare doesn't gather data for

transactions that fall under normal category based on automatic behavior profiling. This reduces the

overhead significantly.

As shown in the example below getSpecialRequirements method is not instrumented at the beginning but

does get instrumented by IntelliTrace as the behavior changes.

IntelliTrace Analyzer

IntelliTrace Analyzer

IntelliSense works in a similar way to IntelliTrace, but gathers individual transaction execution call graphs

for transactions that take longer than usual. IntelliSense automatically detects transactions and

builds/maintains performance baselines for the detected transactions. IntelliSense saves the full call graph

whenever a transaction execution deviates significantly from the baseline. This helps to track down

performance issues that may happen occasionally, and that are hard to reproduce.

A slow transaction execution detected by IntelliSense

IntelliSense: Full call graph of the detected transaction

Business Transaction Monitoring (BTM)

Applicare Business Transaction Monitoring (BTM) lets you monitor transactions important to your

business. You can adjust the level of monitoring from very high-level performance stats to very detailed

performance stats (where all performance data, including full call graphs for every individual execution, is

saved). Applicare BTM can track transaction that span multiple servers and will show the consolidated call

graph consisting of executions on multiple servers in a single UI.

Since IntelliTrace automatically detects transactions, you can easily define business transactions and can

specify SLAs. BTM gives a clear picture into the health and performance of business transaction executions

across all application servers.

Health and performance stats of business transactions across servers

Individual transaction execution details with full call graphs

Individual transaction execution details with full call graphs

The above call graph shows a transaction where a business transaction on a different server was invoked,

ValidateCreditCard on server Tomcat, as part of the main transaction. BTM displays the merged call graph

allowing the user to track performance of transactions that spans multiple servers.

BTM Dashboard shows # and type of transactions going through the system, average response time and

load on the system.

Business Transaction Analyzer Dashboard

Tuxedo Monitoring

Applicare monitors following in Tuxedo

• Services execution timing

• Service Queue monitoring & alerting

• Transaction correlation between JAVA & Tuxedo

According to this call graph, over 99% of the time is spent on the Tuxedo (Jolt) call: CheckShippingAddr.

From here, you can go to the Tuxedo Analyzer to see the performance of your services on Tuxedo server.

That way, you can see if Tuxedo services are causing the performance issue.

Tuxedo Analyzer Transactions View

In the above image of the Tuxedo Analyzer, the CheckShippingAddr is taking over 5 seconds to execute

on Tuxedo-2 server, compared to the same service running under half a second on Tuxedo-1 server. This

indicates a performance issue with Tuxedo-2 server.

Tuxedo Analyzer

Tuxedo Analyzer SLA View

Tuxedo Queue Analyzer

Applicare monitors the queue for each service available in Tuxedo and can alert on certain thresholds.

Application Performance Analytics

In addition to the transaction tracing capabilities described above, Applicare provides extensive real time

and historical data analysis capabilities.

Applicare Dashboard

Applicare dashboard shows the status of each server, transactions going through the system, historic/real

time resource utilization, errors in the log files and alerts. It also allows testing of network speed between

various servers participating in transaction.

http Profiling SQL Profiling

Connection Leak Profiling SQL Error Detection

Heap Analyzer (Real Time)

OS Analyzer (Real Time)

JVM Thread Dumps Analyzer

Processes Analyzer (Real Time)

Application Server Analyzer

(Historic & Real Time)

Garbage Collection Analyzer

JVM Threads Analyzer (Real Time)

The JVM Analyzer provides a very detailed view into a server’s thread behavior, detects Deadlocks, detects

stuck threads, and provides an interface for viewing stack traces.

Deadlock Detection

When a deadlock occurs, the tab will turn red to alert you that a deadlock exists. You can see the deadlock

threads on deadlock tab. This tab shows all available information about the deadlock situation. Then you

can capture the data and send it to development so they can analyze the code. You can also take a thread

dump (via Applicare) to display the current threads and look for the BLOCKED state.

Stuck Thread Detection

A stuck thread is a thread that is blocked and can't return to the thread pool within a defined period of

time. When an application thread is blocked, it can't quickly finish its job and be reused. In most

situations, the cause of stuck threads is also the cause of poor system performance. This is because the

stuck thread interferes with regular task execution.

The “Stuck Threads” tab will turn red and display the diagnostic information to help you identify the root

cause of the stuck thread.

Stuck Threads are often caused by the code opening a network connection without specifying a network

connect or read timeout. When a timeout isn’t configured for each method call that involves networking,

it will potentially create a block because it will wait infinitely for a response. The fallback timeout in this

case comes from the OS which is by default set very long (e.g. 5-10 minutes). Especially in the case of

SOA and Web Services where there are many network calls, timeouts need to be defined properly.

Stack Trace Browser

The Stack Trace tab provides a convenient way to view the stack traces of each active thread

Thread Dumps

This view allows you to save a thread dump of a running server and analyze the saved dumps at a later

time. The saved snap shots contain all thread information displayed on the real time view and any

deadlock or stuck threads detected at the time of creating the thread dump.

Applicare auto detects dead lock or stuck thread situations and automatically creates a thread dump and

sends an alert of the situation.

In addition to various pre built analyzer & reports, Applicare allows creation of custom analyzer (historic &

real-time) for any device & property that is accessible either via SNMP or JMX.

Log Analyzer & its integration with BTM

Applicare has a built in log search engine that provides easy searching capability.

Another feature that comes handy when dealing with legacy systems is BTM integration with Log

Monitoring. This gives and easy way to find data in log files that relates to a certain transaction. Just right

click on the parameters passed to a method and Applicare will find the data in the log files for that

parameter at the time of the transaction.

Knowledge Sharing

Knoms (short for Knowledge Modules) is a way to build knowledge in Applicare that is executable and

sharable. This knowledge can be best practices or patterns causing problems. Knoms allow preventing

outages & performance slowdowns from the known patterns. With KnomBank.com we have opened up

knoms to the whole world and any expert in any part of the world can share/demonstrate their expertise.

We have a couple of 100 members on KnomBank from almost all the continents. This way ones a pattern is

discovered anywhere in the world and shared on KnomBank, Arcturus certifies that pattern and sends it to

Applicare via knowledge update.

Knoms can be build on any property or combination of properties that is accessible via JMX or SNMP.

An added benefit of Knom Builder is that it can be used to test various conditions on an environment. It

comes very handy when you are doing an analysis on a very complex environment and want to see if

certain conditions occurred e.g. did requests have to wait for a connection pool, ejb or mdb at any point of

time or did packet error rate on the network exceed a certain level.

SUMMARY

In today’s economy, your organization must find innovative solutions to improve performance and stability

while cutting costs. Applicare provides a performance management solution that not only generates

instant ROI, but also helps you improve performance over time. This reduces the frequency and duration

of costly outages.

Applicare prevents problems that might occur in your system. By automatically analyzing and detecting, it

simplifies the entire process to just a few clicks—providing better performance and resource utilization.

With Applicare, you can become proactive in your task to keep application performance at its peak

(because you and your customers cannot afford the outages and downtimes that come from being

reactive).

Applicare intelligently measures application performance, auto analyzes and auto tunes your

environment, and proactively warns of system downtime symptoms and their probable causes before

your customers are impacted. With Applicare™, Arcturus Technologies takes enterprise application

performance management to the next level, with predictable and peak-performing SOA environments.

Know exactly who is impacted, why, and when – and how to find a solution to this situation.

To request a free demo, visit http://www.arcturustech.com, or contact sales at 866.262.7971.

ABOUT ARCTURUS Arcturus Technologies, Inc. is the pioneer and leader of knowledge based technology for SOA application

performance optimization and health management.

Arcturus Technologies provides an artificial intelligence based APM solution that leverages embedded

expert knowledge to proactively monitor, analyze and detect potential problems in advance so that

companies can avoid serious and costly downtime. With Applicare™ installed, companies can have the

peace of mind that their applications will run efficiently and if problems develop Applicare™ will provide

timely notification and targeted, specific and actionable resolution recommendations.