wwrug13 investigating performance issues outside of the norm

Danny KellettJava System Solutions

Investigating performance outside of workflow and indexes

© 2013 WWRUG Canada Inc. All Rights Reserved 2

Latency- What is it and why is it such a big deal?- How do I test for it?- How to use this information to demonstrate performance for your users

Queues & Threads- What are they- How to read the logs- How to let your AR Server tell you how many it needs

Plugins- What types there are- How they can impact performance- The process of diagnosis and fix

Agenda


Objects and Results

Objectives- To understand that no matter how many CPUS and RAM you have, a poor

network can bring an application to its knees- Demystify the confusion about queues and threads- Understand the black boxes called plugins

Results- Understand what these are and to give you the tools to understand your

own AR System

Skills developed- To sit through a very technical and possibly boring tech talk and live to talk

about it- The knowledge to understand the “not so well documented” parts of the AR

System

© 2013 WWRUG Canada Inc. All Rights Reserved 4© 2012 WWRUG Canada Inc. All Rights Reserved

Latency“Higher latency decreases app response time, user performance and perceived app quality”


Latency :: What is it and why is it such a big deal?

What is it?- In a network, latency, a synonym for delay, is an expression of how much

time it takes for a packet of data to get from one designated point to another. There are two typical types: One way

– The time from the source sending a packet to the destination receiving it

Round trip– The one-way latency from source to destination plus the one-way

latency from the destination back to the source

Latency is not bandwidth- Two key elements of network performance are bandwidth and latency. The

average person is more familiar with the concept of bandwidth as that is the one advertised by manufacturers of network equipment. However, latency matters equally to the end user experience



Why is it such a big deal to us and BMC Software?- The BMC AR System architecture has multiple

network node points Each line in the diagram is effected by latency If each line added even milliseconds, it all adds

up! If any point adds a delay then the whole “trip” is

effected

The AR System API (ARAPI) is very “chatty”- Instead of eating with a big spoon it eats with a

small spoon



Real life example- ITSM 7.6.04 SP2, load balanced environment as the

diagram. Browser was Firefox v21.0. URL is HTTPS

Test- From the incident console, double clicking an

incident.

Baseline (Initial, first load example)- Took a measurement on the current network to get

the number of trips, amount of data and response time in seconds

Second test with latency- With added latency of 50 ms from the client

(browser on desktop) through the load balancer and to one of the Mid Tier servers

50ms}



Results of Baseline (Initial, first load example)

50ms}# Trips Amount of Data Time (seconds)

Client to server 65 126.6k

Server to client 103 318.6k

Total 168 445.2k 9s

Results of test with 50ms added latencyTotal 168 445.2k 12s

Test summaryJust 50ms of network latency in just one piece of the BMC architecture, from the browser to the Mid Tier, can add ⅓ to your end user response times!


Latency :: How did I do those tests?

Before you start, understand these things- What happens when you type the Mid Tier address in the URL bar

Your browser will use your desktop network configuration to get the network details of your Mid Tier.

First is will look at your local hosts file for the Mid Tier host name. If it is not in there it will ask the Domain Name Service (DNS) If your Mid Tier has an IP address of 192.168.0.5 and it is configured in

your DNS database, then the browser will connect to it and everything works, you see the application etc

BUT if you added a line in your local hosts file (c:\windows\system32\drivers\etc\hosts) so that your desktop believes its not that IP address but a different one e.g. 127.0.0.1 then the browser will try and connect to that.

127.0.0.1 is something called a loopback adaptor and its basically means your own machine you are typing on. And unless you have a Mid Tier running on your machine, it will fail.



Before you start, understand these things- What happens when you type the Mid Tier address in the URL bar (part 2)

What if you had a Mid tier on your desktop and you added the same Mid Tier host name to your local hosts file with 127.0.0.1?– Your browser would still say the correct URL address but you would be

connecting to the Mid Tier on your desktop and not the one on the network.

- OK so why do I need to know that? If you installed a piece of software on your desktop that wasn’t a Mid Tier

but something that connected to your REAL Mid Tier on the network, 195.168.0.5, BUT delayed all connections, adding latency …. Then this is called a Proxy and this is what I used.



Add a proxy on the desktop to simulate latency



Find your Mid Tier real IP using ping or nslookup- <<TODO>> Insert screenshot of ping itsm7604.javasystemsolutions.local and

get back 195.168.0.5

Insert that value into the proxy app as the MAP IP

Add the Mid Tier URL host name to the loopback address in your local hosts file

Click Start on the proxy. Use your browser as before and record the timings.


Latency :: How can this predict your response times

Obtain your users latency times to the Mid Tier server.

Latency Open Incident

London 15

Paris 45

Houston 484

EXAMPLE



Obtain your users latency times to the Mid Tier server.- Using ping, which uses ICMP but sometimes turned off on network

equipment- Those times are round trip, so the time its taken from client to server AND

back from server to client. Therefore when testing, half those values!



Obtain your users latency times to the Mid Tier server.- Using http-ping, which uses the actual web server and this will always be

slower than a ping due to the fact it waits for the web server to respond with a HTTP code

- Those times are round trip, so the time its taken from client to server AND back from server to client. Therefore when testing, half those values!


Latency :: How can this predict my response times

TODO INSERT SCREENSHOT OF FREE LATENCY APPOR maybe do a Demo if the Demo laptop could run vmware etc. Need to find out from the WWRUG people


Latency :: How can this predict my response times

Using free tools, you can test the response times of all your users from your own desktop!

Latency Open Incident

London 15 6s

Paris 45 8s

Houston 484 30s

EXAMPLE


Latency :: If your latency is high

Speak to your network teams about Quality of Service (QOS)- Some network equipment can prioritise certain protocols. The Mid Tier uses

either HTTP typically on port 80 or HTTPS typically on port 443

Make sure your architecture has as little latency as possible between the Mid Tiers and AR Servers and more importantly between the AR Server and the database.- ITSM 7.6.04 with approx 900 concurrent users fires approx 127 SQL

statements per second at the database. Latency would bring the app to its knees!!

Install local Mid Tier instances near your end users.- There is more traffic between the browser and the Mid Tier than there is

from the Mid Tier to the AR Server.

Or if you DARE! Make customisations to workflow to remove trips altogether.- Tell boring story at large outsourcer


Threads & Queues


A queue is an entry point into the AR System- They are identified by a number, and sometimes referred to as an RPC

queue. Here are some examples 390600 = Admin 390603 = Escalation 390620 = Fast API calls (just a name without intending to indicate

performance) 390635 = List API calls (just a name as well but was aimed at things that

search and return lists/large amount of data)

A queue can have one or more threads defined for them. On startup, each thread creates a connection to the database that it uses throughout its existence. Threads only close when you shutdown the server or it cannot connect to the databaseOne queue has one or more threads

Threads :: Lets all get up to speed on queues & threads

20


If an API call gets routed to a queue and all the current threads are being used, it will look at the Max Threads value configured for that RPC queue. If the current thread number is lower, then it will create another thread and use that.If the list queue is at max resource, it will put the work on fast queue and vice versaIf both are full, it will move the work to the admin threadSo, the system has a set of queues -- some pre-defined, some private and defined per instance -- and each of them has processing threads as configured in the ar.cfg/ar.conf

Threads :: Lets all get up to speed on queues & threads

21


Google/Search BMC Support/Search ARSlist/Search BMC SupportBMC Atrium Core 8.0.00_20120921_docs.pdf- Page 87 - set Min Threads to 5 and Max Threads to 10- 262, same information repeated on 356, 383

Fast threads — At minimum, the same number as you have CPU cores; at maximum, 3 times the number of CPU cores, but no exceeding 32

List threads — At minimum, the same number as you have CPU cores; at maximum, 5 times the number of CPU cores, but not exceeding 32

- Page 2048 CPU x 1.5 for the Private queue.

SW00427239 - Fast and List threads are not set as per the recommended Queue settings.Doug Mueller ARSList post - In theory, there is no reason you cannot have 10s or even 100s of threads in a queue.

Threads :: Confusing or mixed messages

22


I do not believe the amount of threads is based on the number of CPUs alone. In all my tests, the CPU usage never went over 55% (excluding Developer Studio work)If the infrastructure can handle it E.g. an MSSQL database has a maximum 32767 connections. So theoretically, if the AR Server could fire that many connections and process them, then why not?Just for now think about if a connection to a database is doing some long query and is held up, the CPU on the AR Server is still the quickest component in the architecture and will have to wait and therefore it will do “other things”Its like saying a car can only handle 100BHP- Sure it can handle more if the rest of the car components and driver can

handle it!

Threads :: Confusing or mixed messages

23


There are so many variables that make your system unique- CPU types, HT, SMT, Cores etc

Virtual CPU vs a bare metal CPU differs http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensi

tive-Workloads.pdf http://scn.sap.com/thread/1646435

- Virtualisation or bare metal- Operating systems and settings

Therefore after 14 years of experience, researching hours and hours, googling the WHOLE internet on system architecture, posting on so many forums my answer is - Every environment is different so suck it and see! Get the system to tell you!- And here’s how I do it...

Threads :: In my experience then answer is…

24

http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf

http://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf


In a queue, if all the threads are constantly busy then you need to do some investigation- If you have a high number of threads already, and these threads are taking

too long to complete. E.g time to look at the queries to the database and work with your DBA to speed them up

- If the above doesn’t work, or the DBA doesn’t want to play ball, then it’s time to increase the Max Thread count in the AR System Administration Console

Each queue and thread takes system resources such as CPU power and memory. If those resources are maximised, then it’s time for an upgrade or another AR Server in the server group

Threads :: Here’s how :: Simple principles

25


In case you didn’t know, you can have the multiple log data in one file. Therefore start a log of API, SQL and filter

Threads :: Here’s how :: Step 1 – Create logs

26

Run this log in your peak periods if you can. Or if you are not live, then use a volume and performance testing applicationThis log will get large so make sure you have enough space


There are a couple of tools you can use- ARLogAnalyzer

https://communities.bmc.com/docs/DOC-2973- AR Log File Analyser

http://www.missingpiecessoftware.com/products/ar-log-file-analyser

Threads :: Here’s how :: Step 2 – Run Log Analyzer

27

https://communities.bmc.com/docs/DOC-2973

http://www.missingpiecessoftware.com/products/ar-log-file-analyser


Understanding the detail in the AR logs- Green -- The same user- Purple -- The same RPC 390620 which is the Fast queue- Red -- TWO RPC ID’s meaning two different API calls being processed

Every call that the dispatcher thread receives is assigned an RPC ID that can be used to identify the call from the time the call is placed into the queue until a response is sent back to the client

- Cyan -- One thread executing both API calls one after the other

Threads :: Here’s how :: Step 2 – Run Log Analyzer

28


Verify the number of threads is actually running. If the numbers in the log file match the Max Threads then you know at some point all threads were utilised!

Threads :: Here’s how :: Step 3 – What to look for?

29

ARLogAnalyzer AR Log File Analyser

Both results show : Fast Max is 30 but 27 being usedList Max is 40 but 35 being used


Thread idle time is the time from when a thread completes some work and then has to start work againTherefore the lower the idle time, the busier the thread isRemember that this will probably spike during the working day but this is truly the best way to monitor when your busy periods are and how stressed your system isThere is no such thing as 0 idle time. Even getting work from the dispatcher takes at least some time- Therefore ignore the MIN Idle Time examples include 0.0007

Look for very small numbers on the AVE idle time column

Threads :: Here’s how :: Step 4 – Idle time

30


You can identify when the AR Server has needed to increase the thread count on a queueWe can see one queue 390626, which is configured to start with 6 threads (Min Threads value below)

Threads :: Identifying busy periods

31

Looking through the log, we can see the thread number synchronously increment for the first 6, 28 to 33, then we see 130


Note the thread id 0000000130 underlined in redSearch your log file for TID: 0000000130

Threads :: Identifying busy periods

32

The above screenshot of the log entry is on one line but I had to cut it to fit on the slideFind the first instance of the thread id (TID) and note the time in this example is 13:08. This is when the AR Server decided it needed to create a new thread on the 390626 queue


Another way of identifying number of threads started- AR System Administration Console > System > General > Review Statistics

Threads :: Server Statistics

33


Plugins…


Three types of plugins. Each do a different thing- AREA – External Authentication- ARF – Filter plugin which are used to extend actions of filters- ARDBC – Access data outside of forms but mimic the behaviour of forms

There are two types of plugins. C and Java- Which obviously relates to the programming language they are built with

C- Runs through an executable file arplugin.exe (windows), arplugin (*NIX)

Java- Surprisingly runs from separate java processes, or java virtual machines

They have completely separate configuration, logging output etc

Plugins :: Lets all get up to speed on plugins

35


C specific- arplugin.exe / arplugin started via armonitor and configured to run in

armonitor.cfg / armonitor.conf E.g. /opt/bmc/ARSystem/bin/arplugin –s srv1 –i /opt/bmc/ARSystem

- Configured through the ar.cfg / ar.conf- “Plugin:”, “Plugin-Path:” & “Plugin-Port:” apply only to the C plugin daemon.- How to identify them? In the ar.cfg/ar.conf

Plugin: Then .dll on Windows, or .so or .a on NIX systems E.g. Plugin: ServerAdmin.so E.g. Plugin: ardbcconf.dll

- Logging is controlled through the AR System Administration Console


36


Java specific- A Java process is started via armonitor and configured to run in armonitor.cfg

/ armonitor.conf E.g. /usr/java/jdk1.6.0_06/jre/bin/java -Xmx512m -classpath

/opt/bmc/ARSystem/pluginsvr:/opt/bmc/ARSystem/pluginsvr/arpluginsvr75.jar com.bmc.arsys.pluginsvr.ARPluginServerMain -x svr1 -i /opt/bmc/ARSystem

- Typical ITSM instance has 4 Java plugin servers running Primary plugin server Full Text Search Engine 2 CMDB plugin servers

- Configured via three seporate pluginsvr_config.xml files- How to identify them? Within pluginsvr_config.xml files and some are

aliased in the ar.cfg/ar.conf- Logging is controlled through each log4j_pluginsvr.xml files


37


2 main types of functionality within the plugins1-Way and 2-Way- 1-Way is the AR Server calls the plugin and the plugin returns a response- 2-Way is the AR Server calls the plugin but in order to complete that request,

the plugin must connect back to the AR Server and lookup some data, and then returns a response

The two way plugins are typically the ones to look out for with regards to performance- E.g. REMEDY.ARDBC.APPQUERY


38


Same process of assigning queue numbers permitting the monitoring of data within the logs- E.g. If the a plugin is configured on 390624 then what API and SQL the plugin

fires against the AR Server, will be in the API and SQL logs with: <Client-RPC: 390624 >

Plugins :: Monitoring the configuration in logs

39


Seen in the log files when viewing the Overview ConsoleFind if its a C or a Java plugin by looking in the ar.cfg/ar.conf- Server-Plugin-Alias: REMEDY.ARDBC.APPQUERY REMEDY.ARDBC.APPQUERY

srv1:9999- See port number as :9999 so if Plugin-Port: 9999 then its a C plugin.

Otherwise you can tell its a Java plugin.

Searched for in the java plugin config xml (pluginsvr_config.xml)

Plugins :: Example - REMEDY.ARDBC.APPQUERY

40

Now look for the above classname line in the log4j_pluginsvr.xml


Change warn to trace, restart and open the arjavaplugin.log

Plugins :: Example - REMEDY.ARDBC.APPQUERY

41

Now look for the above line in the log4j_pluginsvr.xml

ITSM OOTB, this plugin is not configured to run on its own queue. Add the line in the pluginsvr_config.xml


Plugins that connect back to the AR Server are clients just like the User Tool and Mid Tier etcMost, if not all, are not configured to run on queues OOTBIt’s OK to run trace logs in production as long as it is managed!- No better log than a real users, working system, filled one- Have some automated scripts for log rotation with email notification- Just to get a weeks worth of usage statistics

Include this analysis in your capacity management assessment- E.g. If we add 1000 users, will my need to increase my thread count?

Plugins :: Summary

42


Full screen slide example


Add your conclusions here.

Conclusion


Add your discussion points here…

Question/Discussions


Contact information

Wrap-up

wwrug13 investigating performance issues outside of the norm

Documents

wwrug canada

big deal

latencyhigher latency

big spoon

current network

poor network

user performance

bmc ar system architecture