ainars galvans - hints on complex system performance optimization

Partners:

April 12, 2023 www.ExigenServices.com

Hints on complex system performance optimization

Ainars Galvans

Exigen Services Latvia

3 CONFIDENTIAL – NOT FOR DISTRIBUTION www.ExigenServices.com

System Performance

Introduction: context, terms and basics


Terminology

Testing

Load

Stress

Scalability

Endurance

…

Performance …

Targets

Thresholds

Objectives

Other

Stability

Workload

Response time

Scenario


Performance is:

Requirement MeasurementNumber of users ThroughputHardware spec. CPU utilization“Response time” Response time

i.e. time user wait for page to load in the browser


A simple 3-tier Web Application

Prepare request

Parse, process

Select or update

Select or update

More Logic

Render HTML

Render UI

Client App. DB

Response tim

e

User E

xperience

Repeated


A simple 3-tier Web Application

Client1

App. DB

Client2


With 3rd party WebServices

Client1App. DB

Client2


Other Apps share the same DB?

App. DB Other (Corp.)App.


Performance is:

Requirement MeasurementNumber of users ThroughputHardware spec. CPU utilization“Response time” Response time


Rules of thumb

Response time– >2 seconds: is it all CPU time?– 0.5-2.0 seconds: why?– <0.5 seconds : OK

Concurrent users– The term is misleading– What matters technically is throughput

Resource utilization:– DB CPU should be below 50%


My issue localization techniques

First step: where (what host) is the bottleneck?– Single user repeat the same action (i.e. with Selenium)– Monitor “CPU time” delta (secs.), not CPU (%)– Use excel to understand how it scales

If it’s the App Server: analyze HTML– FireFox “Save Page As” -> complete page – Look for large, unused data

If it’s DB or Web Services: analyze traffic, logs– Put milliseconds to log files, see what takes most of it– Look for repeated requests for single page rendering

If it’s client (browser): profile JavaScript– IE -> Dev. Tools (F12) -> Profiler– Are all the JavaScripts cached (or compiled each time)?


System Performance

Performance optimization stories


Locating database call issues

Client1

App. DB

Client2


Story #1: Just a few tables

Problem description– Web Page shows a number of tables– Page response time is a bit above 2 seconds– Under load DB becomes overloaded quickly– All DB queries takes <0.1 sec

Investigation results:– Single page load cause about 100 select executions– Each took 0.02 secs– There were two distinct statements

Solution– After adjusting DB caching – Response times reduced to 0.4– Almost no load to DB under load


Locating WebService call issues

Client1App. DB

Client2


Story #2: One call only takes 0.5 secs.

Problem description– Web Page shows user’s data overview– Typical customer page response time 5 seconds– Response time up to 30 seconds for specific customers– Under load response times don’t increase too much

Investigation results:– Data in tables is requested via web services– Separate call made on each business item– Call response time is 0.5 secs

Solution– Protocol adjusted to request all info with one call– Response times reduced to 0.9


Application Logic (HTML processing)

Client1

App. DB

Client2


Story #3: Tables, again: .NET

Problem description:– Page shows main page and a dynamic table with hot links– Response times >5 secs. – App server loaded significantly– Under load response times increase

Investigation– In page source (HTML) I’ve discovered huge callbackstate

Solution– http://www.devexpress.com/Support/Center/Question/Details/Q42

5597– Response times < 3secs. now [Browsable(false)]

public string UnboundLargeData { get; set; }

http://www.devexpress.com/Support/Center/Question/Details/Q425597

http://www.devexpress.com/Support/Center/Question/Details/Q425597


Client Side (rare case though)

Client1

App. DB

Client2


Story #4: Client side

Problem– Visually page loads 8-12 seconds and vary– Looks like response time depend on client workstation

Investigation– Server response time is only 3 seconds– It takes browser 1-2 seconds to send request– It takes browser 3-4 seconds to render response– It takes more if workstation CPU is slow (<3 GHz)

Solution– Optimize JavaScript and remove features from page– Browser choice recommendation– Unfortunately: optimal client CPU recommendation


… everything. The business itself

Client1

App. DB

Client2


Story #5. The reports and queries

Production problem – Special feature “accounting reports” usually work 1-2 mins– Sometimes 10 times as much– Suspected reason a “morning lag” (all users)

Investigation– One report loads DB CPU to 100% for 1-2 minutes – Only technical solution: rewrite whole app, changing DB

structure and logic The actual solution

– Nightly batch jobs on demand– The default nightly batch jobs– Daily execution only with special privileges and only one at a

time


Conclusions

Localizing the problem


My issue localization techniques

First step: where (what host) is the bottleneck?– Single user repeat the same action (i.e. with Selenium)– Monitor “CPU time” delta (secs.), not CPU (%)– Use excel to understand how it scales

If it’s the App Server: analyze HTML– FireFox “Save Page As” -> complete page – Look for large, unused data

If it’s DB or Web Services: analyze traffic, logs– Put milliseconds to log files, see what takes most of it– Look for repeated requests for single page rendering

If it’s client (browser): profile JavaScript– IE -> Dev. Tools (F12) -> Profiler– Are all the JavaScripts cached (or compiled each time)?


Happy to share more details

Skype name: ainarsgalvans

Phone: +371 29432698

Facebook: Ainars Galvans

Email: [email protected]

Blog: find me at SoftwareTestingClub.com

Feedback

Partners:

ainars galvans - hints on complex system performance optimization

Technology

cpu time

load response times

rules of thumb response

browsablefalse response

page loads

client browser

time user

single page rendering