alm113 large sap systems performance references and load testing
TRANSCRIPT
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 1/26
ALM113
Large SAP Systems, Performance
References and Load Testing
William Adams
Performance & Scalability
SAP AG
Nikolai Sauerwald
Performance & Scalability
SAP AG
© 2010 S AP AG. All rights reserved. / Page 2
Disclaimer
This presentation outlines our general product direction and should not be relied on in making a
purchase decision. This presentation is not subject to your license agreement or any otheragreement with SAP. SAP has no obligation to pursue any course of business outlined in this
presentation or to develop or release any functionality mentioned in this presentation. This
presentation and SAP's strategy and possible future developments are subject to change and
may be changed by SAP at any time for any reason without notice. This document is provided
without a warranty of any kind, either express or implied, including but not limited to, the implied
warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP
assumes no responsibility for errors or omissions in this document, except if such damages
were caused by SAP intentionally or grossly negligent.
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 2/26
© 2010 SAP AG. All rights reserved. / Page 3
Agenda
1. Introduction
2. KPIs and Scalability of SAP Software Architecture
Frontend
Backend – Application Layer
Backend – Database Layer
3. Verify Scalability with Load Testing
4. Performance Metrics and Key Volume Drivers
5. Miscellaneous
© 2010 S AP AG. All rights reserved. / Page 4
"Large" systems can be described with very different characteristics
Likely additional users, higher
throughput
Likely legacy/third party software
Higher level of integration More load on interfaces
Less controlled
Possible Evolvement of Large Systems
"home-made" growth
Increasing user numbers, higher
throughput
Increasing number of applications Rather controlled
Same software, perhaps
template approach
Organic Growth Mergers / Acquisitions
Some Characteristics for Large Systems
Need to consolidate
Users connect over WAN
"Central system"
One database One client
All processes are centrally
managed
Application operations
Downtimes, planned or
unplanned
Single Global Instance
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 3/26
© 2010 SAP AG. All rights reserved. / Page 5
Excerpt Gartner Study on ERP TCO aboutSingle Global Instance ERP
"…
1.1.Will you deploy single-instance ERP
One of the main drivers of ERP TCO is the
number of instances that are deployed and how
they are managed for distributed enterprise. …
Deploying a single instance typically reduces
support requirements, increases consistency in
processes and data, and enables an integrated
view of the organization's data. …
A single instance is not always possible or
preferable. The more diverse a business, the
more difficult it will be to identify and achieve the
synergies of a single-instance strategy. …
…"
Source: The Ten 'How' Facto rs Th at Can Affect ERPTCO , Gartner RAS Core Research Note G00172356,
Denise Ganly, 1 February 2010, V1RA9 04082011
© 2010 S AP AG. All rights reserved. / Page 6
Feasibility of Large Systems
Feasibility Custom coding
Can SAP software payroll 3,000,000
employees and pensioners in less than
three hours?―
Can SAP software fulfill our requirements
concerning 30 million business partners?―Can SAP software meet the service level
agreement of < 1 sec server response time
in our corporate portal?―Can SAP software handle database tables
of more than 1.5 TB and what are the
implications?―
The most common frequency of
performance problems is quarterly or
monthly and custom code the number one
culprit.ERP ―Terabyte Club‖ - Critical Operations
from ASUG/AMR, Dec, 2007
―
Research at customers indicates that most
performance pain points (i.e. long response
times, long-running jobs) were caused by
custom coding
[...]
Furthermore, studies reveal that much of
the custom-coded functionality is either
already part of the standard applications or
is hardly ever used Computerwoche, November 2007
―
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 4/26
© 2010 SAP AG. All rights reserved. / Page 7
Excerpt ASUG/AMR Study on Large ERPSystems
"The charts below show the frequency and top 3
root causes of performance problems for Terabyte
Club companies, The most common frequency of
performance problems is quarterly or monthly andcustom code the number one culprit, There are
many excellent performance tools in the SAP
ecosystem which can greatly assist
troubleshooting and proactive performance
tuning."
"For Terabyte Club companies capacity planning
is an essential best practice to support
performance management, not a luxury, Again,
specialist tools are available to help model and
predict performance throughout project roll outand the entire ERP Lifecycle."
Source: ERP “Terabyte Club” - Critical Operations from
ASUG/AMR, December 2007
Performance Problems:The Most Likely Root Causes
0,0
0,1
0,2
0,3
0,40,5
0,6
0,7
0,8
0,9
1: Custom
Code
2: Database
Too Big
2: Batch
Processing
FREQUENT PERFORMANCE PROBLEMS
Custom code
Large database
Background processing
© 2010 S AP AG. All rights reserved. / Page 8
Agenda
1. Introduction
2. KPIs and Scalability of SAP Software Architecture Frontend
Backend – Application Layer
Backend – Database Layer
3. Verify Scalability with Load Testing
4. Performance Metrics and Key Volume Drivers
5. Miscellaneous
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 5/26
© 2010 SAP AG. All rights reserved. / Page 9
Typical Load Distribution on the Different Layers*
Browser
Browser
Application
Application
Web server
Web server
Typical SAP Software Architecture in Layers
Database
Application
Frontend
Database
Application
Database
Application
Browser
Web server
Central System, 2-Tier 3-Tier Multi-Tier
Frontend Frontend Other
Frontend: 10-30%
Middleware: 5-10%
Application: 50%
Database: 10-20%
Frontend: 10-30%
Backend: 70-90%
Frontend: 10-30%
Application: 50-70%
Database: 10-20%
* The percentages can vary significant ly, depending on the complexity of the business logic, the user interface, the programming language or whether huge amounts ofdata are manipulated at DB level (Business Intelligence).
© 2010 SAP AG. All rights reserved. / Page 1 0
Possible performance metrics:
Response time – user experience / productivity
System throughput – business requirement,
system resource consumption, IT budget
…
Good performance:If the performance metrics match the business
requirements or Service Level Agreements
(―expectation handling‖)
Several parts that constitute the end-to-end(E2E) response time:
Server response time
Network time
Frontend time
Performance, Response Time, and Throughput
Application Server
Application Server(s)
Database
Web Server(s)
R e s p o n s e T i m e = Σ
t [ i ]
t[1]
t[2]
t[3]
t[4]
t[5]
t[6]
t[i]
Browser
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 6/26
© 2010 SAP AG. All rights reserved. / Page 11
KPIs for Network Performance in a Wide AreaNetwork (WAN)
Transferred data volume – bandwidth of a network connection
Number of network roundtrips – latency time of a network connection
Performance Design Pattern One network roundtrip per user interaction step
Keep the data volume as low as possible (approx. 10 ~ 20 KB per user interaction step)
Major concepts to optimize WAN performance for browser-based UIs (HTTP)
Browser caching, supported in entire SAP NetWeaver infrastructure
Only for static content like icons, JS, CSS, etc.
Dynamic and security-relevant content doesn't appear in browser cache
HTTP Compression, supported in entire SAP NetWeaver infrastructure
Achieving a factor 4 -10 for data reduction, depending on the content and size
Use SAP NetWeaver Accelorated Application Delivery (AccAD)
Tunneling network connections using highly efficient data compression
Use Windows Terminals Server (WTS)
Only for specific groups of users with extremly bad network qualities
Addtional hardware and software needed
© 2010 SAP AG. All rights reserved. / Page 1 2
Example Connections Single Distance in Kilometers Theoretical RoundtripTime*
Actual Avg. Measured in aProject
Satellite connection 45,000 miles / 75,000 km 500 ms 600 ms
Europe – US East Coast 4000 miles / 6400 km 70 ms 170 ms
US East Coast – US West Coast 2600 miles / 4200 km 42 ms 85 ms
US West Coast – Europe 6600 miles / 10,000 km 100 ms 210 ms
Asia (China) – Europe 5500 miles / 8600 km 85 ms 400 ms
Impact of Latency on E2E Response Times
E2E response time measurements
of user interaction steps, using aWAN simulation
Number of network roundtrips of a
user interaction step drives the
increase of the network time
4 roundtrips: User Logon
2 roundtrips: SO OWL, New SO
1 roundtrip: WoC SO,
Sold2Party, Add Row, Line
Item, Save
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 7/26
© 2010 SAP AG. All rights reserved. / Page 13
Recommendations to Improve WANPerformance
1 Consider employing a browsersetting policy to avoid roundtrips
Cache sizes: ~ 100 MB
HTTP: allow compression by selecting HTTP 1.1
(accept encoding) De-select
Do not save encrypted data to disk
Empty temporary Internet folder
Keep roundtrips and amount oftransferred data at a minimum incustom applications
Use compression techniques
Avoid synchronous round trips in WAN
Typical avg, bandwidth per user interaction step
(browser): 10-20 KB
Consider potential additional load Central services, such as local printing will need to be sent
over the WANIf you upload data from scanners (e.g. barcode) or RFID
WAN traffic
Balance infrastructure costs withperceived performance
Consider a mixed access approach: e.g. BI users may
work through WTS
Consider SAP NetWeaver Accelerated Application Delivery
Tool-based optimization of network traffic in browser-based
applications
-
-
-
-
Since latency cannot be tuned, avoid "chatty" applications
-
© 2010 SAP AG. All rights reserved. / Page 1 4
Agenda
1. Introduction
2. KPIs and Scalability of SAP Software Architecture Frontend
Backend – Application Layer
Backend – Database Layer
3. Verify Scalability with Load Testing
4. Performance Metrics and Key Volume Drivers
5. Miscellaneous
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 8/26
© 2010 SAP AG. All rights reserved. / Page 15
Small test
system Small data
source
One user
Thousands of users Terabytes of data
High throughput Large objects
Globally distributed users
Hardware adaptivity
Many servers
Dimensions of internal and external scalability Effects of scalability
Dimensions and Effects of Scalability
Size of
data
Number ofobjects in
the DB
Number ofprocessed
objects per
time
Concurrent
users
Number
of servers /
cores /
threads
Scalability
Number of
parallel jobs
© 2010 SAP AG. All rights reserved. / Page 1 6 Time
Ensure Linear Scalability in ResourceConsumption
Processing time
Linear behavior can be best checked withvarying object numbers and sizes, and varying
number of concurrent users
Memory consumption
Efficient data design and release no moreneeded memory as early as possible
Using short living transactions / sessions in
ABAP can easily eleminate memory leaks
Extensive garbage collection activity in Java
impacts the performance and scalability
significantly
Number of processed units / conc. users
CPU utilization(%)
increases linearly
to 100%
10,000
T i m
e
100
J a v a h e a p u s a g e
CPU consumption per processed
unit / conc. user remains constant
Response timebehaves according
to queuing theory
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 9/26
© 2010 SAP AG. All rights reserved. / Page 17
Most Important Performance Topics on theApplication Layer
Linear resource consumption CPU time
Linear to the number/size of
processed objects
Memory usage
Unintended long-lived/growth
of internal tables / strings
Pseudo- garbage
Anonymous data objects that
are pseudo-garbage
Communication load Potentially less communication
overhead in single instance
Potential impacts on
performance
CPU consumption on sending
and on receiving system
Granularity: Fixed overhead per
message
Serialization, compression and
authorization
Choose the right granularity of
message size, especially in WAN
Interfaces
Many processes in parallel Higher demand on system
components (CPU, memory, I/O
system, network, …)
Balance workload
Fixed or dynamic distribution
Give preference to higher
number of small packages than
only a few large packages
Avoid deadlocks
Follow sorted sequence
Keep locks as short as possible
choose right granularity
Consider impact on disk I/O
Parallel Processing CPU and Memory Resources
Effects on the Application Layer
© 2010 SAP AG. All rights reserved. / Page 1 8
Parallel Processing: Choose the RightGranularity of the Work Items
Often, customers tend to choose rather large
job sizes For example they do their billing according to
country
If the country is very large, the individual job may
take very long and block central resources
Processing time of single step processing time of
the whole chain
Advantages of configuring small jobs
More even load distribution decrease of lockingtime on central objects
No preprocessing needed to take the largest jobs
first
In case of non-performance-optimized coding,
negative effects are not as negative as in large jobs
Advantages of configuring large jobs
Easier defined and established, for example on
business units
So, what is a useful package size?
Too small message overhead
Too big possible problems with workload
balancing, data transfer, memory requirements,
runtime restriction for DIA work processes
Therefore: test for optimal throughput
Throughput
optimalNumber of parallel processes
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 10/26
© 2010 SAP AG. All rights reserved. / Page 19
Interface Design – Classify the Interfaces You
Need
According to time constraints
Batch-type bulk interfaces
Bulk transfer smaller performance footprint
Advanced scheduling possible, perhaps once perday
Near real-time interfaces
Single transfer larger performance footprint
Asynchronous, immediately after transaction viamessage protocol, for example
– Asynchronous services calls
– Notification messages to one receiver
– Information messages (multiple receivers)
– Asynchronous q/t-RFCs
Synchronous interfaces
Single transfer larger performance footprint
Wait for the answer: for example Available-to-Promise check
Two systems up and running
According to technical aspects
Standard protocols such as Web Services
via SOAP potentially mediated via PI
Easy and standardized integration
Large flexibility
larger performance footprint
SAP-proprietary protocols such as RFC or
simple non standardized file interfaces
Larger initial integration efforts
Less flexible
smaller performance footprint
In the end, it's a price performancequestion
© 2010 SAP AG. All rights reserved. / Page 2 0
Recommendations for the Application Layer
1 Many scalability issues on the
application layer can be solvedby adding hardware capacity
Number of CPUs/cores/threads
Many slower ones versus few fast ones mayimprove parallel processing
A faster CPU yields better response times in dialog
mode
Memory Make sure there are no memory leaks in your custom-
coded application
Linear CPU time Analyze different dimensions of scalability (number,
size), use more than two measurement points
Response time measurements do not prove scalability
Programs designed to run inparallel
Release locks as soon as you can
Analyze the program for bottlenecks and remove them
as soon as you can
Consider locking time, granularity of the ENQ locks and
importance of the object for the system
Are there work items competing for the same locking
objects?
Interface If you need to design interfaces, carefully weigh the pros
and cons
-
-
Beware: Adding hardware does not necessarily solve performance issues!
-
-
-
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 11/26
© 2010 SAP AG. All rights reserved. / Page 21
Agenda
1. Introduction
2. KPIs and Scalability of SAP Software Architecture Frontend
Backend – Application Layer
Backend – Database Layer
3. Verify Scalability with Load Testing
4. Performance Metrics and Key Volume Drivers
5. Miscellaneous
© 2010 SAP AG. All rights reserved. / Page 2 2
Possible Effects of Large Databases (1/2)
System availability Use of resources System performance
Downtime Lossesby Application
Typical HourlyLoss of
UnplannedDowntime (US$)
Financial/Trading $2,400,000
Supply Chain $600,000
ERP $600,000
CRM $480,000
E-Commerce $480,000
E-Business $480,000
Busine ss Applica tion $300,000
Database $300,000
Messaging $60,000
Infrastructure $42,000
Data Records10 GB
Mirrors20 GB
A r c h i v i n g
2 GB(Compre ssion 20%)
System Copies80 GB
0
500
1000
1500
2000
2500
Month 1 Month 2 Month 3 Month 4
MB or msec
MSEG MB MB51 ms ec
Upgrade to higher software
releases, Runtime for backup and
recovery
Hardware costs for disk, CPU,
memory as well as administration
costs
Response times in dialog mode
for all employees,
Optimal coding ensures access
times independent of DB size
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 12/26
© 2010 SAP AG. All rights reserved. / Page 23
Background jobs usually have a fairly high DB load
If there is no night (i.e., in single global instances),batch jobs may compete with (and win) user
requests increase in end-to-end response time
Danger of time-outs because of runtime limitation
caused by system parameter settings (i,e, adjust
max_wprun_time)
Possible Effects of Large Databases (2/2)
Disk I/O Parallel dialog and background job usage
Size of single spindle vs. access speed, one access
at anyone time CPU processing capabilities much faster than
read/write operations
Database cache must grow linearly with DB size
often overlooked
Size of file system cache
0
10
20
30
40
50
60
70
80
90100
0 2 4 6 8 1 0 1 2 1 3 5 7 9 1 1
User load
Batch load
Example: CPU load profile in parallel user/batch mode
© 2010 SAP AG. All rights reserved. / Page 2 4
Most Frequent Mistake
Poor DB performance due to missing or sub-optimal indexes
Ensure proper index design
All frequently executed database accesses (selects, updates and deletes) must be supported by
indexes
A highly modified table, i.e., a table with many INSERT or UPDATE or DELETE operations,
should not have too many indexes, The indexes must be maintained by the DBMS, too
The more columns an index has, the higher the chance that an UPDATE will affect one of the
indexed columns
More than 10 msec in the column Minimum Time / Record in Performance Trace (ST05) suggest
improper indexing use explain function
Good index design requires coordination and a potential compromise among alldevelopers that need to access the data
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 13/26
© 2010 SAP AG. All rights reserved. / Page 25
Recommendations for Database Performance
1 Ensure proper index design All frequently executed database accesses (selects,
updates and deletes) must be supported by indexes
Make sure you do not keep datain the database any longer thanneeded
Combination of data archiving, retention management
(Bear in mind legal implications of system shut down )
Parallel Processing High load requirements demand careful implementation
of DB accesses
Efficient database access (result set, search area, use
of caches and proper indexes)
Check the I/O rate Use transaction ST04 to check the I /O rate
DB file sequential read for read operation
Logfile synch for synchronous write logging most
important operation, If slow (exceeds 15 msec), overallperformance affected
Another indicator is if you observe an increased runtime
over several days
With high throughput parallelprocessing spend extra time onI/O design
Disk I/O (sufficient number of spindles) virtual I /O in
high-load systems not yet best practice
-
-
-
-
-
© 2010 SAP AG. All rights reserved. / Page 2 6
Agenda
1. Introduction
2. KPIs and Scalability of SAP Software Architecture
Frontend
Backend – Application Layer
Backend – Database Layer
3. Verify Scalability with Load Testing
4. Performance Metrics and Key Volume Drivers
5. Miscellaneous
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 14/26
© 2010 SAP AG. All rights reserved. / Page 27
Some Key Performance Indicators
CPU Processing times of business transactions or tasks
Cost factor: Number and processing power of servers
Disk size
Disk I/O
Data that resides on the database
File read and write activity to storage
Cost factors Backup/recovery depends on size of database
Storage capacity
Memory
Allocated to a user or background process
Garbage collection, acceleration, planningcapabilities, buffers, caches
Cost factor: Physical memory slots
Front-endNetwork
Load
Transferred amount of data
Network time and roundtrips
Cost factor: Leasing bandwidth
© 2010 SAP AG. All rights reserved. / Page 2 8
Ensuring Scalability With Performance Tests –
Approaches
Single User Test
Small test system
(QA, development), one user
Volume Test
Equivalent to multi-user test, stresstest, load test, benchmark
Quality and implications ofaccesses to persistence layer
Linear resource consumption
Parallel processingmechanisms, load balancing
Memory usage and memory
leaks
Analyze & measure
scalable behavior
Performance predictions
for high volume environment
Verify assumptions
and ...
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 15/26
© 2010 SAP AG. All rights reserved. / Page 29
Scalability Testing Test Cases
The test cases are:
Representative
Reproducible
Right to the point
Representative in the sizing environment includes
Bread-and-butter transactions because they cause the greatest load, such as sales orderentry
Critical business cases that must run in a tight time frame, such as settlement runs at the endof a period, nightly MRP runs
Reproducible test cases
Conducting the test several times must not change the load behavior
Can be rapidly understood and followed through
Remain the same over the years/releases
Right to the point
Pragmatic, we don't want to squeeze out the last 10%
© 2010 SAP AG. All rights reserved. / Page 3 0
Testing for Linear Resource Consumption
To test whether the solution is scalable, perform ―linearity tests―.
Scalability tests are basically a repeatable pattern for (example):
Scalability with an increasing number of objects
1 object, five line items
10 objects, five line items
100 objects, five line items
Scalability with an increasing of line items
1 object, 1 line item
1 object, 10 line items
1 object, 100 line items
The results of these controlled tests can be analyzed to determine if there are anyperformance bottlenecks
Again by understanding the resource consumption of a system that was writtento scale
Tests like these enable more detailed questionnaires, where you do not rely onparticular test scenarios but can ask for variations
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 16/26
© 2010 SAP AG. All rights reserved. / Page 31
Scalability Testing EvaluatingMeasurement Results
Application
DB
Total
0.0000
0.0200
0.0400
0.0600
0.0800
0.1000
0.1200
0.1400
0.1600
0.1800
0.2000
0 10 20 30 40 50 60 70
No. of bidders
C P U
t i m e
Make sure you define at least three measurement points per scenario – the more, thebetter
Example: Scalability with the number of bidders – three tests
Another set of tests would include, for example, 10 bids with 5, 50, and 150 line items, respectively
© 2010 SAP AG. All rights reserved. / Page 3 2
Scalability Testing Steps
Provide CPU processing time, Memory consumed, Disk Access, and Network timefor all tests described using the following:
For AS ABAP and AS JAVA:
Disk Analysis DB02
CPU Analysis ST03N, STAD
User Analysis ST07, STAD
Memory Analysis SM04, STAD
Front-End Network Load STAD, ST03N
CPU and memory status ST06
For AS JAVA:
NWA: Open SQL Monitors
Wily Introscope
SAP JVM Profiler
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 17/26
© 2010 SAP AG. All rights reserved. / Page 33
Scalability Testing System Characteristics
Aspect Comment
Generalperformanceand stability
Transports to the test system are not allowed as they may change Coding, DDIC objects and customizing settings
Try to be alone If this cannot be helped, make sure the additional users do not change coding, DDIC objects and
customizing data or settings
Optimally, no other processes, systems or DBs should be active on the server If there are, they should not incur extra load
Prefer central system: Appl. Server and DB on one machine A system with one application server only may do as well For batch process testing a central server is best
CPU Measurements are extremely platform-dependent Impaired by technologies that simulate several logical CPUs
(hyperthreading, hardware-multithreading, …)
For reliable measurements: turn it off If you can't, you must make sure the load on the system is kept at a minimum, below 10%
in peak
Allocate at least two CPUs Three or four would be even better The less CPUs you have available, the more diff icult it is, because of potential "interfering" users,
batch jobs, etc...
Memory Memory must not become a bottleneck SAP memory (Extended Memory, Roll, Paging) Physical memory (avoid OS paging by all means)
© 2010 SAP AG. All rights reserved. / Page 3 4
Scalability Testing Steps (1/5)
1. We assume that your process fulfills one of the criteria given previously for
performing load tests
2. Determine the expected throughput figures and/or user and system response
times
For example:
Upload of 500,000 sales order line items in two hours
23 million emails in eight hours
Processing of 100,000 workflow items per hour
Processing of nn HTTP requests per sec from mm parallel users
Usually these figures are part of a requirement or
specification document
3. Determine the complete set of functionalities needed for the load test and the
resulting software components, for example:
ATP (Available-to-Promise) Check in APO system,
Pricing in IPC,
Workflow or interfaces to other systems
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 18/26
© 2010 SAP AG. All rights reserved. / Page 35
4. Establish test scenarios for the processes
Specify in detail and document well
Process steps Input/output for the steps
Customizing environment
5. Determine the data distribution – i.e., the type of master data you need
Avoid artificial bottlenecks; for example, all orders use the same customer and products
It is not enough to create a vast amount of data by simply copying it
6. Perform the initial capacity planning for the load test system
Based on measurements in the test or development system of the same or very similartest scenario
Scalability Testing Steps (2/5)
© 2010 SAP AG. All rights reserved. / Page 3 6
7. Define procedure for repeatable test executions
Starting conditions must allow reproducible measurements
It might be necessary to ensure this, using a backup and restore strategy; in other
cases, a clean-up program may be sufficient
Even if you test in a small system, you ought to have regular back-ups in order to start a
test from scratch again or to repeat a test phase
8. Create test system and system infrastructure
Ensure that the system landscape is well-configured
Prepare test environment
In some cases it makes sense to start testing with a small system that you gradually
extend
9. Determine the methods of load and user simulation
Decide which tools are used for generating load
Some tools require expert knowledge
Scalability Testing Steps (3/5)
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 19/26
© 2010 SAP AG. All rights reserved. / Page 37
10. Create the actual test data environment
For example: Create the business partners which are used in sales order processing
11. Run the single-user test
Measure and analyze the performance
Eliminate performance bugs
Check scalability of application
Linear behavior
Critical path lengths and locking
Proper indexes
12. Run the multi-user load test Start with a small load and increase load step-by-step towards the final throughput
target
Measure the throughput
Measure the resource consumption using appropriate monitoring tools
Optimize system parameter settings and repeat the test executions if necessary
Scalability Testing Steps (4/5)
© 2010 SAP AG. All rights reserved. / Page 3 8
13. Evaluate the test results
Document the achieved throughput numbers and the corresponding resource
consumption
Document the test system environment (hardware and software releases) carefully
The numbers may be needed for sizing of further systems
Scalability Testing Steps (5/5)
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 20/26
© 2010 SAP AG. All rights reserved. / Page 39
Agenda
1. Introduction
2. KPIs and Scalability of SAP Software Architecture
Frontend
Backend – Application Layer
Backend – Database Layer
3. Verify Scalability with Load Testing
4. Performance Metrics and Key Volume Drivers
5. Miscellaneous
© 2010 SAP AG. All rights reserved. / Page 4 0
What Are Typical Metrics to Define LargeSystems?
Number of CPUs/cores (ondatabase on app servers)
Number of dialog steps* Size of Database & databasegrowth
Number of servers Number of concurrent users Data volume per time period(e.g. n messages / second)
Total number of systems(incl, legacy)
Traffic on the interfaces Size of memory
Actual total CPU time pertime period
... ...
* User interaction steps frontend-backend where data are transferred
On the one hand very easy and on the other hand not really meaningful
For example: what are users?
How many dialog steps would a background-driven system have?
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 21/26
© 2010 SAP AG. All rights reserved. / Page 41
Remote Office
Remote Office
Remote OfficeRemote Office
Remote Office
Data Cente r
Disclaimer 2
The data on large customers are examples
SAP does not systematically collect performance references
Exception Early Watch Alert: customers send extracted technical data to SAP (voluntary)
Other examples are first-hand experience no general statements can be derived
Out of date as of today
Out of scope
TCO discussion
Business value for moving single instance
Price/performance
Architecture/landscaping/
configuration/virtualization
© 2010 SAP AG. All rights reserved. / Page 4 2
A Little Analysis: Relationships BetweenActive Users And Various KPIs
Source: Early Watch Alert
Top 50 systems accordingto the number of
interaction/dialog steps
per week
Users high
Users
mediuUs
ers
low Users DB Size in GB DB Year Dialog steps
(week)
CPU Time
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 22/26
© 2010 SAP AG. All rights reserved. / Page 43
Examples for "Large" or "Challenging"Systems
Performance is driven by the application demands, not necessarily by technology
Customer A: Consumer Industry
Processing of customer orders
24 million items per day, 1.2 million sales orders
Assume an evenly distributed work day of 10 hours
2.4 million items per hour
Doable, confirm with sap.com/benchmark SD 2-
tier
Locks on a central DB table
At least one identical product is contained in each
order
120,000 updates (i.e. locks) on the same table per
hour
33 requests per second that cannot be parallelized
The Catch
Customer B: Consumer Industry
Keeping historical information Sales history from Point-of-Sale kept for 2 years
100,000+ products in 5000+ stores
Quite a large database table One database table with 20 billion entries
The table had ~ 7 columns and the size was ~7 TB
The Catch
© 2010 SAP AG. All rights reserved. / Page 4 4
Examples for "Large" or "Challenging"Systems
Performance is driven by the application demands, not necessarily by technology
Customer C: Automotive Industry
Processing of sales order items
30 legacy systems overall, SAP is currently only
one of many
SAP load profile: 75,000 order line items per day
Time constraints
Processing needed to be done in 15 minutes!
75,000/(15*60) = 80+ order items per second
Including Available-to-Promise check
The Catch
Customer D: Consumer industry
100,000 users in corporate portal
More than 40,000 logins anticipated per peak hour
1 sec, server response time for log-in
Keep the home page very lean
20 physical servers, 40 J2EE server processes, 2
J2EE server processes per node (~40,000 SAPS)
The Catch
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 23/26
© 2010 SAP AG. All rights reserved. / Page 45
Examples for "Large" or "Challenging"Systems
Performance is driven by the application demands, not necessarily by technology
Customer E: Consumer Industry
Managing large databases
30 TB in ERP system
One financials table with 1.2 TB
Another financials table with 2 billion entries
Handling system downtimes
"Database growth cannot be stopped, it can only
be controlled" archive 24 TB per year
Archived data are already compressed, projected
growth would have been 100 TB
The Catch
Customer F: Service Industry
Thousands of downloads from extranet One central global extranet for users to download
software packages
RTT up to 20sec, plus download time for package
Packages Sizes ranged from KB to GB Split Extranet into regions, have all packages
available there
Use third party vendor to create network of reverse
proxies and cache data near where it is being
downloaded
The Catch
© 2010 SAP AG. All rights reserved. / Page 4 6
Business, Architecture & Technology
Performance feasibility evaluations must beginn with the business process
Technology is secondary, as it is a moving target
Business Process Evaluation
Are the key business processes ready for high-volume/multi-user?
How close to the original template idea can you stick?1
Software Architecture Evaluation
Can you avoid bottlenecks in the standard design? Do you consider performance/sizing already in the design phase
of your custom code?2
Technology Evaluation
Is the hardware capable enough?
Is the hardware within economical limits?3
3
2
1
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 24/26
© 2010 SAP AG. All rights reserved. / Page 47
Agenda
1. Introduction
2. KPIs and Scalability of SAP Software Architecture
Frontend
Backend – Application Layer
Backend – Database Layer
3. Verify Scalability with Load Testing
4. Performance Metrics and Key Volume Drivers
5. Miscellaneous
© 2010 SAP AG. All rights reserved. / Page 4 8
Ensure Scalability: Improve Performance
Add hardware
Additional application servers
(consider impact on DB server)
Additional CPUs/memory on server(s)
Tune the system
Profile parameters
Increase buffer sizes
Re-schedule load
Distribute load more evenly acrossservers and time
Tune the application
Customizing, configuration
Function and performance
Optimize the coding
Requires time, knowledge, …
0
1
2
3
4
5
Expertise
Veryfast
Testingeffort
Sustainability
Farreach
Highrisk
Add hardwareTune the systemRe-schedule loadOptimize the codingTune the application
Possible Measures Consequences
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 25/26
© 2010 SAP AG. All rights reserved. / Page 49
Concluding Remarks
Performance is hardly ever described by technical terms alone
Demands from the application design must be considered
Challenge to turn business requirements into technical requirements
Application/implementation team and IT-team must work together
"Largeness" is defined by very different characteristics
make sure to include them in your considerations
In principle, SAP software scales very well
Proven by SAP Standard Application Benchmarks (www.sap.com/benchmark)
Proven by customer load tests
There are a number of potential bottlenecks
Network environment, especially latency limitations
Linear scalability on the application server, memory leaks, batch processing
Database access design and implementation
Disk I/O, especially in mass-processing environments
© 2010 SAP AG. All rights reserved. / Page 5 0
Further Information
SAP Service Marketplace
http://service.sap.com/performance Information about load and volume tests: procedure, examples
Recommendations for system tuning (http://service.sap.com/performance --> Media
Library)
Guidelines for efficient programming (http://service.sap.com/performance --> MediaLibrary)
http://service.sap.com/sizing
http://service.sap.com/quicksizing
8/9/2019 ALM113 Large SAP Systems Performance References and Load Testing
http://slidepdf.com/reader/full/alm113-large-sap-systems-performance-references-and-load-testing 26/26
Contact
Feedback
Please complete your session evaluation.
Be courteous — deposit your trash,
and do not take the handouts for the following session.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG.The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.
IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries,eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+,POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex,MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or othercountries.
Oracle is a registered trademark of Oracle Corporation.
UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.
Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.
HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology.
Java is a registered trademark of Sun Microsystems, Inc.
JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.
SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer and other SAP products and services mentioned herein as well as their respectivelogos are trademarks or registered trademarks of SAP AG in Germany and other countries.
Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcels ius, and other Business Objects products andservices ment ioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. in the United States and in othercountries.
All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only.National product specifications may vary.
The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without theexpress prior written permission of SAP AG.
This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies,developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/ordevelopment. Please note that this document is subject to change and may be changed by SAP at any time without notice.
SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or otheritems contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of
© 2010 SAP AG. All Rights Reserved