sap 2-tier sales and distribution tunings for linux on power7

Blueprints

SAP 2-tier Sales and Distribution Tunings for Linux onPOWER7

��

NoteBefore using this information and the product it supports, read the information in “Notices” onpage 43.

Second Edition (December 2011)

© Copyright IBM Corporation 2010, 2011.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Chapter 1. Scope, requirements, andsupport . . . . . . . . . . . . . . . 1

Chapter 2. SAP software architectureoverview . . . . . . . . . . . . . . 3

Chapter 3. System configuration . . . . 5

Chapter 4. System tuning . . . . . . . 7Using the CFQ I/O scheduler . . . . . . . . 7Adjusting CFS parameters . . . . . . . . . . 7Disabling barriers in the ext3 file system . . . . . 8Using SMT4 mode . . . . . . . . . . . . 8Data Stream Control Register . . . . . . . . . 9

Chapter 5. SAP application settings . . 11Setting communication between DB2 and SAP. . . 11Setting DB2 to use Linux hugepage . . . . . . 11Setting the number of SAP instances . . . . . . 12Setting SAP to use Linux hugepage . . . . . . 13NUMA overview . . . . . . . . . . . . 13

Binding resources for SAP processes . . . . . 14Binding resources for DB2 processes . . . . . 15

Chapter 6. SAP 2-tier Sales andDistribution Tunings for Linux onPOWER7 . . . . . . . . . . . . . . 17Scope, requirements, and support . . . . . . . 17SAP software architecture overview . . . . . . 18System configuration . . . . . . . . . . . 20System tuning . . . . . . . . . . . . . 21

Using the CFQ I/O scheduler . . . . . . . 22

Adjusting CFS parameters . . . . . . . . 22Disabling barriers in the ext3 file system . . . 23Using SMT4 mode . . . . . . . . . . . 23Data Stream Control Register . . . . . . . 24

SAP application settings . . . . . . . . . . 25Setting communication between DB2 and SAP. . 25Setting DB2 to use Linux hugepage . . . . . 26Setting the number of SAP instances . . . . . 26Setting SAP to use Linux hugepage . . . . . 27NUMA overview . . . . . . . . . . . 28

Binding resources for SAP processes . . . . 28Binding resources for DB2 processes . . . . 29

Appendix A. SAP Sales andDistribution (SD) benchmark overview . 31

Appendix B. SD benchmark scenario 33

Appendix C. SD benchmarkperformance metrics. . . . . . . . . 35

Appendix D. SAP binding script for usewith NUMA nodes . . . . . . . . . . 37

Appendix E. Socket scaling . . . . . . 39

Appendix F. Related information. . . . 41

Notices . . . . . . . . . . . . . . 43Trademarks . . . . . . . . . . . . . . 44

© Copyright IBM Corp. 2010, 2011 iii

iv Blueprints: SAP 2-tier Sales and Distribution Tunings for Linux on POWER7

Chapter 1. Scope, requirements, and support

This blueprint applies to PowerLinux™. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Systems to which this information applies

PowerLinux

Intended audience

This blueprint is intended for system administrators and those who have an interest in understandinghow to leverage the performance of IBM® POWER7® systems running SLES 11 SP1 for SAP Sales andDistribution workload. It is assumed that the audience of this document has a general understanding ofLinux system administration, fundamental operating system concepts, and background knowledge ofSAP system architecture.

Scope and purpose

This blueprint provides the SLES 11 SP1 tuning recommendations for the SAP 2-tier SD applicationrunning on POWER7 processor-based systems. The information given here was verified using the SAP SDstandard application benchmark. The detailed information about how to set up and tune the SAPbenchmark is not the goal of this blueprint.

Test environment

The information provided is gathered from engineering runs using the SAP enhancement package 4 forSAP ERP 6.0 and IBM DB2® version 9.7 running on SUSE Linux Enterprise Server 11 SP1.

Author namesChakarat Skawratananond

Other contributorsDamon BullHeather CrognaleRichard HendricksonJennifer HopperMonza Lui

IBM Services

Linux offers flexibility, options, and competitive total cost of ownership with a world class enterpriseoperating system. Community innovation integrates leading-edge technologies and best practices intoLinux.

IBM is a leader in the Linux community with over 600 developers in the IBM Linux Technology Centerworking on over 100 open source projects in the community. IBM supports Linux on all IBM servers,storage, and middleware, offering the broadest flexibility to match your business needs.

For more information about IBM and Linux, go to ibm.com/linux (https://www.ibm.com/linux)

© Copyright IBM Corp. 2010, 2011 1

http://www.ibm.com/linux

IBM Support

Questions and comments regarding this documentation can be posted on the developerWorks® SystemsManagement Blueprint Community Forum:

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1484

The IBM developerWorks discussion forums let you ask questions, share knowledge, ideas, and opinionsabout technologies and programming techniques with other developerWorks users. Use the forumcontent at your own risk. While IBM attempts to provide a timely response to all postings, the use of thisdeveloperWorks forum does not guarantee a response to every question that is posted, nor do wevalidate the answers or the code that are offered.

2 Blueprints: SAP 2-tier Sales and Distribution Tunings for Linux on POWER7


Chapter 2. SAP software architecture overview

An instance of the SAP application server is a group of operating system processes that offers one ormore services. An SAP instance consists of a dispatcher, one or more SAP work processes, and a commonset of SAP buffers in the shared memory. Services provided by these work processes include dialog,update, SAP enqueue management, background processing, printing, and SAP gateway.

The dispatcher acts as a coordinator, distributing work to other available work processes. Dialog workprocesses are, for example, used to process requests from users working online. Update work processesare responsible for updating the data in the database.

The following figure shows a sample SAP environment consisting of 33 SAP instances.

The SAP ERP application suite is based on multi-tier client and server architecture consists of threelayers: database, application, and users. The Application layer represents the SAP application serverfunctionality. In SAP environments, there are two types of multi-tier configurations:v 2-Tier: SAP application and database server layers reside on a single systemv 3-Tier: SAP application and database layers reside on separate systems

In this blueprint, the focus is on the 2-Tier configuration, which is shown in the following figure withfour SAP instances running.

Figure 1. Sample SAP environment consisting of 33 SAP instances


Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Figure 2. 2-tier SAP configuration


Chapter 3. System configuration

To better understand the tuning steps that are presented in this blueprint, you can learn more about thetest environment.

As mentioned earlier, the SAP 2-tier SD standard benchmark was used to verify the tuningrecommendations. The hardware and software setups described are therefore geared toward achieving thebest performance for the benchmark. For more information about the SAP SD benchmark, seeAppendix A, “SAP Sales and Distribution (SD) benchmark overview,” on page 31.

Hardware configuration

The following list includes the hardware components used in the test environment:v System Under Test (SUT) – the server where application and database layers reside

IBM Power 750 Express– 32 POWER7 cores running at 3.55 GHz– 256 GB (32x8 GB) DDR3 1066 Mhz RAM– 8x146.8 GB internal SAS disks– 1x4 GBS Dual Port FibreChannel PCI-E HBA

v Disk enclosure – external disk storage for the SUTIBM TotalStorage DS4500– Six EXP-710 enclosures, with 14x18 GB, 10 KB RPM drives, for a total of 84 disks– Five volumes were created from the 84 disks:

- 1 RAID0 volume stripped across all eight disks for the database log- 1 RAID0 volume stripped across all 29 disks for the main DB2 database- 1 RAID0 volume stripped across all 10 disks for the multiplexed VBDATA tables- 1 RAID5 volume stripped across all 14 disks for database backup files- 1 RAID5 volume stripped across all six disks for the SAP binaries, DB2 home, and others

v Driver Machine – the client machine driving the workload through the serverIBM System p5® 550 Express– Four POWER5+ cores, running at 2.1 Ghz– 16 GB RAM

The following figure shows how the hardware components are connected to each other.


The driver machine and the SUT communicate through a private network. SUT and the Disk enclosureare directly attached through a FibreChannel.

Software components

The following list includes the software packages that were required to set up the test environment.v SUSE Linux Enterprise Server 11 SP1v SAP Applications and required runtime library

– SAP enhancement package 4 for SAP ERP 6.0– XL Compiler runtime environment V9

v IBM DB2 v9.7 for Linux on POWER®


Figure 3. Connection of hardware components


Chapter 4. System tuning

You can use various options, such as the SMT4 mode, barriers in the ext3 file system, schedulingparameters, and the CFQ I/O scheduler to fine-tune your system.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Using the CFQ I/O schedulerThe Completely Fair Queueing (CFQ) I/O scheduler controls the way the Linux kernel commits readsand writes to disks and works to optimize disk access times.

Without an I/O scheduler, the kernel would issue each request in the order that it is received, resulting inthrashing: if one process reads from one part of the disk, and one writes to another, the heads would haveto seek back and forth across the disk for every operation. You can use the CFQ scheduler to control howthe kernel reads and writes to disks to avoid thrashing.

Note: Using other schedulers with the SAP SD workload in the test environment did not result in aperformance gain.

Ensure that CFQ is the I/O scheduler enabled on your system with the following command:# cat /sys/block/<device>/queue/scheduler# noop anticipatory deadline [cfq]

where device is the name of the disk on which you want CFQ to run.

The brackets ([]) indicate the scheduler that is currently in effect.

Or, you can change the assigned scheduler with the following command:# echo cfq > /sys/block/<device>/queue/scheduler

where device is the name of the disk on which you want to change the assigned scheduler.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Adjusting CFS parametersThe Linux Completely Fair scheduler (CFS) includes parameters that you can adjust to fit your specificneeds.

The following parameters can be accessed through the proc file system. The default values (innanoseconds) of these parameters from SLES 11 SP1 are shown in parentheses.v sched_min_granularity_ns (16000000): Minimum preemption granularity for processor-bound tasks.

Tasks are guaranteed to run for this minimum time before they are preempted.v sched_latency_ns (80000000): Period over which CFQ tries to fairly schedule the tasks on the runqueue.

All of the tasks on the runqueue are guaranteed to be scheduled once within this period.


v sched_wakeup_granularity_ns (20000000): Ability of tasks being woken to preempt the current task.The smaller the value, the easier it is for the task to force the preemption.

To verify the value of each parameter, use the following command:# cat /proc/sys/kernel/sched_latency_ns

To modify the value of each parameter, use the following command:# echo 20000 > /proc/sys/kernel/sched_latency_ns

In the test environment, best results were achieved with the following values for these three parameters:kernel.sched_min_granularity_ns = 100000kernel.sched_wakeup_granularity_ns = 25000kernel.sched_latency_ns = 1000000

If you want to make these changes permanent, you can modify these parameters in the /etc/sysctl.conffile and run the sysctl -p command.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Disabling barriers in the ext3 file systemYou can turn barriers off to improve performance.

About this task

Journaling file systems, like ext3, ensure the integrity of their journal through the use of barriers, whichprevent the writing of any blocks after the barrier until all blocks written before the barrier arecommitted to the media. Note that while turning barriers off increases performance, it also decreases faulttolerance.

Procedure1. Modify the /etc/fstab file by adding the barrier=0 file system option to your mount point, as

shown:/dev/sdc /SAP ext3 acl,user_xattr,barrier=0 1 1

Note: Take care to not introduce any error when you edit this file, or the system might not be able toboot.

2. Reboot the system.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Using SMT4 modeSimultaneous Multi-Threading (SMT) allows the concurrent execution of multiple instruction streams onthe same processor core. POWER7 offers three types of SMT: 1-way, 2-way and 4-way. With 4-way SMT,you can increase the number of instruction streams your system can run on the same processor core.

In 2-way SMT (SMT2), the number of the logical processors that the operating system sees is double thenumber of the physical processor cores in the system. That number of logical processors becomesquadrupled with the 4-way SMT (SMT4). Consequently, the overall system capacity increases as the


number of instruction streams increases. This is important for workloads like SAP SD where multipleusers are sharing a set of application server processes. Capacity increase from SMT4 means more userscan simultaneously get their work through the system by using POWER7.

On SLES 11 SP1, the system operates in SMT4 mode by default. If the SMT mode has been switched, youcan switch it back with the ppc64_cpu command as shown, where X is replaced with a numbercorresponding to the SMT mode (2 or 4):# ppc64_cpu –smt=X # Set SMT state to X

In a typical SAP environment, capacity in terms of number of users is often the limiting factor. With theincrease in the overall system capacity from SMT4 mode, more SAP SD users can be supported with agiven server configuration.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Data Stream Control RegisterData Stream Control Register (DSCR) is a 64-bit register on POWER7 processors that controls the degreeof aggressiveness of prefetching data between RAM and caches. For example, you can enable or disablehardware detection and initiation of load or store streams.

You can also set a prefetch depth for hardware-detected streams or software-defined streams. Thefollowing table describes all of the possible settings for DSCR.

Table 1. DSCR settings

Bits Name Description

58 LSD Load Stream Disable

Disables hardware detection andinitiation of load streams.

59 SNSE Stride-N Stream Enable

Enables the hardware detection andinitiation of load and store streamsthat have a stride greater than asingle cache block. Such load streamsare detected only when LSD is alsozero, and store streams are detectedonly when SSE is also one.

60 SSE Enables the hardware detection andinitiation of store streams.

61:63 DPFD Default Prefetch Depth

Supplies a prefetch depth for streams.The values of this field are thefollowing:

0 default1 none2 shallowest3 shallow4 medium5 deep6 deeper7 deepest

Chapter 4. System tuning 9

On Linux, you can manage the content of this register by echoing a (hexadecimal) number into the/sys/devices/system/cpu/cpuX/dscr directory for each cpuX in the system. For an SAP Sales andDistribution workload, set DSCR to 0x8. That is, Bit-60 (SSE) is set to 1 to enable hardware detection andinitiation of store streams, and Bits 61 to 63 (DPFD) are set to 000 to set the prefetch depth to the defaultvalue.

You can use the following scripts to set DSCR to 0x8 for all 128 processors in your system:!/bin/bash

i="0"

while [ $i -lt 128 ]doecho 8 > /sys/devices/system/cpu/cpu$i/dscrcat /sys/devices/system/cpu/cpu$i/dscri=$[$i+1]done

For more information about DSCR, go to page 674 at https://www.power.org/resources/downloads/PowerISA_V2.06B_V2_PUBLIC.pdf.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.


https://www.power.org/resources/downloads/PowerISA_V2.06B_V2_PUBLIC.pdf


Chapter 5. SAP application settings

SAP application settings that you can use to increase overall performance include communicationbetween DB2 and SAP, hugepages, SAP instances, and resource binding.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Setting communication between DB2 and SAPTo enable the communication between a SAP application server and the database management system, adatabase client must be installed on every application server. Starting with SAP NetWeaver 7.0 SR3, anew DB2 client setup was introduced. In this new setup, the DB2 Call Level Interface (CLI) driver is usedinstead of the DB2 Runtime Client.

The DB2 CLI Driver, also referred to as Thin Client, provides runtime support for applications that use theOpen Database Connectivity (ODBC) or Call Level Interface application programming interface. It is notpart of any of the other clients and must be installed separately.

The DB2 Runtime Client provides support for handling database connections and the underlying networkcommunication, however, it contains no administration or configuration tools.

In the test environment setup, the CLI driver is the default client type that the SAP application serveruses to communicate with DB2. However, with the CLI driver that comes with DB2 V9.7 for Linux onPOWER, the SAP application server performs poorly due to the low level semget calls issued by DB2.Instead, use the DB2 Runtime Client in your environment.

You can direct the SAP application server to use the DB2 Runtime Client by setting theDB2DB6_FORCE_RUNTIME_CLIENT environment variable to 1, as shown, if you are running the csh shell:# setenv DB2DB6_FORCE_RUNTIME_CLIENT 1

Alternatively, you can edit the .dbenv_<hostname>.csh file in the home directory of the SAP administratoruser (for example, <sapsid>adm) to have the environment variable set automatically each time the userlogs in, as shown:# set Environment Variable DB2DB6_FORCE_RUNTIME_CLIENT to force the usage# of the DB2 Runtime Client#-----------------------------------------------------------------------setenv DB2DB6_FORCE_RUNTIME_CLIENT 1#-----------------------------------------------------------------------


Setting DB2 to use Linux hugepageTo reduce the number of TLB misses, use larger page sizes.


About this task

To translate a virtual page address to its corresponding physical address, the operating system performs alook up in the Page Table. To avoid this issue, modern architectures provide a faster hardware look-upcache called Translation Lookaside Buffer (TLB). When a valid address translation is not present in TLB,this is called TLB misses. By using larger page sizes, the number of TLB misses can be reduced.

Most modern architectures provide multiple page sizes. The ppc64 architecture, for example, supports 4KB, 64 KB, and 16 MB page sizes. In Linux, large page is supported through a virtual file system calledhugetlbfs. This support is also referred to as the hugepage support. An application can make use of largepages in two ways:v by using the mmap system callv by using the standard Unix System V (SysV) shared memory system calls (shmget, shmat)

The SLES 11 SP1 (on ppc64) kernel comes with hugepage support enabled. The default base page size is64 KB, and the hugepage size is 16 MB. To configure DB2 to leverage hugepages, complete the followingsteps.

Procedure1. Login as the DB2 administrator, and stop DB2:

# su - db2bea# db2stop

2. Login as root, and allocate largepages:# echo 342 > /proc/sys/vm/nr_hugepages# mount -t hugetlbfs hugetlbfs /libhugetlbfs

The number of largepages you need to allocate depends on the size of your database and tablespaces.Consult your DB administrator to obtain this information. The test environment used 342 largepages.

3. Instruct DB2 to use hugepage:# db2set DB2_LARGE_PAGE_MEM=DB

4. Restart DB2.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Setting the number of SAP instancesGenerally, the number of SAP instances used in the environment depends upon the size of yourworkload, which in turn dictates the size of your server.

As mentioned earlier, the test environment used the IBM Power 750 Express® as the host server in theSAP 2-tier configuration. This system is equipped with 32 POWER7-processor cores, and 256 GB ofmemory. By using the SAP SD benchmark as a sample workload, it was determined that by using 32 SAPinstances, the system can serve the highest number of users with an average response time for each userat less than 1 second.

As a result, consider ensuring that the number of the SAP instances in your environment are the same asthe number of processor cores you have in your system.



Setting SAP to use Linux hugepageComplete the following steps to configure SAP to use 16 MB page size for its heap and shared memorysegments.

About this task

With the libhugetlbfs library, you can configure existing SAP applications to use Linux hugepages. Thelibhugetlbfs library interacts with the Linux hugetlbfs to make large pages available to applications in atransparent manner. Heap and shared memory segments of an existing program can be backed bylargepages without explicitly linking with the libhugetlbfs library.

Procedure1. Log in as root, and make sure that the libhugetlbfs library is installed with the following command. If

it is not installed, you must install it using the rpms included with your distribution.rpm -qa | grep libhugetlbfs

2. While still logged in as root, allocate largepages with the following commands. The number oflargepages you need to allocate depends on the number of users you want to support in your SAPenvironment. For example, in the test environment, 500 largepages supported approximately 5,300users.echo 500 > /proc/sys/vm/nr_hugepagesmount -t hugetlbfs hugetlbfs /libhugetlbfs

To confirm the allocation of the largepages, run the following command:# cat /proc/meminfo | grep HugePages_Total

3. Log in as the SAP administrator, and use the limit command (if you use c-shell) to increase the limitof resources used by the process and its children..

Important: You might also need to ask the root user to increase the hard limit of the memorylockedoption before you run this commandlimit memorylocked unlimited

4. Add the following statements in the START profile of all of the SAP instances:SETENV_03 = LD_PRELOAD=/usr/lib64/libhugetlbfs.soSETENV_04 = HUGETLB_MORECORE=yesSETENV_05 = HUGETLB_SHM=yes

5. Start all of the SAP instances.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

NUMA overviewA system is Non Uniform Memory Access (NUMA)-based if its memory access time depends on thememory physical location relative to a processor. In other words, the distance between processors andmemory varies depending on the location of the memory that the processor is trying to get to.

Chapter 5. SAP application settings 13

The test server, the IBM Power 750 Express, is a NUMA-based system. It is equipped with four processorcards, and each card contains eight processor cores and eight memory slots. These memory are, therefore,considered local to the processor cores in that card. In total, the test system contains 32 processor coresand 256 GB of memory.

In NUMA terms, a set of processors and all of their local memory are collectively called a numa node. Inthe case of the test environment, there are four numa nodes in the system, and each numa node has eightprocessor cores and 64 GB of local memory. With SMT4 mode, the operating system sees 128 (32x4)logical processors in the system with 32 logical processors in each numa node.

The output of the numactl --hardware command from the test system is shown below. This commandoutputs the information about the number of processors and the amount of memory in each numa node.Moreover, it also shows the distance between each numa node in the form of a matrix at the end.

# numactl --hardwareavailable: 4 nodes (0-3)node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31node 0 size: 61184 MBnode 0 free: 59734 MBnode 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63node 1 size: 65280 MBnode 1 free: 63848 MBnode 2 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95node 2 size: 65024 MBnode 2 free: 63426 MBnode 3 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117118 119 120 121 122 123 124 125 126 127node 3 size: 65024 MBnode 3 free: 63631 MBnode distances:node 0 1 2 30: 10 20 20 201: 20 10 20 202: 20 20 10 203: 20 20 20 10


Binding resources for SAP processesYou can use the numactl command to tune a system with NUMA.

To get the best overall performance, use the numactl command to direct each of the SAP process to usethe memory closest to the processor that it is running on. For example, if a process is running on a logicalprocessor located in numa node 1, this process should only use the memory that is located in numa node1 as well. You can do this using the following two options for the numactl command:v --physcpubind=<cpus> specifies on which logical processors the process should run onv --membind=<nodes> specifies the numa nodes from which the process should allocate memory

In the test system, there are eight processor cores in each numa node, so the eight SAP instances arebound to each numa node in the system. The sample command given below shows how the D01 SAPinstance is bound to logical processors 0 to 31 (in numa node 0), and memory in numa node 0:# numactl –-physcpubind=0-31 –-membind=0 startsap D01

A script that was used in the test environment to bind all of the SAP instances in the test system isprovided for your use in Appendix D, “SAP binding script for use with NUMA nodes,” on page 37. Inthis script, you can see that there are eight instances bound to each numa node and its closest memory,and there are four numa nodes in the system.



Binding resources for DB2 processesYou can use a different memory placement policy for DB2 processes to give you better overallperformance.

In the test environment, the interleave placement policy for the DB2 processes was used. With this policy,memory is allocated from the specified node using a round-robin method. In the test environment, thesystem selected the processors for DB2 processes at random. To select this policy, complete the followingsteps:1. Use the --interleave=<nodes> option with the numactl command, as shown below.

# numactl –-interleave=all db2start# numactl –-interleave=all db2 connect to <DB2 instance>

2. Instruct DB2 to use hugepage with the following command:# db2set DB2_LARGE_PAGE_MEM=DB


Chapter 5. SAP application settings 15

Chapter 6. SAP 2-tier Sales and Distribution Tunings for Linuxon POWER7

You can learn about performance tuning information for the SAP 2-tier Sales & Distribution (SD)application running on the IBM POWER7 based systems. The SAP Sales & Distribution module is a partof the SAP Enterprise Resource Planning (ERP) suite. After applying the recommended tuning steps inthis blueprint, the test environment performance increased by a factor of 2.

Key tools and technologies discussed in this demonstration include NUMA, the CFQ scheduler, SAPinstances, SD benchmark, SMT4 mode, and barriers.ld

Scope, requirements, and supportThis blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Systems to which this information applies

PowerLinux

Intended audience

This blueprint is intended for system administrators and those who have an interest in understandinghow to leverage the performance of IBM POWER7 systems running SLES 11 SP1 for SAP Sales andDistribution workload. It is assumed that the audience of this document has a general understanding ofLinux system administration, fundamental operating system concepts, and background knowledge ofSAP system architecture.

Scope and purpose

This blueprint provides the SLES 11 SP1 tuning recommendations for the SAP 2-tier SD applicationrunning on POWER7 processor-based systems. The information given here was verified using the SAP SDstandard application benchmark. The detailed information about how to set up and tune the SAPbenchmark is not the goal of this blueprint.

Test environment

The information provided is gathered from engineering runs using the SAP enhancement package 4 forSAP ERP 6.0 and IBM DB2 version 9.7 running on SUSE Linux Enterprise Server 11 SP1.

Author namesChakarat Skawratananond

Other contributorsDamon BullHeather CrognaleRichard HendricksonJennifer HopperMonza Lui


IBM Services

Linux offers flexibility, options, and competitive total cost of ownership with a world class enterpriseoperating system. Community innovation integrates leading-edge technologies and best practices intoLinux.

IBM is a leader in the Linux community with over 600 developers in the IBM Linux Technology Centerworking on over 100 open source projects in the community. IBM supports Linux on all IBM servers,storage, and middleware, offering the broadest flexibility to match your business needs.

For more information about IBM and Linux, go to ibm.com/linux (https://www.ibm.com/linux)

IBM Support

Questions and comments regarding this documentation can be posted on the developerWorks SystemsManagement Blueprint Community Forum:


The IBM developerWorks discussion forums let you ask questions, share knowledge, ideas, and opinionsabout technologies and programming techniques with other developerWorks users. Use the forumcontent at your own risk. While IBM attempts to provide a timely response to all postings, the use of thisdeveloperWorks forum does not guarantee a response to every question that is posted, nor do wevalidate the answers or the code that are offered.

SAP software architecture overviewAn instance of the SAP application server is a group of operating system processes that offers one ormore services. An SAP instance consists of a dispatcher, one or more SAP work processes, and a commonset of SAP buffers in the shared memory. Services provided by these work processes include dialog,update, SAP enqueue management, background processing, printing, and SAP gateway.

The dispatcher acts as a coordinator, distributing work to other available work processes. Dialog workprocesses are, for example, used to process requests from users working online. Update work processesare responsible for updating the data in the database.

The following figure shows a sample SAP environment consisting of 33 SAP instances.


http://www.ibm.com/linux


The SAP ERP application suite is based on multi-tier client and server architecture consists of threelayers: database, application, and users. The Application layer represents the SAP application serverfunctionality. In SAP environments, there are two types of multi-tier configurations:v 2-Tier: SAP application and database server layers reside on a single systemv 3-Tier: SAP application and database layers reside on separate systems

In this blueprint, the focus is on the 2-Tier configuration, which is shown in the following figure withfour SAP instances running.

Figure 4. Sample SAP environment consisting of 33 SAP instances

Chapter 6. SAP 2-tier Sales and Distribution Tunings for Linux on POWER7 19


System configurationTo better understand the tuning steps that are presented in this blueprint, you can learn more about thetest environment.

As mentioned earlier, the SAP 2-tier SD standard benchmark was used to verify the tuningrecommendations. The hardware and software setups described are therefore geared toward achieving thebest performance for the benchmark. For more information about the SAP SD benchmark, seeAppendix A, “SAP Sales and Distribution (SD) benchmark overview,” on page 31.

Hardware configuration

The following list includes the hardware components used in the test environment:v System Under Test (SUT) – the server where application and database layers reside

IBM Power 750 Express– 32 POWER7 cores running at 3.55 GHz– 256 GB (32x8 GB) DDR3 1066 Mhz RAM– 8x146.8 GB internal SAS disks– 1x4 GBS Dual Port FibreChannel PCI-E HBA

v Disk enclosure – external disk storage for the SUTIBM TotalStorage DS4500– Six EXP-710 enclosures, with 14x18 GB, 10 KB RPM drives, for a total of 84 disks– Five volumes were created from the 84 disks:

Figure 5. 2-tier SAP configuration


- 1 RAID0 volume stripped across all eight disks for the database log- 1 RAID0 volume stripped across all 29 disks for the main DB2 database- 1 RAID0 volume stripped across all 10 disks for the multiplexed VBDATA tables- 1 RAID5 volume stripped across all 14 disks for database backup files- 1 RAID5 volume stripped across all six disks for the SAP binaries, DB2 home, and others

v Driver Machine – the client machine driving the workload through the serverIBM System p5 550 Express– Four POWER5+ cores, running at 2.1 Ghz– 16 GB RAM

The following figure shows how the hardware components are connected to each other.

The driver machine and the SUT communicate through a private network. SUT and the Disk enclosureare directly attached through a FibreChannel.

Software components

The following list includes the software packages that were required to set up the test environment.v SUSE Linux Enterprise Server 11 SP1v SAP Applications and required runtime library

– SAP enhancement package 4 for SAP ERP 6.0– XL Compiler runtime environment V9

v IBM DB2 v9.7 for Linux on POWERRelated concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

System tuningYou can use various options, such as the SMT4 mode, barriers in the ext3 file system, schedulingparameters, and the CFQ I/O scheduler to fine-tune your system.

Figure 6. Connection of hardware components



Using the CFQ I/O schedulerThe Completely Fair Queueing (CFQ) I/O scheduler controls the way the Linux kernel commits readsand writes to disks and works to optimize disk access times.

Without an I/O scheduler, the kernel would issue each request in the order that it is received, resulting inthrashing: if one process reads from one part of the disk, and one writes to another, the heads would haveto seek back and forth across the disk for every operation. You can use the CFQ scheduler to control howthe kernel reads and writes to disks to avoid thrashing.

Note: Using other schedulers with the SAP SD workload in the test environment did not result in aperformance gain.

Ensure that CFQ is the I/O scheduler enabled on your system with the following command:# cat /sys/block/<device>/queue/scheduler# noop anticipatory deadline [cfq]

where device is the name of the disk on which you want CFQ to run.

The brackets ([]) indicate the scheduler that is currently in effect.

Or, you can change the assigned scheduler with the following command:# echo cfq > /sys/block/<device>/queue/scheduler

where device is the name of the disk on which you want to change the assigned scheduler.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Adjusting CFS parametersThe Linux Completely Fair scheduler (CFS) includes parameters that you can adjust to fit your specificneeds.

The following parameters can be accessed through the proc file system. The default values (innanoseconds) of these parameters from SLES 11 SP1 are shown in parentheses.v sched_min_granularity_ns (16000000): Minimum preemption granularity for processor-bound tasks.

Tasks are guaranteed to run for this minimum time before they are preempted.v sched_latency_ns (80000000): Period over which CFQ tries to fairly schedule the tasks on the runqueue.

All of the tasks on the runqueue are guaranteed to be scheduled once within this period.v sched_wakeup_granularity_ns (20000000): Ability of tasks being woken to preempt the current task.

The smaller the value, the easier it is for the task to force the preemption.

To verify the value of each parameter, use the following command:# cat /proc/sys/kernel/sched_latency_ns

To modify the value of each parameter, use the following command:# echo 20000 > /proc/sys/kernel/sched_latency_ns


In the test environment, best results were achieved with the following values for these three parameters:kernel.sched_min_granularity_ns = 100000kernel.sched_wakeup_granularity_ns = 25000kernel.sched_latency_ns = 1000000

If you want to make these changes permanent, you can modify these parameters in the /etc/sysctl.conffile and run the sysctl -p command.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Disabling barriers in the ext3 file systemYou can turn barriers off to improve performance.

About this task

Journaling file systems, like ext3, ensure the integrity of their journal through the use of barriers, whichprevent the writing of any blocks after the barrier until all blocks written before the barrier arecommitted to the media. Note that while turning barriers off increases performance, it also decreases faulttolerance.

Procedure1. Modify the /etc/fstab file by adding the barrier=0 file system option to your mount point, as

shown:/dev/sdc /SAP ext3 acl,user_xattr,barrier=0 1 1

Note: Take care to not introduce any error when you edit this file, or the system might not be able toboot.

2. Reboot the system.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Using SMT4 modeSimultaneous Multi-Threading (SMT) allows the concurrent execution of multiple instruction streams onthe same processor core. POWER7 offers three types of SMT: 1-way, 2-way and 4-way. With 4-way SMT,you can increase the number of instruction streams your system can run on the same processor core.

In 2-way SMT (SMT2), the number of the logical processors that the operating system sees is double thenumber of the physical processor cores in the system. That number of logical processors becomesquadrupled with the 4-way SMT (SMT4). Consequently, the overall system capacity increases as thenumber of instruction streams increases. This is important for workloads like SAP SD where multipleusers are sharing a set of application server processes. Capacity increase from SMT4 means more userscan simultaneously get their work through the system by using POWER7.

On SLES 11 SP1, the system operates in SMT4 mode by default. If the SMT mode has been switched, youcan switch it back with the ppc64_cpu command as shown, where X is replaced with a numbercorresponding to the SMT mode (2 or 4):# ppc64_cpu –smt=X # Set SMT state to X


In a typical SAP environment, capacity in terms of number of users is often the limiting factor. With theincrease in the overall system capacity from SMT4 mode, more SAP SD users can be supported with agiven server configuration.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Data Stream Control RegisterData Stream Control Register (DSCR) is a 64-bit register on POWER7 processors that controls the degreeof aggressiveness of prefetching data between RAM and caches. For example, you can enable or disablehardware detection and initiation of load or store streams.

You can also set a prefetch depth for hardware-detected streams or software-defined streams. Thefollowing table describes all of the possible settings for DSCR.

Table 2. DSCR settings

Bits Name Description

58 LSD Load Stream Disable

Disables hardware detection andinitiation of load streams.

59 SNSE Stride-N Stream Enable

Enables the hardware detection andinitiation of load and store streamsthat have a stride greater than asingle cache block. Such load streamsare detected only when LSD is alsozero, and store streams are detectedonly when SSE is also one.

60 SSE Enables the hardware detection andinitiation of store streams.

61:63 DPFD Default Prefetch Depth

Supplies a prefetch depth for streams.The values of this field are thefollowing:

0 default1 none2 shallowest3 shallow4 medium5 deep6 deeper7 deepest

On Linux, you can manage the content of this register by echoing a (hexadecimal) number into the/sys/devices/system/cpu/cpuX/dscr directory for each cpuX in the system. For an SAP Sales andDistribution workload, set DSCR to 0x8. That is, Bit-60 (SSE) is set to 1 to enable hardware detection andinitiation of store streams, and Bits 61 to 63 (DPFD) are set to 000 to set the prefetch depth to the defaultvalue.

You can use the following scripts to set DSCR to 0x8 for all 128 processors in your system:


!/bin/bash

i="0"

while [ $i -lt 128 ]doecho 8 > /sys/devices/system/cpu/cpu$i/dscrcat /sys/devices/system/cpu/cpu$i/dscri=$[$i+1]done

For more information about DSCR, go to page 674 at https://www.power.org/resources/downloads/PowerISA_V2.06B_V2_PUBLIC.pdf.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

SAP application settingsSAP application settings that you can use to increase overall performance include communicationbetween DB2 and SAP, hugepages, SAP instances, and resource binding.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Setting communication between DB2 and SAPTo enable the communication between a SAP application server and the database management system, adatabase client must be installed on every application server. Starting with SAP NetWeaver 7.0 SR3, anew DB2 client setup was introduced. In this new setup, the DB2 Call Level Interface (CLI) driver is usedinstead of the DB2 Runtime Client.

The DB2 CLI Driver, also referred to as Thin Client, provides runtime support for applications that use theOpen Database Connectivity (ODBC) or Call Level Interface application programming interface. It is notpart of any of the other clients and must be installed separately.

The DB2 Runtime Client provides support for handling database connections and the underlying networkcommunication, however, it contains no administration or configuration tools.

In the test environment setup, the CLI driver is the default client type that the SAP application serveruses to communicate with DB2. However, with the CLI driver that comes with DB2 V9.7 for Linux onPOWER, the SAP application server performs poorly due to the low level semget calls issued by DB2.Instead, use the DB2 Runtime Client in your environment.

You can direct the SAP application server to use the DB2 Runtime Client by setting theDB2DB6_FORCE_RUNTIME_CLIENT environment variable to 1, as shown, if you are running the csh shell:# setenv DB2DB6_FORCE_RUNTIME_CLIENT 1

Alternatively, you can edit the .dbenv_<hostname>.csh file in the home directory of the SAP administratoruser (for example, <sapsid>adm) to have the environment variable set automatically each time the userlogs in, as shown:




# set Environment Variable DB2DB6_FORCE_RUNTIME_CLIENT to force the usage# of the DB2 Runtime Client#-----------------------------------------------------------------------setenv DB2DB6_FORCE_RUNTIME_CLIENT 1#-----------------------------------------------------------------------


Setting DB2 to use Linux hugepageTo reduce the number of TLB misses, use larger page sizes.

About this task

To translate a virtual page address to its corresponding physical address, the operating system performs alook up in the Page Table. To avoid this issue, modern architectures provide a faster hardware look-upcache called Translation Lookaside Buffer (TLB). When a valid address translation is not present in TLB,this is called TLB misses. By using larger page sizes, the number of TLB misses can be reduced.

Most modern architectures provide multiple page sizes. The ppc64 architecture, for example, supports 4KB, 64 KB, and 16 MB page sizes. In Linux, large page is supported through a virtual file system calledhugetlbfs. This support is also referred to as the hugepage support. An application can make use of largepages in two ways:v by using the mmap system callv by using the standard Unix System V (SysV) shared memory system calls (shmget, shmat)

The SLES 11 SP1 (on ppc64) kernel comes with hugepage support enabled. The default base page size is64 KB, and the hugepage size is 16 MB. To configure DB2 to leverage hugepages, complete the followingsteps.

Procedure1. Login as the DB2 administrator, and stop DB2:

# su - db2bea# db2stop

2. Login as root, and allocate largepages:# echo 342 > /proc/sys/vm/nr_hugepages# mount -t hugetlbfs hugetlbfs /libhugetlbfs

The number of largepages you need to allocate depends on the size of your database and tablespaces.Consult your DB administrator to obtain this information. The test environment used 342 largepages.

3. Instruct DB2 to use hugepage:# db2set DB2_LARGE_PAGE_MEM=DB


Setting the number of SAP instancesGenerally, the number of SAP instances used in the environment depends upon the size of yourworkload, which in turn dictates the size of your server.


As mentioned earlier, the test environment used the IBM Power 750 Express as the host server in the SAP2-tier configuration. This system is equipped with 32 POWER7-processor cores, and 256 GB of memory.By using the SAP SD benchmark as a sample workload, it was determined that by using 32 SAPinstances, the system can serve the highest number of users with an average response time for each userat less than 1 second.

As a result, consider ensuring that the number of the SAP instances in your environment are the same asthe number of processor cores you have in your system.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Setting SAP to use Linux hugepageComplete the following steps to configure SAP to use 16 MB page size for its heap and shared memorysegments.

About this task

With the libhugetlbfs library, you can configure existing SAP applications to use Linux hugepages. Thelibhugetlbfs library interacts with the Linux hugetlbfs to make large pages available to applications in atransparent manner. Heap and shared memory segments of an existing program can be backed bylargepages without explicitly linking with the libhugetlbfs library.

Procedure1. Log in as root, and make sure that the libhugetlbfs library is installed with the following command. If

it is not installed, you must install it using the rpms included with your distribution.rpm -qa | grep libhugetlbfs

2. While still logged in as root, allocate largepages with the following commands. The number oflargepages you need to allocate depends on the number of users you want to support in your SAPenvironment. For example, in the test environment, 500 largepages supported approximately 5,300users.echo 500 > /proc/sys/vm/nr_hugepagesmount -t hugetlbfs hugetlbfs /libhugetlbfs

To confirm the allocation of the largepages, run the following command:# cat /proc/meminfo | grep HugePages_Total

3. Log in as the SAP administrator, and use the limit command (if you use c-shell) to increase the limitof resources used by the process and its children..

Important: You might also need to ask the root user to increase the hard limit of the memorylockedoption before you run this commandlimit memorylocked unlimited

4. Add the following statements in the START profile of all of the SAP instances:SETENV_03 = LD_PRELOAD=/usr/lib64/libhugetlbfs.soSETENV_04 = HUGETLB_MORECORE=yesSETENV_05 = HUGETLB_SHM=yes

5. Start all of the SAP instances.



NUMA overviewA system is Non Uniform Memory Access (NUMA)-based if its memory access time depends on thememory physical location relative to a processor. In other words, the distance between processors andmemory varies depending on the location of the memory that the processor is trying to get to.

The test server, the IBM Power 750 Express, is a NUMA-based system. It is equipped with four processorcards, and each card contains eight processor cores and eight memory slots. These memory are, therefore,considered local to the processor cores in that card. In total, the test system contains 32 processor coresand 256 GB of memory.

In NUMA terms, a set of processors and all of their local memory are collectively called a numa node. Inthe case of the test environment, there are four numa nodes in the system, and each numa node has eightprocessor cores and 64 GB of local memory. With SMT4 mode, the operating system sees 128 (32x4)logical processors in the system with 32 logical processors in each numa node.

The output of the numactl --hardware command from the test system is shown below. This commandoutputs the information about the number of processors and the amount of memory in each numa node.Moreover, it also shows the distance between each numa node in the form of a matrix at the end.

# numactl --hardwareavailable: 4 nodes (0-3)node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31node 0 size: 61184 MBnode 0 free: 59734 MBnode 1 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63node 1 size: 65280 MBnode 1 free: 63848 MBnode 2 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95node 2 size: 65024 MBnode 2 free: 63426 MBnode 3 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117118 119 120 121 122 123 124 125 126 127node 3 size: 65024 MBnode 3 free: 63631 MBnode distances:node 0 1 2 30: 10 20 20 201: 20 10 20 202: 20 20 10 203: 20 20 20 10


Binding resources for SAP processesYou can use the numactl command to tune a system with NUMA.

To get the best overall performance, use the numactl command to direct each of the SAP process to usethe memory closest to the processor that it is running on. For example, if a process is running on a logicalprocessor located in numa node 1, this process should only use the memory that is located in numa node1 as well. You can do this using the following two options for the numactl command:v --physcpubind=<cpus> specifies on which logical processors the process should run onv --membind=<nodes> specifies the numa nodes from which the process should allocate memory


In the test system, there are eight processor cores in each numa node, so the eight SAP instances arebound to each numa node in the system. The sample command given below shows how the D01 SAPinstance is bound to logical processors 0 to 31 (in numa node 0), and memory in numa node 0:# numactl –-physcpubind=0-31 –-membind=0 startsap D01

A script that was used in the test environment to bind all of the SAP instances in the test system isprovided for your use in Appendix D, “SAP binding script for use with NUMA nodes,” on page 37. Inthis script, you can see that there are eight instances bound to each numa node and its closest memory,and there are four numa nodes in the system.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Binding resources for DB2 processesYou can use a different memory placement policy for DB2 processes to give you better overallperformance.

In the test environment, the interleave placement policy for the DB2 processes was used. With this policy,memory is allocated from the specified node using a round-robin method. In the test environment, thesystem selected the processors for DB2 processes at random. To select this policy, complete the followingsteps:1. Use the --interleave=<nodes> option with the numactl command, as shown below.

# numactl –-interleave=all db2start# numactl –-interleave=all db2 connect to <DB2 instance>

2. Instruct DB2 to use hugepage with the following command:# db2set DB2_LARGE_PAGE_MEM=DB



Appendix A. SAP Sales and Distribution (SD) benchmarkoverview

You can use the SAP Standard Application Benchmarks to find the appropriate hardware configurationfor your IT solutions.

Benchmark results provide basic sizing recommendations by running a workload through the servers,software components, and the relational database systems. You can also use the benchmark results toevaluate the proof-of-concept scenarios, and the scalability and manageability of your large SAPinstallations.

The procedure to run the benchmark is standardized, well-defined, and monitored by the SAP benchmarkcouncil.

An SAP Standard Application Benchmark consists of script files that simulate the most typicaltransactions and work flow of a user scenario. It has a predefined SAP client database that containssample company data against which the benchmark is run. Benchmark users have their own master data,such as inventory, vendor, or customer data to avoid data-locking situations. In general, a number ofsimultaneous benchmark users can be simulated per client. For the SD benchmark, 1000 users can besimulated in parallel with each client.

However, the SD benchmark kit is available only to SAP partners. Customers are generally not allowed touse the benchmark kit unless they get permission from SAP, and SAP provides the benchmark kit to thecustomer directly.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.


Appendix B. SD benchmark scenario

The Sales and Distribution benchmark is one of the most processor-intensive benchmarks available andhas become the standard for SAP's platform partners and the ERP (Enterprise Resource Planning)environment.

SD benchmark covers a sell-from-stock scenario, which includes the creation of a customer order withfive line items and the corresponding delivery with subsequent goods movement and invoicing. Thescenario consists of the following transactions:v Create an order with five line itemsv Create a delivery for this orderv Display the customer orderv Change the delivery and post goods issuedv List 40 orders for one sold-to partyv Create an invoice

Each of the simulated users repeats this series of transactions from the start to the end of a benchmarkrun. A benchmark run consists of three phases:v Ramp-up phase, where all users log on one after another.v High Load phase, where all users run their actions simultaneously.v Ramp-down phase, where all users log off one after another.

For a certifiable SD benchmark, its high-load phase must last at least 15 minutes. The following figureshows how the benchmark progresses through time.


You can increase or decrease the length of a benchmark run by setting the number of repetitions of thescenario (loops) for the users. By default, users are started with a delay of 1 second to avoid the situationthat all users are running the same transaction at the same time. This is, however, a configurableparameter in the benchmark. You can also configure the number of users that log in per second.

An important part of the information that must be provided to the SAP benchmark council is the type ofclient configuration and type of server configuration used in the benchmark run. The multi-tier client andserver architecture consists of a database, application, and presentation layers. The presentation layerrepresenting the user transactions is normally handled by one or more machines (usually referred to asbenchmark drivers). The application layer represents the SAP application server functionality.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.

Figure 7. Benchmark progress through time


Appendix C. SD benchmark performance metrics

In 2009, some changes were made to the SD benchmarks.

The key performance metrics for the SD benchmark are:v the average user response timev the number of supported users (with the response time of less than 1 second)

A typical result, therefore, would read as the following:2500 SD benchmark users with an average dialog response time of 0.98 seconds

In 2009, SAP made changes to all of their ERP benchmarks (including SD). The changes related to SDbenchmarks are:v All benchmarks now must use a Unicode codepagev The response time for the SD benchmark must now be below one second (previously, response times

below 2 seconds were required)

As a result, the SD benchmark is now more resource-intensive. Careful consideration must be made whencomparing the pre-2009 results with results from 2009 and later.Related concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.


Appendix D. SAP binding script for use with NUMA nodes

You can use the following script to help you bind SAP instances to NUMA nodes.#!/usr/bin/perl# This script does the following:# Bind SAP D01-D08 to CPU 00-31 and Memory 0# Bind SAP D09-D16 to CPU 32-63 and Memory 1# Bind SAP D17-D24 to CPU 64-95 and Memory 2# Bind SAP D25-D32 to CPU 96-127 and Memory 3

$start_cpu=0;$end_cpu=31;$inst_boundry = 9 ;$node = 0;$sleep = 8;

$inst_a = 1 ;$inst_b = 2 ;$inst_c = 3 ;$inst_d = 4 ;

system("numactl --physcpubind 96-127 --membind=3 startsap DVEBMGS00 &") ;for ($sock = 1 ; $sock <= 4 ; $sock++){

print "starting instances in node $node: \n";print "Node: $node\n";

while ( $inst_a < $inst_boundry ) {$cpu_range = $start_cpu . "-" . $end_cpu;$a_inst = "D" . sprintf("%02d",$inst_a);$b_inst = "D" . sprintf("%02d",$inst_b);$c_inst = "D" . sprintf("%02d",$inst_c);$d_inst = "D" . sprintf("%02d",$inst_d);

print "$cpu_range: $a_inst $b_inst $c_inst $d_inst memory node: $node\n";

system("numactl --physcpubind $cpu_range --membind=$node startsap $a_inst &") ;system("numactl --physcpubind $cpu_range --membind=$node startsap $b_inst &") ;system("numactl --physcpubind $cpu_range --membind=$node startsap $c_inst &") ;system("numactl --physcpubind $cpu_range --membind=$node startsap $d_inst &") ;

print "Waiting for instances to start. Sleeping $sleep seconds . . .\n";sleep $sleep;

$inst_a = $inst_a + 4 ;$inst_b = $inst_b + 4 ;$inst_c = $inst_c + 4 ;$inst_d = $inst_d + 4 ;

}print "\n\n";$inst_boundry += 8;$start_cpu +=32;$end_cpu +=32;$node++;

}



Appendix E. Socket scaling

Modern servers come equipped with multiple processor cards (also referred to as chips) enablingapplications to scale up if they need to. In this section, additional performance data is provided using theSAP 2-tier SD benchmark to demonstrate the scalability of the SAP SD workload on the testenvironment's multi-chip testing system.

As previously described, the test environment system is the IBM Power 750 Express and contains fourprocessor chips (each with eight processor cores), for a total of 32 processor cores. The term chip, as usedin this description, is synonymous with chips.

To gather the performance data for using 1, 2, 3, and 4 chips, the test system was first configured withonly 1 chip activated. Then, the benchmark was run and data was collected. The system was thenreconfigured with 2 chips enabled, and the benchmark was rerun. This same method continued with 3and 4 chips. In total, the benchmark ran four times:v 1-chip, 8 cores, 64 GB RAM, 9 SAP instancesv 2-chip, 16 cores, 128 GB RAM, 17 SAP instancesv 3-chip, 24 cores, 192 GB RAM, 25 SAP instancesv 4-chip, 32 cores, 256 GB RAM, 33 SAP instances

In each set, the same method for binding the processors and memory to SAP instances was used. Forexample, in the 2-chip run, the first eight SAP instances were bound to processors and memory in thefirst chip (numa node 0), and the other nine instances were bound to the second chip (numa node 1). Theinterleave memory policy (with memory from node 0 and node 1) was used for all DB2 processes.

Results from these runs are shown in the following figure. As you can see, the SD benchmark scaleslinearly on this server when you increase the number of chips from 1 to 4.


Figure 8. The benchmark scales linearly as the number of chips increase


Appendix F. Related information

You can find additional information about the processes and tools described in this blueprint.v CFS Scheduler

http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txtv Linux I/O scheduler

http://www.wlug.org.nz/LinuxIoSchedulerv Powerpc-utils

http://powerpc-utils.ozlabs.org/v Barrier and Journaling Filesystems

http://lwn.net/Articles/283161/v Linux Hugepage support

http://www.kernel.org/doc/Documentation/vm/hugetlbpage.txtv SAP Benchmark: Sales and Distribution

http://www.sap.com/solutions/benchmark/sd.epxRelated concepts:Chapter 1, “Scope, requirements, and support,” on page 1This blueprint applies to PowerLinux. You can learn more about this blueprint, including the intendedaudience, the scope and purpose, and the types of support available to you.


http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt

http://www.wlug.org.nz/LinuxIoScheduler

http://powerpc-utils.sourceforge.net/

http://lwn.net/Articles/283161/

http://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt

http://www.sap.com/solutions/benchmark/sd.epx

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries.Consult your local IBM representative for information on the products and services currently available inyour area. Any reference to an IBM product, program, or service is not intended to state or imply thatonly that IBM product, program, or service may be used. Any functionally equivalent product, program,or service that does not infringe any IBM intellectual property right may be used instead. However, it isthe user's responsibility to evaluate and verify the operation of any non-IBM product, program, orservice.

IBM may have patents or pending patent applications covering subject matter described in thisdocument. The furnishing of this document does not grant you any license to these patents. You can sendlicense inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where suchprovisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATIONPROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS ORIMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFNON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Somestates do not allow disclaimer of express or implied warranties in certain transactions, therefore, thisstatement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodicallymade to the information herein; these changes will be incorporated in new editions of the publication.IBM may make improvements and/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) theexchange of information between independently created programs and other programs (including thisone) and (ii) the mutual use of the information which has been exchanged, should contact:

IBM CorporationDept. LRAS/Bldg. 90311501 Burnet RoadAustin, TX 78758-3400U.S.A.

Such information may be available, subject to appropriate terms and conditions, including in some cases,payment of a fee.

The licensed program described in this document and all licensed material available for it are providedby IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement orany equivalent agreement between us.


For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual PropertyDepartment in your country or send inquiries, in writing, to:

IBM World Trade Asia CorporationLicensing2-31 Roppongi 3-chome, Minato-kuTokyo 106-0032, Japan

IBM may use or distribute any of the information you supply in any way it believes appropriate withoutincurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, theirpublished announcements or other publicly available sources. IBM has not tested those products andcannot confirm the accuracy of performance, compatibility or any other claims related to non-IBMproducts. Questions on the capabilities of non-IBM products should be addressed to the suppliers ofthose products.

Any references in this information to non-IBM Web sites are provided for convenience only and do not inany manner serve as an endorsement of those Web sites. The materials at those Web sites are not part ofthe materials for this IBM product and use of those Web sites is at your own risk.

This information contains examples of data and reports used in daily business operations. To illustratethem as completely as possible, the examples include the names of individuals, companies, brands, andproducts. All of these names are fictitious and any similarity to the names and addresses used by anactual business enterprise is entirely coincidental.

TrademarksIBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International BusinessMachines Corporation in the United States, other countries, or both. If these and other IBM trademarkedterms are marked on their first occurrence in this information with a trademark symbol (® and ™), thesesymbols indicate U.S. registered or common law trademarks owned by IBM at the time this informationwas published. Such trademarks may also be registered or common law trademarks in other countries. Acurrent list of IBM trademarks is available on the Web at Copyright and trademark information atwww.ibm.com/legal/copytrade.shtml

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarksof Adobe Systems Incorporated in the United States, and/or other countries.

Java™ and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/orits affiliates.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Other company, product, or service names may be trademarks or service marks of others.


��

Printed in USA

sap 2-tier sales and distribution tunings for linux on power7

Documents