active memory expansion

Active Memory Expansion performance

Updated: 2 November 2012

By Dirk Michel

Power Systems Performance

2012 updates by Bob Welgan,

IBM Systems ISV Enablement

SAP Performance Benchmarking


Active Memory Expansion Performance.doc

Preface 3

Acknowledgments: 3

Disclaimer: 3

1 Introduction 4

2 Active Memory Expansion overview 5

2.1 Active Memory Expansion concept 5

2.2 Active Memory Expansion value 6

3 Active Memory Expansion performance considerations 7

3.1 Memory expansion factor 7

3.2 Application response time 8

4 Active Memory Expansion performance measurements 9

4.1 ERP overview 9

4.1.1 ERP workload performance measurements 9

4.1.2 ERP workload — single partition throughput 11

4.1.3 ERP workload — summary 13

5 Conclusion 14

Appendix A: POWER7 ERP workload — server throughput 15

Trademarks and special notices 18



Preface This document introduces the basic concepts of Active Memory Expansion, showing the principles of operation and performance characteristics of this new component of the AIX® operating system. Active Memory Expansion is available on IBM POWER7® systems starting with AIX 6.1 TL04 SP2. This paper is an update to an earlier paper and it contains data collected from experiments on an IBM POWER7+™ system.

The audience for this document consists of computer users, administrators, programmers, and performance analysts, as well as team leaders and management who need to understand the basics of this technology.

You can find a detailed description of design points and detailed instructions for the configuration of Active Memory Expansion in the AIX information center at http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.prftungd/doc/prftungd/intro_ame_process.htm and the Active Memory Expansion white paper ibm.com/systems/power/hardware/whitepapers/am_exp.html.

Acknowledgments:

We would like to thank the people who made invaluable contributions to this paper as well as the developers of this remarkable technology. Contributions included authoring, insights, reviews, critiques and reference documents.

Co-authors of this document are:

Thuy Nguyen, Boyd Murrah, Walter Orb, Joerg Droste, and Jose Escalera

Disclaimer:

All performance data contained in this publication was obtained in the specific operating environment and under the conditions described below and is presented as an illustration. Performance obtained in other operating environments may vary and customers should conduct their own testing.



1 Introduction All computers have a limited amount of random access memory (RAM) in which to run programs. Therefore, one of the perennial design issues for all computer systems is how to make the best use of the entire RAM that is physically available in the system, in order to run as many programs concurrently as possible, in the limited space available.

Active Memory Expansion, originally a POWER7 feature, supplies a new technique for making better use of RAM: Portions of programs that are infrequently used are compressed into a smaller space in RAM. This, in turn, expands the amount of RAM available for the same or other programs. Starting with POWER7+, Active Memory Expansion memory page compression and decompression is offloaded to a hardware accelerator.

This white paper describes the process of Active Memory Expansion in more detail in the next sections, and then reports measurements on a typical workload which illustrates the beneficial effects. Among the benefits of Active Memory Expansion, this paper shows the following scenarios and their performance results:

1. Reducing the physical memory requirement of an logical partition (LPAR) resulting in 125% memory expansion on a POWER7+ system

2. Increasing the effective memory capacity and throughput of a memory-constrained LPAR, resulting in a 54% increase in application throughput on a POWER7+ system



2 Active Memory Expansion overview RAM is allocated in contiguous units called pages. In an LPAR with Active Memory Expansion enabled, a certain proportion of the RAM is dedicated to serve as auxiliary storage to hold compressed data. When the operating system determines that a page of uncompressed memory is not currently being referenced, and there is demand for memory, then the content of the page can be compressed and migrated to the auxiliary storage dedicated for Active Memory Expansion. Starting with POWER7+ systems, Active Memory Expansion logic is implemented in both hardware and software.

Compression of temporarily unreferenced data reduces the total space requirement for that data and frees up more memory than it costs, leaving more RAM to execute programs. Data is expanded when it is referenced again. Capacity of the total system can be increased with the appropriate combination of available RAM and available processor time to compress and uncompress the content of memory pages.

2.1 Active Memory Expansion concept

The effect of memory compression and the expansion of active memory with the Active Memory Expansion feature are illustrated in the following figure:

Memory

Expanded Memory

UncompressedPool

AME

Real M

em

ory

Siz

e

Real M

em

ory

Siz

e

Expa

nd

ed

Mem

ory

Siz

e

Compressed Pool

Figure 1. Effect of memory compression and the expansion of active memory

Figure 1 demonstrates an example of taking memory away from an existing LPAR and running it with less physical memory.

The figure shows:

1. For a given data set in RAM, compression of data in inactive (or seldom active) pages reduces the total memory requirement for that data set. If the logical memory size in the LPAR is not changed, this expands the space available for active pages in running programs. Alternatively, the extra space can be given to another LPAR in the same frame, or the number of LPARs in the frame can be increased.

2. In view of the fact that compressed memory cannot be directly used by running programs, the diagram illustrates that the logical memory in any LPAR using Active Memory Expansion must be partitioned into active (uncompressed) and inactive (compressed) page pools.



3. The degree of compression illustrated in the figure above is purely for purposes of illustration. The compressibility of workloads will vary.

4. Pages in the uncompressed pool will be migrated to the compressed pool when they have been unreferenced for a sufficient length of time and there is a demand for memory pages. Pages in the compressed pool are migrated back on demand by referencing them. These operations are transparent to the application.

5. The size of the compressed memory pool is dynamically controlled by the operating system. It is not fixed. Expansion factor and compressibility of the data determine the size of the compressed pool.

2.2 Active Memory Expansion value

As described in 2.1, Active Memory Expansion increases the memory capacity of a partition. The real value is the potential usage of the extra memory to add more workload, leading to a higher throughput and increased utilization of the partition.

Figure 2 illustrates another way Active Memory Expansion can be used to increase utilization and throughput:

System Memory

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

LPAR LPAR LPAR

System Memory

Active

MemoryExpansion

Figure 2. Reducing LPAR physical memory consumption to free memory

In environments where the number of LPARs is limited by the physical amount of memory that is available to the system, Active Memory Expansion reduces the physical memory consumption of the individual LPARs and frees up memory that can be used for additional LPARs to drive more workload on the system.



3 Active Memory Expansion performance considerations Application performance in an Active Memory Expansion environment depends on multiple factors such as the memory expansion factor, application response time sensitivity and how compressible the data is.

The graph below illustrates the general relationship between application response time, application throughput, processor utilization and the percentage memory expansion. The processor utilization increases with larger percentage memory expansion due to more compression and decompression activity, which has an impact on the application response time. An increase in application response time often results in lower application throughput.

% Memory

Expansion

Application Response Time

Application Throughput

CPU Utilization

3.1 Memory expansion factor

The memory expansion factor has a direct impact on performance because it has an impact on the sizes of the compressed and uncompressed pools and therefore, the amount of active memory that can be held in uncompressed memory. As the amount of active memory becomes larger than the current size of the uncompressed memory pool, more compression and decompression activity occurs.

Figure 3 illustrates the impact on the sizes of the compressed and uncompressed pools for reasonable and aggressive memory-expansion factors.



Expanded Memory

Uncompressed

Pool

Compressed Pool

Expanded Memory

UncompressedPool

Compressed Pool

Reasonable Memory Expansion Factor

Aggressive Memory

Expansion Factor

Large Uncompressed

Pool

Small Compressed

Pool

Small Uncompressed

Pool

Large Compressed

PoolTru

e

Mem

ory

Siz

e

Tru

e

Me

mo

ry S

ize

Exp

and

ed

M

em

ory

Siz

e

Exp

and

ed

M

em

ory

Siz

e

Figure 3. Impact on the sizes of the compressed and uncompressed pools for memory-expansion factors

• A reasonable memory-expansion factor maintains a small compressed pool and a relatively large uncompressed pool.

• An aggressive memory-expansion factor leads to a larger compressed pool and a smaller uncompressed pool.

The performance of a workload is less affected by an aggressive memory-expansion factor if the workload has a small active working set and therefore a low amount of compression and decompression activity. However, the amount of compression and decompression activity increases if the memory-compression factor is set too aggressively. The performance of workloads with a large active memory set is more sensitive to an aggressive memory expansion factor.

A reasonable memory-expansion factor can be determined by running the Active Memory Expansion Planning Tool, amepat from the command line.

3.2 Application response time

When an application accesses a page that is currently compressed, the operating system automatically decompresses the page and makes it available to the application. This can increase an applications response time because the application has to wait during the decompression process. Although the time to decompress a single page is very short, a large amount of decompression activity can have a noticeable impact on response time.



4 Active Memory Expansion performance measurements You can use Active Memory Expansion for a wide variety of workloads like enterprise resource planning (ERP), online transaction processing (OLTP), online stock trading, and so on. This section describes the performance results for ERP workloads in an Active Memory Expansion environment.*

4.1 ERP overview

ERP workloads such as SAP ERP provide an effective metric to determine systems performance running common business transactions. In the case where the system is constrained by available memory, there are likely to be unused processor resources available. Active Memory Expansion takes advantage of this by using the idle cycles to manage the physical memory in a way that makes more free space available for the application to run.

The following two scenarios will be described in more detail in subsequent sections; they show that you can use Active Memory Expansion to:

1. Reduce the physical memory requirement for an LPAR

2. Increase the memory capacity and throughput of a memory-constrained LPAR

The application workload chosen for this study represented a sample set of standard SAP ERP business transactions. Multiple simulated users all ran the same set of predefined transactions.

The software configuration for these tests was based on SAP ERP 6.0 (Unicode) and the IBM DB2® 10.1 database running on the IBM AIX® 7.1 operating system and the POWER7+ processor.

4.1.1 ERP workload performance measurements

The following example demonstrates the use of Active Memory Expansion to reduce the physical memory requirements of an LPAR on a POWER7+ server that is running an SAP ERP workload. The SAP ERP workload was configured to run a set number of users to simulate concurrent business transactions. For the baseline measurement, the workload was run with enough users to drive the processor utilization of the system to approximately 60%. Active Memory Expansion was disabled during the baseline measurement.

The next step was to reconfigure the partition with a physical memory configuration matching the virtual memory demand of the SAP ERP workload near the 60% utilization level. Then Active Memory Expansion was enabled and a test sequence was run using the same workload as before with a number of different memory configurations. The results are shown in Table 1.

* Users on systems with large amounts of memory might experience performance degradation when using Active

Memory Expansion if their application used and benefitted from memory pages that are larger than 4 Kb. Active

Memory Expansion automatically reconfigures the system to use only 4 Kb memory pages when enabled.



Memory GB 64.5 43.25 38.75 32.25 30.25 25.75 23.25 21.25 Memory expansion 0% 49% 67% 100% 113% 150% 177% 204% Relative performance

1.00 1.00 1.00 1.00 1.00 0.99 0.99 0.96

Processor % 59.9 59.8 60.8 62.0 64.4 80.6 86.1 83.4

Table 1. Results with Active Memory Expansion enabled

The memory expansion metric in Table 1is calculated by:

((baseline physical memory - new physical memory) / (new physical memory)) * 100

0

10

20

30

40

50

60

70

80

90

100

0% 25% 50% 75% 100% 125% 150% 175% 200% 225%

Memory Expansion %

CP

U %

Figure 4. Processor utilization at various memory-expansion rates

Figure 4 demonstrates the processor utilization of the sample SAP ERP workload at different memory expansions.

Figure 5 shows the relative performance of the same workload as in the previous graph. There is virtually no measurable performance impact due to Active Memory Expansion until the memory expansion exceeds 125% and even then the performance impact is very slight. This demonstrates how Active Memory Expansion can help maintain application performance even in a reduced physical memory footprint.



0.60

0.80

1.00

0% 49% 67% 100% 113% 150% 177% 204%

Memory Expansion %

Re

lati

ve

pe

rfo

rma

nc

e

Figure 5. Relative performance

As described in the previous section, the SAP ERP workload tests the ability of a configuration to support multiple users following through the simulated steps to complete a number of business transactions.

Figure 5 shows the relative application performance of Active Memory Expansion enabled configurations compared to a base configuration that has all the necessary physical resources available.

In this memory-constrained environment, as physical memory is reduced, Active Memory Expansion compensates by using available processor cycles to make more memory available. In spite of the additional work dispatched to the processor, the number of transactions per second completed is not affected. The memory expansion, as shown, can continue to increase as long as processor resources are available. After a threshold of processor availability is crossed, performance stability decreases.

As long as there are processor cycles available, Active Memory Expansion allows the user to size the configuration for a workload beyond the ordinary limitations imposed by the lack of available physical memory.

4.1.2 ERP workload — single partition throughput

Figure 6 and Figure 7 show the results of increasing the workload in a memory-constrained partition on a POWER7+ server. The first workload was run in a single partition with Active Memory Expansion turned off, and then the same workload was run with Active Memory Expansion enabled. Figure 6 shows the achieved throughput in transactions per second (TPS) at a specific workload level. Figure 7 shows the respective processor utilization for that workload. The columns in the Figure 7 represent the average number of virtual memory page-in operations per second as an indicator of operating system (OS) paging. The paging statistics are only shown for the test sequence with Active Memory Expansion disabled, as no OS paging occurred during the test sequence with Active Memory Expansion turned on.



Throughput

0

100

200

300

400

500

600

2496

2624

2752

2880

3008

3136

3264

3392

3520

3648

3776

3904

4032

4160

4288

4416

4544

4672

4800

4928

5056

Users

Tra

nsacti

on

s p

er

seco

nd

AME off AME on

Figure 6. Throughput without and with Active Memory Expansion turned on

CPU Utilization

0

10

20

30

40

50

60

70

80

90

100

2496 2624 2752 2880 3008 3136 3264 3392 3520 3648 3776 3904 4032 4160 4288 4416 4544 4672 4800 4928 5056

Users

CP

U %

Bu

sy

0

20

40

60

80

100

120

140

160

180

200

pag

e-i

ns p

er

sec

Avg pi AME off AME off AME on

Figure 7. Processor utilization without and with Active Memory Expansion turned on

Up to the 3,264 user level, the throughput and processor utilization are nearly identical in both cases. With Active Memory Expansion turned off, there is already a significant amount of OS paging at the 3,264 user level starting to affect the throughput. At 3,392 users the throughput actually decreased because of heavy OS paging. At this level, you are using only 60% of the available processor resources; however the simulated users are suffering from increased response times (and thus decreased throughput). After turning Active Memory Expansion on, you can easily run the same workload and then scale the load up all the way to 5,058 users and a processor utilization of 90%. The throughput increased by 54% going from 3,392 users to 5,056 users.

OS paging



4.1.3 ERP workload — summary

The test results in the previous sections show that partitions running a standard SAP ERP workload are good candidates for turning Active Memory Expansion on. Note that the simulated environment used SAP ABAP application servers only. SAP Java™ servers were not part of the test-case.

The test showed that you can use Active Memory Expansion to reduce the physical memory requirement of a partition with little impact to transaction throughput. You can also use Active Memory Expansion to overcome memory constraints in a partition and thus enable the partition to handle additional workload.

However, the test cases proved that you can add new partitions to the server as well. A common scenario could be to add partitions for additional two-tier test, development, or training systems on servers with spare processor and limited memory capacity.

The scenarios used in these tests showed very good compressibility for our workloads — two-tier systems running the database and application server instances within the same partition.



5 Conclusion The scenarios of reducing physical memory requirement of a partition, and increasing a partition’s memory capacity outlined in this paper have shown that Active Memory Expansion for AIX on POWER7+ technology provides a significant improvement in system throughput and utilization by expanding system memory capacity.



Appendix A: POWER7 ERP workload — server throughput This appendix shows the results of testing Active Memory Expansion on a POWER7 server. There were no test updates on POWER7+ for the update of this paper. However, you can use Active Memory Expansion in a similar way on a POWER7+ server to achieve the following goals:

• Reduce the physical memory requirement of a single partition without impact to the throughput

• Increase the throughput in a single memory constrained partition

The next set of measurements conducted on a POWER7 system was intended to demonstrate Active Memory Expansion in a production-like setup with multiple partitions running on a single server. This test was conducted on an AIX 6.1 system with DB2 Version 9.5. This test simulated a SAP 3-tier setup with a single SAP system on a server with 48 GB of physical memory. The first partition was running the SAP ERP database server and an application instance. Three more partitions were configured as SAP application servers connected to the database server running in the first partition.

Table 2 shows the configuration of the server partitions:

Partition memory (GB) Partition Number of processors Active

Memory Expansion off

Active Memory Expansion on

Role

1 8 20 18 Database sever and application server

2 8 14 10 Application server

3 8 14 10 Application server

4 8 0* 10 Application server

* Partition 4 was deactivated during the runs with Active Memory Expansion turned off

Table 2. Configuration of the server partitions

For the runs with Active Memory Expansion disabled, the simulated users were distributed over the first three partitions. The fourth partition was deactivated, as all of the server’s installed physical memory was used. After turning Active Memory Expansion on, the physical memory allocation for each of the first three partitions was reduced. This freed up 10 GB of physical memory, which allowed the activation of the fourth partition. The simulated users were then distributed over all four partitions.

Figure 8 and Figure 9 show the results of scaling up the workload for both configurations. Note that these are shown as column graphs because the increase in workload was not done in equidistant steps (unlike in the previous section).

Figure 8 the combined throughput of all active partitions in transactions per second (TPS) at a specific workload level. Figure 9 shows the respective processor utilization of the physical server for that workload. For the test case with Active Memory Expansion



turned off, the eight deactivated processors of partition 4 were included in the processor utilization calculation as idle processors.

Server Throughput

250

300

350

400

450

500

2900 3200 3780 4500 5000

Users

TP

S

AME off AME on

heavy OS paging

Figure 8. Server throughput

Server Utilization

0

20

40

60

80

100

2900 3200 3780 4500 5000

Users

CP

U %

AME off AME on

heavy OS paging

Figure 9. Server utilization

For the first two data points, the throughput and processor utilization are almost identical for both test cases, regardless of whether the simulated users were distributed over three partitions (with Active Memory Expansion turned off) or over four partitions (with Active Memory Expansion turned on). The partitions that had Active Memory Expansion turned off started to page in the operating system at the 2900 user level. Because paging is unwanted in a production environment, the test used the 2900 user level as



baseline for our comparison in this section. At the 3780 user level, the throughput for the configuration with Active Memory Expansion turned off decreased caused by heavy paging in the operating system. The processor utilization actually dropped as well as the runable user threads were slowed down by having to wait for their required memory pages to be paged in.

In this particular test-case, the OS statistics showed an average of 400 page-in operations per second in a partition, with peaks of more than 4800 page-ins per second. The average response time of the simulated users went up from below 200 milliseconds to over 4 seconds. In real life, a system with such a heavy amount of OS paging would be basically unusable, even though there is plenty of spare processor capacity to handle additional workload.

The configuration with four partitions and Active Memory Expansion enabled was easily able to handle the same workload and subsequently, the workload was scaled up to 5000 simulated users with a subsecond response time. The throughput increased by 60% going from 3200 users to 5000 users.

The total virtual memory demand for all partitions at the 5000 user level was a little more than 65 GB, so the Memory Expansion for this test-case was about 35%. This number is lower than the ones shown in section 4.1.1 ERP workload performance measurements for two reasons:

• The workload was significantly increased and the major part of the spare processor capacity was used to process the increased business workload.

• The virtual memory footprint of an application server-only partition is relatively smaller than a partition running both a database and an application server. For the same workload, the percentage of actively used memory pages in an application server-only system is higher than in a database and application server partition.

A test case with four partitions each running its own SAP database and application server instance would result in overall higher virtual memory demand and lead to higher memory expansion numbers.

A test case similar to the one in section 4.1.1 ERP Workload Performance Measurements using the four-partition setup with a fixed workload of 3600 users was also run. During the test sequence, the physical memory configuration for each partition was reduced. Table 3 shows the test results for this test-case:

Memory GB 48 44 39 34 32 28

Memory expansion 0% 9% 23% 41% 50% 71%

Throughput (TPS) 356 356 355 353 354 343

Processor % 60 60 71 77 81 90

Table 3. ERP workload performance with 3600 users

The first row shows the combined physical memory allocation for all partitions. The measured throughput stays virtually the same up to 50% memory expansion and decreased slightly at the 71% memory expansion level.



Trademarks and special notices

© Copyright. IBM Corporation 2012. All rights reserved.

References in this document to IBM products or services do not imply that IBM intends to make them

available in every country.

AIX, DB2, IBM, the IBM logo, ibm.com and POWER7+ are trademarks or registered trademarks of

International Business Machines Corporation in the United States, other countries, or both:

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other

countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

The information provided in this document is distributed “AS IS” without any warranty, either express or

implied.

The information in this document may include technical inaccuracies or typographical errors.

All customer examples described are presented as illustrations of how those customers have used IBM

products and the results they may have achieved. Actual environmental costs and performance

characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published

announcement material, or other publicly available sources and does not constitute an endorsement of

such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly

available information, including vendor announcements and vendor worldwide homepages. IBM has not

tested these products and cannot confirm the accuracy of performance, capability, or any other claims

related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the

supplier of those products.

All statements regarding IBM future direction and intent are subject to change or withdrawal without

notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller

for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a

definitive statement of a commitment to specific levels of performance, function or delivery schedules with

respect to any future products. Such commitments are only made in IBM product announcements. The

information is presented here to communicate IBM's current investment and development activities as a

good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled

environment. The actual throughput or performance that any user will experience will vary depending

upon considerations such as the amount of multiprogramming in the user's job stream, the I/O

configuration, the storage configuration, and the workload processed. Therefore, no assurance can be

given that an individual user will achieve throughput or performance improvements equivalent to the ratios

stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.



Any references in this information to non-IBM Web sites are provided for convenience only and do not in

any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part

of the materials for this IBM product and use of those Web sites is at your own risk.

POW03038-USEN-01

active memory expansion

Documents