z13 capacity planning - epv technologies white papers/z13 capacity... · epv technologies z13...

EPV Technologies

z13 Capacity Planning 1

z13 Capacity Planning

Fabio Massimo Ottaviani – EPV Technologies

April 2015

1 Introduction On January 14th IBM announced its new generation of the mainframe. The new system is simply called IBM z13 while the family model is 2964. Experienced capacity planners know that every new generation of machines provides a major challenge to their skills. They also know that their best friends are the IBM LSPR benchmarks, the IBM zPCR tool, the Measurement Facility counters provided in SMF 113 and an up to date performance database. At the time of writing this paper the only thing available are the IBM LSPR benchmarks so in the first part of this paper, after a quick look at the most important capacity characteristics of the IBM z13, we will start from them to calculate the MIPS capacity of each IBM z13 processor model. We will also compare z13 single CP capacity and workload variability with previous machine generations.

EPV Technologies


2 z13 capacity highlights It’s interesting to note that, for some aspects, the z13 is more an evolution of the zBC12 than the zEC12. As the zBC12 it uses CPC drawers, instead of books, and single chip modules (SCM), instead of multi chip modules (MCM). The processor speed is slightly less than in the previous machine generations but thanks to the new processor architecture the single processor capacity has increased. See next chapter.

System z13 zEC12 z196 Type 2964 2827 2817

HW Models N30, N63, N96,

NC9, NE1 H20, H43, H66,

H88, HA1 M15, M32, M49,

M66, M80 Cycle rate (Ghz) 5,0 5,5 5,2

Max CP 141 101 80 Max LPARs 85 60 60

zAAP support N Y Y Subcapacity models 4xx, 5xx, 6xx 4xx, 5xx, 6xx 4xx, 5xx, 6xx

Entry MIPS 250 240 240 Max MIPS 111.556 78.426 52.286 Entry MSU 31 30 30 Max MSU 13.078 9.194 6.140

Entry Memory (GB) 64 32 32 Max Memory (GB) 10.000 3.000 3.000

Figure 1 In the table above we compare some of the most important capacity characteristics of z13, zEC12 and z196 machines. From the point of view of CP capacity the magic number is 40%. Compared to zEC12 the z13 provides:

About 40% more CPs; About 40% more total capacity; About 40% more LPARs.

Memory is a different story; the minimum size is 64 GB while the maximum size is 10 TB, versus a maximum of 3 TB available with zEC12. The message behind this is that memory is a critical factor in order to improve response time and reduce CPU consumption of modern applications. First rumours say that IBM will be very aggressive on memory pricing. As usual, subcapacity models (4xx, 5xx and 6xx) are available. The entry point is not very different from zEC12, about 250 MIPS (31 MSU). As expected zAAP are not supported anymore. Finally the announced SMT (Symmetric Multi-Thread) revolution has only partially started. IBM decided in fact to provide SMT for zIIP and IFL but not for the standard CPU.

EPV Technologies


The reason is that SMT will increase the overall throughput but it will introduce very big challenges from the point of view of single address space performance, variability, measurement and accounting. IBM wisely chose a gentle approach which will allow them and their customers to gain experience with these much less critical resources first.

EPV Technologies


3 IBM LSPR benchmarks and MIPS values On the same day as the announcement a new set of IBM LSPR benchmarks for z/OS 2.1 has been published on the web. These benchmarks are available for all the IBM machines including z13. However for all the machines, except z13, the published values are exactly the same as those already available in the z/OS 1.13 benchmark table. As usual benchmarks for three workload categories are provided: LOW RNI (Relative Nest Intensity): this category represents workloads lightly using the

memory nest (shared processor caches and memory). This would be similar to past high scaling primitives.

AVERAGE RNI (Relative Nest Intensity): this category represents workloads with an average use of the memory nest (shared processor caches and memory) hierarchy. This would be similar to the past LoIO-mix workload and is expected to represent the majority of production workloads.

HIGH RNI (Relative Nest Intensity): this category represents workloads heavily using the memory nest (shared processor caches and memory). This would be similar to the past DI-mix workload.

Benchmark values are the ITR ratio between the capacity of each processor model and the capacity of a reference processor model which, as in the z/OS 1.13 table, is the 2094-701. Starting from the published values, shown in the columns with a dark blue header in the Figure below, we calculated the capacity of each processor model, in MIPS, by multiplying the benchmark values by the suggested “capacity scaling factor”1. Only the starting and ending rows of the table are shown here. You can find the full list of z13 processor models in Appendix A. You can note that a PCI value is also provided. It is very close to the AVG MIPS capacity. The difference is due to the lack of precision, only 2 decimals, of the published benchmarks.

Processor #CP PCI MSU Low* Average* High* Low MIPS AVG MIPS High MIPS

2964-401 1 250 31 0,47 0,45 0,41 263,1 251,9 229,5

2964-402 2 478 60 0,92 0,85 0,75 515,0 475,8 419,8

2964-403 3 697 88 1,35 1,25 1,09 755,7 699,7 610,2

2964-404 4 910 114 1,78 1,63 1,42 996,4 912,5 794,9

2964-405 5 1118 140 2,20 2,00 1,74 1.231,5 1.119,6 974,0

2964-406 6 1321 165 2,61 2,36 2,05 1.461,1 1.321,1 1.147,6

………… ….. ……… …….. ……… ……… ……… ……… ……… ………

2964-7D6 136 108593 12731 251,21 193,99 161,99 140.625,3 108.594,1 90.680,7

2964-7D7 137 109188 12800 252,65 195,05 162,75 141.431,4 109.187,4 91.106,1

2964-7D8 138 109782 12870 254,08 196,11 163,49 142.232,0 109.780,8 91.520,4

2964-7D9 139 110375 12940 255,52 197,17 164,22 143.038,1 110.374,2 91.929,0

2964-7E0 140 110966 13009 256,95 198,23 164,95 143.838,6 110.967,6 92.337,7

2964-7E1 141 111556 13078 258,38 199,28 165,66 144.639,1 111.555,3 92.735,1

Figure 2

1 More information at https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex?OpenDocument

EPV Technologies


In the next figure you can find the MIPS to MSU ratio of every processor model of the last 3 mainframe generations. You can see that the shape of the graphs is very similar. So IBM definitely stopped using the machine MSU capacity to make discounts. The ratio is also very stable. It starts from 8,1 (8,0 for z196) for single CP processor models and rises up to become flat at 8.5 (from a capacity of about 26.000 MIPS).

Figure 3

-

1,0

2,0

3,0

4,0

5,0

6,0

7,0

8,0

9,0

2817

-704

2817

-710

2817

-716

2817

-722

2817

-728

2817

-734

2817

-740

2817

-746

2817

-752

2817

-758

2817

-764

2817

-770

2817

-776

2827

-705

2827

-711

2827

-717

2827

-723

2827

-729

2827

-735

2827

-741

2827

-747

2827

-753

2827

-759

2827

-765

2827

-771

2827

-777

2827

-783

2827

-789

2827

-795

2827

-7A1

2964

-703

2964

-709

2964

-715

2964

-721

2964

-727

2964

-733

2964

-739

2964

-745

2964

-751

2964

-757

2964

-763

2964

-769

2964

-775

2964

-781

2964

-787

2964

-793

2964

-799

2964

-7A5

2964

-7B1

2964

-7B7

2964

-7C3

2964

-7C9

2964

-7D5

2964

-7E1

MIPS/MSU

EPV Technologies


4 Single CP capacity The graph below shows the maximum capacity of the single CP processor model of each of the last five mainframe generations.

Figure 4

You can see that the relative single CP capacity improvement is continuing to decline:

from z9 to z10 it improved by 61% from z10 to z196 it improved by 33% from z196 to zEC12 it improved by 26% from zEC12 to z13 it improved by 12%.

So it really seems that Moore’s Law is approaching its limits for current commercial technologies. Capacity growth is therefore based on an increasing number of processors and, in the near future, on SMT exploitation.

560

902

1202

1514

1695

0

200

400

600

800

1.000

1.200

1.400

1.600

1.800

2094-701 2097-701 2817-701 2827-701 2964-701

MIPS

Uniprocessor Capacity Full speed modes

z9 z10 z196 zEC12 z13

EPV Technologies


5 Capacity variability The following graph shows a comparison between the last 5 mainframe generations of a variability index calculated using the workload MIPS capacity as follows:

Low RNI MIPS – High RNI MIPS ------------------------------------------

Average RNI MIPS

You can see that capacity variability grows very quickly with the number of processors then flattens. You can also note that, with the z10 exception, workload variability is increasing with every new generation.

Figure 5

Very big z13 processor models show a variability higher than 45%. It means that the difference in z13 capacity between LOW RNI and HIGH RNI is 45% the AVG RNI capacity. So if you consider a 2094-7C2 model (122 CP) the LOW RNI capacity is 129.161 MIPS, the AVG RNI capacity is 100.124 MIPS and the HIGH RNI capacity is 84.148 MIPS. The difference between LOW RNI and HIGH RNI is about 45.000 MIPS (about 45% of AVG RNI capacity).

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

2094

-701

2094

-711

2094

-721

2094

-731

2094

-741

2094

-751

2097

-704

2097

-714

2097

-724

2097

-734

2097

-744

2097

-754

2097

-764

2817

-707

2817

-717

2817

-727

2817

-737

2817

-747

2817

-757

2817

-767

2817

-777

2827

-704

2827

-714

2827

-724

2827

-734

2827

-744

2827

-754

2827

-764

2827

-774

2827

-784

2827

-794

2964

-710

2964

-720

2964

-730

2964

-740

2964

-750

2964

-760

2964

-770

2964

-780

2964

-790

2964

-7A0

2964

-7B0

2964

-7C0

2964

-7D0

2964

-7E0

Capacity variabilityby processor family

z9 z10 z196 zEC12 z13

EPV Technologies


Figure 6

The graph in Figure 6 shows the absolute capacity difference between LOW RNI and HIGH RNI workloads. The bottom line is that it is more and more important to correctly classify your system workloads in order to use the right benchmark in capacity planning studies. However to do that we need an update of the SMF 113 records to provide hardware measurement facility counters for the z13 machines.

0

10.000

20.000

30.000

40.000

50.000

60.000

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101

105

109

113

117

121

125

129

133

137

141

MIPS

Difference between LOW RNI and HIGH RNI capacityby processor model and mainframe generation

z196 zEC12 z13

EPV Technologies


6 IBM zPCR A new version (V8.7a) of the IBM zPCR free tool, supporting the z13 machines, is already available on the web. You can download it at: https://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS1381.

Figure 7

zPCR is a “must have” tool for capacity planners. With it you can estimate the capacity of a machine taking into consideration LPAR configuration, operative system level and workload characteristics. As you can see in Figure 7 the reference CPU is still a 2094-701 estimated at 593,00 MIPS. This is the base of zPCR capacity studies and also the base for the ratio or MIPS estimates provided in the LSPR Multi-Image Capacity table. It’s worth noting that the zPCR MIPS values are a bit more precise than the ones you can calculate starting from LSPR benchmarks and provided in the Appendix to the first part of this paper. A snapshot of the table is provided in Figure 8.

EPV Technologies


Figure 8

As you can see zPCR also provides Low-Avg and Avg-High values which are calculated as an harmonic mean of the Low, Average and High RNI benchmarks. They should be used when workload characteristics are on the border between Low and Average RNI or between Average and High RNI. To understand which benchmark best represents your system workload you need to collect the hardware measurement facility counters (recorded in SMF 113) and pass them as input to zPCR which will automatically select the appropriate LSPR benchmark2. A better solution is collecting SMF 113 in a tool, such as EPV for z/OS, and analysing system workload behaviour in multiple days and at different times of the day. Whatever method you choose, the benchmark to use depends on the number of misses in the Level 1 cache and on the RNI values of the system workload. Starting from these two values you can classify it by using the rules in Figure 9.

2 The z13 processor cache architecture and the hardware measurement facility counters will be discussed in the third part of this paper.

EPV Technologies


%L1 Miss RNI Benchmark< 3% >= 0,75 AVG < 3% < 0,75 Low

3% to 6% > 1,00 High 3% to 6% 0,60 to 1,00 AVG 3% to 6% < 0,60 Low

> 6 % >= 0,75 High > 6 % < 0,75 AVG

Figure 9

Beside z13 support, the biggest change introduced in this zPCR version is the possibility to take into account the effect of Simultaneous Multi-Threading (SMT) on zIIP engines.

Figure 10

A new option button to“Add SMT benefit to Capacity Results” is provided in the Partition Detail Report. When clicking it, a small box appears allowing you to choose if you want to apply a capacity increase due to SMT to zIIP, IFL or both. Default values are 25% for zIIP and 20% for IFL. Of course you can change them.

EPV Technologies


Figure 11

In Figure 12 you can see the result: zIIP capacity increased from 1.681 to 2.102 MIPS.

Figure 12

EPV Technologies


7 z13 Simultaneous Multi-Threading3 7.1 CPU or core?

Originally the CPUs (hardware chip) had a single central processing unit on it. So the term “CPU” was used to indicate both of them.

To increase performance, manufacturers started to increase the number of central processing units in a chip. They called them cores. A multi-core chip appears to the operating system (e.g. z/OS) as multiple processing units which can be used by different processes at the same time. This is what is relevant from a measurement and performance analysis perspective.

Mainframe machines have exploited multi-core chips for many years so we should be accustomed to the term “core”. In reality all mainframe commands, tools, manuals and people still use the term “CPU” to indicate a core.

Figure 13

In the figure above you can see the structure of the z13 PU Single Chip Module (from “IBM z13 Technical Guide”). Eight cores are hosted on the SCM.

3 Most of the content of this chapter has been inspired by “Simultaneous Multithreading and System z” written by Bob Rogers and published in number 3-2014 of Cheryl Watson’s TUNING Letter.

EPV Technologies


7.2 Advantages and issues of SMT Mainframe cores process instructions in multiple pipes composed of a number of stages each performing one step in the processing of an instruction, similar to an assembly line. However a traditional core can operate on a single instruction stream. A big part of the core capacity is normally wasted when an instruction stream gets stalled waiting for a cache miss to be resolved. To address this issue with z13 machines IBM decided to start exploiting Simultaneous Multi-Threading (SMT). By using SMT multiple instruction streams can be processed simultaneously so when a thread is waiting for a cache miss the core can continue doing work on behalf of the other threads. Unfortunately, the additional throughput from SMT does not scale very well with the number of threads. This is because all the threads on a core share some limited resources (e.g. pipes, processor cache, TLB). We saw in the previous chapter that the default expected increase of zIIP capacity when using SMT-2 (two threads) is only 25% in zPCR.4 As already mentioned IBM has been very cautious with SMT on z13: only SMT-2 can be used and only on zIIP and IFL. The reason of this approach is that, while SMT may generally increase the overall throughput, it introduces some important issues. a. Reduced speed; a thread in an SMT environment is slower than a thread using a dedicated core;

the main reason is the fact that the Level 1 and Level 2 caches are shared among the threads; the effect on the application is similar to running on more but slower engines; the more threads the stronger the effect.

b. Throughput variability; as discussed in the first part of this paper, variability has been increasing with each new mainframe model as the processor designs get ever more complex. With SMT that variability will increase much more because the throughput will also depend on the characteristics of the threads sharing the core. If all threads need the whole Level 1 cache, throughput could be even worse than running without SMT. On the other hand if all threads have a small Level 1 cache footprint the overall throughput could be up to 100% more (with SMT-2) than running without SMT.

c. zIIP measurements; all the zIIP measurements have to be reviewed. The current CPU timer

implementation accounts processor time both when using the processor and when waiting (normally for a Level 1 cache miss); using it with SMT, the time waiting for other threads will be accounted as processor time too. Even zIIP busy may become tricky: if we have only one zIIP core, only one thread is running at 100% busy and we use SMT-2 we could say that the overall zIIP busy is 50% because we have another thread to use. But if we assume that activating the second thread the maximum throughput increase we can get is about 25%, we should say that zIIP busy is 80% because by adding 20% (80%*25%) more work we will reach 100% busy5.

4 On P7 machines the average throughput increase with SMT-2 is about 40%; it will probably be about the same on the mainframe. 5 A solution to this issue has been implemented on P7 machines.

EPV Technologies


7.3 Settings and commands

To activate the SMT-2 function on z/OS, you have to: define the PROCVIEW CORE option in LOADxx; if you do not want to use SMT-2 you

can omit the PROCVIEW parameter or specify PROCVIEW CPU which is the default; set MT_ZIIP_MODE=2 in IEAOPTxx.

When you define PROCVIEW CORE, you cannot use the word CPU in z/OS commands. You must use CORE instead of CPU. If you want to continue to use CPU in z/OS commands, you have to define PROCVIEW CORE,CPU_OK. This parameter causes z/OS to treat CPU as an acceptable alias for CORE.

Figure 14

You can see that the output of D M=CORE is quite different from the output of D M=CPU6. For each CORE ID there is a range with two ids and each thread appears as a logical processor to z/OS when SMT-2 is used as you can see in the CPU column of CORE ID 0004 (online zIIP).

6 From “IBM z13 Configuration Setup”.

EPV Technologies


8 Processor cache architecture Generally speaking zEC12 and z13 architectures are very similar. If the data and instructions to be processed are found in the Level 1 cache (L1) dedicated to each processor7, this is called a “cache hit”. In this case the speed of the clock can be exploited well. If the data and instructions cannot be found in L1 then the hardware tries to load them from the Level 2 cache, which is still a cache dedicated to each processor, then from the the Level 3 (L3) cache which is a cache serving all the processors on the same chip, from Level 4 cache (L4) of the same book8, from the Level 4 cache (L4) of another book, from local memory or remote memory in this order. This is a “cache miss” and clock cycles are lost while waiting for data and instructions to be loaded into the L1 cache. The number of lost cycles depends on the cache level accessed, it can range from a few cycles for L2 to hundreds of cycles for memory.

Figure 15 shows a simplified view of the zEC12 processor cache architecture.

Figure 15

zEC12 uses books up to a maximum of 4, each book includes 6 chips and each chip includes 6 processors.

One of the limits of this architecture is that there is no direct communication between L3 caches. Data and instructions moving from one L3 to the other have to pass through the L4 cache which is the coherence manager. So all memory fetches must be in the L4 cache before that data can be used by the processor.

7 There are two L1 caches, one for data the other for instructions, dedicated to each processor but for simplicity only one cache is depicted in the figure. 8 L4 serves all the processors in a book.

L3 – 48MB

L2L1

+4 L3

L1 L1 L1 L1 L1L2 L2 L2 L2 L2

L3 – 48MB

L2L1 L1 L1 L1 L1 L1

L2 L2 L2 L2 L2

L4 – 384MB +3 books

EPV Technologies


In the z13 cache design, represented in Figure 169, some lines of the L3 cache are not included in the L4 cache. The L4 cache has a non-data inclusive coherent (NIC) directory that has entries pointing to the non-inclusive lines of L3 cache. This design ensures that L3 locally owned lines can be accessed by using the intra L3 node interface without being included in L4.

Figure 16 Another big improvement introduced is in the total amount of cache provided. The new z13 technology gives an increase in the size of cache levels (L1 and L2) without increasing access latency. This has a direct influence on productivity reducing the number of L1 and L2 misses and allowing better exploitation of the processor speed.

Up to 8 processors can be served by a L3 cache in z13 so, even though the L3 size increased, the average amount of MB per processor is the same.

Processor cache zEC12 z13 Level 1 (instructions) 64K 96K Level 1 (data) 96K 128K Level 2 (instructions) 1MB 2MB Level 2 (data) 1MB 2MB Level 3 (single chip) 48MB 64MB Level 3 (per CP) 8MB 8MB

Figure 17

Finally the size of the L4 cache has been hugely increased from about 1,5GB to about 5,5GB. So even though there is a larger number of processors to be supported, the average L4 cache per processor has been doubled in z13.

A bigger L4 cache has a key role in reducing the accesses to memory which is still much slower even than the L4 cache.

9 Only half of one CPC drawer node is represented.

L4 – 480MB

L3 – 64MB

L2L1

+1 L3

L1 L1 L1 L1 L1L2 L2 L2 L2 L2

L3 – 64MB

L2L1 L1 L1 L1 L1 L1

L2 L2 L2 L2 L2

+7 nodesNIC dir 224MB

Intra L3 node interface

L1 L1L2 L2

L1 L1L2 L2

+7 nodes

EPV Technologies


The bottom line is always the same: “Workload capacity performance will be quite sensitive to how deep into the memory hierarchy the processor must go to retrieve the workload’s instructions and data for execution. Best performance occurs when the instructions and data are found in the cache(s) nearest the processor so that little time is spent waiting prior to execution; as instructions and data must be retrieved from farther out in the hierarchy, the processor spends more time waiting for their arrival.10” The two main factors determining workload performance are:

Percentage of L1 misses over total searches; Percentage of L1 misses satisfied by each cache level (including memory).

In the next chapter we will show how to calculate them for z13 machines.

10 From IBM Large Systems Performance Reference

EPV Technologies


9 SMF 113 counters The CPU Measurement Facility (CPU MF), introduced with z10 machines, provides the ability to obtain measurements (counters) on processor cache effectiveness. Collected information are recorded in SMF 113 subtype 2.11 Starting with z/OS 2.1 SMF 113 subtype 1 is also written. IBM stated that subtype 2 will be frozen and all the new information will be added to subtype 1. They already started this process by adding two new counter sections:

z/OS counters MT counters

However at the moment the major advantage of subtype 1 versus subtype 2 is that it provides de-accumulated counters. All the counters and formulas discussed in this chapter apply to both subtypes. Most important groups of counters are:

basic counters; which should be used to calculate the percentage of L1 misses; extended counters; which should be used to calculate the percentage of L1 misses sourced

by each cache level and, starting from them, the RNI value. BASIC COUNTERS Six metrics are provided in the Basic Counters section:

B0, CYCLE COUNT B1, INSTRUCTION COUNT B2, L1 I-CACHE DIRECTORY-WRITE COUNT B3, L1 I-CACHE PENALTY CYCLE COUNT B4, L1 D-CACHE DIRECTORY-WRITE COUNT B5, L1 D-CACHE PENALTY CYCLE COUNT

The basic counters’ meaning is the same whatever the machine model is (z10, z196, z114, zBC12, zEC12 and z13). Starting from these measurements the percentage of L1 misses over total searches can be calculated for z13 machines by using the following formula:

z13 %L1 Miss = ((B2 + B4) / B1) * 100

z13 EXTENDED COUNTERS

The extended counters’ meaning depends on the machine model. The SMF113_2_CTRVN2 field allows us to identify the model: it is 1 for z10, 2 for z196 and z114, 3 for zEC12 and zBC12, 4 for z13. The number of Li misses sourced by each cache levels can be calculated as follows:

L2d, data sourced from L2 = E133;

11 They are also collected in a USS file written in the HIS started task HOME directory.

EPV Technologies


L2i, instructions sourced from L2 = E136; L3d, data sourced from L3 = E144 + E145; L3i, instructions sourced from L3 = E162 + E163; L4Ld, data sourced from L4 Local = E146 + E147 + E148; L4Li, instructions sourced from L4 Local = E164 + E165 + E166; L4Rd, data sourced from L4 Remote = E149 + E150 + E151 + E152 + E153 + E154 + E155

+ E156 + E157; L4Ri, instructions sourced from L4 Remote = E167+ E168 + E169 + E170 + E171 + E172 +

E173 + E174 + E175; MEMLd, data sourced from Local Memory = E158 + E161; MEMRd, data sourced from Remote Memory = E159 + E160; MEMLi, instructions sourced from Local Memory = E176 + E179. MEMRi, instructions sourced from Remote Memory = E177 + E178.

Starting from these measurements the percentage of L1 misses sourced by each cache level can be calculated by using the following formulas12:

%L2 = (L2d + L2i)/(B2 + B4) * 100 %L3 = (L3d + L3i)/(B2 + B4) * 100 %L4L = (L4Ld + L4Li)/(B2 + B4) * 100 %L4R = (L4Rd + L4Li)/(B2 + B4) * 100 %MEM = (MEMLd + MEMLi + MEMRd + MEMRd)/(B2 + B4) * 100

The following formula allows you to calculate the RNI of a system when running on a z13 machine, starting from the Extended Counters: z13 RNI = 2.6 x (0.4 x %L3 + 1.6 x %L4L + 3.5 x %L4R + 7.5 x %MEM) / 100 The coefficients (in bold) are used to weight cache and memory accesses so in the above formula:

accessing the chip cache (%L3) is weighted 0,4; accessing the local book cache (%L4L) is weighted 1,6; accessing a remote book cache (%L4R) is weighted 3,5: accessing memory (%MEM), including both local and remote book memory, is weighted

7,5; an additional coefficient (2,6) is used to adjust the resulting RNI value.

IBM always states that these coefficients may change in the future. However very small changes have been done to previous machines RNI formulas up to now. As already discussed, workload capacity performance is quite sensitive to how deep into the memory hierarchy the processor must go to retrieve the workload’s instructions and data to be executed. So the higher the RNI, the worse will be the workload capacity performance. In practical terms the machine will look less powerful to a workload presenting High RNI characteristics than to a workload presenting AVG RNI or Low RNI characteristics.

12 More details in “The CPU-Measurement Facility Extended Counters Definition for z10, z196/z114, zEC12/zBC12 and z13” manual (SA23-2261-03).

EPV Technologies


By using the %L1 Miss and RNI values together with the rules in Figure 9 - Chapter 6, you can understand which benchmark best represents the workload running in each system.

EPV Technologies


10 The CPI index The CPI index represents the average number of cycles needed per instruction. It can be calculated by using basic counters and the following simple formula:

CPI = B0 / B1 As you can imagine there is not a Rule of Thumb for the ideal CPI value. However it’s intuitive that to exploit the processor power the CPI value should be as low as possible. Measuring this index on a regular basis will allow you to evaluate the effect of changes in:

hardware configuration; microcode; exploitation of HiperDispatch; LPAR configuration such as weights, number of logical processors, number of LPARs, etc.; system and subsystem levels; workload mixture.

Using this knowledge you will be able, in case of performance degradation after a change, to quickly identify the problem and solve it. In the figure below the effect on the CPI values of moving two production systems (SYSA and SYSB) from z10 to zEC12 is shown. The graph refers to the weeks from March to May which are the systems peak period every year.

Figure 18

-

1,00

2,00

3,00

4,00

5,00

6,00

7,00

10 11 12 13 14 15 16 17 18 19 20 21 22

CPI

Week

CPI values in the peak weeksz10 vs zEC12

SYSA 2014 - zEC12

SYSB 2014 - zEC12

SYSA 2012 - z10

SYSB 2012 - z10

EPV Technologies


Generally speaking you should always expect a CPI reduction when moving to a new generation machine; we think it will apply to z13 too. You can also get a deeper understanding of CPI by splitting it into:

finite_CPI; cycles needed because L1 cache is not infinite; instruction_complexity_CPI; cycles needed even with an infinite L1 cache.

They can be estimated by using the following simple formulas:

finite_CPI13 = E143 / B1

instruction_complexity_CPI = CPI – finite_CPI

13 The E143 extended counter provides the number of cycles where a level-1 cache or level-1 TLB miss is in progress.

EPV Technologies


11 Conclusions z13 looks a very powerful machine; its cache processor architecture presents interesting improvements compared to zEC12. Single processor capacity is increasing only by 12% and capacity variability depending on the workload continues to increase. SMT will introduce some throughput improvement but also new challenges both for IBM and customers. More information is needed about CPU measurements which will radically change with SMT. The new SMF 113 subtype 1 record, available since z/OS 2.1, provides de-accumulated counters and introduces new counter sets. At the moment you can use any of the SMF 113 record subtypes to calculate all the indexes relevant for capacity planning activities, such as %L1 Miss, RNI and CPI. However IBM said they will freeze the “old” subtype 2 records so you should prepare to switch to the subtype 1 as soon as you will move to z/OS 2.1.

EPV Technologies


Appendix A – z13 MIPS table This table is provided “as is”. While EPV Technologies believes the information included in this table to be accurate, EPV Technologies cannot be held responsible for any consequential damages resulting from the application of information contained in this table.

Processor #CP PCI MSU Low* Average* High* Low MIPS AVG MIPS High MIPS

2964-401 1 250 31 0,47 0,45 0,41 263,1 251,9 229,5

2964-402 2 478 60 0,92 0,85 0,75 515,0 475,8 419,8

2964-403 3 697 88 1,35 1,25 1,09 755,7 699,7 610,2

2964-404 4 910 114 1,78 1,63 1,42 996,4 912,5 794,9

2964-405 5 1118 140 2,20 2,00 1,74 1.231,5 1.119,6 974,0

2964-406 6 1321 165 2,61 2,36 2,05 1.461,1 1.321,1 1.147,6

2964-407 7 1520 189 3,02 2,72 2,36 1.690,6 1.522,6 1.321,1

2964-408 8 1716 213 3,42 3,06 2,66 1.914,5 1.713,0 1.489,0

2964-409 9 1907 236 3,82 3,41 2,95 2.138,4 1.908,9 1.651,4

2964-410 10 2093 258 4,21 3,74 3,24 2.356,7 2.093,6 1.813,7

2964-411 11 2277 281 4,59 4,07 3,52 2.569,4 2.278,4 1.970,5

2964-412 12 2456 303 4,97 4,39 3,80 2.782,2 2.457,5 2.127,2

2964-413 13 2631 325 5,34 4,70 4,07 2.989,3 2.631,0 2.278,4

2964-414 14 2803 346 5,71 5,01 4,34 3.196,4 2.804,6 2.429,5

2964-415 15 2971 367 6,07 5,31 4,60 3.397,9 2.972,5 2.575,0

2964-416 16 3138 387 6,43 5,61 4,86 3.599,5 3.140,4 2.720,6

2964-417 17 3305 407 6,79 5,90 5,11 3.801,0 3.302,8 2.860,5

2964-418 18 3470 427 7,14 6,20 5,37 3.996,9 3.470,7 3.006,1

2964-419 19 3635 448 7,50 6,49 5,63 4.198,4 3.633,1 3.151,6

2964-420 20 3799 468 7,85 6,79 5,88 4.394,4 3.801,0 3.291,6

2964-421 21 3962 488 8,20 7,08 6,13 4.590,3 3.963,3 3.431,5

2964-422 22 4125 508 8,55 7,37 6,38 4.786,2 4.125,7 3.571,5

2964-423 23 4286 527 8,90 7,66 6,63 4.982,1 4.288,0 3.711,4

2964-424 24 4447 547 9,25 7,94 6,88 5.178,1 4.444,7 3.851,4

2964-425 25 4607 567 9,59 8,23 7,13 5.368,4 4.607,1 3.991,3

2964-426 26 4766 586 9,94 8,51 7,38 5.564,3 4.763,8 4.131,3

2964-427 27 4924 605 10,28 8,80 7,62 5.754,7 4.926,2 4.265,6

2964-428 28 5082 624 10,62 9,08 7,87 5.945,0 5.082,9 4.405,6

2964-429 29 5239 643 10,96 9,36 8,11 6.135,3 5.239,7 4.539,9

2964-430 30 5394 661 11,30 9,64 8,35 6.325,6 5.396,4 4.674,3

2964-501 1 746 94 1,40 1,33 1,21 783,7 744,5 677,3

2964-502 2 1417 177 2,72 2,53 2,24 1.522,6 1.416,3 1.253,9

2964-503 3 2067 255 4,01 3,69 3,23 2.244,8 2.065,6 1.808,1

2964-504 4 2694 333 5,27 4,81 4,19 2.950,1 2.692,6 2.345,5

2964-505 5 3306 407 6,52 5,91 5,12 3.649,8 3.308,4 2.866,1

2964-506 6 3903 481 7,74 6,97 6,04 4.332,8 3.901,8 3.381,1

2964-507 7 4485 552 8,95 8,01 6,93 5.010,1 4.483,9 3.879,4

2964-508 8 5052 620 10,13 9,03 7,80 5.670,7 5.054,9 4.366,4

2964-509 9 5606 687 11,30 10,01 8,64 6.325,6 5.603,5 4.836,6

2964-510 10 6145 753 12,46 10,98 9,47 6.975,0 6.146,5 5.301,2

2964-511 11 6671 818 13,59 11,92 10,28 7.607,6 6.672,7 5.754,7

2964-512 12 7183 880 14,71 12,83 11,07 8.234,5 7.182,1 6.196,9

2964-513 13 7683 942 15,81 13,73 11,84 8.850,3 7.685,9 6.627,9

EPV Technologies


2964-514 14 8171 1000 16,89 14,60 12,59 9.454,9 8.173,0 7.047,8

2964-515 15 8646 1056 17,96 15,44 13,32 10.053,9 8.643,2 7.456,4

2964-516 16 9119 1112 19,03 16,29 14,05 10.652,8 9.119,0 7.865,1

2964-517 17 9589 1168 20,09 17,13 14,78 11.246,2 9.589,2 8.273,7

2964-518 18 10057 1223 21,14 17,97 15,50 11.834,0 10.059,5 8.676,8

2964-519 19 10523 1279 22,19 18,80 16,22 12.421,8 10.524,1 9.079,8

2964-520 20 10986 1333 23,23 19,63 16,94 13.004,0 10.988,7 9.482,9

2964-521 21 11447 1388 24,27 20,45 17,65 13.586,2 11.447,7 9.880,3

2964-522 22 11906 1442 25,31 21,27 18,35 14.168,3 11.906,8 10.272,2

2964-523 23 12363 1495 26,34 22,08 19,06 14.744,9 12.360,2 10.669,6

2964-524 24 12817 1550 27,36 22,90 19,76 15.315,9 12.819,2 11.061,5

2964-525 25 13269 1603 28,39 23,70 20,46 15.892,5 13.267,1 11.453,3

2964-526 26 13719 1656 29,40 24,51 21,15 16.457,9 13.720,5 11.839,6

2964-527 27 14166 1709 30,41 25,31 21,84 17.023,3 14.168,3 12.225,9

2964-528 28 14611 1762 31,42 26,10 22,53 17.588,7 14.610,6 12.612,1

2964-529 29 15054 1815 32,43 26,89 23,21 18.154,1 15.052,8 12.992,8

2964-530 30 15495 1867 33,42 27,68 23,89 18.708,2 15.495,0 13.373,4

2964-601 1 1068 134 2,00 1,91 1,73 1.119,6 1.069,2 968,4

2964-602 2 2019 249 3,91 3,61 3,19 2.188,8 2.020,8 1.785,7

2964-603 3 2938 363 5,76 5,25 4,60 3.224,4 2.938,9 2.575,0

2964-604 4 3827 471 7,56 6,84 5,96 4.232,0 3.829,0 3.336,4

2964-605 5 4690 577 9,33 8,38 7,28 5.222,9 4.691,1 4.075,3

2964-606 6 5528 678 11,07 9,88 8,56 6.196,9 5.530,7 4.791,8

2964-607 7 6342 777 12,77 11,33 9,81 7.148,5 6.342,4 5.491,6

2964-608 8 7132 874 14,45 12,74 11,01 8.089,0 7.131,8 6.163,3

2964-609 9 7899 968 16,09 14,11 12,18 9.007,1 7.898,7 6.818,3

2964-610 10 8644 1056 17,69 15,44 13,32 9.902,7 8.643,2 7.456,4

2964-611 11 9368 1141 19,27 16,73 14,42 10.787,2 9.365,3 8.072,2

2964-612 12 10070 1224 20,82 17,99 15,48 11.654,9 10.070,7 8.665,6

2964-613 13 10752 1305 22,34 19,21 16,52 12.505,8 10.753,6 9.247,8

2964-614 14 11414 1384 23,83 20,39 17,53 13.339,8 11.414,2 9.813,2

2964-615 15 12057 1460 25,29 21,54 18,50 14.157,1 12.057,9 10.356,2

2964-616 16 12695 1535 26,74 22,68 19,46 14.968,8 12.696,1 10.893,6

2964-617 17 13328 1610 28,19 23,81 20,42 15.780,5 13.328,6 11.431,0

2964-618 18 13956 1684 29,62 24,93 21,37 16.581,0 13.955,6 11.962,8

2964-619 19 14579 1759 31,05 26,04 22,30 17.381,5 14.577,0 12.483,4

2964-620 20 15196 1831 32,47 27,15 23,23 18.176,4 15.198,4 13.004,0

2964-621 21 15809 1904 33,88 28,24 24,15 18.965,8 15.808,5 13.519,0

2964-622 22 16417 1975 35,28 29,33 25,05 19.749,5 16.418,7 14.022,8

2964-623 23 17020 2046 36,67 30,40 25,95 20.527,6 17.017,7 14.526,6

2964-624 24 17618 2116 38,06 31,47 26,84 21.305,7 17.616,7 15.024,8

2964-625 25 18212 2186 39,43 32,53 27,72 22.072,6 18.210,0 15.517,4

2964-626 26 18801 2255 40,80 33,59 28,60 22.839,5 18.803,4 16.010,1

2964-627 27 19385 2324 42,16 34,63 29,46 23.600,8 19.385,6 16.491,5

2964-628 28 19964 2394 43,51 35,66 30,32 24.356,5 19.962,2 16.972,9

2964-629 29 20539 2463 44,85 36,69 31,16 25.106,7 20.538,8 17.443,1

2964-630 30 21109 2531 46,19 37,71 32,00 25.856,8 21.109,8 17.913,3

2964-701 1 1695 210 3,18 3,03 2,75 1.780,1 1.696,2 1.539,4

2964-702 2 3196 394 6,17 5,71 5,06 3.453,9 3.196,4 2.832,5

2964-703 3 4644 571 9,08 8,30 7,28 5.082,9 4.646,3 4.075,3

2964-704 4 6041 740 11,93 10,79 9,40 6.678,3 6.040,2 5.262,0

EPV Technologies


2964-705 5 7392 905 14,72 13,21 11,45 8.240,1 7.394,9 6.409,6

2964-706 6 8700 1062 17,44 15,54 13,43 9.762,8 8.699,2 7.518,0

2964-707 7 9964 1212 20,11 17,80 15,34 11.257,4 9.964,3 8.587,2

2964-708 8 11188 1356 22,73 19,99 17,18 12.724,1 11.190,2 9.617,2

2964-709 9 12371 1496 25,29 22,10 18,96 14.157,1 12.371,4 10.613,7

2964-710 10 13515 1632 27,80 24,14 20,68 15.562,2 13.513,4 11.576,5

2964-711 11 14622 1764 30,25 26,12 22,33 16.933,7 14.621,8 12.500,2

2964-712 12 15693 1891 32,65 28,03 23,93 18.277,2 15.691,0 13.395,8

2964-713 13 16729 2011 35,00 29,88 25,47 19.592,7 16.726,6 14.257,9

2964-714 14 17731 2129 37,30 31,67 26,96 20.880,2 17.728,6 15.092,0

2964-715 15 18700 2244 39,56 33,41 28,39 22.145,4 18.702,7 15.892,5

2964-716 16 19665 2358 41,80 35,13 29,82 23.399,3 19.665,5 16.693,0

2964-717 17 20624 2472 44,03 36,84 31,24 24.647,6 20.622,7 17.487,9

2964-718 18 21579 2584 46,26 38,55 32,66 25.896,0 21.580,0 18.282,8

2964-719 19 22529 2695 48,47 40,25 34,07 27.133,1 22.531,6 19.072,1

2964-720 20 23475 2801 50,68 41,93 35,47 28.370,3 23.472,1 19.855,8

2964-721 21 24415 2905 52,87 43,61 36,86 29.596,2 24.412,5 20.633,9

2964-722 22 25351 3009 55,05 45,29 38,25 30.816,5 25.353,0 21.412,0

2964-723 23 26282 3111 57,23 46,95 39,63 32.036,9 26.282,2 22.184,6

2964-724 24 27209 3212 59,39 48,60 41,00 33.246,0 27.205,9 22.951,5

2964-725 25 28130 3313 61,54 50,25 42,36 34.449,6 28.129,5 23.712,8

2964-726 26 29048 3414 63,69 51,89 43,72 35.653,2 29.047,6 24.474,1

2964-727 27 29960 3516 65,82 53,52 45,07 36.845,5 29.960,1 25.229,8

2964-728 28 30868 3619 67,95 55,14 46,42 38.037,9 30.866,9 25.985,5

2964-729 29 31772 3725 70,07 56,76 47,76 39.224,6 31.773,8 26.735,7

2964-730 30 32671 3830 72,17 58,36 49,09 40.400,2 32.669,5 27.480,2

2964-731 31 33566 3935 74,27 59,96 50,42 41.575,8 33.565,1 28.224,7

2964-732 32 34456 4040 76,35 61,55 51,74 42.740,1 34.455,2 28.963,6

2964-733 33 35341 4143 78,43 63,13 53,05 43.904,5 35.339,7 29.697,0

2964-734 34 36223 4247 80,49 64,71 54,36 45.057,7 36.224,1 30.430,3

2964-735 35 37099 4349 82,55 66,27 55,67 46.210,8 37.097,4 31.163,6

2964-736 36 37972 4452 84,59 67,83 56,97 47.352,8 37.970,7 31.891,4

2964-737 37 38840 4553 86,62 69,38 58,26 48.489,2 38.838,4 32.613,5

2964-738 38 39703 4655 88,65 70,93 59,55 49.625,6 39.706,0 33.335,6

2964-739 39 40563 4755 90,66 72,46 60,84 50.750,7 40.562,5 34.057,7

2964-740 40 41418 4855 92,66 73,99 62,12 51.870,3 41.419,0 34.774,3

2964-741 41 42269 4955 94,66 75,51 63,39 52.989,9 42.269,9 35.485,2

2964-742 42 43115 5055 96,64 77,02 64,66 54.098,3 43.115,2 36.196,2

2964-743 43 43958 5153 98,61 78,52 65,93 55.201,1 43.954,9 36.907,1

2964-744 44 44796 5252 100,58 80,02 67,19 56.303,9 44.794,6 37.612,4

2964-745 45 45630 5349 102,53 81,51 68,44 57.395,5 45.628,6 38.312,2

2964-746 46 46459 5447 104,48 82,99 69,69 58.487,1 46.457,1 39.011,9

2964-747 47 47285 5543 106,41 84,47 70,93 59.567,5 47.285,6 39.706,0

2964-748 48 48107 5641 108,33 85,94 72,17 60.642,3 48.108,5 40.400,2

2964-749 49 48924 5737 110,25 87,40 73,41 61.717,1 48.925,8 41.094,3

2964-750 50 49737 5833 112,15 88,85 74,64 62.780,7 49.737,5 41.782,9

2964-751 51 50546 5927 114,05 90,30 75,86 63.844,3 50.549,2 42.465,8

2964-752 52 51352 6022 115,94 91,73 77,08 64.902,3 51.349,7 43.148,8

2964-753 53 52153 6116 117,81 93,16 78,29 65.949,1 52.150,2 43.826,1

2964-754 54 52950 6209 119,68 94,59 79,50 66.995,9 52.950,7 44.503,5

2964-755 55 53743 6300 121,54 96,01 80,71 68.037,1 53.745,6 45.180,8

EPV Technologies


2964-756 56 54532 6393 123,39 97,42 81,91 69.072,7 54.534,9 45.852,6

2964-757 57 55318 6485 125,23 98,82 83,10 70.102,8 55.318,6 46.518,7

2964-758 58 56099 6577 127,06 100,21 84,29 71.127,2 56.096,8 47.184,9

2964-759 59 56876 6668 128,88 101,60 85,48 72.146,0 56.874,9 47.851,0

2964-760 60 57650 6758 130,69 102,98 86,66 73.159,2 57.647,4 48.511,6

2964-761 61 58419 6849 132,50 104,36 87,84 74.172,4 58.419,9 49.172,1

2964-762 62 59185 6938 134,29 105,73 89,01 75.174,5 59.186,8 49.827,1

2964-763 63 59947 7028 136,07 107,09 90,17 76.170,9 59.948,1 50.476,4

2964-764 64 60706 7117 137,85 108,44 91,34 77.167,3 60.703,8 51.131,4

2964-765 65 61462 7205 139,62 109,79 92,49 78.158,2 61.459,6 51.775,2

2964-766 66 62215 7294 141,39 111,14 93,64 79.149,0 62.215,3 52.418,9

2964-767 67 62965 7382 143,14 112,48 94,79 80.128,6 62.965,4 53.062,7

2964-768 68 63712 7469 144,89 113,81 95,93 81.108,3 63.709,9 53.700,8

2964-769 69 64456 7556 146,64 115,14 97,07 82.087,9 64.454,5 54.339,0

2964-770 70 65196 7643 148,37 116,47 98,20 83.056,3 65.199,0 54.971,6

2964-771 71 65934 7730 150,10 117,78 99,32 84.024,8 65.932,3 55.598,5

2964-772 72 66669 7816 151,82 119,10 100,44 84.987,6 66.671,2 56.225,5

2964-773 73 67401 7902 153,54 120,40 101,56 85.950,5 67.399,0 56.852,5

2964-774 74 68130 7987 155,25 121,71 102,67 86.907,7 68.132,3 57.473,8

2964-775 75 68857 8072 156,95 123,00 103,77 87.859,4 68.854,4 58.089,6

2964-776 76 69580 8157 158,64 124,30 104,87 88.805,4 69.582,1 58.705,4

2964-777 77 70300 8242 160,33 125,58 105,97 89.751,5 70.298,7 59.321,2

2964-778 78 71018 8326 162,01 126,86 107,06 90.691,9 71.015,2 59.931,3

2964-779 79 71733 8409 163,69 128,14 108,15 91.632,4 71.731,7 60.541,5

2964-780 80 72444 8493 165,35 129,41 109,23 92.561,6 72.442,7 61.146,1

2964-781 81 73153 8576 167,01 130,68 110,30 93.490,9 73.153,6 61.745,1

2964-782 82 73859 8659 168,67 131,94 111,37 94.420,1 73.859,0 62.344,0

2964-783 83 74563 8741 170,32 133,20 112,44 95.343,8 74.564,3 62.943,0

2964-784 84 75263 8823 171,96 134,45 113,50 96.261,8 75.264,0 63.536,4

2964-785 85 75961 8905 173,59 135,69 114,56 97.174,3 75.958,2 64.129,8

2964-786 86 76656 8987 175,22 136,94 115,61 98.086,8 76.657,9 64.717,6

2964-787 87 77348 9068 176,84 138,17 116,66 98.993,6 77.346,5 65.305,3

2964-788 88 78037 9149 178,46 139,40 117,70 99.900,5 78.035,0 65.887,5

2964-789 89 78724 9229 180,06 140,63 118,74 100.796,1 78.723,5 66.469,7

2964-790 90 79408 9309 181,67 141,85 119,77 101.697,4 79.406,5 67.046,3

2964-791 91 80089 9389 183,26 143,07 120,80 102.587,5 80.089,4 67.622,9

2964-792 92 80767 9469 184,85 144,28 121,82 103.477,6 80.766,8 68.193,9

2964-793 93 81443 9548 186,43 145,49 122,84 104.362,0 81.444,1 68.764,8

2964-794 94 82116 9627 188,01 146,69 123,85 105.246,5 82.115,9 69.330,2

2964-795 95 82786 9705 189,58 147,89 124,86 106.125,4 82.787,6 69.895,6

2964-796 96 83453 9783 191,14 149,08 125,87 106.998,6 83.453,8 70.461,0

2964-797 97 84119 9862 192,71 150,27 126,87 107.877,5 84.119,9 71.020,8

2964-798 98 84783 9939 194,26 151,45 127,87 108.745,2 84.780,5 71.580,6

2964-799 99 85444 10017 195,82 152,64 128,86 109.618,5 85.446,7 72.134,8

2964-7A0 100 86104 10094 197,37 153,81 129,84 110.486,1 86.101,6 72.683,4

2964-7A1 101 86761 10171 198,92 154,99 130,82 111.353,8 86.762,2 73.232,0

2964-7A2 102 87417 10248 200,46 156,16 131,80 112.215,9 87.417,1 73.780,6

2964-7A3 103 88071 10325 202,01 157,33 132,77 113.083,6 88.072,1 74.323,6

2964-7A4 104 88722 10401 203,54 158,49 133,74 113.940,1 88.721,4 74.866,6

2964-7A5 105 89372 10477 205,08 159,65 134,70 114.802,1 89.370,8 75.404,0

2964-7A6 106 90020 10553 206,61 160,81 135,65 115.658,6 90.020,2 75.935,8

EPV Technologies


2964-7A7 107 90666 10629 208,14 161,96 136,60 116.515,1 90.663,9 76.467,6

2964-7A8 108 91310 10705 209,67 163,11 137,55 117.371,6 91.307,7 76.999,4

2964-7A9 109 91952 10780 211,19 164,26 138,49 118.222,5 91.951,4 77.525,6

2964-7B0 110 92592 10855 212,72 165,40 139,43 119.079,0 92.589,6 78.051,8

2964-7B1 111 93230 10930 214,23 166,54 140,36 119.924,2 93.227,8 78.572,4

2964-7B2 112 93866 11004 215,75 167,68 141,29 120.775,1 93.865,9 79.093,0

2964-7B3 113 94501 11079 217,26 168,81 142,21 121.620,4 94.498,5 79.608,0

2964-7B4 114 95133 11153 218,77 169,94 143,13 122.465,7 95.131,1 80.123,0

2964-7B5 115 95764 11227 220,27 171,07 144,05 123.305,4 95.763,6 80.638,0

2964-7B6 116 96392 11300 221,78 172,19 144,96 124.150,7 96.390,6 81.147,4

2964-7B7 117 97019 11374 223,28 173,31 145,86 124.990,4 97.017,6 81.651,3

2964-7B8 118 97644 11447 224,77 174,43 146,76 125.824,4 97.644,5 82.155,1

2964-7B9 119 98267 11520 226,27 175,54 147,66 126.664,1 98.265,9 82.658,9

2964-7C0 120 98888 11593 227,76 176,65 148,55 127.498,2 98.887,3 83.157,1

2964-7C1 121 99507 11666 229,24 177,76 149,44 128.326,7 99.508,6 83.655,3

2964-7C2 122 100125 11738 230,73 178,86 150,32 129.160,8 100.124,4 84.147,9

2964-7C3 123 100740 11810 232,21 179,96 151,20 129.989,3 100.740,2 84.640,6

2964-7C4 124 101354 11882 233,69 181,06 152,07 130.817,8 101.355,9 85.127,6

2964-7C5 125 101966 11954 235,17 182,15 152,94 131.646,3 101.966,1 85.614,6

2964-7C6 126 102576 12025 236,64 183,24 153,81 132.469,2 102.576,3 86.101,6

2964-7C7 127 103184 12097 238,11 184,33 154,67 133.292,1 103.186,5 86.583,0

2964-7C8 128 103791 12168 239,57 185,41 155,52 134.109,4 103.791,0 87.058,9

2964-7C9 129 104395 12239 241,04 186,49 156,37 134.932,3 104.395,6 87.534,7

2964-7D0 130 104998 12309 242,50 187,57 157,21 135.749,6 105.000,2 88.004,9

2964-7D1 131 105601 12380 243,96 188,64 158,04 136.566,9 105.599,2 88.469,5

2964-7D2 132 106202 12450 245,41 189,72 158,86 137.378,6 106.203,7 88.928,6

2964-7D3 133 106801 12521 246,87 190,79 159,66 138.195,9 106.802,7 89.376,4

2964-7D4 134 107400 12591 248,32 191,86 160,45 139.007,5 107.401,7 89.818,6

2964-7D5 135 107997 12661 249,76 192,92 161,23 139.813,6 107.995,1 90.255,3

2964-7D6 136 108593 12731 251,21 193,99 161,99 140.625,3 108.594,1 90.680,7

2964-7D7 137 109188 12800 252,65 195,05 162,75 141.431,4 109.187,4 91.106,1

2964-7D8 138 109782 12870 254,08 196,11 163,49 142.232,0 109.780,8 91.520,4

2964-7D9 139 110375 12940 255,52 197,17 164,22 143.038,1 110.374,2 91.929,0

2964-7E0 140 110966 13009 256,95 198,23 164,95 143.838,6 110.967,6 92.337,7

2964-7E1 141 111556 13078 258,38 199,28 165,66 144.639,1 111.555,3 92.735,1

z13 capacity planning - epv technologies white papers/z13 capacity... · epv technologies z13...

Documents