ip platforms best practices for performance 010810

8/3/2019 IP Platforms Best Practices for Performance 010810

1/45

2010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone

IP Platforms BestPractices for Performance

Pierre Lamy

Technical Lead Ottawa TACApril 2010


2/45

2


Intro and revision history

This document describes methods and

techniques that users can apply on variousCheck Point IP Security Appliances to achieveoptimal performance.

Version 1.0 October 2009 Word format Version 1.1 January 2010 Word + PPT, minor

revisions

Version 1.2 April 2010 Updates


3/45

3


General Performance Best Practices

These guidelines are Appliance independent and do not require any specialtuning.

Always use the latest versions of Check Point products. Always upgrade to themost recent HFA (HotFix Accumulator) for a given version.

Create a small block of rules near the top of your rulebase, containing themost heavily used rules. These rules should be fully accelerated withSecureXL.

Keep the rulebase simple and small. Reduce the number of rules bycombining similar rules together. Rules which disable SecureXL accelerationshould be placed very low in the rulebase.

If not using VPN (encryption) on the module, make sure the VPN product is

disabled on that module.


4/45

4



Do not use QoS from IPSO or Floodgate.

Avoid using Domain Objects. DNS lookup takes additional CPU cycles.

Avoid using UFP URI Filtering Protocol as this is resource intensive.

Use Networks instead of address ranges for Network Address Translation.

Keep logging to a minimum. Only business critical rules which will beanalyzed should have logging enabled. Drop rules, Accept rules, Stealth rules,

Cleanup rules and Implied rules should not log unless there is a clear businesscase and the customer intends on analyzing the logs on a regular basis.Otherwise logging should only be used for debug purposes.

IP Cluster members should have exactly the same package lists, havingdissimilar packages can cause state sync issues resulting in a performancereduction.


5/45

5



When you install an ADP interface module in an appliance, the networkprocessor in the card performs all VPN encryption and decryption, even for

VPN packets that ingress or egress through non-ADP interfaces. The built-inNokia encryption accelerator continues to accelerate IKE traffic but does notperform any other processing.

If VPN traffic ingresses or egresses through a non-ADP interface, throughputis negatively affected because the packets must transit the appliancesbackplane to reach the network processor in the ADP module. It isrecommended that one configure VPNs to use only ADP interfaces to avoidthis performance loss.


6/45

6



Uniprocessor systems (IP152, IP292, IP395, IP565, IP1265) shoulduse IPSO 4.2 (this is correct in April 2010) while Multiprocessorsystems (IP695, IP1285, IP2455) should use IPSO 6.2. The latestbuild of any major release should always be used.

Multiprocessor systems should have Check Point R70 installed to take

full advantage of CoreXL technology. sk40465 has some more detailsabout this.

Use R70 + IPSO 6.2 on uniprocessor systems where there is a needfor a specific feature

Do not persist in the use of IPSO 4.2 or R65 once support is no longeroffered for these products.


7/45772010 Check Point Software Technologies Ltd. | [Unrestricted] For everyone


Using interface flow control can reduce network

throughput on busy interfaces and we do not suggest it beenabled.

Avoid using SmartView Monitor to constantly monitor

system performance or collect historical data, asSmartView Monitor itself has an impact on performance.

Avoid using custom scripts on systems which have

performance issues, as the scripts will incur CPUresources.




A CST from the IPSO system as well as a cpinfo from themanagement station, are CRITICAL to provide to Check Point Supportwhen opening a case for assistance in troubleshooting performanceissues. Without at least those files, Check Point Support will be unableto assist the customer. For systems with extremely high CPU, orwhere running CST may cause problems, it is recommended to run itwith the following syntax: nice +20 cst. The command may take

hours to complete but will not divert critical system resources fromprocessing traffic in a live environment.

Any cpinfo that is provided to Check Point Support MUST begenerated using the latest cpinfo tool downloaded from the Support

site. This requires uninstalling the old cpinfo, and then installing thelatest one. Providing old cpinfo output to Check Point Support willdelay Support response.




Avoid using Standalone installations. Separate the management station fromthe enforcement point by running the management station on another system.

Use the default settings in the capacity optimization tab of the enforcementpoints properties, changing only the total connections number.

General recommendations for these platforms are to use onboard quad portsfor:

Security Gateway State Synchronization traffic Cluster protocol network traffic Policy and Appliance management traffic A path from the enforcement point to the Check Point Log server

SecureXL options should be matched between SXL and the Security Gatewaysettings; for example Sequence Validation and Delayed Notifications.



System Resources

High performance relies on the availability ofkey system resources: CPU, memory, andnetwork interface bandwidth.

Tuning involves better using the currenthardware, not simply upgrading.



System Resources - CPU

Small packet size traffic: The amount of traffic any network device can processis not determined by byte throughput numbers, but rather by the packets per

second. A small packet uses as many resources as a large full size ethernetpacket.

CPU utilization is incurred on a per-packet level, rather than per-byte.Therefore it is critical to note that a system that is processing a large numberof small packets, works as hard as a system processing the same number of

large packets. There may be a very large difference in the apparent bytethroughput between the systems.

State synchronization particularly demands high CPU cycles because thenodes in a cluster perform synchronization for every connection theyencounter. This ensures high availability but causes high CPU usage. This is

especially important to consider, when deciding whether or not to synchronizeshort lived connection types like DNS when using a VRRP pair the failovertime of VRRP exceeds the DNS timeout. It would not be advisable to syncDNS connections in that scenario.




High number of logging rules affects CPU. Logging uses CPUcycles and is discouraged where there is no need.

The Active Log feature in SmartView Tracker will severelycompromise the ability of an enforcement point to processtraffic, and for performance reasons should not be used.

Accounting Logging: By default, accounting logging producestwo kinds of log tracking (one in Log Viewer and one in AccountLog View) for the same connection. Alerts as a tracking option,also have a significant negative impact on performance.




Any configuration which disables SecureXL or forces traffic touse Slowpath affects CPU. This includes rule configurations,SmartDefense protections, and Floodgate.

NAT traffic incurs slightly more CPU impact that non-NATedtraffic. NAT in an ADP environment is strongly discouraged asconnection establishment rate acceleration does not work onNAT traffic.

Traffic which is not connection-rate accelerated, uses moreCPU resources than traffic which is. Connection establishment

and teardown uses (relatively) a lot of CPU resources, evenwhen all SecureXL acceleration is in use.



System Resources Memory

Check Point recommends upgrading any Appliances memory to its full capacity toimprove performance.

The main factors that demand high RAM usage are: Concurrent connections Concurrent VPN tunnels NAT connections Security Servers

Use the Web User Interface to determine how much memory the Appliance hasinstalled. In the Web User Interface navigation tree, select Monitor --> System Utilization--> CPU-Memory Live Utilization. Look for the Total Real Memory value. The topcommand can also be used on recent IPSO versions to view the information on thecommand line.

The amount of memory, allocated to the Check Point Security Gateway to processnetwork traffic, is determined under the capacity optimization tab of the gateway object

properties. The Automatic values should always be used, and the manual setting ofmemory allocation should NEVER be used unless directed to by Check Point Support.

You should always set the capacity optimization to the maximum values (connections)supported by the platform and memory configuration as detailed in the release notes forIPSO. There is no drawback to doing this, but leaving the low defaults in place willresults in insufficient memory on busy systems.



System Resources Memory

Two factors can reduce the number of Security Gatewayconnections that can be supported

Concurrent VPN tunnels are dependent upon the amount ofmemory available in the Appliance. As you add more VPNtunnels, the number of Security Gateway connections anAppliance can support will decrease.

Security Servers will reduce the maximum number ofsupported connections as they write to temporary files anduse 8 entries in the connections table.



System Resources - Network InterfaceBandwidth

When the Security Gateway performance reaches the limitation of theinterface bandwidth and the CPU is still not fully utilized then the bottleneck isthe interface. One option to increase performance in this case is to use moreports via Link Aggregation to achieve the maximum performance.

The limitation of the network interface is determined by the amount of packetsper second it can process. Assuming 1518 byte frames, a 100megabit NICport can sustain ~8234 packets per second (pps ) in each direction (fullduplex). A 1gb NIC port can sustain ~82340 pps in each direction (full duplex).

While the network port may sustain much higher numbers of pps than these and this is often seen in the field and QA there is no guarantee that it WILLsupport more than these standard pps numbers for a given link speed.



System Resources - Network InterfaceBandwidth

If the number of packets per second exceeds those numbers, using Link Aggregation tocombine 2 or more links together is possible, thus increasing the amount of bandwidth inpps that can be processed on that logical network interface.

IPSO Sync should not need more than a 1gb interface, and there can be problems whenIPSO Sync is run over a Link Aggregation group. The speed of the IPSO and CheckPoint sync interface should be as fast as the fastest NIC port on the system. In a 10gbenvironment, typically 1gb for sync is sufficient. State sync should always be on anisolated VLAN or network segment. Note that it is NOT supported to run Check Pointstate sync over any interface VLAN other than 0 / 1 / untagged. Any VLANing done onthe switch access port is fine, but trunking is not supported as Check Point state sync isnot designed to support the extra frame sizes.

Note: Security Gateway state sync link aggregated interfaces should be directlyconnected using cross-over cables, unless you are using 3 or more cluster members.IPSO and Check Point state sync should always be on an isolated VLAN or networksegment.


18/45


IP Clustering

IP Clustering provides both high availability and scalability. IPClustering is useful when the performance of one system aloneis insufficient to provide the desired level of performance. Forexample, when an Appliance CPU reaches ~30%, it would berecommended to add another Appliance to form a two-membercluster that can scale the Security Gateway performance. This

is a capacity planning exercise that Check Point Salesengineers can help with. The 30% number is considered anindustry standard measurement or indicator that suggests morecapacity should be added.

IP Clustering is especially beneficial when using SmartDefensefeatures. With all SmartDefense features enabled, a two-member clusters HTTP transaction rate is about 40% higherthan a Standalone Appliance.


19/45


IP Clustering

Use dedicated interfaces for cluster protocol networks and statesynchronization; do not share interfaces with the production traffic.

It is strongly recommended to use separate interfaces for clusterprotocol network and Security Gateway synchronization traffic so thatthey are separate Broadcast domains.

Use a bandwidth of at least 100 Mbps full duplex for IPSO sync

interface(s). 1gb is recommended.

Use switches, not hubs, and never use crossover cables for IPClustering protocol networks.


20/45


IP Clustering

Do not use IP Clustering Forwarding mode, when performanceis a concern. Unicast and Multicast provide better performanceand less latency. Forwarding mode is a fallback mode, for whenfeature-poor network switches are in use.

If IGMP snooping is in use on the switch, disable it or configurestatic CAMs in order to allow Multicast traffic on specific ports.

Use dynamic cluster work assignment for optimum loadbalancing. This allows the cluster to periodically rebalance theload by moving active connections between nodes.

Use delayed synchronization if your system processes manyshort-lived connections, you are in VRRP or Standalone, andSXL templates are in use. A 30 second delay in synchronizingconnections can boost the performance by about 20%. If youuse Check Point delayed notifications, you must also enableSecureXL delayed notifications.


21/45


ADP

Addition of ADP will increase performance of the appliance with somelimitations explained below. The decision to purchase ADP add oncards should be made in consultation with Check Point Sales. ADPshould be considered if the performance improvement desired falls into one of the following categories:

Packet throughput performance, specifically for small packets. Performance improvements for packet streams with mixed packet

sizes. Encrypted traffic (VPN) forwarding. Long-lived connections performance.

For example data transfer rates for protocols like ftp, http etc. NAT Performance only for long-lived NAT connections. (ADP

accelerates NAT throughput. Connection-rate Acceleration is

not currently supported for NAT connections. XMC cardsshould be used instead for high NAT & CPS.)

Latency for both un-encrypted and encrypted traffic. Multicast throughput performance.


22/45


ADP

Performance issues with mixing ADP with non-ADP interfaces:

The best performance one can get is by not mixing trafficbetween ADP and non-ADP interfaces. Running in mixed mode

will have performance impacts. When run in dual-mode having separate ADP traffic flows and separate non-ADP flowsone can see the Appliance performance scale as opposed tonot using any ADP interfaces at all.


23/45


ADP Benefits

Throughput Acceleration:

The first packet in a connection is sent up the stack to the Security Gateway thatvalidated the packet based on the defined rule base. Once validated the SecurityGateway application tells IPSO via the SecureXL API to handle future packets in thesame connection. IPSO then instructs the ADP sub-system to create a bi-directionalflow for that connection. All future packets for that connection will now be processed bythe ADP sub-system.

The following protocols benefit from SecureXL & ADP Throughput acceleration:

TCP, UDP, & traffic carried over those protocols IPSec VPN acceleration Multicast forwarding PIM (from IPSO 3.9 for IP2250 & IP2255; from IPSO 4.2 for all platforms) GRE & ESP


24/45


ADP Benefits

Connection Rate Acceleration

The first packet in a connection is validated by the Security Gateway application. Oncevalidated, the Security Gateway instructs IPSO to create a template so that IPSO canvalidate future connections where only the Source Port differs. A template consists ofthe following attributes: SrcAddr, SrcPort, Proto, DestAddr, & Dest-Port. IPSO comparesthe first packet in the next connection to its template table. If the packet matches atemplate then IPSO adds the connection to its table, then instructs ADP to create a bi-directional flow for the connection and lastly informs the Security Gateway about thenew connection. All future packets are processed by the ADP module.

The following protocols benefit from SecureXL & ADP Connection-rate acceleration: Unencrypted TCP, UDP, & traffic carried over those protocols Particularly effective on HTTP 1.1 traffic Even more effective on HTTP 1.0 traffic HTTP 1.0: Separate connection for each HTTP component


25/45


ADP Best Practices

Configure traffic to flow in/out of the same ADP subsystem, sincetraversing to another ADP subsystem or worse to a non-ADP interface

will negatively impact throughput performance. There is a 10gb fullduplex bandwidth over the crossbar between 2 ADP subsystems.

Do not use ports connected to the ADP subsystems for clusterprotocol network or for Security Gateway state synchronization. Useonboard ports for Security Gateway state synchronization; this will

guarantee that the synchronization data goes to its own channel andwill avoid sync packets being lost. This also prevents the sync datafrom disrupting the data passing between the ADPs and the mainCPU. Likewise do not log to the management station via the ADPinterfaces.


26/45


ADP Best Practices

Note that the backplane connecting the ADP subsystems to themain CPU has limited bandwidth, and this bottleneck will impactthroughput performance when there is a lot of non-acceleratedtraffic.

Do not combine non-ADP ports and ADP ports in a link

aggregation group or redundancy group.

Do not include interfaces on different ADP I/O cards in the samelink aggregation or redundancy group. IP Security Appliances donot support cross ADP link aggregation.

SecureXL is enabled by default. Note that its critical not to disableSecureXL because SecureXL is required for ADPs to function.


27/45


ADP Best Practices

Avoid performing tcpdump or fw monitoron ADP

interfaces when the interfaces are under heavy load.Performing a tcpdump or fw monitoron an ADP interfaceforces all traffic received or transmitted by the ADPsubsystem, to be copied and piped to IPSO through thebackplane. SecureXL will still be used,and the ADP willstill accelerate, but the backplane will be choked withdata. This causes a significant degradation inperformance due to constricted backplane capacity.


28/45


Limitations of ADP

Traffic that is not throughput or connection rateaccelerated will not benefit from ADP acceleration. Alllimitations of SecureXL apply to ADP.

Transparent mode will accelerate traffic normally, howevera special design consideration is that there must be routespointing out of the xmode interfaces, as SecureXLdepends on caching route table lookups.


29/45


Limitations of ADP

Enabling Sequence verifier: this solution requires enabling sequence verifier option onIPSO as well as in SmartDashboard.

This solution was suggested after analyzing the CST, where it was observed that most(60 million out of 80 million) of the TCP connections were getting closed with RSTsinstead of the usual 3-way handshake for terminating TCP connections. As part of thestateful inspection, the Security Gateway needs to monitor all TCP RSTs if sequenceverifier is not turned ON, as they are categorized as untrusted RSTs. This behavior ofterminating TCP connections causes additional load on ADP backplane interfaces,where the packet drops were observed.

By turning ON sequence verifier ADP will perform sequence verification on all TCPconnections thereby validating even RSTs that are used to terminate the TCPconnections. Once the TCP RST is validated by ADP and is accepted there is no needto send this packet to the Security Gateway, thereby reducing the backplane traffic andthe overhead of the Security Gateway having to inspect these packets.


30/45


Limitations of ADP

Once the sequence verification is turned ON, you should see significantreduction in packets going over the backplane to the Security Gateway. This

can be monitored by executing the following command

ipsctl i net:dev:adp:if:stats with an r option to monitor the rate of tcp_rst

As a result of the reduction in TCP RSTs going over the backplane, we should

observe less drops of data packets on backplane which can be monitored byexecuting the following command.

ipsctl i net:dev:bp:msg:stats with an r option to monitor the rate ofrx_fc_drops


31/45


Limitations of ADP

Too many control messages queued up on eth1, will result in data loss on thedata channels eth2-4. The queue depth for the control channel is tunable, thedefault is 64 in IPSO 6.2. Potential values are 128 and 256:

ipsctl -w net:dev:bp:msg:delay_drop_limit 128

You should see a decrease in the rate of dropped packets on data channel.This can be monitored by executing the following command:

ipsctl i net:dev:bp:msg:stats with an r option to monitor the rate ofrx_fc_drops

If this solution does not yield the desired result then the delay_drop_limit canbe easily set back to its default value of 64. Setting of these ipsctl variablestakes effect immediately and is non-intrusive.


32/45


Limitations of ADP Turning off the delay_drop variable: this solution requires changing the default value of

delay_drop, an ipsctl tunable in IPSO. IPSO pro-actively drops data packets, when controlchannel is congested. This option can be completely turned off by executing the followingcommand

ipsctl w net:dev:bp:msg:delay_drop 0

The option of dropping data packets when control channel is congested was developed undercertain performance benchmarking conditions, where the box is tested for limits and theaggressive load conditions persist for an extended period of time. This is the reason foraggressive drop_delay_limit to 64 by default.

Unfortunately, this condition also comes into effect when there is transient congestion on thecontrol channel. By turning off this feature, we do not drop the data packets pre-maturely.

Current congestion level and the max congestion level on the control channel can bemonitored by executing the following command:

ipsctl i net:dev:bp:msg:stats with an r option to monitor the rate of bms_scheds andbms_scheds_max

If this solution does not yield the expected result, then you can revert back to default behaviorimmediately by executing the following command.

ipsctl w net:dev:bp:msg:delay_drop 1


33/45


Limitations of ADP

The PSL acceleration feature should be enabled on multicoresystems using SecureXL and CoreXL, with or without ADP. Theipsctl tunable can be found using

ipsctl a net:sxl

PSL acceleration allows full acceleration of all but the lastpacket containing the application level Protocol Data Unit

IPS / SmartDefense takes care of a go/no-go to drop deniedconnections.


34/45


Security Gateway Performance Tuning

NAT

IP Appliances do not support Network Address Translation(NAT) connection acceleration. The first packets of the firstconnection on the same service are forwarded to the SecurityGateway application. Then a template of that connection is

created so that subsequent TCP establishments on the sameservice, where only the source port is different, will beaccelerated by SecureXL. NAT connections setup and teardowncannot be accelerated because NAT templates are notsupported.

While each connection uses two entries in the flows table,connections involving NAT use four entries. NAT connectionsuse more CPU and memory resources compared to ordinaryconnections.


35/45


Security Gateway Performance Tuning

Rulebase Size

Although there is no limit to the number of rules in a Security Gateway database, thereis a performance impact as the number of rules grows. The more rules an Appliancehas, the more it will cost the Appliance in compilation time and runtime efficiency.

Rulebase size affects connections rate performance.

Rulebase order is important and can affect performance. Use the following guidelinesfor organizing the rulebase: The rulebase should be as simple as possible. With fewer rules the rulebase will be

more efficient and less error prone. When creating a rule, be specific. Narrow down the source, destination, and

service. Avoid using Any in the service field. The most active NAT rules should be at the top of the NAT rulebase. Defining Group Objects for networks allows the policy compiler to superset traffic in

the actual rule for a performance gain. Anti spoofing should be configured for all the Security Gateway interfaces.

Avoid using negate in the rulebase (For example, a network exclusion)


36/45


Performance Troubleshooting

Follow the guidelines to troubleshoot performance issues and use the bestpractices outlined above to optimize the Appliance overall performance.

Do NOT use fw monitor for performance troubleshooting. Connecting aspan port via a switch, is preferred to tcpdump. Traffic captures sent toCheck Point Support should not exceed 80mb uncompressed. Particularlines within a packet capture should be referenced by the customer as

needed.

Check free disk space using the df k command line tool.


37/45



Check currently used CPU statistics using the vmstat 1 tool. The last3 columns are significant, customers should never concernthemselves with the other columns. The very last column is CPU Idletime, this is the amount of free CPU cycles, in percentage, since thelast vmstat iteration. The second-last column is System CPU usage in

percentage, this includes IPSO and Check Point kernels as well asinterrupts. The third-from-last column is User CPU utilization, this isusually due to Policy Installation, SmartDefense, Security Servers,and user scripts or commands.


38/45



The top command line utility is used on recent IPSO versions, thisutility will provide per-process CPU and memory utilization, as well asglobal statistics, and more granular CPU statistics such as % ofinterrupts. The percentage of interrupts includes both softwareinterrupts and hardware interrupts. Hardware interrupts are virtually

never the cause of performance issues, performance issues arevirtually always caused by software interrupts.

If it can be shown that the performance problem is due to a high ratioof interrupts compared to overall CPU utilization, the fix is always first

to properly tune SecureXL.

P f T bl h i


39/45



SecureXL Acceleration statistics can be verified using the followingcommands:

fwaccel stat

fwaccel stats

fwaccel stats s fwaccel templates s

These commands should provide a good overview of how much SecureXL isin use. The SecureXL and Nokia IPSO Guide

(http://downloads.checkpoint.com/dc/download.htm?ID=10036) should beused to help tune the rulebase, and ensure that as many connections andpackets are accelerated as possible.

P f T bl h i


40/45



If dropped packets are a concern, they can be checked as a snapshot in IPSOby running the command ipsctl a ifphys | grep qdrop. This will provide anindication of which interfaces are dropping traffic. Please note that thesecounters are incremental since system boot time. For more information aboutqdrops, consult sk39462.

To view realtime statistics for a particular counter, run ipsctl i use use the r command to toggle the rate per second counter refresh.

Any qdrop which is logged will have a corresponding reason codeincremented, in ifphys::errors

ifphys::stats

Common drops are rx_mpc, which represents the Receive, Missed PacketCount, and symerrs, Symbol Errors.

P f T bl h ti


41/45



rx_mpc is due to the operating system not being able to flush the receivebuffer for the interface fast enough. The receive buffer queues up incomingdata which has been received on the physical media, and is flushed from thequeue every time there is an interrupt. The interrupt is triggered under 2conditions when the rx_ring has reached full, or after a timer is hit. Moreinformation about these processed is detailed in sk39176. There are advanced

tunable variables that can be used under direction of Check Point Support toinfluence this behavior.

Symbol Errors are due to a bad fiber network cable, dirty or dusty NIC port orfiber connectors. It can also be due to a bad NIC card or switch port but this is

very uncommon. Symbol errors only increment for received data for the localside, not sent data. More information about symerrs is detailed in sk39733.

P f T bl h ti


42/45



A lot of short lived connections transiting the enforcement point cancause slowdowns, as connection establishment and teardown incursCPU utilization. This can be partly mitigated, by ensuring templatescan be created for the most heavily used rules. You may also be ableto use the Fast Expire SecureXL feature.

Fast Expire should be used primarily for short lived connections suchas DNS.

P f T bl h ti


43/45



Use ifconfig to verify that no interfaces have the PROMISC flag set. Aninterface in promiscuous mode forwards all frames seen on the physicalmedia, to the operating system for Layer 2 filtering. An interface in non-promiscuous mode, uses a MAC chip to filter Layer 2 frames to ensure thatonly frames which are destined for the local machine are passed to theoperating system. This is determined based on Unicast/Broadcast/MulticastMAC Address lists in the Receive Address High and Low registers. PROMISCis set for Transparent Mode and this is normal behavior.

Customers may wish to view the Security Gateway connection tables inhuman readable format to help with rules optimization. Check Point Supporthas internal-only tools to read the output of fw tab u. Customers may beinterested in an unsupported third-party script which can be found athttp://www.fw-1.de/aerasec/download/fw1-tool/fw1-tool.pl

Check Point makes no guarantees about this product, and provides thisinformation as reference only.

Ad d d b i d t i
http://www.fw-1.de/aerasec/download/fw1-tool/fw1-tool.plhttp://www.fw-1.de/aerasec/download/fw1-tool/fw1-tool.pl


44/45


Advanced debugging and tuning

Advanced debugging and tuning should onlybe carried out under the direction of theEscalations group, or Development.

Th E d Q ti ?


45/45

The End Questions?

ip platforms best practices for performance 010810

Documents