advanced performance measurement service assurance … · • designing your network to deploy...

Copyright © 2003, Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr

111© 2003, Cisco Systems, Inc. All rights reserved.NMS-40417950_05_2003_c2


Advanced Performance Measurement with Cisco IOS® Service Assurance Agent

Session NMS-4041



Objective of this Presentation


Objective of this Presentation

• Give a clear understanding of SA Agent features and capabilities

• Understand the internals of SA Agent

• Be able to configure various probe types

• Advanced topics (performance, scalability,…)

• Some deployment recommendations



This Presentation Is Not About…

• Designing your network to deploy tight SLA services

• QoS configuration or recommendations

• We will talk about SA Agent only—not the various network management applications to use on the back-end


Prerequisites

• Before attending this session, you should:Be familiar with SA Agent

Understand the SLA concepts

Know the basic concepts of network management



Network Disturbance(When Theory and Practice Diverge)


Murphy’s Law

• If anything can go wrong, it will

• If anything just cannot go wrong, it will anyway

• Left to themselves, things tend to go from bad to worse

• If everything seems to be going well, you have obviously overlooked something



SLA Components

• Delay

• Jitter

• Bandwidth

• Availability/connectivity

• Packet loss

• Out of Sequence (OoS)

• [Add your favourite here]


Latency (Delay)

• Propagation delay: the time it takes to the physical signal to traverse the path; (add 6 ns per meter for fibre, ie 36 ms for a transatlantic 6000 km link)

• Serialization delay is the time it takes to actually transmit the packet; depends on the bit-rate

• Queuing delay is the time a packet spends in router queues; depends on queue length and type

• Comfortable human-to-human audio is only possible for round-trip delays not greater than 100ms



Jitter

• This is the variation of the delay, a.k.a the ‘latency variance,’ can happen because:

• Variable queue length generates variable latencies

• Load balancing with unequal latency

• Harmless for many applications but real-time voice and video


Packet Loss

•• Loss of one of more packets, can happen Loss of one of more packets, can happen because…because…

•• CRC errorCRC error

•• Full queue (tail drop) or out of contractFull queue (tail drop) or out of contract

•• Route change (temporary drop) or Route change (temporary drop) or blackholeblackhole route (persistent drop)route (persistent drop)

•• Interface or router downInterface or router down

•• MisconfiguredMisconfigured accessaccess--listlist

•• ……



Misordering [1/2]

• This is not a rare situation…

• According to a study, roughly 25% of the hosts monitored on the Internet exhibit reordering

• For the hosts that exhibited reordering on average 8 of the 50 packets were identified as being out of order

(Results Are Based on “Packet Reordering Is Not Pathological Network Behavior, Jon C. R. Bennett, Craig Partridge and Nicholas Shectman, IEEE/ACM Transactions on Networking, Vol. 7, No. 6, December 1999, p789”)


Misordering [2/2]

• Out-of-order packet delivery, can happen because…

• Load balancing through multiple paths having different latencies

• Typically happening on parallel architectures (equivalent to multiple parallel routers)

• …



But Also…

• Packet alteration—the content is randomly modified

• Packet duplication—the same packet arrives multiple times (generally combined with misordering)


To Summarize…

•• The network is like a live ecosystemThe network is like a live ecosystem

•• There are harmless and harmful species There are harmless and harmful species living togetherliving together

•• They cannot always be under controlThey cannot always be under control

•• But at least we can vigilantly observe But at least we can vigilantly observe what’s going onwhat’s going on



SAA Overview


Current Solutions to Measure SLAs?

• Wait for problem to happen, and customer to complain

Reactive approach

• Manually Monkey approach

• Custom, home-made application The geeky approach

• Special hardware probes The expensive approach



Current Solutions Drawbacks

• Requires additional hardware

• New software, protocols

• Additional configuration skills

• Eventually adding a new vendor, support contract…


The Idea behind SAA

• If you have a running Cisco IOS router, turn it into a probing device

The smart approach

• Reuse your current equipment and enhance existing network management applications

(ex: CiscoWorks, VPNSC, Infovista, Concord eHealth, Agilent Firehunter…)



Supported Cisco IOS Version

XXXXSNMP Support

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

12.2(2)T

XAPM

XICMP Path Jitter

XFrame-Relay (CLI)

XMPLS/VPN Aware

XXFTP Get

XXUDP Jitter with One Way Latency

XXXDLSw+

XXXDHCP

XXXDNS

XXXHTTP

XXXUDP Jitter

XXXXTCP Connect

XXXXUDP Echo

XXXXSSCP(SNA)

XXXXXICMP Echo Path

XXXXXICMP Echo

12.2(11)T(Eng2)

12.1(1)T12.2

12.0(5)T12.0(8)S

12.0(3)T11.2Feature/Release


Measurement Capabilities

HTTPHTTPDLSwDLSw

JitterJitterPathEchoPathEcho

EchoEchoDNS/DHCP

FTPFTP

Increasing Service ValueIncreasing Service ValuePath JitterPath Jitter

ConnectConnect

EchoEcho

SNASNA

Cisco IOS-BasedService Assurance*

Agent

TCP

QoS Support

MPLS VPN Aware

Frame Relay

APM

UDP ICMP

*With Cisco IOS 12.2(9)T

(TOS)(TOS)

ATM*



Availability

Cat5K

Cisco GSR, 10K

Cisco 6400/7200/7500,uBR7200

Cisco 800/100x/14xx/16xx/17xx

Cisco 25xx/26xx

MC3810 Cisco 36xx

Cisco AS5300/5800

Cisco 4500/4700Catalyst

5K/6K with RSM/MSFC

All Cisco IOS-Based Platforms


SA Agent History

• Used to be called RTR, renamed SA Agent in 12.0(5)T; we call it Engine 1

• Initially only for ICMP Echo

• New Engine 2 introduced in 12.2(11)T, and will be present in all 12.3 trains

Major rewrite of the SAA code

Faster and more scalable

Memory usage reduced by a factor 2 to 5

ATM and Frame Relay L2 probes



To Summarize…

• Wide measurement capabilities(UDP, TCP, ICMP…)

• Millisecond precision (do not use Cisco IOS timers)

• Accessible using CLI and SNMP• Proactive notification via SNMP traps• Already in Cisco IOS—available on most

platforms at no additional cost• All IP interfaces supported, physical

and logical


SAA Architecture



Network

Global Architecture Overview

SA Agent

Responder

Responder

IP Server

Management

Targets

Source

SNMP/CLI

Probe


SAA Sender

• Cisco IOS box that sends probes

• Where the probes are configured

• Where all the results are calculated and stored

• Target might be another entity running Cisco IOS, or another system like a server



SAA Responder

• Runs on Cisco IOS• To activate, add ‘rtr responder’ to the

config, or set rttMonApplResponder.0=1 with SNMP

• Sender uses the SAA control protocol to communicate with responder before sending the test packets

• Responder knows the type of operation, the port used, the duration

• Communication on UDP 1967 and can be authenticated with MD5, not encrypted


SAA Operation with Responder [1/2]

SAA Sender SAA ResponderControl Message Ask Receiver to

Open Port 2020 on UDP)

Responder Says OK

Sending Test Packets…

Start Listening onUDP Port 2020

UDP, 2020

Done: Stop Listening

ControlPhase

ProbingPhase



SAA Operation with Responder [2/2]

• The responder, based on the type of operation, may insert in/out timestamps in the packet’s payload

• Processing time spend on the responder can therefore be calculated and deduced

• The response time is always calculated by the sender


MD5 Configuration Example

key chain saa_kckey 1key-string key

rtr key-chain saa_kc

04:02:24.271: responder receives request04:02:24.271: Ver: 1 ID: 22 Len: 52 04:02:24.275: process port enable clsd04:02:24.275: cmd: command: , ip: 10.52.132.68, port: 6666,

duration: 5200

17:01:28.745: RTR 1: Starting An Echo Operation - IP RTR Probe 117:01:28.745: source=10.52.132.69(52653) dest-ip=10.52.132.68(6666)17:01:28.745: sending control msg:17:01:28.745: Ver: 1 ID: 21 Len: 52 17:01:28.749: cmd: command: RTT_CMD_JITTER_PORT_ENABLE, ip: 10.52.132.68,

port: 6666, duration: 520017:01:28.757: receiving reply17:01:28.761: Ver: 1 ID: 21 Len: 8

SAA Sender SAA Responder

RTT_CMD_JITTER_PORT_ENABLERTT_OK



SAA Control—Scanning

• SAA responder can be detected with a port scanner and generates an “RTR responder: bad format” when debug RTR error is enabled

• On the top of MD5, configure an access-list to restrict it’s access if it runs on a public network

linux-f7-1:~ # nmap -sU 10.52.132.68 -p 1967Starting nmap V. 3.00 ( www.insecure.org/nmap/ )Interesting ports on c26f7-11.nsite.cisco.com (10.52.132.68):Port State Service1967/udp open unknownNmap run completed -- 1 IP address (1 host up) scanned in 1 second


SA Agent ConfigurationVia Command Line (CLI)



Configuring an Operationc26f7-12(config)#rtr 1

c26f7-12(config-rtr)#type ?

dhcp Perform DHCP Operation

dlsw Perform DLSw Keepalive Operation

dns Perform DNS Query

echo Perform Point to Point Echo Operations

ftp Perform ftp operation

http Perform HTTP Operations

jitter Perform Jitter Operation

pathEcho Perform Path Discovered Echo Operations

tcpConnect Perform TCP Connect Operations

udpEcho Perform UDP Echo Operations


Parameters

router (config-rtr)#?

frequency Operation Frequency Value in Seconds

lsr-path Loose Source Routing Path

owner Owner of Entry

request-data-size Requested Request Payload Size

response-data-size Requested Response Payload Size

tag User Defined Tag

threshold Operation Threshold in msec

timeout Operation Timeout Value in msec

tos Type Of Service

verify-data Verify Data



ICMP Echo Operation

• Target can be any IP host

• Round-trip time computed by measuring the time taken between sending an ICMP Echo request message and receiving the reply

• Processing delays on the source router only is subtracted


ICMP Echo Operation (Measurement)

• The round trip time is T = T2-T1

• The processing time spend on the target host cannot be measured, nor predicted, so it will be included in the total round trip time (see later)

• Because it might be inaccurate, use this probe for connectivity measurement (check link connectivity, if a server is still online, if a dial-on-demand line is up,…)

T1

T2T3 Target HostSAA

Sender



ICMP Echo Operation (Example)

rtr 1type echo protocol ipIcmpEcho 10.32.130.2tos 0x20frequency 120

rtr schedule 1 life forever start-time now


ICMP Echo Operation (Output)

c26f7-12#sh rtr op 1Current Operational State

Entry Number: 1Modification Time: 11:28:21.000 CET Thu Aug 22 2002Diagnostics Text: Last Time this Entry was Reset: NeverNumber of Octets in use by this Entry: 1490Connection Loss Occurred: FALSETimeout Occurred: FALSEOver Thresholds Occurred: FALSENumber of Operations Attempted: 1Current Seconds Left in Life: infinite - runs foreverOperational State of Entry: activeLatest Completion Time (milliseconds): 2Latest Operation Start Time: 11:28:21.000 CET Thu Aug 22 2002Latest Operation Return Code: okLatest 10.52.130.2



SAA Accuracy…ICMP Echo Probe

• With unloaded receiver, SAA measures 1.5 ms

• With high CPU load on the receiver: 45 ms!!

ICMP Echo Probe

Any System Will Report Wrong Results when Too Much CPU Time Is Spent on the Receiver between

the ICMP Echo Request and Echo Reply

Fortunately, We Have a Solution…

(90% Process Load)

ResponderSender


Processing Time Measurement

• When running the responder, we have a clear advantage, because…

• There is a mechanism to evaluate the processing time spend on the receiving router

• Insert a timestamp when the responder receives the packet, and when it replies

• Receive timestamp done at interrupt level, as soon as the packet is dequeued from the interface driver;absolute priority over everything else

• With SA Agent, this mechanism is implemented for both UDP Echo and UDP Jitter probes



UDP Echo Operation

• Uses either well-known UDP port 7 (echo service)—or any other custom port

• Can run with or without the responder

• However, it requires the responder for more accurate results; the processing delay spend on both source and destination is measured and deduced from the total RTT


T2

UDP Echo Operation (w/SAA Responder)

• We have no control on the queuing delay on the source and destination, but this is experienced by real traffic too, and must be accounted as such

T5

T4

T3

Processing Delay on the Source: Tps = T5-T4

Processing Delay on the Destination: Tpd = T3-T2

Round Trip Time Delay: T = […] = T2 - T1 + T4 - T3

Sender

T1

Responder



UDP Echo Operation (Example)

rtr 1type udpEcho dest-ipaddr 10.52.132.68 dest-port 7threshold 200

rtr schedule 1 start-time now


UDP Echo Operation (Output)

c26f7-12#sh rtr op 1Current Operational State

Entry Number: 1Modification Time: 13:55:05.000 CET Thu Aug 22 2002Diagnostics Text: Last Time this Entry was Reset: NeverNumber of Octets in use by this Entry: 1490Connection Loss Occurred: FALSETimeout Occurred: FALSEOver Thresholds Occurred: FALSENumber of Operations Attempted: 1Current Seconds Left in Life: 3595Operational State of Entry: activeLatest Completion Time (milliseconds): 2Latest Operation Start Time: 13:55:05.000 CET Thu Aug 22 2002Latest Operation Return Code: okLatest 10.52.132.68



The SAA Responder Processing Delay Will Be Subtracted to the Final Results

The SAA Responder Processing Delay Will Be Subtracted to the Final Results

• With unloaded receiver: 1.5 ms

• With 90% CPU receiver: 1.8 ms

SAA Accuracy—UDP Echo Probe

UDP Echo Probe

ResponderSender

(90% Process Load)


UDP Jitter Operation

• Measures the delay, delay variance (jitter) and packet loss by generating periodic UDP traffic

• Measures: per-direction jitter, per-direction packet-loss and round trip time

• Detect and report out-of-sequence and corrupted packets

• One-way delay requires Cisco IOS 12.2(2)T or laterand clock synchronization between source and destination

• One-way jitter does not require clock sync

• Always requires SAA responder



UDP Jitter—Measurement Example

SAA RTx = Receive Tstamp for Packet x

Send Packets

ST2 ST1

P1P1P2P2P2i1

AT1 AT2

Reflected Packets

P2P2P1P1i4

Responder

Dx = Processing Time Spent between Packet Arrival and Treatment

IP Core

STx = Sent Tstamp for Packet x

With Each Packet Is Associated STWith Each Packet Is Associated STxx, RT, RTxx, AT, ATxx, d, dxx——so so the Source Can Now Calculate:the Source Can Now Calculate:JitterSDJitterSD = (RT= (RT22--RTRT11))--(ST(ST22--STST11) =) = ii22--ii11JitterDSJitterDS = (AT= (AT22--ATAT11))--((RT((RT22+d+d22))--(RT(RT11+d+d11)) =)) = ii44--ii33

ATx = Receive Tstamp for Packet x

Receive Packets

RT2 RT1

P1P1P2P2P2i2

RT1+d1 RT2+d2

Reply to Packets

P2P2P1P1i3


UDP Jitter Operation Jitter Computation

• If packets are sent with 10ms interval, positive jitter means they have been received with more than 10ms interval

• Negative jitter means less than 10ms interval

• Zero jitter means they are received with the same inter-packet delay (the variance is zero)

• Jitter should remain as low as possible for real-time traffic such as voice over IP

• No need to have clocks synchronized



UDP Jitter Operation One-Way Delay Computation

• One-way delay measurement requires the clocks on source and target routers synchronized

• Use Network Time Protocol (NTP) server, eventually with GPS device as reference

• Use a GPS device on the auxiliary port of a 7200 (ex: Trimble Palisade GPS)

• If the time is not synchronized, SAA skips the one-way delay results; we tolerate a drift of 10% of the RTT; the shorter the delay, the stricter it will be

• GPS requires clear-sky view: not always feasible

• CDMA clocks works where a cell-phone work


Ack: (1,1)Ack: (1,1)

UDP Jitter OperationPacket Loss

Sender

Tim

e

Result:PacketLossSD = 1PacketLossDS = 1

Result:PacketLossSD = 1PacketLossDS = 1

Send Counter: 1

Responder

Rx: (5,4)Rx: (5,4)

R Missed a Packet (Only 4 Received while 5 Sent): PacketLossSD += 1

R Missed a Packet (Only 4 Received while 5 Sent): PacketLossSD += 1

Index of the ACKd Packet

Index of the ACKd Packet

Rx Counter: 1

Rx Counter: 2

Rx Counter: 3

Rx Counter: 4

Rx: (1,1)Rx: (1,1)

Rx: (3,3)Rx: (3,3)

XX Ack: (2,2)Ack: (2,2)

Ack: (3,3)Ack: (3,3)

Ack: (5,4)Ack: (5,4)

Send Counter: 3

XXSend Counter: 4

Send Counter: 5

Current RxCountCurrent RxCount

Send Counter: 2

R Received the Packet, but Did Not Received the ACK: PacketLossDS += 1

R Received the Packet, but Did Not Received the ACK: PacketLossDS += 1



rtr 1type jitter dest-ipaddr 10.52.130.68 \

dest-port 3456 num-packets 20


UDP Jitter Operation (Example)

• Simple example:

UDP Jitter Probe to 10.52.130.68, Port 3456

Send 20 Packets Each Time


UDP Jitter Operation (Example)

• Simulating G.711 VoIP call• Use RTP/UDP ports 16384 and above, the packet size is

200 bytes (160 bytes of payload + 40 bytes of header)• Packets are sent every 20 milliseconds• Marked with DSCP value of 8 (TOS equivalent 0x20)

rtr 1 type jitter dest-ipaddr 10.52.130.68 dest-port 16384 \

num-packets 1000 interval 20tos 0x20frequency 60request-data-size 200rtr schedule 1 life forever start-time now

AB C

A = 20 msB = 20 s (1000 x 20 ms)C = 40 s (60 s – 20 s)



etychon-vpn#sh rtr op 1Current Operational State

Entry Number: 1Modification Time: 08:22:34.000 PDT Thu Aug 22 2002Diagnostics Text: Last Time this Entry was Reset: NeverNumber of Octets in use by this Entry: 1594Number of Operations Attempted: 1Current Seconds Left in Life: 574Operational State of Entry: activeLatest Operation Start Time: 08:22:34.000 PDT Thu Aug 22 2002Latest Oper Sense: okRTT Values:NumOfRTT: 997 RTTSum: 458111 RTTSum2: 238135973Packet Loss Values:PacketLossSD: 3 PacketLossDS: 0PacketOutOfSequence: 0 PacketMIA: 0 PacketLateArrival: 0InternalError: 0 Busies: 0(cont…)

UDP Jitter Operation (Output) [1/3]

Average RTT Was 458111/997 = 459ms

3 Packets Lost S->Dout of 1000 Sent


(…cont)Jitter Values:MinOfPositivesSD: 1 MaxOfPositivesSD: 249NumOfPositivesSD: 197 SumOfPositivesSD: 8792 Sum2PositivesSD: 794884MinOfNegativesSD: 1 MaxOfNegativesSD: 158NumOfNegativesSD: 761 SumOfNegativesSD: 8811 Sum2NegativesSD: 139299MinOfPositivesDS: 1 MaxOfPositivesDS: 273NumOfPositivesDS: 317 SumOfPositivesDS: 7544 Sum2PositivesDS: 581458MinOfNegativesDS: 1 MaxOfNegativesDS: 183NumOfNegativesDS: 603 SumOfNegativesDS: 6967 Sum2NegativesDS: 336135Interarrival jitterout: 16 Interarrival jitterin: 35One Way Values:NumOfOW: 0OWMinSD: 0 OWMaxSD: 0 OWSumSD: 0 OWSum2SD: 0OWMinDS: 0 OWMaxDS: 0 OWSumDS: 0 OWSum2DS: 0


Follow RFC1889 (RTP) to Measure Jitter with Noise Reduction

No Synchro between Clocks: All ZeroesNo Synchro between Clocks: All Zeroes

Source to Destination Jitter

Destination to Source Jitter

See Next Slide



MinOfPositivesSD: 1 MaxOfPositivesSD: 249

NumOfPositivesSD: 197 SumOfPositivesSD: 8792 Sum2PositivesSD: 794884

MinOfNegativesSD: 1 MaxOfNegativesSD: 158

NumOfNegativesSD: 761 SumOfNegativesSD: 8811 Sum2NegativesSD: 139299


Smallest Positive Jitter

Number of Packets with a Positive Jitter

Sum of All Positive Jitter

Sum the Squares of All Positive Jitter

Sum the Squares of All Negative Jitter

Smallest Negative Jitter

Number of Packets with a Negative Jitter

Biggest Positive Jitter

Biggest Negative Jitter

Sum of All Negative Jitter


UDP Jitter Operation—Calculate Jitter

• There is no average jitter on SAA output

• You can calculate it with:

NumOfRTT

DS}|SDNegative}{|tiveSumOf{Posi∑=AvgLat



SA Agent Configuration(Using SNMP—Quick Overview)


SAA Configuration via SNMP

• Uses the RTTMON MIB• For all the probes should be set at least:

rttMonCtrlAdminRttTypeThe type of SAA operation to be performed; this value must be set in the same PDU or before setting any type specific configuration

rttMonEchoAdminProtocolSpecifies the protocol to be used to perform the SAA operation; the following list defines what protocol should be used for each probe type: echo, pathEcho, ipIcmpEcho, udpEcho, ipUdpEchoAppl, tcpConnect, ipTcpConn http, httpAppl jitter, jitterAppl dlsw, dlswAppl dhcp, dhcpAppl ftp, ftpAppl



SA Agent via SNMP (Example)

• SNMP set for ICMPEcho:rttMonCtrlAdminStatus.1 1

rttMonCtrlAdminRttType.1 1

rttMonEchoAdminProtocol.1 2

rttMonEchoAdminTargetAddress.1 "05 00 00 02"

rttMonEchoAdminTOS.1 5

rttMonScheduleAdminRttStartTime.1 1

rttMonScheduleAdminRttLife.1 200Last for 200 SecondsLast for 200 Seconds

Start NowStart NowTOS = 5TOS = 5

Destination IP = 5.0.0.2Destination IP = 5.0.0.2IpIcmpIpIcmp

Echo ProbeEcho Probe

Probe Is ActiveProbe Is Active

Probe Index = 1Probe Index = 1


SNMP Configuration—More

• Additionally each probe requires specific variables to be set; see the RTTMON MIB for more details

• For the probe to be visible in the running configuration, and hence saved, you must set rttMonCtrlAdminNvgen to 1; by default, the value is 0

• A running probe cannot be changed; this is also valid for CLI (an exception is made for trigger admin variables)

• Not everything is configurable by SNMP, and not everything can be retrieved by SNMP; check the MIB and CCO documentation for details



SA Agent Options


VRF—Awareness Issue (for MPLS/VPN)

• How to send a probe from the SA Agent to a specific VPN?

• By default, local processes are not executed in a VRF context

• Route lookup is done in the global routing table, and the wrong route is selected

10.10.10.110.10.10.1

10.11.10.110.11.10.1

10.12.10.110.12.10.1

SA Agent(PE)

CEs with VRFs Red, Blue and Yellow

We Need a Way to Execute a Probe in a VRF Context—E.G. Red, Blue or YellowWe Need a Way to Execute a Probe in a VRF Context—E.G. Red, Blue or Yellow



Solution—SAA for MPLS/VPN

• SAA probes are vrf-aware since 12.2(2)T

• Supported on ICMP Echo, ICMP Path Echo, UDP Echo and UDP Jitter probes

• Allows measurement from PE to anything

• Use vrf vrf-name option


rtr 1type jitter dest-ipaddr 10.52.130.68 \

dest-port 3456vrf blue


SAA for MPLS/VPN (Example)

• Sends a probe to a remote SAA in the context of the blue VPN:



TOS Marking

• Probes can be TOS marked

• Only TOS setting is supported, no diffServ(see next slide to perform translation)


Always Zero

Converting between TOS and DiffServ

128 64 32 16 8 4 2 1

32 16 8 4 2 1

554444176 (0xB0)176 (0xB0)101 100101 10011141456 (0x38)56 (0x38)001 110001 110

554040160 (0xA0)160 (0xA0)101 000101 000PrecedencePrecedenceDSCPDSCPToSToSBinaryBinary

Multiply by 4 Divide by 8

In Cisco IOS the 8 Bits of TOS Are Set from Right to Left

TOS(RFC795)TOS(RFC795)

DiffServ(RFC2474)DiffServ(RFC2474)

Precedence

4 2 1P2P2 P0P0P1P1 CUCU

D4D4 D3D3 D2D2 D1D1 D0D0DSCP (6 Bits)

D5D5

ToST3T3 T2T2 T1T1 T0T0

CUCU CUCU



Reaction Threshold [1/2]

martel(config)#rtr reaction-configuration <n> ?

action-type RTR Reaction Action Type

connection-loss-enable RTR Enable Connection LossReaction

threshold-falling RTR Falling Threshold Value

threshold-type RTR Reaction Threshold Type

timeout-enable RTR Enable Timeout Reaction

martel(config)#rtr reaction-trigger <probe> <targetProbe>


Reaction Threshold [2/2]

• Allows to set threshold on conditions (connection lost, threshold exceeded, single of multiple violations,…)

• Allows to set a reaction: send an SNMP trap, start another probe for problem diagnosis, or both



Low Water Mark

• Every time a probe is created, the router checks if there is more free memory than ‘LowWaterMark’; if not, the probe is not created

• By default, it’s set to 25% of the free memory on the router after bootup; if ‘0’, routers can create probes till run out of memory (not recommended)

etychon-vpn(config)#rtr low-memory ?

<0-4294967295> Low Water Memory Mark


Loose Source Routing (LSR) [1/2]

Source(207.139.2.10)

Destination(195.207.139.1)

Problem: If We Have Two Equal Paths, How to Measure One Specific Path?Problem: If We Have Two Equal Paths, How to Measure One Specific Path?

rtr 3type echo protocol ipIcmpEcho 195.207.139.1lsr-path 193.121.249.55rtr schedule 3 start-time now

Hop (193.121.249.55)

Solution: Specify a Hop with Loose Source Routing

Example:



Loose Source Routing (LSR) [2/2]

• To compute response time while the sender gives one or more hops that the packet must go through

• Valid for ICMP Echo probes• Intermediate devices might not support LSR,

which is often the case for security reasons• More processing time will be spend on the

intermediate hops than with regular probe; (LSR packets are sometimes process switched)


SAA Performance and Scalability



SAA Performance Measured(Cisco 2600 Running SAA Engine 1)

14401440

39003900

20002000

14401440

20002000

# of Source # of Source Probe Probe

Operations Operations per Minuteper Minute

51.5351.5397K Total97K TotalResponder (UDP JitterResponder (UDP Jitter

49.449.458K Total58K TotalResponder (UDP Echo)Responder (UDP Echo)

6.336.3311K per Probe11K per ProbeICMP EchoICMP Echo

54.3154.3117K per Probe17K per ProbeUDP JitterUDP Jitter

28.8328.8313K per Probe13K per ProbeUDP EchoUDP Echo

Average CPU Average CPU Usage (%)Usage (%)

Average Average Memory Usage Memory Usage

(Bytes)(Bytes)TypeType

(Cisco 2600 Running SAA Engine 1)


SAA Performance (Eng1 vs. Eng2, Jitter)

1200120010801080960960840840720720600600480480360360240240

Probes per Probes per MinuteMinute

38385757202034345151181830304646161627274040141423233434121220202727101016162020881313141466888844

CPU Load on CPU Load on Eng2Eng2

CPU Load on CPU Load on Eng1Eng1

Probes per Probes per SecSec

(Cisco 2600—12.2(8)T5 vs. Latest Eng. Build)



SAA Performance (Eng1 vs. Eng2, Jitter)

8

20

34

46

57

8

16

23

30

38

0

10

20

30

40

50

60

4 8 12 16 20

Probes per Second

CP

U L

oad

(%

)

Engine 1Engine 2

(Cisco 2600—12.2(8)T5 vs. Latest Eng. Build


SAA Performance—CPU Load by Platform

35

31

27

23

21

15

10

7

6

5

4

3

2

1

1

7200/2257200/225

57

56

45

41

40

34

29

25

16

9

6

4

3

1

1

37253725

36003600

33603360

31203120

28802880

26402640

24002400

21602160

19201920

16801680

14401440

12001200

960960

720720

480480

240240

Probes/Probes/MinuteMinute

95859963636

1464654040

1970714444

2276774848

2381825252

2595965656

44945772828

65652883232

276060

34239662424

33532572020

32827461616

32113341212

31272088

388844

7500/ 7500/ RSP8RSP8364036402650XM2650XM26002600Probes/Probes/

SecondSecond

(Jitter Probe Running Eng 1—500 Active Probes—Cisco IOS 12.2(8)T5)



SAA Performance—by Platform(Jitter Probe Running Eng 1—500 Active Probes—Cisco IOS12.2(8)T5

0

20

40

60

80

100

120

4 12 20 28 36 44 52 60

Probes per Second

CP

U L

oad

(%

) 26002650XM36407200/22537257500/RSP8


Relation between Probes and Frequency

• Each operation’s results have to be stored into a hierarchical structure

• So, the processing time increase with the number of configured probes

• With the same amount of probes starting every second, the higher the configured probes, the higher the CPU



Probes/Frequency—Graphical View

01020304050607080

4 12 20 28 36 44 52 60

Probes per Second

CP

U L

oad

(%

)

500 Probes

1000 Probes

2000 Probes

Cisco 7200VXR /NPE-225 Running Eng1 12.2(8)T5


Probes/Frequency—Graphical View

Cisco 7500/RSP8/250Mhz Running Eng1 12.2(8)T5

0

20

40

60

80

100

4 12 20 28 36 44 52 60

Probes per Second

CP

U L

oad

(%

)

500 Probes

1000 Probes

2000 Probes



SAA Memory Usage

< 3.2 KB< 3.2 KB< 17 KB< 17 KBICMP EchoICMP Echo

< 3.5KB< 3.5KB< 19 KB< 19 KBUDP EchoUDP Echo

< 12KB< 12KB< 24 KB< 24 KBUDP JitterUDP Jitter

Eng2Eng212.2(13)T12.2(13)T

Eng1Eng112.2(8)T512.2(8)T5

Engine 2 Reduce the Memory Usage by a Factor 2 to 5


SAA—Scalability Recommendation on Memory Usage

• Do not use more than 50% of the router’s total memory for SAA

• This amount depends on which image and features you are running…up to you to fine tune



SAA Accuracy—The Dilemma

• A router is, basically, a forwarding machine

• SAA is a time sensitive application running on a forwarding machine

• Cisco IOS processes uses a non-premptive scheduler

• This creates potential issues…but we have solutions


SAA Accuracy—ICMP vs. UDP

• As seen before—for RTT accuracy, always use UDP Echo or jitter with SAA responder

• Only in this case, processing time spent on the sender and responder routers will be subtracted

• Results more accurate regardless of the sender and receiver CPU process load

• But…if we have a high CPU interrupt load, like packet forwarding on centralized platforms, things may change…



UDP Echo Probe

ResponderSender

SAA Accuracy—Forwarding

Traffic Generator

Traffic Generator

(90% Fwd Load)

• With unloaded receiver: 1.5 ms

• With 90% CPU receiver, loaded by forwarded traffic: 10 ms!!

SA Agent Timestamping Routines Are in Competition with the Forwarded Traffic Done at

Interrupt Level Too

SA Agent Timestamping Routines Are in Competition with the Forwarded Traffic Done at

Interrupt Level Too


SAA Accuracy—Forwarding Router

• SAA may be inaccurate on a router loaded with forwarded traffic

• Reason is that interrupt level code (ie: interface) is in competition with SA Agent

• Actual solution: use a dedicated, non forwarding router (called shadow router)



SAA Accuracy—Test Results

• Tests have shown good accuracy if the router’s forwarding CPU load is below 30%. This is Cisco’s recommendation.

• Results become unrealistic when the forwarding CPU load reach the 60% utilization

• Process load has a negligible effect on UDP probes. Remaining at 60% process load is a comfortable value.


Accuracy with TOS-Marked Packets

• SAA probes can be sent with a specific Type of Service (TOS) value

• The right precedence will be applied when routing the packet, but what about the sending router?

• It depends…



Accuracy—Per Platform TOS Queuing

• Non-distributed platforms and 7500:Locally originated packets with proper TOS marking will go through the same outgoing queuing treatment; so SA Agent packets go through the corresponding WFQ queues

• For the Cisco 12K (GSR) and 10K (ESR):Queuing is done on the line cards; locally originated packets, like SAA probes, are all going to the default queue regardless of their original precedence; the default queue is typically slower


Best Practice and Design Recommendations



Why Using a Shadow Router?

• If your PE is already well loaded, or…

• If your PE lacks memory, or…

• If your PE is a distributed platform, or…

• If you want to isolate SAA and routing, or…

• If you want to be able to upgrade the SAA engine without disturbing the network, then…

• Use a Shadow Router


What Is a Shadow Router (SR)?

• Dedicated router to run the SAA engine, behave as a CE or a PE in a POP

• If behave as PE, member of VRFs but advertise no route

• If behave as CE, use one link per VRF; ideally use Ethernet with VLAN



Shadow Router as CE—Pros and Cons

• Cheap as any router from Cisco 800 will suffice

• Use one shadow per VRF, or set one VLAN per VRF [this requires Cisco 1700+ for dot1q]

• Cannot cope with overlap addresses


dot1q

Shadow Router as CE—Example

Blue VPNBlue VPN

Green VPNGreen VPN

PE

Shadow Router(As a CE)

interface fastethernet 4/1.100encapsulation dot1q 100ip vrf forwarding blueip address …

interface fastethernet 4/1.101encapsulation dot1q 101ip vrf forwarding greenip address …

interface fastethernet 1/1.100encapsulation dot1q 100ip address …

interface fastethernet 4/1.101encapsulation dot1q 101ip address …



Shadow Router as PE—Pros and Cons

• Requires at least a Cisco 3600 for PE functionality, often the remote access router can be leveraged

• Works for all VRFs without touching the configuration


Blue VPNBlue VPN

Green VPNGreen VPN

Shadow Router as PE—Example

Evaluate PE-to-CE for Green VPN

Evaluate PE-to-PE, or PE-to-CE

for Any VPN

The Shadow Router Emulates a PE with VRF-Aware SAA

PE

Shadow

Evaluate PE-to-CE for Blue VPN



How to Probe?

• Full mesh

• Partial mesh

• Composite SLAs


Full Mesh

• No of probes is a square function of nodes

• Does not scale

NodesNodes ProbesProbes22 11

33 33

44 66

55 1010

66 1515

77 2121

88 2828

…… ……

100100 49504950



Partial Mesh

San JoseAmsterdam

Raleigh

Brussels

Paris

London

• Full mesh is not always desirable• Select only critical path, like branch offices

to headquarters• Dramatically reduces the number of probes


Composite SLA for Delay

A

B

C

D

E

F G H

10 ms

8 ms

12 ms

5 ms

11 ms10 ms

7 ms

C->F = 24 msC->F = 24 ms

• Total delay can be easily calculated by adding the measured delay

• For n nodes, full mesh requires n(n-1)/2 probes while composite requires n-1 probes

• Measurement is less accurate, as each measurement carry it’s own error tolerance



Composite SLA for Jitter

• Short answer: NO!

• This is not a valid approach to calculate total jitter based on measured jitter, because we don’t know how to do it…

• Too many factors: positive jitter, negative jitter, percentile-95 of jitter, average jitter,…

• You’d better measure it, not calculate it

2 ms 4 ms 3 ms

Can We Add a Jitter Value to a Jitter Value?


SA Agent Management Applications



Management Applications Supporting SA Agent

Internetwork Performance Monitor (IPM)Service Management Suite (SMS)VPN Solution CenterCNS Performance Engine

eHealthVistaViewPowerView

Firehunter

UpTimeIPInsight

Brixworx

…and Many More


SA Agent Applications

TrendReportsTrend

Reports

HealthReports

HealthReports

Report for Thu 1/15/98

Service LevelReports

Service LevelReports

ExceptionsReports

ExceptionsReports

01/15/1998

09/13/1997

09/13/1997



Screenshot—Infovista Vistaview


Screenshot—Cisco IPM Probe Configuration

Target for SA AgentDoes Not Have to Be

a Cisco Router

Device whereSA Agent Is Configured

and Statistics Are Collected Type of Probe andCharacteristics



Screenshot—Cisco IPM Hourly Jitter


References(If You Want to Go Deeper)



References

• Cisco IOS documentation on CCO:http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120t/120t5/saaoper.htm

• SAA User Guide:http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/saaug_ai.htm

• Measuring Delay, Jitter, and Packet Loss with Cisco IOS SAA and RTTMON:

http://www.cisco.com/warp/public/126/saa.html


References

• SA Agent Support for Frame Relay, VoIP, and MPLS VPN Monitoring:

http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122newft/122t/122t2/ft1csaa.htm

• Deploying Tight-SLA Services on an Internet Backbone (RIPE-41):

http://www.ripe.net/ripe/meetings/archive/ripe-41/eof.html



Feedback

• Any question, feedback or suggestion regarding SAA, please use the eMail alias:

[email protected]


Conclusion/Summary

• SA Agent is a powerful and accurate toolbox providing a comprehensive set of measurement capabilities

• Easy integration into your current network, without additional equipment or skills

• It is free, and available today on almost all platforms running Cisco IOS



Other Network Management Sessions

• Network ManagementNMS-1001 Introduction to Network ManagementNMS-2001 Network Troubleshooting Tools and Techniques

• FaultNMS-1011 Principles of Fault Management

• ConfigurationNMS-2021 Configuration of Large-Scale Networks with CiscoWorks NMS-4021 Advanced Configuration Methods

• AccountingNMS-1031 Introduction to Collecting Traffic Accounting InformationNMS-4031 Advanced NetFlow Accounting

• PerformanceNMS-1041 Introduction to Performance ManagementNMS-2041 Performance Measurement with Cisco IOS Software

• SecurityNMS-2051 Securely Managing Your Network

• ServicesNMS-1101 Understanding DNS and DHCPNMS-2102 Deploying and Troubleshooting NAT

• High AvailabilityNMS-1201 Improving Network AvailabilityNMS-2201 Deploying Highly Available Enterprise Networks


Questions?




Please Complete Your Evaluation Form

Session NMS-4041




Annex


SA Agent vs. IETF IPPM



What Is IETF IPPM?

• An IETF working group

• Stands for IP Performance Metrics

• Exists since 1995

• Develop a set of standard metrics that can be applied to the quality, performance, and reliability of IP

• A lot of mathematics and statistics


SAA vs. IPPM

• Is an application

• Has it’s own way to perform measurements

• Address specific protocols and problems

• Is a specification

• Focussed on defining metrics for repeatable and interoperable measurements

• Generic for IP

SAA IPPM



Some of IPPM’s RFCs

• Framework for IP Performance Metrics (RFC 2330)

• IPPM Metrics for Measuring Connectivity (RFC 2678)

• A One-Way Delay Metric for IPPM (RFC 2679)

• A One-Way Packet Loss Metric for IPPM (RFC 2680)

• A Round-Trip Delay Metric for IPPM (RFC 2681)

• A Framework for Defining Empirical Bulk Transfer Capacity Metrics (RFC 3148)

• One-Way Loss Pattern Sample Metrics (RFC 3357)


SA Agent in the Future



SA Agent—Upcoming Features

• New features coming soon: VoIP active monitoring (Eng2)

Voice over IP network assessment and health monitoring; generate real RTP traffic

VRF backport to Eng1

Port the current VRF-aware SAA features to 12.0(26)S commonly used by SP

Bulk scheduling of SAA probes (Eng2)

Increase scalability when using CLI configuration

Ease of use improvements (Eng2)

Enhanced and more understandable show commands, and a new easy to use MIB


Statistical Considerations



Mean

• The mean gives some sort of ‘average’

• This is not the median

• It is sensible to noise (spikes)

• To calculate the mean, use:

∑=

=n

kkxn

m1

1


Mean (with SAA)

• With SAA output, the mean can be calculated with:

NumOfRTTRTTSum

=m



Example of Mean

• This example shows two latency samples having the same mean of 120ms; this clearly shows that the mean is not sufficient to determine the behaviour

0

50

100

150

200

1 3 5 7 9 11 13 15 17 19

Sample

Lat

ency

(m

s)

Result 1 Result 2


Variance

• The variance is a measure of how spread out a distribution is

• To calculate the variance:

mean. the and samples ofnumber theis Where

)(1 2

1

2

xn

xxn

n

kk∑

=

−=σ



Standard Deviation

• It is the square root of the variance

• Measures the spread of the data around the mean value

• A smaller standard deviation is better

• Use the formula:

2

1

)(1 ∑

=

−=n

kk xx

nσ


Standard Deviation with SAA [1/3]

• When SAA runs jitter probe, about the latency we have only the number of packets, the sum and the sum of squares

• To apply the std dev formula we need the square difference between samples and mean; let’s tweak the formula

• Calculate a corrective factor gamma to add to the sum of squares (RTTSum2), so that:

∑ ∑ +=−n

k

n

kkk xxx γ22)(




• Major steps, the details can easily be done offline (proceed by distribution then simplification):

−=

+−++−=+++=−++−

∑n

kk

n

nn

xxnx

xxxxxx

xxxxxx

2

2...2

...)(...)(22

1

221

221

γ

γγ



• After due substitution in the standard deviation formula, and simplifications, obtain this formula—that gives the standard deviation for latency based on SA Agent standard output:

• The same approach can be used for one-way latency, one-way jitter and so on

2

NumOfRTTRTTSum

NumOfRTTRTTSum2

−=σ



Example

• Result 1 standard deviation is 20,6• Result 2 standard deviation is 2,65

• This is a good indicator

0

50

100

150

200

1 3 5 7 9 11 13 15 17 19

Sample

Lat

ency

(m

s)

Result 1 Result 2


Applied Example [1/2]

• Here is an execution of a SA Agent Jitter probe, where we will calculate mean and standard deviation:

etychon-vpn#sh rtr op 1Current Operational State

Entry Number: 1[…]RTT Values:NumOfRTT: 10 RTTSum: 193 RTTSum2: 3741Packet Loss Values:PacketLossSD: 3 PacketLossDS: 0PacketOutOfSequence: 0 PacketMIA: 0 PacketLateArrival: 0InternalError: 0 Busies: 0[…]



Applied Example [2/2]

268.110194

103741

NumOfRTTRTTSum

NumOfRTTRTTSum2

2

2

=

−=

−=

σ

σ

3.1910193

NumOfRTTRTTSum

===m

advanced performance measurement service assurance … · • designing your network to deploy...

Documents