monitoring e2e performance on high-speed networks

7/20/047/20/04 Joint Techs Columbus, OHJoint Techs Columbus, OH

Monitoring e2e Performance Monitoring e2e Performance

on High-speed Networkson High-speed Networks Margaret Murray (CAIDA)Margaret Murray (CAIDA)


AcknowledgementsAcknowledgements Shava Smallen: TeraGrid INCA Test Harness Framework at SDSCShava Smallen: TeraGrid INCA Test Harness Framework at SDSC Omid Khalili: INCA reporter programmingOmid Khalili: INCA reporter programming Nevil Brownlee: NeTraMet development and configNevil Brownlee: NeTraMet development and config Johnny Chang, Alok Shriram: bwest tool evaluation experimentsJohnny Chang, Alok Shriram: bwest tool evaluation experiments Tony Lee, Tuan Le: bwest tool automationTony Lee, Tuan Le: bwest tool automation Jiri Navratil, Ravi Prasad, Vinay Ribeiro: remote testbed usersJiri Navratil, Ravi Prasad, Vinay Ribeiro: remote testbed users Grant Duvall, Nathaniel Mendoza, Brendan White: router configGrant Duvall, Nathaniel Mendoza, Brendan White: router config Kevin Walsh: CalNGI, NPRL accessKevin Walsh: CalNGI, NPRL access

Spirent SmartBits 6000 with SmartFlow softwareSpirent SmartBits 6000 with SmartFlow software Foundry Big Iron routerFoundry Big Iron router

Cisco: GSR12008 router Cisco: GSR12008 router Juniper: M20 routerJuniper: M20 router Endace: gigE DAG card for passive monitoring with NeTraMet and Endace: gigE DAG card for passive monitoring with NeTraMet and

CoralReefCoralReef

Department of Energy SciDAC grant DE-FC02-01ER25466Department of Energy SciDAC grant DE-FC02-01ER25466


Talk OutlineTalk Outline

Monitoring/Measurement goalsMonitoring/Measurement goals Terms and ConditionsTerms and Conditions Bandwidth estimation toolsBandwidth estimation tools Evaluating and comparing toolsEvaluating and comparing tools

Lab tests with SmartBitsLab tests with SmartBits Lab tests with tcpreplayLab tests with tcpreplay

TeraGrid tests using the INCA architectureTeraGrid tests using the INCA architecture Future DirectionsFuture Directions


Why measure e2e Why measure e2e available bandwidth?available bandwidth?

Configure overlay routes Select “best” content distribution server Adjust encoding rate on streaming applications Verify SLA and QoS Use as criterion for end-to-end admission control Construct a peer-to-peer application topology Select inter-domain egress ISP and…


End-to-end performance End-to-end performance perspectivesperspectives

User goals:User goals: Optimize my application performanceOptimize my application performance Move my data… FASTMove my data… FAST With whom am I sharing network bandwidth?With whom am I sharing network bandwidth?

Sysadmin goals:Sysadmin goals: Identify problemsIdentify problems Set realistic performance expectationsSet realistic performance expectations

Common denominator:Common denominator: MaximizeMaximize available bandwidth available bandwidth


Terms Terms “Bottleneck” is not a meaningful term“Bottleneck” is not a meaningful term

e2e Capacitye2e Capacity (C): min (C): min link capacity in the link capacity in the pathpath

e2e avail-bwe2e avail-bw (A): min (A): min unused bandwidth at unused bandwidth at time time

BTCBTC: max achievable : max achievable TCP throughputTCP throughput

Tight link A3 (avail-bw)

Narrow link C1 (capacity)


……and Conditionsand Conditions(factors impacting e2e net performance)(factors impacting e2e net performance)

Cross-traffic (load level, burstiness)Cross-traffic (load level, burstiness) Traffic type (TCP/UDP) mixTraffic type (TCP/UDP) mix

We assume that 80%+ of apps are TCPWe assume that 80%+ of apps are TCP Number of competing streams Number of competing streams

Host TCP settingsHost TCP settings MTU sizeMTU size

Clock synchronization Clock synchronization Router buffer sizes and COS or QoSRouter buffer sizes and COS or QoS


Measuring end-to-end Measuring end-to-end Available BandwidthAvailable Bandwidth

It’s not easy, and tools haven’t been validated.It’s not easy, and tools haven’t been validated. Even fewer tools developed and validated on high speed links.Even fewer tools developed and validated on high speed links.

CAIDA is performing first comprehensive tool evaluation on high CAIDA is performing first comprehensive tool evaluation on high speed links in CAIDA/SDSC lab.speed links in CAIDA/SDSC lab.

Well-known Iperf Well-known Iperf (persistent TCP connection w/ large advertised (persistent TCP connection w/ large advertised window)window) Can be intrusive: can saturate the path and increase path delays Can be intrusive: can saturate the path and increase path delays

and jitter…depending on time scale and if no limits on its bw useand jitter…depending on time scale and if no limits on its bw use Measures “brute force” avail-bw Measures “brute force” avail-bw

Pathload (Self-Loading Periodic Streams)Pathload (Self-Loading Periodic Streams) Attempts to be non-intrusive over time (uses < 10% avail-bw)Attempts to be non-intrusive over time (uses < 10% avail-bw) Measures the dynamics of avail-bw over timeMeasures the dynamics of avail-bw over time


Current e2e ToolsCurrent e2e Tools

Tool ClassTool Class ToolTool AuthorsAuthors MethodologyMethodology ToolTool AuthorsAuthors MethodologyMethodology

Per-hop Per-hop CapacityCapacity

clink √clink √ DowneyDowney VPSVPS pathchar √pathchar √ JacobsonJacobson VPSVPS

pchar √pchar √ MahMah VPSVPS

End-to-End End-to-End CapacityCapacity

bprobebprobe CarterCarter pkt pairpkt pair pathrate √pathrate √ Dovrolis-Dovrolis-PrasadPrasad

pkt pairs,trainpkt pairs,train

nettimernettimer LaiLai pkt pairspkt pairs sprobe √sprobe √ SaroiuSaroiu pkt pairspkt pairs

End-to-EndEnd-to-End

AvailableAvailable

BandwidthBandwidth

abing √abing √ NavratilNavratil unknownunknown netest √netest √ JinJin unknownunknown

cprobecprobe CarterCarter pkt trainspkt trains pathload √pathload √ Jain-DovrolisJain-Dovrolis SLoPSSLoPS

IGI √IGI √ HuHu SLoPsSLoPs SpruceSpruce StraussStrauss Mod. SLoPSMod. SLoPS

Bulk TransferBulk Transfer

CapacityCapacity

capcap AllmanAllman emulate TCP emulate TCP tputtput

trenotreno MathisMathis std TCP tputstd TCP tput

Achievable Achievable TCPTCP

ThroughputThroughput

iperf √iperf √ NLANRNLANR || TCP connect|| TCP connect ttcpttcp MuussMuuss TCP connectTCP connect

NetperfNetperf NLANRNLANR || TCP connect|| TCP connect


Sidebar: How pathload works…Sidebar: How pathload works…

Send Send 100 probes of equal-sized packets at rate 100 probes of equal-sized packets at rate R and measure one-way delays; iterate while R and measure one-way delays; iterate while modifying R (and limit probing rate to < 10%)modifying R (and limit probing rate to < 10%)

One-way delays only increase when the stream One-way delays only increase when the stream rate R is rate R is largerlarger than the avail-bw A than the avail-bw A

…find the range of the kneeConcept:


Summer 2004 Tool EvalSummer 2004 Tool Eval

Tools to testTools to test PathloadPathload PathchirpPathchirp SpruceSpruce AbingAbing IperfIperf

Performance metricsPerformance metrics ErrorError Overhead trafficOverhead traffic Time to measureTime to measure

Testing metricsTesting metrics Test frequencyTest frequency Test schedulingTest scheduling

Tools not (or no Tools not (or no longer) under testlonger) under test AbwAbw BprobeBprobe CprobeCprobe ClinkClink PathcharPathchar PcharPchar PipecharPipechar traceratetracerate


Lab Tests with SmartBitsLab Tests with SmartBits

Use reproducible test conditionsUse reproducible test conditions Test against saturated linksTest against saturated links Validate tool and cross trafficValidate tool and cross traffic Test “black box” e2e tools against same Test “black box” e2e tools against same

scenariosscenarios Identify conditions where tools work wellIdentify conditions where tools work well Give developers an environment for refining their Give developers an environment for refining their

toolstools

(synthetic traffic, and unresponsive to TCP)(synthetic traffic, and unresponsive to TCP)


SmartBitstraffic gen

passivemonitor

OC48/gigE 4hop OC48/gigE 4hop configurationconfiguration

Foundryrouter

endhosts

endhosts

outer looptwo loopse2e pathgigE linkOC48 link

JuniperM20 router

Ciscorouter

regentap


Lab Tests with tcpreplayLab Tests with tcpreplay

Use the same anonymized trace for Use the same anonymized trace for all toolsall tools

Estimate the load level using Estimate the load level using CoralReefCoralReef

One end host generates tcpreplay One end host generates tcpreplay cross-traffic cross-traffic

One end host runs the tool under testOne end host runs the tool under test

(real traffic, but unresponsive to TCP)(real traffic, but unresponsive to TCP)


Tests with real trafficTests with real traffic

INCA Test Harness and Monitoring INCA Test Harness and Monitoring InfrastructureInfrastructure

Take advantage of INCA’s:Take advantage of INCA’s: Full mesh deploymentFull mesh deployment Data repository/archiveData repository/archive Web interfaceWeb interface Scheduling optionsScheduling options

To collect network performance data: To collect network performance data: Add Network Reporter Add Network Reporter

Reporter-Pair - a new variationReporter-Pair - a new variation Same wrapper can work with multiple avail-bw toolsSame wrapper can work with multiple avail-bw tools


Inca ArchitectureInca Architecture Data consumerData consumer - user- - user-

friendly web interface, friendly web interface, application, etc. application, etc.

FrameworkFramework - daemons - daemons Planning and execution of Planning and execution of

reportersreporters Centralized data collectionCentralized data collection PublishingPublishing

ReporterReporter - a script or - a script or executableexecutable


Gathering performance dataGathering performance data

1.1. Write reporter to wrap benchmark and Write reporter to wrap benchmark and print XML output according to Inca print XML output according to Inca reporter specification reporter specification

2.2. Write configuration file to express:Write configuration file to express:a)a) InputsInputsb)b) Frequency of executionFrequency of executionc)c) Data to archiveData to archive

3.3. Write web page to display dataWrite web page to display data


Writing performance reporterWriting performance reporter

Resource A Resource B

1. Reporter unpacks network tool

2. Reporter starts sender

3. Reporter launches receiver

4. Network tool runs

5. Reporter returns data

Perl API to enable running of network Perl API to enable running of network probes across sites (uses globusrun)probes across sites (uses globusrun)


Executing reporterExecuting reporter

Now: Cron schedulingNow: Cron scheduling Schedule far enough apart so they don’t Schedule far enough apart so they don’t

collidecollide Not foolproofNot foolproof

Move to token-passing protocol (NWS)?Move to token-passing protocol (NWS)?


Graphing dataGraphing data

Calls rrdtool commands to generate graphsCalls rrdtool commands to generate graphs CGI script currently uses SOAP call to get graph CGI script currently uses SOAP call to get graph

from Inca archivefrom Inca archive


More graphs from CGI formMore graphs from CGI form

User selects:User selects: SourceSource DestDest Start date/timeStart date/time End date/timeEnd date/time

Planned:Planned: Weather map styleWeather map style


Future Directions…Future Directions…

Justify test scheduling frequencyJustify test scheduling frequency Now: once/hrNow: once/hr Check result distributionsCheck result distributions Refine scheduling: Move to token-passing protocol Refine scheduling: Move to token-passing protocol

(NWS)? (NWS)? Compare results of multiple toolsCompare results of multiple tools

pathload, pathchirp, Spruce, iperfpathload, pathchirp, Spruce, iperf Consider error and overheadConsider error and overhead

Refine graphs and web interfaceRefine graphs and web interface Run network probes across different OSesRun network probes across different OSes Consider more e2e paths than just between login nodesConsider more e2e paths than just between login nodes

(especially aggregated bandwidth between site gridftp servers?)(especially aggregated bandwidth between site gridftp servers?)


Discussion: SOBAS for appsDiscussion: SOBAS for apps

Socket Buffer Auto-Sizing (SOBAS) [Prasad, Socket Buffer Auto-Sizing (SOBAS) [Prasad, Jain & Dovrolis, GaTech]Jain & Dovrolis, GaTech] Apps use a SOBAS enabled socket library.Apps use a SOBAS enabled socket library. Concept: Limit the send window after reaching avail-Concept: Limit the send window after reaching avail-

bw to avoid “self-induced” packet loss.bw to avoid “self-induced” packet loss. Experimental results show 20-80% increase in Experimental results show 20-80% increase in

throughput compared to TCP transfers using max throughput compared to TCP transfers using max possible socket buffer size.possible socket buffer size.

R. Prasad, M. Jain and C. Dovrolis, “Socket Buffer Auto-Sizing for R. Prasad, M. Jain and C. Dovrolis, “Socket Buffer Auto-Sizing for High-Performance Data Transfers” Journal of Grid Computing June High-Performance Data Transfers” Journal of Grid Computing June 2004. 2004. http://http://www.cc.gatech.edu/~ravi/tools/sobas.tar.gzwww.cc.gatech.edu/~ravi/tools/sobas.tar.gz


SummarySummary

CAIDA is evaluating bwest tools in both lab and CAIDA is evaluating bwest tools in both lab and real high-speed environments. real high-speed environments.

TeraGrid’s INCA architecture now supports TeraGrid’s INCA architecture now supports available bandwidth measurements.available bandwidth measurements.

Pathload reports a range variation of available Pathload reports a range variation of available bandwidth on an e2e path.bandwidth on an e2e path.

INCA/pathload measures available bandwidth on INCA/pathload measures available bandwidth on TGrid e2e paths (login node to login node).TGrid e2e paths (login node to login node).

monitoring e2e performance on high-speed networks

Documents

qosjoint techs columbus

timejoint techs columbus

availbw aconcept

pathe2e availbw

dynamics of avail

monitoring e2e performance

stream rate r

link capacity