23 June 2003

Internet2 End-to-End Performance Monitoring Initiative Update

Internet2 End-to-End Performance Monitoring Initiative Update

Eric L. Boyd, Internet2

May 21, 2003

Eric L. Boyd, Internet2

May 21, 2003

6/23/2003 2

Agenda

� Motivation

� piPEs / AMI Overview

� piPEs / AMI Deployment Plan

� OWAMP Release

� Demo

6/23/2003 3

Context for E2E Performance

Even with high bandwidth

network links, the Internet2

community often does not see

expected performance.

6/23/2003 4

The Wizard Gap

6/23/2003 5

A Problem

Applications

Developer

System

Administrator

LAN

Administrator

Campus

Networking

Gigapop Gigapop

Backbone

Campus

Networking

LAN

Administrator

System

Administrator

Applications

Developer

How do you solve

a problem along a path?

Hey, this is not

working right!

The computer

Is working OK

Talk to the other guys

Everything is

AOK

No other

complaints

The network is lightly loaded

All the lights

are green

We don�t see

anything wrong

Looks fine

Others are

getting in ok

Not our problem

6/23/2003 6

Good Work Already Going On

Many projects and research is underway

by others investigating performance on

portions of the end-to-end path.

E2Epi seeks to bring these efforts

together to look at the end-to-end path

as a complete system.

6/23/2003 7

Can You Go Direct to the Problem?

How can you tell where is the problem?

Need a tool to tell you:� Where the problem is.

� The type of problem

� Who to contact to get it fixed

Terry Gray, University of Washington�We Need a Finger Pointing Tool�

6/23/2003 8

Gray Finger Pointing Tool

Applications

Developer

System

Administrator

LAN

Administrator

Campus

Networking

Gigapop Gigapop

Backbone

Campus

Networking

LAN

Administrator

System

Administrator

Applications

Developer

Locate the ProblemGray Finger Pointing Tool

Terry Gray

University of Washington

Finger Pointing

Tool Time!

6/23/2003 9

Gray Finger Pointing Tool

Applications

Developer

System

Administrator

LAN

Administrator

Campus

Networking

Gigapop Gigapop

Backbone

Campus

Networking

LAN

Administrator

System

Administrator

Applications

Developer

Finger Pointing

Tool Time

You ARE the

Weakest Link!

OK, I�ll fix it

6/23/2003 10

Agenda

� Motivation

� piPEs / AMI Overview

� piPEs / AMI Deployment Plan

� OWAMP Release

� Demo

6/23/2003 11

BNI AMI

� Project: Backbone Network

Infrastructure Abilene Measurement

Infrastructure

� Goal: Instrument Next Generation

Abilene network with extensive

performance measurement capabilities

� Approach: Collaborative project among

BNI, Engineering, and E2E.

6/23/2003 12

E2E piPES

� Project: End-to-End Performance Initiative

Performance Environment System (E2E

piPES)

� Goal: To allow end-users and network

operators to determine performance

capabilities, locate problems, and contact the

right person to get a problem resolved.

� Approach: Collaborative project combining

the best work of many organizations.

6/23/2003 13

The Measurement System

PMP

PMP

PMP

PMP = Performance Measurement Point (at each Abilene Node)

The New Abilene will have measurement devices as part of its structure

Abilene

6/23/2003 14

The Measurement System

PMP

PMP

PMP

PMP at each Campus Border

Extend the System to Campuses

AbilenePMP

Campus X

PMP

Campus Y

6/23/2003 15

Performance Reporting

PMP

PMP

PMP

Perform regular tests to provide web based Performance

Reports across Abilene and for Campus-Abilene paths

AbilenePMP

Campus X

PMP

Campus YCampus-Abilene Capability

Campus W � 155 Mbps

Campus X � 612 Mbps

Campus Y � 1000 Mbps

Campus Z � 2 Mbps

Abilene Capability

SNVA/DNVR � 10 Gbps

DNVR/IPLS � 10 Gbps

IPLS/NYCM � 10 Gbps

6/23/2003 16

Reaching the End-User

PMP

PMP

PMP

Provide a Testing Tool for the End-Users Computer

to test across campus and beyond

AbilenePMP

Campus X

PMP

Campus Y

Host A Host B

The tool reports performance of the path to a specified destination by using active tests

(like to the campus border) or using stored test results from regularly scheduled tests.

It also performs tests on the host system to check host tuning parameters

6/23/2003 17

Adding Path Information

PMP

PMP

PMP

Add PMP along paths to provide additional detail

AbilenePMP

Campus X

PMP

Campus Y

Host A Host B

Regular tests are performed between adjacent PMP and stored in a Data Repository.

This information is used by the End-User tool to reduce the number or active tests needed.

PMP

PMP

Regional

Network J

PMP

PMP

Regional

Network M

Gigapop S

Gigapop T

6/23/2003 18

Who You Gonna Call?

PMP

PMP

PMP

E2E piPES will provide contact information when a

segment in the path appears to have a problem

AbilenePMP

Campus X

PMP

Campus Y

Host A Host B

The test results obtained by the End-User will be passed on to the contact person

PMP

PMP

Regional

Network J

PMP

PMP

Regional

Network M

Gigapop S

Gigapop T

piPES Test Analysis

Looks like a problem

In Gigapop T. Pass

these test result to:

Joe@gigapopT.net

6/23/2003 19

E2E piPES Architecture

End User

Interface

Network Administrator

Interface

Culprit

DB

�Turn

Yourself

In� Culprit

Interface

ID Self

AAA

Query

CulpritID

Culprit

Specify end points &

application family

Request Test

ResultTesting

Engine

Analysis

Engine

Specify Test &

End NodesSend

Conclusion

Source

Administrative

Domain

InterfaceReply Ready,

Rejected, Patience

Virtual Performance

Measurement DB (local to

Domain)

Target

Administrative

Domain

Interface

AAA and Tool

AgreementRequest Test

Result

Source

PMC

Target

PMC

AAA & Schedule

Agreement

Source

PMP

Target

PMP

AAA & Schedule

Agreement

Database

Gatekeeper

Web-

based

Display

Engine

Human

Analysis

Engine

6/23/2003 20

Testing / Analysis Engine

Problem: Encode Matt Z.�s Brain

Open Questions:

�What are measurement results are �acceptable� for a given application (family)?

�What tools generate those results?

�How do you handle incomplete data?

�How do you rank multiple result-generators?

�What is the iterative decision tree to understand E2E problem?

6/23/2003 21

Measurement Schema

Measurement Types?

e.g. BW, Latency, Loss, Jitter

Measurement Units?

e.g. Seconds vs. Microseconds

Map Tools to Measurements?

e.g. OWAMP -> 1-way Latency

Measurement Metadata?

Database Table Design?

6/23/2003 22

Access, Authentication, Authorization

Roles for Access� Standard End-User?

� Near Neighbor (Test Buddy)?

� NOC Staff / Network Engineer?

Shibboleth for Implementation� Each campus decides who in each role

� Others trust campus designation

6/23/2003 23

Other R&D Issues

Design system to scale (eliminate

centralization)?

How to �discover� PMPs?

How to insert On-Demand tests into

Regularly-Scheduled test set

Balance centralization and distributed

database requirements

6/23/2003 24

Agenda

� Motivation

� piPEs / AMI Overview

� piPEs / AMI Deployment Plan

� OWAMP Release

� Demo

6/23/2003 25

E2E piPES Architecture v1.0

End User

Interface

Network Administrator

Interface

Culprit

DB

�Turn

Yourself

In� Culprit

Interface

ID Self

AAA

Query

CulpritID

Culprit

Specify end points &

application family

Request Test

ResultTesting

Engine

Analysis

Engine

Specify Test &

End NodesSend

Conclusion

Source

Administrative

Domain

InterfaceReply Ready,

Rejected, Patience

Virtual Performance

Measurement DB (local to

Domain)

Target

Administrative

Domain

Interface

AAA and Tool

AgreementRequest Test

Result

Source

PMC

Target

PMC

AAA & Schedule

Agreement

Source

PMP

Target

PMP

AAA & Schedule

Agreement

Database

Gatekeeper

Web-

based

Display

Engine

Human

Analysis

Engine

6/23/2003 26

piPEs / AMI Rollout

AMI

PMP

AMI

PMP

AMI

PMP

OWAMP

PERF

DB

Visualization Package

OWAMP OWAMP

OWAMP

6/23/2003 27

piPEs / AMI Rollout

AMI

PMP

AMI

PMP

AMI

PMP

OWAMP

PERF

DB

Visualization Package

OWAMP OWAMP

OWAMP

Web Service

6/23/2003 28

piPEs / AMI Rollout

AMI

PMP

AMI

PMP

AMI

PMP

OWAMP

PERF

DB

Visualization Package

OWAMP

IPERF

Traceroute

SNMP

OWAMP

IPERF

Traceroute

SNMP

OWAMP, IPERF, Traceroute, SNMP

Web Service

Traceroute

PERF

DB

IPERF

PERF

DB

SNMP

PERF

DB

6/23/2003 29

piPEs / AMI Rollout

AMI

PMP

AMI

PMP

AMI

PMP

OWAMP

PERF

DB

Visualization Package

OWAMP

IPERF

Traceroute

SNMP

OWAMP

IPERF

Traceroute

SNMP

OWAMP, IPERF, Traceroute, SNMP

Web Service

Traceroute

PERF

DB

IPERF

PERF

DB

SNMP

PERF

DB

Campus

PMP

Campus

PMP

6/23/2003 30

piPEs / AMI Rollout

AMI

PMP

AMI

PMP

AMI

PMP

OWAMP

PERF

DB

Visualization Package

OWAMP

IPERF

Traceroute

SNMP

OWAMP

IPERF

Traceroute

SNMP

OWAMP, IPERF, Traceroute, SNMP

Web Service

Traceroute

PERF

DB

IPERF

PERF

DB

SNMP

PERF

DB

Campus

PMP

Campus

PMP

Scheduler

Web Service

Configuration Package

6/23/2003 31

piPEs / AMI Development & Testing Environment

nmsX

Server

Server

nmsX

Server

Development

TestingTests Tests

New Code

Base Production Code

Measurement

Node

New Production Code

Master

Measurement

Node

Measurement

Node

Measurement

Node

Measurement

Node

Abilene

Measurement

Nodes

. . . . . . . . . . . . . . . . . . . . . .

Base/New Production Code

Tests Tests Tests

Server

Test Results

Database

Server

Performance

Visualization Tool

Measurement

Node

Router

Switch nms1

nms3 nms2nms4

CDMA

6/23/2003 32

piPEs / AMI Internal Resources

Eric Boyd � piPEs development, project coordination

Jeff Boote � OWAMP, piPEs development

Prasad Calyam � piPEs development

Chris Heermann � AMI

Matt Zekauskas - AMI

Susan Evett - Documentation

Russ Hobby � Campus Deployment

George Brett � Schema, �Grid Service�

6/23/2003 33

piPEs External Resources

Yee-Ting Li � UCL (50%)

Paul Mealor � UCL (up to 100%)

PhD Student � UCL (contingent on grant)

Warren Matthews � SLAC

Jim Ferguson et al � NLANR / DAST

Nicolas Simar et al � DANTE

Many early adopter volunteers

6/23/2003 34

Agenda

� Motivation

� piPEs / AMI Overview

� piPEs / AMI Deployment Plan

� OWAMP Release

� Demo

6/23/2003 35

Latency Tests

Problem: Existing platforms are not

interoperable (SURVEYOR,RIPE,�)

Solution: Standards

6/23/2003 36

OWAMP

One-Way-Active-Measurement-Protocol� Possible due to growing availability of good time

sources

� Wide deployment of �open� servers would allow

measurement of one-way delay to become as

commonplace as measurement of RTT using

ICMP tools such as ping.

� Current Draft: draft-ietf-ippm-owdp-05.txt

�Shalunov,Teitelbaum,Zekauskas

6/23/2003 37

OWAMP Implementation

Basically:NTP system call interface

Multiple processes for recv/send loops

Written as an API to allow one-off implementations

6/23/2003 38

OWAMP implementation status

Sample implementation

http://owamp.internet2.edu/

� Beta Release:

�No �policy�

�No authentication/encryption

�FreeBSD only

6/23/2003 39

Abilene OWAMP deployment

2 overlapping full meshes (IPv4 & IPv6)� 11 measurement nodes = 220 ongoing tests

� UDP singletons

� Rate: 10 packets/second*

� Packetsize: (32 byte payload)*

� Results are continuously streamed back to

�Measurement Portal� for long-term archive and

data dissemination (Near real-time)

6/23/2003 40

Agenda

� Motivation

� piPEs / AMI Overview

� piPEs / AMI Deployment Plan

� OWAMP Release

� Demo

6/23/2003 41

More Internet2 Information

On the Web� www.internet2.edu

� e2epi.internet2.edu

Email� Info-e2epi@internet2.edu

6/23/2003 42

Questions?

www.internet2.edu