The TeraPaths TeraPaths Collaboration
Presented byPresented by
Dimitrios Katramatos, BNLDimitrios Katramatos, BNL
2
Outline
Background: the TeraPaths projectBackground: the TeraPaths project Objective
View of the network
System architecture
Collaborative effortCollaborative effort The problem of being distributed…
Testing and deployment strategy
Milestones
What is the next step?What is the next step?
3
Objective
Provide QoS guarantees at the individual data flow level, Provide QoS guarantees at the individual data flow level, between actual end hosts, transparentlybetween actual end hosts, transparently Data flows have varying priority/importance
Video streams Critical data Long duration transfers
Default “best effort” network behavior treats all data flows as equal Capacity is not unlimited
Congestion causes bandwidth and latency variations Performance and service disruption problems, unpredictability
Dynamic flow-based SLAs = schedule network utilizationDynamic flow-based SLAs = schedule network utilization Regulate and classify (prioritize) traffic Select routing (if possible)
4
View of the Network
WAN
ctrl
WAN 1
WAN 2
WAN 3
TeraPaths
Domain ctrl
TeraPaths
RN
RN
TeraPaths
WAN
ctrl
WAN
ctrl
Site A
Site B
Site C
Site D
MPLS tunnelDynamic circuitDomain control
5
TeraPathsTeraPaths Web Services Architecture
Domain Controller
DSM
Web Interface
NDCNDCNDC • • •
Database
protected network
API
local
WAN controllers
• • •
Domain controllers(non-TeraPaths)
WAN serviceclients (proxies)
CLI s/w clientWeb browser
NDC database
Domain service clients (proxies)
Site controller
Site service
hardware
“virtualnetwork
engineer”
remote
6
Collaborative Effort
TeraPaths is a set of domain controllers tied together by TeraPaths is a set of domain controllers tied together by
distributed services modules (DSMs)distributed services modules (DSMs)
DSMs interoperate with WAN domain controllersDSMs interoperate with WAN domain controllers
End site controllers directly configure network devicesEnd site controllers directly configure network devices This can be a problem…
Modifying router configs on the fly using experimental software
requires trust and assurances
Thanks to our collaborators we were able to move from Thanks to our collaborators we were able to move from
proof-of-concept to prototype running currently at 3 sites: proof-of-concept to prototype running currently at 3 sites:
BNL, UMich, and BU - SLAC also participatesBNL, UMich, and BU - SLAC also participates
7
“Do no harm”
BNL testbed (2 X Cisco 6509) not part of production network, i.e. BNL testbed (2 X Cisco 6509) not part of production network, i.e.
breaking it is okbreaking it is ok Try software w/o risk to production network
Use only proven software to configure other site routersUse only proven software to configure other site routers
Bring hosts under TeraPaths control selectivelyBring hosts under TeraPaths control selectively Static part of configuration
Reduce adverse effects of bugs
SecuritySecurity “Circle of trust” for servers running TeraPaths modules
X.509 certificates, accounts, grid membership, etc.
“Virtual network engineer” web service to configure routers
There is always some residual riskThere is always some residual risk
8
Milestones
Proof of concept code demonstrated at SC’05Proof of concept code demonstrated at SC’05
Automatic invocation of OSCARS for MPLS tunnels in early 2006 Automatic invocation of OSCARS for MPLS tunnels in early 2006
Prototype code deployed at BNL and UMich, true end-to-end path Prototype code deployed at BNL and UMich, true end-to-end path
setup using MPLS tunnels through ESnet in summer 2006setup using MPLS tunnels through ESnet in summer 2006
Started looking into dynamic circuits in early 2007Started looking into dynamic circuits in early 2007
Modified code utilizes MPLS tunnels and dynamic circuitsModified code utilizes MPLS tunnels and dynamic circuits
Testbed expansion to BU through NoX and 2nd connection to UMich Testbed expansion to BU through NoX and 2nd connection to UMich
through Merit/MiLRthrough Merit/MiLR
Demo at SC’07: basic circuit utilizationDemo at SC’07: basic circuit utilization
Demo at JTW’08: traffic regulation within circuitsDemo at JTW’08: traffic regulation within circuits
9
MPLS Tunnels vs. Dynamic Circuits
WANborder routerborder router
MPLS tunnel ingress/egress
router
MPLS tunnel ingress/egress
router
WANswitch switch
border routerborder router
vs.
10
VLAN Setup for L2
TeraPaths-controlled“virtual border” router(directs flows w/PBR)
e.g.,1 to X, 2 to Y
WAN Site’sBorderRouter
trunked VLAN pass-thru50 VLAN ids (3550-3599)
3550 X Y 3599interfaces trust DSCP
TeraPaths-controlledhost router
#X
#Y
DSCP-friendly LAN
host 1 host nhost 2 . . .
1 to X
2 to
Ycan be the same device
RegionalProvider’s
Router
11
Current TeraPaths Testbed
US ATLAS T2 sites
BNL
OU
UC/IU UMich BU
SLACESnet
UTA
I2
NLR
NoX
StarLight
UltraLight
MiLR/Merit
L2 (dynamic circuit)
L3 (MPLS tunnel)
L2 and L3
JTW08 weathermap
12
Traffic Regulation (demo)
1
2
2
13
What is the next step?
Side effect of utilizing dynamic circuits: path selectionSide effect of utilizing dynamic circuits: path selection Common ground with Lambda Station
End sites need domain controllers in order to utilize dynamic circuits End sites need domain controllers in order to utilize dynamic circuits
and MPLS tunnels transparently and effectivelyand MPLS tunnels transparently and effectively
We would like to:We would like to: At the very least, interoperate with all types of domain controllers
Maybe develop a common-core domain controller for end sites,
customizable to site requirements through plug-ins?
Deploy domain controllers to all sites willing to participate All sites serviced by ESnet and Internet2 qualify
Setup a “protoduction” [?] testbed encompassing these sites
Eventually, move system into production