nanog24 miami 12th february 2002

Agnès Pouélé ([email protected]), Jan Novak ([email protected])

Inter-domain Multicast in European Research Networking:

TEN-155 Operational Experience and Deployment on GÉANT

NANOG24

Miami 12th February 2002

Agnes Pouélé, DANTE Ltd.Network Engineer

Jan Novak, Cisco Systems Inc. Network Consulting Engineer

1


Agenda• DANTE

• TEN-155 Operational Experience– Evolution of the TEN-155 multicast topology from

1998 to 2000– Operational Experience

• GÉANT deployment– Overview of GÉANT Network– GÉANT’s Multicast design– GÉANT Multicast and Unicast Coverage– GÉANT Multicast Service and Monitoring

• Conclusion2


DANTE, TEN-155, GÉANT

• DANTE • DANTE is a not-for-profit company set up in 1993

by European National Research Network organizations.

• TEN-155 • was an ATM based network built initially on OC3

links and then upgraded in 2000 • GÉANT

• successor of TEN-155 • launched in December 2001• 10 Gbps Pan-European Network

3


Part I TEN-155 Operational Experience

4


Typical mess of DVMRP tunnels on Sun WSs with usual tunnel routing problems.

5

Starting pointMBONE TUNNELED TOPOLOGY

Agnès Pouélé ([email protected]), Jan Novak ([email protected]) 6

OSLO IETF transmitted over both native STM-1 ATM based network and DVMRP tunnelled infrastructure

1999: PIM-SM and DVMRP set-up

PIM-SM domain


Multicast deployment in TEN-155

• Based on this first set-up and successful operation, we concluded to enable multicast on all production routers in TEN-155.

• Multicast code stable, CPU usage ok, M-BGP ok• PIM-SM “only” for production service

• PIM-SM to DVMRP border works, but not possible to operate routinely (NOC)

• From 1999 to 2000 each country was migrated from the DVMRP cloud to a interconnection with TEN-155 using PIM-SM/MBGP/MSDP

7


2000: TEN-155 final topology

NLUnited Kingdom

France

Italy

Greece

Belgium

Spain

CHFR

UK

SE

IT

DE

Slovenia

Czech Rep

Germany

AT

Switzerland

Portugal

AS8933OSPF + internal MBGP

Poland

Israel

NetherlandsNordics

e x t e r n a l

e x

t e

r n

a l e x t e r n

a l

B G P

AT & T

UUNET

Luxembourg

Hungary

US

ABILENE

8

tunnel

unicast/mcast linededicated mcast


TEN-155 Operational ExperiencePerformance Impact

• Parameters to be considered for the exploitation of Multicast– CPU - parameters

• about 1600 forwarding (mroute) entries• max 8 outgoing interfaces (average 2-3)• max 20 Mbit/s of data forwarded by one router resulting in 5-10% of CPU increase (mainly PIM)

9


TEN-155 Operational ExperiencePerformance Impact

• Parameters to be considered for the exploitation of Multicast– Memory

• about 3000 SA messages in the cache• about 1600 forwarding entries (mroute)• about 10 000 routes in the MBGP table

• These parameters didn’t represent a significant memory usage

10


Usage of MSDP – new component – new storms

• Default peer in redundant topology

• “Redundant” mesh-groups

MSDP peers default peer

Group A

Group C Group B

Impact of a MSDP storm on the CPU load

11


Usage of MSDP – early implementation problems

• Cisco IOS 12.0.6S and lower– ghost SA entries in the MSDP cache

• SA messages recreated by the incoming (S,G) joins

• FIXED– Origination of SA messages only when source registered

to the originating RP

– “ip msdp redistribute” command – without arguments• re-originated all known SAs

• caused huge increase of the SA counts worldwide• FIXED

12


TEN-155 MSDP monitoring

•MSDP usage monitoring •number of RPs, average 90 (40 EU)•number of groups•number of pairs source, group (S,G)•number of SA messages per minute

•Monitoring set-up

WS - DANTE DE.TEN-155router

•Software – adapted C++ MSDP •implementation of Steve Rubin

13

MSDP peering


TEN-155 MSDP monitoring

Number of RPs announced to TEN-155 before and after the loss of US connectivity

number of RPs

14


TEN-155 MBGP monitoring

•MBGP monitoring •stability/updates•number routes, max about 9000 (760 EU)•number of ASNs, average 240 (80 EU)

•Monitoring set-up

WS - DE PoP DE.TEN-155router

•Software - Merit’s MRTD - •modified SAFI definition for MBGP

15

MBGP peering


TEN-155 MBGP monitoring

•In blue, •the number of routes originated from one AS

•In green, •the number of updates originated by the same AS

stability/updates

16


TEN-155 Multicast Monitoring tools

• http://www.dante.net/pubs/dip/40/40.html

• http://www.dante.net/pubs/dip/41/41.html

• http://www.dante.net/mbone/

• http://sigma.dante.org.uk/stats/mrtg/msdp/data/

• http://www.dante.net/mbone/mbgp

The graph values from mbgp and msdp monitoring are historical values.

17


TEN-155 Operational ExperienceConclusion

• Move to native and SM multicast in national networks (NRENs)

• Unicast and multicast non congruent• MSDP peer doesn’t need to be RP.• Concept of two BGP tables and “multi-protocol” RPF

check often still misunderstood.• Inter-domain Multicast debugging

=> Almost impossible to fix problems in just one week

18


PART IINEXT GENERATION

GÉANT DEPLOYMENT

19

Agnès Pouélé ([email protected]), Jan Novak ([email protected]) 20

www.dante.net/nep/GÉANT-MULTICAST/map.html


GÉANT Services • GÉANT Standard IP Service

– IP traffic from NREN to NREN and Research peerings.

• Multicast Service (rolling out now)

• Replacement of TEN-155 Managed Bandwidth Service– GÉANT Premium IP Service – Layer-2 VPNs (forthcoming)

• Upcoming

• Security and Dos attack detection, IPV621


GÉANT’s Customers and Other Peerings

• GÉANT Unicast Customers– 27 countries in Europe

• GÉANT Multicast Customers – 24 countries in Europe

• GÉANT Unicast and Multicast research and commercial peerings– Abilene, Canarie and ESnet via GTREN– Infonet, UUNET

22


GÉANT Multicast Customers

NREN ACCESS MCAST Type

Austria POS STM-4 NO Mcast

Slovenia POS STM-4* NATIVE

Belgium POS STM-16 NATIVE

Croatia ATM E3 NATIVE

Czech Rep. POS STM-16 NATIVE

Cyprus E3 TBA

Germany POS STM-16 NATIVE

ESTONIA POS STM-1 NATIVE

Portugal POS STM-4 NATIVE

Greece POS STM-16* NATIVE

Ireland POS STM-1 NATIVE

Hungary POS STM-16 NATIVE

Italy POS STM-16 NATIVE

Israel T3 NATIVE

United Kingdom POS STM-16 NATIVE

Latvia E3 NATIVE

Lithuania T3NATIVE

Norway, Sweden, Finland,Island

POS STM-16 NATIVE

Poland POS STM-16 NATIVE

Spain POS STM-16 NATIVE

France POS STM-16 NATIVE

Luxembourg POS STM-1 NATIVE

Roumania E3 NATIVE

Slovak Rep. POS STM-1 NATIVE

Netherlands POS STM-16 NATIVE

Switzerland POS STM-16 NATIVE/TBA

Bulgaria E3 TBA

23



GÉANT Multicast Implementation

• Current design built on– The experience gained from TEN-155– Guidelines from multicast experts– Juniper Laboratory tests – GÉANT Multicast services

–Multicast transit domain NRENs to NRENs–Multicast transit domain NRENs to other

PEERS–Beacon monitored backbone

• GÉANT Multicast routing policy at:–

http://www.dante.net/nep/GEANT-MULTICAST/routing-policy.html

24


HostX

Manganese

Iridium

HeliumGallium

Sulfur

HostYXenon

Network 10.2/16 –AS 102

Network 10.0/16-AS 100

Network 10.1/16-AS 101

HostZ

Tin

Network 10.3/16 –AS 103

Group 233.1.10.1Source 10.1.10.2

Source 10.2.20.2Group 233.2.20.1 Group 233.3.30.1

Source 10.3.30.2

Group 233.0.1.1Source 10.0.1.2

Group 233.0.3.1Source 10.3.30.2

TEST-BED LAB 26/27th Nov 2001

M-BGP peering

MSDP peering

I-MBGP full mesh

•Junos 5.0R3.3

• mesh group

25

I-MSDP full mesh


PIM SMv2 GÉANT domain

• GÉANT single PIM-SMv2– Version 2 , enabled on all interfaces.

• Three Rendez-vous Points with private anycast address. – backup for internal sources and receivers.– Private Anycast address (filtered out)– closest RP based on the OSPF cost

• All other interconnected administrative domain have to be PIM-SM v2 enabled with their dedicated RP.

26


PHYSICAL TOPOLOGY: RP

uk

fr

at

ch

cz

pl

hu

sk

gr

ie

10

10

9

7 640

20

40

20

40

10

35

40

160

159

77

7

40

35

630

10si

160

170

be

nl

40

40

40

lu

640

640

5

it

es

de

se

IUCCJanet NY4-1 Nordunet

Eenet

Latnet

Litnet

Posnan

DFNNY4-2

Infonet

Multicast access

Rendez-vous Point

Cesnet

Sanet

HungarnetRoEduNet

Arnes

Grnet

Switch&Cern

Renater

Rediris

FCCN

Surfnet

Belnet

Restena

Heanet

Carnet

INFN

Aconet

Unicom-bCynet

STM64/OC192STM16/OC48STM4/OC12STM1/OC3

27

PoP



GÉANT Design: MBGP and MSDP• MBGP

• Separate multicast routing table (inet.2) • Currently congruent BGP and MBGP topology in

Europe

• iMSDP • MSDP is fully meshed between 19 PoPs• Use of mesh group• i-MSDP Peering with loopback addresses (Not

the anycast address !!)

• eMSDP• NREN <---> GÉANT Access Router

28


MSDP peering Logical view

iMSDP Mesh Group

iMBGP Full Mesh

ukse

de1

cz

huat

es

fr

Iucc Janet Ny4-1

Renater

Switch&Cern

Rediris

FCCN

INFN

Arnes

RoEdunet

Sanet Posnan

Cesnet

DFN

NY4-2

Infonet

GRnet

Nordunet

Latnet

Litnet

Eenet

Carnet

gr

CynetUnicom-b

ch

lu

Restena

beBelnet

it

si

Hungarnet

sk

de2

plie

Heanet

pt

Surfnetnl

Aconet

External MSDP peering NRN ----- GÉANT router

29

RP

GTREN

Abilene

www.dante.net/nep/GEANT-MULTICAST/map.html


MSDP SA Filtering

• A list of filtered SA is defined at:– http://www.dante.net/nep/GEANT-MULTICAST/

deployment-msdp.html

• We filter the recommended list.

• We authorise 239.194.0.0/16 from the IPV4 Organisation Local Scope through GÉANT.

30


GÉANTAS20965

CAnet

Third party providerEuropean Distributed Access

Commodity Internet Access

DFN

NREN2

JANET

NREN4

GTRENRESEARCH Peerings

STM-16

UKDE

Esnet

... ...

Abilene

31

STM-4

www.dante.net/nep/gtren.html

GÉANT Multicast and Unicast Coverage


GÉANT Multicast Service

• Access to the service– Via the primary access to GÉANT– Via a GRE tunnel (currently nobody)– Support of PIM-SM v2 only

• Operational procedures (rolling out now)– Goal: same level of service as Unicast.

• Troubleshooting – Extension of the trouble ticket systems to multicast

incidents

32


GÉANT multicast monitoring

• Beacon– Tool initially developed by Kai Chen from

NLANR• dast.nlanr.net/projects/Beacon/

• Relies on a number of Agents spread over the network which simultaneously send and receive multicast packets carrying a packet sequence number and a timestamp.

• Communicates with a central server which displays matrices of Agents via web pages.

33


GÉANT multicast monitoring• Recommended by TF-NGN group

– (www.dante.net/tf-ngn)

• To use from day 1– monitoring of multicast inside and outside of GÉANT.

• Server code enhanced with historical functionality http://noc.man.poznan.pl/noc/index/strony (Menu item

“Applications”)

[email protected]

Multicast Beacon Agent written in C http://www.cesnet.cz/tf-ngn/multicast/

34


GÉANT multicast monitoring• Beacon’s matrices

– One for the internal sources of GÉANT• Each GÉANT POP has a beacon agent installed

– One for the external sources of GÉANT• http://beaconserver.geant.net:19999/

– We have assigned two multicast groups from GLOP range [RFC2770] for each matrix

• Parameters monitored• Loss• Delay• Jitter

35


Beacon internal/external matrix

36

www.dante.net/nep/GÉANT-MULTICAST/deployment-beacon.html


Other Monitoring Tools

• Per group monitoring from TEN-155– Based on the IETF IP-MROUTE MIB – Shows traffic per multicast group per interface– Under installation on ws1.se.geant.net

• MSDP and MBGP monitoring tools– Not yet available

37


CONCLUSION

• Conclusion– From TEN-155 to GÉANT Unicast and multicast

moves to a congruent topologyacross Europe and towards research peerings.

• Deployment status

• Links– http://www.dante.net/nep/GEANT-MULTICAST/– http://beaconserver.geant.net:19999/

38


We would like to thank and acknowledge the help of the people who worked and are working with

us on these projects, mainly from all EU and US research networks

39

THANKS


Questions ?

40


Peering relationship Address DescriptionAll 224.0.1.2/32 SGI "Dogfight" gameAll 224.0.1.3/32 RWHODAll 224.0.1.8/32 Sun's NIS+All 224.0.1.22/32 SVRLOCAll 224.0.1.24/32 MICROSOFT-DSAll 224.0.1.25/32 NBC-proAll 224.0.1.35/32 SVRLOC-DAAll 224.0.1.39/32 Cisco's Rendezvous Point Announcement ProtocolAll 224.0.1.40/32 Cisco's Rendezvous Point Discovery ProtocolAll 224.0.1.60/32 HP's Device Discovery ProtocolAll 224.0.2.1/32 rwho group (BSD)All 224.0.2.2/32 Sun's Remote Procedure Call ProtocolAll 229.55.150.208/32 Norton "Ghost" disk duplication softwareAll 234.42.42.42/32 ImageCast disk duplication softwareAll 234.142.142.142/32 ImageCast disk duplication softwareAll 232.0.0.0/8 Default SSM-range. Do not do MSDP in this range.Out of GÉANT 239.0.0.0/8 Administratively Scoped IPv4 Group Addresses

In GÉANT

239.0.0.0/8 with exception of 239.194.0.0/16

Administratively Scoped IPv4 Group Addresses. Th 239.194.0.0/16 address address range is reserved for GÉANT only multicast traffic.

All 10.0.0.0/8 Private addresses (RFC 1918)All 172.16.0.0/12 Private addresses (RFC 1918)All 192.168.0.0/16 Private addresses (RFC 1918)All 127.0.0.1/8 Loopback address

SA Filter list

41

nanog24 miami 12th february 2002

Documents