- 1 -

Internet Access for Academic Networks in Lorraine - France Infrastructure and Services

Alexandre Simon

© C.I.R.I.L : Centre Interuniversitaire des Ressources Informatiques de Lorraine

- Inter-university center for computer resources of Lorraine - Rue du Doyen Roubault F - 54500 VANDOEUVRE

Phone : +33 (0)3.83.44.74.32

Fax :+33 (0)3.83.44.02.62 E-Mail : Alexandre.Simon@ciril.fr

Abstract Lorraine, a region of eastern France well known for its past, is today recognized for its regional network -Lothaire- and its metropolitan academic networks -StanNet and AmpereNet- that cover the two most important cities of the region, Nancy and Metz. Lothaire is built around a high bandwidth ATM backbone which connects six strategic points (called MAP, Metropolitan Access Point) and classical leased lines that are used to connect less important points. Lothaire is directly connected to the national academic network -Renater- and because Lothaire and Renater use the same ATM technology, all the services provided by Renater can be easily transported on Lothaire. StanNet -the private metropolitan academic network of Nancy- is the most important MAN of Lothaire. It can be compared to a "large campus network" interconnecting 19 establishments spread over 45 sites. Four high-end core switches/routers make a Gigabit Ethernet backbone which allows all the sites to be connected together and to the Internet with bandwidth from 100Mb/s to 1Gb/s. VLAN technology and central routing on the backbone provide a highly extensible topology and high security of the network with reduced administration. The Ethernet backbone, which transports the classical IPv4 traffic, is doubled by an ATM backbone with four ATM switches. This backbone supports specific traffic and services from Renater or Lothaire such as IP over ATM, multicast, IPv6, ... and probably in the near future telephone traffic. Lothaire and StanNet are managed by a team of 6 engineers from the C.I.R.I.L who guarantee the connectivity and the good health of the network. This team uses and provides several tools to manage the network : device configuration, IP and level 2 connectivity supervision, security supervision and configuration, metrology and statistics of all the regional traffic... With intent to stay close to network evolutions, this team keeps an eye on new technologies and is involved in national experimental projects. Keywords MAN, optical fiber infrastructure, ATM, Gigabit Ethernet, network/system services

- 2 -

Introduction

Internet access for academic networks in France is based on hierarchical levels : a user who wants to access the Internet will go through different levels of networks. First the user is connected to his local network (site network), after this to a campus network or a metropolitan network, then to a regional network. Finally, the national network is the last door to the Internet. Figure 1 illustrates this hierarchical notion.

Figure 1 : Hierarchical Internet access in France

Lorraine, a region of eastern France, follows this hierarchical model to provide Internet access to its academic users. The size of the Lorraine region and the large number of users explain the presence of a regional network, two metropolitan networks and several site networks. Figure 2 shows a map of France illustrating that Renater, the national academic network of France, federates all regions. At the northeast appears Lorraine and its regional network Lothaire, Le réseau LOrrain de Télécommunications à Haut débit pour les Applications Informatiques de la Recherche et de l’Enseignement supérieur (High bandwidth telecommunication network of Lorraine for research and academic computer applications). Federated by Lothaire, two metropolitan academic networks : StanNet –Stanislas Network- and AmpereNet –Ampère Network- provide network services and Internet access to the two most important cities of the region, Nancy and Metz.

- 3 -

Figure 2 : Hierarchical levels in Lorraine

To give some landmarks, we can present these networks with some key points :

• Renater can be compared to an “Internet service provider” for the universities in France. It differs from a commercial ISP in that it cannot have commercial use and services must be closely related to technology, education and research. Renater interconnects 27 regions of France together and to the Internet through 4 points (START TAP, TEN155 and 2 transatlantic links). Network topology is based on point to point ATM links from Paris to each region.

• Lothaire can be compared to an “Internet service provider” for the Universities of Lorraine. Because it’s connected to the Internet through Renater, it is governed by the same rules concerning use and access. Lothaire interconnects the 2 MAN of Lorraine and 32 smaller peripheral points together and to the Internet through Renater. Lothaire is built around a high bandwidth ATM backbone which connects 6 strategic points (called MAP, Metropolitan Access Point) and classical leased lines that are used to connect less important points. Services provided by Renater can be easily transported on Lothaire because Renater and Lothaire use the same ATM technology inside the network and at their interconnections.

• StanNet and AmpereNet, the metropolitan academic networks of Nancy and Metz, interconnect 26 establishments (19 StanNet, 7 AmpereNet) spread over 58 sites (45 StanNet, 13 AmpereNet) together, to Lothaire and to the Internet through Renater. StanNet and AmpereNet can be compared to “private large campus networks” –this notion will be discussed later- built around a Gigabit Ethernet backbone and, in the case of StanNet, a GE backbone doubled by an ATM backbone.

In the next section of this paper, we will use StanNet as an example to show that the infrastructure and the use of certain technologies – ATM, GE, VLAN, central routing- provide a good extensibility and flexibility of the network and suitable answers to user needs. After a brief presentation of the operating team that manages StanNet –who they are, what missions they have, what they provide-, we’ll treat Lothaire to show that it has a “user dimension” and how user needs can be satisfied on this regional network.

- 4 -

StanNet : Infrastructure

StanNet is known as the “private large campus network” of Nancy. Its topology and the technologies it uses are not really revolutionary but what is really important is to understand how the initial project was undertaken, how it’s deployed and how it’s operated.

We use the term “private” to emphasize the fact that StanNet is financed and operated by and

for the Universities of Nancy. In 1998, the 3 Universities of Nancy came to an agreement to find enough resources –political/financial/technological solutions, common projects, users- to build their own network, from the physical infrastructure to the exploitation. The first challenge was to build an optical fiber infrastructure. To do this, the universities seized the opportunity that the Communauté Urbaine du Grand Nancy –CUGN, a public organization which manages the town planning for Nancy and its surrounding cities and villages- wanted to build an optical fiber infrastructure for its public metropolitan telecommunication network, RMT1. The universities, associated with the CUGN, built their physical infrastructure according to these general terms :

• some optical fibers are entirely owned by the CUGN : they can be exploited, rented or resold

• other optical fibers are financed by the universities and can only be exploited.

The initial project defined 4 converging points and 39 peripheral points. Of course converging points were placed in strategic locations to easily serve peripheral points where users are present.

Figure 3 : StanNet, optical fiber infrastructure

Figure 3 illustrates the optical fiber infrastructure for StanNet. The 4 converging points –CIRIL,

Campus Sciences, Présidence UHP, Pôle Lorraine Gestion- constitute a square connected by optical fiber cables of 24 mono-mode fibers. Peripheral points are served by optical fiber cables of 12 to 84 mono-mode fibers depending on the number of cascading points behind a peripheral point. In some specific cases optical fiber paths with multi-mode fiber were built but don’t appear on this figure. The physical infrastructure make an optical fiber cable path of about 120km in the streets of Nancy.

- 5 -

We use the terms “large campus network” to emphasize the size of StanNet which is a multi-

site, multi-establishment network at the scale of a city. StanNet is today composed of 19 establishments spread over 45 sites. Figure 4 shows all the sites and establishments constituting and connected to StanNet.

Figure 4 : StanNet a multi-site, multi-establishment network

We can also talk about a “large campus network” because StanNet includes :

• 4 main routers • 165 switches (6 GE, 5 ATM, 154 Ethernet) • 250 VLANs declared on the backbone • 230 class C networks owned by the establishments of StanNet

StanNet serves a community of about 50 000 users (students, faculty, researchers) and about

15 000 computers (servers and clients) are permanently connected to StanNet.

During the physical solution studies in 1998, the universities chose a transport technology based on a simple ATM backbone. But with the emergence of the Gigabit Ethernet technology it was rapidly decided to double the initial ATM backbone with a Gigabit Ethernet backbone, and since 1999 StanNet has a double backbone, GE and ATM. We can describe these two backbones in a few words.

The Gigabit Ethernet backbone :

• is composed of 4 high-end core Gigabit Ethernet switches/routers • provides bandwidth from 100Mb/s (peripheral sites) to 1Gb/s (backbone and

peripheral sites) • uses VLAN technology and central routing, providing :

o transport of a VLAN wherever users are on StanNet o routing to other VLAN or to the Internet on central backbone routers

• transports mainly IPv4 traffic, but using VLAN technology it’s very simple to create dedicated VLAN for specific traffic : IPv6, voice over IP, …

- 6 -

The ATM backbone :

• is composed of 4 ATM switches with OC-3 interfaces • provides maximum bandwidth up to 155Mb/s • allows the configuration of dedicated virtual channels/paths providing dedicated links

with reserved resources and Quality of Service capabilities • transports mainly specific traffic : IPv4 over ATM VPN, IPv6, voice over IP, telephone

traffic, …

Figure 5 shows the StanNet network with its double backbone and all the sites connected to the backbone. Figure 5 shows only active devices of the backbone (high-end core GE switches/routers) and site-entry edge devices (classic Ethernet switches). Of course there are other switches behind a site-entry edge device but they don’t appear on this figure. To make a figure of about 165 active devices could be quite difficult !

Figure 5 : StanNet double backbone GE-ATM

All the active devices are devices and Figure 6 illustrates the device series used on StanNet.

- 7 -

Figure 6 : Cisco active devices on StanNet

The extensibility and flexibility of the infrastructure –considering physical and active infrastructure- can be easily demonstrated with 2 examples.

1. If a new client wants to connect to StanNet :

• if this new client site is not in an existing site of StanNet then we have to provide a new optical fiber path. This construction can be easily done because of the strategic locations of converging points or because this new site can be cascaded to an existing peripheral point

• we have to deploy a new switch connected directly to the backbone or through an existing point. This switch is called site-entry edge device

• if this new client can not be linked up to an existing VLAN, then we have create a new VLAN for this new client

• and to finish, to provide inter-VLAN communications and Internet access we have to activate central routing of this new VLAN on backbone routers

2. If an existing client (having his own VLAN) wants to migrate or to have an extension to another

existing site : • if switches or ports are not available in the site, then we have to deploy a new switch

connected like the other switches of the site • and to finish, we just have to transport the VLAN to the new site in order to provide intra-

VLAN and inter-VLAN communications and Internet access. Normally, central routing of the VLAN has already been done

What is important to remember about the StanNet infrastructure is that :

• it’s a “private” infrastructure because of the association between the universities and the urban community (CUGN) to build their own optical fiber infrastructure

• it’s a “large campus” infrastructure because of the multiple sites and multiple establishments at the scale of a city, and because of the number of active devices

• it’s an extensible and flexible infrastructure because of the general deployment of strategic converging points and peripheral points and because of the use of 2 technologies –GE and ATM- which provide the transport of all traffic and the extensibility and flexibility required by users

- 8 -

StanNet : Services

StanNet is a transport network but also a service network. Two main domains of services are provided : network services and system services.

Network services Transport is of course a network service but not only. Level 2 services don’t guarantee level 3

connectivity and so we provide IPv4 and IPv6 connectivity within the MAN, and to the Internet. IPv4 traffic is mainly present on StanNet and its working is quite simple, as illustrated in Figure 7.

Figure 7 : IPv4 routing on StanNet

In fact, StanNet has about 250 VLAN declared on the 4 main routers of the backbone. The

local routing on a router is obvious, but for inter-router routing, the traffic goes through a backbone routing VLAN connecting all the routers of StanNet. Cisco EIGRP routing protocol is used to distribute routing information on the backbone.

For the IPv6 connectivity we use the existing Gigabit Ethernet backbone to transport the IPv6 VLAN whereas we use dedicated IPv6 routers, because IPv6 IOS are still in experimental state. Figure 8 shows the IPv6 deployment on StanNet with the creation of 2 new VLAN : IPv4 backbone admin IPv6 for the IPv4 administration of the IPv6 routers and the IPv6 backbone for the real IPv6 traffic. One can see the dedicated routers for each site that wants to be connected with IPv6.

- 9 -

Figure 8 : IPv6 routing on StanNet

These preceding examples illustrate another network service, that is to say, the extensibility and the scalability of LAN with VLAN. That functionality is mainly due to the use of Gigabit Ethernet technology with its VLAN.

Security on the backbone is assured with ACL –Cisco Access Control Lists- applied to all the VLAN declared on the backbone routers. This provide inter-VLAN security with the capabilities to permit or to deny certain traffic between hosts, subnets for certain protocols, services, … Note that central routing on backbone routers facilitates security configurations.

The last network service is the availability of the services provided by Renater for StanNet. Indeed, because Renater and StanNet use the same ATM technology, it’s very simple for StanNet to receive these services and to redistribute them on StanNet with native ATM or with VLAN. Today Renater provides specific services such as : IPv6, multicast for the FMBone, virtual paths/channels for VPN.

System services The first one is DNS, the Domain Name Service. We manage all StanNet domains except for

a few which are delegated to sites that have enough resources to manage them by themselves. We also do e-mail management and web hosting for certain sites that don’t have the

capabilities to operate their own e-mail server and web server. Today we manage about :

• 2 000 e-mail boxes • 50 web sites

With intent to reduce Internet access bandwidth and to provide safe information, we manage central news service and we provide today about 1 000 000 articles for the StanNet community.

- 10 -

StanNet : Exploitation and operating team

All the infrastructure –physical and active devices- and the services that we have presented, are managed by a team of 5 network engineers and 3 system engineers. This team is located in the C.I.R.I.L : Centre Interuniversitaire des Ressources Informatiques de Lorraine (Inter-university center for computer resources of Lorraine). The team and its resources are shared by the 4 Universties of Lorraine. The CIRIL is not an autonomous center; it’s under the authority of the Université Henri Poincaré, one of the 3 Universities of Nancy.

The CIRIL and its team have some specific missions :

• provide network and system services • guarantee the connectivity and the good health of the network and the system by

doing preventive and corrective maintenance • propose network and system evolution with technological survey through development

and testing projects

When one talks about active devices, one must think about device configuration. The CIRIL team uses the well-known but effective telnet to apply basic device configurations. For more complex operations, we use of course some dedicated scripts to do administration. We have also developed web-based administration to facilitate users’ queries and the processing of their requests. The web-based tools are available for the CIRIL team but also for some “special” users who are authorized to request modifications of services managed by the CIRIL (DNS, e-mail boxes, VLAN, …).

Figure 9 shows an example of the web-based DNS administration tools developed by the

CIRIL team, allowing StanNet “special” users to see DNS bases and to manage their domains.

Figure 9 : Web-based DNS administration

Figure 10 shows an example of the web-based e-mail box administration allowing StanNet “special” users to consult and to modify e-mail boxes managed and hosted by the CIRIL.

- 11 -

Figure 10 : Web-based e-mail box administration

Figure 11 shows an example of web-based active devices administration allowing StanNet “special” users to consult their VLAN configuration (IP addresses, ACL, …) with the availability of archives and real-time notification of ACL violation.

Figure 11 : Web-based active device administration

- 12 -

To guarantee the connectivity and the good health of the network, the CIRIL team does IP and level 2 supervision based on dedicated tools using basic mechanisms. IP supervision is based on a tool called NetUp [netup] which processes pings to a pre-defined list of critical servers (mail, web, news servers) and active devices. Note that, on StanNet, all the active devices are critical !. NetUp provides alerts with e-mail and a X11 and web-based consultation of the current state. To validate routing information and IP routes on StanNet we use the well-known command traceroute.

Level 2 supervision is based on a tool developed by the CIRIL team called SwitchUp that

supervises all the critical ports of the backbone, using SNMP information. Indeed, due to mecanisms like the Spanning Tree Protocol and Cisco Uplink Fast, sometimes a link breakdown can be transparent, but we must nevertheless have an alert because even if the network still works, there is a potential problem. So, the SwitchUp tool supervises all ports of the backbone, all site-entry edge device ports and cascading device ports.

Security configuration and supervision are done with dedicated scripts that we have

developed. Remember that the security is based on Cisco ACL and applied to all the VLAN declared on the backbone. So we use scripts to configure the ACL on routers and to parse syslog information sent by the routers corresponding to ACL violations. For ACL violations, StanNet “special” users can have daily reports or real-time information.

To collect statistics about bandwidth used on the backbone and on the Internet connection we use MRTG [mrtg], Multi-Router Traffic Grapher. Figure 12 shows an example of MRTG statistics for the Internet connection to Renater2.

Figure 12 : MTRG on the Internet access Renater2

But MRTG capabilities are limited to byte statistics and it can not provide statistics for an

identified protocol or an identified service. Moreover, MRTG statistics be can distorted because of the use of new-generation routers working in traffic acceleration mode (Cisco IP Fast Switching, Cisco Express Forwarding, …). For this reason, the CIRIL team has developed a tool called netMET –Network’s METrology- based on Cisco Netflow information. Indeed, Cisco routers are able to send flows information to a server, which can process this to provide statistics. Cisco Netflow does not

- 13 -

generate the problems found with MRTG because it is based on routing information and flow detection, whereas MRTG is based on SNMP counters.

Today netMET is used to collect statistics on the Internet connection to Renater2, and it

provides such as information :

• top n by host and establishment • detailed metrology information with statistics by protocols (TCP, UDP, …) and

services (www, ftp, mail, …) • statistics by establishments similar to MRTG statistics but more accurate • daily, weekly and monthly archives

Figure 13 shows the top10 traffic by establishment available with netMET

Figure 13 : netMET top10 traffic by establishment

Note that netMET is used not only by StanNet and Lothaire but also by other regions of

France, and this tool seems the best current solution for campus/metropolitan/regional network statistics.

- 14 -

Lothaire : a WAN for MAN and sites

After this presentation of StanNet and its infrastructure, services and exploitation, we would like to treat the regional network Lothaire and its “user dimension”.

Lothaire is a high bandwidth wide area network providing network transport and services for 2 MAN –StanNet and AmpereNet- and 32 peripheral points. Bandwidths available on Lothaire are from 2Mb/s for peripheral points with leased lines to 102Mb/s on the ATM backbone connecting the 6 strategic points (called MAP, Metropolitan Access Point).

Lothaire is financed and operated by and for the Universities of Lorraine. : 3 universities at Nancy and 1 university at Metz. The Lothaire project is quite similar to StanNet : the Universities of Lorraine worked together to build their own regional network. But whereas StanNet has its own “private” physical infrastructure, Lothaire uses leased lines and optical fibers on the ATM backbone leased from France Telecom, the public telecommunication provider of France. Lothaire is entirely operated by the CIRIL and its team of network and system engineers.

So what are the differences between StanNet and Lothaire exploitation ? In fact there are few, despite the difference in infrastructure :

• all the active devices are owned by the universities • a new site can be rapidly connected

o StanNet : a new fiber path –if necessary-, a new switch and a new VLAN o Lothaire : a new leased line –if necessary-, a new router and a new point to

point interconnection • IPv4 network service is available • IPv6 network service is provided on StanNet but not yet deployed on Lothaire. But it

could be done simply with the use of virtual ATM paths/channels and dedicated routers or with IPv4 tunneling

• native VLAN technology can only be provided on StanNet, although VLAN can be transported on Lothaire with bridging technology on routers

• ACL-based security is used • all the system services –DNS, e-mail, web, news- are available • these networks are operated by the same team of engineers : CIRIL team • operating tools and methods are the same on StanNet and Lothaire

Lothaire includes today :

• 5 ATM links, 7 Transfix links at 2Mb/s, 18 HDSL links at 2Mb/s • 31 routers, 8 central switches (2 LS1010 and 6 France Telecom switches) • about 20 000 computers, servers and clients permanently connected (including 15

000 for StanNet)

Figure 14 shows the entire Lothaire network. Of course StanNet and AmpereNet don’t appear, nor does the local network behind each peripheral router, but what’s important in this figure is the configuration of the 6 strategic MAP around the ATM backbone and the distribution to peripheral points with leased lines and routers.

- 15 -

Figure 14 : Lothaire, the regional network of Lorraine

Lothaire is a regional network, but we can say that it has a “user dimension” because unlike typical regional networks, which federate only MAN or campus, networks, on Lothaire sites can be directly connected to the network. Moreover, Lothaire provides the same functionality as a MAN with the same exploitation. Even if Lothaire is a network at the scale of a region, the CIRIL team keeps listening to the regional user needs and therefore can bring necessary evolution to the network on a day-to-day basis. A good example is the bandwidth increase planning illustrated by Figure 15 and the current projects like IPv6, video conferencing, voice over IP, video on demand for the region.

Figure 15 : Lothaire ATM backbone bandwidth planning

- 16 -

Conclusion

StanNet and Lothaire are networks with a “user dimension”. They try to take care of user needs and to respond as accurately as possible to these needs. StanNet and Lothaire have physical infrastructures and services that are designed to be as flexible and extensible as possible. This flexibility and extensibility are assured by the entire centralized control of the active devices on both StanNet and Lothaire. Indeed, StanNet with its Gigabit Ethernet backbone and Lothaire with its ATM backbone and leased lines peripheral distribution, provide the capabilities to interconnect easily new sites with all the associated network and system services. The good health of the network and system is not enough, that’s why the operating team must also look for new technologies and try to suggest network adaptations and evolutions.

Today, we can not divide networks and users “lives” because they are closely linked. Users

bring about the emergence of new needs and they have important effects on networks. And of course network exploitation must take care of these user requests and effects. Our metropolitan and regional experiences lead us to say that a homogeneous operation of the network guarantees this interaction between a network and users, and if this interaction is preserved, it promises a long and successful life for the network.

- 17 -

Links CIRIL - http://www.ciril.fr Université Henri Poincaré - http://www.uhp-nancy.fr StanNet network - http://www.stannet.net Lothaire network - http://www.lothaire.net Cisco - http://www.cisco.com [netup] Netup - http://www.pasteur.fr/recherche/unites/sis/netup [mrtg] MRTG - http://ee-staff.ethz.ch/~oetiker/webtools/mrtg