team 11 - jagadeesh dhandapaani - apr 25 2014 542 pm - group11 viability of deploying ebgp as an igp

24
VIABILITY OF DEPLOYING eBGP AS IGP IN DATACENTER NETWORKS Chavan, Prathamesh Dhandapaani, Jagadeesh Kavuri, Mahesh Babu Mohankumar, Aravind Faculty Advisors: Jose Santos & Mark Dehus “A capstone paper submitted as partial fulfilment for the degree of Masters in Telecommunications at the University of Colorado at Boulder “

Upload: an-uj

Post on 09-Dec-2015

3 views

Category:

Documents


0 download

TRANSCRIPT

VIABILITY OF DEPLOYING eBGP AS IGP IN DATACENTER NETWORKS

Chavan, Prathamesh

Dhandapaani, Jagadeesh

Kavuri, Mahesh Babu

Mohankumar, Aravind

Faculty Advisors: Jose Santos & Mark Dehus

“A capstone paper submitted as partial fulfilment for the degree of Masters in Telecommunications at the University of Colorado at Boulder “

Viability of implementing eBGP as IGP in datacenters 2

ABSTRACT:

Advent of cloud based infrastructure and continuous growth of web based services has motivated

service providers to set up large scale Datacenters (DC). Traditionally, DCs utilize Border

Gateway Protocol (BGP) to route the customer traffic to and from the Internet. Link state protocols

such as Open Shortest Path First (OSPF) and Intermediate System-Intermediate System (IS-IS)

are used for routing the traffic within DC. The disadvantages of OSPF and IS-IS are: they require

large number of CPU cycles to process the link state updates and maintain an identical link state

database within the area; these protocols also impose a challenge due to the necessity of re-

designing DC routing architecture when the number of network devices increase. Few large scale

DC operators have taken a novel approach in the routing design by adopting exterior Border

Gateway Protocol (eBGP) as an Interior Gateway Protocol (IGP). The significant advantages of

using eBGP as an IGP are:

i) It is a stable and robust routing protocol consuming less number of CPU cycles

ii) It can handle large number of routes without having to re-design the network

iii) It is simple to operate and troubleshoot

Our research will focus on the feasibility, architecture and the impacts of implementing eBGP as

an IGP in large scale DC environments. We will simulate this environment with eBGP as the only

routing protocol and compare the network performance against traditional IGP’s. We will also

analyze the impact of the fundamental datacenter services such as seamless mobility for virtual

machines, extension of Layer 2 adjacency (VLANs) within the datacenter and load balancing

applications critical for the operation of high density server environments. The goal of our

research is to evaluate the incentives and the viability for large scale datacenters to migrate to an

e-BGP only design.

Viability of implementing eBGP as IGP in datacenters 3

Table of Contents

I. Introduction .......................................................................................................................... 4

Statement of the Problem ........................................................................................................ 5 Research Problem .................................................................................................................... 6

Research Sub-problems ........................................................................................................... 6 II. Literature Review................................................................................................................. 7

Traditional Datacenter topology .............................................................................................. 7 Advantages of using CLOS design .......................................................................................... 8

Choosing the appropriate routing protocol to be used in the Datacenter ................................ 9

Impact on legacy datacenter applications .............................................................................. 10

Current state of the art technology ........................................................................................ 10

III. Research Methodology ...................................................................................................... 11

Layer 2 Performance Analysis .............................................................................................. 12

Layer 3 performance analysis OSPF vs BGP ........................................................................ 12 IV. Research Results ................................................................................................................ 17

Layer 2 Performance evaluation ............................................................................................ 17

Layer 3 performance results under normal operations .......................................................... 18

Layer 3 performance analysis during convergence ............................................................... 19

V. Discussion of Research results: ......................................................................................... 19

VI. Conclusion and Future work: ............................................................................................. 20

VII. Appendix ............................................................................................................................ 22

VIII. References ........................................................................................................................ 23

Viability of implementing eBGP as IGP in datacenters 4

I. Introduction

The growing demand for virtualization and cloud based services along with traditional network

practices has created challenges in designing large scale datacenters (DC), which typically includes

more than 20,000 Virtual Machines and nearly 2000 networking devices [1]. Following are some

of the significant requirements of a large scale DC [2]:

i) Selecting an appropriate topology that supports the increasing demand for server to

server communication (east-west traffic) within the Datacenter, by adding more

number of links and cheap commodity devices with identical port density.

ii) Choosing a single control plane protocol (if possible) that could handle large number

of routes, eliminates the need for inter-operability with other control plane protocols

and are supported by most of the network equipment vendors.

iii) Choosing a routing protocol that has a simple implementation in terms of the protocol

design and ease of operational support.

iv) Reduce the scope of failure caused by equipment or protocol issues.

Designing the control plane architecture is the most challenging part. Control plane is a critical

component of the router or a switch architecture that has sufficient intelligence to build the network

map (dynamically or statically) and determines how an individual network device interacts with

its neighbors. It is the signaling component of the network. Few of the control plane functions

include management/configuration of the network device and exchange of routing information.

Control plane protocols are essential for efficient forwarding of the data traffic between any two

devices. Multiple Spanning Tree (MST) may be considered as the control plane protocol on a pure

Layer 2 environment, whereas OSPF and BGP may be considered as the control plane protocol

used in a Layer 3 environment.

Viability of implementing eBGP as IGP in datacenters 5

CLOS topology is preferred in large scale DCs as this topology can be horizontally scaled by

increasing the port density in the devices or by adding similar devices to the topology. This saves

the cost involved in buying high performance devices and avoiding the re-design of the network

[3]. In a large scale DC, the cost of the networking infrastructure amounts to 10-15% of the entire

Capital Expenditure (CAPEX) [4]. There is a constant drive to cut down costs and therefore DCs

prefer networking elements of the same hardware type with minimal features [2]. Traditionally

datacenters utilize Border Gateway Protocol (eBGP), a distance vector protocol, on Wide Area

Networks(WAN) to route the traffic to and from the Internet and link state protocols such as OSPF

and IS-IS in the Local Area networks (LAN) [5] to efficiently route the traffic within the

datacenter. Having two control plane protocols within a DC increases the probability of network

misconfiguration and poses challenges in making both the protocols interact with each other,

thereby demanding a design with simpler control plane protocol that reduces the chance of network

failure [2]. A single control plane protocol makes the network design simple and easy to

troubleshoot; which is one of the fundamental requirements of a large scale data center.

Statement of the Problem

BGP is mostly viewed as the exterior routing protocol used for routing traffic between networks

managed by different administrative bodies. The significant reasons for using BGP are; it provides

a better control over the traffic flow because of its built-in attribute set, it can handle a large number

of routes because of its limited flooding scope and simpler protocol design [6]. Link state Protocols

such as OSPF and IS-IS are designed for routing traffic within the same administrative domain; as

they have a complete view of the network topology within the area and can converge faster [5].

However, as the number of network devices and routes continues to increase, it creates the

following scalability challenge in the network design:

Viability of implementing eBGP as IGP in datacenters 6

a. The scope of Link State Advertisements (LSA) flooding is the entire area which causes

all the network devices within the area to process all the LSAs generated and compute

SPF algorithms. The above processes may consume large number of CPU cycles on a

network device as the instability in the network and number of network devices

increase.

The most widely implemented solution to address the above challenges is to re-design the network

by segmenting a single area into multiple areas, so as to reduce the processing load on the network

device [2]. Re-designing DC network requires careful planning, scheduled network outages,

possibility of misconfiguration and administrative overhead. Currently, the IETF group is working

on standardizing an alternate and innovative solution to the above challenges by deploying BGP

as an IGP, instead of traditional OSPF and IS-IS [2]. The above routing architecture is the basis of

our research in analyzing the practicality and viability of deploying eBGP as an IGP in large scale

Datacenters.

Research Problem

What are the incentives for firms operating large scale datacenters to migrate to an eBGP only

design? Although the design permits, what is the viability of implementing eBGP as an IGP? Does

the new design of deploying BGP as the only control plane protocol affect any existing services

and applications offered by datacenters?

Research Sub-problems

i. Simulating an optimally designed large scale DC network to analyze the viability of

deploying eBGP as an IGP

ii. Network Performance evaluation of Layer 2 and Layer 3 control protocols

a. Layer 2 performance analysis

Viability of implementing eBGP as IGP in datacenters 7

b. Layer 3 performance analysis (OSPF vs. BGP)

iii. Analyzing the possible impacts of deploying eBGP as an IGP

II. Literature Review

Traditional Datacenter topology

Traditional DC environment utilizes a mix of Ethernet based Layer 2 technologies with three levels

of hierarchy namely access, aggregation and core layers [7]. A typical DC network consist of a

hierarchical tree like topology with progressively sophisticated high performance network devices

at the core layer of the network and commodity network devices at the access layer of the network.

These traditional datacenters utilize Spanning Tree Protocols between the access and distribution

layer and link state routing protocols between the distribution and core layer. BGP is used for

providing external connectivity for the DC. The use of a Layer 2 solution at the access layer causes

redundant uplinks on the access layer switches to be blocked by the Spanning Tree Protocol (STP)

to avoid Layer 2 loops; this effectively reduces the net utilization of the up-links [3] because of an

active-passive approach. The advent of Multi-Chassis Link Aggregation protocol (M-LAGs)

provides an active-active approach and offers increased throughput. However, most of the M-LAG

implementations are proprietary and have a little multi-vendor support [2]. The need of STP in a

flat Layer 2 network can be avoided with the introduction of TRILL that addresses most of the

challenges faced by STP. However, TRILL requires new network devices and has limited

implementation experiences. Finally, both TRILL and M-LAG does not eliminate the primary

concerns of a shared broadcast domain such as: large amounts of broadcast and unknown-unicast

storms [2].

Viability of implementing eBGP as IGP in datacenters 8

Advantages of using CLOS design

Until recently, the traffic flows were commonly North-South traffic (enter and exit the datacenter)

and tree-topology served the requirements of these type of traffic with high oversubscription ratios

and high density line cards on the core network devices. But, recently many large scale DC host

applications that require distributed computing across the servers, causing large amount of East-

West traffic flow. The traditional tree topologies do not scale horizontally to match these

bandwidth demands. The fat-tree topology (also known as CLOS or leaf-spine topology), on the

other hand, uses similar type of commodity Ethernet switches to provide cheaper scalable

interconnection bandwidth between the servers. The fat tree topology can be easily scaled by

adding more number of stages or increasing the port density of the network element as shown in

the figure below:

Figure 1.a: Two stage CLOS topology to support 4 server subnets

Figure 1.b: Scaling CLOS topology

Viability of implementing eBGP as IGP in datacenters 9

The above discussed CLOS topology presents a new interconnect architecture that leverages

commodity network elements and meets the scalability, bandwidth and cost requirements of a large

scale datacenter. Figure 1.a shows a two stage fully non-blocking CLOS topology to support four

servers. The CLOS topology can be scaled to support eight servers by either increasing the port

density on the two stage CLOS topology or by moving to a three stage CLOS topology as shown

in figure 1.b.By appropriate choice of routing protocol, that inherently supports load balancing,

the hosts in the network can communicate at the full bandwidth of its local interface.

Choosing the appropriate routing protocol to be used in the Datacenter

IP routing is generally used at the core layer and above. However, Next-generation datacenters

incorporate IP routing intelligence to the access – distribution layers to have efficient utilization

of the uplinks, improved network stability and scalability [7]. It also confines Layer 2 broadcast

domains and thereby significantly reducing the OPEX. Usually the layer 3 routing protocol is

either OSPF or IS-IS, as it is assumed to provide faster convergence and better loop avoidance

features in comparison to their distance vector counterparts [5]. However, as the DC grows in size

and scale, the existing IGPs had to be constantly revisited and re-designed to reduce the amount

of processing required on the routers within an area.

Lapukhov from Microsoft and Premji from Arista Networks have recently identified several

problems with the existing IGPs [8]. They are actively working with the IETF group in proposing

a novel design to utilize BGP as the only IGP in large-scale datacenter environments because of

the following advantages [2]:

i. A network device running BGP advertises only the best path available to its directly

connected neighbors and is less prone to flooding events, which makes it more scalable

without the need to re-design [9].

Viability of implementing eBGP as IGP in datacenters 10

ii. BGP is the most widely implemented WAN protocol in a datacenter environment and

extending its implementation in the LAN would simplify the operation overhead.

iii. BGP has several attributes built-in to control the flow of routing information. It also

supports third party resolved next hops that allow unequal cost load balancing and

application defined forwarding paths [10].

iv. BGP is simple to troubleshoot when compared to OSPF because of the availability of

adjacent Routing Information Base – In (RIB-In) and Routing Information Base – out

(RIB-out) structures with the corresponding Network Layer reachability Information that

can be easily correlated [10].

Impact on legacy datacenter applications

Incorporating Layer 3 intelligence close to the access layer will break the layer 2 adjacency

requirements of large scale DCs. There are several critical Datacenter applications such as

seamless mobility for virtual machines, application load balancing using Layer 2 DSR architecture

that require layer 2 adjacency for their operation.. However with the advent of several Layer 2

tunneling protocols such as GRE tunnel, L2TPv3, OTV and VXLAN, these services can still be

achieved by extending the Layer 2 domain across the Layer 3 backbone. The pros and cons of

these tunneling protocols will be discussed in the appendix section.

Current state of the art technology

The current research by IETF does not compare the performance of OSPF and BGP in a large scale

DC environment and evaluate the incentives for firms to migrate to an e-BGP only design. Our

research will attempt to extend the current IETF’s work and analyze the viability of deploying

BGP as an IGP in datacenter networks and the incentives to migrate to the “eBGP-only” design

by

Viability of implementing eBGP as IGP in datacenters 11

i) Comparing the CPU utilization of OSPF and BGP during normal operations and re-

convergence

ii) Comparing the convergence time of OSPF and BGP

iii) Exploring the most scalable Layer 2 tunneling solutions to address the Layer 2

adjacency requirement of critical datacenter applications.

III. Research Methodology

Our research evaluates the feasibility of deploying eBGP as an IGP in large scale Datacenter

networks. The first sub-problem of our research was to simulate this environment in a lab; while

real world datacenter networks consist of large number of network devices, we replicated a small

subset of DC network to conduct our measurements.

Fig 2.a Data plane Fig 2.b Management plane

We implemented the above specialized and scalable network topology (Fig2.a) that satisfies the

first requirement of the large scale datacenters as mentioned in Section I [3]. This topology is

broken down into three layers: core (Tier 1), distribution/aggregation (Tier 2) and access (Tier 3)

layers. The above network topology was built using 12 Cisco 3550 multi-layer switches, and 1000

Viability of implementing eBGP as IGP in datacenters 12

loopbacks were added on each Tier-3 device to simulate large number of servers. Next, to prevent

the production traffic being affected by the management traffic, we implemented a separate data

plane and management plane as shown in Figure 2.a and 2.b. The management plane was used to

monitor and collect the network performance statistics through automated scripts.

Layer 2 Performance Analysis

We conducted a layer 2 performance analysis by considering three factors: CPU utilization, delay

and bandwidth. A layer 2 network was simulated by configuring Multiple Spanning Tree (MST)

in the test bed. We simulated instability in the network to test the MST convergence and measure

the CPU utilization while the network converged. Next, we simulated traffic flows from one cluster

to another to measure the delay and bandwidth in the network under various levels of broadcast.

Layer 3 performance analysis OSPF vs BGP

The following section describes the design and performance analysis of the layer 3 network with

OSPF and BGP as the only control plane protocol.

OSPF Design and optimization:

Figure 3: IGP design using OSPF

Viability of implementing eBGP as IGP in datacenters 13

The network consisted of 12 devices in a single OSPF area (Area 0). We optimized OSPF to

achieve faster convergence and utilize less CPU cycles by incorporating the following measures:

a. Point-to-Point networks – To avoid Designated Router (DR) elections

b. Reduced Hello timers- To achieve faster re-convergence

c. Incremental SPF (ISPF)- To reduce the CPU utilization for SPF computations

d. Throttle timers- To achieve Faster convergence by reducing the time between SPF

computations and LSA flooding

All the above features are supported by most of the commodity switches, and hence satisfies

requirement II of the large scale datacenters as mentioned in Section 1.

Performance analysis of OSPF:

We tested the CPU utilization under two states: normal operations (stable) and during re-

convergence (unstable). For both the states, we simulated two scenarios: one with 8 device CLOS

topology and other with 12 device CLOS topology to check whether the performance degrades

with more number of devices. The following table illustrates the methodology used to conduct

performance analysis for both states:

State Reason for CPU utilization Methodology Normal operations

Link state database (LSDB) refreshed every 30 minutes (LS Refresh time).

CPU utilization from all network devices was recorded every 30 minutes

Re-convergence

Flapping the server subnets (Loopbacks in Tier 3 devices)

CPU utilization from all network devices was recorded till the network converged (for approx. 2 minutes).

Flapping the transit links

Viability of implementing eBGP as IGP in datacenters 14

IGP Network design using eBGP:

The following diagram illustrates the BGP design scheme used in our network topology:

Figure 4: IGP design using eBGP

Internal Routing:

This section describes the eBGP design scheme used for internal routing. External BGP (eBGP)

peering sessions were established over point-to-point links between all network devices. All the

devices in Tier-1 were assigned a unique Autonomous System Number (ASN), each group of Tier

2 devices in a cluster was assigned a unique ASN, and each Tier 3 device within a cluster was

assigned a unique ASN as shown in figure 4. The 16bit ASNs allows the use of only 1023 unique

private ASNs within a datacenter. Since, large scale datacenters may have more than 2000 network

devices, the ASNs will have to be re-used at the Tier 3 level. Considering that ASN’s are used in

BGP as a loop avoidance feature [11]; the reuse of ASN’s at the Tier 3 level will prevent routes

advertised from a tier 3 device in one cluster being accepted by a tier 3 device in another cluster.

In order to overcome this challenge, we utilized “Allow AS In” feature of BGP which allows a

BGP enabled device to accept routes that contain its own ASN in the AS-PATH attribute. This

feature is mostly supported on all commodity network devices [12]. It is assumed that the

Viability of implementing eBGP as IGP in datacenters 15

Datacenter network topology is symmetrical and does not cause routing loops when implementing

this feature.

External routing:

The above BGP design routes the traffic within the datacenter, however doesn’t provide Internet

access. To do so, we have used another border cluster as shown in Figure 4. The Edge devices of

the border cluster peer with the ISP and the core routers of the Border cluster peer with the Tier-1

devices of the datacenter. As private ASNs were used inside the datacenter; we use the “Remove

private-as” feature in the border cluster to strip off the private AS numbers when advertising the

Datacenter routes to the Internet.

BGP optimization:

As BGP is a stable routing protocol consuming less CPU cycles, we modified BGP to achieve faster

convergence by incorporating the following measures:

a. Reduced the default BGP advertisement interval value to zero so that a BGP peer exchanges

the routing updates immediately. Even though the signaling overhead increases, it does not

cause a considerable effect on the performance

b. BGP, by default does not install multiple paths in the routing table. In order to utilize

redundant paths, we enabled “maximum paths” feature available in BGP.

BGP Performance evaluation:

Performance evaluation for BGP was carried out in a similar manner as OSPF performance

evaluation in order to achieve comparable results. The following table illustrates the methodology

used to conduct performance analysis for both states:

Viability of implementing eBGP as IGP in datacenters 16

State Reason for CPU utilization Methodology Normal operations

BGP scanner process that runs every 60seconds by default

CPU utilization from all network devices was recorded after every minute.

Re-convergence

Flapping the server subnets (Loopbacks on Tier 3 devices)

CPU utilization from all network devices was recorded till the network converged (for approx. 2 minutes).

Flapping the transit links

Convergence testing for OSPF and BGP:

In order to test the convergence time of BGP and OSPF, continuous traffic was sent from a server

connected to a Tier-3 device ( S9) in cluster 1 to a server connected to another Tier -3 device (S12)

in cluster 2. The worst case scenario (in terms of processing overhead and convergence) for both

the protocols happens when (Refer figure 5):

a) We disable a redundant link L1, and force all the traffic through L2

b) We again enable link L1 and disable link L2 immediately.

The above scenario was simulated, because the convergence depends on the neighbor

establishment, advertisement of routes or link state packets and calculation of the best path.

Figure 5: Network performance evaluation

Viability of implementing eBGP as IGP in datacenters 17

IV. Research Results

In this section we present the performance evaluation of MST, OSPF and BGP, and briefly

describe our results with graphs wherever necessary.

Layer 2 Performance evaluation

We observed that the CPU utilization of all the network devices was under 2% (for both stable and

unstable cases) with MST and we were able to achieve convergence under one second in worst

case scenarios The following figure illustrates the relationship between the amount of broadcasts

and delay in a flat Layer 2 network.

Figure 6: Layer 2 Performance evaluation

0

5

10

15

20

25

0 5 10 15 20 25 30 35 40 45

Traf

fic D

elay

(Sec

onds

)

Percentage of Broadcasts

Performance of a Layer 2 network

Viability of implementing eBGP as IGP in datacenters 18

Layer 3 performance results under normal operations

Figure 7a: CPU Utilization of OSPF under normal operations

Figure 7b: CPU Utilization of BGP under normal operations

Figure 7.a and 7.b show the CPU utilization of OSPF and BGP under stable conditions. From the

above graphs, it can be observed that the CPU utilization of OSPF increases as the number of

network devices increase. However, it can be observed from the graph that the CPU utilization of

BGP stays constant with the addition of more number of network devices.

010203040506070

0 20 40 60 80 100 120

Perc

enta

ge C

PU U

tiliza

tion

Number of Nodes

OSPF CPU utilization in stable conditions

Tier - 1 Tier - 2

Number of Routes = 4000

00.5

11.5

22.5

3

0 20 40 60 80 100 120

Perc

enta

ge C

PU U

tiliza

tion

Number of nodes

BGP CPU utilization in stable conditions

Tier - 1 Devices Tier - 2 Devices

Number of Routes = 4000

Viability of implementing eBGP as IGP in datacenters 19

Layer 3 performance analysis during convergence

Figure 8: Performance metrics of OSPF and BGP during convergence.

From the figure 8, it can be observed that the CPU utilization of OSPF increases to a maximum

value of 56% (approx.) during convergence. Similarly it can be also observed that the CPU

utilization of BGP increases to a maximum value of 37% (approx.) during convergence.

V. Discussion of Research results:

The CPU utilization and convergence of a Layer 2 network with MST was comparable with the

Layer 3 network performance. However, broadcasts are inherently present throughout the Layer 2

network and this effectively reduces the network capacity and increases the delay in the network.

It also consumes the CPU cycles of all the hosts connected in the DC, as all the hosts present in

the same broadcast domain will have to process the broadcast traffic. Therefore, as the number of

hosts in the Datacenter network increase, the flat Layer 2 design does not scale well.

From the Layer 3 performance analysis tests, we observed that the CPU utilization of BGP was

better than the CPU utilization of OSPF under stable conditions and also during network

0

10

20

30

40

50

60

OSPF - Tier 1 OSPF - Tier 2 OSPF - Tier 3 BGP - Tier 1 BGP - Tier 2 BGP - Tier 3

9.2815.45

12.24

1.48

9.134.71

27.1

56.5

29.74

11.9

37.8

19.6

Perc

enta

ge o

f CPU

util

izatio

nCPU utilization of OSPF and BGP during convergence

Average utilization over 60 sec Average utilization over 5 sec

Viability of implementing eBGP as IGP in datacenters 20

convergence. Inherent properties of link state protocols such as SPF computation complexity and

periodic link state updates cause the CPU utilization to increase as the number of network devices

increase. BGP being a distance vector protocol is not impacted by the above factors.

Normally, the convergence of link state protocols is better than distance vector protocols [5].

However, based on our performance test results, we were able to achieve equivalent convergence

time for BGP by optimizing it as discussed in section IV.

VI. Conclusion and Future work:

We first determined that a flat Layer 2 network performance in terms of CPU and convergence

was comparable with the Layer 3 network performance. However, we also identified that as the

size of the network increases, the Layer 2 design will have high amount of broadcasts which would

cause delays and affect the overall throughput of the network.

As we moved towards a Layer 3 approach, we found that the performance of OSPF degrades as

the number of network devices increases; this performance degradation can be addressed by re-

designing OSPF in to multiple areas. However, the redesigning approach requires planning and

scheduled maintenance often, as the network scales. The use of eBGP as the IGP overcomes the

above design challenges and does not require datacenter operators to re-design the complete

routing architecture. Further, our results also showed that the CPU utilization of BGP is better than

OSPF, and BGP can be optimized to achieve comparable convergence time as OSPF.

We also found that a fully routed design with BGP/OSPF breaks the layer 2 adjacency, which is

required for high availability services in a datacenter. However, we found out that several overlay

technologies like GRE tunneling, L2TPv3, OTV and VXLAN can be used to achieve layer 2

adjacency over a fully routed design.

Viability of implementing eBGP as IGP in datacenters 21

In summary, our research shows that using eBGP as an IGP provides several operational

advantages such as: single control plane protocol within the network which is easy to operate and

troubleshoot, avoids network misconfigurations due to inter-operability of different control plane

protocols, and capability to handle large number of routes without having to re-design the routing

architecture. The above advantages provide large scale datacenter operators an incentive to migrate

to an eBGP only design. Further, our approach of using a single control plane protocol within the

network simplifies the migration steps towards Software Defined Networking.

Viability of implementing eBGP as IGP in datacenters 22

VII. Appendix - I

Overlay Technologies

The eBGP as an IGP design would have a complete Layer 3 design in the datacenters. There are

critical applications such as Virtual machine migration, Layer 2 direct server return, etc. which

require Layer 2 adjacency. This requirement can be satisfied by use of overlay technologies such

as Layer 2 Generic routing encapsulation [GRE], Layer 2 transport protocol [L2TP], Overlay

transport virtualization [OTV] and Virtual extensible local area network [VXLAN].

The above mentioned overlay technologies were analyzed to ensure ease of deployment and

scalability. The advantage of GRE and L2TP is that they are easy to deploy and have less

configuration overhead, but each of them create point to point tunnels and therefore become

complex to deploy as the network scales.

The recently developed overlay technologies such as OTV and VXLAN are scalable as they create

dynamic tunnels to traverse through the routed access layer. The performance and configuration

overhead between OTV and VXLAN is comparable. OTV primarily addresses overlaying layer 2

over layer 3 whereas VXLAN also addresses the shortage of vlan ids (only 4094) which is a

realistic issue with increasing scale of datacenters. Also, OTV is a Cisco proprietary protocol

which would necessitate all the devices in the Tier 3 to be cisco devices whereas VXLAN is

currently in IETF draft stage with wide vendor support.

Appendix – II

Further information relating to the implementation and configurations used for various tests conducted can be found at:

https://docs.google.com/document/d/1gl5cW9zeh3HDbwLrbWNs2Ejq8iKARRpTRPwcfOAFTSs/edit?usp=sharing

Viability of implementing eBGP as IGP in datacenters 23

VIII. References

[1] Sengupta, S. (2013, August 16). Data center networking. Retrieved from

http://conferences.sigcomm.org/sigcomm/2013/ttldc.php

[2] Lapukhov, P. L., & Premji, A. P. (2013). Using BGP for routing in large-scale data

centers. IETF. Retrieved from

http://tools.ietf.org/id/draft-lapukhov-bgp-routing-large-dc-06.txt

[3] Al-Fares, M., Loukissas,A., & Vahdat,A. (2008). A scalable, commodity data center network

architecture. ACM SIGCOMM Computer Communication Review, 38(4), 63.

[4] Greenberg, A., Hamilton, J., Maltz, D. A., & Patel, P. (2008). The cost of a cloud: research

problems in data center networks. ACM SIGCOMM Computer Communication Review, 39(1), 68–

73.

[5] Carroll, J. D., & Doyle, J. (2005). Routing TCP/IP: Vol. 1. Indianapolis, Ind: Cisco Press.

[6] Carroll, J. D., & Doyle, J. (2012). Routing TCP/IP: Vol. 2. Indianopolis, Ind: Cisco Press.

[7] Cisco Data Center Infrastructure 2.5 Design Guide. (2007, December 6).

Retrieved August 12, 2013, from

https://www.cisco.com/application/pdf/en/us/guest/netsol/ns107/c649/ccmigration_09186a00807

3377d.pdf

[8] Lapukhov, P. (2012, June 4). Nanog 55. Retrieved September 20, 2013, from

http://www.nanog.org/meetings/nanog55/presentations/Monday/Lapukhov.pdf

Viability of implementing eBGP as IGP in datacenters 24

[9] Halabi, S., & McPherson, D. (2001). Internet routing architectures. Indianapolis, IN: Cisco

Press.

[10] Hankins, G. (2013, October 10). BGP as a Data Center IGP - Brocade Community.

Retrieved October 20, 2013, from

[11] Fernando, R., Kumaki, K., McPherson, D., Patel, K., & Raszuk, R. E. (2012). Distribution

of Diverse BGP Paths. IETF. Retrieved from http://tools.ietf.org/html/rfc6774

http://community.brocade.com/t5/Service-Providers/BGP-as-a-Data-Center-IGP/ba-p/649

[12] Walton, D., Retina, A., Chen, E., & Scudder, J. (2009). IETF. Advertisement of Multiple Paths

in BGP, 06. Retrieved from

http://tools.ietf.org/html/draft-walton-bgp-add-paths-06