disaster recovery design with cisco application centric infrastructure

White Paper

Disaster Recovery Design with Cisco Application Centric

Infrastructure

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. of 46


Contents

Overview ................................................................................................................................................................... 3

Naming Conventions, IP Addresses, and VLANs ................................................................................................. 4 Naming Conventions ............................................................................................................................................. 4 IP Addresses ......................................................................................................................................................... 6 VLANs ................................................................................................................................................................... 8

Design Requirements .............................................................................................................................................. 8 Tenant DMZ ........................................................................................................................................................ 10 Tenant Server Farm ............................................................................................................................................ 10 Traffic Flow ......................................................................................................................................................... 10

Internet Traffic Flows ...................................................................................................................................... 10 DCI Traffic Flows ............................................................................................................................................ 12 Internet Access through Disaster Recovery ................................................................................................... 13

Disaster Recovery Topology and Service Flows ................................................................................................ 13 Leaf and Spine Connectivity ............................................................................................................................... 14 Server Connectivity with Leaf Switches .............................................................................................................. 15 Layer 4 Through 7 Device Connectivity to Leaf Switches ................................................................................... 16 Cisco ASR Router WAN Connectivity ................................................................................................................. 17 Cisco APIC Connectivity ..................................................................................................................................... 17 External Networking ............................................................................................................................................ 18

External Layer 2 Networks ............................................................................................................................. 18 Service Architecture Design ................................................................................................................................ 20

Firewall as the Gateway (NLB) ....................................................................................................................... 21 Load Balancer in One-Armed Mode ............................................................................................................... 25

Traffic Flow ............................................................................................................................................................. 27 Load Balancer as the Gateway ...................................................................................................................... 31

Services Integration............................................................................................................................................... 34 Service Device Packages ................................................................................................................................... 34

Automated Service Insertion .......................................................................................................................... 35 Cisco ASA Integration with Cisco ACI ................................................................................................................. 36 F5 Integration with Cisco ACI .............................................................................................................................. 36

Virtual Machine Networking .................................................................................................................................. 37 VMware vSphere Integration ............................................................................................................................... 37 VMM Domain Configuration ................................................................................................................................ 38

Management Network in Cisco ACI ...................................................................................................................... 40 Out-of-Band Management Network .................................................................................................................... 41

Simple Network Management Protocol .......................................................................................................... 43 Syslog ............................................................................................................................................................ 44 Network Time Protocol ................................................................................................................................... 45

Conclusion ............................................................................................................................................................. 45

For More Information ............................................................................................................................................. 45


Overview

One of the largest universities in the Europe, Middle East, Africa, and Russia (EMEAR) region is building a

completely new (greenfield) disaster recovery site to meet service-level agreement (SLA) requirements for critical

applications and services.

Cisco® Application Centric Infrastructure will be used to deliver these main services:

● Hosted applications for all faculty

● Research and development services for students

● Security, monitoring, and capacity planning

● Data center disaster recovery site capabilities

● Full benefits of F5 Layer 4 through 7 load-balancing capabilities

● Security enforcement for the application services using the Cisco Adaptive Security Appliances (ASA)

firewall

Figure 1 shows the building blocks of the network infrastructure. The existing primary IT services (ITS) data center

is connected to the disaster recovery site through a data center interconnect (DCI). The Internet block at the

disaster recovery site will be used for backup; in the event that the primary data center Internet goes down, the

disaster recovery site Internet will be used. The ACI fabric will connect the servers, Layer 4 through 7 services, and

the Internet and WAN blocks.

Figure 1. Disaster Recovery Network Block

The disaster recovery data center will house a number of devices, including blade servers, load balancers,

firewalls, and virtualization appliances, all interconnected by networking equipment. These devices will require

physical cabling with an increasing demand for higher performance and flexibility, requiring a reliable, scalable, and

manageable cabling infrastructure.

This document makes the following assumptions:

● On the basis of discussions with customer teams, initially the system will have two tenants.

● One VLAN and one IP network block per application tier is deployed.

● Data center servers generally are physical, with the use of some virtualized servers.


● The disaster recovery site hosts mission-critical applications: Microsoft SharePoint, Exchange, and Active

Directory; Banner; Oracle E-Business Suite; web and portal content; file sharing, e-learning solutions; etc.

● The primary data center is the ITS site. The point-of-presence (POP) room at the Women’s College of

Science MDF (Main distribution frame) is the secondary site within the campus. The disaster recovery site is

enabled for ACI.

● Dual 40-Gbps connectivity is used on leaf switches to connect to the spines.

● 1- and 10-Gbps connections are used on the servers to connect to the leaf switches.

● The disaster recovery site deploys three application tiers, in which all servers have gateways defined on the

firewall. All front-end applications are network load balanced with SSL offload. All application and database

servers are in same subnet. The F5 network load balancer is in single-arm mode, and for the Microsoft

Exchange 2013 servers, the default gateway is on the F5 load balancer.

● All servers have static IP addresses and no need of Domain Host Configuration Protocol (DHCP).

● The disaster recovery site has a standalone Cisco ASA firewall deployed with two contexts and operates in

routed mode.

● On the basis of the application flow, the F5 load balancer will be deployed using one-armed mode and

routed mode.

● Cisco Catalyst® 3750 Switches are used for out-of-band (OOB) network connectivity to all network devices.

● One Cisco Aggregation Services Router (ASR) 1001 Router will be connected to both leaf switches and the

perimeter Internet router with two 1-Gbps fiber.

● Two Cisco ASR 1001 Routers are used at the disaster recovery site for DCI connectivity to the primary and

secondary (co-location) sites.

● Role-based access control (RBAC) is used to define user administrative privileges and roles on the fabric.

● Services, including Simple Network Management Protocol (SNMP), Network Time Protocol (NTP), syslog,

Domain Name System (DNS), FTP, Secure FTP (SFTP), and TACACS servers, use the out-of-band

management network, and all these services are hosted in the ITS data center.

● VMware vCenter and Virtual Machine Manager (VMM) and service-graph integration for Cisco ASA firewall

and F5 applications use the out-of-band management network.

Naming Conventions, IP Addresses, and VLANs

This section provides a summary of the naming conventions used by the customer network team and Cisco

Advanced Services.

Naming Conventions

This section presents the naming conventions for customer disaster recovery site physical devices (Table 1), ACI

logical constructs (Table 2), and access policies (Table 3).


Table 1. Naming Conventions for Physical Devices

Building Floor Type and Manufacturer

Device Type Sequence Number

Host Name

DRC 04 DR SP = Spine 01 DRC4DRSP01

DRC 04 DR SP = Spine 02 DRC4DRSP02

DRC 04 DR LF = Leaf 01 DRC4DRLF01

DRC 04 DR LF = Leaf 02 DRC4DRLF02

DRC 04 DR AP = Cisco Application Policy Infrastructure Controller (APIC)

01 DRC4DRAP01

DRC 04 DR AP = APIC 02 DRC4DRAP02

DRC 04 DR AP = APIC 03 DRC4DRAP03

DRC 04 DR RINT = Internet router 01 DRC4DRRINT01

DRC 04 DR RDCI = DCI router 01 DRC4DRRDCI01

DRC 04 DR F5 = F5 load balancer 01 DRC4DRF501

DRC 04 DR GTM = F5 Global Traffic Manager 01 DRC4DRGTM01

DRC 04 DR ASA = Cisco ASA firewall 01 DRC4DRASA01

Table 2. Naming Convention for Cisco ACI Logical Constructs (Tenant, Contexts, Bridge Domains, Endpoint Groups, etc.)

Policy Construct Naming Format Naming Format Example Use Example

Tenant <Name in CAPITAL letters identifying tenant function and transit-device details>

INT_DMZ Tenant used for DMZ services

Contexts <Name in CAPITAL letters with end-device type and traffic type and transit-device details>

Transit_DMZ Context used for Internet transit traffic between Cisco ASA Instance <instance name> and Cisco ASR

Bridge domain <Name in CAPITAL letters with end-device type and traffic type and transit-device details>

FW_Internet_To_ASR_Internet

Bridge domain used for VLAN traffic between the firewall and the Internet

Application profile <Name in CAPITAL letters with end-device type and traffic type and transit-device details>

Transit_ASR Application profile used to group endpoints; part of the traffic between the firewall and the Internet

Endpoint group (EPG) <Name in CAPITAL letters with end-device type and traffic type and transit-device details>

FW_Internet_To_ASR_Internet

EPG used to connect to firewall interface <x/y> carrying Internet traffic


Table 3. Naming Conventions for Cisco ACI Access Policies

Policy Construct Naming Format Naming Format Example Use Example

VLAN pool <Maximum 10-letter PLATFORM NAME><Device or pair number if any>

UCS-SHAREPOINT <Platform name>

Physical domain <Maximum 10-letter PLATFORM NAME><Device or pair number if any>


Attached Entity Profile <Maximum 10 letter PLATFORM NAME><Device or pair number if any>


Link level <1/10 Gbps> 1G -

Lightweight Access Control Protocol (LACP)

<LACP-ACTIVE/PASSIVE> LACP-ACTIVE -

Interface policy group <3- or 4-letter PLATFORM NAME>-<Interface type = vPC<number>/PO<number>/IND<1/10 Gbps>-<Pair number>-<End device number>

UCS-SHAREPOINT -vPC1-01 <Platform name>

Interface profile <3- or 4-letter PLATFORM NAME>-<Interface type = vPC<number>/PO<number>/IND<1/10 Gbps>-<Pair number>-<End device number>


Switch profile <3- or 4-letter PLATFORM NAME>-<Interface type = vPC<number>/PO<number>/IND<1/10 Gbps>-<Pair number>-<End device number>


Port selector identity <3- or 4-letter PLATFORM NAME>-<Interface type = vPC<number>/PO<number>/IND<1/10 Gbps>-<Pair number>-<End device number>


IP Addresses

ACI fabric requires a block of addresses to enable ACI infrastructure to initialize the fabric and assign IP addresses

to different types of nodes. Here are some examples:

● Infrastructure loopback IP addresses (spine and leaf)

◦ Node-level communication with the APIC

◦ Tunnel endpoint (TEP) termination on the top-of-rack (ToR) switch

◦ Peering address for internal Border Gateway Protocol (iBGP), diagnostics, etc.

● Leaf loopback virtual PortChannel (vPC) TEP IP addresses

◦ Address of the logical virtual TEP (VTEP) shared between vPC peers


● Leaf loopback fabric TEP IP addresses

◦ VTEP address used to communicate to downstream virtual switch (vSwitch) VTEPs

◦ Identical across all leaf switches to allow downstream mobility of VTEP devices

● Spine loopback proxy anycast IP addresses

◦ Common anycast IP address shared by each fabric proxy redundancy group

◦ IPv4, IPv6, and Layer 2 MAC address proxies

The fabric administrator specifies the infrastructure addressing scope. By default, the APIC assigns IP addresses

from the 10.0.0.0/16 block (this setting is user configurable). For the disaster recovery project discussed here, new

unique address spaces are used, as shown in Tables 4, 5, and 6.

ACI provides both in-band and out-of-band capabilities to manage the ACI infrastructure. Table 4 shows the

address spaces reserved for the ACI network.

Table 4. IP Addresses for Cisco ACI Network

Description IP Range

TEP and infrastructure (infra) address 10.xxx.11.0/23

In-band management (mgmt) address 10.xxx.13.0/24

Out-of-band management address 10.xxx.14.0/24

Router ID 10.xxx.0.128/29

Table 5 shows the address spaces used for fabric interconnectivity.

Table 5. IP Addresses for Fabric Interconnectivity


Point-to-point, fabric, and Cisco ASR connectivity 10.xxx.1.128/27

Point-to-point, fabric, and Cisco ASA connectivity 10.xxx.1.160/27

Table 6 shows the address spaces used for management services.

Table 6. IP Addresses for Management Services


SNMP yy.xxx.xxx.xxxx

TACACS 10.xxx.32.21

10.51.200.135

Syslog 10.xxx.32.22

yy.xxx.xxx.xxxx

NTP yy.xxx.xxx.xxz

10.0.0.1

10.0.0.2

Secure Copy (SCP) and FTP yy.xxx.xxx.xxxx

DNS 10.xxx.35.35

10.xxx.35.36


VLANs

Table 7 lists the VLANs allocated for disaster recovery in an ACI implementation.

Table 7. VLANs in Fabric

VLAN Description VLAN Range

Server farm VLAN (static) 1-1500

Server farm VLAN (dynamic) 1501-3000

Layer 2 in-band management 3001-3010

Internet routers VLAN 3011-3020

WAN/DCI routers VLAN 3021-3030

Layer 2 connectivity from external network 3031-3050

Testing VLAN 3500-3600

Connectivity to the F5 load balancer 3601-3700

Connectivity to the Cisco ASA 3701-3800

Fabric infrastructure VLAN 4093

Design Requirements

In Cisco ACI terminology, a tenant is the highest-level logical container in which objects and policies of a given

group or organization can reside. A tenant itself isn’t mapped directly to existing (legacy) network constructs such

as virtual routing and forwarding (VRF) instances, virtual private networks (VPNs), interfaces, and security policies.

Tenants provide the following high-level properties:

● Isolation: Tenants can be totally isolated from one another or share certain resources.

● RBAC: Users can be created that have visibility and specific privileges over only one or more specific

tenants (this is called a domain).

● Inheritance: Objects inside a tenant inherit that tenant’s top-level policies.

In cases in which the departments within an organization do not all adhere to the same approach to network

design, a single-tenant model can quickly become tedious to maintain at the design and operation levels.

When a given department or service function is confined to its corresponding tenant in ACI, troubleshooting and

fault isolation can be performed more efficiently. A tenant can choose to export (or import) objects to (or from) other

tenants, thereby relaxing the strict isolation initially provided by the multitenant model.

Implementation of more than one tenant provides a clean model for controlling inter-tenant communications when

desirable. The use of multiple tenants lets you easily totally isolate one environment from another. Although it is

true that inside a tenant, inter-EPG communication is never allowed explicitly, the “one environment = one tenant”

model lets you isolate or permit communication in a highly specific way.

The ACI fabric design proposed in the context of a customer disaster recovery deployment addresses the needs of

multiple internal network blocks. Those blocks, summarized in Table 8, are:

● Internet DMZ tenant at the disaster recovery perimeter

● Intranet server farm tenant


The Internet DMZ tenant provides services from the external world. The Cisco ASR Internet routers and WAN DCI

routers are part of this tenant, as is the Internet context of the firewall. The server farm tenant addresses the

internal applications. Each of these service functions has its specific applications and policies as well as traffic

patterns. Mixing all of these within one overarching tenant results in operational complexity and little administrative

flexibility. It also reduces the ease with which isolation and fault containment can be provided between

departments. RBAC applies roles to domains. Domains can group from 1 to N tenants. With this model, you can

easily create highly specific access control policies.

Table 8. Tenant Description and Naming

Description Tenant

Internet DMZ tenant at disaster recovery perimeter INT_DMZ

Intranet server farm SF

Figure 2. High-Level View of Tenants and Required Services


Tenant DMZ

The tenant DMZ provides Internet services and a pool of shared network and application services to the customer’s

users. The customer has an Internet link from two different service providers at the primary data center. The BGP

routing protocol is used to provide dynamic failover between the two Internet service providers (ISPs). If the

primary data center site is down, the Internet link at the disaster recovery site provides backup Internet services for

campus and external users.

The tenant DMZ provides the following services:

● Internet services for the primary data center in the event that the primary data center Internet service is

down

● Internet services for all the services at the disaster recovery site

Tenant Server Farm

The tenant server farm provides mission-critical application services to the customer’s users. The customer’s

secondary POP site within the campus functions as an alternative site for some core internal services that are

required within the campus: for example, DHCP, DNS, IP telephony, and read-only domain controllers.

The tenant server farm provides the following services:

● Microsoft SharePoint

● Microsoft Exchange 2013

● Oracle ERP and E-Business Suite (Exadata)

● Banner systems (Oracle Exadata)

● E-learning (Blackboard)

● File sharing

● Web portal and content

● Domain controller

Traffic Flow

This section discusses the business- and customer-initiated traffic flows destined for the customer’s disaster

recovery data center. The traffic flows are of two main types:

● Internet traffic

● DCI traffic

Internet Traffic Flows

The Internet traffic flows provide access from the DMZ servers and load-balancing servers to the Internet, and also

the access from the Internet to these servers. In the customer environment, the servers can access the Internet

through a secure channel for software patching.

● Internet DMZ server access to Internet (outbound); flow (a) in Figure 3: All DMZ servers have a gateway on

the Cisco ASA Internet firewall context, which sends traffic to the perimeter Cisco ASR router to access the

Internet.

● Internet traffic reaching the Internet DMZ server (inbound); flow (b) in Figure 3: F5 GTM determines the

application availability site (primary data center or disaster recovery site) based on DNS entries. All


incoming traffic travels through the perimeter firewall context (INT), and if incoming traffic is allowed,

perimeter firewall traffic will go to servers in the DMZ.

● Server access to the Internet (outbound traffic); flow (c) in Figure 3: The server gateway is defined in the

server farm context on the Cisco ASA firewall, which sends traffic to the Cisco ASA Internet context through

the fabric (the fabric acts as Layer 2 transport). After the traffic reaches the perimeter firewall of the Cisco

ASA Internet context, depending on the destination IP address, the traffic is forwarded to the Internet ASR

router to reach the Internet. All front-end servers are load balanced with F5, which works in one-arm mode.

● Internet traffic reaching the servers (inbound traffic); flow (d) in Figure 3: GTM determines the application

availability site (primary data center or disaster recovery site) based on DNS entries. All incoming traffic

travels through the Cisco ASA perimeter firewall context (Internet context). If incoming traffic is allowed,

after passing the perimeter, it is routed through the leaf (in this case, the fabric acts as Layer 2 transport) to

the internal server farm firewall context. The server farm context on the Cisco ASA firewall’s route to reach

the virtual IP address or network load-balancing (NLB) server’s IP address depends on the destination IP

address. If the request is for NLB servers, the traffic will be forwarded to the F5 Local Traffic Manager

(LTM), which will forward the traffic to a real server.

Figure 3. One-Arm Load-Balancing Internet Traffic Flow


DCI Traffic Flows

The DCI traffic flows provide access from the DMZ servers and load-balancing servers to the primary data center

and co-location site and also access from the ITS data center applications to these servers.

● DMZ server access to ITS data center (outbound); flow (a) in Figure 4: All DMZ servers have a gateway on

the Cisco ASA Internet DMZ firewall context, which sends traffic to the DCI Cisco ASR routers to access the

primary and secondary sites.

● ITS traffic reaching the Internet DMZ server (inbound); flow (b) in Figure 4: All incoming traffic from the ITS

data center travels through the perimeter firewall context (INT_DMZ), and if incoming traffic is allowed, after

passing the perimeter firewall, the traffic will go to the servers in the DMZ.

● Server farm server access to ITS site (outbound traffic); flow (c) in Figure 4: The tenant server farm server’s

gateway is defined in the server farm SF context on the Cisco ASA firewall. On the basis of the destination

IP address, the traffic will be forwarded to the DCI ASR routers to reach the primary or secondary site. All

front-end servers are load-balanced with F5, which works in one-arm mode.

● ITS traffic reaching the disaster recovery server farm servers (inbound traffic); flow (d) in Figure 4: All

incoming traffic from ITS travels directly to the server farm context on the Cisco ASA firewall. Its route to

reach the virtual IP address or NLB server’s IP address depends on the destination IP address. If the

request is for load-balancing servers, the traffic is forwarded to the F5 LTM, and then the traffic is forwarded

to a real server.

Figure 4. DCI Traffic Flow for One-Arm Load Balancing


Internet Access through Disaster Recovery

If the primary data center site is down in the primary data center, the Internet link at the disaster recovery site will

be used for backup Internet service. Figure 5 shows the flow.

Figure 5. Internet Access Through Disaster Recovery

Disaster Recovery Topology and Service Flows

Figure 6 shows the logical topology of both the ITS and disaster recovery data centers.

Figure 6. Customer Data Center High-Level Topology Overview


Figure 6 is summarized here:

● Three main components of Cisco ACI

◦ Fixed-chassis spine (Cisco Nexus® 9336PQ ACI Spine Switch) with 36 ports to connect with leaf

switches

◦ Leaf (Cisco Nexus 9396PX Switch) with 48 x 1/10-Gbps fiber ports for host connectivity and 12 x 40-

Gbps ports to connect with spines

◦ Cisco APIC appliances (Cisco UCS® C220 M3 Rack Server)

● 40-Gbps BiDi optics between leaf and spine switches (OM4 cables)

● Two 1-Gbps connection between disaster recovery site and primary data center in ITS data center (through

Cisco ASR routers)

● Two 100-Gbps connection used to connect each data center with Multiprotocol Label Switching (MPLS)

cloud

● Hardware summary

◦ Two spine switches (Cisco Nexus 9336PQ)

◦ Two leaf switches (Cisco Nexus 9396PX)

◦ Two Cisco ASR 1000 Series routers used for WAN DCI connectivity

◦ One Cisco ASR 1000 Series router used for Internet connectivity

◦ One Cisco ASA firewall (Cisco ASA 5585-X)

◦ One F5 BIG-ADC10200V

◦ Two F5 GTM F5-BIG-DNS-2000S

Leaf and Spine Connectivity

Figure 7. Customer Disaster Recovery Physical Connectivity Overview


The main concepts of leaf and spine connectivity are summarized here:

● Endpoint devices (servers, firewalls, load balancers, etc.) and Cisco APIC connected to leaf nodes only

● Spine switches connected to leaf switches only

● Leaf switches rely on the spine’s knowledge of all endpoints

Depending on bandwidth requirements, all leaf nodes connect to both spines with one 40-Gbps uplink. A maximum

of 12 uplinks can be used from the leaf switches to the spine switches.

Server Connectivity with Leaf Switches

The servers connect to leaf nodes only. Each leaf has a total 48 x 1/10-Gbps fiber ports available for endpoints,

supporting individual port connectivity, active and standby mode connectivity, and vPC or Port Channel

connectivity or both.

● Leaf switches support vPC.

● No vPC peer link is required. Peer communication occurs over the fabric.

● Path recovery also occurs over the fabric.

● Within the fabric, the vPC interfaces use an anycast VTEP that is active on both vPC peers.

● The servers can connect to leaf switches using individual interfaces, PortChannels, active and standby

modes, and vPC.

Figure 8. Server Connectivity Options with Leaf Switches


Layer 4 Through 7 Device Connectivity to Leaf Switches

As mentioned earlier, the disaster recovery project uses a total of two leaf switches. The same leaf switches are

used to connect Layer 4 through 7 devices:

● Cisco ASA firewall

● F5 load balancer

The customer uses the F5 BIG-IP load balancer and Cisco ASA firewall services devices, enabled for Cisco ACI

and configured and managed by Cisco APIC. The ACI fabric supports traditional service devices with features such

as flooding bridge domain (Figure 9).

Figure 9. F5 LTM and GTM and Cisco ASA Firewall Connectivity to Leaf Switches


Cisco ASR Router WAN Connectivity

Figure 10 shows the connectivity between the Cisco ASR routers and the WAN.

Figure 10. Cisco ASR and Leaf Switch Connectivity

Cisco APIC Connectivity

Three Cisco APIC devices are installed in the disaster recovery site using two 10-Gbps interfaces in active-standby

mode and standard Linux interface bonding (Figure 11). This configuration is provided by default, and no additional

configuration is required. The APIC cluster is a fully redundant multinode cluster, and upgrades and downgrades

are performed through the cluster redundancy and not through In-Service Software Upgrade (ISSU) on individual

nodes. (This solution uses a transactional redundancy model, and as in Hadoop and other distributed systems, it is

designed to operate with node state changes. The addition of new nodes and the upgrading and removal of

existing nodes are simple forms of cluster scale up and scale down.)


Figure 11. Cisco APIC Connectivity

External Networking

The ACI fabric uses the concepts of “inside” and “outside” networks. Anything designated as within, or inside, the

fabric is subject to the forwarding rules and policy enforcement. However, external, or outside, entities almost

always need to be connected to the fabric (WANs, existing networks, etc.), so a way is needed to designate these

external entities as not being part of the fabric. To achieve this, the concept of external, or outside, networks is

used. External networks can be at Layer 2 or Layer 3. In this scenario, the fabric is acting as Layer 2 between the

firewall and Cisco ASR, so neither Layer 3 nor peering is required.

External Layer 2 Networks

External (outside) bridged and Layer 2 networks are required whenever the Layer 2 domain needs to be extended

beyond the ACI fabric: for example, when a Layer 2 interconnect is required between data center sites.

When a Layer 2 external network is configured, essentially a mapping is created between a fabric bridge domain

and a VLAN external to the fabric. In addition, ports on one or more leaf switches are designated as border ports.

These interfaces connect to the external device.

The customer wants to extend a number of Layer 2 segments across data center sites. To support this

requirement, the Cisco ASR 1000 Series routers use Cisco Overlay Transport Virtualization (OTV) technology to

transport VLANs between the disaster recovery site and the primary data center.


To support Layer 2 extension from the ACI fabric, a Layer 2 outside construct is used to extend bridge domains

beyond the fabric. A Layer 2 outside construct works by extending a particular bridge domain using a specified

VLAN.

Figure 12 shows how the Layer 2 outside construct is used to extend bridge domains outside the fabric.

Figure 12. Layer 2 Extension from the Cisco ACI Fabric

In Figure 12, the fabric is Layer 2 connected to the DCI ASR 1001 Routers for Layer 2 extension between the ITS

data center and the disaster recovery site.

The Cisco ASR 1001 is connected to a leaf port, which is configured as a border leaf port during the creation of the

Layer 2 outside construct. Note the following points about border ports:

● Border ports have Bridge Protocol Data Unit (BPDU) guard disabled (regular fabric edge ports have this

feature enabled).

● Spanning-tree BPDUs are forwarded between border ports within the same bridge domain.

When a bridge domain is extended using a Layer 2 outside construct, the bridge domain must operate in regular

mode: in other words, Address Resolution Protocol (ARP) optimization should be disabled, and unknown unicast

flooding should be enabled. Depending on the specific bridge domain being extended, these settings may already

be configured (for example, bridge domain settings are different depending on where the default gateway resides).

Layer 2 External constructs are configured from the APIC using the External Bridged Networks configuration option

under Networking. Figure 13 shows how this item is configured in the APIC GUI.


Figure 13. Layer 2 Outside Configuration

Note that the Layer 2 outside construct must be mapped to a bridge domain. A VLAN must also be specified for the

external encapsulation. Finally, the physical node and interface to which the external device (in this case, the Cisco

ASR 1000 Series router) connects must be specified.

Service Architecture Design

The customer deploys most applications in a three-tiered architecture (web, application, and database tiers, or

similar layers). Note the following points about three-tiered applications:

● The servers in a three-tiered architecture use one VLAN per application tier.

● The server farm and DMZ tenants use Cisco ASA firewalls for security enforcement.

● A standalone F5 load-balancer appliance is used for load balancing.

● Additional VRF instances would map to additional firewall contexts.

● Additional firewall contexts would use shared EPG and bridge domains for connection to the Cisco ASR and

intercontext communication.

● There is one bridge domain per application tier, contained within a VRF instance, but a gateway is not

configured on the fabric.

● The firewall is the default gateway for all hosts (except for the Microsoft Exchange application). Therefore,

all east-west and north-south traffic passes through the firewall.

● The F5 load balancer is used in proxy Source Network Address Translation (SNAT) mode for most mission-

critical services. Mainly the web and application tiers use the load-balancer service.

● For Microsoft Exchange application F5 BIG-IP LTM is deployed in Layer 3 inline (two-arm) mode, where the

source IP address of the client is visible to server (direct server return).

● Firewalls and F5 BIG-IP run static routing with distribution switches.

● Distribution switches run dynamic routing protocol with WAN edge and provider edge routers.

● One load-balancing context is created per application group.


For the solution discussed here, application-tier bridge domains are configured as follows:

● ARP flooding: Enabled. Within a bridge domain, servers must use ARP for the default gateway (the firewall).

If the firewall is silent (that is, if it does not send out Generic ARP (GARP) or similar packets), the ACI fabric

will not learn the MAC address information for the firewall. In this situation, ARP requests from the host to

the firewall will fail. To avoid this scenario, regular ARP flooding is enabled in the application-tier bridge

domains. In other words, ACI-optimized ARP handling is not enabled for these bridge domains.

● Unicast routing: Disabled. If unicast routing is enabled within a bridge domain, learning occurs for a host’s

IP and MAC addresses. In situations in which the fabric is not participating in routing (such as when a

firewall acts as the gateway), IP addresses do not need to be learned. Therefore, this option is disabled for

each of the application-tier bridge domains.

● Layer 2 unknown unicast: Flood. Hardware proxy handling of unknown unicast traffic is the default option.

For topology 1, regular Layer 2 handling is enabled (flooding).

● Gateway and subnet addresses: Not configured. In this topology, all Layer 3 gateway services are running

on the firewall, not the ACI fabric. Therefore, a subnet or gateway should not be configured in the

application-tier bridge domains.

For the customer’s disaster recovery design, several topologies are supported for three-tiered applications. These

topologies provide different options for integration of firewall and load-balancing services, as follows:

● Scenario 1

◦ Firewall is the gateway for the servers.

◦ Load balancer service is not required.

● Scenario 2

◦ Firewall is the gateway for the servers.

◦ Load balancer is in one-armed mode.

● Scenario 3

◦ Load balancer acts as default gateway for hosts (load balancer operating in inline mode).

◦ Firewall is used in Layer 3 routed mode.

Note that a combination of these topologies can be used for flexibility: for example, if one application group (web,

application, and database) needs to use the firewall as the default gateway, and a second application group needs

to use the load balancer as the gateway. The customer indicated that all application tiers will use the firewall as the

default gateway except Microsoft Exchange, which uses the load balancer as the gateway.

Firewall as the Gateway (NLB)

In scenario 1, the firewall is used as the default gateway for all hosts and servers on a given VLAN. This means

that all communication in and out of that VLAN is secured by the firewall.


Figures 14 through 17 show how this topology works.

Figure 14. NLB DMZ Servers: Firewall as Default Gateway (Internet Traffic)

Figure 15. NLB DMZ Servers: Firewall as Default Gateway (DCI Traffic)


Figure 16. NLB Server Farm Servers: Firewall as Default Gateway (Internet Traffic)

The following points relate to the topology in Figure 16:

● The Cisco ASA firewall contexts operate in routed mode.

● An EPG is created on the ACI fabric for each application tier.

● A bridge domain is created on the ACI fabric for each application tier.

● A separate bridge domain is created to handle dynamic routing between the Cisco ASA Internet context and

the Cisco ASR 1001 Routers.

● A separate bridge domain is created to handle routing (static or Open Shortest Path First [OSPF]) between

the Cisco ASA Internet context and the server farm context.

● The fabric does not actively participate in routing, but instead acts as the Layer 2 transport.


Figure 17. NLB Server Farm Servers: Firewall as Default Gateway (DCI Traffic)

The following points relate to the topology shown in Figure 17:

● The Cisco ASA firewall contexts operate in routed mode.



● A separate bridge domain is created to handle dynamic routing between the Cisco ASA Internet context and

the Cisco ASR 1001 Routers.

● The fabric does not actively participate in routing, but instead acts as the Layer 2 transport.

As mentioned previously, an EPG is created for each application tier: web, application, and database.

By keeping the firewall interfaces in the same EPG as the hosts, you do not need to configure contracts between

the host and the firewall (endpoints within the same EPG can communicate freely).

For communication between the firewalls and the Cisco ASR 1001 Routers, a similar EPG arrangement is used. In

this topology, the firewall and the Cisco ASR router are treated as regular hosts by the fabric; therefore, both the

firewalls and the Cisco ASR 1001 Routers can reside in a dedicated INT_ASR EPG.

On the basis of the current understanding of the applications to be deployed in the disaster recovery site, one

context will be configured per tenant.

In addition to the regular application tier bridge domains, a separate bridge domain is required for dynamic routing

between the firewalls and the Cisco ASR 1001 Routers. In this topology, the dynamic-routing bridge domain should

be configured with the same settings as the application-tier bridge domain.


Load Balancer in One-Armed Mode

There are many ways to insert the F5 load balancer into the traffic path in the ACI fabric. The actual traffic flow will

depend on the service being load balanced and the configuration of the core components including the ACI, F5

load balancer, firewalls, and connecting infrastructure.

According to customer application requirements, the gateway to all the traffic resides on the firewall, and F5

performs SNAT. The F5 BIG-IP LTM is connected to its gateway (firewall) using a single interface and VLAN only.

It is not inline: that is, it does not sit in the traffic path, and only traffic that needs to be load balanced is directed to

it.

The traffic that the client initializes for the virtual server virtual IP address travels to the load balancer through

service graph insertion. When the F5 load balancer operates in one-arm mode, it uses a pool of addresses, known

as SNAT IP addresses. When traffic reaches the virtual IP address of the load balancer, the source IP address is

replaced with an address from the SNAT pool. Return traffic from the real server is directed to the SNAT IP

address; therefore, the load balancer sees all return traffic.

In this topology, the firewall is used as the default gateway for all hosts and servers on a given VLAN. This means

that all communication in and out of that VLAN is secured by the firewall. The load-sharing algorithm picks a

physical server, and the load balancer forwards the traffic to the physical IP address of this server.

In this topology, the F5 BIG-IP LTM load balancer operates in automated mode: that is, the device package is

uploaded to the APIC, service graphs are used to redirect traffic to the load balancer, and configuration tasks are

moved to the APIC.

Figure 18 and 19 show how this topology works.

Figure 18. Load Balancer in One-Arm Mode: Firewall as Gateway (Internet Traffic)


Figure 19. Load Balancer in One-Arm Mode: Firewall as Gateway (DCI Traffic)

The following points relate to the topology in Figure 19:

● The firewalls operate in routed mode.

● The load balancer operates in one-arm mode.



● An EPG is created for the transit connection to F5 BIG-IP LTM.

● A bridge domain is created for the transit connection to F5 BIG-IP LTM.

● A separate bridge domain is created to handle dynamic routing between the firewalls and the Cisco ASR

1000 Series routers.

● Dynamic routing (OSPF) runs between the firewalls and Cisco ASR routers. This routing exchange occurs

through the fabric (that is, the fabric does not actively participate in OSPF, but instead acts as the Layer 2

transport).

● No contract is necessary for host-to-firewall communication.

● The load-balancer configuration is provisioned using the service graph.

● A contract is created between the provider (server) and the consumer (client side) traffic, and the same

contract binds the load-balancer graph function.

By keeping the firewall interfaces in the same EPG as the hosts, contracts between the host and the firewall are

not needed (endpoints within the same EPG can communicate freely).

Because the load balancer is operating in one-arm mode, it is attached to the ACI fabric using a single arm. To

accommodate this connection, a dedicated EPG is set up in which both the load-balancer interface and the firewall

interface facing the load balancer reside. The load balancer and firewall can communicate freely without an ACI

contract.


For communication between the firewalls and the Cisco ASR 1000 Series routers, a similar EPG arrangement is

used. In this topology, the firewall and the Cisco ASR router are treated as regular hosts by the fabric; therefore,

both the firewalls and the Cisco ASR 1000 Series routers can reside in a dedicated INT_ASR EPG.

A dedicated bridge domain is configured for each application tier. This design was chosen (instead of a single

bridge domain) to prevent broadcast traffic from unnecessarily traversing application tiers.

In addition to the regular application bridge domains, a separate bridge domain is required for dynamic routing

between the firewalls and the Cisco ASR 1000 Series routers. In this topology, the dynamic-routing bridge domain

should be configured with the same settings as the application-tier bridge domains.

A dedicated bridge domain is also required for the transit connection between the firewall and the load balancer.

Dynamic routing is required between the firewalls and the Cisco ASR 1000 Series routers to exchange routing

information to and from external networks. Because the ACI fabric is not responsible for any Layer 3 services in

this topology, dynamic routing is performed directly between the firewalls and the routers. Therefore, the ACI is

operating purely as a Layer 2 transport.

Although dynamic routing is taking place between the firewalls and the Cisco ASR 1000 Series routers, the fabric

does not play an active role in this exchange; therefore the EPG and bridge domain configuration to achieve this

behavior is identical to the configuration for the application tiers. Firewalls and Cisco ASR 1000 Series routers can

all reside in the same EPG, eliminating the need to configure contracts to allow communication.

Traffic Flow

This section describes a traffic flow from an external client destined for a web server and the associated return

flow.

Figure 20 shows the traffic flow from the client to the server.

Figure 20. Traffic Flow: Client to Server (DCI Traffic)

Note: In the example in Figure 20, the database, application, and web servers are using the firewall as the

default gateway. The database and application servers are not load balanced in this scenario.


In Figure 20, the traffic enters the data center through the Cisco ASR 1000 Series router, where it is routed to the

virtual IP address of the load balancer through the firewall. The firewall needs a route pointing to the virtual IP

address through the load balancer interface (accessible through the bridge domain named LB_ONE-ARM).

Upon receiving this traffic, the load balancer creates a new session with a source IP address from the IP address

pool. This new flow is destined for the real server and is routed through the firewall. Finally, the firewall sends the

traffic to the real server using the relevant application bridge domain and EPG (in this case, SRV1).

The traffic flow is summarized here:

● A flow from outside is destined for the virtual IP address.

● The Internet ASR traffic is routed to the virtual IP address through the Cisco ASA DMZ context.

● The firewall DMZ context needs to route traffic to the virtual IP address.

● The firewall DMZ context has a route pointing to the virtual IP address through the firewall server farm

context.

● The firewall server farm context has a route pointing to the virtual IP address of F5 BIG-IP LTM.

● Upon receiving the traffic, the load balancer creates a new session with a source IP address from the SNAT

pool defined in the service graph parameters and changes the destination IP address on the packet to point

to the back-end server.

● The new flow is destined now for the real servers based on the load-balancing method and is routed

through the firewall as a gateway.

● The firewall server farm context routes the flow to the correct application bridge domain.

● Finally, the firewall sends the traffic to the real server using the relevant bridge domain and EPG (web EPG

and web bridge domain in this case).

The flow in Figure 21 is summarized here:

Figure 21. Traffic Flow: Client to Server (DCI Traffic)


● A flow from outside is destined for the virtual IP address.

● The DCI ASR route to the virtual IP address is through the Cisco ASA server farm context.

● The firewall server farm context has a route pointing to the virtual IP address of F5 BIG-IP LTM.

● Upon receiving the traffic, the load balancer creates a new session with a source IP address from the SNAT

pool defined in the service graph parameters and changes the destination IP address on the packet to point

to the back-end server.

● The new flow is destined now for the real servers based on the load-balancing method and is routed

through the firewall as a gateway.

● The firewall server farm context routes the flow to the correct application bridge domain.

● Finally, the firewall sends the traffic to the real server using the relevant bridge domain and EPG (web EPG

and web bridge domain in this case).

Figures 22 and 23 show the return traffic flow.

Figure 22. Traffic Flow: Server-to-Client Return Traffic (to Internet)


Figure 23. Traffic Flow: Server-to-Client Return Traffic (to DCI)

For the return flow, the web server sends traffic to the IP address, which was derived from the address pool. This

traffic is then routed through the firewall. The firewall routes the traffic to the load balancer’s interface. Finally, the

firewall sends the return traffic back to the original client through the firewall and Cisco ASR 1001 Router.


The return flow is summarized here:

● The real server responds to the load-balancer address through the default gateway (firewall).

● The firewall routes the traffic to the load balancer address through the load-balancer interface.

● The load balancer sends traffic back to the client through the firewall.

● The firewall routes traffic back to the client through the Cisco ASR.

Load Balancer as the Gateway

In this topology (Figure 24), the load balancer is used as the default gateway for all hosts and servers on a given

VLAN. This approach means that all communication in and out of that VLAN also must traverse the firewall.

The major difference in this topology is that the load balancer operates in inline mode.

Figure 24. Load Balancer as Default Gateway (Inline Mode Internet Traffic)

In the example in Figure 24, the web server uses the load balancer as the default gateway. The application and

database servers use the firewall as the gateway and are not load balanced in this scenario.

The following main points relate to this topology (Figure 25):

● The load balancer and firewalls operate in inline routed mode.



● A separate bridge domain is created to handle traffic between the load balancers and the firewalls.

● A separate bridge domain is created to handle dynamic routing between the firewalls and the Cisco ASR

1000 Series routers.

● An EPG is deployed for the external side of the load balancer (in this case named LB_INLINE).


● A contract is deployed between the server EPG (for example, Web) and the load balancer external EPG (for

example, LB_INLINE). This contract refers to a service graph, which redirects the incoming traffic to the

load balancer.

● The load-balancer configuration is provisioned using the service graph.

● A contract is created between the provider (server) and the consumer (client side) traffic, and the same

contract binds the load-balancer graph function.

Figure 25. Load Balancer as Default Gateway (Inline Mode DCI Traffic)

Figures 26 and 27 show the return traffic for the inline mode deployment.

● The server sends the return traffic pointing the client IP address to F5 BIG-IP LTM.

● F5 BIG-IP has a route pointing to the Cisco ASA server farm context firewall and to the client IP address.

● Depending on the client’s location, the traffic is forwarded to the DCI ASR or Internet ASR.


Figure 26. Load Balancer as Default Gateway (Inline Internet Return Traffic)

Figure 27. Load Balancer as Default Gateway (Inline DCI Return Traffic)


Services Integration

Service Device Packages

A device package is an archive of information containing the details required to manage an external service device

(either from Cisco or a third party). A device package is presented as a compressed zip file containing the

following:

● XML-based device specification

● Python script implementing the device script API

The device specification contains XML code that specifies parameters such as version information, functions, and

configuration options for the device.

The device script handles the interface with the service device using its API (preferred) or the command-line

interface (CLI).

A device package can be uploaded to the APIC easily using the GUI. The L4-L7 Services tab contains a Packages

submenu from which packages can be uploaded in zip file format as shown in Figure 28

Figure 28. Importing Device Packages

Figure 29. Importing Device Packages from the Packages Menu


Automated Service Insertion

ACI supports a significantly enhanced service insertion model compared to traditional VRF and VLAN stitching-

based approaches. By using the APIC as a centralized point of policy control, the traditional traffic steering

challenges associated with services integration can be overcome.

The first step in achieving automated service integration is importing the device package (described in the previous

section). After a device package has been uploaded, the APIC is aware of the device features and functions, and

configuration tasks can be moved from the device itself to the APIC.

The second major part of the automated service insertion architecture is the service graph. A service graph is

essentially a chain of service devices that traffic should pass through. A policy specifies that traffic should be

redirected to a service graph, which may consist of firewalls, load balancers, or any other services (Figure 30).

Figure 30. Logical Configuration

After the service graph has been defined, it can be selected when defining a contract subject for an application

policy.

Figure 31 shows packet processing for the initial traffic requiring redirection to a service graph. Note that the initial

traffic must be sent to the egress leaf node for redirection to the correct leaf switch on which the service resides;

subsequent traffic will be redirected by the ingress leaf switch after learning occurs on that node.

Figure 31. Services Redirection: Initial Packet Processing


Cisco ASA Integration with Cisco ACI

The customer uses the Cisco ASA 5585-X firewall for Internet perimeter termination and for connection to the

intranet server farm. The Cisco ASA firewall has the following logical and physical connectivity:

● Connectivity to the ACI fabric uses dual 10-Gbps fiber interfaces.

● One Cisco ASA 5586-X is connected to a pair of Cisco Nexus 9396PX leaf switches.

● A pair of 10 Gigabit Ethernet PortChannels connect to each of the Cisco Nexus 9396PX Switches.

● The Cisco ASA firewall peers with the Cisco ASR router using OSPF as the routing protocol.

● The Cisco ASA firewall operates in routed mode.

● The standalone deployment mode is used (no high availability).

● The Cisco ASA has two contexts defined: intranet server farm and Internet perimeter firewalls.

● Another 1-Gbps interface connects the Cisco ASA firewall connected to the Cisco Catalyst 3750 out-of-

band switch for management connectivity.

● The Cisco ASA firewall and ACI integration will make up the out-of-band management connectivity.

The Cisco APIC automates insertion of services (such as an Cisco ASA firewall) northbound between applications,

also called EPGs. The APIC uses northbound APIs to configure the network and services. You use these APIs to

create, delete, and modify a configuration using managed objects.

When configuration is controlled through the APIC, you cannot change the configuration through the Cisco ASA

CLI. This means that the CLI for any feature that you configure through the APIC are disabled on the Cisco ASA.

However, you can use the CLI to configure management access to the Cisco ASA. Operational and status

commands, such as troubleshooting commands and show commands, are also available through the CLI.

When a service function is inserted in the service graph between applications, traffic from these applications is

classified by the APIC and identified using a tag in the overlay network. Service functions use the tag to apply

policies to the traffic. For Cisco ASA integration with the APIC, the service function forwards traffic using either

routed firewall operation.

F5 Integration with Cisco ACI

With the Cisco ACI and F5 solution’s fully programmable load-balancing services technology, customer can

implement application-centric deployments with the Cisco ACI fabric, using contracts, filters, and service graphs to

control traffic between application tiers. This model provides stateless load balancing for a three-tiered application

in the data center with agility and full automation. Traffic can be redirected to F5 BIG-IP LTM load-balancing

devices (both physical and virtual) using an appropriate device package integrated into Cisco APIC. The F5 load

balancer has the following logical and physical connectivity considerations:

● One LTM (F5-Big-IP-10050) and one GTM (F5-Big-IP-2000) are used.

● Both F5 LTM and GTM support 1/10-Gbps connections (copper and fiber). However, the customer plans to

use four 10-Gbps fiber connections for LTM and two 1-Gbps copper connections for GTM. This

configuration may change in the final design.

● Both F5 LTM and GTM use Layer 2 LACP connectivity to both leaf switches.

● F5 LTM operates in mixed mode (one-arm anywhere mode) for all the servers; only for Microsoft Exchange

2013 does it operate in routed inline mode.

● The F5 LTM and Cisco ACI integration uses out-of-band management connectivity.


The Cisco APIC automates the insertion and provisioning of network services through the F5 BIG-IP platform: for

example, SSL offload, server load balancing (SLB), and Microsoft SharePoint services. F5 BIG-IP LTM integrates

with Cisco APIC through well-established and open APIs (Simple Object Access Protocol [SOAP] or

Representation State Transfer [REST]). This integration automates network and service provisioning across the F5

services fabric, providing end-to-end telemetry and visibility of applications and tenants. Cisco APIC acts as a

central point of configuration management and automation for Layer 4 through 7 services and tightly coordinates

service delivery, serving as the controller for network automation. A service appliance (device) performs a service

function defined in the service graph. One or more service appliances may be required to render the services

required by a service graph. A single service device can perform one or more service functions.

A maximum of two F5 BIG-IP LTM concrete device clusters in active-standby mode can be set up. Concrete

devices have concrete interfaces connected to physical F5 BIG-IP interfaces, or to virtual network interface cards

(vNICs) if F5 BIG-IP VE is used. When a concrete device is added to a logical device cluster, the concrete

interfaces are mapped to the logical interfaces (Figure 32).

Figure 32. F5 Logical Device Cluster with Pair of Concrete Devices

Virtual Machine Networking

VMware vSphere Integration

Cisco ACI can closely integrate with the server virtualization layer. In practice, this means that instantiating

application policies through ACI results in the equivalent constructs at the virtualization layer (that is, port groups)

being created automatically and mapped to the ACI policy.

Integration with the server virtualization layer is defined through the creation of a policy construct known as a virtual

machine management (VMM) domain. A VMM domain is defined as a container for one or more VMM systems (for

example, VMware vCenter) with similar network policy requirements. Each VMM domain represents a live

migration domain: that is, virtual machines can be live-migrated within the VMM domain, but not beyond it (Figure

33).


Figure 33. VMM Domains

Each VMM domain is associated with a name space. The name space defines a range of identifiers (normally

VLANs or VXLANs) that are used to identify traffic within the domain and map that traffic to EPGs. If VLANs are

used to identify the traffic, then each VMM domain can support up to 4000 EPGs.

VMM Domain Configuration

To implement a VMM domain, the ACI administrator must establish connectivity with vCenter through the GUI or

API. This is accomplished through the VM Networking tab in the APIC GUI and the Policies submenu (Figure 34).

Figure 34. Creating a VMM Domain


Note the selection of the VMware Distributed Switch in this step. After the connection to vCenter has been

established, a new virtual distributed switch (vSwitch) will be automatically created in vCenter, as shown in

Figure 35.

Figure 35. Creating a Distributed Switch

Note that physical uplinks (vmnics) are not automatically added to the new vSwitch; the server administrator must

perform this step.

Assuming a successful connection, the APIC will now reflect information about the virtualized infrastructure, such

as virtual machine information and vmnics, as shown in Figure 36.

Figure 36. VMM Domain Inventory Display

After the integration with vCenter is complete, the fabric or tenant administrator creates EPGs, contracts, and

application profiles as usual. Upon the creation of an EPG, the corresponding port group is created at the

virtualization level. The server administrator then connects virtual machines to these port groups (Figure 37).


Figure 37. Port Group Creation on VMware vCenter

The vCenter connectivity can be established using two different modes: in-band and out-of-band mode. For the

customer’s disaster recovery design, out-of-band mode is used for vCenter and other Layer 4 through 7 integration.

Note: In the customer’s environment, services such as syslog, NTP, and TACACS as well as the VMM domain, F5,

and Cisco ASA use the out-of-band network for management connectivity.

Management Network in Cisco ACI

Two tenants are created by default within Cisco ACI for management purposes:

● Infra: Used for TEP-to-TEP (leaf to leaf) traffic within the fabric and for bootstrap protocols within the fabric.

● Mgmt: Used for management connectivity between the APICs and switch nodes, as well as for connectivity

to other management systems (authentication, authorization, and accounting [AAA] servers; vCenter; etc.).

The infra tenant is preconfigured for settings related to the fabric infrastructure, including the context and bridge

domain used for the fabric VXLAN overlay. Only EPGs configured under the infra tenant are allowed to associate

with the overlay bridge domain, and these EPGs are implicitly allowed to communicate with each other with no

contracts needed or allowed. The infra tenant can also be used to extend the fabric infrastructure to outside

systems that support overlay protocols such as VXLAN and NVGRE. For example, the infra tenant’s default EPG

can be used to configure a fabric infrastructure VLAN on leaf ports that are connected to hypervisors or switches

supporting VXLAN encapsulation.


It is highly recommended that users do not modify the infra tenant. This tenant is not used for general management

functions.

The mgmt tenant is used for general in-band and out-of-band management of the ACI solution. The mgmt tenant

has a default context and private network named inb (for in-band). A single bridge domain is also created by

default, also named inb.

ACI switch nodes have:

● In-band access to infra and mgmt tenants

● On-board management port connected to the out-of-band management network

● Console ports connected to the terminal server

APIC nodes have:

● Two 10 Gigabit Ethernet links to leaf nodes for control-plane and data-plane traffic

● Two management connections (connected to the out-of-band network)

● One Cisco Integrated Management Controller (IMC) port for appliance management

● One console port connected to the terminal server

APIC can be configured to send a variety of system data to Call Home, SNMP, or syslog destinations. When an

event triggers the sending of a report, information from selected queries is included in the report. It is also possible

to configure a query based on a class name or a distinguished name, or to further qualify the query based on

subtrees and object properties.

As mentioned previously, the customer’s disaster recovery site reserved the 10.xxx.14.0/24 block for the out-of-

band network. Table 9 shows the distribution of the out-of-band block.

Table 9. Out-of-Band and In-Band IP Block Assignment Details

Type of Device Out-of-Band IP Block In-Band IP Block

APIC 10.xxx.14.11-20 10.xxx.13.11-20

IMC 10.xxx.14.21-30 -

LTM 10.xxx.14.31-46 -

ASA 10.xxx.14.47-62 -

VMM 10.xxx.14.63-78 -

Free 10.xxx.14.63-79-100 -

Leaf 10.xxx.14.101-150 10.xxx.13.101-150

Spine 10.xxx.14.201-204 10.xxx.13.201-204

Out-of-Band Management Network

The APIC and spine and leaf switches have dedicated out-of-band management ports and provide connectivity to

access these devices (Figure 38).


Figure 38. Out-of-Band Management Network

The out-of-band management endpoint group consists of switches (leaf and spine switches) and APICs that are

part of the associated out-of-band management zone. Each node in the group is assigned an IP address from the

address pool associated with the corresponding out-of-band management zone. The allocated IP address is then

configured on the out-of-band management port of the corresponding node (Figure 39). Hosts that are part of

regular endpoint groups cannot communicate with the nodes in the out-of-band management endpoint group. Any

host that is part of a special group known as the instance profile can communicate with the nodes in an out-of-band

management endpoint group using special out-of-band contracts. Regular contracts cannot be used with this

endpoint group.

Figure 39. Create Node Management Addresses


Note: All the management services, including NTP, syslog, and TACACs, use the out-of-band management

network

Simple Network Management Protocol

The management information base (MIB) can be retrieved from switch nodes (leaf and spine), and SNMP traps

can be sent from switch nodes. SNMP support (MIB and trap) is available on fabric switches only.

This section shows the steps for retrieving the MIB from switch nodes.

After enabling the Admin state and assigning community policies, assign the default SNMP policy to the policy

group (Figure 40).

Figure 40. SNMP MIB Policy

Figure 41. SNMP Policy

Push the SNMP configuration to all switch nodes by configuring the fabric policies for SNMP (Figure 41).


The SNMP trap receiver configuration is required to send SNMP traps from the APIC switch nodes. Trap policy is

also pod-level policy; after it is created on the APIC, it will be pushed to all switch nodes.

Configure the destination group for external SNMP and trap servers. Destination servers can be on either an in-

band or out-of-band network, depending on the option selected from the Management EPG drop-down menu

(Figure 42).

Figure 42. SNMP Trap Destination

Note: The ACI APIC does not support SNMP. However, SNMP polling and trapping can be configured on the

APIC for each individual spine and leaf switch.

Syslog

Syslog messages can be sent through both in-band and out-of-band networks (Figure 43).

Figure 43. Syslog Fabric Policy


Fabric monitoring policy for syslog (as shown in Figure 43) is pushed to all leaf and spine switches.

The remote destination group is configured for external syslog servers. Destination servers can be on an in-band or

out-of-band network, depending on the option selected from the Management EPG drop-down menu (Figure 44).

Figure 44. Syslog Remote Destination

Network Time Protocol

NTP synchronizes time keeping among a set of distributed time servers and clients. This synchronization allows

events to be correlated when system logs are created and other time-specific events occur. In the customer’s

disaster recovery setup, APIC and switch nodes (leaf and spine) are synchronized by external NTP servers.

Conclusion

Cisco Application Centric Infrastructure brings a lot of value to customer even when they are trying to build a

greenfield disaster recovery data center. Customers were not able to bring up the network infrastructure quickly as

well as integrate services and virtualization to bring their critical applications and meet their business deadlines.

For More Information

http://www.cisco.com/go/aci

http://www.cisco.com/go/aci


Printed in USA C11-733786-00 02/15

disaster recovery design with cisco application centric infrastructure

Documents