20140205 dfwcug cisco nexus and how it differs from catalyst 6500

8/11/2019 20140205 DFWCUG Cisco Nexus and How It Differs From Catalyst 6500

1/54

CISCO NX-OS DATA

CENTER FEATURESJack RossCCIE #16728

1


2/54

Agenda Brief Hardware Overview Software Versions NX-OS Layer 2

NX-OS Layer 3 FabricPath Virtual Device Contexts (VDCs) Fiber Channel Over Ethernet (FCoE) Overlay Transport Virtualization (OTV) Virtual Port Channels ( vPCs )

2


3/54

Nexus 7000 OverviewNexus 7000/7700

Typically DC core or aggregation High performance, density, & availability

Unified I/OFCoE switch but not a FC switch

Redundant Power, Line Cards and Supervisors

3


4/54

Nexus 7000 Platform Overview Currently 7 form factors

7018, 7010, 7009, 7004, 7718, 7710,7706 Currently 2 types of line cards

M Series Cards - Layer 3 cards Feature rich cards

F Series Cards - Layer 2 cards* Performance oriented cards

4


5/54

Line Card Features M Series Specific

Layer 3 Routing FEX

OTV TrustSec

F Series Specific FabricPath

vPC+ FCoE

5


6/54

Nexus 5000 OverviewNexus 5000/5500

Typically End of Row (EoR) aggregation or Topof Rack (ToR) access

Typically Layer 2 but can do limited Layer 3 with add ondaughter card in the 5500 SeriesUnified I/O

Both FCoE and native FC switching

Redundant power but not supervisors

6


7/54

Nexus 5000 Platform Overview Currently 2 Generations 1st Gen - Nexus 5000 5010 & 5020 2nd Gen - Nexus 5500 5548 & 5596

Mainly layer 2 switching 5500 can support L3 add-in card

Supports Unified I/O Both FCoE Forwarder (FCF) and native FC switching

5500 supports Unified Ports (UP models) Ports can run as Ethernet or native Fibre Channel Ethernet ports allocated at port 1 and counts up Fibre Channel ports allocated at last port and counts down Requires a reboot to re- allocate ports role (like UCS FI)

7


8/54

Nexus 2000 Fabric Extender (FEX) Acts as a remote line card of 7K or 5K chassis All management performed on Parent Switch

No console or VTY ports on FEX NX-OS automatically downloaded from Parent No local switching

Essentially a VN-Tag/802.1BR switch, not an Ethernet

switch Traffic between local ports on FEX must flow north viauplink to Parent and then south back down

Can impact design decision of platform placement

8


9/54

Software Versions for CCIE Lab NX-OS v6.0(2) on Nexus 7000 Switches (6.2(6) latest) NX-OS v5.1(3) on Nexus 5000 Switches NX-OS v4.2(1) on Nexus 1000v

NX-OS v5.2(2) on MDS 9222i Switches UCS Software release 2.0(1x) for UCS -6248

Fabric Interconnect

9


10/54

Nexus NX-OS BasicsNexus at its core is a Layer 2/3 SwitchSimilar in many aspects to Catalyst IOS

VLANs, Trunking, VTP, Rapid PVST, MST,

EtherChannel, PVLANs, UDLD, FHRPs, IGPs,BGP, etc.

Key new features beyond Catalyst IOS FEX, vPC, Fabric Path, OTV, FC Switching, FCoE, etc

10


11/54

NX-OS Port Channels/EtherChannels

Unlike IOS, NX -OS does not support PAgP Channels must be statically on or LACP negotiated

no switchport mode desireable

LACP must be enabled with feature lacp One of the killer apps of NX -OS is

Virtual Port Channels (vPC) Multi-Chassis EtherChannel (MEC/MCEC)

Analogous to 3750 Cross StackWise Channel & 6500Virtual Switching System (VSS)

11


12/54

NX-OS Spanning Tree Unlike IOS, NX -OS does not support legacy CST/PVST+

Default STP mode is Rapid-PVST+ i.e. per-VLAN, but uses 802.1w Rapid STP

Also supports 802.1s Multiple Spanning Tree (MST) NX-OS defines three STP port types

spanning-tree port-type [normal | edge | network]

12


13/54

NX-OS Switchport Typesspanning-tree port-type normal

Normal ports act like Catalyst IOS ports Default STP port type, run Rapid Per VLAN STP

spanning-tree port-type edge Edge ports are STP PortFast portsspanning-tree port-type network

Network ports run STP Bridge Assurance

13


14/54

Bridge Assurance All STP Network Ports send BPDUs regardless of STP port state

Legacy 802.1d only sends BPDUs from Root Bridgedownstream

Primary goal is to protect against unidirectional links BPDU becomes a bidirectional keepalive Replaces LoopGuard functionality

Secondary result is same functional effect as VTP Pruning VLANs stop forwarding on trunk links that you do not receive

BPDUs for that VLAN in Enabled on interfaces with spanning-tree port type network

14


15/54

Bridge Assurance Diagram

15

N5K-1 N5K-2

VLANS 10, 20, 30 VLANS 20, 30, 40

10

20

30

20

3040

20, 30

Bridge Assurance

switchport trunk allowed vlan


16/54

NX-OS Layer 3Like Catalyst IOS, NX-OS supports Native layer 3 routed interfaces

I.e. no switchport

Switched Virtual Interfaces (SVIs) I.e. VLAN interfaces Must be enabled with feature interface-vlan

16


17/54

NX-OS Routing Protocols Like IOS, NX -OS supports routing with

Static routing RIPv2 & RIPng

EIGRP & EIGRPv6 OSPF & OSPFv3 IS-IS BGP

Policy Routing No network command in IGPs, activated on link

Not all protocols use the same license

17


18/54


19/54

NX-OS Redistribution Unlike IOS, route -maps are required to performredistribution on NX-OS

Same route-map match/set logic as IOS

Redistribution does not include directly connectedinterfaces

Requires redistribute direct route-map

19


20/54


21/54

22


22/54

Virtual Device Contexts (VDC)VDCs used to virtualize physical hardware of Nexus 7000

Loosely analogous to SDRs in IOS XR or Contexts in ASA VDCs also virtualize control plane protocols of Nexus 7000

Not analogous to VLANs or VRFs in IOS Separate control plane per VDC

VLAN 40 in VDC 1 is not VLAN 40 in VDC 2

OSPF PID 20 in VDC 1 is not OSPF PID 20 in VDC 2

22

23


23/54

Why VDCs? Multiple logical roles per physical chassis

E.g. Core & Aggregation/Distribution on same box

Multi-Tenancy E.g. VDCs as a managed service to customers

Test Lab Environment for later Production UseRequired for certain features FCoE/Storage

23

24


24/54

VDC CaveatsSome features cant co -exist in same VDC

OTV and VLAN interfaces (SVIs) F2 cards and M1/F1

FCoE requires its own Storage VDC

Hardware and Software version dependent, check therelease notes

24

25


25/54

VDC Maximums 4 VDCs per chassis with SUP 1 4+1 VDCs per chassis with SUP 2 8+1 VDCs per chassis with SUP 2E*

No internal cross VDC communication E.g. no route leaking like in VRFs Physical cable can be used to connect VDCs

25

26


26/54

The Default VDC Default VDC 1 always exists and cannot be removed Used to create and manage other VDCs

Controls VDC port allocations

All ports allocated to default VDC at initialization Controls VDC resource allocations

Number of VLANs, VRFs, Routing table memory, etc. Can be used for normal data plane operations

Recommended for management of chassis only

26

27


27/54

Default VDC TasksSome tasks can only be performed in the default VDC

VDC creation/deletion/suspend Resource allocation interfaces, memory, MACs NX-OS Upgrade across all VDCs ISSU or EPLD Upgrade Ethanalyzer captures control plane traffic Feature-set installation for Nexus 2000, FabricPath, FCoE Control Plane Policing (CoPP) Port Channel load balancing hash Hardware IDS check control ACL Capture feature enable System-Wide QoS

27

28


28/54

Converged Ethernet or FCoE

Lots of terms that essentially mean thesame thing

Unified Fabric

Unified Wire Converged Ethernet Converged Enhanced Ethernet Data Center Ethernet Data Center Bridging

What they all really mean You are running the physical framing for both Ethernet and FibreChannel over the same physical links

28

29


29/54

FCoE Terms FCoE Initialization Protocol (FIP) FCoE Forwarder (FCF) ENode End Device

Virtual Fibre Channel (VFC) Interface

29

30


30/54

FC, FCoE, FCIP and iSCSI

30

31


31/54

How FCoE Works FCoE replaces layer 1 & 2 transport for FC All upper layer FC services remain

Domain IDs, FSPF, FCNS, FLOGI, Zoning.

New FCoE Initialization Protocol (FIP) to negotiatebetween Fabric and Node Fabric is the FCF Node is the ENode

31

32


32/54

FCoE Control and Data Planes

FIP is the control plane of FCoE FCoE is the actual data plane FIP

New EtherType 0x8914 Used to discover FCFs and perform FLOGI _ UCS C when FCoE turned on uses LLDP to begin

negotiation

FCoE New Ethertype 0x8906 Min length of 2240 bytes, FC has larger payload

Implies Jumbo Frames are required

32

33


33/54

OTV Basics Overlay Transport Virtualization (OTV)

Layer 2 VPN over IPv4 Specifically OTV is

IPv4/IPv6 over Ethernet over MPLS over GRE over IPv4

34


34/54

OTV vs. other Layer 2 DCIs Layer 2 DCI is needed for Virtual Machine Workload Mobility i.e. VMwareVMotion

Many possible options for L2 DCI Dark Fiber (CWDM/DWDM) Layer 2 Transport Protocol (L2TPv3) Any Transport over MPLS (AToM) Virtual Private LAN Services (VPLS) Bridging over GRE Spanning Tree Bridge Group

These options can be used for DCI, but OTV was made for DCI

Optimizes ARP flooding over DCI Demarc of the STP domain Can overlay multiple VLANs without complicated design Allows multiple edge routers without complicated design

35


35/54

OTV Terms OTV Edge Device

Edge router(s) running OTV Authoritative Edge Device (AED)

Active edge router for a particular VLAN Allows multiple redundant edge routers while

preventing loops Extend VLANs

VLAN being bridged over OTV Site VLAN

Internal VLAN used to elect AED

36


36/54

OTV Terms Continued Site Identifier

Unique ID per DC site, shared between AEDs Internal Interface

Layer 2 interface where traffic to be encapsulated isreceived Overlay Interface

The logical OTV tunnel interface that performs the OTV

encapsulation OTV Join Interface

The Layer 3 physical link or port-channel that you useto route upstream towards the DCI

37


37/54

OTV Overview

VLANS10 - 70

Server 1VLAN 60

Server 2VLAN 60

OTV Overview

DCI Any Layer 3Connection

OTV OverlayLogical Address

OTV OverlayLogical Address

JoinInterface Internal

InterfaceExtendVLANS50 -70

SiteVLAN 4

VLANS50 - 90

OTV Overlay

Logical Address

OTV Overlay

Logical Address

SiteVLAN 3

AEDVLANS50 -59

AEDVLANS60 - 70

VLANS10 - 70

VLANS50 - 90

NK7-1-2

NK7-1-1

NK7-2-2

NK7-2-1

38


38/54

OTV Control Plane Uses IS -IS to advertises MAC addresses between AEDs

Is its own transport and extensible

ISIS Encapsulated as Control Group Multicast IS-IS over Ethernet over MPLS over GRE over

IPv4 Multicast Implies that DCI must support ASM Multicast


39/54

40


40/54

OTV Adjacency Server Normally OTV requires that the DCI runs multicast

Needed to find and form IS-IS adjacencies and to tunnelmulticast data traffic OTV Adjacency Server removes multicast requirement

One (or more) AEDs are configured as the adjacencyserver

All other AEDs register with the adjacency server Now all endpoints are known

All control and data plane traffic is now unicast encapsulated Will result in Head End Replication when more than 2

DCs connected over the DCI

41


41/54

OTV DCI Optimizations Other DCI options bridge all traffic over DCI

STP, ARP, L2 Flooding, broadcast storms, etc.

OTV reduces unnecessary flooding by Proxy ARP/ICMPv6 Cache on AED Terminating the STP domain on AED

42


42/54

vPC Port Channels

Port Channels, EtherChannels, & NIC Teaming/Bondingterms used interchangeably

Regardless of vendor, 802.3ad (LACP) refers to Port Channeling

Used to aggregate bandwidth of multiple links between devices E.g. 4 x physical 1GigE links form a 4GigE logical Port Channel

Appears as one logical link from STPs perspective Avoids active/standby and allows active/active

43


43/54

vPC Port Channels Data flows are load balanced between member links

Single flow cannot exceed BW of any physical member link E.g. increases lanes on the highway but not the speed limit Does not perform LFI like PPP Multilink

Flows are load balanced based on L2, L3, & L4 header information

SRC/DST VLAN, MAC, IP, & TCP/UDP Port Default is SRC/DST L3 for IPv4/IPv6 and SRC/DST MAC for non IP

Can result in over/under subscribed links

44


44/54

Port Channels Port Channeling was original between only 2 devices

1 downstream device & 1 upstream device E.g. end host to Catalyst 3550 via 2 x FE links

Increases BW but still has single point of failure Multi Chassis EtherChannel (MCEC/MEC) is between3 devices

1 downstream device & 2 upstream devices

E.g. end host to 2 x Catalyst 3750s via 2 x GigE links Increases BW and resiliency Logically appears the same as a 2 device Port Channel

45


45/54

Multi Chassis Ethernet Channels 3750 StackWise & 6500 VSS single control plane

StackWise via Stacking Cable to connect BP VSS via Virtual Switch Link (VSL)

vPC uses two separate control planes Configurations managed independently Separate control plane protocol instances

STP, FHRPs, IGPs, BGP, etc. Synchronization via a Peer Link

Similar logic to VSSs VSL

46


46/54

vPC Peer Switches vPC made up of 2 physical switches

The vPC Peers vPC Peers each have

vPC Peer Link vPC Peer Keepalive Link vPC Member Ports

47


47/54

vPC Overview

N5K-1 N5K-2

N7K-1 N7K-2

Access Switch

Peer Link

Peer Keepalive

Member Ports Member Ports

48


48/54

vPC Peer Link Layer 2 trunk link used to sync control plane between vPC peers

CAM table, ARP cache, IGMP Snooping DB, etc. Uses Cisco Fabric Service over Ethernet (CFSoE) protocol Used to elect a vPC Primary and vPC Secondary Role

Normally not used for the data plane Peer Link generally much lower BW than aggregate of

vPC Member Ports If Peer Link used in the data plane, it is the bottleneck

49


49/54

vPC Peer Keepalive Layer 3 link used as heartbeat in the control plane

Used to prevent active/active or Split Brain vPC Roles Not used in the vPC data plane

Uses unicast UDP port 3200 Peer Keepalive Link can be Mgmt0 port

Back to back or over routed infrastructure

Ideally in an isolated VRF

50


50/54

vPC Member Ports Data plane port channel towards downstream neighbor Each vPC Peer has at least one member port per vPC

Can be more, up to hardware platform limits

From perspective of downstream neighbor, upstream vPC Peers are one switch Physical result is a triangle Logical result is a point-to-point Port Channel with no

STP blocking ports


51/54

52


52/54

vPC Loop Prevention Goal of vPC is to hide redundant links from STP

Could result in layer 2 flooding loops Loops are prevented via vPC Check behavior

Frames received in the vPC Peer Link cannot flood out a vPCMember Port while the remote vPC Peer has active vPCMembers in the same vPC

vPC Check Exception If vPC Peers Member Ports are down, the

vPC Member Ports become Orphan Ports and thevPC Check is disabled

vPC Peer Link is essentially a last resort connection

53


53/54

vPC and FHRP Nexus 7000 is typically L2 & L3 network boundary

N7K is vPC Peer but also end hosts FHRP Default Gateway FHRP behavior changes to accommodate active/active forwarding over vPC

Traffic received in vPC Member Port of FHRP Standby to FHRPVirtual MAC is not forwarded over Peer Link to Active FHRP member

Essentially HSRP Standby acts as HSRP Active FHRP vPC can break in certain non-standard vendor applications

Frames sent to FHRP Standby with physical DST MAC of FHRP

Active are sent out the Peer Link peer-gateway allows FHRP Standby to forward frames onbehalf of DST MAC of FHRP Active without going over Peer Link

54


54/54

vPC and Multicast

When source is reachable via vPC Member Port, bothvPC Peers act as PIM DR

Called Dual DR or Proxy DR

Allows either vPC Primary or Secondary to receive trafficfrom source and forward it north without having to cross thevPC Peer Link

Respects vPC check rule

20140205 dfwcug cisco nexus and how it differs from catalyst 6500

Documents