hong kong next generation server access architecture with ... · next generation server access...
TRANSCRIPT
© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Connect 1 1 © 2012 Cisco and/or its affiliates. All rights reserved.
Hong Kong
March 6-7, 2013
Eddie Tan Principal Engineer
Next Generation Server Access architecture with Cisco Nexus 6000
Agenda
Introducing Nexus 6000 Series
Nexus 6004 and Nexus 6001 Overview
Nexus 6000 Positioning
Nexus 6000 Designs
Key Takeaways
Q& A
IT is Moving Towards a World of Many Clouds
Cost
Agility
Flexibility
As Apps Evolve They Demand Infrastructure
Adapt at an Ever Increasing Rate
Public
Private
Hybrid
Communit
y
Is the
Network
Ready?
Key Attributes
Cloud Infrastructure
High-Performance
Intelligent
Programmable
Flexible
Lowest TCO
Introducing
Nexus 6001 1RU 10G switch
Nexus 2248-PQ 1RU 1G/10G FEX
Q2CY13
Q1CY13
48 SFP+ ports and 4 QSFP+ ports
48 SFP+ ports and 4 QSFP+ Uplinks
Performance and Scale
L2 + L3
….. 1 2 32*
1us
NX-OS
7.68 Tbps in 4 RU
FEX
3 X port density of
nearest competitor
Single Management Point
1500+ 10G Servers
Industry leading 10G/
40G ports per RU
First 40G FCoE switch
* Roadmap
Intelligent
Sampled Netflow*
1
31
2
.. Network C
Network B
Network A
* Software Roadmap
Span: Drop,
High Latency*
Microburst Monitoring*
Active Buffer Monitoring*
Latency Monitoring*
Flexible- Platform
40G
FET**
10G
100G*
40G
FC Port*
* Nexus 6004 Roadmap
** Roadmap
….. 1 2 32*
FEX
Connectivity Flexible Choice of 10G or 40G
Convergence FCoE or FC*
Versatile Design with FEX
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 10
Nexus 6000 Overview
Nexus 6004 Feature Highlights
High Performance High Scalability Feature-Rich Visibility and Analytics
• Line rate L2 and L3
with all ports and all
features and all frame
sizes
• 1-microsecond port-to-
port latency with all
frame sizes
• 40-Gbps flow
• 40-Gbps FCoE
• Cut-through switching
for 40 and 10 GE
• 25-MB buffer per three
QSFP interfaces
• Line-rate SPAN
• Sampled NetFlow*
• Buffer monitoring
• Latency monitoring*
• Conditional SPAN-
SPAN on drop-SPAN
on higher latency*
• Micro-burst monitoring*
• L2 and L3 features
• FEXlink
• vPC FabricPath TRILL
• FabricPath with
segment ID
• Adapter-FEX
• /VM-FEX
• 96x40 GE in 4RU
• 384x10 GE in 4RU
• Up to 256,000 MAC
• Up to 128,000 ARP
• 32,000 LPM
• 16,000 bridge
domains*
• 31 Bidirectional SPAN
sessions * Some features in this slide are not supported by software at FCS
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 12
Cisco Nexus 6004 Chassis Port-Side View
• Chassis depth: 30 in.
• Chassis width: 17.5 in.
30 in.
4RU
48 Fixed QSFP
Interfaces
12 QSFP ports
Expansion Module
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 13
Cisco Nexus 6004 Chassis Power Supply and Fans
• The chassis has six power supply slots; a minimum of three is required.
They support both 3 + 1 and 3 + 3 redundancy.
• Each power supply is rated 1100W, 90—240 VAC.
• The chassis has four fan trays. A minimum of three is
required. Power Supply 3 + 3
Grid Redundancy
or 3 + 1 Redundancy
Console Mgmt0 USB
Fan Module 3+1
Redundancy
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 14
Cisco Nexus 6004 Airflow
• Front-to-back or back-to-front airflow
• Port-side exhaust at FCS
• Port-side intake (reversed airflow) with Harbord maintenance release –Q2CY13
• New power supply and fan modules required for different airflow directions
Airflow direction at
FCS
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 15
12xQSFP Line-Card Expansion Modules
Provide 12 QSFP interfaces
Support 40, 10GE, FCoE
Offer same performance, features,
and scalability as fixed ports
Support OIR
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 16
Unified Port Expansion Module
• 16 SFP+ unified ports expansion
module
• 2/4/8/FC port
• 10GE and FCoE
• For Nexus 6004 only
• SFP+ port allows support for more
variety of optical transceiver
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Target
2H CY13
Pre-CC
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 17
100G Expansion Module
• 4 port 100G unified ports expansion
module
• 100GE support including FCOE
• For Nexus 6004 only
• Not CC’d and EC’d. Form factor and
features subject to change
Target
1H CY14
Pre-CC
1
2
3
4
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 18
Nexus 6004 Control Plan Elements
• Single built-in supervisor
• Support L2 ISSU with first software release
CPU Four core 2-GHz Intel Gladden
DRAM 16 GB of DDR3 in four DIMM slots
Program Store 80 GB of iSATA flash for base system storage and partitioned to store image, configuration, log
Boot/BIOS flash memory
8 GB to store upgradable and golden version of (BIOS + bootloader) image
Onboard fault log (OBFL)
64 MB of flash memory to store hardware-related fault and reset reason
NVRAM 6 MB of SRAM to store syslog and licensing information
Management
Interfaces
RS-232 console port
10/100/1000BASE-T management port
USB port
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 19
Cisco Nexus 6000 Key Forwarding Tables
• Host table: 256,000 -entry hashing table;
actual capacity is slightly less than 256,000
• Host table: Shared between MAC, ARP, and
ND and /32 host route
• Host table FCS carving: 128,000 MAC, 128,000 IP
host
• LPM table: 32,000 entries. Also known as summary
routes
• Mroute table: 64,000 entries. 8K by softwrae at FCS
IP Host Region (ARP/ND/Host
route/(*,G))
MAC Region Summary Routes (S,G)
Host Table LPM Table(32,000) Mroute Table(64,000)
256,0
00 E
ntr
ies
* Hardware table size. Please check configuration limit for software scaling
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 20
Multi-tenant with FabricPath Segment ID
L3 Cloud
• 24-bit segment ID is embedded in FabricPath
frame. Extend “VLAN” space to 16 million
• Provide great scale to support large number of
tenants in the fabric
• Supported by N7k F3, N6k and N3548
Traffic from VM with local VLAN
Ingress leaf switch maps local
VLAN to 24-bit segment ID
Egress leaf switch maps
segment ID back to local VLAN
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 21
Flexible Buffer Management
• More Packet Buffer
• 25MB per ASIC(3x40G or 12x10G)
• 800MB for Nexus 6004
Dedicated Buffer
• Predictable performance
• No interference among ports with congestion
• Shared Buffer
• Increase burst absorption capacity
• Efficient use of buffer
• Tunable Buffer Allocation
• Flexible and get benefit of both dedicated and shared buffer scheme
• VOQ Architecture
• Leverage buffer from multiple ASIC
Port 1
Dedicated
Port 2
Dedicated
Port 3
Dedicated
Shared Packet Buffer
SPAN Control
Ingress Buffer 16MB per UPC
Multicast Buffer
Egress Buffer 9MB per UPC
Unicast Buffer
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 22
Virtual Output Queuing
• Take advantage of ingress buffer from multiple port or ASIC for unicast burst
absorption.
• Ensure fairness among multiple ingress ports
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 23
Long Distance FCoE
300m FCoE at FCS (optical transceiver distance limitation)
• 10-km FCoE for 10 GE and 40 GE ports with global QoS policy (roadmap)
‒ Current software implements global network-qos policy to tune the buffer for long-distance FCoE.
‒ Global network-qos policy increases FCoE buffer for all the ports.
‒ >10-km FCoE for 10 GE is possible from perspective of the hardware.
300m FCoE
FCS
10-km FCoE
Future Software Roadmap
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 24
Intelligent SPAN for Better Visibility
Large number of active SPAN sessions
• 31 bidirectional active SPAN sessions supported by
hardware
• 16 bidirectional active SPAN sessions supported at FCS
Line-rate SPAN
• Line-rate SPAN for multiple 40 GE ports
• Built-in extra capacity in fabric to accommodate SPAN
traffic
Intelligent SPAN
• Prioritize data over SPAN in case of fabric link
oversubscription due to SPAN traffic
• Conditional SPAN: SPAN on drop. SPAN on high latency
• Sampled SPAN
• Truncated SPAN
• ERSPAN with 1588 timestamp
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 25
Visibility with SPAN-on-Drop
• Copy the dropped packet to capture device.
• Correlate the packet drop with application
• Software support at Q3CY13
Data Flow
Data Flow
SPAN-on-drop
packets Capturing
Device
Nexus 6000
N6004# sh monitor session 1
session 1
---------------
type : span-on-drop
state : up
acl-name : acl-name not specified
source ports : Eth1/8 Eth1/9
destination ports : Eth1/11
1/8
1/9 1/11
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 26
High Performance SPAN
• No penalty to production data traffic with SPAN
• Extra bandwidth capacity built in for SPAN traffic
• Line-rate SPAN throughput with extra fabric link bandwidth
• First product in the market to support 16 line-rate SPAN sessions
Unified
Crossbar
Fabric
120-Gbps
Data
448
Gbps
448
Gbps
Scheduler Prioritizes
Production Data
Data
Span
Extra Fabric
Bandwidth
for SPAN
224
Gbps
Separate Buffer
Pool for SPAN
120-Gbps
Data
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 27
Cisco Nexus 6004 Features at FCS
In General All Cisco Nexus® 5000 Features up
to 5.2 Release(with the exception of 1G, FC
port, PTP, Trustsec and Flexlink) Are Supported:
• FEX support: 24 FEX for L2 and L3; all Cisco
Nexus 2200 FEX and B22 HP and B22 Fujitsu
are supported.
• vPC
• FabricPath
• IPv4 and IPv6
• PIM-SM, PIM-SSM, BiDir PIM, and MSDP
• FCoE including 40-Gbps FCoE
• Adapter-FEX/VM-FEX
• QoS with flexible buffer carving
• SPAN and ERSPAN source sessions: 16
sessions
• Egress interface drop counters
• PoAP, Python for PoAP
• Increased table scalability
• VRF-Lite
• PACL, VACL, and RACL logging
• VRF Route Leak
• BGP Local-AS
• BGP Allow-AS in
• BGP AS-Path Relax
• BGP disable-peer-as-check
• BGP passive neighbor
• BGP MD5 authentication for passive neighbors
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 28
Cisco Nexus 6000 Feature Timeline*
*Not comprehensive…Subject to change
• FEX support
• vPC
• FabricPath
• IPv4 and IPv6
• PIM-SM, PIM-SSM, BiDir
PIM, and MSDP
• FCoE
• Adapter-FEX/VM-FEX
• QoS with flexible buffer
carving
• ACL and ACL logging
• SPAN and ERSPAN source
sessions: 16 sessions
• PoAP
• Increased table scalability
• VRF-Lite
• VRF route leaking
• BGP enhancement
• Reverse airflow for Cisco
Nexus® 6004
• QSFP-LR4
• Nexus 6001
• Dell FEX
• Ingress policing
• QSFP-FET
• FEX storm control
• PBR
• BFD
• EEM
• 32 FEX support in L2
• TCAM carving
• Forwarding table carving
Harbor(EC)—Q1CY13 Harbord Maintenance
(EC)—Q2CY13
Harbord Plus
(CC)—Q3CY13
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 29
Cisco Nexus 6004 Key Scalability Numbers at FCS
FEXLink
ACL
SPAN
FabricPath
L3
L2
* Single-dimension scalability. Multidimension scalability will be provided at FCS.
• 24 FEX per Cisco Nexus® 6004 in L2 mode
• 24 FEX per Cisco Nexus 6004 in L3 mode
• 4,000 VLANs
• 115,000 MAC
• 32,000 summary routes
• 8,000 mroutes
• 115,000 ARP/host routes
• 128 switch IDs
• 4,000 VLANs
• 16 bidirectional SPAN
• VACL 1024 ACE
• 64,000 STP logical interfaces
• 8,000 IGMP snooping entries
• 256 OSPF peers
• 256 BGP peers
• 256 EIGRP peers
• 2 topologies
• 4 multidestination trees
• IFACL 1152 ACE
• QoS ACL 448 ACE
• 256 PIM peers
• 500 HSRP groups
• 4,000 VRF instances
• RACL 1024 ACE
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 30
Physical Connection Choices
• FCS support: QSFP-SR4 (100m over OM3); copper 40 GE–40 GE cable and copper breakout cable
• FCS stretch: QSFP-CSR4 (300m over OM3); post FCS: QSFP-LR4 (10 KM over SMF)
40G QSFP+
Interfaces
40G QSFP+
Interfaces
40G QSFP+
Interfaces
40G QSFP+
Interfaces
40G QSFP+
Interfaces
40G QSFP+
Interfaces
10G QSFP+
Interfaces
10G QSFP+
Interfaces
QSFP-SR4 QSFP-SR4
1M, 3M, 5M Passive
7M, 10M Active
QSFP-SR4
QSFP-4x10G-7M
QSFP-4x10G-10M
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 31
Supported Transceiver
Type Distance
QSFP-40G-LR4 10 km with SMF
QSFP-40G-SR4 100m with OM3
150m with OM4
QSFP-40G-CSR4 300m with OM3
400m with OM4
QSFP-4x10G-AC7M 7m
QSFP-4x10G-AC10M 10m
QSFP-H40G-1M 1m
QSFP-H40G-3M 3m
QSFP-H40G-5M 5m
QSFP-H40G-AC7M 7m
QSFP-H40G-AC10M 10m
QSFP QSFP
10 km
QSFP SFP+
300m
QSFP SFP+
300m
QSFP QSFP
10 km
FEX
Cisco Nexus®
2248PQ-10G
• QSFP-40G-LR4 will be supported with Q2CY13 software release
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 32
Low Cost FEX Connectivity FET-40G
Cisco Nexus 6000 Cisco Nexus 6000
Cisco Nexus 2232PP and 2232TM-E 2232TM
Cisco Nexus 2248TP-E 2248TP Cisco Nexus 2248PQ-10G
FET-40G
FET-40G
FET-10G
FET-40G
• Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus® 6004
• Supported on Cisco Nexus 6004 and Nexus 2248PQ-10G
• Interoperable with FET-10G
• Support for 100m distance with OM3
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 33
Nexus 6000 Positioning
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 34
LAN/SAN LAN SAN
Cisco Unified Fabric Switching Innovations
CISCO NX-OS: From Hypervisor to Core CISCO DCNM: Single Pane of Management
DELIVERING TO YOUR DATA CENTER NEEDS
Resilient, High Performance,
Scalable Fabric
Operational Efficiency/
Consistency—P-V-C
Workload Mobility
Within/Across DCs
Secure Separation/
Multitenancy
LAN+SAN Convergence
Cisco MDS 9200 Series
Cisco ® MDS 9500 Series
Cisco Nexus 1000V
Cisco Nexus ®
1010
Cisco Nexus 4000
Cisco MDS 9100 Series
Cisco Nexus 5000
Cisco Nexus 3000
Cisco Nexus 2000
Cisco Nexus 7000 Cisco
Nexus 6000
• 10GbE Access Scale/Compact 40GbE Aggregation • Low Latency Spine/Leaf • 10GbE/40GbE Converged Fabric
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 35
Data Center Switching Nexus 7000 and Nexus 6000: DC Considerations
Customer Requirements: Decision Points
Decision Criteria in the Aggregation/Spine
Virtualization Scalability DCI/Mobility Environmentals
L4-7 Services High Availability Latency Investment Protection
Recommended when:
Scale and Flexibility 10G/40G/100G
Highest Availability (HA)
Investment Protection
Multi-Protocol / Services
FEX, VDC, OTV, MPLS/VPLS, LISP
Recommended when:
High density compact 10/40G
Low footprint & low power
Low latency & jitter
Advanced FEX Features
Nexus
7000
Series
17 Tbps
Nexus
6004
Up to 7.68 Tbps
Lead Platform: Modular, High-End Solution Fixed, Mid-Range Solution
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 36
Data Center Switching DC Access Layer Considerations
* Roadmap
Customer Requirements: Decision Points
Decision Criteria in the Access Layer
Connectivity Options Table Sizes 40G Density Environmentals
Line Rate L3 FC Latency FEX Scale
Recommended when:
Medium to High density compact
10G deployments
Integrated and line rate L3
Enhanced scale
(Tables, FEX, throughput)
NAT, PIM-BiDir, Analytics
Recommended when:
Low to medium density 10G deployments
UP/Native FC support
Nexus
6001
Series
Nexus
5500
Series
Nexus 6001 Nexus 5500
Nexus 6004 Hardware Pricing Summary
Starting as low as $90,000 for a 24 port configuration
Flexible software and hardware upgrade options
24 ports
HW bundle
$90,000
SW Lic
$40,000
SW Lic
$40,000 LEM
$40,000
LEM
$40,000 LEM
$40,000
LEM
$40,000
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 38
Nexus 6000 Designs
Access Nexus 6001/Nexus 5000/Nexus 2000
• Different connectivity
options 1G/10G/10GBase-
T/FCoE
• Support all HA design
• N6001: 40G uplink, large
MAC table
Aggregation
Nexus 6004 • High density 10G/40G
• Compact form factor
• Power efficient
• Large MAC and ARP
table
• vPC/vPC+
Nexus 6000 Access Agg vPC Design
N6004-1 N6004-2
vPC domain 100
L2
L3
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
N5k/N6k
N7k/N6k
L3 Core
Nexus 6000 Design FabricPath • . Easy to scale out. Up to 8000x10G ports with 4x Nexus 6004 as spine
• 3:1 oversubscription at FEX
• Leaf to spine 1:1 or 2:1 oversubscription ratio
• Physical hosts and VMs mobility across entire fabric
• Leverage big host table size on N6k for gateway function
10GE 10GE
10G FEX: 2248PQ
Nexus 6004
10GE 10GE
24xFEX
Nexus 6004
Nexus 6004
10GE 10GE
10G FEX: 2248PQ
10GE 10GE
24xFEX
Nexus 6004 L3 Core
FabricPath
L2 Spine
L2 leaf Routed
leaf
FabricPath Architecture Evolution Anycast HSRP Gateway(Future)
Default
Gateway
Default
Gateway
Anycast Gateway
Today(vPC+ with HSRP)
• A pair of FP node running vPC+ support
HSRP Active/Active
• Limit to 2 FP nodes as gateway for a given
VLAN
• Can be placed at spine or have dedicated leaf
to provide routing
FP Anycast HSRP Gateway
• More than two FP nodes as default gateway
• N-way ECMP for Inter-VLAN traffic
• Can be placed at spine or routed leaf
• No need for vPC+ related configuration
Vlan 100-200 Vlan 201-300
Vlan 100-300
FabricPath
FabricPath
L3
L2
L3
L2
Nexus 6000 Design vPC+L3 High Density 1G and 10G
10GE 10GE
10G FEX: 2248PQ
Nexus 7K/6004
10GE 10GE
24xFEX
L3
L2
L3
Nexus 6004
10G POD
1G FEX: 2248TM-E 22xFEX
L3
L2 Nexus 6001
1G POD
L3 Core
• N6k+N2k as POD
• L2 L3 boundary at leaf N6k.
• Easy to scale out spine layer and add more POD for future growth
• 32K ARP support at leaf N6k with first software release. Higher ARP per N6k with future software release
10GE 10GE
10G FEX: 2248PQ
10GE 10GE
24xFEX
L3
L2 Nexus 6004
10G POD
Key Takeaways • High performance compact 10G/40G L2 L3 switch for Datacenter
• Address the challenges of scalability in Datacenter
• Flexible for different architecture design
• Innovative features
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Connect 44
Thank you.