implementing qos with nexus and nx-osd2zmdbbm9feqrf.cloudfront.net/2012/eur/pdf/brkrst-2930.pdf ·...
TRANSCRIPT
BRKRST-2930
Implementing QoS with Nexus and NX-OS
Follow us on Twitter for real time updates of the event:
@ciscoliveeurope, #CLEUR
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 2
Housekeeping
We value your feedback- don't forget to complete your online session evaluations after each session & the Overall Conference Evaluation which will be available online from Thursday
Visit the World of Solutions and Meet the Engineer
Visit the Cisco Store to purchase your recommended readings
Please switch off your mobile phones
After the event don’t forget to visit Cisco Live Virtual: www.ciscolivevirtual.com
Follow us on Twitter for real time updates of the event: @ciscoliveeurope, #CLEUR
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 3
Session Goal
This session will provide a technical description of the NX-OS QoS capabilities and hardware implementations of QoS functions on the Nexus 7000, 5500/5000, 3000 and Nexus 2000 I
t will also include a design and configuration level discussion on the best practices for use of the Cisco Nexus family of switches in implementing QoS for Medianet in additional to new QoS capabilities leveraged in the Data Centre to support FCoE, NAS, iSCSI and vMotion.
This session is designed for network engineers involved in network switching design. A basic understanding of QoS and operation of the Nexus switches 2000/5000/5500/7000 series is assumed.
3
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 4
Housekeeping
We value your feedback- don't forget to complete your online session evaluations after each session & the Overall Conference Evaluation which will be available online from Thursday
Visit the World of Solutions and Meet the Engineer
Visit the Cisco Store to purchase your recommended readings
Please switch off your mobile phones
After the event don’t forget to visit Cisco Live Virtual: www.ciscolivevirtual.com
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 5
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
New QoS Requirements
New QoS Capabilities
Understanding Nexus QoS Capabilities and Configuration
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Applications of QoS with Nexus
Converting a Voice/Video IOS (Catalyst 6500) QoS Configuration to an NX-OS (Nexus 7000) Configuration
Configuring Storage QoS Policies on Nexus 5500 and 7000 (FCoE & iSCSI)
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 6
Evolution of QoS Design Switching Evolution and Specialization
Quality of Service is not just about protecting voice and video anymore
Campus Specialization
Desktop based Unified Communications
Blended Wired & Wireless Access
Data Center Specialization
Compute and Storage Virtualization
Cloud Computing
Consolidation of more protocols onto the fabric
Storage – FCoE, iSCSI, NFS
Inter-Process and compute communication (RCoE, vMotion, … )
VMotion
FCoE
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 7
NX-OS QoS Design Requirements Where are we starting from
VoIP and Video are now mainstream technologies
Ongoing evolution to the full spectrum of Unified Communications
High Definition Executive Communication Application requires stringent Service-Level Agreement (SLA)
Reliable Service—High Availability Infrastructure
Application Service Management—QoS
© 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14497_04_2008_c1 7
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 8
NX-OS QoS Design Requirements QoS for Voice and Video is implicit in current Networks
Application
Class
Per-Hop
Behavior
Admission
Control
Queuing &
Dropping
Application
Examples
VoIP Telephony EF Required Priority Queue (PQ) Cisco IP Phones (G.711, G.729)
Broadcast Video CS5 Required (Optional) PQ Cisco IP Video Surveillance / Cisco Enterprise TV
Realtime Interactive CS4 Required (Optional) PQ Cisco TelePresence
Multimedia Conferencing AF4 Required BW Queue + DSCP WRED Cisco Unified Personal Communicator, WebEx
Multimedia Streaming AF3 Recommended BW Queue + DSCP WRED Cisco Digital Media System (VoDs)
Network Control CS6 BW Queue EIGRP, OSPF, BGP, HSRP, IKE
Call-Signaling CS3 BW Queue SCCP, SIP, H.323
Ops / Admin / Mgmt (OAM) CS2 BW Queue SNMP, SSH, Syslog
Transactional Data AF2 BW Queue + DSCP WRED ERP Apps, CRM Apps, Database Apps
Bulk Data AF1 BW Queue + DSCP WRED E-mail, FTP, Backup Apps, Content Distribution
Best Effort DF Default Queue + RED Default Class
Scavenger CS1 Min BW Queue (Deferential) YouTube, iTunes, BitTorent, Xbox Live
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 9
Time
Critical Data
Realtime
4-Class Model
Best Effort
Signaling / Control Call Signaling
Critical Data
Interactive Video
Voice
8-Class Model
Scavenger
Best Effort
Streaming Video
Network Control
Network Management
Realtime Interactive
Transactional Data
Multimedia Conferencing
Voice
12-Class Model
Bulk Data
Scavenger
Best Effort
Multimedia Streaming
Network Control
Broadcast Video
Call Signaling
http://www.cisco.com/en/US/docs/solutions/Enterprise/WAN_and_MAN/QoS_SRND_40/QoSIntro_40.html#wp61135
NX-OS QoS Design Requirements QoS for Voice and Video is implicit in current Networks
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 10
20 msec
Voice Packets
Bytes
200
600
1000
Audio
Samples
1400
Time
200
600
1000
1400
33 msec
Video Packets Video
Frame
Video
Frame
Video
Frame
NX-OS QoS Design Requirements Attributes of Voice and Video
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 11
Access-Edge Switches
Conditionally Trusted Endpoints
Example: IP Phone + PC
Secure Endpoint
Example: Software-protected PC
With centrally-administered QoS markings
Unsecure Endpoint
Tru
st
Bo
un
dary
Tru
st
Bo
un
dary
NX-OS QoS Design Requirements Trust Boundaries – What have we trusted?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 12
Spectrum of Design Evolution
Ultra Low Latency
• Queueing is designed out of the network whenever possible
• Nanoseconds matter
MSDC
• ECN & Data Center TCP • Hadoop and Incast Loads on
the server ports
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8
blade1 blade2 blade3 blade4 blade5 blade6 blade7 blade8
Virtualized Data Center
• vMotion, iSCSI, FCoE, NAS, CIFS
• Multi Tenant Applications • Voice & Video
HPC/GRID
• Low Latency • Bursty Traffic
(workload migration) • IPC • iWARP & RCoE
NX-OS QoS Design Requirements What else do we need to consider?
The Data Center adds a number of new traffic types and requirements
No Drop, IPC, Storage, Vmotion, …
New Protocols and mechanisms
802.1Qbb, 802.1Qaz, ECN, …
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 13
Data Centre architecture can be provide a new set of trust boundaries
Virtual Switch extends the trust boundary into the memory space of the Hypervisor
Converged and Virtualized Adapters provide for local classification, marking and queuing
vPC
vPC
VM #4
VM #3
VM #2
N1KV – Classification,
Marking & Queuing
COS Based Queuing in the
extended Fabric
NX-OS QoS Requirements What do we trust and where do classify and mark?
Trust Boundary
CNA/A-FEX - Classification and Marking
N2K – CoS Marking
COS Based Queuing in the
extended Fabric
N5K – CoS/DSCP Marking,
Queuing and Classification
N7K – CoS/DSCP Marking,
Queuing and Classification
COS/DSCP Based Queuing in the extended
Fabric
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 14
PCP/COS Network
priority Acronym Traffic characteristics
1 0 (lowest) BK Background
0 1 BE Best Effort
2 2 EE Excellent Effort
3 3 CA Critical Applications
4 4 VI Video, < 100 ms latency
5 5 VO Voice, < 10 ms latency
6 6 IC Internetwork Control
IEEE 802.1Q-2005
NX-OS QoS Requirements CoS or DSCP?
We have non IP based traffic to consider again
FCoE – Fibre Channel Over Ethernet
RCoE – RDMA Over Ethernet
DSCP is still marked but CoS will be required and used in Nexus Data Center designs
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 15
Traffic Type Network Class COS Class, Property, BW
Allocation
Infrastructure Control 6 Platinum, 10%
vMotion 4 Silver, 20%
Tenant
Gold, Transactional 5 Gold, no drop, 30%
Silver, Transactional 2 Bronze, 15%
Bronze, Transactional 1 Best effort, 10%
Storage FCOE 3 No Drop, 15%
NFS datastore 5 Silver
Non Classified Data 1 Best Effort
In this example of a Virtualized Multi-Tenant Data Center there is a potential overlap/conflict with Voice/Video queueing assignments, e.g.
COS 3 – FCoE ‘and’ Call Control
COS 5 – NFS ‘and’ Voice bearer traffic
NX-OS QoS Requirements Where do we put the new traffic types?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 16
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
New QoS Requirements
New QoS Capabilities
Understanding Nexus QoS Capabilities
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Nexus 1000v
Applications of QoS with Nexus
Voice and Video
Storage & FCoE
Future QoS Design Considerations (Data Center TCP, ECN, optimized TCP)
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 17
Data Center Bridging Control Protocol DCBX Overview - 802.1Qaz
DCBX Switch
DCBX CNA
Adapter
Negotiates Ethernet capability’s : PFC, ETS, CoS values between DCB capable peer devices
Simplifies Management : allows for configuration and distribution of parameters from one node to another
Responsible for Logical Link Up/Down signaling of Ethernet and Fibre Channel
DCBX is LLDP with new TLV fields
The original pre-standard CIN (Cisco, Intel, Nuova) DCBX utilized additional TLV’s
DCBX negotiation failures result in:
per-priority-pause not enabled on CoS values
vfc not coming up – when DCBX is being used in FCoE environment
dc11-5020-3# sh lldp dcbx interface eth 1/40
Local DCBXP Control information:
Operation version: 00 Max version: 00 Seq no: 7 Ack no: 0
Type/
Subtype Version En/Will/Adv Config
006/000 000 Y/N/Y 00
<snip>
https://www.cisco.com/en/US/netsol/ns783/index.html
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 18
Packet
R_R
DY
Fibre Channel
Transmit Queues Ethernet Link
Receive Buffers
Eight
Virtual
Lanes
One One
Two Two
Three Three
Four Four
Five Five
Seven Seven
Eight Eight
Six Six
STOP PAUSE
B2B Credits
Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1p
When link is congested, CoS assigned to “no-drop” will be PAUSED
Other traffic assigned to other CoS values will continue to transmit and rely on upper layer protocols for retransmission
Not only for FCoE traffic
Priority Flow Control FCoE Flow Control Mechanism – 802.1Qbb
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 19
Offered Traffic
t1 t2 t3
10 GE Link Realized Traffic Utilization
3G/s HPC Traffic
3G/s
2G/s
3G/s Storage Traffic
3G/s
3G/s
LAN Traffic
4G/s
5G/s 3G/s
t1 t2 t3
3G/s 3G/s
3G/s 3G/s 3G/s
2G/s
3G/s 4G/s 6G/s
Prevents a single traffic class of “hogging” all the bandwidth and starving other classes
When a given load doesn’t fully utilize its allocated bandwidth, it is available to other classes
Helps accommodate for classes of a “bursty” nature
Enhanced Transmission Selection (ETS) Bandwidth Management – 802.1Qaz
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 20
Data Center TCP Explicit Congestion Notification (ECN)
ECN is an extension to TCP that provides end-to-end congestion notification without dropping packets. Both the network infrastructure and the end hosts have to be capable of supporting ECN for it to function properly. ECN uses the two least significant bits in the Diffserv field in the IP header to encode four different values. During periods of congestion a router will mark the DSCP header in the packet indicating congestion (0x11) to the receiving host who should notify the source host to reduce its transmission rate.
N3K-1(config)# policy-map type network-qos traffic-priorities
N3K-1(config-pmap-nq)# class type network-qos class-gold
N3K-1(config-pmap-nq-c)# congestion-control random-detect ecn
The configuration for enabling ECN is very similar to the previous WRED example, so only the policy-map configuration with the ecn option is displayed for simplicity.
ECN Configuration:
Diffserv field Values in the IP Header
0x00 – Non ENC-Capable Transport
0x10 - ECN Capable Transport (0)
0x01 – ECN Capable Transport (1)
0x11 – Congestion Encountered
WRED and ECN are always applied to the system policy
Notes: When configuring ECN ensure there are not any queuing policy-maps
applied to the interfaces. Only configure the queuing policy under the system policy.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 21
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
Nexus and NX-OS
New QoS Capabilities and Requirements
Understanding Nexus QoS Capabilities
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Nexus 1000v
Applications of QoS with Nexus
Voice and Video
Storage & FCoE
Hadoop and Web 2.0
Future QoS Design Considerations (Data Center TCP, ECN, optimized TCP)
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 22
Nexus 7000 I/O Module Families M and F Series Line Cards
M family – L2/L3/L4 with large forwarding tables and rich feature set
F family – High performance, low latency, low power and streamlined feature set
N7K-M108X2-12L
N7K-M132XP-12/
N7K-M132XP-12L
N7K-M148GT-11/N7K-M148GT-11L
N7K-M148GS-11/N7K-M148GS-11L
N7K-F132XP-15 N7K-F248XP-25
Now Shipping
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 23
Nexus 7000 M1 I/O Module QoS Capabilities
Modular QoS CLI Model
3-step model to configure and apply policies:
– Define match criteria (class-map)
– Associate actions with the match criteria (policy-map)
– Attach set of actions to interface (service-policy)
Two types of class-maps/policy-maps (C3PL provides option of type)
– type qos – to configure marking rules (default type)
– type queuing – to configure port based QoS rules
PHY R2D2 EARL EARL R2D2 PHY
Ingress Queuing
policies enforced here Ingress QoS policies
enforced here
Egress QoS policies
enforced here
Egress Queuing
policies enforced here
Ingress Linecard Egress Linecard
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 24
CoS mutation
IP Prec mutation
IP DSCP mutation
Class-map matching criteria: ACL-based
(SMAC/DMAC, IP SA/DA, Protocol, L4 ports, L4 protocol fields)
CoS
IP Precedence
DSCP
COS-to-queue mapping
Bandwidth allocation (DWRR)
Buffer allocation
Congestion Avoidance (WRED1 and tail drop)
Set COS
IP Prec
IP DSCP
QoS Group
Discard Class
1-rate 2-color and 2-rate 3-color aggregate policing
Shared policers
Color-aware policing
Policing actions: Transmit
Drop
Change CoS/IP-Prec/DSCP
Markdown
Set QoS Group or Discard Class
Input Queuing
& Scheduling Ingress Classification Marking
Ingress
Mutation Ingress Policing
Applied at
ingress port
ASIC
Applied at ingress forwarding
engine (ingress pipe)
1. WRED on ingress GE ports only
Nexus 7000 M1 I/O Module QoS Ingress Capabilities
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 25
1-rate 2-color and 2-rate 3-color aggregate policing
Shared policers
Color-aware aggregate policing
Policing actions: Transmit
Drop
Change CoS/IP-Prec/DSCP
Markdown
Egress Policing
CoS mutation
IP Prec mutation
IP DSCP mutation
COS-to-queue mapping
Bandwidth allocation
Buffer allocation
Congestion avoidance (WRED & tail drop)
Priority queuing
SRR (no PQ)
Output Queuing
& Scheduling
Egress
Mutation
Class-map matching criteria: ACL-based (L2
SA/DA, IP SA/DA, Protocol, L4 port range, L4 protocol specific field)
CoS
IP Precedence
DSCP
Protocols (non-IP)
QoS Group
Discard Class
Egress Classification
Applied at
egress port
ASIC
Applied at ingress forwarding
engine (egress pipe)
CoS
IP Prec
IP DSCP
Marking
Nexus 7000 M1 I/O Module QoS Egress Capabilities
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 26
How to Configure Queuing on Nexus 7000
Key concept: Queuing service policies
Queuing service policies leverage port ASIC capabilities to map traffic to queues and schedule packet delivery
Define queuing classes
Class maps that define the COS-to-queue mapping
i.e., which COS values go in which queues?
Define queuing policies
Policy maps that define how each class is treated
i.e., how does the queue belonging to each class behave?
Apply queuing service policies
Service policies that apply the queuing policies
i.e., which policy is attached to which interface in which direction?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 27
Queuing Classes
class-map type queuing – Configure COS-queue mappings
Queuing class-map names are static, based on port-type and queue
tstevens-7010(config)# class-map type queuing match-any
1p3q4t-out-pq1 1p7q4t-out-q-default 1p7q4t-out-q6 8q2t-in-q1 8q2t-in-q6
1p3q4t-out-q-default 1p7q4t-out-q2 1p7q4t-out-q7 8q2t-in-q2 8q2t-in-q7
1p3q4t-out-q2 1p7q4t-out-q3 2q4t-in-q-default 8q2t-in-q3
1p3q4t-out-q3 1p7q4t-out-q4 2q4t-in-q1 8q2t-in-q4
1p7q4t-out-pq1 1p7q4t-out-q5 8q2t-in-q-default 8q2t-in-q5
tstevens-7010(config)# class-map type queuing match-any 1p3q4t-out-pq1
tstevens-7010(config-cmap-que)# match cos 7
tstevens-7010(config-cmap-que)#
Configurable only in default VDC
Changes apply to ALL ports of specified type in ALL VDCs
Changes are traffic disruptive for ports of specified type
10G ingress port type
1G ingress port type
10G egress port type 1G egress port type
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 28
Queuing Policies
policy-map type queuing – Define per-queue behavior such as queue size, WRED, shaping
tstevens-7010(config)# policy-map type queuing pri-q
tstevens-7010(config-pmap-que)# class type queuing 1p3q4t-out-pq1
tstevens-7010(config-pmap-c-que)#
bandwidth no queue-limit set
exit priority random-detect shape
tstevens-7010(config-pmap-c-que)#
Note that some “sanity” checks only performed when you attempt to tie the policy to an interface
e.g., WRED on ingress 10G ports
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 29
Queue Attributes
priority — defines queue as the priority queue
bandwidth — defines WRR weights for each queue
shape — defines SRR weights for each queue Note: enabling shaping disables PQ support for that port
queue-limit — defines queue size and defines tail-drop thresholds
random-detect — sets WRED thresholds for each queue
Note: WRED and tail-drop parameters are mutually exclusive on a per-queue basis
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 30
Queuing Service Policies
service-policy type queuing – Attach a queuing policy-map to an interface
Queuing policies always tied to physical port
No more than one input and one output queuing policy per port
tstevens-7010(config)# int e1/1
tstevens-7010(config-if)# service-policy type queuing input my-in-q
tstevens-7010(config-if)# service-policy type queuing output my-out-q
tstevens-7010(config-if)#
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 31
QoS “Golden Rules”
Assuming DEFAULTS –
For bridged traffic, COS is preserved, DSCP is unmodified
For routed traffic, DSCP is preserved, DSCP[0:2] (as defined by RFC2474) copied to COS
For example, DSCP 40 (b101000) becomes COS 5 (b101)
Changes to default queuing policies, or application of QoS marking policies, can modify this behavior
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 32
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
Nexus and NX-OS
New QoS Capabilities and Requirements
Understanding Nexus QoS Capabilities
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Applications of QoS with Nexus
Voice and Video
Storage & FCoE
Hadoop and Web 2.0
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 33
Nexus 5000/5500 QoS QoS Capabilities and Configuration
Nexus 5000 supports a new set of QoS capabilities designed to provide per system class based traffic control
Lossless Ethernet—Priority Flow Control (IEEE 802.1Qbb)
Traffic Protection—Bandwidth Management (IEEE 802.1Qaz)
Configuration signaling to end points—DCBX (part of IEEE 802.1Qaz)
These new capabilities are added to and managed by the common Cisco MQC (Modular QoS CLI) which defines a three-step configuration model
Define matching criteria via a class-map
Associate action with each defined class via a policy-map
Apply policy to entire system or an interface via a service-policy
Nexus 5000/7000 leverage the MQC qos-group capabilities to identify and define traffic in policy configuration
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 34
Nexus 5000/5500 QoS Packet Forwarding: Ingress Queuing
In typical Data Center access designs multiple ingress access ports transmit to a few uplink ports
Nexus 5000 and 5500 utilize an Ingress Queuing architecture
Packets are stored in ingress buffers until egress port is free to transmit
Ingress queuing provides an additive effective
The total queue size available is equal to [number of ingress ports x queue depth per port]
Statistically ingress queuing provides the same advantages as shared buffer memory architectures
Egress Queue
0 is full, link
congested
Traffic is Queued on all ingress interface
buffers providing a cumulative scaling of
buffers for congested ports
v
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 35
Nexus 5000/5500 QoS Virtual Output Queues
Unified Crossbar
Fabric
Egress
Queue 0
is full
Egress
Queue 0
is free
Packets
Queued for
Eth 1/20
Eth 1/20
VoQ Eth
1/20 VoQ Eth
1/8
Eth 1/8
Packet is able to
be sent to the
fabric for Eth 1/8
Nexus 5000 and 5500 use an 8 Queue QoS model for unicast traffic
Traffic is Queued on the Ingress buffer until the egress port is free to transmit the packet
To prevent Head of Line Blocking (HOLB) Nexus 5000 and 5500 use a Virtual Output Queue (VoQ) Model
Each ingress port has a unique set of 8 virtual output queues for every egress port (1024 Ingress VOQs = 128 destinations * 8 classes on every ingress port)
If Queue 0 is congested for any port traffic in Queue 0 for all the other ports is still able to be transmitted
Common shared buffer on ingress, VoQ are pointer lists and not physical buffers
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 36
Nexus 5000/5500 QoS QoS Policy Types
There are three QoS policy types used to define system behavior (qos, queuing, network-qos)
There are three policy attachment points to apply these policies to
Ingress interface
System as a whole (defines global behavior)
Egress interface
Ingress UPC
Egress UPC
Unified Crossbar
Fabric
Policy Type Function Attach Point
qos Define traffic classification rules system qos
ingress Interface
queuing Strict Priority queue
Deficit Weight Round Robin
system qos
egress Interface
ingress Interface
network-qos System class characteristics (drop or no-
drop, MTU), Buffer size, Marking system qos
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 37
Nexus 5500 QoS QoS Defaults
# Predefined FCoE service policies
service-policy type qos input fcoe-default-in-policy
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type network-qos fcoe-default-nq-policy
QoS is enabled by default (not possible to turn it off)
Three default class of services defined when system boots up
Two for control traffic (CoS 6 & 7)
Default Ethernet class (class-default – all others)
Cisco Nexus 5500 switch supports five user-defined classes and the one default drop system class
FCoE queues are ‘not’ pre-allocated
When configuring FCoE the predefined service policies must be added to existing QoS configurations
Gen 2 UPC
Unified Crossbar Fabric
Gen 2 UPC
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 38
Nexus 5500 Series Layer 3 QoS Configuration
Internal QoS information determined by ingress Carmel (UPC) ASIC is ‘not’ passed to the Lithium L3 ASIC
Need to mark all routed traffic with a dot1p CoS value used to:
Queue traffic to and from the Lithium L3 ASIC
Restore qos-group for egress forwarding
Mandatory to setup CoS for the frame in the network-qos policy, one-to-one mapping between a qos-group and CoS value
Classification can be applied to physical interfaces (L2 or L3, including L3 port-channels) not to SVIs
Gen 2 UPC
Unified Crossbar Fabric Gen 2
Gen 2 UPC
Gen 2 UPC
Layer 3 Forwarding Engine
Gen 2 UPC
Routed packet is queued on egress from
Lithium based on dot1p
On initial ingress packet QoS matched and packet is associated with a qos-group for queuing and policy
enforcement
Packet qos-group is not passed to Lithium, leverages CoS dot1p
class-map type network-qos nqcm-grp2 match qos-group 2 class-map type network-qos nqcm-grp4 match qos-group 4 policy-map type network-qos nqpm-grps class type network-qos nqcm-grp2 set cos 4 class type network-qos nqcm-grp4 set cos 2
If traffic is congested on ingress to L3 ASIC it is queued on ingress UPC ASIC
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 39
Nexus 5500 Series Layer 3 QoS Configuration
Apply “type qos” and network-qos policy for classification on the L3 interfaces and on the L2 interfaces (or simply system wide)
Applying “type queuing” policy at system level in egress direction (output)
Trident has CoS queues associated with every interface
8 Unicast CoS queues
4 Multicast CoS queues
The individual dot1p priorities are mapped one-to-one to the Unicast CoS queues
This has the result of dedicating a queue for every traffic class
With the availability of only 4 multicast queues the user would need to explicitly map dot1p priorities to the multicast queues
wrr-queue cos-map <queue ID> <CoS Map>
Gen 2 UPC
Unified Crossbar Fabric Gen 2
Gen 2 UPC
Gen 2 UPC
Layer 3 Forwarding Engine
Gen 2 UPC
8 Unicast Queues
4 Multicast
Queues
Nexus-5500(config)# wrr-queue cos-map 0 1 2 3 Nexus-5500(config)# sh wrr-queue cos-map MCAST Queue ID Cos Map 0 0 1 2 3 1 2 4 5 3 6 7
8 Unicast Queues
8 Multicast
Queues
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 40
Nexus 5000/5500 QoS Mapping the Switch Architecture to ‘show queuing’
dc11-5020-4# sh queuing int eth 1/39
Interface Ethernet1/39 TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 50
1 WRR 50
Interface Ethernet1/39 RX Queuing
qos-group 0
q-size: 243200, HW MTU: 1600 (1500 configured)
drop-type: drop, xon: 0, xoff: 1520
Statistics:
Pkts received over the port : 85257
Ucast pkts sent to the cross-bar : 930
Mcast pkts sent to the cross-bar : 84327
Ucast pkts received from the cross-bar : 249
Pkts sent to the port : 133878
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
<snip – other classes repeated>
Total Multicast crossbar statistics:
Mcast pkts received from the cross-bar : 283558
SFP SFP SFP SFP
Unified
Crossbar
Fabric
UPC
Egress (Tx) Queuing
Configuration
Packets Arriving on this port
but dropped from ingress
queue due to congestion on
egress port
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 41
Configuring QoS on the Nexus 5500 Create New System Class
Step 1 Define qos Class-Map
Step 2 Define qos Policy-Map
Step 3 Apply qos Policy-Map under
“system qos” or interface
N5k(config)# ip access-list acl-1
N5k(config-acl)# permit ip 100.1.1.0/24 any
N5k(config-acl)# exit
N5k(config)# ip access-list acl-2
N5k(config-acl)# permit ip 200.1.1.0/24 any
N5k(config)# class-map type qos class-1
N5k(config-cmap-qos)# match access-group name acl-1
N5k(config-cmap-qos)# class-map type qos class-2
N5k(config-cmap-qos)# match access-group name acl-2
N5k(config-cmap-qos)#
N5k(config)# policy-map type qos policy-qos
N5k(config-pmap-qos)# class type qos class-1
N5k(config-pmap-c-qos)# set qos-group 2
N5k(config-pmap-c-qos)# class type qos class-2
N5k(config-pmap-c-qos)# set qos-group 3
N5k(config)# system qos
N5k(config-sys-qos)# service-policy type qos input policy-qos
N5k(config)# interface e1/1-10
N5k(config-sys-qos)# service-policy type qos input policy-qos
Create two system classes for
traffic with different source
address range
Supported matching criteria N5k(config)# class-map type qos class-1
N5k(config-cmap-qos)# match ?
access-group Access group
cos IEEE 802.1Q class of service
dscp DSCP in IP(v4) and IPv6 packets
ip IP
precedence Precedence in IP(v4) and IPv6 packets
protocol Protocol
N5k(config-cmap-qos)# match
Qos-group range for user-
configured system class is 2-5
Policy under system qos applied
to all interfaces
Policy under interface is preferred
if same type of policy is applied
under both system qos and
interface
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 42
Configuring QoS on the Nexus 5500 Create New System Class(Continue)
Step 4 Define network-qos Class-Map
Step 5 Define network-qos Policy-Map
Step 6 Apply network-qos policy-map under
system qos context
N5k(config)# class-map type network-qos class-1
N5k(config-cmap-nq)# match qos-group 2
N5k(config-cmap-nq)# class-map type network-qos class-2
N5k(config-cmap-nq)# match qos-group 3
N5k(config)# policy-map type network-qos policy-nq
N5k(config-pmap-nq)# class type network-qos class-1
N5k(config-pmap-nq-c)# class type network-qos class-2
N5k(config-pmap-nq-c)# system qos
N5k(config-sys-qos)# service-policy type network-qos policy-nq
N5k(config-sys-qos)#
No action tied to this class
indicates default network-qos
parameters.
Policy-map type network-qos will
be used to configure no-drop
class, MTU, ingress buffer size
and 802.1p marking
Default network-qos parameters
are listed in the table below
Network-QoS
Parameters
Default Value
Class Type Drop class
MTU 1538
Ingress Buffer Size 20.4KB
Marking No marking
Match qos-group is the
only option for network-
qos class-map
Qos-group value is set by
qos policy-map in
previous slide
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 43
Configuring QoS on the Nexus 5500 Strict Priority and Bandwidth Sharing Create new system class by using policy-map qos and network-
qos(Previous two slides)
Then Define and apply policy-map type queuing to configure strict priority and bandwidth sharing
Checking the queuing or bandwidth allocating with command show queuing interface
N5k(config)# class-map type queuing class-1
N5k(config-cmap-que)# match qos-group 2
N5k(config-cmap-que)# class-map type queuing class-2
N5k(config-cmap-que)# match qos-group 3
N5k(config-cmap-que)# exit
N5k(config)# policy-map type queuing policy-BW
N5k(config-pmap-que)# class type queuing class-1
N5k(config-pmap-c-que)# priority
N5k(config-pmap-c-que)# class type queuing class-2
N5k(config-pmap-c-que)# bandwidth percent 40
N5k(config-pmap-c-que)# class type queuing class-fcoe
N5k(config-pmap-c-que)# bandwidth percent 40
N5k(config-pmap-c-que)# class type queuing class-default
N5k(config-pmap-c-que)# bandwidth percent 20
N5k(config-pmap-c-que)# system qos
N5k(config-sys-qos)# service-policy type queuing output policy-BW
N5k(config-sys-qos)#
Define queuing class-map
Define queuing policy-map
Apply queuing policy under
system qos or egress interface
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 44
Configuring QoS on the Nexus 5500 Set Jumbo MTU
Nexus 5000 supports different MTU for each system class
MTU is defined in network-qos policy-map
No interface level MTU support on Nexus 5000
Following example configures jumbo MTU for all interfaces
N5k(config)# policy-map type network-qos policy-MTU
N5k(config-pmap-uf)# class type network-qos class-default
N5k(config-pmap-uf-c)# mtu 9216
N5k(config-pmap-uf-c)# system qos
N5k(config-sys-qos)# service-policy type network-qos policy-MTU
N5k(config-sys-qos)#
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 45
Configuring QoS on the Nexus 5500 Adjust N5k Ingress Buffer Size
Step 1 Define qos class-map
Step 2 Define qos policy-map
N5k(config)# ip access-list acl-1
N5k(config-acl)# permit ip 100.1.1.0/24 any
N5k(config-acl)# exit
N5k(config)# ip access-list acl-2
N5k(config-acl)# permit ip 200.1.1.0/24 any
N5k(config)# class-map type qos class-1
N5k(config-cmap-qos)# match access-group name acl-1
N5k(config-cmap-qos)# class-map type qos class-2
N5k(config-cmap-qos)# match access-group name acl-2
N5k(config-cmap-qos)#
N5k(config)# policy-map type qos policy-qos
N5k(config-pmap-qos)# class type qos class-1
N5k(config-pmap-c-qos)# set qos-group 2
N5k(config-pmap-c-qos)# class type qos class-2
N5k(config-pmap-c-qos)# set qos-group 3
N5k(config)# system qos
N5k(config-sys-qos)# service-policy type qos input policy-qos
Step 3 Apply qos policy-map under
system qos
Step 4 Define network-qos Class-Map
Step 5 Set ingress buffer size for class-1 in network-qos policy-map
N5k(config)# class-map type network-qos class-1
N5k(config-cmap-nq)# match qos-group 2
N5k(config-cmap-nq)# class-map type network-qos class-2
N5k(config-cmap-nq)# match qos-group 3
N5k(config)# policy-map type network-qos policy-nq
N5k(config-pmap-nq)# class type network-qos class-1
N5k(config-pmap-nq-c) queue-limit 81920 bytes
N5k(config-pmap-nq-c)# class type network-qos class-2
N5k(config-pmap-nq-c)# system qos
N5k(config-sys-qos)# service-policy type network-qos policy-nq
N5k(config-sys-qos)#
Step 6 Apply network-qos policy-map
under system qos context
Step 7 Configure bandwidth allocation
using queuing policy-map
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 46
Configuring QoS on the Nexus 5500 Configure no-drop system class
Step 1 Define qos class-map
Step 2 Define qos policy-map
N5k(config)# class-map type qos class-nodrop
N5k(config-cmap-qos)# match cos 4
N5k(config-cmap-qos)#
N5k(config)# policy-map type qos policy-qos
N5k(config-pmap-qos)# class type qos class-nodrop
N5k(config-pmap-c-qos)# set qos-group 2
N5k(config)# system qos
N5k(config-sys-qos)# service-policy type qos input policy-qos
Step 3 Apply qos policy-map under
system qos
Step 4 Define network-qos Class-Map
N5k(config)# class-map type network-qos class-1
N5k(config-cmap-nq)# match qos-group 2
N5k(config)# policy-map type network-qos policy-nq
N5k(config-pmap-nq)# class type network-qos class-nodrop
N5k(config-pmap-nq-c) pause no-drop
N5k(config-pmap-nq-c)# system qos
N5k(config-sys-qos)# service-policy type network-qos policy-nq
N5k(config-sys-qos)#
Step 5 Configure class-nodrop as no-drop class in network-qos policy-map
Step 6 Apply network-qos policy-map
under system qos context
Step 7 Configure bandwidth allocation
using queuing policy-map
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 47
Configuring QoS on the Nexus 5500 Configure CoS Marking
Step 1 Define qos class-map
Step 2 Define qos policy-map
N5k(config)# ip access-list acl-1
N5k(config-acl)# permit ip 100.1.1.0/24 any
N5k(config-acl)# exit
N5k(config)# class-map type qos class-1
N5k(config-cmap-qos)# match access-group name acl-1
N5k(config-cmap-qos)#
N5k(config)# policy-map type qos policy-qos
N5k(config-pmap-qos)# class type qos class-1
N5k(config-pmap-c-qos)# set qos-group 2
N5k(config)# system qos
N5k(config-sys-qos)# service-policy type qos input policy-qos
Step 3 Apply qos policy-map under
system qos
Step 4 Define network-qos Class-Map
Step 5 Enable CoS marking for class-1 in network-qos policy-map
N5k(config)# class-map type network-qos class-1
N5k(config-cmap-nq)# match qos-group 2
N5k(config)# policy-map type network-qos policy-nq
N5k(config-pmap-nq)# class type network-qos class-1
N5k(config-pmap-nq-c) set cos 4
N5k(config-pmap-nq-c)# system qos
N5k(config-sys-qos)# service-policy type network-qos policy-nq
N5k(config-sys-qos)#
Step 6 Apply network-qos policy-map
under system qos context
Step 7 Configure bandwidth allocation for
new system class using queuing policy-
map
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 48
Configuring QoS on the Nexus 5500 Check System Classes
N5k# show queuing interface ethernet 1/1
Interface Ethernet1/1 TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 20
1 WRR 40
2 priority 0
3 WRR 40
Interface Ethernet1/1 RX Queuing
qos-group 0:
q-size: 163840, MTU: 1538
drop-type: drop, xon: 0, xoff: 1024
Statistics:
Pkts received over the port : 9802
Ucast pkts sent to the cross-bar : 0
Mcast pkts sent to the cross-bar : 9802
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 18558
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
qos-group 1:
q-size: 76800, MTU: 2240
drop-type: no-drop, xon: 128, xoff: 240
Statistics:
Pkts received over the port : 0
Ucast pkts sent to the cross-bar : 0
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
Continue…
qos-group 2:
q-size: 20480, MTU: 1538
drop-type: drop, xon: 0, xoff: 128
Statistics:
Pkts received over the port : 0
Ucast pkts sent to the cross-bar : 0
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx
(Inactive)
qos-group 3:
q-size: 20480, MTU: 1538
drop-type: drop, xon: 0, xoff: 128
Statistics:
Pkts received over the port : 0
Ucast pkts sent to the cross-bar : 0
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 0
Pkts sent to the port : 0
Pkts discarded on ingress : 0
Per-priority-pause status : Rx (Inactive), Tx
(Inactive)
Total Multicast crossbar statistics:
Mcast pkts received from the cross-bar : 18558
N5k#
Strict priority and WRR
configuration
class-default
class-fcoe
User-configured system
class: class-1
User-configured system
class: class-2
Packet counter
for each class
Drop counter for
each class
Current PFC status
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 49
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
Nexus and NX-OS
New QoS Capabilities and Requirements
Understanding Nexus QoS Capabilities
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Nexus 1000v
Applications of QoS with Nexus
Voice and Video
Storage & FCoE
Hadoop and Web 2.0
Future QoS Design Considerations (Data Center TCP, ECN, optimized TCP)
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 50
Nexus 2000 QoS Tuning the Port Buffers
Each Fabric Extender (FEX) has local port buffers
You can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host)
You can use a lower queue limit value on the Fabric Extender to prevent one blocked receiver from affecting traffic that is sent to other non-congested receivers ("head-of-line blocking”)
A higher queue limit provides better burst absorption and less head-of-line blocking protection
Gen 2 UPC
Unified Crossbar Fabric
Gen 2 UPC
Nexus 2000
FEX ASIC
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 51
N5k/N2k QoS Processing Flow
1 Incoming traffic is classified based on CoS.
2 Queuing and scheduling at egress of NIF
3 Traffic classification, buffer allocation, MTU check and CoS marking at N5k ingress
4 Queuing and scheduling at N5k egress.
5 CoS based classification at the ingress of NIF ports
6 Queuing and scheduling at egress of HIF ports. Egress tail drop for each HIF port
FEX(2148, 2248, 2232)
Unified Port
Controller
Unified Swtich Fabric
Unified Port
Controller
1
2
3 4
5
6
Nexus 5000
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 52
Nexus 2000 QoS Tuning the Port Buffers
Each Fabric Extender (FEX) has local port buffers
You can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host)
You can use a lower queue limit value on the Fabric Extender to prevent one blocked receiver from affecting traffic that is sent to other non-congested receivers ("head-of-line blocking”)
A higher queue limit provides better burst absorption and less head-of-line blocking protection
Gen 2 UPC
Unified Crossbar Fabric
Gen 2 UPC
Nexus 2000
FEX ASIC
40G Fabric
1G Sink
10G Source (NFS)
# Disabling the per port tail drop threshold
dc11-5020-3(config)# system qos
dc11-5020-3(config-sys-qos)# no fex queue-limit
dc11-5020-3(config-sys-qos)#
# Tuning of the queue limit per FEX HIF port
dc11-5020-3(config)# fex 100
dc11-5020-3(config-fex)# hardware N2248T queue-limit 356000
dc11-5020-3(config-fex)# hardware N2248T queue-limit ?
<CR>
<2560-652800> Queue limit in bytes
10G
NF
S
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 53
• Speed mismatch between 10G NAS and 1G server requires QoS tuning
• Nexus 2248TP-E utilizes a 32MB shared buffer to handle larger traffic bursts
• Hadoop, NAS, AVID are examples of bursty applications
• You can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host)
• You can use a lower queue limit value on the Fabric Extender to prevent one blocked receiver from affecting traffic that is sent to other non-congested receivers ("head-of-line blocking”)
N5548-L3(config-fex)# hardware N2248TPE queue-limit 4000000 rx
N5548-L3(config-fex)# hardware N2248TPE queue-limit 4000000 tx
N5548-L3(config)#interface e110/1/1
N5548-L3(config-if)# hardware N2348TP queue-limit 4096000 tx
VM
#4
VM
#3
VM
#2
NAS
iSCSI
10G Attached Source (NAS Array)
1G Attached Server
10G
NF
S
Nexus 2248TP-E 32MB Shared Buffer
Tune 2248TP-E to support a extremely
large burst (Hadoop, AVID, …)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 54
Nexus 2248TP-E Counters N5596-L3-2(config-if)# sh queuing interface e110/1/1
Ethernet110/1/1 queuing information:
Input buffer allocation:
Qos-group: 0
frh: 2
drop-type: drop
cos: 0 1 2 3 4 5 6
xon xoff buffer-size
---------+---------+-----------
0 0 65536
Queueing:
queue qos-group cos priority bandwidth mtu
--------+------------+--------------+---------+---------+----
2 0 0 1 2 3 4 5 6 WRR 100 9728
Queue limit: 2097152 bytes
Queue Statistics:
---+----------------+-----------+------------+----------+------------+-----
Que|Received / |Tail Drop |No Buffer |MAC Error |Multicast |Queue
No |Transmitted | | | |Tail Drop |Depth
---+----------------+-----------+------------+----------+------------+-----
2rx| 5863073| 0| 0| 0| - | 0
2tx| 426378558047| 28490502| 0| 0| 0| 0
---+----------------+-----------+------------+----------+------------+-----
<snib>
Ingress queue
limit(Configurable)
Egress queue limit(Configurable)
Egress queues:
CoS to queue mapping
Bandwidth allocation
MTU
Per port per
queue
counters
Drop due to oversubscription
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 55
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
Nexus and NX-OS
New QoS Capabilities and Requirements
Understanding Nexus QoS Capabilities
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Nexus 1000v
Applications of QoS with Nexus
Voice and Video
Storage & FCoE
Hadoop and Web 2.0
Future QoS Design Considerations (Data Center TCP, ECN, optimized TCP)
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 56
Nexus 3000 QoS Overview
QoS is enabled by default on the Nexus 3000 (NX-OS Default)
All ports are “trusted” (CoS/DSCP/ToS values are preserved) by default
Default interface queuing policy uses QoS-Group 0 (Best Effort - Drop class),
WRR (tail drop), 100% throughput (bandwidth percent)
Unicast and Multicast traffic defaults to a 50% WRR bandwidth ratio of the egress
interface traffic data rate. (system wide Configuration)
The default interface MTU is 1500 bytes (system wide configuration)
Control plane traffic destined to the CPU is prioritized by default to improve
network stability.
QoS Policy Types (CLI):
Type (CLI) Description Applied To…
QoS Packet Classification based on Layer 2/3/4 (Ingress) Interface or System
Network-QoS Packet Marking (CoS), Congestion Control WRED/ECN (Egress) System
Queuing Scheduling - Queuing Bandwidth % / Priority Queue (Egress) Interface or System
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 57
Multi-Level Scheduling
Per-port Per-group
Deficit Round Robin
Buffer/Queuing Block
UC Queue 0 UC Queue 1
UC Queue 2
UC Queue 3 UC Queue 4
UC Queue 5
UC Queue 6
UC Queue 7
MC Queue 0
MC Queue 1
MC Queue 2
MC Queue 3
UC Queue 0 UC Queue 1
UC Queue 2
UC Queue 3 UC Queue 4
UC Queue 5
UC Queue 6
UC Queue 7
MC Queue 0
MC Queue 1
MC Queue 2
MC Queue 3
UC Queue 0 UC Queue 1
UC Queue 2
UC Queue 3 UC Queue 4
UC Queue 5
UC Queue 6
UC Queue 7
MC Queue 0
MC Queue 1
MC Queue 2
MC Queue 3
….
Egress
port 2
….
Egress
port 1
Egress
port 64
9MB
Total
80%
Shared
20% Per Port
Reserved
Nexus 3000 QoS Shared Memory Architecture
A pool of 9MB Buffer space
is divided up among Egress
reserved and Dynamically
shared buffer
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 58
Nexus 3000 QoS Configuration WRR Example
The next six slides contain configuration and verification examples for creating an ingress classification policy and an egress queuing policy to prioritize egress traffic if congestion occurs on the egress interface. The ingress classification policy trusts the IP DSCP values assigned by hosts and maps them into QoS-Groups. The egress queuing policy assigns a predefined bandwidth percentage to traffic class.
Traffic Class QoS-Group Throughput Percentage
Gold 1 40
Silver 2 30
Bronze 3 20
Best Effort (Default) 0 10
Ingress Egress
Traffic is prioritized based on traffic bandwidth percentage
Excess traffic is dropped based on bandwidth ratios
10% 20% 30% 40%
Example Traffic Class Definitions:
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 59
Traffic Classification Configuration
N3K-1(config)# class-map type qos match-all qos-group-1
N3K-1(config-cmap-qos)# description Gold
N3K-1(config-cmap-qos)# match dscp 46
N3K-1(config-cmap-qos)# class-map type qos match-all qos-group-2
N3K-1(config-cmap-qos)# description Silver
N3K-1(config-cmap-qos)# match dscp 36
N3K-1(config-cmap-qos)# class-map type qos match-all qos-group-3
N3K-1(config-cmap-qos)# description Bronze
N3K-1(config-cmap-qos)# match dscp 26
N3K-1(config)# policy-map type qos traffic-classification
N3K-1(config-pmap-qos)# class qos-group-1
N3K-1(config-pmap-c-qos)# set qos-group 1
N3K-1(config-pmap-c-qos)# class qos-group-2
N3K-1(config-pmap-c-qos)# set qos-group 2
N3K-1(config-pmap-c-qos)# class qos-group-3
N3K-1(config-pmap-c-qos)# set qos-group 3
N3K-1(config)# interface ethernet 1/30
N3K-1(config-if)# service-policy type qos input traffic-classification
Ingress traffic is classified based on IP DSCP values and associated to different QoS-Groups. In this example, the hosts are trusted and set the IP DSCP values. However, if the hosts were not trusted, a classification policy could be configured to set/rewrite the DSCP values.
Apply the Policy-Map to the interface or system
Define the Class-Maps and match the DSCP values
Define the Policy-Map and set the QoS-Groups
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 60
Queuing (WRR) Configuration
Egress traffic is matched on QoS-Group and guaranteed a percentage of bandwidth when traffic exceeds the egress Ethernet interface throughput. It is important to note that the class-default has to be modified to prevent the bandwidth percentage from being greater than 100%. In the example below the class-default has been reduced to 10% from 100%.
Apply the Policy-Map to the interface or system
Define the Class-Maps and match the QoS-Group values
Define the Policy-Map and set the bandwidth percentages
N3K-1(config)# class-map type queuing qos-group-1
N3K-1(config-cmap-que)# description Gold
N3K-1(config-cmap-que)# match qos-group 1
N3K-1(config-cmap-que)# class-map type queuing qos-group-2
N3K-1(config-cmap-que)# description Silver
N3K-1(config-cmap-que)# match qos-group 2
N3K-1(config-cmap-que)# class-map type queuing qos-group-3
N3K-1(config-cmap-que)# description Bronze
N3K-1(config-cmap-que)# match qos-group 3
N3K-1(config)# policy-map type queuing traffic-priorities
N3K-1(config-pmap-que)# class type queuing qos-group-1
N3K-1(config-pmap-c-que)# bandwidth percent 40
N3K-1(config-pmap-c-que)# class type queuing qos-group-2
N3K-1(config-pmap-c-que)# bandwidth percent 30
N3K-1(config-pmap-c-que)# class type queuing qos-group-3
N3K-1(config-pmap-c-que)# bandwidth percent 20
N3K-1(config-pmap-c-que)# class type queuing class-default
N3K-1(config-pmap-c-que)# bandwidth percent 10
N3K-1(config)# interface ethernet 1/10
N3K-1(config-if)# service-policy type queuing output traffic-priorities
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 61
Queuing - Network-QoS Configuration
The network-qos policy instantiates the QoS-Groups when applied to the system policy. This enables the QoS-Groups and interface statistics collection per QoS-Group.
Apply the Policy-Map to the system
Define the Class-Maps and match the QoS-Group values
Define the Policy-Map and match the Class-Maps previously defined
N3K-1(config)# class-map type network-qos qos-group-1
N3K-1(config-cmap-nq)# match qos-group 1
N3K-1(config-cmap-nq)# class-map type network-qos qos-group-2
N3K-1(config-cmap-nq)# match qos-group 2
N3K-1(config-cmap-nq)# class-map type network-qos qos-group-3
N3K-1(config-cmap-nq)# match qos-group 3
N3K-1(config)# policy-map type network-qos qos-groups
N3K-1(config-pmap-nq)# class type network-qos qos-group-1
N3K-1(config-pmap-nq)# class type network-qos qos-group-2
N3K-1(config-pmap-nq)# class type network-qos qos-group-3
N3K-1(config)# system qos
N3K-1(config-sys-qos)# service-policy type network-qos qos-groups
Notes: QoS-Group 0 is already included in the default QoS policy.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 62
Queuing – Priority Queue
N3K-1(config)# class-map type queuing dscp-priority
N3K-1(config-cmap-que)# match qos-group 1
N3K-1(config)# policy-map type queuing dscp-priority
N3K-1(config-pmap-que)# class type queuing dscp-priority
N3K-1(config-pmap-c-que)# priority
N3K-1(config)# interface ethernet 1/10
N3K-1(config-if)# service-policy type queuing output dscp-priority
Egress Queue Configuration: The queuing service-policy can be applied per interface or per system
N3K-1(config)# class-map type qos match-all dscp-priority
N3K-1(config-cmap-qos)# match dscp 46
N3K-1(config)# policy-map type qos dscp-priority
N3K-1(config-pmap-qos)# class dscp-priority
N3K-1(config-pmap-c-qos)# set qos-group 1
N3K-1(config)# interface ethernet 1/30
N3K-1(config-if)# service-policy type qos output dscp-priority
Ingress Classification Configuration: The qos service-policy can be applied per interface or per system
This configuration example puts all packets with an IP DSCP value of 46 (EF) into a priority queue that is scheduled before any other queues (i.e. QoS-Group 0). The ingress interface classifies packets on the Ingress interface matching DSCP 46 and puts it in QoS-Group 1. The Egress Queue matches QoS-Group 1 and configures that QoS-Group as a priority queue. All other traffic is is placed the QoS-Group 0 (Best Effort/Drop queue).
N3K-1(config)# class-map type network-qos qos-group-1
N3K-1(config-cmap-nq)# match qos-group 1
N3K-1(config)# policy-map type network-qos qos-groups
N3K-1(config-pmap-nq)# class type network-qos qos-group-1
N3K-1(config)# system qos
N3K-1(config-sys-qos)# service-policy type network-qos qos-groups
Network-QoS Configuration: The network-qos service-policy is applied per system
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 63
Queuing – WRED
The default WRR Queue behavior is to tail drop packets when congestion is experienced. A network-qos policy can be configured to enable WRED, which drops packets prior to experiencing congestion (based on min/max/probability ratios). This is beneficial for applications that use TCP, since the source can reduce its transmission rate when the TCP stream experiences lost packets.
N3K-1(config)# class-map type qos match-all class-gold
N3K-1(config-cmap-qos)# match dscp 8
N3K-1(config)# policy-map type qos traffic-classification
N3K-1(config-pmap-qos)# class class-gold
N3K-1(config-pmap-c-qos)# set qos-group 1
N3K-1(config)# interface ethernet 1/20
N3K-1(config-if)# service-policy type qos input traffic-classification
N3K-1(config)# class-map type network-qos class-gold
N3K-1(config-cmap-nq)# description Gold
N3K-1(config-cmap-nq)# match qos-group 1
N3K-1(config)# policy-map type network-qos traffic-priorities
N3K-1(config-pmap-nq)# class type network-qos class-gold
N3K-1(config-pmap-nq-c)# congestion-control random-detect
N3K-1(config)# system qos
N3K-1(config-sys-qos)# service-policy type network-qos traffic-priorities
Traffic Classification: Match packets with a IP DSCP 8 and transmit them in QoS-Group 1
Network-QoS: Match packets in QoS-Group 1 and enable WRED for the QoS-Group
Notes: Bandwidth percentages were not configured in this example to keep it simple.
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 64
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
Nexus and NX-OS
New QoS Capabilities and Requirements
Understanding Nexus QoS Capabilities
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Nexus 1000v
Applications of QoS with Nexus
Voice and Video
Storage & FCoE
Hadoop and Web 2.0
Future QoS Design Considerations (Data Center TCP, ECN, optimized TCP)
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 65
Converting Catalyst 6500 to Nexus 7000 What’s Different?
Biggest change is introduction of “queuing policies” to apply port-based QoS configuration
Catalyst 6500 uses platform-specific syntax for port QoS
“mls”, “rcv-queue”, “wrr-queue”, etc. commands
Nexus 7000 uses “modular QoS CLI” (MQC) to apply both queuing and traditional QoS (marking/policing) policies
Class maps to match traffic
Policy maps to define actions to take on each class
Service policies to tie policy maps to interfaces/VLANs in a particular direction
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 66
Typical Catalyst 6500 Egress Port QoS Configuration
mls qos
!
interface range gig3/1-48
mls qos trust dscp
wrr-queue bandwidth 100 150 200 !For 1p3q8t
wrr-queue bandwidth 100 150 200 0 0 0 0 !For 1p7q8t
wrr-queue cos-map 1 1 1
wrr-queue cos-map 1 2 0
wrr-queue cos-map 2 8 4
wrr-queue cos-map 2 2 2
wrr-queue cos-map 3 4 3
wrr-queue cos-map 3 8 6 7
priority-queue cos-map 1 5
Enable QoS. Not
needed – QoS is always
enabled on Nexus 7000 Define trust
behavior. Not
needed – DSCP is
preserved (trusted)
by default
Define DWRR weights.
Nexus 7000 uses
bandwidth statements in
queuing policy-maps.
Define COS-to-queue
mapping, and COS-to-
threshold mapping.
Nexus 7000 uses
match statements and
queue-limit commands.
Example: 2 8 4 =
map COS 4 to
queue 2, threshold 8
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 67
Equivalent Nexus 7000 Egress Queuing Policy class-map type queuing match-any 1p7q4t-out-pq1
match cos 5
class-map type queuing match-any 1p7q4t-out-q2
match cos 3,6-7
class-map type queuing match-any 1p7q4t-out-q3
match cos 2,4
class-map type queuing match-any 1p7q4t-out-q-default
match cos 0-1
!
policy-map type queuing 10G-qing-out
class type queuing 1p7q4t-out-pq1
priority level 1
queue-limit percent 15
class type queuing 1p7q4t-out-q2
queue-limit percent 25
queue-limit cos 6 percent 100
queue-limit cos 7 percent 100
queue-limit cos 3 percent 70
bandwidth remaining percent 22
class type queuing 1p7q4t-out-q3
queue-limit percent 25
queue-limit cos 4 percent 100
queue-limit cos 2 percent 50
bandwidth remaining percent 33
class type queuing 1p7q4t-out-q-default
queue-limit percent 35
queue-limit cos 1 percent 50
queue-limit cos 0 percent 100
bandwidth remaining percent 45
!
int e1/1
service-policy type queuing output 10G-qing-out
Define behavior for
each queue in
queuing policy-map
Define COS-to-queue
mapping in queuing class-
maps (configurable for
each port type in each
direction)
Define priority queue
Size the queue
Define COS-to-
threshold mapping
Define DWRR weight for queue
(“bandwidth remaining” required when using PQ)
Tie policy-map as service-policy
on appropriate interface type in
appropriate direction
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 68
ESE QoS SRND for Catalyst 6500 interface range TenGigabitEthernet4/1 - 4
wrr-queue queue-limit 5 25 10 10 10 5 5
wrr-queue bandwidth 5 25 20 20 20 5 5
wrr-queue random-detect 1
wrr-queue random-detect 2
wrr-queue random-detect 3
wrr-queue random-detect 4
wrr-queue random-detect 5
wrr-queue random-detect 6
wrr-queue random-detect 7
wrr-queue random-detect min-threshold 1 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 2 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 3 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 3 100 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 4 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 4 100 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 5 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 5 100 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 6 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 6 100 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 7 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 7 100 100 100 100 100 100 100 100
wrr-queue cos-map 1 1 1
wrr-queue cos-map 2 1 0
wrr-queue cos-map 3 1 4
wrr-queue cos-map 4 1 2
wrr-queue cos-map 5 1 3
wrr-queue cos-map 6 1 6
wrr-queue cos-map 7 1 7
priority-queue cos-map 1 5
SRND queuing
configuration for
Catalyst 6500 1p7q8t
port type
Allocates buffer space
to non-PQs
Sets the DWRR weights
for non-PQs
Enables WRED on
non-PQs
Sets WRED min thresholds
for the non-PQs
Sets WRED max thresholds
for the non-PQs
Assigns
scavenger/
bulk to Q1
WRED
threshold 1
Q2:
Best
effort Q3:
Video
Q4: NMS/
transactional data Q5: Call sig and
critical data Q6: RPs Q7: STP PQ: VoIP
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 69
Mapping QoS SRND to Nexus 7000 (1)
class-map type queuing match-any 1p7q4t-out-pq1
match cos 5
class-map type queuing match-any 1p7q4t-out-q2
match cos 7
class-map type queuing match-any 1p7q4t-out-q3
match cos 6
class-map type queuing match-any 1p7q4t-out-q4
match cos 4
class-map type queuing match-any 1p7q4t-out-q5
match cos 3
class-map type queuing match-any 1p7q4t-out-q6
match cos 2
class-map type queuing match-any 1p7q4t-out-q7
match cos 0
class-map type queuing match-any 1p7q4t-out-q-default
match cos 1
PQ: VoIP
Q-Default:
Scavenger/ bulk
Q7: Best effort
Q6: Video
Q5: NMS/
transactional data
Q4: Call sig and
critical data
Q3: RPs
Q2: STP
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 70
Mapping QoS SRND to Nexus 7000 (2) policy-map type queuing 10G-SRND-out
class type queuing 1p7q4t-out-pq1
priority level 1
queue-limit percent 10
class type queuing 1p7q4t-out-q2
queue-limit percent 10
bandwidth remaining percent 5
random-detect cos-based
random-detect cos 7 minimum-threshold percent 80 maximum-threshold percent 100
class type queuing 1p7q4t-out-q3
queue-limit percent 10
bandwidth remaining percent 5
random-detect cos-based
random-detect cos 6 minimum-threshold percent 80 maximum-threshold percent 100
class type queuing 1p7q4t-out-q4
queue-limit percent 15
bandwidth remaining percent 20
random-detect cos-based
random-detect cos 4 minimum-threshold percent 80 maximum-threshold percent 100
class type queuing 1p7q4t-out-q5
queue-limit percent 10
bandwidth remaining percent 20
random-detect cos-based
random-detect cos 3 minimum-threshold percent 80 maximum-threshold percent 100
Defines the PQ
Sizes the PQ
I actually question enabling WRED on network control
queues as described in SRND… Your choice…
Sets the DWRR
weight for the queue
Sets WRED min and max
thresholds for the queue
Enables COS-based WRED
for the queue
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 71
Mapping QoS SRND to Nexus 7000 (3)
class type queuing 1p7q4t-out-q6
queue-limit percent 10
bandwidth remaining percent 20
random-detect cos-based
random-detect cos 2 minimum-threshold percent 80 maximum-threshold percent 100
class type queuing 1p7q4t-out-q7
queue-limit percent 30
bandwidth remaining percent 25
random-detect cos-based
random-detect cos 0 minimum-threshold percent 80 maximum-threshold percent 100
class type queuing 1p7q4t-out-q-default
queue-limit percent 5
bandwidth remaining percent 5
random-detect cos-based
random-detect cos 1 minimum-threshold percent 80 maximum-threshold percent 100
!
int e1/1
service-policy type queuing output 10G-SRND-out
I chose slightly different queue-limit sizes vs SRND –
when all 8 queues enabled, sum of queue-limit
percentages must equal 100
Tie the policy-map to the interface as an
output queuing service-policy
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 72
Summary
MQC configuration for both queuing and marking/policing polices
Departure from platform-specific Catalyst 6500 configuration model
Initially, queuing policy configuration model generates some confusion
But, it’s modular and self-documenting
99% of needed QoS features exist in NX-OS
DSCP-to-queue perhaps biggest gap
A few key default changes:
QoS always enabled
Default port behavior is « trust »
Port QoS config conversion from Catalyst 6500 IS possible ;)
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 73
Implementing QoS with Nexus and NX-OS Agenda
Nexus and QoS
Nexus and NX-OS
New QoS Capabilities and Requirements
Understanding Nexus QoS Capabilities
Nexus 7000
Nexus 5500
Nexus 2000
Nexus 3000
Nexus 1000v
Applications of QoS with Nexus
Voice and Video
Storage & FCoE
Hadoop and Web 2.0
Future QoS Design Considerations (Data Center TCP, ECN, optimized TCP)
1K Cisco Nexus
x86
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 74
On Nexus 5000 once feature fcoe is configured, 2 classes
are made by default
Priority Flow Control – Nexus 5000/5500 Operations Configuration – Switch Level
FCoE DCB Switch
DCB CNA Adapter
class-fcoe is configured to be no-drop with an MTU of 2158
policy-map type qos default-in-policy
class type qos class-fcoe
set qos-group 1
class type qos class-default
set qos-group 0
policy-map type network-qos default-nq-policy
class type network-qos class-fcoe
pause no-drop
mtu 2158
system qos
service-policy type qos input fcoe-default-in-policy
service-policy type queuing input fcoe-default-in-policy
service-policy type queuing output fcoe-default-out-policy
service-policy type network-qos fcoe-default-nq-policy
Enabling the FCoE feature on Nexus 5548/96 does ‘not’ create no-drop policies automatically as on Nexus 5010/20
Must add policies under system QOS:
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 75
Configs for
3000m no-drop
class
Buffer size Pause Threshold
(XOFF)
Resume
Threshold (XON)
N5020 143680 bytes 58860 bytes 38400 bytes
N5548 152000 bytes 103360 bytes 83520 bytes
Tuning of the lossless queues to support a variety of use cases
Extended switch to switch no drop traffic lanes
Support for 3km with Nexus 5000 and 5500
Increased number of no drop services lanes (4) for RDMA and other multi-queue HPC and compute applications
Support for 3 km no drop switch to
switch links Inter Building DCB
FCoE links
Nexus 5000/5500 QoS Priority Flow Control and No-Drop Queues
5548-FCoE(config)# policy-map type network-qos 3km-FCoE
5548-FCoE(config-pmap-nq)# class type network-qos 3km-FCoE
5548-FCoE(config-pmap-nq-c)# pause no-drop buffer-size 152000 pause-threshold 103360
resume-threshold 83520
Gen 2 UPC
Unified Crossbar Fabric
Gen 2 UPC
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 76
show policy-map system
Type network-qos policy-maps
=====================================
policy-map type network-qos default-nq-7e-policy
class type network-qos c-nq-7e-drop
match cos 0-2,4-7
congestion-control tail-drop
mtu 1500
class type network-qos c-nq-7e-ndrop-fcoe
match cos 3
match protocol fcoe
pause
mtu 2112
Priority Flow Control – Nexus 7K & MDS
Operations Configuration – Switch Level
N7K-50(config)# system qos
N7K-50(config-sys-qos)# service-policy type network-qos default-nq-7e-policy
Policy Template choices
No-Drop PFC w/ MTU 2K set for Fibre Channel
show class-map type network-qos c-nq-7e-ndrop-fcoe
Type network-qos class-maps
=============================================
class-map type network-qos match-any c-nq-7e-ndrop-fcoe
Description: 7E No-Drop FCoE CoS map
match cos 3
match protocol fcoe
Template Drop CoS (Priority) NoDrop CoS (Priority)
default-nq-8e-policy 0,1,2,3,4,5,6,7 5,6,7 - -
default-nq-7e-policy 0,1,2,4,5,6,7 5,6,7 3 -
default-nq-6e-policy 0,1,2,5,6,7 5,6,7 3,4 4
default-nq-4e-policy 0,5,6,7 5,6,7 1,2,3,4 4
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 77
Enhanced Transmission Selection - N5K Bandwidth Management
When configuring FCoE by default,
each class is given 50% of the available
bandwidth
Can be changed through QoS
settings when higher demands for
certain traffic exist (i.e. HPC traffic,
more Ethernet NICs)
1Gig FC HBAs
1Gig Ethernet NICs
Traditional Server
Best Practice: Tune FCoE queue to provide equivalent capacity
to the HBA that would have been used (1G, 2G, …)
N5k-1# show queuing interface ethernet 1/18
Ethernet1/18 queuing information:
TX Queuing
qos-group sched-type oper-bandwidth
0 WRR 50
1 WRR 50
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 78
Create classification rules first by defining and applying policy-map type qos
Define and apply policy-map type queuing to configure strict priority and bandwidth sharing
pod3-5010-2(config)# class-map type queuing class-voice
pod3-5010-2(config-cmap-que)# match qos-group 2
pod3-5010-2(config-cmap-que)# class-map type queuing class-high
pod3-5010-2(config-cmap-que)# match qos-group 3
pod3-5010-2(config-cmap-que)# class-map type queuing class-low
pod3-5010-2(config-cmap-que)# match qos-group 4
pod3-5010-2(config-cmap-que)# exit
pod3-5010-2(config)# policy-map type queuing policy-BW
pod3-5010-2(config-pmap-que)# class type queuing class-voice
pod3-5010-2(config-pmap-c-que)# priority
pod3-5010-2(config-pmap-c-que)# class type queuing class-high
pod3-5010-2(config-pmap-c-que)# bandwidth percent 50
pod3-5010-2(config-pmap-c-que)# class type queuing class-low
pod3-5010-2(config-pmap-c-que)# bandwidth percent 20
pod3-5010-2(config-pmap-c-que)# class type queuing class-fcoe
pod3-5010-2(config-pmap-c-que)# bandwidth percent 30
pod3-5010-2(config-pmap-c-que)# class type queuing class-default
pod3-5010-2(config-pmap-c-que)# bandwidth percent 0
pod3-5010-2(config-pmap-c-que)# system qos
pod3-5010-2(config-sys-qos)# service-policy type queuing output policy-BW
pod3-5010-2(config-sys-qos)#
Enhanced Transmission Selection – N5K Changing ETS Bandwidth Configurations
FCoE Traffic given
30% of the 10GE link
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 79
Ingress Queuing for Ethernet4/17 [System]
------------------------------------------------------------
Trust: Trusted
-----------------
Group Qlimit%
-------------------
0 70
1 30
---------------------------------------------------
Que# Group Qlimit% IVL CoSMap
---------------------------------------------------
0 0 45 0 0-1
1 0 10 5 5-7
2 1 100 3 3
3 0 45 2 2,4
Egress Queuing for Ethernet4/17 [System]
---------------------------------------------------------
Template: 4Q7E
-----------------------------------------------------
Group Bandwidth% PrioLevel Shape%
-----------------------------------------------------
0 80 - -
1 20 - -
--------------------------------------------------------------------------
Que# Group Bandwidth% PrioLevel Shape% CoSMap
---------------------------------------------------------------------------
0 0 - High - 5-7
1 1 100 - - 3
2 0 50 - - 2,4
3 0 50 - - 0-1
n7k-50-fcoe-2# show queuing interface ethernet 4/17
ETS – Nexus 7000 Bandwidth Management
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 80
DC Design Details iSCSI Storage Considerations
VM
#4
VM
#3
VM
#2
NAS
iSCSI
• iSCSI and DCB
• Where does PFC make sense in the non FCoE design?
• Extending buffering from switch to connected device
End to End – Need to consider network oversubscription carefully!
Fibre Channel and FCoE leverage very low levels of oversubscription
No Drop for FC works due to capacity planning
• Where does ETS make sense?
• Anywhere you want to guarantee capacity
Flow Control from the array to the switch
Flow Control from the server to the switch
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 81
iSCSI
1G 1G 1G 1G
4G
10G
10G
1. Steady state traffic is
within end to end network
capacity
2. Burst traffic from a source
5. All sources are eventually
flow controlled
3. ‘No Drop’ traffic is queued
DC Design Details iSCSI Storage Considerations – TCP or PFC
4. Buffers begin to fill and
PFC flow control initiated
• TCP not invoked
immediately as frames
are queued not dropped
• Is the optimal behaviour
for your oversubscription?
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 82
Nexus 5500 Switch
DCB CNA Adapter
class-map type qos class-iscsi
match protocol iscsi
match cos 4
class-map type queuing class-iscsi
match qos-group 4
policy-map type qos iscsi-in-policy
class type qos class-fcoe
set qos-group 1
class type qos class-iscsi
set qos-group 4
• iSCSI TLV will be supported in the 5.2 release (CY12) – 3rd Party Adapters not validated until that release
• Functions in the same manner as the FCoE TLV
• Communicates to the compatible Adapter using DCBX (LLDP)
• Steps to configure
Configure Class Maps to identify iSCSI traffic
Configure Policy Maps to identify marking, queueing and system behaviour
Apply policy maps
Identify iSCSI traffic
Nexus 5500 and iSCSI - DCB PFC (802.1Qbb) & ETS 802.1Qaz
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 83
policy-map type queuing iscsi-in-policy
class type queuing class-iscsi
bandwidth percent 10
class type queuing class-fcoe
bandwidth percent 10
class type queuing class-default
bandwidth percent 80
policy-map type queuing iscsi-out-policy
class type queuing class-iscsi
bandwidth percent 10
class type queuing class-fcoe
bandwidth percent 10
class type queuing class-default
bandwidth percent 80
class-map type network-qos class-iscsi
match qos-group 4
policy-map type network-qos iscsi-nq-policy
class type network-qos class-iscsi
set cos 4
pause no-drop
mtu 9216
class type network-qos class-fcoe
system qos
service-policy type qos input iscsi-in-policy
service-policy type queuing input iscsi-in-policy
service-policy type queuing output iscsi-out-policy
service-policy type network-qos iscsi-nq-policy
Nexus 5500 Switch
DCB CNA Adapter
Define policies to be
signaled to CNA
Define switch queue
BW policies
Define iSCSI MTU
and ‘if’ single hop
topology no-drop
behaviour
Nexus 5500 and iSCSI - DCB PFC (802.1Qbb) & ETS 802.1Qaz
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 84
Conclusion
You should now have a good understanding of QoS implementation using the Nexus Data Center switches …
Any questions?
84
© 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public BRKRST-2930 86
Please complete your Session Survey
Don't forget to complete your online session evaluations after each session.
Complete 4 session evaluations & the Overall Conference Evaluation
(available from Thursday) to receive your Cisco Live T-shirt
Surveys can be found on the Attendee Website at www.ciscolivelondon.com/onsite
which can also be accessed through the screens at the Communication Stations
Or use the Cisco Live Mobile App to complete the
surveys from your phone, download the app at
www.ciscolivelondon.com/connect/mobile/app.html
We value your feedback
http://m.cisco.com/mat/cleu12/
1. Scan the QR code
(Go to http://tinyurl.com/qrmelist for QR code reader
software, alternatively type in the access URL above)
2. Download the app or access the mobile site
3. Log in to complete and submit the evaluations