introduction to storage

Technical whitepaper

Dual-Hop FCoE with HP Virtual Connect modules Cookbook

Table of contents

Introduction 3

Requirements and support 3 Virtual Connect 3 Upstream Switch 4 FlexFabric CNAs 4 Storage Array 4

Virtual Connect Limitations 5

Dual-Hop FCoE with Nexus 5xxx Series ToR switch in FCF mode scenario 8

Requirements 9 Guidelines 9 Nexus configuration when using a single VC to Nexus link 10 Nexus configuration when using a port channel between VC and Nexus 13

Virtual Connect configuration 19 Converged Shared Uplink Sets (SUS) Details and Restrictions 20 Defining two Converged Shared Uplink Sets 21

QoS Configuration 25 Quality of Service configuration on the Nexus 5xxx series 25 Quality of Service configuration on the Virtual Connect modules 28

Nexus Troubleshooting commands 33 Useful Show commands 33 Per-Priority Flow Control commands 37 Issues with FCoE performance 38 Interface Status commands 43 FCoE Frames Analysis 44

Support and Other Resources 47

2

Contacting HP 47 Related documentation 48

3

Introduction

Dual-Hop FCoE support is one of the main features of VC 4.01. This new feature allows the FCoE traffic to be propagated out of the enclosure to an external bridge which will handle the conversion of FCoE to FC traffic.

Dual-Hop FCoE solution has several advantages when compared to a traditional FC + Ethernet design. With FCoE, the number of cable is reduced, this results in a simplification of the in-rack cable installation and consequently eliminates expensive equipment and reduces the overall solution cost.

This feature is described as FCoE Dual Hop because there is two FCoE hops between the server and the storage the VC Module and the external FCoE ->FC bridge that connects the HP Virtual Connect modules to the storage. No additional external bridges are currently allowed in this configuration in order for Virtual Connect to guarantee the lossless of the FCoE connection.

Requirements and support

Virtual Connect Dual-Hop FCoE with HP Virtual Connect is currently only supported with the following modules:

HP Virtual Connect FlexFabric 10Gb/24-port Module

HP Virtual Connect Flex-10/10D Ethernet Module

Firmware requirement

Dual-Hop FCoE with HP Virtual Connect is supported with VC 4.01 or later.

FcoE-capable uplink ports

VC FlexFabric module can only support FCoE on uplink port X1 to X4

VC Flex-10/10D module can support FCoE on ALL uplink ports (X1-X10)

X1 X4Uplink ports available

for FCoE connection

X1 X10Uplink ports available

for FCoE connection

VC FlexFabric 10Gb/24-port VC Flex-10/10D

FcoE-enabled uplink port limitation

Long Range Ethernet Optical transceivers (LR SFP+) are not supported on Virtual Connect FCoE-enabled uplink port.

4

Upstream Switch Virtual Connect Dual-Hop FCoE is currently supported only with Cisco Nexus 5000, and Cisco Nexus 5500 product families (i.e. Nexus 5010, 5020, 5548UP and 5596UP).

Supported Nexus configuration details are available in the C-Series FCoE Switch Connectivity stream document available on the following HP Single Point of Connectivity Knowledge (SPOCK) web page: http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_switches.html

(Requires HP Passport; if you do not have an HP Passport account, follow the instructions on the webpage).

This SPOCK document provides detailed information on Nexus switches, recommended NX-OS versions, supported transceivers, etc.

Nexus support

This cookbook does not intent to provide any detailed list of Cisco supported configurations, for information on Nexus FCoE supported configuration and design, refer to the Cisco web site.

Nexus 5000 Series Design Guides http://www.cisco.com/en/US/products/ps9670/products_implementation_design_guides_list.html

Data Center Design with Cisco Nexus Switches and Virtual PortChannel http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572831-00_Dsgn_Nexus_vPC_DG.pdf

FlexFabric CNAs SPP 2013.02 is recommended as a Baseline Release Set with the addition of Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01 compatibility (includes Emulex CNA FW 4.2.401.2215 and new drivers for RHEL 5.8/5.9 - RHEL 6.2/6.3 - SLES 10 SP4 - SLES 11 SP2).

More details about FlexFabric CNA Firmware and FCoE Drivers are available in the Virtual Connect FCoE Modules section of the following SPOCK webpage: http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_virtual_connect.html

(Requires HP Passport; if you do not have an HP Passport account, follow the instructions on the webpage).

These Virtual Connect Connectivity Stream documents provide a detailed table of the FlexFabric CNA firmware and FCoE drivers that are supported by HP.

Note: For more information about HP Service Pack for ProLiant and Hotfixes, see http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

Storage Array SPOCK is the HP authoritative source of interoperability information for HP storage products. A particular configuration is supported if and only if, it is listed on SPOCK.

HP Storage Arrays supported with Virtual Connect Dual-Hop configuration are listed in the Virtual Connect Connectivity Stream documents available on the following SPOCK webpage: http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_virtual_connect.html

For information about Non-HP Storage support, firmware versions and specific supported configuration information, consult your storage equipment vendor.

5

Virtual Connect Limitations

Only Dual-Hop FCoE is supported by HP at this time. Figure 1: Dual-Hop FCoE Split FC & Ethernet at Top of Rack FCoE switch

C7000 with Virtual Connect

SANLAN

C7000 with Virtual Connect

FCoE switchFCoE switch

FCoEFCoE

Only one FCoE network can be associated with any single set of uplink ports. FCoE-capable Shared Uplink Set (SUS) can be used to allow concurrent Ethernet and FCoE traffic, but only one of the networks in the SUS can be an FCoE Network. Figure 2: Virtual Connect SUS configuration when using FCoE

SUS-1

FCoE-1

VLAN-30

VLAN-20

VLAN-10

LACP

FCoE Lossless Network

VC Ethernet networks

Converged-SUS defined to pass FCoE traffic and traditional

Ethernet traffic

6

Virtual Connect requires the creation of an Active/Active Shared Uplink Set configuration for the FCoE traffic. Figure 3: Active-Active FCoE Virtual Connect configuration

FCoE ToRSwitch 1

FCoE ToRSwitch 2

FCoE FCoE

VC Domain

FCoE ToRSwitch 1

FCoE ToRSwitch 2

FCoE FCoE

SUS-1

VC Domain

SUS-2

SUS

Lossless Ethernet Network

Lossless Ethernet Network

Active Active

FCoE-capable Shared Uplink Sets must always contain ports from a single VC module in order to maintain the SAN-A/B isolation. Figure 4: FCoE-capable Shared Uplink Set configuration support

FCoE ToRSwitch 1

FCoE ToRSwitch 2

FCoE FCoE

VC Domain

FCoE ToRSwitch 1

FCoE ToRSwitch 2

FCoE FCoE

SUS-1

VC Domain

SUS-2

SUS-1 SUS-2

7

FCoE-capable Shared Uplink Sets must always be connected to a single Top of Rack switch in order to maintain the SAN-A/B isolation. Figure 5: FCoE-capable Shared Uplink Set LACP configuration support

FCoE ToRSwitch 1

FCoE ToRSwitch 2

FCoE FCoE

VC Domain

FCoE ToRSwitch 1

FCoE ToRSwitch 2

FCoE FCoE

SUS-1

VC Domain

SUS-2

SUS-1 SUS-2

Active Active

Lossless

Ethernet

Network

Active Active

Lossless Ethernet

Network

LACPLACP

LACPLACP

Ethernet forwarding through a link aggregation can break

SAN A/B separation

8

Dual-Hop FCoE with Nexus 5xxx Series ToR switch in FCF mode scenario

In this scenario, the Nexus switches operate as Fiber Channel Forwarders (FCF). This is the default Cisco Nexus 5xxx Series switches mode, its also called the fabric mode. In this mode, the switch provides standard Fibre Channel switching capability and features.

Figure 6: Cisco Nexus 5xxx switches operating as FC Forwarder (FCF)

E_Port N_PortVF_PortVN_Port

Virtual Connect(FIP Snooping mode)

FCF

FCoE

FC

Native FC

Lossless Ethernet network

FC

Storage

Array

F_Port

N_Port

Server(ENode)

Cisco

MDS

FC

Storage

ArrayE_Port

F_Port

Native FC

Dual-Hop FCoE

Nexus 5xxx

Series(Fabric Mode)

This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of Nexus FCoE configuration guidelines, refer to the Cisco NX-OS Fibre Channel over Ethernet Configuration Guides:

Cisco Nexus 5000 Series NX-OS Fibre Channel over Ethernet Configuration Guide, Release 5.2(1)N1(1) http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/fcoe/521n11/b_5k_FCoE_Config_521N11.pdf

Cisco Nexus 5500 Series NX-OS Fibre Channel over Ethernet Configuration Guide, Release 6.0(2)N1(1) http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/fcoe/602_N1_1/b_5500_FCoE_Config_602N11.pdf

9

Figure 7: Physical view

FCoEFCoE

FCFC

FCoEHP Virtual Connect FlexFabric

or Flex-10/10D modules

FC-FCoE

FC Storage Array

Nexus 5xxx Series

ToR Switch using

Fabric Mode

VC Domain

LAN

FC SAN BFC SAN A

Cisco MDS Series

SAN Switch

Requirements

Minimum NX-OS version 5.2(1)N1(3)

FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG)

Note: The license is an option which is activated when the Nexus switch is shipped from Cisco.

Guidelines

Nexus switches must either bridge to native FC infrastructure or directly connect to FC/FCoE-based Storage devices. Refer to the Cisco Nexus or Storage vendor interoperability Matrix.

When bridged to a native FC infrastructure, it is mandatory to use Cisco MDS directors or fabric switches in order to provide interoperability between fabrics.

To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is recommended to use different FCoE VLANs and VSANs numbers between the two fabrics.

The FCoE VLAN should be dedicated to FCoE traffic (i.e. it should not carry IP traffic).

The FCoE VLAN must not be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by default VLAN 1).

Interfaces connecting to VC must be configured as trunk ports and STP edge ports. (STP does not run on FCoE VLANs between FCFs (VE_Ports) but does run on FCoE VLANs towards the host (VF_Ports)).

10

Nexus configuration when using a single VC to Nexus link Details about the configuration:

Interfaces eth1/5 are connected to the VC modules.

Interfaces fc2/1 are directly connected to Cisco MDS 9148 switches.

VLAN IDs 200 and 201 are used for the FCoE networks.

VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-FCoE networks).

The vfc interface (virtual Fibre Channel interface) binds to eth1/5.

Figure 8: Physical diagram

HP BladeSystem c7000

fc2/1

LAN

fc2/1eth1/5 eth1/5

FCoE/DCBFC

LAN

X3 X3

eth1/17eth1/17

FCoE VLAN 200 FCoE VLAN 201

VLAN 1,10,20 VLAN 1,10,20

VSAN 201VSAN 200

VLAN 1,10,20 VLAN 1,10,20

vfc 2005 vfc 2005

FCF Switch ANexus-5010

VC FlexFabricBay 1

VC FlexFabricBay 2

Blade Server

FCoE NIC

CNA Port 1

CNAPort 2

FCoE NIC

WWN:

50:06:0b:00:00:c3:1a:22

WWN:

50:06:0b:00:00:c3:1a:20

3PAR Storage Array

FCF Switch BNexus-5010

FCoE / DCB

NativeEthernet

NativeFibre Channel

WWN:

20:53:00:02:ac:00:15:9d WWN:

21:53:00:02:ac:00:15:9d

SAN-A SAN-B Cisco MDS9148

11

Nexus switch-A configuration

Upgrade the first Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)

Enable FCoE on the switch (disabled by default):

o configure terminal

o feature fcoe

Note: The Nexus will require a reload.

Associate a VLAN with a VSAN for FCoE traffic:

o vlan 200

o fcoe vsan 200

Create a virtual Fibre Channel interface to carry the FCoE traffic through eth1/5:

o interface vfc 2005

o bind interface ethernet 1/5

o no shutdown

Create the VLANs for the IP traffic:

o vlan 1,10,20

Create a trunk on the interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):

o interface Ethernet1/5

o description FCoE uplink to FlexFabric

o switchport mode trunk

o switchport trunk allowed vlan 1,10,20,200

o spanning-tree port type edge trunk

Assign the vfc interface and the Fibre Channel port to the FCoE VSAN:

o vsan database

o vsan 200

o vsan 200 interface fc2/1

o vsan 200 interface vfc 2005

Configure the interface connected to the datacenter LAN:

o interface eth 1/17


o switchport trunk allowed vlan 1,10,20

Configuration of the zone:

a. Create zones: o zone name fcoe-zone vsan 200

o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port}

o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the Blade FlexFabric Adapter port 1}

b. Create zoneset: o zoneset name zoneset1 vsan 200

o member fcoe-zone

c. Activate zoneset: o zoneset activate name zoneset1 vsan 200

12

Nexus switch-B configuration

Upgrade the second Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)

Enable FCoE on the switch (disabled by default):


o feature fcoe



o vlan 201

o fcoe vsan 201

Create a virtual Fibre Channel interface to carry the FCoE traffic through eth1/5:


o bind interface ethernet 1/5

o no shutdown


o vlan 1,10,20

Create a trunk on the interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):

o interface Ethernet1/5

o description FCoE uplink to FlexFabric





o vsan database

o vsan 201








a. Create zones: o zone name fcoe-zone vsan 201

o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port}


b. Create zoneset: o zoneset name zoneset1 vsan 201

o member fcoe-zone

c. Activate zoneset: o zoneset activate name zoneset1 vsan 201

13

Nexus configuration when using a port channel between VC and Nexus

Current NX-OS limitations

All ports in the FCoE port channel must belong the same Unified Port Controller (UPC) ASIC; you cannot configure port channels across UPCs.

Nexus 5010/5020 switches can support a maximum of four ports per port-channel when carrying FCoE traffic.

Nexus 5548/5596 switches can support a maximum of eight ports per port-channel when carrying FCoE traffic.

Note: These limitations will be removed in a future NX software release.

Details about the configuration:

Interfaces eth1/5 and eth1/6 are connected to the VC modules.

Interfaces fc2/1 are directly connected to Cisco MDS 9148 switches.

VLAN IDs 200 and 201 are used for the FCoE networks.

VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-FCoE networks).

The vfc interface (virtual Fibre Channel interface) binds to the Port Channel 200 configured with eth1/5 and eth1/6.

Figure 9: Physical diagram

HP BladeSystem c7000

3PAR Storage Array

fc2/1

LAN

fc2/1eth1/5 eth1/6 eth1/5 eth1/6

FCoE/DCBFC

LAN

X3 - X4 X3 X4

eth1/17eth1/17

VLAN 1,10,20 VLAN 1,10,20VSAN 201VSAN 200

FCoE VLAN 200VLAN 1,10,20

FCoE VLAN 201VLAN 1,10,20

vfc 2005 vfc 2005

FCF Switch ANexus-5010

FCF Switch BNexus-5010

VC FlexFabricBay 1

VC FlexFabricBay 2

Blade Server

FCoE NIC

CNA Port 1

CNA Port 2

FCoE NIC

FCoE / DCB

NativeEthernet

NativeFibre Channel

WWN:

50:06:0b:00:00:c3:1a:22

WWN:

20:53:00:02:ac:00:15:9d WWN:

21:53:00:02:ac:00:15:9d

WWN:

50:06:0b:00:00:c3:1a:20

po200po200

SAN-A SAN-B Cisco MDS9148

14

Identifying ports that belong to the same Unified Port Controller

It is necessary while configuring the FCoE port channel group to use interfaces that belong to the same UPC ASIC (e.g. Eth1/5 and Eth1/6 because both interfaces are mapped to the same ASIC, UPC#1). In order to identify ports that belong to same UPC, it is necessary to use the show hardware internal command.

On Nexus 5010/5020 switches: The Nexus 5010 and 5020 switches use Gatos Unified Port Controller ASIC (UPC Gen 1). Each UPC ASIC have 4 ports mapped to front panel ports. This can be seen with the following command: Nexus-5010# show hardware internal gatos all-ports

Gatos Port Info:

name |log|gat|mac|flag|adm|opr|c:m:s:l|ipt|fab|xgat|xpt|if_index|diag

-------+---+---+---+----+---+---+-------+---+---+----+---+--------+----

xgb1/4 |3 |0 |0 |b7 |dis|dn |0:0:0:f|0 |55 |0 |2 |1a003000|pass



xgb1/2 |1 |0 |3 |b7 |en |up |1:3:3:f|3 |57 |0 |6 |1a001000|pass

xgb1/8 |7 |1 |0 |b7 |en |up |0:0:0:f|0 |50 |1 |2 |1a007000|pass


xgb1/5 |4 |1 |2 |b7 |en |up |1:2:2:f|2 |53 |1 |4 |1a004000|pass

xgb1/6 |5 |1 |3 |b7 |en |up |1:3:3:f|3 |52 |1 |6 |1a005000|pass

xgb1/19|18 |2 |0 |b7 |en |up |0:0:0:f|0 |45 |4 |0 |1a012000|pass

xgb1/11|10 |2 |1 |b7 |dis|dn |0:1:1:f|1 |44 |2 |0 |1a00a000|pass

xgb1/20|19 |2 |2 |b7 |en |up |1:2:2:f|2 |49 |4 |2 |1a013000|pass

xgb1/12|11 |2 |3 |b7 |dis|dn |1:3:3:f|3 |48 |2 |2 |1a00b000|pass

xgb1/18|17 |3 |0 |b7 |dis|dn |0:0:0:f|0 |41 |4 |6 |1a011000|pass

xgb1/10|9 |3 |1 |b7 |en |dn |0:1:1:f|1 |40 |2 |6 |1a009000|pass

xgb1/17|16 |3 |2 |b7 |en |up |1:2:2:f|2 |46 |4 |4 |1a010000|pass



xgb1/15|14 |4 |1 |b7 |dis|dn |0:1:1:f|1 |36 |3 |0 |1a00e000|pass

Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc. Column 3- Indicates the UPC ASIC number. Column 4- Indicates the UPC ASIC interface number. 0-3 are the 4 port available on the Gen-1 ASIC.

Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example, Eth1/1 to Eth1/4 can be used in the same port channel because they are all mapped to UPC#0.

15

Figure 10: Interface Eth1/5 cannot be part of a same Port Channel as Eth1/1 Eth1/4

0 2 3

Eth1/1 . . .

10G Ethernet Interfaces

Only these interfaces can be part of the same

FCoE Port Channel

UPC #0

xgb1/3 = Eth1/3

LACP

0 1 2 3

UPC #1

Eth1/2 Eth1/3 Eth1/6Eth1/4 Eth1/5

1

On Nexus 5548/5596 switches:

The Nexus 5548 and 5596 switches use Carmel Unified Port Controller ASIC (UPC Gen 2). Each UPC ASIC have 8 ports mapped to front panel ports. This can be seen with the following command:

Nexus-5548# show hardware internal carmel all-ports Carmel Port Info: name |log|car|mac|flag|adm|opr|m:s:l|ipt|fab|xcar|xpt|if_index|diag|ucVer

-------+---+---+---+----+---+---+-----+---+---+----+---+--------+----+-----

xgb1/1 |0 |0 |0 -|b7 |dis|dn |0:0:f|0 |92 |0 |0 |1a000000|pass| 0.00









xgb1/10|9 |1 |1 p|b7 |en |up |1:1:f|1 |87 |0 |0 |1a009000|pass| 0.00

xgb1/11|10 |1 |2 -|b7 |dis|dn |2:2:f|2 |81 |0 |0 |1a00a000|pass| 0.00

xgb1/12|11 |1 |3 -|b7 |dis|dn |3:3:f|3 |86 |0 |0 |1a00b000|pass| 0.00

Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc. Column 3- Indicates the UPC ASIC number. Column 4- Indicates the UPC ASIC interface number. 0-7 are the 8 port available on the Gen-2 ASIC.

Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example, Eth1/1 to Eth1/8 can be used in the same port channel because they are all mapped to UPC#0.

16

Nexus switch-A configuration


Enable FCoE and LACP on the switch (disabled by default):


o feature fcoe

o feature lacp

Note: The Nexus can require a reload.


o vlan 200

o fcoe vsan 200

Create a port channel with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010:

o interface ethernet 1/5

o channel-group 200 mode active

o lacp rate fast



o lacp rate fast


o vlan 1,10,20

Create a trunk on the port channel interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):

o interface port-channel 200




Create a virtual Fibre Channel interface to carry the FCoE traffic through the port channel:


o bind interface port-channel 200

o no shutdown


o vsan database

o vsan 200







17


a. Create zones:

o zone name fcoe-zone vsan 200

o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port}


b. Create zoneset:

o zoneset name zoneset1 vsan 200

o member fcoe-zone

c. Activate zoneset:

o zoneset activate name zoneset1 vsan 200

Nexus switch-B configuration


Enable FCoE and LACP on the switch (disabled by default):


o feature fcoe

o feature lacp



o vlan 201

o fcoe vsan 201

Create a port channel with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010:



o lacp rate fast



o lacp rate fast


o vlan 1,10,20

Create a trunk on the port channel interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):

o interface port-channel 200




Create a virtual Fibre Channel interface to carry the FCoE traffic through the port channel:


o bind interface port-channel 200

o no shutdown

18


o vsan database

o vsan 201








a. Create zones:

o zone name fcoe-zone vsan 201

o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port}


b. Create zoneset:

o zoneset name zoneset1 vsan 201

o member fcoe-zone

c. Activate zoneset:

o zoneset activate name zoneset1 vsan 201

19

Virtual Connect configuration

Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS) which contain one FCoE network and optionally associated non-FCoE network(s). Figure 11: Virtual Connect Active-Active Converged Shared Uplink Sets scenario

VC Module bay 2

Active

FCF Mode

FlexHBA FlexNIC FlexNICFlexNIC

VC Domain

VC Module bay 1

FCF Mode

Active


vSwitch or NIC Teaming

FlexFabric Adapter Port 1


Multi Pathing

Nexus 5xxx Series ToR Switch using

Fabric Mode

X1 X1X2 X2

DCB-1 DCB-2

802.1q802.3ad

802.1q802.3ad Lossless

Ethernet Network

ConvergedShared Uplink Sets

Blade Server

20

An alternative scenario can be created using dedicated Share Uplink Sets for the FCoE traffic combined with Shared Uplink Sets for the standard Ethernet traffic since it is not mandatory to associate non-FCoE networks to a Converged Shared Uplink Set network. A typical scenario for this is when the Virtual Connect Domain cannot be modified and is already configured with Shared Uplink Sets that are using either Non-FCoE capable VC uplink ports or Dual Homed LACP configurations (it is not supported to connect a FCoE-capable Shared Uplink Set to two Top of Rack switches). This scenario is using additional uplink ports however these extra links used by the FCoE traffic can provide a better service level because the link is dedicated to the FCoE traffic and is not shared with any other types of network. Figure 12: Alternative Virtual Connect Active-Active Converged Shared Uplink Sets scenario with Dedicated SUS for the FCoE traffic.

FCF Mode


VC Domain

FCF Mode

Blade Server


vSwitch or NIC Teaming



Multi Pathing

Nexus 5xxx Series ToR Switch using

Fabric Mode

SUS-1

FCoE

Shared Uplink Set (Non-FCoE traffic)

SUS-FCoE-1 SUS-FCoE-2

FCoE

SUS-2

Active Non-FCoE

Active Non-FCoE

vpcpeer link

VC Module bay 1 VC Module bay 2

Converged Shared Uplink Set (FCoE traffic)Lossless Ethernet Network

Dual Homed LACP SUS configuration

Converged Shared Uplink Sets (SUS) Details and Restrictions

FCoE-capable Shared Uplink Sets (SUS) can contain both the FCoE network and non-FCoE networks.

FCoE-capable SUS can support port aggregation (802.3ad).

FCoE-capable SUS can support only one FCoE network.

FCoE-capable SUS must always contain ports from a single VC module.

Note: For Multi Enclosure Stacking (MES) environments, all corresponding ports in remote enclosures will be included in the same SUS. (e.g. selecting enc0:bay1:X1 means bay1:X1 in all remote enclosures is also included).

21

Defining two Converged Shared Uplink Sets

Create a SUS and name it DCB-1

On the Virtual Connect Manager screen, click Define, Shared Uplink Set to create a SUS.

Enter the Network Name of DCB-1.

Select Add Port, then add one or more ports from Bay 1.

Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of uplink to reach a good oversubscription ratio.

For Connection Mode, use Auto.

For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing in-service upgrades on upstream switch firmware.

For Associated FCoE Network, click on Add then enter:

o Network Name: FCoE-1.

o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the upstream switch.

For Associated Networks, click on Add then enter all Ethernet networks that are required.

o Select Smart Link for all defined networks.

22

Create a SUS and name it DCB-2

Enter the Network Name of DCB-2.

Select Add Port, then add one or more ports from Bay 2.

Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of uplink to reach a good oversubscription ratio.

For Connection Mode, use Auto.

For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing in-service upgrades on upstream switch firmware.

For Associated FCoE Network, click on Add then enter:

o Network Name: FCoE-2.

o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the upstream switch.

For Associated Networks, click on Add then enter all Ethernet networks that are required.

o Select Smart Link for all defined networks.

Note: For the removal of ports from a Converged Shared Uplink Set, it is recommended to first shutdown or remove a port from the vfc (interface vfc xx - no bind interface ethx/x) or channel group (interface ethx/x no channel-group xx mode active) on the FCF prior to removing the corresponding port from the Converged Shared Uplink Set in order to allow the FCF to properly halt traffic on the link. Failure to do so may result in loss of storage access.

23

FCoE Server traffic configuration

Unifying storage traffic and network traffic on the same link can present some challenges and usually requires more careful bandwidth management than standard Ethernet environments. One easy solution is to maintain the server traffic to reasonable connection speed by using the VC Maximum connection speed option.

This option limits the server traffic by providing a maximum connection speed option for defined networks. This option is recommended for high oversubscribed environment where the number of FCoE uplinks are limited which can increase the risk of congestion and could badly impact the network performance.

Note: Maximum bandwidth optimization requires the Flex-10 and FlexFabric adapter firmware and drivers to be updated to SPP version 2013.02.00 with Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01 compatibility, to take advantage of this enhancement. This feature excludes support for the following adapters:

HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter

HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter

HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter

To configure a maximum bandwidth limit from Server for the FCoE traffic:

Go to the Ethernet Networks and select the first FCoE network and click Edit.

Check the Advanced Network Settings option.

Check Set Maximum Connection Speed.

Select the best suitable speed for the maximum connection speed in order to meet your network design. Click Apply and Apply again to save the new FCoE network configuration.

This value will limit the maximum port speed from Server for the FlexHBA associated with this FCoE network.

24

Note: With 2 x 10Gb uplinks per VC modules connected to a Top of Rack switch using 50/50 bandwidth share and 16 servers using the FCoE network, it might be necessary to limit the maximum connection speed to get a reasonable FCoE over-subscription:

o For a 2:1 oversubscription, use 1.3Gb

o For a 4:1 oversubscription, use 2.5Gb

o For a 8:1 oversubscription, use 5Gb

o For a 16:1 oversubscription, use 10Gb

Repeat the same steps for the second FCoE network.

25

QoS Configuration

The default Virtual Connect configuration provides lossless service of the FCoE traffic. PFC (Priority Flow Control) mechanisms are used to provide the FCoE lossless service and prevent congestion. It is therefore not necessary to enable any particular QoS settings in the VC Domain.

Since Virtual Connect provides only best effort service for Ethernet traffic (i.e. flow control does not take place, in the event of congestion, packets get dropped) Administrator can decide to configure QoS in the Virtual Connect Domain and on the Nexus switches to improve the Ethernet traffic service level control.

The bandwidth allocation for FCoE-capable SUS uplinks always allocates only half the link bandwidth to FCoE traffic, with the other half allocated to Ethernet traffic. With VC 4.01, this allocation is fixed and cannot be modified.

Therefore it is important to configure the Nexus switches with the same Virtual Connect Domain QoS settings as follow:

FCoE lossless traffic class 3.

50/50% of bandwidth sharing between regular and lossless traffic.

Conflicting QoS parameter settings between the Virtual Connect Domain and the Nexus environment can badly impact the traffic and affect the performance.

Quality of Service configuration on the Nexus 5xxx series QoS on Nexus 5010 and 5020

On the Nexus 5010 and 5020 it is not necessary to modify the QoS settings because by default once the FCoE feature is configured (by entering feature fcoe) the default QoS template for shared links allocates the same shares and priorities than Virtual Connect. The following two classes of service are automatically configured:

class-fcoe is configured to be no-drop with a COS priority 3 and a MTU of 2158 with an ingress/egress bandwidth allocation of 50%.

class-default is configured to be drop with an MTU of 1500 with an ingress/egress bandwidth allocation of 50%.

26

To verify that the FCoE policy maps can be found in the running configuration, enter:

show policy-map system

QoS on Nexus 5500 series

On the Nexus 5500 series, enabling the FCoE feature does not create automatically no-drop policies like on the Nexus 5000 series, so before enabling FCoE, it is necessary to configure the QoS policies manually by including class-fcoe in each of the following policy types:

Network-Qos

Queuing

Qos

The following is an example of a service policy that needs to be configured:

o class-map type qos class-fcoe

o class-map type queuing class-fcoe

o match qos-group 1

o class-map type queuing class-all-flood

o match qos-group 2

27

o class-map type queuing class-ip-multicast

o match qos-group 2

o class-map type network-qos class-fcoe

o match qos-group 1

o class-map type network-qos class-all-flood

o match qos-group 2

o class-map type network-qos class-ip-multicast

o match qos-group 2

o system qos

o service-policy type qos input fcoe-default-in-policy

o service-policy type queuing input fcoe-default-in-policy

o service-policy type queuing output fcoe-default-out-policy

o service-policy type network-qos fcoe-default-nq-policy

To verify the FCoE system class is active on the interface connected to VC, enter:

show queuing interface Ethernet 1/5

For more detailed description of the QoS configuration on Nexus switches, see the Cisco Nexus 5000 Series NX-OS Quality of Service Configuration Guide at the following URL: http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html

28

Quality of Service configuration on the Virtual Connect modules It is not necessary to enable any particular QoS settings in the VC Domain to provide Quality of Service for the FCoE traffic. The moment FCoE is configured, Virtual Connect provides lossless service for the FCoE traffic.

By default, the VC QoS configuration mode is Passthrough. In this mode, two classes of service are provided, one for FCoE with lossless service and one for Ethernet without any form of flow control.

To improve the Ethernet traffic service level control and make VC to take a specific action on a specified classified traffic, the VC QoS settings can be changed from Passthrough to Custom (with FCoE Lossless). In this mode, VC monitors the Ethernet traffic for Layer 2 802.1p Priority bits, or Layer 3 DSCP or ToS markings and places packets on the pre-defined egress queues. The Ethernet traffic is then prioritized based on the traffic classification.

Optional VCM QoS Configuration

On the Virtual Connect Manager screen, click Configure then Quality of Service (QoS).

Change the default Pass-through mode to Custom (with FCoE Lossless).

29

In this mode, VC supports up to 8 configurable traffic classes:

o 1 predefined system class for Best Effort.

o 1 predefined system class for FCoE Lossless.

o 6 user defined classes.

Note: The default 802.1p priority for the Lossless FCoE traffic class is 3.

Note: The FCoE Lossless traffic share is based on the VC server profile configuration. The MAX Share is based on the FCoE Fabric configuration.

For each user defined classes you want to use, check the Enabled box and enter the appropriate Share/Max share and 802.1p priority.

Then click Apply.

30

Connecting Servers to the FCoE networks

To connect Blade servers to FCoE networks that have been configured, go to the Server Profiles interface.

Select a profile and click Edit.

Select the first FCoE HBA Connection and select the first FCoE network FCoE-1.

The server administrator can select the FCoE network like any other storage network when configuring profiles.

31

Select the second FCoE HBA Connection and select the second FCoE network FCoE-2.

The two Active/Active FCoE networks are now assigned to the server FCoE HBA.

To configure a Minimum Allocated Port Speed for the FCoE traffic:

Select Custom from the Port Speed Type drop down menu of the first FCoE port.

Then select the port speed, this speed is the minimum guaranteed bandwidth.

Maximum Connection

Speed that had been

defined for FCoE-1

Note: Make sure to select the best suitable Minimum Port Speed connection in order to meet your network design.

Click Ok.

Note: The Minimum Allocated Port Speed feature is an additional solution to overcome the challenges of network convergence discussed before. This Virtual Connect option guarantees that the FCoE traffic from server will get its minimum allocated bandwidth even if other FlexNICs on the same port generate too much traffic. Furthermore, if a congestion event occurs, the bandwidth is allocated based on the ratio of the Minimum setting. This option can really improve the Virtual connect bandwidth utilization and network performance.

32

Note: Minimum and maximum bandwidth optimization requires the Flex-10 and FlexFabric adapter firmware and drivers to be updated to SPP version 2013.02.00 with Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01 compatibility, to take advantage of this enhancement. This feature excludes support for the following adapters:

o HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter

o HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter

o HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter

Repeat the same steps for the second FCoE port.

Click Apply to activate the profile configuration.

The Allocated Port Speed should now be displayed with the Min and Max that have been defined.

The next step is to configure under the Server Operating System the NIC teaming and the FC multi-pathing for failover.

33

Nexus Troubleshooting commands

For more detailed information about Nexus FCoE troubleshooting guidance, see http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_fcoe.html

Disclaimer: we used the Cisco Nexus 5000 FCoE Troubleshooting Guide to create this section.

Useful Show commands

show flogi database

o This command displays all logins (server nodes and storage devices) which have logged into the Fabric using

FC and FCoE. Each FCoE Adapter port logs into the Fabric using the same virtual FC interface. The vfc are listed along with the VC assigned WWN addresses, the VSAN numbers and the FCIDs assigned by the Fabric to every single server.

show fcoe database

o This command is the same as the previous one but only displays the FCoE logins.

o It provides also the VC assigned MAC address of the FCoE adapter.

o Interfaces are listed with their FCIDs assigned by the Fabric and their PWWNs and MAC addresses assigned by Virtual Connect.

show fcoe

o This command shows if FCoE is enabled.

Server #1 Server #2 Server #3

34

o It displays the FCF-MAC and the FC-MAP of the Nexus device. This command is useful because every FCoE frame sent by servers always use this FCF-MAC address as their Destination MAC Address.

show fcdomain

show vsan 200 membership

35

show policy-map

o 2 traffic classes are defined, one for the FCoe traffic and one for Ethernet.

show class-map class-fcoe

o The FCoE traffic class is defined with priority 3.

Note: COS 3 is the default Industry standard value for the FCoE Lossless traffic class (IEEE 802.1p). The same priority is used by the Virtual Connect Domain.

36

show queuing interface ethernet 1/5

o This commands shows the ETS (Enhanced Transmission Selection - IEEE 802.1Qaz) configuration of an interface.

o It shows two traffic classes, one for the FCoE traffic (no-drop, MTU 2240) and one for the Ethernet traffic (drop, MTU 1500).

o The bandwidth allocation for each traffic class is 50% of the link bandwidth.

show logging last 20

o This command displays the last 20 system log messages of the device.

o Always very helpful, the Nexus logs can provide useful messages to help the troubleshooting.

show logging | grep vfc

o This command is useful to check the status of the virtual interface used for the FCoE traffic.

o The vfc can be up, down, administratively down, etc.

o A down but waiting for flogi can mean that the setup is ok but no blade server has been turned on yet.

50% of the link bandwidth is

reserved to the FCoE traffic

qos-group 1 defines the

lossless type of service (FCoE).

37

Per-Priority Flow Control commands Make sure PFC (Per-priority Flow Control - 802.1Qbb) is enabled on the interface connected to VC. Flow Control must be disabled. PFC ensures the lossless behavior of the FCoE traffic.

Check the status of Flow Control; it must be turned off on interfaces connected to Virtual Connect.

o show interface ethernet 1/5 flowcontrol

Check the status of the PFC. It must be turned on on interfaces connected to Virtual Connect.

o show interface ethernet 1/5 priority-flow-control

If the command shows Mode as Off or On, it should be changed to Auto. To force PFC mode Auto on an interface, enter:

interface ethernet 1/5

priority-flow-control mode auto

If the command shows Mode as Auto and Oper as Off or if the command returns nothing, you should check the setup of the Shared Uplink Set in VCM by making sure that an associated FCoE Network has been properly defined.

38

Issues with FCoE performance If FCoE throughput on servers is very low, this can be due to pause frames. It is necessary to check if the switch is sending pause frames or if it is getting paused.

Possible Cause:

If the egress FC port is congested, then the switch will send PFC frames to the Virtual Connect uplink port. The PFC frames are sent to reduce its FCoE rate and avoid a drop.

If the server is slow or congested, then the server will send PFC frames to the Virtual Connect downlink port. As a result Virtual Connect will eventually send PFC Frames to the Nexus interface.

To verify this situation, it is necessary to check the Nexus interface that is connected to the Virtual Connect modules.

When slow FCoE performance is not due to Pause Frames

For a few iterations using the "show interface ethx/x priority-flow-control" command, the PFC frame count (RX/TX) is not incrementing:

For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for both RX

and TX as Inactive:

39

Solution:

The slow FCoE performance does not seem to be due to network congestion. Check the Nexus interface to find information about possible link issues:

Use the show interface command to verify that the Nexus interface connected to VC is not showing errors, drops, discards, etc.

Use the show interface command to verify that the vfc interface is not showing errors or discards.

Use the show queuing interface to verify the number of discards.

When slow FCoE performance is due to Pause frames transmitted by the Nexus switch.

Figure 13: Nexus sending PFC Pause frames due to FC congestion.

FC Target

LAN

PPP

Congestion Point

Nexus interface connected

to Virtual Connect that needs to be monitored

40

When the Nexus switch is sending Pause frames to VC, the Nexus interface that is connected to the Virtual Connect module is showing the following:

For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for TX as Active:

For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause TxPPP counter is incrementing:

Solution:

Identify the source of the congestion and try to resolve it by increasing the FC bandwidth (or by adding additional Nexus FC ports to connect to the SAN environment).

41

When slow FCoE performance is due to Pause frames transmitted by VC/Servers.

Figure 14: Virtual Connect sending PFC Pause frames due to congestion.

FC Target

LAN

PPP

Congestion Point

PPP

Congestion Point

Nexus interface connected

to Virtual Connect that needs to be monitored

When the Virtual Connect uplink is sending Pause frames to the Nexus, the Nexus interface that is connected to the Virtual Connect module is showing the following:.

For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for Rx as Active as shown below:

42

For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause RxPPP counter is incrementing:

Solution:

Identify the source of the congestion and try to resolve it with the following action:

1. Increase the FCoE connection speed (Min and Max) of every servers.

2. Increase the number of VC FCoE Uplink ports connected to the Nexus.

Note: If congestion is expected, then Pause is expected for FCoE traffic.

Note: PFC Pause frames are currently not available in the Virtual Connect statistics. Only the standard Ethernet Pause frames are displayed.

43

Interface Status commands To check the status of the port connected to VC, you can enter:

show interface status err-disabled

VFC is usually down due to FIP solicitation failure and possible reasons are:

o Fabric is not available for flogi.

o Zoning or Storage presentation are misconfigured.

show interface vfc2005

show interface vfc2005 brief

44

FCoE Frames Analysis For network troubleshooting, capturing frames with a protocol analyzer can be very instructive but it is necessary to understand the way FCoE works. FCoE traffic use a different MAC address than the MAC address of the FCoE Adapter, the physical MAC address of the FCoE adapter (i.e. the MAC displayed in the Virtual Connect interface) is only used during FIP negotiation (FCoE Initialization Protocol). Figure 15: MAC address of the FCoE adapter found in the Server profile interface.

During the FIP negotiation, the FCF assigned a Fabric Provided MAC address (FPMA) to the server. This address is used for FCoE forwarding and is the one Wireshark will detect during FCoE network captures. The FPMA is built by appending the FCID (assigned by the Fabric to the server) to the Nexus FC-MAP. Figure 16: The way the FPMA is built.

FC-MAPs0E-FC-00 to 0E-FC-FF

FC-IDE5.01.00

FC-MAP0E-FC-00

FC_IDE5.01.00

24 Bits 24 Bits

FPMA

48 bits

0E-FC-00-E5-01-00

45

FPMA addresses assigned to servers cannot be found from the Virtual Connect interface, it is necessary to launch the Nexus console and use the following commands:

show fcoe

show fcoe database

Figure 17: How to find Fabric Provided MAC Addresses.

FPMA of server 1 : Oe:fc:00:e5:01:00



FCoE Frames sent by servers:

Destination MAC: the destination of FCoE frames sent by servers is always the MAC address of the Fibre Chanel Forwarder (i.e. the FCF-MAC address of the Nexus switch).

Source MAC: the source of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server by the FCF during the FIP negotiation.

FCoE Frames received by servers:

Destination MAC: the destination of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server by the FCF during the FIP negotiation.

Source MAC: the source of FCoE frames sent by the FCF to a server is always the MAC address of the Fibre Chanel Forwarder (i.e. the FCF-MAC address of the Nexus switch).

46

Figure 18: Sample of an FCoE frame sent by a server

FPMA (FC-MAP+FCID)

FCID of the FCoE Server

FCID of the Storage Target

47

Support and Other Resources

Contacting HP Before you contact HP

Be sure to have the following information available before you call contact HP:

Technical support registration number (if applicable)

Product serial number

Product model name and number

Product identification number

Applicable error message

Add-on boards or hardware

Third-party hardware or software

Operating system type and revision level

HP contact information

For help with HP Virtual Connect, see the HP Virtual Connect webpage: http://ww.hp.com/go/virtualconnect

For the name of the nearest HP authorized reseller:

See the Contact HP worldwide (in English) webpage: http://www.hp.com/country/us/en/wwcontact.html

For HP technical support:

In the United States, for contact options see the Contact HP United States webpage: http://welcome.hp.com/country/us/en/contact_us.html

To contact HP by phone:

Call 1-800-HP-INVENT (1-800-474-6836). This service is available 24 hours a day, 7days a week. For continuous quality improvement, calls may be recorded or monitored.

If you have purchased a Care Pack (service upgrade), call 1-800-633-3600. For more information about Care Packs, refer to the HP website: http://www.hp.com/hps

In other locations, see the Contact HP worldwide (in English) webpage: http://welcome.hp.com/country/us/en/wwcontact.html

Subscription service

HP recommends that you register your product at the Subscriber's Choice for Business website: http://www.hp.com/country/us/en/contact_us.html

After registering, you will receive email notification of product enhancements, new driver versions, firmware updates, and other product resources.

48

Related documentation HP Virtual Connect Manager 4.01 Release Notes http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03801912/c03801912.pdf HP Virtual Connect for c-Class BladeSystem Version 4.01 User Guide http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03791917/c03791917.pdf HP Virtual Connect Version 4.01 CLI User Guide http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03790895/c03790895.pdf HP Virtual Connect for c-Class BladeSystem Setup and Installation Guide Version 4.01 and later http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03801914/c03801914.pdf HP Virtual Connect FlexFabric Cookbook http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02616817/c02616817.pdf HP Virtual Connect Fibre Channel Networking Scenarios Cookbook http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01702940/c01702940.pdf HP Virtual Connect with iSCSI Cookbook http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02533991/c02533991.pdf HP Virtual Connect Multi-Enclosure Stacking Reference Guide http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02102153/c02102153.pdf Converged Networks and Fibre Channel over Ethernet Technical Whitepaper http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03440617/c03440617.pdf

HP welcomes your feedback. To make comments and suggestions about product documentation, send a message to [email protected]. Include the document title and manufacturing part number. All submissions become the property of HP.

Get connected hp.com/go/getconnected

Current HP driver, support, and security alerts delivered directly to your desktop

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omiss ions contained herein.

Trademark acknowledgments, if needed.

4AA4-xxxxENW, Created Month 2013

Get connected hp.com/go/getconnected

Current HP driver, support, and security alerts delivered directly to your desktop

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Trademark acknowledgments, if needed.

c03808925, Created June 2013

introduction to storage

Documents

fcoe support

fcoe dual hop

fcoe solution

fcoe traffic

fcoe performance

conversion of fcoe

fcoe hops

fcoe frames analysis