storage networking fcf_co_eiscsivsn_technology

1 © Copyright 2014 EMC Corporation. All rights reserved. © Copyright 2014 EMC Corporation. All rights reserved.

Storage Networking (FC/FCoE/iSCSI) technology and best practice update Erik Smith


Agenda • What’s new with Connectivity

– Protocols – Connectrix-B (Brocade) – Connectrix-C (Cisco) – Things you should know – Hot Topics

• OpenStack and Fibre Channel

• Virtual Storage Networks


Protocol Overview • Storage Capacity forecast by access protocol

Source IDC


Protocol Overview • Traditional Enterprise Storage versus “Server SAN”

Source: Wikibon “The Rise of Server SAN”

Today


Protocol Overview Block NAS Object


Protocol Overview Block

FC Family iSCSI Family Server SAN

NAS Object



FCoEFC


NAS Object

• FC is the most popular – 99.99999 (7 nines) of Availability – Outstanding performance (16G today, 32G and 128G soon) – Inertia (well established) – Specialized skill sets to install and maintain – Scalability limits – Difficult to automate

- - -



Loss-less RDMAiSCSIFCoEFC


NAS Object

• iSCSI - As defined in RFC 7143 (uses TCP) – Does not require specialized HW or skills – Easier to automate (but at the end devices) – Requires more CPU and memory (generally)

• Not a huge issue on the host • Becomes a bigger concern on the storage array

-


Protocol Overview - Performance

Source: Manoj Wadekar - QLogic

READ IOPS

Block Size (kB)




WRITE IOPS

Block Size (kB)





NAS Object

• iSCSI…but uses DCB Ethernet – Does improve iSCSI peformance in certain topologies – Could decrease buffering requirements, but… – DCB can be painful to configure in arbitrary topologies

• If you’re going to go through all of this trouble, why not just use RDMA??

-





NAS Object

• iSCSI using iSER - As defined in RFC 7145 – Reduces CPU and memory utilization as well as Latency – RoCE, iWARP and IB can be used – With RoCE, it’s not clear how this will be supported over

arbitrary L3 topologies due to DCB configuration – May be susceptible to the same congestion issues as FC – Why not just use FCoE?

-

-





Ethernet

NAS Object

• ScaleIO, VSAN, CEPH, etc – A potentially disruptive technology – Uses Server capacity to create pools of capacity – Easier to configure – Lacks enterprise features (today) – Resource utilization concerns as features are added – Network utilization, multi-tenancy, Disaggregation?

- - -





EthernetSMB SMB

DirectNFS

NAS Object

• Allows for file access on a remote system – Supports many different applications from file shares to

shared storage for server virtualization – Easy to configure and manage – Lower performance but RDMA helps -




READ IOPS

Block Size (kB)




WRITE IOPS

Block Size (kB)





EthernetSMB SMB

DirectNFS

NAS Object

REST S3

• Data treated as Objects (e.g, OS image, SNAP) – Very simple to use – Not well suited to transactional workloads -


Connectrix-B (Brocade) updates • New products

– DS-6520B 16G switch (released April 13)

• Firmware releases – Target releases

• 7.0.2c • 6.4.3d

– New releases • 7.1.0

– Support for DS-6520B – Increased ICL Distance support from 50m to 100m on DCX8510 optical ICLs – ClearLink (formerly known as D_Port) Enhancements – Enhancements to In-Flight Encryption / Compression


Connectrix-B (Brocade) updates • New releases (cont)

– 7.2.0d • Fabric Vision

• Monitoring and Alerting Policy Suite (MAPS) • Flow Vision • ClearLink Diagnostics (D_Ports) • Dashboards


Connectrix-C (Cisco) updates • New products

– 9710 (May 2013) • Line rate 16G • Up to 512 ports per chassis • No FCoE or FICON

– 9250i (April 2014) • FCIP • FCoE (non converged) • No FICON


Connectrix-C (Cisco) updates • Firmware releases

– Target releases • 3.3.5b • 4.2.9 • 5.0.8 (Gen 3) • 5.2.8b (Gen 4) • 6.2.1 (9710)

– New releases • 5.2.8c • 6.2.1 • 6.2.3 • 6.2.5


Things you should know - Overview • There are concepts that you should be familiar with

if you’re working with a SAN on a regular basis: – Congestion (next) – Bit errors and the impact to the ULP (e.g., SCSI-FCP) – Over-subscription – NPIV – Login, Name Server and Domain scalability limitations

• For more information see the “Networked Storage Concepts and Protocols” EMC TechBook


Monitor For Congestion • Oversubscription – A condition

that exists when the potential demand exceeds the capacity of the system


Monitor For Congestion • Congestion - A condition

that exists when the actual demand exceeds the capacity of the system


• Slow drain device – May be unable to keep up with line rate due to workload – Could cause congestion spreading

Monitor For Congestion (continued)


Monitor For Congestion (continued) • Slow drain device

– May be unable to keep up with line rate due to workload – Could cause congestion spreading


Monitor For Congestion (continued) • Slow drain impact on other N_Ports


Hot topics - Overview • Hot Topics - Problems that seem to be causing the

most pain to our customers. – Gathered by working with our support organization and

engineering teams – There are several categories:

• Common • Connectrix-B • Connectrix-C • Distance extension • Remote replication

• Workarounds are provided where available


Hot topics – Common • Problem: Slow Drain device

• Symptom: – Poor performance

• Especially after a new end device (or application) is added to the SAN

• Solutions: – When possible analyze how a planned change will impact overall

SAN health / performance – Use SAN performance monitoring tools

• Brocade Fabric Vision or Virtual Instruments – Monitor for Slow Drains

• Cisco - Configure Slow Drain Device Detection and Congestion Avoidance • Brocade – Fabric Vision or Enable bottleneck detection


Hot topics - Connectrix-B (Brocade) • Problem: Stuck Virtual Channel (VC)

• Symptom: – Performance issues – Switch Log contains:

• 2011/11/21-15:37:01:534223, [C2-5021], 115166/0, SLOT 6 " CHASSIS, WARNING, ED_48000B, S10,P-1(5): Link Timeout…

• Solution: – See Brocade DEFECT000342738

• Only an issue with 8G Brocade switches • Fixed at >= 6.4.2a – Contact support for procedure (if upgrading) • 8G 64 Port line card seems to be particularly susceptible


Hot topics - Connectrix-B (Brocade) • Problem: Webtools/Java compatibility

• Symptom: Webtools fails to launch after Java is upgraded to version 7 update 40 or higher

• Solution(s): – downgrade Java to Java 7u25 or earlier – Modify java.security configuration file

• See EMC 000177076 for more information

• Prevention: – Refer to release notes for supported versions


Hot topics - Connectrix-B (Brocade) • Problem: FCIP stability

• Symptom: FCIP tunnels bouncing or stuck at In-Progress

• Solution(s): – Brocade is recommending that all environments using

FCIP upgrade to 7.1.0c and above.


Hot topics - Connectrix-C (Cisco) • Problem: FCoE performance

• Symptom(s): – Poor I/O performance is observed. – FCOE attached hosts (using CNAs) log abort errors – The FCOE interfaces on the Nexus switch where the FEX (Nexus 2000

series) module attaches will log increasing pause frames • From the SAN towards the CNA on the FEX.

– Storage attached interfaces on the MDS switch log high buffer transitions to zero

• Solution: – Configure Jumbo frames on lossless priority – Upgrade to NX-OS 5.1(3)N1(1a) or above (see CSCtz63490)


Hot topics - Connectrix-C (Cisco) • Problem: DCNM login display issue

• Symptom: Web GUI does not display the NPV host device under inventory

• Solution: None – See CSCui69324


Hot topics - Connectrix-C (Cisco) • Problem: GUI performance

• Symptom: – General performance issues, slowness, – The web gui will occasionally log an error that tells you

specifically to vacuum the database to improve performance

• Solution: – EMC 000176676 - Cisco: How to vacuum Postgres

Database for Windows based DCNM Installs


Hot topics – Distance Extension • Best practices

– For all client-side ports (i.e. E_ports) attached to distance devices:

• Hard-set the port-speed to the highest expected link rate (i.e. remove auto-negotiation). This will reduce the link initialization time after unexpected link-bounces

– For switches that support BBScN, use it. • There are times when credits/R_RDYs are lost and this feature is used

to recover them. • For Brocade switches this is under the “Brocade Credit Recovery”

feature. • For Cisco switches, there’s a BBScN feature/mode.


Hot topics – Distance Extension • Best practices

– Verify optimum optical medium conditions: • Verify clean Fiber Optic cables • No mismatching of different Fiber Optic cable core sizes • Verify light levels are within thresholds


Hot topics – Remote Replication • Problem: Degraded SRDF Performance

• Symptom(s): – Poor I/O performance is observed. – High BBC=0 on RF or ISL ports

• Contributing factors – SAN switch ISLs extended over DWDM or Wan extender – ISLs throttled to 1-2Gb for extender connectivity

• Solution: – Add additional ISLs to compensate for reduced speeds – Verify no oversubscription on ISL and RF ports on switches – Verify extender configuration: speed limits, compression, etc – Verify RFs spread evenly across all Fabrics/Wan paths


Hot topics – Remote Replication • Problem: Degraded Performance

• Symptom(s): – Poor I/O performance or response time is observed. – Logging link errors

• Solution: – Guaranteed bandwidth (use network qos for shared networks) – Verify speed limits to match guaranteed bandwidth – Verify host workload has not out-paced replication links – may need to add

more links – For synchronous replication, enough RA links to maintain <40% utilization


OpenStack - Overview • Management and Orchestration (M&O) layer

– Infrastructure as a Service (IaaS) – Open source – See openstack.org for additional information

• FC support introduced in the “Grizzly” release – No support for automated zoning

• Brocade started a project to add zoning automation – EMC participated in the review process – Support is available in “Icehouse” release – At least three workflows are possible.


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API

Orchestration (REST, AMQP)

Nova Neutron

Volume Driver


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova Neutron

Volume Driver


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova Neutron

Volume Driver

R


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova Neutron

Volume Driver

R

Connector: Host = H1

Storage = ??


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova Neutron

Volume Driver

R


Storage = ??


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova Neutron

Volume Driver

R


Storage = S1


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova

FC Zone Driver

FC Zone Manager

Neutron

Volume Driver


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova

FC Zone Driver

FC Zone Manager

Neutron

Volume Driver

R


Storage = ??


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova

FC Zone Driver

FC Zone Manager

Neutron

Volume Driver


Storage = ??


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova

FC Zone Driver

FC Zone Manager

Neutron

Volume Driver

Zoning Mode = Fabric


Storage = S1 ZONE: H1, S1


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN AStorage

WWPN: S1

Volume API


Nova

FC Zone Driver

FC Zone Manager

Neutron

Volume Driver



Storage = S1


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

Neutron

Volume Driver


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver


R Connector: Host = H1

Storage = ??


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver



Storage = ??

Candidates: Storage = S1 Storage = S2


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver



Storage = ??

Connectable: Host = H1

Storage = S1


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver



Storage = S1

Provision: Host = H1

Storage = S1


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver



Storage = S1

ZONE: H1, S1


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver

X

X

ViPR

Zoning Mode = None


OpenStack and FC

Host A

Cinder

Volume Manager

WWPN: H1

SAN A

SAN B

StorageWWPN:

S1WWPN:

S2

Volume API


Nova

FC Zone Driver

FC Zone Manager

FC Lookup Service

Neutron

Volume Driver

X

X

ViPR OR

Zoning Mode = None


Virtual Storage Networks • The Virtual Storage Network concept seeks to enable

the dynamic composition of multi-tenant Storage Services


Virtual Storage Network(VSN)

VPLEX

Data Domain

VMAX

Network

Hypervisor

Hypervisor

On-demand composition

of multi-tenant

storage services



VPLEX

Data Domain

VMAX

Network

Hypervisor

Hypervisor


of multi-tenant

storage services



VPLEX

Data Domain

VMAX

Network

Hypervisor

Hypervisor


of multi-tenant

storage services

VSN-CS



VPLEX

Data Domain

VMAX

L2 or L3 Network

VXLAN Tunnels

HypervisorOVS

HypervisorOVS


of multi-tenant

storage services

VSN-CS


VPLEX

Data Domain

VMAX

VTEP A

VTEP Access

VTEP B

VTEP Access

VTEP C

VTEP Trunk

L2 or L3 Network

VXLAN Tunnels

HypervisorOVS

HypervisorOVS


VXLAN Tunnel End Point (VTEP) VTEP enabled switch

VTEP enabled switch

VTEP Trunk port

UDPIPEth FCS

VXLAN VLAN

FCSVXLAN DATATCPIPEth DATATCPIPEth


VXLAN Tunnel End Point (VTEP) VTEP enabled switch

VTEP enabled switch

VTEPVXLAN 1 VLAN 1Trunk

VXLAN 2

VXLAN 3

VLAN 2VLAN 3


VPLEX

Data Domain

VMAX

VTEP A

VTEP Access

VTEP B

VTEP Access

VTEP C

VTEP Trunk

L2 or L3 Network

VXLAN Tunnels

HypervisorOVS

HypervisorOVS


NSX Based Storage Use Case

VNXESX

Network (L2||L3)VM 1 N

SX-VS

VDM1

3 4

NSX – Multi-HypervisorControl Cluster

OVSDB

VLANVXLANVTEP

ToRM

2

C1 C2 C3

VSN Connectivity ServiceVSN Connectivity Service


NSX Based Storage Use Case File based storage access from within a VM

VTEP enabled switch

VTEP

VXLAN 1 VLAN 1

Trunk

VXLAN 2VXLAN 3

VLAN 2VLAN 3

VDM RED

VDM GRN

VDM BLUE

VNX

VM RED

VM GRN

VM BLUE

Hypervisor


Vs.

Virtual Storage Networks Multi-tenancy


Vs.

Virtual Storage Networks Multi-tenancy

“Storage Networking: Network Virtualization & Multi-Tenant Storage (Birds-of-a-Feather)”

Wednesday at 1:00


New E-Lab Navigator Released on April 14th!


Related Sessions • Converged Data Center: FCoE, iSCSI & The Future

Of Storage Networking

• Presented by David Black – Monday 1:30 – 2:30 – Tuesday 1:30 – 2:30


EMC SAN Health Quiz

Visit the VMAX side of the Enterprise and Mid-Range Systems Division booth!

Ever wondered how healthy your SAN really is?

storage networking fcf_co_eiscsivsn_technology

Technology