brocade fabric technology with the violin memory 6232 ...€¦with the violin memory 6232 flash...

53-1004096-01December 2015

Brocade Fabric Technologywith the Violin Memory6232 Flash ArrayValidation Test Report

Supporting Fabric OS 7.4.0

© 2015, Brocade Communications Systems, Inc. All Rights Reserved.

ADX, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, The EffortlessNetwork, VCS, VDX, Vplane, and Vyatta are registered trademarks, and Fabric Vision and vADX are trademarks of BrocadeCommunications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned may betrademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning anyequipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this documentat any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not becurrently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained inthis document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to theaccuracy of this document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs thataccompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other opensource license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable tothe open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

http://www.brocade.com/support/oscd

Contents

Preface..................................................................................................................................... 5Overview........................................................................................................... 5Purpose of This Document................................................................................5Audience........................................................................................................... 5Objectives......................................................................................................... 5Related Documents...........................................................................................6Document History..............................................................................................6About Brocade.................................................................................................. 6About Violin Memory......................................................................................... 6

Configure Equipment................................................................................................................ 7Task 1: Brocade FC Fabric Configuration.........................................................7Task 2: Violin Memory Array Configuration.......................................................9Task 3: Host Setup..........................................................................................13

Test Plan.................................................................................................................................17Scope.............................................................................................................. 17What's New in This Report..............................................................................17Test Configuration...........................................................................................18DUT Descriptions............................................................................................ 18DUT Specifications..........................................................................................19Test Equipment............................................................................................... 20

Test Cases.............................................................................................................................. 211.0 Fabric Initialization—Base Functionality................................................... 22

1.0.1 Storage Device—Physical and Logical Login with SpeedNegotiation.................................................................................... 22

1.0.2 Zoning and LUN Mapping.......................................................... 231.0.3 Storage Device Fabric I/O Integrity............................................ 231.0.4 Storage Device Multipath Configuration—Path Integrity............24

1.1 Fabric—Advanced Functionality............................................................... 241.1.1 Storage Device Bottleneck Detection Using MAPS FPI—

With Congested Hosts...................................................................251.1.2 Storage Device Bottleneck Detection Using MAPS FPI—

With Congested Fabric.................................................................. 261.1.3 Storage Device—QoS Integrity with QoS Zone-Based

Traffic Prioritization........................................................................271.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic

Prioritization...................................................................................271.1.5 Storage Device—FC Protocol Jammer Test Suite.....................301.1.6 Clear Link Diagnostics (D_Port) Test.........................................31

1.2 Stress and Error Recovery with Device Multipath.....................................321.2.1 Storage Device Fabric I/O Integrity—Congested Fabric............ 331.2.2 Storage Device Nameserver Integrity—Device Recovery

with Port Toggle.............................................................................331.2.3 Storage Device Nameserver Integrity—Device Recovery

with Device Relocation.................................................................. 34

Brocade Fabric Technology with the Violin Memory 6232 Flash Array Validation Test Report 353-1004096-01

1.2.4 Storage Device—Device Recovery with Device PortToggle—Extended Run............................................................... 35

1.2.5 Storage Device Recovery—ISL Port Toggle ...........................361.2.6 Storage Device Recovery—ISL Port Toggle (Entire Switch)... 361.2.7 Storage Device Recovery—Director Blade Maintenance........ 371.2.8 Storage Device Recovery—Switch Offline...............................391.2.9 Storage Device Recovery—Switch Firmware Download......... 39

1.3 Storage Device—Fibre Channel Routing Internetworking Tests............ 401.3.1 Storage Device Internetworking Validation with the FC

Host............................................................................................. 401.3.2 Storage Device Internetworking Validation with FCoE Test

Using VDX FlexPort.....................................................................411.3.3 Storage Device Edge Recovery After FCR Disruptions...........421.3.4 Storage Device Backbone Recovery After FCR Disruptions... 43

1.4 Optional/Additional Tests........................................................................ 441.4.1 Performance I/O Testing with Varying Workload

Dimensions..................................................................................441.4.2 VMware I/O Analyzer Workload Test.......................................45

Test Conclusions...................................................................................................................47

4 Brocade Fabric Technology with the Violin Memory 6232 Flash Array Validation Test Report53-1004096-01

Preface

● Overview........................................................................................................................... 5● Purpose of This Document................................................................................................5● Audience........................................................................................................................... 5● Objectives......................................................................................................................... 5● Related Documents...........................................................................................................6● Document History..............................................................................................................6● About Brocade.................................................................................................................. 6● About Violin Memory......................................................................................................... 6

OverviewThe Solid State Ready (SSR) program is a comprehensive testing and configuration initiative to provideFibre Channel SAN and IP interoperability with flash storage. This program provides testing of multiplefabrics, heterogeneous servers, NICs, and HBAs in large port-count Brocade environments. Testingcovers future Brocade OS versions and vendor software releases. The SSR qualification program helpsverify seamless interoperability and optimum performance with solid state storage in Brocade SANfabrics.

Purpose of This DocumentThis document provides the validation of Brocade fabric technology with the Violin 6232 all flash storagearray, using multiple switch platforms, HBAs, and server operating systems. This validation shows thatthe Violin 6232 interoperates properly within a Brocade Fibre Channel fabric, while supporting theperformance and low latency associated with solid state storage.

AudienceThe content in this document is written for a technical audience, including solution architects, systemengineers, and technical development representatives.

Objectives1. Test the Violin 6232 array with Brocade FC fabrics in single and routed configurations for different

stress and error recovery scenarios and validate the interoperability and integration of the array withBrocade FC fabrics.

2. Validate the performance of the FC fabric in a solid state storage environment for high-throughputand low-latency applications.


Related Documents• Brocade Fabric OS Administrator's Guide, 7.4.0• Brocade SAN Design and Best Practices• Brocade Monitoring and Alerting Policy Suite Administrator's Guide, 7.4.0• Brocade Network OS Layer 2 Switching Configuration Guide. 5.0.1• Emulex ExpressLane Configuration• Emulex OneCommand Manager User Manual• QLogic QLE2672 Adapter User's Guide• QLogic BR-1860 Adapter Administrator's Guide

Document History

Date Description

December 2015 Initial version.

About BrocadeBrocade networking solutions help the world's leading organizations transition smoothly to a worldwhere applications and information reside anywhere. This vision is realized through the BrocadeOne™ strategy, which is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.

Innovative Ethernet and storage networking solutions for data center, campus, and service providernetworks help reduce complexity and cost while enabling virtualization and cloud computing toincrease business agility.

To help ensure a complete solution, Brocade partners with world-class IT companies and providescomprehensive education, support, and professional services offerings.

To learn more, visit www.brocade.com.

About Violin MemoryBusiness in a Flash. Violin Memory transforms the speed of business with high performance, alwaysavailable, low-cost management of critical business information and applications.

Violin’s All-Flash optimized solutions accelerate breakthrough CAPEX and OPEX savings for buildingthe next generation data center. Violin’s Flash Fabric Architecture (FFA) speeds data delivery withchip-to-chassis performance optimization that achieves lower consistent latency and cost pertransaction for cloud, enterprise, and virtualized mission-critical applications. Violin’s All-Flash arraysand appliances and enterprise data management software solutions enhance agility and mobility whilerevolutionizing data center economics.

Founded in 2005, Violin Memory is headquartered in Santa Clara, California.

Related Documents


http://www.brocade.com/content/dam/common/documents/content-types/administration-guide/fos-740-adminguide.pdf

http://www.brocade.com/downloads/documents/best_practice_guides/san-design-best-practices.pdf

http://www.brocade.com/content/dam/common/documents/content-types/administration-guide/fos-740-maps.pdf

http://www.brocade.com/downloads/documents/product_manuals/B_VDX/NOS_L2Guide_v501.pdf

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0CCMQFjAB&url=http%3A%2F%2Fwww.emulex.com%2Fartifacts%2FBBE3C568-802C-4CD0-8471-C7B1680232CC%2Felx_wp_all_expresslane.pdf&ei=K-BLVcDZIpfpoAT41oC4DQ&usg=AFQjCNEpdBY_8qiiBUcmP9Puh2o4EC0elQ&sig2=W_7n3oUL3TtegJYFUYbEPA&bvm=bv.92765956,d.aWw

http://www-dl.emulex.com/support/elx/rt10.4.0/ga/Docs/final/apps/ocm_gui_manual_elx.pdf

http://filedownloads.qlogic.com/Files/TempDownlods/90481/UsersGuide_2600_FCAdapter_FC0054609-00G.pdf

http://filedownloads.qlogic.com/Files/TempDownlods/90546/AdminGuide_BRSeriesAdapters_BR0054501-00C.pdf

http://www.brocade.com

Configure Equipment

● Task 1: Brocade FC Fabric Configuration.........................................................................7● Task 2: Violin Memory Array Configuration.......................................................................9● Task 3: Host Setup..........................................................................................................13

Task 1: Brocade FC Fabric ConfigurationConfiguration settings of the Brocade switches in the test bed are covered here.

1. Zoning is configured using the Peer Zoning feature in Fabric OS 7.4.0. Peer zoning allows a"principal" device to communicate with the other devices in the zone. The principal device manages apeer zone. Other "nonprincipal" devices in the zone can communicate with the principal device only;they cannot communicate with each other.

root> zonecreate –peerzone violin_peer –principal “ 21:00:00:24:ff:39:21:53; 21:00:00:24:ff:39:fc:7b;50:01:43:80:09:af:5e:e6; 50:01:43:80:09:af:67:66;21:00:00:24:ff:39:21:52; 21:00:00:24:ff:39:fc:7a;50:01:43:80:09:af:5e:e4; 50:01:43:80:09:af:67:64” –members “10:00:8c:7c:ff:24:4c:00; 10:00:8c:7c:ff:24:4c:01 …”root> zoneshow --peerzone allzone: violin_peer Property Member: 00:02:00:00:00:03:00:08 Created by: User Principal Member(s): 21:00:00:24:ff:39:21:53; 21:00:00:24:ff:39:fc:7b; 50:01:43:80:09:af:5e:e6; 50:01:43:80:09:af:67:66; 21:00:00:24:ff:39:21:52; 21:00:00:24:ff:39:fc:7a; 50:01:43:80:09:af:5e:e4; 50:01:43:80:09:af:67:64 Peer Member(s): 10:00:8c:7c:ff:24:4c:00; 10:00:8c:7c:ff:24:4c:01 10:00:00:90:fa:61:92:3b; 10:00:00:90:fa:61:92:3c; 10:00:8c:7c:ff:05:60:01; 10:00:8c:7c:ff:05:60:00; 10:00:8c:7c:ff:14:e0:01; 10:00:8c:7c:ff:14:e0:00; 10:00:8c:7c:ff:03:bc:01; 10:00:8c:7c:ff:03:bc:00; 10:00:8c:7c:ff:03:9b:00; 10:00:8c:7c:ff:03:9b:01; 10:00:8c:7c:ff:05:72:02; 10:00:8c:7c:ff:05:72:03

2. Configure MAPS on the switches (requires a Fabric Vision license). This enables reporting of latencyand congestion alerts on each switch, as well as a number of switch and fabric health metrics. Formore information on configuring and using MAPS, see the Monitoring and Alerting Policy SuiteAdministrator's Guide, 7.4.0. Enable the desired MAPS policy using any available default policy orcreating a custom policy. For this test, the default Aggressive Policy is used. This provides the mostsensitive threshold levels for detection of latency and congestion.

> mapsconfig --enablemaps -policy dflt_aggressive_policyDefine the reporting actions that MAPS takes. In this test, errors discovered by MAPS generate aRASLog entry and send e-mail to the configured recipients:

> mapsconfig --actions raslog,emailThe Fabric Performance Impact (FPI) category within MAPS monitors the current condition of thelatency seen on F_Ports over different time windows, and it uses that to determine the performanceimpact to the fabric and network. In order to use FPI monitoring, legacy 'bottelneckmon' must bedisabled:

> bottleneckmon –disable




Enable FPI monitoring:

> mapsConfig –enableFPImonTo examine the effective MAPS configuration, use the following command:

> mapsconfig –showConfigured Notifications: RASLOG,EMAILMail Recipient: [email protected],[email protected] Monitoring: EnabledPaused members :================PORT :CIRCUIT :SFP :Use the following command to view the summary of the events or rules triggered and the objects onwhich the rules were triggered over a specified period of time.

> mapsdb –show1 Dashboard Information:========================DB start time: Wed Dec 17 20:59:09 2014Active policy: dflt_aggressive_policyConfigured Notifications: RASLOG,EMAILFenced Ports : NoneDecommissioned Ports : None

3. Configure FlexPorts on the VDX switches for Fibre Channel support (an FCoE Base license isrequired). These ports are used as uplinks from the VCS fabric to the Fibre Channel fabric in thistest. For detailed information on configuring VDX 6740 switches for Fibre Channel support, see the Network OS Layer 2 Switching Configuration Guide, 5.0.

Example of configuring a Fibre Channel port on a Brocade VDX 6740 switch:

# conf t(config)# hardware(config-hardware)# flexport 112/0/38(config-flexport-112/0/38)# type fibre-channelFlexPort speeds are configured in connector groups. In this test-bed example, the HighMixed speedis chosen. This speed allows the Fibre Channel FlexPort to operate at an 16-Gb speed, which isoptimal because the uplink is attaching to a 16-Gb Fibre Chanel SAN. Also, not all ports in thisconnector group are configured as Fibre Channel ports; some are still configured as Ethernet ports,and the HighMixed setting will support the Ethernet speeds on these ports as well. For detailedinformation on configuring VDX 6740 connector group speeds, see the Network OS Layer 2Switching Configuration Guide.

• LowMixed—2/4/8G Fibre Channel, and Ethernet speeds (default)• HighMixed—16G Fibre Channel, and Ethernet speeds• FibreChannel—2/4/8/16G Fibre Channel

Use the following command to set the connector-group speed to HighMixed:

(config-hardware)# connector-group 111/0/3(config-connector-group-111/0/3)# speed HighMixed

4. Configure zones for FCoE initiators on the VDX switches.

Example zone on a Brocade VDX 6740 switch:

# show zoning enabled-configurationzoning enabled-configuration cfg-name NOS_SSRzoning enabled-configuration enabled-zone lsan_hb067166_violin member-entry 10:00:8c:7c:ff:1f:7b:00 member-entry 10:00:8c:7c:ff:1f:7b:01 member-entry 21:00:00:24:ff:51:7e:86 member-entry 21:00:00:24:ff:51:7d:ae

Configure Equipment





member-entry 21:00:00:24:ff:51:7d:e8 member-entry 21:00:00:24:ff:51:7d:9c

5. Configure Fibre Channel Routing (an Integrated Routing license is required). Detailed information onFCR setup can be found in the Fabric OS Administrator's Guide, 7.4.0.

Example FCR configuration is shown:

> fcrconfigure –bbfid 100> fosconfig --enable fcr> portcfgexport [port#] -a1 –m[0/5] -f 10 {0=Brocade FC fabric; 5=Brocade NOS fabric}The prefix lsan is used when configuring zones for use in Fibre Channel Routing.

Example of a zone prefixed with lsan:

> zoneshow lsan_hb067166_violin zone: lsan_hb067166_violin 10:00:8c:7c:ff:23:b7:00; 10:00:8c:7c:ff:23:b7:01; 52:4a:93:7d:f3:5f:61:00; 52:4a:93:7d:f3:5f:61:11Example output of exported devices:

> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 10 21:00:00:24:ff:48:b9:6a 02f001 20 551a00 Imported 10 21:00:00:24:ff:48:b9:6b 02f101 20 541e00 Imported 10 52:4a:93:7d:f3:5f:61:00 02f201 20 550e00 Imported 10 52:4a:93:7d:f3:5f:61:01 02f401 20 540400 Imported

Task 2: Violin Memory Array ConfigurationConfiguration steps for the Violin Memory array are covered here.

1. Create a host initiator group for each host.

Task 2: Violin Memory Array Configuration


http://www.brocade.com/content/html/en/administration-guide/fos-740-adminguide/index.html

FIGURE 1 New Host Initiator Group Creation on the Violin Memory Array

2. Add WWPNs to the host.

Configure Equipment


FIGURE 2 Adding Host Ports to the Host Configuration on the Violin Memory Array

3. Create new LUNs. For this test, 8 x 10GB LUNs are configured.

Configure Equipment


FIGURE 3 Creating New LUNs on the Violin Memory Array

4. Export LUNs to the host.

Configure Equipment


FIGURE 4 Exporting LUNs to the Host on the Violin Memory Array

Task 3: Host SetupConfiguration settings of the servers in the test bed are covered here.

1. Provision a minimum of two uplinks from the host to the FC fabric for redundancy, and use nativemultipath tools to manage the available paths and load-balance across them.

2. Multipath configuration on Linux host. This configuration allows all paths to be used in a round-robinfashion, providing superior performance to the default Linux settings, which would use only a singleactive path per LUN. Recommended /etc/multipath.conf entry on Linux systems:

device { vendor "VIOLIN" path_selector "round-robin 0" path_grouping_policy multibus rr_min_io 1 path_checker tur fast_io_fail_tmo 10 dev_loss_tmo 30 }Example multipath configuration on Linux:

# multipath -llmpathbp (36001b970b716b179b716b1795996a912) dm-13 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 13:0:5:3 sdbp 68:48 active ready running |- 13:0:6:3 sdbx 68:176 active ready running |- 14:0:5:3 sdcj 69:112 active ready running |- 13:0:4:3 sdcg 69:64 active ready running |- 13:0:7:3 sdcw 70:64 active ready running |- 14:0:6:3 sdcz 70:112 active ready running |- 14:0:7:3 sddl 71:48 active ready running `- 14:0:4:3 sddt 71:176 active ready runningmpathbo (36001b970b716b179b716b1794df3d665) dm-15 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active

Task 3: Host Setup


|- 13:0:5:8 sdbu 68:128 active ready running |- 13:0:6:8 sdcc 69:0 active ready running |- 14:0:5:8 sdcs 70:0 active ready running |- 13:0:4:8 sdcp 69:208 active ready running |- 14:0:6:8 sddi 71:0 active ready running |- 13:0:7:8 sddh 70:240 active ready running |- 14:0:7:8 sddq 71:128 active ready running `- 14:0:4:8 sddy 128:0 active ready runningmpathbn (36001b970b716b179b716b179453c26a3) dm-12 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 13:0:5:1 sdbn 68:16 active ready running |- 13:0:6:1 sdbv 68:144 active ready running |- 13:0:4:1 sdcd 69:16 active ready running |- 14:0:5:1 sdcf 69:48 active ready running |- 14:0:6:1 sdcv 70:48 active ready running |- 13:0:7:1 sdct 70:16 active ready running |- 14:0:7:1 sddj 71:16 active ready running `- 14:0:4:1 sddr 71:144 active ready runningmpathbm (36001b970b716b179b716b17975c58392) dm-11 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 13:0:5:4 sdbq 68:64 active ready running |- 13:0:6:4 sdby 68:192 active ready running |- 14:0:5:4 sdcl 69:144 active ready running |- 13:0:4:4 sdci 69:96 active ready running |- 14:0:6:4 sddb 70:144 active ready running |- 13:0:7:4 sdcy 70:96 active ready running |- 14:0:7:4 sddm 71:64 active ready running `- 14:0:4:4 sddu 71:192 active ready runningmpathbl (36001b970b716b179b716b1798ebafd67) dm-18 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 13:0:5:7 sdbt 68:112 active ready running |- 13:0:6:7 sdcb 68:240 active ready running |- 13:0:4:7 sdcn 69:176 active ready running |- 14:0:5:7 sdcr 69:240 active ready running |- 14:0:6:7 sddg 70:224 active ready running |- 13:0:7:7 sdde 70:192 active ready running |- 14:0:7:7 sddp 71:112 active ready running `- 14:0:4:7 sddx 71:240 active ready runningmpathbs (36001b970b716b179b716b17909052d48) dm-16 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 13:0:5:5 sdbr 68:80 active ready running |- 13:0:6:5 sdbz 68:208 active ready running |- 13:0:4:5 sdck 69:128 active ready running |- 14:0:5:5 sdco 69:192 active ready running |- 14:0:6:5 sddd 70:176 active ready running |- 13:0:7:5 sdda 70:128 active ready running |- 14:0:7:5 sddn 71:80 active ready running `- 14:0:4:5 sddv 71:208 active ready runningmpathbr (36001b970b716b179b716b179c367a45c) dm-17 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 13:0:5:6 sdbs 68:96 active ready running |- 13:0:6:6 sdca 68:224 active ready running |- 14:0:5:6 sdcq 69:224 active ready running |- 13:0:4:6 sdcm 69:160 active ready running |- 13:0:7:6 sddc 70:160 active ready running |- 14:0:6:6 sddf 70:208 active ready running |- 14:0:7:6 sddo 71:96 active ready running `- 14:0:4:6 sddw 71:224 active ready runningmpathbq (36001b970b716b179b716b179072e9a2a) dm-14 VIOLIN,SAN ARRAYsize=5.0G features='0' hwhandler='0' wp=rw`-+- policy='round-robin 0' prio=1 status=active |- 13:0:5:2 sdbo 68:32 active ready running |- 13:0:6:2 sdbw 68:160 active ready running |- 13:0:4:2 sdce 69:32 active ready running |- 14:0:5:2 sdch 69:80 active ready running |- 14:0:6:2 sdcx 70:80 active ready running |- 13:0:7:2 sdcu 70:32 active ready running |- 14:0:7:2 sddk 71:32 active ready running `- 14:0:4:2 sdds 71:160 active ready running

3. Configure multipathing on Windows hosts. Use the Windows MPIO dialog to discover and managemultipath entries for the Violin SAN array:

Configure Equipment


FIGURE 5 Windows MPIO Multipathing with the Violin Memory Array

4. Multipath configuration on VMware hosts. Change the path selection to Round Robin. This providessuperior performance to the "Most Recently Used" setting, which would use only a single active pathper LUN.

Configure Equipment


FIGURE 6 Round Robin Multipath Configuration on VMware

5. Apply any additional host tuning (for Linux systems). The following settings help support high IOPSoperation of the Violin Memory flash array with Linux hosts. Create a file named /etc/udev/rules.d/99-violin-storage.rules, and add the following lines:

# Use noop scheduler for high-performance solid-state storageACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", nimbENV{ID_VENDOR}=="VIOLIN", ATTR{queue/scheduler}="noop"# Reduce CPU overhead due to entropy collectionACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="VIOLIN", ATTR{queue/add_random}="0"# Spread CPU load by redirecting completions to originating CPUACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="VIOLIN", ATTR{queue/rq_affinity}="2"

6. Set up workload generators. On Windows and Linux systems, Medusa Labs Test Tools is installed.On VMware systems, the VMware I/O Analyzer is installed.

Configure Equipment


Test Plan

● Scope.............................................................................................................................. 17● What's New in This Report..............................................................................................17● Test Configuration...........................................................................................................18● DUT Descriptions............................................................................................................ 18● DUT Specifications..........................................................................................................19● Test Equipment............................................................................................................... 20

ScopeTesting is performed with a mix of GA and development versions of Brocade Fabric OS (FOS) in aheterogeneous environment. Test beds include Brocade directors and switches configured in routedand nonrouted fabric configurations.

Testing is centered on interoperability and optimal configuration. Performance is observed within thecontext of best-practice fabric configuration; however absolute maximum benchmark reporting ofstorage performance is beyond the scope of this test.

The storage array is connected to two SAN fabrics and multiple server hosts to drive I/O in a multipathconfiguration. Error injection is introduced, and failover and recovery behaviors are observed. I/Operformance is observed across different workload configurations.

Details of the test steps are covered in the "Test Cases” section. Standard test-bed setup includesIBM/HP/Dell chassis server hosts with Brocade/QLogic/Emulex HBAs with two uplinks from every hostto a Brocade FC fabric. I/O generators included Medusa Labs Test Tools and VMware I/O Analyzer.

What's New in This Report1. The array firmware version under test is A6.3.2/G5.6.2.2. The Brocade Fabric OS version under test is 7.4.0.3. Emulex and QLogic adapters have updated firmware and drivers.4. Two new test cases have been added:

• 1.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic Prioritization• 1.1.6 Clear Link Diagnostics (D_Port) Test

For detailed information, see the "DUT Descriptions" section.

The previous version of the test report covering Brocade Fabric OS 7.3.1 and Violin array firmwareV6.3.1.2 is available here: Brocade Fabric Technology with the Violin 6232 Validation Test Report.


http://www.brocade.com/content/html/en/validation-test-report/brocade-fabric-violin-6232-vt/index.html

Test Configuration

DUT DescriptionsThe following tables provide details about the devices under test (DUTs).

Storage Array TABLE 1

DUT ID Model Vendor Description

Violin Memory 6232 6232 Violin Memory The Violin Memory 6232 flashstorage array is an all-flash arraythat supports up to 64 MLC VIMMs(Violin Intelligent Memory Modules).The unit under test is populated with32 VIMMs. Each controller supports4x8Gb Fibre Channel connections.

Switches TABLE 2

DUT ID Model Vendor Description

6510-1...9 BR-6510 Brocade 48-port 16-Gb FC switch

Test Configuration


Switches (Continued)TABLE 2

DCX-1 DCX 8510-8 Brocade 8-slot 16-Gb FC chassis

DCX-2 DCX 8510-4 Brocade 4-slot 16-Gb FC chassis

VDX-1,2 VDX 6740 Brocade 48-port 10-Gb switch(48x10Gb/4x40Gb)

DUT Specifications

Storage Version

Violin Memory 6232 array A6.3.2/G5.6.2

Brocade Switches Version

DCX 8510-8 Fabric OS 7.4.0

DCX 8510-4 Fabric OS 7.4.0

6510 + Integrated Routing, Fabric Vision licenses Fabric OS 7.4.0

VDX 6740 Network OS 5.0.1

Adapters Version

Brocade 1860 2-port 16-Gb FC HBA Driver and firmware: 3.2.5.1

QLogic QLE2672 2-port 16-GB FC HBA Driver 8.07.00.16, firmware 7.04.01

QLogic QLE2562 2-port 8-GB FC HBA Driver 9.1.11.28, firmware 7.04.00

Emulex LPe 16202-X 2-port 16-Gb FC HBA Driver 10.4.246.0, firmware 10.4.255.23

Brocade 1020 2-port CNA adapter Driver and firmware: 3.2.5.1

DUT ID Servers RAM Processor OS

SRV-1 HP ProLiant DL380PG8

160 GB Intel Xeon E5-2640 VMware 5.5 [cluster]


160 GB Intel Xeon E5-2640 VMware 5.5 [cluster]

SRV-3 IBM System x3630M4

24 GB Intel Xeon E5-2420 Windows Server 2012R2

DUT Specifications


SRV-4 Dell PowerEdgeR720

160 GB Intel Xeon E5-2640 RHEL 7.0 x86_64

SRV-5 HP ProLiant DL385pG8

16 GB AMD Opteron 6212 Windows Server 2012

SRV-6 Dell PowerEdgeR720

16 GB Intel Xeon E5-2620 RHEL 6.5 x86_64


32 GB Intel Xeon E5-2690v2 Windows Server 2012R2

SRV-8 IBM System x3630M4

16 GB Intel Xeon E5-2620 SLES 11.3 x86_64

Test Equipment

Device/Software Tools Version

Finisar 16-Gb Analyzer/Jammer XGIG5K2001153

Medusa Labs Test Tools 6.0.1.156759

VMware I/O Analyzer 1.6.0

Test Equipment


Test Cases

1.0 Fabric Initialization—Base Functionality

Confirm basic Fibre Channel functionality of the storage array.

1.0.1 Storage Device—Physical and Logical Login with Speed Negotiation

1.0.2 Zoning and LUN Mapping

1.0.3 Storage Device Fabric I/O Integrity

1.0.4 Storage Device Multipath Configuration—Path Integrity

1.1 Fabric—Advanced Functionality

Examine the storage behavior related to more advanced fabric features such as QoS, bottleneck detection,and advanced frame recovery.

1.1.1 Storage Device Bottleneck Detection Using MAPS FPI—With Congested Hosts

1.1.2 Storage Device Bottleneck Detection Using MAPS FPI—With Congested Fabric

1.1.3 Storage Device—QoS Integrity with QoS Zone-Based Traffic Prioritization

1.1.4 Storage Device—QoS Integrity with CS_CTL-Based Traffic Prioritization

1.1.5 Storage Device—FC Protocol Jammer Test Suite

1.1.6 Storage Device—ClearLink Diagnostics (D_Port) Test

1.2 Stress and Error Recovery with Device Multipath

Confirm proper HA/failover behavior of the storage in a multipath environment.

1.2.1 Storage Device Fabric I/O Integrity—Congested Fabric

1.2.2 Storage Device Nameserver Integrity—Device Recovery with Port Toggle

1.2.3 Storage Device Nameserver Integrity—Device Recovery with Device Relocation

1.2.4 Storage Device—Device Recovery with Device Port Toggle—Extended Run

1.2.5 Storage Device Recovery—ISL Port Toggle

1.2.6 Storage Device Recovery—ISL Port Toggle (Entire Switch)

1.2.7 Storage Device Recovery—Director Blade Maintenance

1.2.8 Storage Device Recovery—Switch Offline

1.2.9 Storage Device Recovery—Switch Firmware Download


1.3 Storage Device—Fibre Channel Routing Internetworking Tests

Confirm proper storage functioning within routed fabrics.

1.3.1 Storage Device Internetworking Validation with the FC Host

1.3.2 Storage Device Internetworking Validation with FCoE Test Using VDX FlexPort

1.3.3 Storage Device Edge Recovery After FCR Disruptions

1.3.4 Storage Device Backbone Recovery After FCR Disruptions

1.4 Optional/Additional Tests

1.4.1 Performance I/O Testing with Varying Workload Dimensions

1.4.2 VMware I/O Analyzer Workload Test


1.0.1 Storage Device—Physical and Logical Login with SpeedNegotiation

Test ObjectiveVerify device login to the switch and nameserver with all supported speed settings.

ProcedureTest Execution

1. Set switch ports to 2/4/8/Auto_Negotiate speed settings.

portcfgspeed <port> [2/4/8/0]Result Validation

1. Validate link states on the array, and verify speed negotiation and device login at different speeds.Use the portshow command on the switch to check the link state and speed, and use thenscamshow command to verify device login to the fabric.

2. Check the switch port, status and verify the “actual” and “configured” link speed. Check the nameserver for device login.

# nscamshow# portshow Xroot> portshow 19portIndex: 19portName: port19portHealth: HEALTHYportState: 1 Online



Protocol: FCportWwn of device(s) connected: 10:00:8c:7c:ff:22:f7:81Distance: normalportSpeed: N16Gbps <-

ResultPASS. Storage logs into the fabric and is link up at 2Gb/4Gb/8Gb. Run I/O to verify.


Test ObjectiveVerify that host-to-LUN access exists with valid zoning.


1. Create an FC zone on the fabric with the initiator and target WWNs.2. Create host groups and LUNs on the array with access to the initiator WWN.

Result Validation

Verify that LUNs are discovered on the hosts using host-specific tools.

- Linux: Check the output of lsscsi- Windows: Check the output at Computer Management > Storage > Disk Management- VMware: Check the output at Configuration > Storage > Devices

ResultPASS. For each host, create a zone that contains four storage ports and two host ports. LUNs arepresented to the host as verified with I/O.

1.0.3 Storage Device Fabric I/O Integrity

Test ObjectiveValidate single-path host-to-LUN I/O with read/write/verify testing.


1. Set up read/write I/O to the LUN using Medusa.2. Perform link disruptions with port toggles/cable pulls.

Result Validation

Check Medusa I/O logs, and verify that I/O resumes after a short downtime. Medusa I/O may pause, butshould recover without errors.



ResultPASS. I/O resumes without errors.


Test ObjectiveVerify that multipath configures successfully, and verify the integrity of each path.


1. Set up the host with at least two initiator ports zoned with two target ports on the array.2. Set up multipath on the host.3. Start I/O.4. Isolate paths individually, and run I/O on the separate paths.

Result Validation

1. Check host multipath properties to verify that the toggled path recovers.

- Windows: mpclaim –s –d - Linux: multipath –ll- VMware: Check the paths at Configuration > Storage > Devices > Manage Paths

2. Check the host and storage logs for any failures.3. Check the switch error logs and switch port status after toggling.

# errdumpall# portstatsshow X# portshow Xroot> portshow 19portHealth: HEALTHY <-. . .portState: 1 Online <-. . .portWwn of device(s) connected: 10:00:8c:7c:ff:22:f7:81 <-Distance: normalportSpeed: N16Gbps

4. Check I/O logs, and verify that I/O continues without any errors.

ResultPASS. Each individual path is valid, and the paths combine successfully in a multipath configuration.

1.1 Fabric—Advanced Functionality



1.1.1 Storage Device Bottleneck Detection Using MAPS FPI—WithCongested Hosts

Test ObjectiveVerify that congestion on host ports is detected. Verify storage device and fabric behavior duringcongestion.


1. Configure MAPS and FPI on all switches. (For more information, see the “Configure Equipment”section.)

2. Start I/O from a single host initiator to multiple targets.3. Monitor the switch logs for IO_PERF_IMPACT/IO_FRAME_LOSS warnings.

Result Validation

Check the switch error logs and the MAPS dashboard for bottleneck warnings.

# errdumpall | grep IO_root> errdumpall | grep IO_2014/12/17-11:56:00:672622, [MAPS-1003], 117148/115018, FID 128, WARNING, B6510_066_088, Port 16, Condition=ALL_F_PORTS(DEV_LATENCY_IMPACT==IO_PERF_IMPACT), Current Value:[DEV_LATENCY_IMPACT,IO_PERF_IMPACT, 30.0% in 10 secs], RuleName=defALL_F_PORTS_IO_PERF_IMPACT, Dashboard Category=Fabric Performance Impact., actionHndlr.c, line: 755, comp:md, ltime:2014/12/17-11:56:00:671909# mapsdb --show allroot> mapsdb --show1 Dashboard Information:========================Active policy: dflt_aggressive_policy. . .2 Switch Health Report:=======================Current Switch Policy Status: HEALTHY3.1 Summary Report:===================Category |Today |Last 7 days |--------------------------------------------------------------------------------Port Health |No Errors |Out of operating range |Fru Health |In operating range |In operating range |Security Violations |No Errors |In operating range |Fabric State Changes |No Errors |In operating range |Switch Resource |In operating range |In operating range |Traffic Performance |In operating range |In operating range |FCIP Health |Not applicable |Not applicable |Fabric Performance Impact|Out of operating range |Out of operating range |

ResultPASS. With MAPS reporting configured, performance warnings are reported as expected. Thebottlenecked ports are displayed on the MAPS dashboard, and a RASLog warning is created.

1.1.1 Storage Device Bottleneck Detection Using MAPS FPI—With Congested Hosts


1.1.2 Storage Device Bottleneck Detection Using MAPS FPI—WithCongested Fabric

Test ObjectiveValidate bottleneck detection with congested fabric.


1. Configure MAPS on all switches. (For more information, see the “Configure Equipment” section.)2. Isolate a single ISL in the fabric.3. Start I/O from multiple host initiators to multiple targets.4. Monitor switch logs for traffic performance warnings.

Result Validation

Check the switch error logs and the MAPS dashboard for bottleneck warnings.

# errdumpall | grep Traffic2014/12/18-08:39:03:497512, [MAPS-1003], 9637/5496, FID 128, WARNING, B6510_066_082, Port 33, Condition=ALL_TARGET_PORTS(RX/hour>60.00), Current Value:[RX,65.16 %], RuleName=defALL_TARGET_PORTSRX_60, Dashboard Category=Traffic Performance., actionHndlr.c, line: 755, comp:md, ltime:2014/12/18-08:39:03:496754# mapsdb --show allroot> mapsdb --show1 Dashboard Information:========================Active policy: dflt_aggressive_policy. . .2 Switch Health Report:=======================Current Switch Policy Status: HEALTHY3.1 Summary Report:===================Category |Today |Last 7 days |--------------------------------------------------------------------------------Port Health |No Errors |Out of operating range |Fru Health |In operating range |In operating range |Security Violations |No Errors |In operating range |Fabric State Changes |No Errors |In operating range |Switch Resource |In operating range |In operating range |Traffic Performance |In operating range |In operating range |FCIP Health |Not applicable |Not applicable |Fabric Performance Impact|Out of operating range |Out of operating range |

ResultsPASS. With MAPS reporting configured, performance warnings are reported as expected.

1.1.2 Storage Device Bottleneck Detection Using MAPS FPI—With Congested Fabric


1.1.3 Storage Device—QoS Integrity with QoS Zone-Based TrafficPrioritization

Test ObjectiveValidate QoS functionality.


1. Set up initiator-target pairs with Low/Medium/High QoS zones in the fabric.2. Start I/O across all pairs, and validate traffic priority.

Result Validation

1. Check I/O logs, and verify that I/O continues without errors.2. Check the switch error logs and the switch port status for errors.

# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

3. Verify traffic prioritization by examining performance and VC utilization.

ResultPASS. I/O completes without errors in low, medium, and high priority zones.

1.1.4 Storage Device—QoS Integrity with CS_CTL-Based TrafficPrioritization

Test ObjectiveValidate CS_CTL QoS functionality.


1. Enable ExpressLane on the Emulex adapter driver by setting the value to 1 and the priority to 3(high).

1.1.3 Storage Device—QoS Integrity with QoS Zone-Based Traffic Prioritization


FIGURE 7 Configuring Driver Parameters in the Emulex OneCommand Interface

2. Enable CS_CTL QoS Auto-Mode at the chassis level on all switches in the Brocade fabric.

> configurechassisConfigure...cfgload attributes (yes, y, no, n): [no]Custom attributes (yes, y, no, n): [no] system attributes (yes, y, no, n): [no] fos attributes (yes, y, no, n): [no] y Reboot needed to effect new CSCTL Mode CSCTL QoS Mode (0 = default; 1 = auto mode): (0..1) [0] 1To verify:B6510_066_073:root> configshow -all | grep csctlfos.csctlMode:1

3. Enable CS_CTL QoS on initiator and target ports.

B6510_066_073:root> portcfgqos --enable 7 csctl_modeEnabling CSCTL mode flows causes QoS zone flows to lose priority on such ports.Do you want to proceed?(y/n):yTo verify:

Test Cases


B6510_066_073:root> portcfgshow 7 | grep -i csctlCSCTL mode: ON

4. Enable ExpressLane QoS on selected LUNs. For multipath LUNs, enable ExpressLane on eachdevice path.

FIGURE 8 Enabling LUNs for ExpressLane in the Emulex OneCommand Interface

5. Start high throughput I/O to all LUNs.

Result Validation

1. Check I/O logs, and verify that I/O continues without errors for all LUNs.2. Verify that I/O performance improves on ExpressLane-enabled LUNs.3. Verify CS_CTL prioritization in the fabric by monitoring the high VC buffer credits on the ISLs.4. Check the switch error logs and switch port status for errors. # porterrshow

Test Cases


root> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ResultPASS. Prioritized LUNs display higher performance, the traffic receives prioritization through thefabric, and I/O completes without errors.


Test ObjectivePerform FC Jammer tests including CRC corruption, packet corruption, missing frames, host errorrecovery, and target error recovery.


1. Insert the Jammer device in the I/O path on the storage link.2. Execute the following Jammer scenarios:

• Delete one frame.• Delete R_RDY.• Replace CRC of data frame.• Replace EOF of data frame.• Replace “good status” with “check condition.”• Replace IDLE with LR.• Truncate frame.• Create S_ID/D_ID error of data frame.

3. Verify Jammer operations and recovery using Analyzer.

Result Validation

Check the Medusa log, and verify that I/O recovers and completes without errors.

ResultPASS. Host and target are able to recover from the errors and continue I/O.



1.1.6 Clear Link Diagnostics (D_Port) Test

Test ObjectiveValidate the Clear Link Diagnostics feature on supported HBAs. The feature is tested on EmulexLPe16202, QLogic 2672, and QLogic/Brocade 1860.


1. The Brocade switch can be configured in static, dynamic, and on-demand D_Port modes. The switchsupports dynamic D_Port mode by default at the chassis level.

root> configureConfigure... Fabric parameters (yes, y, no, n): [no] D-Port Parameters (yes, y, no, n): [no] y Dynamic D-Port (on, off): [on]On Demand D-Port (on, off): [off]

2. With the Emulex LPe16202 adapter, the switch port is in dynamic mode and the test is manuallyinitiated on the HBA port from the Emulex OneCommand Manager.

root> switchshow0 0 030000 id N16 In_Sync FC D-Port 10:00:00:90:fa:61:92:3b Dynamic

3. With the QLogic 2672 adapter, the HBA port is in dynamic D_Port mode and the test is initiated byconfiguring the switch port as a static D_Port. The test starts automatically after enabling the D_Portat the switch.

> portdisable <port#>> portcfgdport ––enable <port#>Caution: D_Port functionality is only available on 16Gb-capable platforms with

1.1.6 Clear Link Diagnostics (D_Port) Test


16Gb FC SFPs, 10Gb FC SFPs, 8Gb LWL/ELWL FC SFPs, QSFPs or QSFP+.> portenable <port#>(reverse the above steps to disable D_Port mode)To verify:root> portcfgshow <port#> | grep -i d-portD-Port mode: ONroot> switchshow7 7 581300 id N16 Online FC D-Port Loopback->Port 7

4. With the QLogic/Brocade 1860 adapter, the HBA port can be in static or dynamic D_Port mode.

Enabling static D_Port on the HBA:# bcu port --disable 2/0port disabled# bcu diag --dportenable 2/0D-port mode for port 2/0 enabled.

5. Execute the D_Port test, and verify that all tests pass and that no port and link problems arereported.

Result Validation

Check the D_Port test results on the switch ports and host HBA diagnostic utilities.

root> portdporttest --show 14D-Port Information:===================Port: 14Remote WWPN: 10:00:8c:7c:ff:23:b7:00Mode: AutomaticNo. of test frames: 1 MillionTest frame size: 1024 BytesFEC (enabled/option/active): Yes/No/NoCR (enabled/option/active): No/No/NoStart time: Wed Apr 15 12:02:53 2015End time: Wed Apr 15 12:03:52 2015Status: PASSED================================================================================Test Start time Result EST(HH:MM:SS) Comments================================================================================Electrical loopback 12:03:01 PASSED -------- --------Optical loopback 12:03:31 PASSED -------- --------Link traffic test 12:03:39 PASSED -------- --------================================================================================Roundtrip link latency: 157 nano-secondsEstimated cable distance: 3 metersBuffers required: 1 (for 2112 byte frames at 16Gbps speed)Egress power: Tx: -3.4 dBm, Rx: Not Avail.Ingress power: Rx: -3.3 dBm, Tx: Not Avail.

ResultPASS. The D_Port feature is validated on all tested HBAs.





Test ObjectiveValidate I/O integrity in a congested fabric environment.


From all initiators, start a mixture of read, read/write, and write traffic continuously to all their targets fora 24-hour period.

Result Validation

1. Check the host and storage logs for errors.2. Verify link congestion, and check the switch logs for errors.

# errdumpall# portperfshow# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

3. Check I/O generator tool logs to verify that I/O runs without errors.

ResultPASS. In a congested fabric, I/O runs successfully without errors or failover.

1.2.2 Storage Device Nameserver Integrity—Device Recovery with PortToggle

Test ObjectiveValidate path recovery behavior on storage and host ports.


1. Set up multipath on the host, and start I/O.2. Perform multiple iterations of sequential port toggles across initiator and target switch ports.

Result Validation



1. Check switch port status after toggling, and check for errors in the switch logs.


2. Check the host multipath status on hosts to verify that the toggled path recovers.


3. Check host and storage error logs, and verify that I/O continues without errors.

ResultPASS. Failover between 8 logical paths (2 host x 4 storage) tests successfully. Paths recover, and I/Ocompletes without errors.

1.2.3 Storage Device Nameserver Integrity—Device Recovery withDevice Relocation

Test ObjectiveValidate storage device path recovery after physical port relocation.


1. Perform the test sequentially for each storage device port.2. Disconnect the port and reconnect it to a different switch port in the same fabric.

Result Validation

1. Check for errors in the switch logs, and check the status of the new switch port.

# errdumpall# portstatsshow X# portshow Xroot> portshow 8portHealth: HEALTHY <-. . .portState: 1 Online <-. . .portWwn of device(s) connected: 52:4a:93:7d:f3:5f:61:11 <-

1.2.3 Storage Device Nameserver Integrity—Device Recovery with Device Relocation


Distance: normalportSpeed: N8Gbps

2. Check the multipath status on hosts to verify that the toggled path recovers.


3. Confirm that there are no errors in the Medusa log.

ResultPASS. The storage port successfully recovers after being physically moved. Paths recover, and I/Ocompletes without errors.


Test ObjectiveValidate path recovery behavior on storage and host ports for an extended duration.


1. Set up multipath on the host, and start I/O.2. Sequentially toggle each initiator and target port in the fabric (multiple iterations).3. Run the test for an extended period.

Result Validation

1. Check the switch port status after toggling, and check for errors in the switch logs.


2. Check the multipath properties for iSCSI hosts to verify that the toggled path recovers.

On Windows: mpclaim –s –d On Linux: multipath –llOn VMware: Check the paths at Configuration > Storage > Devices > Manage Paths

3. Check host and storage logs, and verify that I/O continues without errors.

ResultPASS. 48-hour run: paths recover, and I/O completes without errors.




Test ObjectiveValidate path recovery and I/O integrity when ISL links are disabled.


1. Set up multipath on the host, and start I/O.2. Sequentially toggle each ISL path (one at a time) on all switches.

Result Validation

1. Check the FC fabric status after ISL toggles. Verify that all nodes are online.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check the switch logs for errors, and verify that I/O fails over to an alternate ISL path in the fabric.


3. Check the host and storage logs, and verify that I/O continues without errors.

ResultPASS. Paths recover, and I/O completes without errors.

1.2.6 Storage Device Recovery—ISL Port Toggle (Entire Switch)

Test ObjectiveValidate path recovery and I/O integrity when all ISL links on a switch are disabled.




1. Ensure ISL redundancy by provisioning multiple ISLs connected to different switches to providemultiple paths through the fabric.

2. Set up multipath on the host, and start I/O.3. On each switch, disable all ISL links simultaneously.

Result Validation

1. Check the FC fabric status after ISL toggles. Verify that all nodes are online.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------------ 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR






Test ObjectiveValidate path recovery and I/O integrity during director blade maintenance.




1. Uplink edge-switch ISLs to different blades on the directors.2. Sequentially toggle each director blade.3. Include blade disable/enable and blade power on/off.

Syntax:

slotpoweroff/slotpoweron, bladedisable/bladeenable, slotshowResult Validation

1. Check the FC fabric status after the blade toggles. Verify that all nodes are present in the fabric.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name--------------------------------------------------------------------------------- 1: fffc01 50:00:53:39:47:bd:6e:0a 0.0.0.0 0.0.0.0 "fcr_fd_1" 2: fffc02 10:00:00:05:1e:09:15:3e 10.38.51.5 0.0.0.0 "sw5300sus1005" 3: fffc03 10:00:c4:f5:7c:07:c2:0c 10.38.51.3 0.0.0.0 "sw300sus1003" 4: fffc04 10:00:c4:f5:7c:7b:86:1f 10.38.51.4 0.0.0.0 "sw6510sus1004" 5: fffc05 10:00:00:05:33:13:80:ef 10.38.51.24 0.0.0.0 "swBESsus1024" 13: fffc0d 10:00:00:05:1e:4c:ff:00 10.38.51.13 0.0.0.0 "sw8510sus1013" 16: fffc10 10:00:00:05:1e:44:02:00 10.38.51.16 0.0.0.0 >"sw8518sus1016" 22: fffc16 10:00:00:05:1e:07:7d:99 10.38.51.22 0.0.0.0 "sw5100sus1022" 23: fffc17 10:00:00:27:f8:2b:9f:aa 10.38.51.23 0.0.0.0 "sw6520sus1023" 25: fffc19 10:00:00:05:1e:54:8b:d4 10.38.51.25 0.0.0.0 "swBESsus1025" 26: fffc1a 10:00:00:05:33:8f:b6:77 10.38.51.26 0.0.0.0 "sw6510sus1026" 29: fffc1d 10:00:00:05:33:83:a4:00 10.38.51.29 0.0.0.0 "swDCX4sus1029" 32: fffc20 10:00:00:05:1e:b8:c1:00 10.38.51.32 0.0.0.0 "swDCXsus1032"160: fffca0 10:00:00:05:1e:a8:35:9d 10.38.51.21 0.0.0.0 "swVA40FCsus1021"171: fffcab 10:00:00:05:1e:d7:1a:04 10.38.51.171 0.0.0.0 "sw5480sus1171" 2620:100:4:fa00:205:1eff:fed7:1a04183: fffcb7 10:00:00:05:33:00:b2:e2 10.38.51.183 0.0.0.0 "swM5424sus1183"195: fffcc3 10:00:00:05:33:8d:df:d1 10.38.51.195 0.0.0.0 "sw5480sus1195"199: fffcc7 10:00:00:27:f8:44:33:61 10.38.51.199 0.0.0.0 "sw6547sus1199" 2620:100:4:fa00:227:f8ff:fe44:3362220: fffcdc 10:00:00:05:1e:86:1b:1f 10.38.51.220 0.0.0.0 "sw5450sus1220"The Fabric has 19 switches





Test Cases



Test ObjectiveValidate path recovery and I/O integrity during switch offline events.


1. Toggle each switch in sequential order.2. Include switch enable/disable, power on/off, and reboot testing.

Result Validation

1. Check the FC fabric status after the switch toggles. Verify that all nodes are present in the fabric.

# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check the switch logs for errors, and verify that the toggled switch recovers.

# errdumpall# switchshowroot> switchshowswitchName: B6510_066_088switchType: 109.1switchState: Online <-switchMode: NativeswitchRole: SubordinateswitchDomain: 88switchId: fffc58switchWwn: 10:00:00:27:f8:06:23:28zoning: ON (SSR)switchBeacon: OFFFC Router: ONFC Router BB Fabric ID: 100Address Mode: 0Fabric Name: SSR_2


ResultPASS. Paths fail over and recover, and I/O completes without errors.

1.2.9 Storage Device Recovery—Switch Firmware Download

Test ObjectiveVerify that I/O continues with minimal disruption throughout the switch firmware upgrade process.




1. Set up host multipath with links on different switches in the FC fabric, and start I/O.2. Sequentially perform firmware upgrades on all switches in the fabric.

Result Validation

1. Verify that the firmware upgrade completes successfully on each switch node and that the nodesmerge back into the FC fabric.

# versionroot> versionKernel: 2.6.14.2Fabric OS: v7.3.1 <-Made on: Thu Dec 11 14:30:38 2014Flash: Wed Dec 17 09:53:59 2014BootProm: 1.0.11# fabricshowroot> fabricshowSwitch ID Worldwide Name Enet IP Addr FC IP Addr Name------------------------------------------------------------------------------- 1: fffc01 50:00:53:35:b1:d3:df:1b 0.0.0.0 0.0.0.0 "fcr_xd_1_40" 3: fffc03 10:00:00:05:33:13:95:9a 10.38.66.73 0.0.0.0 "B6510_066_073" 19: fffc13 10:00:00:05:33:a5:bf:86 10.38.66.74 0.0.0.0 >"B6510_066_074" 82: fffc52 10:00:00:05:33:13:96:5a 10.38.66.82 0.0.0.0 "B6510_066_082" 83: fffc53 10:00:00:05:33:5b:1d:1d 10.38.66.83 0.0.0.0 "B6510_066_083"The Fabric has 5 switches <-Fabric Name: SSR

2. Check I/O generator tool logs to verify that I/O runs without errors throughout the firmware upgrade.3. Check the switch logs for errors, and verify that I/O resumes on the node after the firmware upgrade

is complete.

# errdumpall# portperfshow

ResultPASS. I/O fails over and recovers during the firmware download. Paths recover, and I/O completeswithout errors.


1.3.1 Storage Device Internetworking Validation with the FC Host

Test ObjectiveValidate that storage targets are imported successfully in a routed Fibre Channel environment.




1. Set up FCR in an edge-backbone-edge configuration.2. Set up LSAN zoning, verify host access to target LUNs, and start I/O.

Result Validation

1. Verify the name server and FCR fabric state.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------------------- 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92"FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------------------- 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported. . . 100 50:05:07:60:5e:80:76:52 01fe02 40 612800 Imported 100 50:05:07:60:5e:80:76:71 01ff02 40 642300 ImportedTotal devices displayed: 20

2. Verify that I/O runs successfully without errors.

ResultPASS. All devices are available via the routed fabric. Running I/O confirms successful routing.

1.3.2 Storage Device Internetworking Validation with FCoE Test UsingVDX FlexPort

Test ObjectiveVerify storage device availability from FCoE initiators routed through the VCS fabric.


1. Set up FCoE host initiator ports on the VCS fabric using Brocade VDX 6740 switches.2. Enable Fibre Channel support on the VDX switch ports (for more details, see the "Configure

Equipment" section).3. Connect the VCS fabric to the FC fabric via FCR ports.4. Configure LSAN zones in the VCS and FC fabrics.

1.3.2 Storage Device Internetworking Validation with FCoE Test Using VDX FlexPort


Result Validation

1. Verify the state of the name server and FCR fabric.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:5b:7d:06, Dom ID: 97,Info: 10.38.66.97, "B6510_066_097" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------------- 18 110 10.38.66.120 10:00:50:eb:1a:62:8c:33 "VDX6740_066_120"FC Router WWN: 10:00:00:05:33:5b:10:e8, Dom ID: 100,Info: 10.38.66.100, "B6510_066_100" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ------------------------------------------------------------------------------- 8 110 10.38.66.119 10:00:50:eb:1a:62:83:7b "VDX6740_066_119"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 110 50:05:07:60:5e:80:76:52 02f001 120 612800 Imported 110 50:05:07:60:5e:80:76:71 02f101 120 642300 Imported 120 10:00:8c:7c:ff:00:48:00 03f101 110 781000 Imported 120 10:00:8c:7c:ff:00:48:01 03f001 110 771000 ImportedTotal devices displayed: 4

2. Verify that I/O runs successfully without errors.

ResultPASS. Storage targets are available through the FCoE/FC routed configuration; I/O completes withouterrors.


Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channelenvironment. Edge-backbone-edge configuration.


1. Set up FCR in an edge-backbone-edge configuration.2. Set up LSAN zoning.3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches

in the backbone fabric.

Result Validation

1. Verify the FCR fabric state throughout the disruptions.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) -----------------------------------------------------------------------------



40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92"FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------------------- 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported. . . 100 50:05:07:60:5e:80:76:52 01fe02 40 612800 Imported 100 50:05:07:60:5e:80:76:71 01ff02 40 642300 ImportedTotal devices displayed: 20

2. Check the switch logs for errors.


3. Check the host and storage logs, and verify that I/O runs without errors.

ResultPASS. Paths fail over and recover from FCR disruptions, and I/O completes without errors.


Test ObjectiveValidate that storage target paths recover successfully from disruptions in a routed Fibre Channelenvironment. Edge-backbone configuration.


1. Set up FCR in an edge-backbone configuration.2. Set up LSAN zoning.3. With I/O running, perform sequential reboots, switch disables, and ISL port toggles on the switches in

the backbone fabric.

Result Validation

1. Verify the FCR fabric state throughout the disruptions.

# fcrfabricshowroot> fcrfabricshowFC Router WWN: 10:00:00:05:33:13:96:5a, Dom ID: 82,Info: 10.38.66.82, "B6510_066_082" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ----------------------------------------------------------------------------- 40 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088" 41 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92"FC Router WWN: 10:00:00:05:33:5b:1d:1d, Dom ID: 83,



Info: 10.38.66.83, "B6510_066_083" EX_Port FID Neighbor Switch Info (enet IP, WWN, name) ---------------------------------------------------------------------------- 40 40 10.38.66.92 10:00:00:27:f8:66:f3:81 "B6520_066_92" 41 40 10.38.66.88 10:00:00:27:f8:06:23:28 "B6510_066_088"# fcrproxydevshowroot> fcrproxydevshow Proxy WWN Proxy Device Physical State Created PID Exists PIDin Fabric in Fabric------------------------------------------------------------------------ 40 10:00:00:05:1e:60:b4:6b 02ff02 100 132400 Imported 40 10:00:00:05:1e:60:b4:6c 02fe02 100 032400 Imported. . . 100 50:05:07:60:5e:80:76:52 01fe02 40 612800 Imported 100 50:05:07:60:5e:80:76:71 01ff02 40 642300 ImportedTotal devices displayed: 20

2. Check the switch logs for errors.


3. Check host and storage logs, and verify that I/O runs without errors.

ResultPASS. Paths fail over successfully, and I/O completes without errors.


1.4.1 Performance I/O Testing with Varying Workload Dimensions

Test ObjectiveRun I/O workload generation with varying dimensions (block size, read/write/mix), and verify thatperformance characteristics are as expected.


1. Run an I/O loop at block transfer sizes of 512, 1k, 2k, 4k, 8k, 16k, 32k, 64k, 128k, 256k, 512k, and1m.

2. Include a nested loop of 100-percent read, 100-percent write, and 50-percent read/write.

Repeat the test for the following configurations:

• One host port to one target port (single path)• Two host ports to multiple target ports (multipath)• Multiple hosts to multiple target ports (multihost multipath)

Result Validation



1. Check the I/O generator tool logs to verify that I/O completes without errors.2. Check the host and storage logs for errors throughout the I/O operations.3. Check the switch logs and port stats for errors or I/O drops.

# errdumpall# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ResultPASS. All workload runs are monitored at the host, storage, and fabric and complete without I/O errorsor faults. Performance behavior is as expected.


Test ObjectiveRun the workload test suite including varying application workloads generated from multiple VMs.


1. Configure a two-host VMware cluster with multipath on two initiator ports per host and four targetports.

2. Configure workload generation from eight worker VMs using the VMware I/O Analyzer.3. Run a variety of application I/O workload patterns from the I/O Analyzer suite. For this round of

testing, the following workloads are used:

• Workstation• Webserver• Video on Demand• Max Write Throughput• Max Write IOPS• Max IOPS• Max Throughput• SQL Server 16k• 4k Read• Exchange 2007• OLTP 4k

Result Validation



1. Check the I/O generator tools logs to verify that I/O completes without errors.2. Check the host and storage logs for errors throughout the I/O operations.3. Check the switch logs and port stats for errors or I/O drops.

# errdumpall# porterrshowroot> porterrshow frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout pcs tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx err 0: 1.7g 2.7g 0 0 0 0 0 0 0 946 1 0 1 0 0 0 0 0 1: 431.1m 431.4m 0 0 0 0 0 0 0 30 1 0 1 0 0 0 0 0 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ResultPASS. All workload runs are monitored at the host, storage, and fabric and complete without I/O errorsor faults.

Test Cases


Test Conclusions

1. A pass rate of 100 percent was achieved on all test cases in the SSR qualification test plan. Thenetwork and the storage were able to handle the various stress and error recovery scenarios withoutany issues.

2. Different I/O workload scenarios were simulated using Medusa and VMware I/O Analyzer tools, andsustained performance levels were demonstrated across all workload types.

3. The results confirm that the Violin Memory 6232 array interoperates seamlessly with Brocade FibreChannel fabrics, and together they demonstrate high availability, performance, and low latency.

4. The Brocade Gen5 16-Gb FC switches handled the sustained throughput and latency performancerequirements efficiently with fewer ISL trunks. Multiple ISLs to different switches in the fabric shouldbe set up to provide path redundancy through the fabric.

5. The Monitoring and Alerting Policy Suite (MAPS) heath monitor should be enabled on all switches inthe FC fabric to report fabric-wide events and traffic performance metrics. The additional MAPSfeature of Fabric Performance Impact monitoring should also be enabled to detect bottlenecks in theform of timeouts and latency. Using MAPS is recommended to maximize the benefit of high-performance low-latency storage.

6. Peer zoning helps reduce the zone database size and the zoning complexity, while providing theRSCN and hardware resource efficiencies of single-initiator zoning.

7. QoS zoning should be used to classify host-target traffic into high, medium, or low priority zones andto provide traffic prioritization through the FC fabric for the desired host-target pair by allocating moreresources to the traffic in the higher priority zone.

8. Enabling Emulex ExpressLane on a LUN provides prioritized queuing on the HBA for traffic to thatLUN and also sets the CS_CTL tag on the frame, which allows the traffic to be prioritized through theFC fabric based on the value of the CS_CTL tag and the corresponding priority level.

9. For optimal availability and performance, consideration should be given to multipath configuration onthe host side. Although Windows 2008 and 2012 provide Round-Robin behavior by default, Linuxsystems benefit from adding a custom entry to /etc/multipath.conf, and VMware hosts systemsshould be changed from the default "Most Recently Used (VMware)" setting to "Round-Robin(VMware)". Actively using all available paths provides a significant improvement in performancethroughput.

10.The ClearLink D_Port diagnostic test is a valuable tool for validating link quality before adding newdevices or ISLs to the fabric and for troubleshooting existing links.
