fc tom keiser

7/31/2019 Fc Tom Keiser

1/60

Fibre ChannelOr, why iSCSI is a BAD IDEA

Tom Keiser

[email protected]

Penn State Linux Users Group

Fibre Channel p.1/6


2/60

Overview

Historical Background

What is Fibre Channel?

FC Hardware

Topologies and Addressing

The FC Protocol

How fast is it?Why is it better than iSCSI?

Fibre Channel p.2/6


3/60

History of Storage Networks

Storage networks evolved for several reasons

To abstract the complexity of storagetechnologies from the host system

To allow multiple systems to share a diskcabinet or array

They were a necessary step in the

development of cluster filesystems forVMS, OS/390, etc.

More recently, they have permitted data

centers to place storage outside ofmetropolitan areas in specialized storagecenters Fibre Channel p.3/6


4/60

History of Storage Networks

DEC developed a precursor to the SAN for its

VMS clusters using the DSSI and CIinterconnects

IBM developed SSA for its RS/6000 systems

The SSA protocol stack is much simplerthan FC

The lowest layers of the SSA stack are astreamlined derivative of FC

SSA is application-specific; FC is generic

shared SCSI is common in small installations

Fibre Channel p.4/6


5/60

History of Fibre Channel

Architecture developed by Sun Microsystems

as a part of Project StoreXProtocol developed by IBM

IBM decided FC was too complex for manyapplications

SSA was IBMs simplified solution

Fibre originally came in several speedsranging from 125Mbps to 1062.5Mbps

Early fibre installations were private loops

Fibre Channel p.5/6


6/60

Fibre Channel

The Basics

Fibre Channel p.6/6


7/60


A high-performance serial storage network

Current implementations run at 1 and 2 Gbps

4 and 8 Gbps implementations are planned

Common Physical Cabling:Copper cabling with HSSDC, HSSDC2, orDB9 connectors

Single-mode or multi-mode fiber with SC orLC connectors

Fibre Channel p.7/6


8/60


FC, like most major data link layer protocols,

encapsulates common transport protocolsSCSI

IP

VI (virtual interface protocol used inclusters)

HIPPIESCON

Fibre Channel p.8/6


9/60

FC Hardware Overview

Host Bus Adapters (HBAs)

Interface PCI, SBUS, XIO64, etc. with FCIntegrate SRAM frame buffers and RISCprocessors to offload framing,segmentation, etc.

Hubs

Make a logical ring topology physicallyappear as a star topology

Automatically detect failed or unplugged

devices and route the ring around them

Fibre Channel p.9/6


10/60


Gigabit Interface Connectors (GBICs)

Abstract the technical details related tocable type and laser type from expensiveI/O processors

Hot-swappable

Available in two form factorsLegacy (GBIC)Small Form-factor Pluggable (SFP)

Available in a variety of connector types

Make transition to new cable technologiescheaper

Fibre Channel p.10/6


11/60


Switches

Incorporate large buffers to ensurelong-distance links for metropolitan-areaSANs are not starved

Handle load balancing and routedetermination across inter-switch links

Provide fabric name services to aid inservice location

Include zoning capabilites to enforce

security policies



12/60

Hardware Manufacturers

HBAs

Emulex, Qlogic, JNI, LSI Logic, SunMicrosystems

Switches

Brocade, McData, Ancor/Qlogic, Cisco

GBICs

IBM, Finisar, AMP, Stratos Lightwave,Methode

Routers and GatewaysPathlight, ADIC, Cisco, QlogicFibre Channel p.12/6


13/60

Fibre Channel

Network Addressing and Topology



14/60

Fibre Network Logon Types

Private

Simplistic protocol that uses ArbitratedLoop Physical Addressing (AL_PA)

Runs on an arbitrated loop topology

Public (Fabric Aware)

Flexible protocol that runs on switched

fabricsUses fabric name services for targetlocation

Switching done on Hardware addresses(WWNs)



15/60

Public Addressing

Every port has a 48-bit hardware address,

called a World-Wide Port Name (WWPN)Each node in the system has a World-WideNode Name (WWNN)

Frames are routed based upon source anddestination WWPN values in frame headers

WWNNs help facilitate multipathingtechnologies by revealing an N:1 mapping



16/60

Arbitrated Loops

Serial Ring Topology

Loop Initialization Primitive (LIP) is used toestablish and maintain loop

An initiator must acquire the loop, and thusform a bidirectional symmetric logical link withanother node on the ring

Only one pair of nodes per loop cancommunicate at a time

Stability can be disrupted by LIP storms

Loop can be constructed using TX RXsim lex cablin , or usin hubs Fibre Channel p.16/6


17/60

Switched Fabric

One or more switches interconnected by

Inter-Switch Links (ISLs)Each port is configured by the administratorfor a specific mode of operation

Switches provide zones (similar to IEEE802.1Q VLANs) to allow coexistence of

multiple security domainsService discovery is aided by the fabric nameservices that run on every switch

WWNs are used to address targets, so muchlarger networks are possible Fibre Channel p.17/6


18/60

Common Port Types

L_Port

Node PortArbitrated Loop Mode

N_PortNode Port

Point-to-Point Mode

FL_Port

Switch Port

Arbitrated Loop Mode



19/60

Common Port Types

F_Port

Switch PortPoint-to-Point Mode

Many older I/O processors do not supportthis (e.g. Qlogic ISP2100, LSIFC909)

T_Port

Qlogic-proprietary ISL PortE_Port

ISL Port



20/60


21/60

Port to Switch Topologies

Node

NL_Port

Switch

FL_Port

Node

NL_Port

Node

NL_Port

Public Loop Topology

Switch

F_Port

Node

N_Port

Public Point-to-Point Topology



22/60

Common Fabric Topologies

Degenerate Case

Single-switch fabric

Cascade

Consists of a line or ring of switchesEvery switch has one or two adjacentswitches

Latency scales with , which is very badNumber of ISLs scales with , which isoptimal

Cascade is well suited to small fabricsFibre Channel p.22/6


23/60


Switch 1 Switch 2 Switch 3

Cascade Fabric Topology



24/60


Mesh

Switches and ISLs form a complete graphGraph diameter is constant, which makeslatency constant (assuming all equipment

operational, and no congestion)Number of ISLs scales with

Worst-case ISL to usable port ratioProhibitively expensive for large fabrics

Highest performance

Good for medium-sized fabricsFibre Channel p.24/6


25/60


Switch 1

Switch 2

Switch 3

Mesh Fabric Topology

Switch 4



26/60


Multi-Stage

Switches belong to one of two disjointclasses

Edge switches contain links to nodes and

core switchesCore switches contain links to edgeswitches

Topology forms a complete bipartite graphEvery edge switch has a link to everycore switch, and vice versa

ISL count scales withFibre Channel p.26/6


27/60


Mult-Stage

Usable port count scales with, Where

is the number of edge switches

is the number of core switchesis the port count of edge switch

is the number of trunks between

edge switch and core switch

Latency is constant, but greater than thelatency for a Mesh

This topology is well suited to large fabricsFibre Channel p.27/6


28/60


29/60

Fibre Channel

The Protocol



30/60

FC Protocol Stack

FC0

Physical LayerHandled by GBIC electronics

FC18b/10b encoding layer developed by IBMNecessary for receiver clock recovery

Encoding tries to keep the ratio of 0s to1s near 1Limits same bit run length to 5

Detects single-bit errorsFibre Channel p.30/6


31/60

FC Protocol Stack

FC1

268 of 1024 values are valid256 data values12 control values

Start of Frame (SOF), End of Frame(EOF), Idle, Receiver Ready (R_RDY),etc.

FC2

Framing, Segmentation, Reassembly

Flow Control LayerFibre Channel p.31/6


32/60


33/60

The Messaging Hierarchy

FC is a frame-based protocol

Frames are either 512, 1024, or 2048 bytesFrame size is negotiated during port login

Practically everything uses 2048 byteframes

An atomic message unit is called a Sequence

Sequences consist of one or more framesA transaction is called an Exchange

Exchanges consist of one or moresequences



34/60


35/60

Frame Types

Link Control Frame

Provides for flow controlExamples include ACK for class 1 or 2frames

Contains no payload

Data Frame

Provides for data transport servicesExamples include encapsulated protocoldata

Contains a payload of 02112 bytesFibre Channel p.35/6


36/60

Frame Contents

Start of Frame Sequence (4 bytes)

Frame Header (24 bytes)

Payload (02112 bytes)

Cyclic Redundancy Checksum (4 bytes)End of Frame Sequence (4 bytes)



37/60

FC Frame Header Layout

Offset Word Contents

0 Routing Ctrl Destination WWN

4 Reserved Source WWN

8 ULP Type Frame Control12 Sequence ID Data Field Ctrl Sequence Count

16 Originator Exchange ID Responder Exchange ID

20 Parameter



38/60

Fibre Channel

Classes of Service and Flow Control



39/60


40/60

Service Class 2

Connectionless

Unordered delivery is possible

Multi-mode flow control

Buffer-to-buffer by fabric-generated R_RDYframes

End-to-end by N_Ports sending ACKs

Fabric may multiplex/demultiplex frames intransit

Fabric attempts to notify originator ofnon-deliveryFibre Channel p.40/6


41/60

Service Class 3

Connectionless

Unordered delivery is possible

Delivery is not guaranteed

Buffer-to-buffer flow control viafabric-generated R_RDY frames

No notification to originator of dropped frames

Fabric may multiplex/demultiplex frames intransit



42/60

Service Class 4

Not defined in ANSI FC-PH; classes 46 are

discussed vaguely on the internetStateful virtual circuit

Frames are delivered in-order

Delivery is guaranteed

Provides a fractional-bandwidth virtual circuit

Meant for isochronous data services

No know implementations



43/60

Flow Control

Each port has a pool of frame buffers

Qlogic switches statically assign buffers toeach port

Thus, they have the lowest latency in the

industryLong-distance ports can steal buffersfrom adjacent ports, at the expense of

deactivating themBrocade switches have a global framebuffer pool



44/60

Flow Control

Each frame buffer is referred to as a credit

A port may send as many frames as it hasavailable credits before receiving an ACK

Thus, link distances are effectively limited

by the number of available buffer credits ateach end of the link



45/60

Flow Control

End-to-End Flow Control

Class 1 and Class 2 frames use thisImplemented as an ACK frame

A single ACK may acknowledge, one,

multiple sequential, or all frames in asequence

Buffer-to-Buffer Flow Control

Class 2 and Class 3 frames use this

Implemented as an R_RDY frame

One R_RDY is required per receivedframeFibre Channel p.45/6


46/60

Fibre Channel

The Practical Side



47/60

Wh i iSCSI b d h l ?


48/60

Why is iSCSI bad technology?

Implementing SCSI over TCP is simply

overkillThe SCSI protocol does not lend itself tostateful transports, hence SCSI is an FC

class 3 serviceSCSI has its own built-in error detectionmethodology, so a reliable transport layer

is redundantThere are simpler ways to get flow controlthan by implementing Reno or Vegas in

SiliconFibre Channel p.48/6

Wh i iSCSI b d t h l ?


49/60


A key issue with SANs is that they are

physically separate from public networks, andthus are inherently more secure

LANs are generally architected in a much

more haphazard manner than SANs, thusleading to stability and robustness concerns

Mixing bulk I/O traffic and time-sensitiveIP-based traffic will lead to disputes over QoSsettings



50/60

Wh i iSCSI b d t h l ?


51/60


The iSCSI protocol family is still very

immatureBecause iSCSI HBAs will be attached tocorporate ethernet backbones, there will

be an extraordinarily greater chance ofmalicious traffic hitting the HBAs.

The complexity of implementing a TCP

stack in silicon coupled with the smallamount of prior-work makes the probabilityof error high


C t D i Ch i


52/60

Customary Design Choices

FC fabric designs follow normal HA theory

Multiple HBAs on separate buses are placedin each node

Two identical, redundant fabrics are built sothat a complete fabric failure will not affectoperations

FC disks are dual-port so they can beattached to both fabrics


E amples


53/60

Examples

Disk 1 Disk 0Disk 2

Bypass CardBypass CardBypass Card

Bypass CardBypass CardBypass CardLoop A

GBIC

Loop B

GBIC

FibreJBOD


Examples


54/60

Examples

Disk Loop BDisk Loop A

Host Loop BHost Loop A

Typical Redundant Fibre RAID System

Controller A Controller B

High-Bandwidth,

Low-Latency CacheCoherency Interconnect

Fabric AFabric B

JBOD JBOD JBODJBOD


Examples


55/60

Examples

Loop B

Loop A

Fabric Switch

Private JBOD Public JBODPublic Initiator

Fibre HubPublic Loop UsingSimplex Cabling

Simple Fibre Channel Fabric

JBOD

JBOD

JBOD

Private Loop

Using a Hub

Public RAID

Controller(s)


Examples


56/60

Examples

Loop B

Loop AFabric Switch

Private JBOD

Public JBOD

Public InitiatorPublic Initiator Private Initiator

Simple Redundant Fibre Channel Fabric

JBOD

JBOD

JBOD

Public RAID

Controller(s)Fabric Switch


Fibre Channel


57/60

Fibre Channel

Conclusions


Conclusions


58/60

Conclusions

The FC protocol is finally reaching maturity

Fibre Channel has become the standard SANinterconnect technology

No replacement technologies are going to beavailable in the next few years

iSCSI is only a replacement for FC in

lower-end installations where performanceand stability are less critical



59/60

Fibre Channel


60/60

Fibre Channel

Because iSCSI is a BAD IDEA

(And thanks for coming to this talk)

created using xemacs, visio, and LATEXwith the prosper class


fc tom keiser

Documents