storage area networking protocols and architecture

© 2004 Cisco Systems, Inc. All rights reserved. Printed in USA.Presentation_ID.scr

1© 2004 Cisco Systems, Inc. All rights reserved.OPT-2T019899_06_2004_X

STORAGE AREA NETWORKING PROTOCOLS AND ARCHITECTURESESSION OPT-2T01


Morning Schedule

• 9:00am–10:30am Introduction to Storage Area Networking

Storage Terms and Acronyms

Storage Networking Devices (Switches, HBAs, Disk)

Storage Networking ApplicationsStorage Networking TopologiesIntro to Storage Protocols (SCSI, FC, FCIP, iSCSI)

• 10:30am–10:45am Break

• 10:50am–12:30pm Storage Protocols in-depth

Introduction to the Standards

SCSIFibre Channel

• 12:30pm–1:30pm Lunch



Afternoon Schedule

• 1:45pm–3:30pm Storage Protocols In-Depth (Cont.)Fibre Channel ServicesiSCSI FCIPiFCPiSNS and SLP

• 3:30pm–3:45pm Break • 3:50pm–6:00pm Storage Network Troubleshooting

Required ToolsRequired Technical Skill SetsStorage Network Architecture

Design PracticesFC Network DesignsIP SANsSAN ExtensionImplementation and Management


Associated Sessions

• OPT-1051 Introduction to Storage Topologies and Applications

• OPT-2051 Fibre Channel Storage Area Network Design • OPT-2052 FCIP Design and Implementation • OPT-2053 iSCSI Design and Implementation • OPT-2054 Storage Networking Security • OPT-3051 Troubleshooting MDS9000 Fibre Channel SAN• OPT-3052 Troubleshooting MDS9000 IP Storage

Area SAN• OPT-4051 Design and Architecture of Storage

Networking Platforms• OPT-4052 Case Study: Cisco IT Storage Strategy



Reference Materials

• Cisco Storage Networkingwww.cisco.com/go/storagenetworking

• Cisco AVVID Storage Networking Partner Programwww.cisco.com/go/partners

• Cisco Metro Optical Product Informationwww.cisco.com/go/comet

• Storage Network Industry Association (SNIA)www.snia.org

• IETF—IP Storagewww.ietf.org/html.charters/ips-charter.html

• ANSI T11—Fibre Channelwww.t11.org/index.htm

INTRODUCTION TO STORAGE AREA NETWORKING




Section Agenda

• Storage Terms and Acronyms

• Storage Networking Devices

• Storage Networking Applications

• Storage Networking Topologies

• Introduction to Storage Protocols


STORAGE TERMS AND ACRONYMS



Technologies Overview (or “Storage in a Nutshell”)

FSPF

Databases

Servers andMainframes

BackupApps

Mirroring

IPCLOUD

IPCLOUD

IPCLOUD

FC Switch FC Switch

Virtual SAN

JBODsand NAS

Storage AreaNetwork (SAN)Technologies

TAPE

EmbeddedManagement

SANApplications

SANProtocols

RAID &VirtualRAID

iSCSI Drivers

FCIP

iSCSI

iSCSI

GenericFibre Channel

EnhancedFibre Channel

SAN

FCHA

Call Home

SupportCenter

StorageVirtualization


Introduction to SAN Terminology

• Block Level I/O

• File Level I/O

• SCSI—Small Computer Systems Interface

• FC—Fibre Channel

• RAID—Redundant Array of Inexpensive Disks

• iSCSI—Internet SCSI

• FCIP—Fibre Channel over TCP/IP

• iFCP—Internet Fibre Channel Protocol

• iSNS—Internet Storage Name Service



RAID Levels

3 without log 4 with log

Stripe with distributed parity with Random I/O

5

N/AFix parity with Random I/O4

N/AFix parity with concert I/O3

N/AHamming Code2

4Mirror then Striping/Concatenation1+0

4Striping/Concatenation then Mirror0+1

2Mirror1

2Striping/Concatenation0

Min DisksDescriptionRAID Level


Terminology Direct Attached Storage (DAS)

• Block level I/O

• Can be internal or external

• Typically SCSI or FC

• Limited scalability

• High cost due to management



Terminology Network Attached Storage (NAS)

• File level I/O

• Used for file sharing applications

• IP-based

• Deployed over existing low-cost Ethernet networks

• Redundant links

• Scalable

• Multiple servers can share same file system

IP

NAS NAS NAS


Terminology Storage Area Network (SAN)

• Block level I/O

• Deployed as separate network

• Servers share storage subsystem

• Scalable

• Multiple paths for high availability


STORAGE NETWORKING DEVICES



SAN ComponentsHost Bus Adapter (HBA)

• Interface between host and storage

• Supports copper or optical

• Typically one port; Can be multiple ports

• 1Gb, 2Gb and 4Gb



SAN ComponentsFabric Switch

• 1Gb, 2Gb, and 4Gb

• 8-40 ports

• Low latency

• Can be copper or optical


SAN ComponentsDirector Class Switch

• 1Gb, 2Gb, 4Gb and 10Gb

• FC and FICON

• 256 ports and growing

• Low latency

• Can be copper or optical

• Multi-service platforms



SAN ComponentsJBOD

• Just a bunch of disks

• Limited scalability

• Typically 2 FC ports

• SCSI or FC disks

• Basic controllers

• No caches


SAN ComponentsStorage Arrays

• 36GB to many TB

• Typically 2 to many interfaces

• Subsystems may mix interfaces

• ESCON/FICON, SCSI, FC, or iSCSI

• SCSI or FC disks

• Intelligent controllers

• Large caches

iSCSI



SAN ComponentsTape Arrays

• Tape speed vary 5MBs—30MBs+

• Capacity vary 20GB—300GB+

• Deployed in servers or external libraries

• SCSI, FC, Ethernet interface

• DLT most common; LTO gaining traction

222222© 2004 Cisco Systems, Inc. All rights reserved.

STORAGE NETWORKING APPLICATIONS

OPT-2T019899_06_2004_X



IT Storage Requirements

• ScalabilityMeet high growth demand for storage capacity (>80% per year)Increase capacity utilization rates

• AvailabilityShare data across distributed data centers via fast speed, long distance connectivity links Provide effective disaster recoveryImprove interoperability across heterogeneous equipmentEnhance security

• ManageabilityAutomate storage management functionsProvide cross-vendor management tools Managing heterogeneous environments


Storage Network Build-Out

• Application-specific islands of networked storage

• iSCSIConvenient extension of existing FC SAN to IP-attached servers

• Extensive IP services for NAS environments

Homogenous Infrastructure“Isolated Islands”

Starting Point

DAS

SAN

NAS



Storage Network Interconnection

• SAN interconnection forBusiness continuanceUnified management

Remote backup

• Metro DWDM solutionsLow-latency option for synch replication

• FCIPLower-cost option for asynch replication and backup consolidation

SAN Interconnectivity

Present Trend

Optical

FCIP

FCIP


Intelligent SAN

• Intelligent services into the network

• Common management framework

• Content, file, and block awareness

• Transport independent

Storage Utility SAN

Data MgmtServices

StorageRouting

HostAwareness

StorageManagement

StorageSwitching

ContentDelivery

StorageVirtualization


STORAGE NETWORKING TOPOLOGIES



SCSI I/O Topology

• SCSI is the protocol used to communicate between servers and storage devices

• SCSI I/O channel provides a half-duplex pipe for SCSI commands and data

• Parallel implementationBus width: 8, 16 bitsBus speed: 5–80 MhzThroughput: 5–320 MBpsDevices/bus: 2–16 devicesCable length: 1.5m–25m

• A network approach can scale the I/O channel in many areas (length, devices, speed)

SCSI

Target

SCSI Adapter

Host System

Initiator



Fibre Channel Topology

• Very common method for networking SCSI

• Fibre Channel provides high-speed transport for SCSI payload

• Fibre Channel overcomes many shortcomings of DAS including:

Addressing for up to 16 million nodes (24 bits)Loop (shared) and Fabric (switched) transportSpeeds of 100 or 200 Mbps (1 or 2 Gbps)Distance of up to 10km (without extenders)Support for multiple protocols

• Combines best attributes of a channel and a network

SCSI

Target

FibreChannel HBA

Host System

Initiator

Fibre ChannelFabric


iSCSI Storage Topology

• IP access to open storage sub-systems

• iSCSI driver is loaded onto hosts on ethernet network

• Able to consolidate servers via iSCSI onto existing storage arrays

• Able to build ethernet-based SANs using iSCSI arrays

• Storage assigned by iSCSI instance

iSCSI-EnabledHosts (Initiators)

iSCSI Array

(Target)

StoragePool (Target)

iSCSIRouter

IPNetwork

FCFabric FC HBA

Attached Host

(Initiator)

iSCSIiSCSI

iSCSI

iSCSI



FCIP SAN Extension Topology

• FCIP gateways perform Fibre Channel encapsulation process into IP packets and reverse that process at the other end

• FC Switches connect to the FCIP gateways through an E_Port for SAN fabric extension to remote location

• A tunnel connection is set up through the existing IP network routers and switches across LAN/WAN/MAN

StandbyStorage

Production

Storage

Production SiteProduction Site

Database Servers

EMC SRDFExisting IP

NetworkLAN/WAN/MAN

FC Switch

FCSAN

Backup Server

FCIPFCIPGatewayGateway

FCIPFCIPGatewayGateway

ServersFC

SAN

FC Switch

Backup, R&D, Shared Storage,Data Warehousing, Etc.

Backup, R&D, Shared Storage,Data Warehousing, Etc.


FCIP and iSCSI: Complementary

• FCIP: SAN-to-SAN over IP• iSCSI: Host to storage over IP

IP Network

FC SAN

Storage Router

FC SAN

iSCSIiSCSIiSCSI

iSCSI

Storage Router

FCIPGateway

FCIPGateway

FCIPFCIP


INTRODUCTION TO STORAGE PROTOCOLS



Introducing SCSI

• SCSI = Small Computer System Interface

• SCSI is a standard that defines an interface between an initiator (usually a computer) and a target (usually a storage device such as a hard disk)

• INTERFACE refers to connectors, cables, electrical signals, optical signals and the command protocol that allow initiators and targets to communicate



SunSun

Opcode (2A = Write 10)

Reserved

LBA

LBALBA

(0010E43)

Reserved

LBA

LBALen

(128)

Control

SCSI Example

Initiator Target 2Target 1

SCSI Connector SCSI Cable

The SCSI Command Protocol Is Used to Communicate Between SCSI Devices

SCSI Command

In this Case, a File is Being Written to the Hard Drive By an Application on the Workstation

TapeDisk


Why Is SCSI Important for SANs?

• SCSI command protocol is the de facto standard that is used extensively in high-performance storage applications

• The command part of SCSI can be encapsulated in FCP—Fibre Channel Protocol or IP and carried across internetworks; This is the core concept behind storage area networking

• To understand the finer points involved with transporting SCSI across a network with FC or ethernet, the basics of SCSI must be well understood



Standards

• SCSI has evolved since it was introduced as SASI in 1979 by Shugart Associates—it was approved as a standard by ANSI in 1986 and is now referred to as SCSI-1

• SCSI-2 was approved by X3 in 1990 and by ANSI in 1994

• SCSI-3 refers to a collection of standards, each of which defines a very specific part of SCSI: physical interface, transport interface, command interface, architecture model, programming interface, etc.


Sample SCSI Standard ComponentsSCSI Parallel Interface: SPI

Sun

Sun

Initiator

Target 1

Target 2



Sample SCSI Standard ComponentsSCSI Primary Commands: SPC

SunSun

Initiator Target 2Target 1

SCSI Primary Commands(SPC-2)

SCSI Block Commands(SBC)

SCSI Stream Commands(SSC)


SCSI Standards: The Big Picture

FC-xx 1394

SPI-x

FCP SBP

ATAPI

SPC-2 / SPC-3

CAM ASPI Generic

SBC SSC SES More…



“This specification describes a reference model for the coordination of standards applicable to SCSI-3 I/O systems and a set of common behavioral requirements which are essential for the development of host software and device firmware that can interoperate with any SCSI-3 interconnect or protocol.”

SCSI Architecture Model November 1995

SCSI Architecture Model


SCSI Architecture Model

• The SCSI architecture model defines generic requirements and implementation requirements

• Each SCSI implementation standard must fulfill the requirements set forth by SAM



SAM Highlights: Client-Server

• SCSI is a client-server protocol

• The client is called the initiator (this is usually the OS I/O subsystem) and issues requests to the server

• The server is called the target (this is usually the SCSI controller that is part of a storage device) and receives, executes and returns initiator requests and their associated responses


SAM Highlights: Initiator: Target

• A single initiator can have multiple application clients

• Targets have ONE task manager and one OR MORE Logical Units (LU), which are numbered (LUN)

• The task manager has the authority to modify service requests that have already been received by the target



SAM Highlights: Logical Units

• Each logical unit within a target is numbered; that number is called a LUN and is the only way to refer to that logical unit

• The device server is the entity that receives, executes and returns requests that are made to its logical unit

• The concept of task setis beyond the scope of this presentation


SAM Highlights: Command Model

• SAM defines two categories of protocol services:Execute command/confirmation services;Data transfer services

• This leads to the three main phases of a data transfer:1. Execute: Send required command and parameters

via CDB;2. Data: Transfer data in accordance with the command;

3. Confirmation: Receive confirmation of command execution



SAM Highlights: Sample Data READ

1. Send SCSI Cmd issued by initiator—the command sent is READ;

2. SCSI command receivedby target;

Data transfers occur during the ‘working’ phase between initiator and target;

3. Send command completeis returned by the target;

4. Command complete received by target

…


SAM Highlights: Parameters

• The data transfer model reflects parameters that will be used by SCSI commands

• This model illustrates that a complete data transfer (right) can be broken up into multiple parts (left)



SAM Highlights: Communication Model

SAM Defines a Hierarchy of Protocols

Let’s Expand on this Portion


SCSI Transport Protocol

IPIP

TCPTCP

iSCSIiSCSI

SCSI ProtocolSCSI Protocol

FibreChannelFibreChannel

FCPFCP Parallel BusParallel Bus

EthernetEthernet

iFCPiFCPFCIPFCIP



SCSI Transport Protocol

IPIP

TCPTCP

iSCSIiSCSI

SCSI ProtocolSCSI Protocol

FibreChannelFibreChannel

FCPFCP Parallel BusParallel Bus

EthernetEthernet

iFCPiFCPFCIPFCIP

Today’s In-Depth Protocol

Discussions

STORAGE PROTOCOLS IN-DEPTH




Section Agenda

• Introduction to Standards

• SCSI Protocol

• Fibre Channel Protocol

• Internet SCSI (iSCSI)

• Fibre Channel over IP (FCIP)

• Internet Fibre Channel Protocol (iFCP)

• iSNS and SLP

INTRODUCTION TO STANDARDS




Standards Groups: Storage

ISO / IECJTC-1

InterNational Committee for InformationTechnology Standards

(INCITS)

American NationalStandards Institute

(ANSI)

Techincal Committee onLower-Level Interfaces

(T10)

Information TechnologyIndustry Council

(ITI)

Techincal Committee onDevice-Level Interfaces

(T11)

CJ11

C++J16

Techincal Committee on ATAttachment Interfaces

(T13)

SCSI Fibre ChannelHIPPI

IPI

ATA (IDE)ATAPI

www.t10.org www.t11.org www.t13.org


Standards Process

• Technical Committees (T10) write drafts

• Drafts are sent to INCITS for approval

• Once approved by INCITS, drafts become standards and are published by ANSI

• ANSI promotes american national standards to ISO as a Joint Technical Committee member (JTC-1)



Standards Work Group: IP Storage

IP Storage Technical Work Group Acts as Primary Technical Focal Point of the Storage Networking Industry Association (SNIA) on IP Storage Issues, Coordinating with the SNIA IP Storage Forum

Transport Area—Has 23 WGs, One which Is the IP Storage WG

ISOC

Internet Society

IESGInternet Engineering

Steering Group

IETF

Internet EngineeringTask Force

Transport Area

IEFT Is the Organization Ratifying the IPS Standards

FIBRE CHANNEL IN-DEPTHFIBRE CHANNEL IN-DEPTH




Fibre Channel Protocol Agenda

• FC Introduction• Fibre Channel Communications Model• Protocol Constructs• FC-PH (Fibre Channel—Physical and Signaling

Interface)• Login Parameters• Frame Processing• Arbitrated Loop• Switch Fabric Operation• Switch and Hub Mixed Topology Network Operations• FC Error Management


Fibre Channel Environment

• Channel reliabilityMultiprotocol supportOvershared serial media

With networking capability and functionality



Fibre Channel Environment

• High bandwidth

• High data integrity

• Highly reliable

• Destination pacedBuffer credits

• Scalable

• High availability

• Shared media

• Circuit/packet

• Multiple protocol support

• Transport flexibilityDedicated conn—Class 1Multiplexed—Class 2

Datagram—Class 3

• Configuration flexibilitySwitch

Loop


What Is It?

Channels• Connection service

• Physical circuits

• Reliable transfers

• High speed

• Low latency

• Short distance

• Hardware intense

Networks• Connectionless

• Logical circuits

• Unreliable transfers

• High connectivity

• Higher latency

• Longer distance

• Software intense



What Is It? Fibre Channel

Channels• Connection service• Physical circuits• Reliable transfers• High speed• Low latency• Short distance• Hardware intense

Networks• Connectionless• Logical circuits• Unreliable transfers• High connectivity• Higher latency• Longer distance• Software intense

Fibre Channel• Circuit and packet

switched

• Reliable transfers

• High data integrity• High data rates

• Low latency• High connectivity

• Long distance


Fibre Channel Protocol LevelsLevels

HIPPI Cluster 370 OEM

SCSI IP ATM

Common Services

FC-4’s

FC-3

Signaling Protocol

Transmission Code

Physical Interface

FC-2

FC-1

FC-0

FC-PHN_Port

F_Port

PC-PH = Physical and Signaling Layer



Fibre Channel Functions

Structure Is Divided into 5 Levels of Functionality• FC-0 defines the physical interface characteristics

Signaling rates, cables, connectors, distance capabilities, etc.

• FC-1 defines how characters are encoded/decoded for transmission

Transmission characters are given desirable characters

• FC-2 defines how information is transportedFrames, sequences, exchanges, login sessions

• FC-3 is a place holder for future functions

• FC-4 defines how different protocols are mapped to use Fibre Channel

SCSI, IP, virtual interface architecture, others


Fibre Channel Topologies

• Point to point

• Arbitrated loop

• Switched fabricFC

N

N N

L

L

L L

L

L

N

N

N

FF

FF



Point to Point

• Dedicated connection between ‘N’ port FibreChannel devices

• All link bandwidth is dedicated to communication between the two nodes

• Suitable for small scale scenarios when storage devices are dedicated to file servers

NNN

N


Arbitrated Loop (FC-AL)

• TX of each node is connected to the RX of the next node until a closed loop is formed

• Maximum bandwidth: 100 MB/sec. (shared amongst all nodes on loop)

• 126 nodes max on loop• Not a token passing scheme—no limit

on how long a device may retain control• Operational sequence:

Arbitrate for control of loopOpen channel to targetTransfer dataClose

• Number of nodes on loop directly affects performance Fibre Channel Hub

FC

L

L

L

L

L

L

LLL



Data Integrity

Upper Level Protocol

Signaling Protocol

Transmission Code

Physical Media

Operation Control and Byte Counts

• Operation• Frame counts• CRC (32 bit)• Frame delimiters

8b/10b Code

Fibre Reliability


Flow Control

• Back pressure technique

• Frame credit Established by receiver during LOGIN

• TransmitterMust have credit to transmit

• ReceiverReinstates credit with ACK


FIBRE CHANNEL COMMUNICATIONS MODEL



The Model

• The Fibre Channel communications model is based on the definition of:

Physical objectsProtocol construct

• These objects and constructs:Define the behavior of the physical elementsControl the transfer on informationProvide for “link” managementProvide the basis for:

HardwareFirmwareSoftware



Physical

• The fundamental physical objects in FibreChannel are:

PortsLink

Nodes

Fabric

Some Logical Items Used in These Discussion Are:• Addressing

• Communications Model


Fibre Channel: Port Types

• ‘N’ port: Node ports used for connecting peripheral storage devices to switch fabric or for point to point configurations; can be considered the end port

• ‘F’ port: Fabric ports reside on switches and allow connection of storage peripherals (‘N’ port devices)

• ‘L’ port: Loop ports are used in arbitrated loop configurations to build storage peripheral networks without FC switches; these ports often also have ‘N’ port capabilities and are called ‘NL’ ports

• ‘E’ port: Expansion ports are essentially trunk ports used to connect two Fibre Channel switches

• ‘G’ port: A generic port capable of operating as either an ‘E’ or ‘F’ port; its also capable of acting in an ‘L’ port capacity; Auto Discovery



N_Port

Host / Device

N_Port

Host/ Device Interface

Serial Data Out Serial Data In


Link

• A link consists of2 unidirectional “fibers” transmitting in opposite directionsMay be either:

Optical fiber

Copper

• Transmitters may be:Long wave laser

Short wave laserLED

Electrical

Link Transfer Rates

25265.5100106.25

Mbytes/secClock Mbaud/sec



Link

Link

Host / Device

N_Port

Host/ Device interface

Serial Data Out Serial Data In


Node

• The equipment which contains one or moreN_Port or NL_Port (topology dependent)May be

Computer

ControllerDevice

Is NOT a switch fabric



Node

N_Port

Link

N_Port

Link

N_Port

Link

N_Port

Link

Controller


Communications Model

• Point to point

• N_Port to N_Port

• Flow control

• AcknowledgedNode Node

Transmitter

Receiver Transmitter

Receiver

N_Port

Node Node

Link

Transmitter

N_Port



Fabric

• FabricThe entity which interconnects N_PortsProvides routing based on destination address

Fabric may be:

Point to point—No routing requiredSwitched—Routing provided by switch

Arbitrated loop—Routing is distributed throughout attached L_Ports


Terms

• TopologyThe physical structure of the interconnect of portsDefines the logical behavior of transactionsFibre channel has 3 topologies

Pt to PtSwitchedArbitrated loop

• FabricThe fabric is the generic item that interconnects nodesA fabric is made of Fibre Channel topologies like Pt to Pt, switches and loops



Point to Point

Communications Model

• Source to destination

• Based on address routing through the fabric

Node Node

Transmitter

Receiver

Receiver

Node Node

Fabric

N_PortN_Port

Transmitter


Switched Fabric

N_Port N_PortN_Port N_Port N_PortN_Port

Fabric

Switch

Communications Model—Source to Destination Based on Address Routing through the Fabric

A B



Communications Model—Source to Destination Based on Address Routing Distributed in the NL_Ports on the Loop

Arbitrated Loop

NL_Node “A”

NL_Node “B”

Link

A B

FC PROTOCOL CONSTRUCTS




What Are Protocol Constructs

• The fundamental protocol structures in the FibreChannel are called constructs, and they are:

FramesSequences

Exchanges

Information Units (IU)Procedures

Upper Layer Protocols (ULP’s)


Construct Introduction

• FC-2 defines these constructs that allow the related information to be:

Grouped togetherCoordinatedHandled in an efficient manner

• To accomplish this we define the notion of:FramesSequencesExchanges

• Also defined are means for the Upper Level Protocols ULP’s to communicate with FC-2:

Information Units (IU)• A procedure called the login defines the operating

environment between the N_PortsExchange of the data describing the characteristics of the ports



Chunks

• The ULP’s deal with “chunks” of data that are moved across the network

• These chunks of data may be eitherControlStatus

Real data


Frames

Frame• FC-2 layer will take this

chunk of data and move it from

Transmitting node to receiving node In the units of what Fibre Channel calls frames

Frame Size• FC-2 will determine the size

of the frames based on operating environment established between the two communicating nodes

Common Services

Signaling Protocol

Transmission CodePhysical Interface

FC-3

FC-2

FC-1

FC-0



Frame Structure

24* 4 24 0-2114 4 4 Bytes

Idles SOF Frame Header Data Field CRC EOF Idles

* 6 Idle Words (24 bytes) Requires by TX

2 Idle Words (8 bytes) Guaranteed to RX

General FC-2 Frame Format

Frame Format

CRC Calculated on Frame Header and Data Field Only


Frame Header

R_CTL Routing D_ID 24bits Destination

07815

16

23

24

31

Word

0

1

2

3

4

5

S_ID 24 bits SourceCS_CTL 8 bits Class Spec TYPE 8 bitsData structure

SEQ_ID 8 bits DF_CTL 8 bitsData field

F_CTL 24 bits Frame Control

SEQ_CNT 8 bits Sequence Count

OX_ID 16 bits Orig Exch ID RX_ID 8 bits Respon Exch ID

Parameter Specific to frame type



Data Field

Optional Headers PayloadFILL

Data Field 0-2114

0 - 64 0 - 2112

1 - 3

0–2048 Typical MTU


Sequence

• SequencesEach chunk of Upper Level Protocol (ULP) data is moved within the envelope of what Fibre Channel calls a Sequence (SEQ)A sequence consists of a set of related framesAs expected there are lots of rules governing sequences

• Information Units (IU)The ULP tells the FC-2 how to transfer theses chunks of data through a structure called a information unitVery few rules for IU’sIU is a convention defined outside of FC-PHIU’s are unique to each upper level protocol



Sequence

• Sequence Initiator (SI)The N_Port which is transmitting the data frames

• Sequence Recipient (SR)The N_Port which is receiving the data frames

FabricSI N_Port

SR N_Port

Data

Chunk

Data

Chunk

Data Frame


Sequence

Fabric

Initiator (SI)

Target

Data Frame

Sequence

Read Command (Chunk)

(SR)

(SI)

Sequence

(SR) Data (Chunk)

Status

Data Frame

Data Frame

Sequence



Sequence Identifier

• Sequence initiator assigns an “identifier” to each sequence

This “identifier” is called the Sequence_Identifier or Seq_IDThe Seq_ID uniquely identifies a given sequence within the context of the operation

Each frame is identified within this operation by Seq_ID and Seq_CNT


Sequences: Active and Open

• Sequence Initiator (SI)A sequence is ACTIVE

From the time the first frame of the sequence is transmitted until the frame with the end sequence flag is sent

A Sequence is OPENFrom the time the first frame is transmitted until the reception of the ACK to the last frame

• Sequence Recipient (SR)A sequence is ACTIVE and OPEN

From the time of the first frame of the sequence is received until the transmission of the ACK to the last frame of that sequence



Sequences: Active and Open

Originator (SI)Responder (SR)First Data_Frame

SOF Received

EOF Transmitted

Frame with End_SEQ set

ACK to first Frame

ACK to last Frame

EOT Received

Open

Active

Active & Open


Sequence Streaming

• Sequence streaming is the ability to Begin transmission of the next sequence while one or more previous sequences are OPEN

• Sequence Recipient (SR) grants permission to have up to “n” streaming sequences; This is determined at N_Node login time

Must Support “n=1” sequence status blocks (state info)

(This Allows for More Data in the Pipe for Distant Connections)



Exchange

• Upper level protocols frequently deal with related bits of data as:

Request/replyCommand/data/status

• These relationships are called “operations”Exchanges• “Operations” of data grouped together into what

Fibre Channel call exchangesAn exchange consists of a set of related sequences

Exchanges are bi-directionalSequences are unidirectional and sequential

• There are other rules that govern exchanges


Exchange

Fabric

Initiator (SI)

TargetData Frame

Sequence

Read Command (Chunk)

(SR)

(SI)

Sequence

(SR) Data (Chunk)

Status

Data Frame

Data Frame

Sequence

Exchange



Exchange

• Exchange originatorThe N_Port which transmitted the FIRST data frame for this exchange

• Exchange responderThe N_Port which is the destination of the FIRST data frame of this exchange

The designation for the originator and responder are fixed for the duration of the exchange

Unlike the SI and SR Which Change Roles Within the Exchange


Exchange Identifiers X_ID

• An exchange has two “identifiers” associated with it

Exchange originator:Assigns an OX_ID which is meaningful to it

Exchange responder:Assigns a RX_ID which is meaningful to it

In general terms it is called the X_ID• Meaningful is that in the exchange there is

“context” with information like state, control, and status with regards to the exchange

• An N_Port will save, create and update this information throughout the exchange based on the assigned X_ID’s



Information Unit

• Upper Level Protocols (ULP’s) know about Information Units (IU’s) but know nothing about:

FramesSequencesExchanges

• A ULP deals with units like:Order of events within the operationWhich node will transmit in the next “phase”

(Command phase, data phase, status phase)Is required to have some knowledge about Fibre Channel

• An information unit is a Fibre Channel sequence


Information Unit

• The IU contains information sets with such items as LUN, task attributes, CDB and the command byte count

• The IU’s are used in protocol mapping from FC-4 to FC-2 and are assigned an identifier that is useful to humans not used by the machine

• All the information needed to support a ULP is formed into a IU table and is listed as a first , middle or last IU in the exchange

We Will See More of these Tables when We Cover SCSI Mapping onto Fibre Channel



FC-2 Hierarchy

Construct Meaning

OX_ID / RX_IDExchange

SEQ_ID

Information Unit

Sequence

Frame SEQ_CNT

Frame Fields

Consists of one or more Sequences for ULP Operation

The structure used by the ULP to define a Sequence (not visible over link)

Consists of one or more related Frames

Contains in its Payload a ULP “chunk” of data

The Hierarchy of Constructs


FC-2 Hierarchy

Frame Fields

OX_ID & RX_ID

…...

…...

EXCHANGE

SEQ_ID

SEQ_CNT

SEQUENCE

Frame FrameFrame

Information Unit Per ULP Terms


FC-PH (FIBRE CHANNEL: PHYSICAL AND SIGNALING INTERFACE) STRUCTURE, PROCEDURES, AND PROTOCOLS



Transmission Code

• Fibre Channel uses a 8b/10b transmission codeEach 8 bit data byte to be transmitted is converted into a 10 bit quantity

The 10 bit quantity is then transmitted over the FC media

The 10 bit quantity is then converted back to the 8 bit data byte by the receiving node

• The 10 bit quantities are called transmission characters

• Transmission characters come in two formsData chartersSpecial characters



8b/10b Code

Why 8b/10b

1. To ensure the sufficient transitions are present in the serial bit stream to make clock recovery possible at the receiver

2. Increase the likelihood of detecting any single or multiple bit errors

3. To provide special characters with distinctive and easily recognizable characters to achieve word alignment on the incoming bit stream


8b/10b Code

Characteristics of 8b/10b• The 10 bit transmission code

Supports all 256 values of the 8 bit data byte

Contains unused code points

Illegal codes(called code violations)

Detection of code violations

May occur on the transmission character in which the error occurred or may be detected on a subsequent character

Contains “special” characters

Running “disparity” with DC balance

(Count of 0’s and 1’s Equal the Same over Time)



8b/10b Code

Running DisparityDisparity: The Difference Between the Number of Ones and

Zeros in a Transmission CharacterRunning Disparity: A Binary Parameter Indicating the Cumulative

Disparity of All Previously Issued Transmission Characters

Transmission Characters Always Have Either:6 Ones and 4 Zeros = Positive Disparity4 Ones and 6 Zeros = Negative Disparity5 Ones and 5 Zeros = Neutral Disparity

Rules: A Positive Disparity Transmission Character Can Not Be Followed By Another Positive Transmission Character

A Negative Disparity Transmission Character Can Not Be Followed By Another Negative Transmission CharacterAt Transmission Character Boundaries the Difference between the Number of Ones and Zeros is + or – 1


8b/10b Code

Code Notation• Each valid transmission character has been

assigned a name in the form of:

Zxx.y“Z” = K or D D=Data K=Special Character

“xx” = Decimal Value of the 5 LSb bits“y” = Decimal Value of the 3 MSb bits



Conversion Table

ABCDEiFGHj

01234567 FC-2 Bits

FC-1 Code Bit

LSBMSB

D or K .

j and i are add as part of the 10b conversion process

0 0 0 0 1 0 0 0

Example D1.0

FC-1 Transmission Character

0 1 1 1 0 1 0 1 0 0 Neg Disp Value

Transmission Order


Special Characters

• K28.5 only special character used in FibreChannel out of the 12 set aside

Has no 8 bit representationThe only FC transmission character with 5 consecutive 1’s or 0’s

Used to find word boundaries and syncUsed in ordered sets

0 0 1 1 1 1 1 0 1 0 + Current Running Disparity

1 1 0 0 0 0 0 1 0 1 - Current Running Disparity



Transmission

• Transmission word consists of 4 continuous transmission characters treated as a unit

40 bits longAligned on a word boundaryThere is a ordered set and a data word

Encoded Data Byte

Encoded Data Byte

Encoded Data ByteK28.5

Byte 0 1 2 3

Ordered Set

Encoded Data Byte

Encoded Data Byte

Encoded Data Byte

Encoded Data ByteData Word

Transmission Order


Ordered Set

• Transmission word starting with the K28.5 special character

• Three classifications of ordered sets are definedDelimitersPrimitive signals

Primitive sequences

Dxx.yDxx.yDxx.yK28.5MSB LSB

The Three Data Characters Define the Meaning of the Ordered Set and Are Repeated for the Third and Fourth Character



Primitive Signals

• Primitive signals are ordered setsTransmission of primitive signals are interrupted occasionally to transmit frames

• Three basic typesReceiver_Ready (R_Rdy)Idle (idle or I)

Arbitrate (ARBx)


Delimiters

• Delimiters are ordered sets that delineate a frameImmediately preceding and following the contents of a frame

• Two basic typesStart_of_Frame (SOF)End_of_Frame (EOF)

• SOF delimitersIdentify the start of a frameIdentify the transmission classUsed to establish a Class_1 connectionIdentify the beginning and continuation of a sequence

• EOF delimitersTerminate framesIdentify the end of a sequenceTerminate connectionsIndicate known frame errors



FC-1 Synchronization

• ProceduresSync acquireInitialization

Loss of sync procedure

• Primitive sequences


Sync Procedures

• Bit synchronizationThe state in which a receiver is delivering retimed serial data at the required bit error rate

• Transmission word synchronizationAchieved when the receiver identifies the same transmission word boundary on the receive bit stream as the established by transmitter at the other end of linkAcquired by detection of three consecutive ordered sets without errors

• Loss of synchronization procedureThe receiver shall enter the loss-of-sync state upon detection of the fourth invalid transmission word

• Synchronization acquired procedureThe receiver shall enter the synchronization-acquired state when it has achieved both bit and transmission word sync



Synch AcquiredLoss of Sync State

Bit Sync Acquired

Rx Ordered set #1

Rx Ordered set #2

Rx Ordered set #3

Sync Acquired

Data Word

Data Word

Data Word

Waiting on Bit Synchronization


Loss-of-Sync ProcedureSync Acquire State

No Invalid Words Detected

First Invalid Word

Second Invalid Word

Third Invalid Word

Loss Of SyncFourth Invalid Word

One Invalid Word in Next

2 Words


2 Words


2 Words


2 Words

Two Consecutive Valid Words





FC-1 Constructs

• Port states

• Primitive sequencesNOS/OLS/LR/LRR

• Primitive sequence protocolsSequence flows

• Relationships

• Port state transition table


Port States

• Four primary operational statesActive stateLink recovery stateLink failure stateOffline state

• Operational states of a portN_PortsF_Ports

• Port state changes occur as a result ofConditions detected within the portIn response to reception of primitive sequencesIn response to upper level controlling entity



Primitive Sequences

• Ordered set that is transmitted continuously to indicate that specific conditions within the port are encountered

• Transmitted while the condition exist

• Four primitive sequencesNot Operational Sequence (NOS)Offline Sequence (OLS)

Link Reset Sequence (LR)

Link Reset Response Sequence (LRR)


Primitive Sequence NOS

Not_Operational Sequence

• Transmitted by the port to indicate thatLink failure had been detected

Loss of sync

Loss of signal

Port is offline

D5.2D31.5D21.1K28.5



Primitive Sequence OLS

Offline Sequence

• Transmitted by port to indicate that it is:Initiating the link initialization protocol

Receiving NOS

Entering the Offline state

D21.2D10.4D21.2K28.5


Primitive Sequence LR

Link Reset Sequence

• Transmitted by port to indicate that it is:Initiating the link reset protocol

To recover from a link timeout

To remove a Class_1 connection

D9.2D31.5D9.2K28.5



Primitive Sequence LLR

Link Reset Response Sequence

• Transmitted by port to indicate that:Link reset is being received

D9.2D31.5D21.1K28.5


Primitive Sequence Protocols

Link Initialization Protocols• Required after

Port power-on

Port internal resetPort has been in offline state

Online to offline protocols• Required to enter offline state



Primitive Sequence Protocols

Link Failure Protocol• Required after

Detection of loss of synchronization for a period of time greater than 100ms which is the receiver-transmitter time-out value (R_T_TOV)

Loss of signal while not in the offline state

Link Reset Protocol• Required after

Link resetLink timeout


Primitive Sequence Flows

Link Failure State (LF)

Offline State (OL)

Link Recovery State (LR)

Active State(AC)

NOS

Link Failure Protocol

Li nk Init iali zat ion Pr oto co l

Link Reset Protocol

Idle

Online t o O

f fl in e Prot ocol



Primitive Sequence Meanings

Idles or R_RDY

Operational Link

• Idles and R_RDY recognizedIDLE

IdlesLink Reset Response

• Link Reset RecognizedLRR

LRRLink Reset

• Remove class_1 Conn

• Reset F_Port

• OLS recognized

LR

LROffline State

• Internal port failure

• Transmitter power down, perform diags, or perform initialization

• Receiver shall ignore Link error or Link Failure

OLS

OLSNot Operational

• Link FailureNOS

Transmit in Response

MeaningCurrently Transmitting


AC = Activity State

LR = Link Recovery State

LF = Link Failure State

OL = Offline State

Link Failure ConditionAC AC

LFLF

LROL

LR

AC

AC

NOS

OLS

LR

Idle

LRR

Idle

Port A Port B

Link Failure



Offline

AC AC

OLOL

LRLR

AC

AC

OLS

LR

LRR

Idle

Idle

Idle

Port A Port BRequest to Go Offline

After 5ms Minimum Diags May Be Preformed

Request to Go Online

AC = Activity State



OL = Offline State


Frame Header Detail

• Routing control (R_CTL)

• Addressing (D_ID) (S_ID)

• Type (TYPE)

• Frame control (F_CTL)

• Sequence identifier (SEQ_ID)

• Sequence count (SEQ_CNT)

• Exchange identifiers (OX_ID) (RX_ID)

• Parameter field



Frame Detail: Routing Control


07815

16

23

24

31

Word

0

1

2

3

4

5

S_ID 24 bits SourceCS_CTL 8 bits Class Spec

TYPE 8 bitsData structure







Routing Control

• The Routing control field is an 8 bit field

• R_CTL consist of two 4 bit sub-fieldsRoutingInformation category

Info CategoryRouting

31 242728



Routing Control

• The R_CTL is used to direct the frame to the process the frame is directed to; For example:

Frames directed to the fabric for extended link services (0x22)

Indication of the function or purpose of the frame payload from the upper level protocol at FC-4 (0x01)


Port Addressing

R_CTL Routing D_ID 24 bits Destination

07815

16

23

24

31

Word

0

1

2

3

4

5










Port Addressing

• D_ID and S_ID fields are 24 bits each

• They provide the address or identifier of theSource and destination port of a frame

• Although the address map is flat, there are several formats depending on:

Topology

Location


Port Address Identifiers

• Applicable to all topologiesPoint to pointSwitched

Loop

• Dynamically assigned or administratively assigned

• Used for frame routingUnique within Fibre Channel network

• Assigned by the “fabric”

• Some address reserved for special functions



Port Address Identifiers

Acquired During Loop InitializationArbitrated Loop

By Switch During Fabric Logon

• Bound to Physical Port on Switch

Switched

By N_Port with Higher Worldwide Name (MAC)

Point To Point

AssignmentTopology


Address Identifiers

Arbitrated Loop Physical Address (AL_PA)0000

AL_PAAreaDomain

DeviceAreaSwitch Domain

8 bits 8 bits 8 bits

Switch Topology Model

Private Loop (Not Connected to a Switch)

Public Loop (Connected to Switch)



Reserved Addresses

• FC-PH has defined a block of addresses for special functions:

High order 16 addresses in the 24 bit address spaceCalled the well known addresses

Main Address Used Today

FF FF FC Directory Server

FF FF FD Fabric Controller

FF FF FE Fabric F_Port which N_Port is attached to


Data Structure Type


07815

16

23

24

31

Word

0

1

2

3

4

5










Type

• The TYPE is a 8 byte field

• Indicates the upper level carried in the payload of the frame

• Examples:SCSI ‘08h’IP ‘05h’

SNMP ‘24h’Fibre Channel services ‘20h’


Frame Control


07815

16

23

24

31

Word

0

1

2

3

4

5










Frame_Control

• The frame control is a 24 bit field

• It contains a number of flags that are used to control the flow of the sequence

• The more common flags are exchange and sequence management, acknowledgement control and error conditions

Bits 16-23 deal with the sequence and exchange settings

Bits 14-15 deal with X_IDBits 13-12 form the ACK level for class 1 & 2

Bits 5-4 used for aborting the sequence


Frame Control Bits 12-13

• Acknowledgment CapabilityProvide assistance to Sequence Recipient (SR) by translating the ACK capabilities bits in the N_Port class parameters

Meaningful only in Class 1 and 2 data frames0 0 = No ACK

0 1 = ACK level 1 –one for every frame1 0 = ACK level “N” N = number of frames

1 1 = ACK Level 0—single ACK for complete exchange, used in video streaming



Sequence Identifier


07815

16

23

24

31

Word

0

1

2

3

4

5









Sequences

Sequences• Deal with chunks of upper level protocol

• Are made up of one or more frames which transport the ULP

• The data phase may be subdivided into multiple sequences

• Uniquely identifiable with SEQ_ID

• The command, data, and status phases of SCSI are examples of sequences



Sequence Identifier

• The Sequence Identifier (SEQ_ID) is a 8 bit field

• All Frames of a sequence will carry the same SEQ_ID value

Data content of these frames are related in some way by the ULP


Sequence Count


07815

16

23

24

31

Word

0

1

2

3

4

5










Sequence Count

• Sequence count (SEQ_CNT) is a 16 bit field

• Identifies the order of the transmission of frames within this sequence

• Used by Sequence Recipient (SR) to account for all transmitted frames

• Used by Sequence Initiator (SI) to account for all transmitted acknowledges (ACK’s) in Class 1 and 2


Sequence Count

• Within a Sequence_InitiativeThe SEQ_CNT of the first data frame will be zero

The SEQ_CNT of each subsequent data frame in the sequence will be incremented by 1

The first data frame of the next sequence may be either zero or one more then the last data frame, this is called “continuously increasing SEQ_CNT”If streamed sequences is used, continuously increasing SEQ_CNT is required



Sequence Count

• Sequence initiatorAssigns SEQ_CNT to data framesKeeps a record of ACK frames received

• Sequence recipient Records SEQ_CNT of data framesTransmits an ACK frame for each valid data frame when Rx buffer is availableKnows that sequence was received without error if all Frames are Rx without errors and are accounted for

• Sequence initiator Knows the sequence was received without error if it has Rx an ACK frame to all frames within the sequence


Exchange Identifiers


07815

16

23

24

31

Word

0

1

2

3

4

5










OX_ID and RX_ID

• 2 byte fields each

• Contain the originator exchange identifier and responder exchange identifier

• They point to state and context information regarding the exchange in the originator port and responder port

OX_ID’s are reused after each exchange is over


Parameter Field


07815

16

23

24

31

Word

0

1

2

3

4

5










Parameter Field

• The parameter is a 4 byte field

• The content of the parameter field is dependent on the specific frame type as identified in the routing field

FC-4 data frames

ACK link controlPort reject and frame reject frames

Port busy and fabric busy frames

LOGIN PARAMETERS




Login

• Exchange “service parameters” done with login frame PLOGI or FLOGI

• Required before communications can be established between the two ports

• Applies to all topologies• Applies to all ports, node and fabric• Bi-directional

ACCEPT Frame contains service parameters of the port addressed

Procedure to Determine the Operating Environment for Communications between Two Ports


Login

• Version of Fibre Channel support• N_Port or F_Port functionality• Service classes supported• Size of receive buffers• Number of sequences supported• Support for Intermix• ACK capability• Error policy supported• Others

Service Parameters Contain the Following “Type” of Information



ACK’s

Informs Transmitter that:• One or more valid data frames were received by the

sequence recipient for the corresponding sequence qualifier

• Interface buffer is available for another data frame, this only applies to class 1 and class 2

Class 3 are not ACK’ed

• Flow controlRe-instates end-to-end credit


ACK’s

• Frame HeaderConstructed from the data frame which is being acknowledged

S_ID and D_ID are swappedF_CTL with both exchange and sequence context bit invertedSEQ_ID is unchangedSEQ_CNT is set to the sequence count of the highest data frame being replied to by the ACK

Parameter FieldBit 16 = History bitBits 0-15 are ACK type specific



ACK’s

• Again there are three types of ACK’sACK_1 default for class 1 and 2 one ACK sent for each SEQ_CNT

ACK_N Class 1 or 2 N=ACK sent by recipient for the support indicated during port loginACK_0 class 1 or 2 single ACK sent at end of sequence

We could spend a lot more time discussing ACK’s but there is little or no class 1 or 2 used in networks today and doubt if we will see any soon


Busy and Reject

Port Reject P_RJT Fabric Reject F_RJT

• Transmitted by destination port or fabric in response to a specific data frame

• Applicable to only Class 1 and 2

• Sent in reply to valid frames

• Transmitted by the “receiver” of the data frame with reason code

• Indicated that the corresponding data frame was NOT delivered to the ULP



Busy and Reject

• Busy sent by fabric if unable to deliver frame due to busy condition

• Busy sent by port if temporarily busy and unable to process a frame

• If F_BSY or P_BSY is sent, fabric or port give reason code

Class 1 busy only allowed on the connection request

Class 2 any frame may Rx busyClass 3, busy is not sent; If a frame can not be delivered it is discarded without notification


Flow Control and Credit

• Flow modelFrames are moved:

From one BufferTo another Buffer

Frame Flow is:From the Source buffer andTo the destination buffer

Depending on the class of serviceMultiple intermediate buffers may be involved

Applies to:All topologies




• Frame flow is controlled by the receiverBack-pressure mechanismACK’s class 1 and 2, RDY’s class 3

• Flow control is based on frame flowWhich frames are flow controlled is dependent on class of service

• Receiver defined parameters during the login procedure

Maximum frame size

Number of buffers



Receiver• Establishes operating environment through login

Size of buffers

Number of buffers (credits) allocated to this transmitting port

• Pumps-up these creditsBy ACK’s when buffer is available

• A receive Buffer is available afterThe frame was verified to be valid, no errors

And the frame has been moved off the interface buffer




Transmitter• Keeps

Credit maximum valueCredit_Count

• Consumes one credit for each “frame” it transmitsCredit_CNT = Credit_CNT –1 for each Data_Frame Tx

• Regenerates credit for each ACK Rx’edCredit_CNT = Credit_CNT + N

• Stops transmitting whenCredit_CNT = 0



• FC-2 defines two type of creditBuffer to Buffer (BB)End-to-End (EE)

• BB credit is the flow of connectionless trafficOver a LINK from Tx to RxClass 2 and 3Signal used = R_RDY

• EE credit is the flow on connection trafficSource to destination nodeClass 1 and 2Signal used = ACK

• Both based onCreditCredit_CNT

• Differ inFrames controlled and acknowledgement signal




Sequence Initiator

Sequence Recipient

Fabric

ACK

R_RDYR_RDY

R_RDY

R_RDYACK

EE_Credit

BB_C BB_C

TX Buf

TX Buf

TX Buf

TX Buf

RX Buf

RX BufRX Buf

RX Buf


Class of Service

• Applicable to all fabric topologiesSwitchedPoint to point

Arbitrated loop

• These three classes of service areClass 1 dedicated connection

Class 2 connectionless multiplexed

Class 3 datagram

• Delimiters used to set required class for a sequence



Class of Service

• SOF delimiterThe required class of service along with basic sequence management are specified in the SOF delimiter of every frame

The SOF delimiter dedicate basic link management functions within the fabric

The SOF delimiter identifies basic Sequence management functions within the destination N_Port in the initial frame of the sequence and the last frame of the sequence

• EOF delimiterLast frame of a sequence is terminated by a special EOF

Dedicated connections are removed by a special EOF


Class of Service

Class 1• Dedicated connection service

Connection oriented service between two N_Ports

Frames received in order transmittedGuaranteed delivery with notification of non-delivery

Guaranteed throughput

Optional IntermixCan mix Class 2 and 3 frames if allowed



Class of Service

Class 1• Requires explicit connection establishment

SOF(C1) delimiter

• Requires explicit removal of connectionACK with EOF(DT) delimiter

• Once connection is establishedBSY and RJT will not occur

• Flow controlBuffer to buffer on SOF(C1) frame: R_RDYEnd to end for all other data frames: ACK


Class of Service: Class 1 Flow

FabricInitiator Recipient

SOF(C1)

R_RDYACK

SOF(n1)

SOF(n1)

SOF(n1) ACK

ACK

R_RDY

Connection Requested

Connection Established

EOF(t)Conn Removed

ACK



Class of Service

Class 2• Multiplexed connectionless service

Connectionless oriented service between two N_Ports

Order of frame reception not guaranteedGuaranteed delivery

Notification of non-delivery

No throughput guaranteesOptional intermix


Class of Service

Class 2• Multiplex on a frame-by-frame basis

Between different destination N_Ports

Among different sequences

• BSY and RJT may occur on any frame

• Flow ControlBuffer-to-buffer for all frames: R_RDY

End-to-end for all data frames: ACK




FabricInitiator Recipient

SOF(C2)

R_RDYACK

SOF(n2)

SOF(n2)

ACK

ACK

R_RDY

R_RDY

R_RDY

R_RDY

R_RDY

R_RDYR_RDY

R_RDY


Class of Service

Class 3• Datagram multiplexed connectionless service

Connectionless oriented service between two N_Ports

Order of frame reception not guaranteedUnacknowledged

Delivery NOT guaranteed

No throughput guaranteesOptional intermix



Class of Service

Class 3• Multiplex on a frame-by-frame basis

Between different destination N_Ports

Among different sequences

• BSY and RJT will not occur on any frame

• Flow controlBuffer-to-buffer for all data frames: R_RDY



FabricRecipientInitiator

Data Frame

Data Frame

R_RDY

R_RDY

R_RDY

R_RDY



EE Credit

NL_Node “A”

NL_Node “B”

EE_Credit

Switch

EE_Credit

EE_Credit

Applies Only to Class 1 and Class 2 Frames for All Topologies


For All Class 2 and Class 3 Frames for All Topologies

BB Credit

BB_Credit

Switch

NL_Node “A”

NL_Node “B”

BB_Credit

BB_Credit


FRAME PROCESSING



Tables

• The N_Port will keep the following informationAvailable X_ID tableExchange context table

Login table



Tables

Available X_ID Table• This table contains a list of available X_ID’s

Can be used for OX_IDs or RX_IDs

A given implementation may choose to keep two tables one for OX_ID and RX_ID

• When a device driver sends a request to transmit a frame, a value will be taken for the OX_ID

• When a port receives a frame for a new exchange, a value will be taken for the RX_ID


Tables

Exchange Context Table• Each exchange ID points to a unique entry in the

exchange context table• Each entry contains the context and state

information for the particular exchangePort_ID involved in exchangeX_ID it assigned to exchangeULP and phase within the operationData source or destination addressData frames transmitted or received (SEQ_CNT)ACK frames transmitted or received (SEQ_CNT)



Tables

Login Table• This table contains one entry for each port to

which this port is logged in with

• Each entry contains service parameters and working EE_Credit count value


Data Frames: Putting It All Together

Data Frame Transmission• Request for a ULP

Initiate some operation with a specific destination port

• Login processIf you are not logged in, initiate login process

Build logging table entry for destination port

• Assign OX_ID if neededGet a value from the available X_ID Table

Build the exchange context table



Data Frames

Data Frame Transmission (Cont.)• Gather information

Exchange context tableReceive buffer size and destination port

Login tableWorking credit count of destination port

Set-up frame header

• Data frame transmissionSegmentation processCredit management


Data Frames

Transmit Request

• ULP passes a request to transmit a chunk of data to the N_Port

Destination Port D_ID is made

• The N_Port must access the “login table” to determine the service parameters on the destination port

Number of Rx buffersValue of the working credit count

And the rest



Data Frames

The Data Transmission• ULP data chunk is moved in frames with the

use of the sequenceAll within the context of the exchange

• A number of processes are involvedInitialization of the frame header fieldsSegmentation and reassemble


First and Last Data Frames

• The first data frame of a sequence is identified bySOF(Ix) Delimiter, where ‘x’ is the Class of Service

• The last data frame of a sequence is identified byF_CTL bit 19, End_SEQ=‘1’

• A sequence consists of all data framesStarting at the SEQ_CNT for the first frame through the SEQ_CNT of the last frame



Sequence Processing

Sequence Count• ULP chunk of data is transmitted IN ORDER

All Frames are sent in order

• Sequence_Count (SEQ_CNT)Frames are assigned sequentially increasing numbers as they are sent

The receiving N_Port will use the SEQ_CNT to insure that Frames are reassembled in order and back in its original chunk


Sequence Initiator (SI)

• Sets F_CTL bit 23“0” If it is the exchange originator“1” If it is the exchange responder

OX_ID and RX_ID set to assigned values

RX_ID = “FFFF” if first sequence of exchangeRouting field (R_CTL) set to “0000” to indicate FC-4 data frameInformation category field of R_CTL set according to payload



Sequence Initiator: Frame Header

• Sequence ID (SEQ_ID)Any value select that is not used

• Sequence count (SEQ_CNT)Assign sequentially as frames are sent

Starts with “0” on first frame of sequenceIncrements by ‘1’ while sequence initiative is held

• ParameterSet to ‘offset’ of the first byte of payload with respects to entire chunkOffset = ‘0’ on first frame and ‘1’ + for second and subsequent frames


Sequence Initiator: Frame Header

• Other important F_CTL bitsBit 23, exchange contextBit 21, first sequence

Bit 20, last sequence

Bit 19, end sequenceBit 16, sequence initiative

Used to pass initiative to other device



Automatic Processes

• These processes are automatic and are performed by the protocol chip

Segmentation and reassemblySEQ_CNT assignment

Higher layers are unaware of these processes


ULP Processing

• The Upper Level Protocol (ULP) uses these fieldsRouting ‘0000’ = FC-4 data frameType ’08 = SCSI/FCP

Info category

Identifies Specific Function of Payload‘01’ = Solicited Data

‘06’ = Unsolicited Command

‘05’ = Data Descriptor

‘07’ = Command Status


ARBITRATED LOOP



Fibre Channel Arbitrated Loop (FC-AL)

• Maximum bandwidth: 100 MB/sec. (shared amongst all nodes on loop)

• 126 nodes max on loop

• Can be combined with switches

• Attaches “NL_Ports”

• Number of nodes on loop directly affects performance

• Defined in it’s own standard

Fibre Channel Hub

FC

L

L

L

L

L

L

LLL



Loop Advantages

• Low cost solution with copper transceivers

• Eliminates the need for a discrete “fabric”Fabric routing decision distributed around the loop

• Compatible with all FC- 0 variantsCopper within a box

Optical between boxes

• Self discovery procedure

• Simple additions to FC-PH


Loop Advantages

• Port bypass network

• High availability configurations possible

• Supports both public and private loops

• Provides access fairness



NL_Port

• N_PortAttaches to the physical transport mediaProvides the Fibre Channel control and protocol

Provides the termination point for Fibre Channel

Resides within the node

• NL_PortProvides all functionality on N_Port with additional function of the loopAn NL_Port can function as a N_Port


FL_Port

• F_PortAttaches to the physical transport media at the edge of the switched fabric

• FL_PortThe switched fabric port which attaches to a loopF_Port functionality with additional function of the loop

G and GL Ports Will Do Both N and F



Private and Public

• Private LoopContains no FL_PortCommunications outside the loop via Fibre Channel is not possible

• Public LoopContains an FL_Port

Communications outside loop via Fibre Channel is possible

• Private DevicesDevices on a public loop may be private, i.e. do not login


Addressing

• Arbitrated Loop Physical Address (AL-PA or PA)Assigned during the loop initialization (soft addressing)A unique 8 bit value

127 valid values

• Arbitrated Loop Destination Address (AL-PD or PD)The AL_PA used to identify the destination L_Port

Target of a primitive signal or D_ID of a frame

• Arbitrated Loop Source Address (AL_PS or PS)The AL_PA used to identify the source L_PortSource of a primitive signal or S_ID of a frame



The Fabric Definition

• The entity that interconnects attached N_Ports

• Provides ‘routing’ based on destination address

• Fabric may be:Point to point—No routing requiredSwitched—Routing provided by the Switch

Arbitrated loop—Routing is distributed throughout the attached NL_Ports


Switched Fabric

N_Port N_PortN_Port N_Port N_PortN_Port

Fabric

Switch



Loop

NL_Port

Node

NL_Port

Node

NL_Port

Node

Node

NL_Port

NodeNL_Port

NodeNL_Port

Fabric

LOOP

= Arbitrated Loop Additional Function


Routing Process: Loop

• The routing function is distributedEach L_Port performs a portion of routing

• Routing is performed through out-of-band signaling using primitive signals

• Connection oriented independent of class of service

Obtain ownership of the loop (Arbitration)Establish a connection (Open)Transfer frames (Data)Remove the connection (Close)Relinquish the loop



Processes and Procedures

• InitializationThe process by which addresses are assigned and recovery is performed

• ArbitrationThe process by which an L_Port acquires ownership of the loop

• OpenThe process by which the L_Port which owns the Loop uses to select the L_Port to which it wants to communicate with

• CloseThe process by which the L_Port which owns the Loop releases control


Fill Words

• FC-PH defines two signals that may be transmitted between frames (when no other information is being transmitted)

IdleR_RDY

• FC-AL defines several additional signals that may be transmitted between frames

• FC-AL defines the “fill word” to beARB(F0)ARB(x)Idle



Primitive Signals and Sequences

• Primitive signalsArbitrateOpenCloseMark

• Primitive sequencesPort bypass enablePort bypass disableLoop initialization

FC-AL Defined the Following Unique Signals and Sequences


Credits Buffers

• Alternate BB_Credit management requested during login

• When activated service parameter BB_Credit = number of buffers available when circuit is established

• The receiving L_Port shall transmit R_RDYs for the additional buffers at anytime when “opened”

Used to pump up BB_Credit_CNT

• Transmitting L_PortDecrements BB_Credit by ‘1’ for each data frame TxIncrements BB_Credit by ‘1’ for each R_RDY RxStops transmitting when BB_Credit =‘0’

Loop Uses Same Credit Method as Previously Discussed But Also Has an Alternate Credit Model



Arbitrated Loop Initialization Procedure

Purpose• An L_Port will perform the loop initialization

procedure to:Determine the Operating environment for the L_Port; Is this a loop?

Acquire an address. AL_PA (Physical Address)Report that an error has been detected


Loop Commands

Command & AL_PA Distribute Position map

132Link Initialization – Loop PositionLILP

Command & AL_PA Collect Position map

132Link Initialization – Report PositionLIRP

Command & AL_PA bit map

20Link Initialization – Soft AssignedLISA


20Link Initialization – Hard AssignedLIHA


20Link Initialization – Previously AssignedLIPA


20Link Initialization – Fabric AssignedLIFA

Command & WWN12Link Initialization – Select MasterLISM

Payload ContentsBytesCommand

Loop Initialization Procedure—LIP Is an Ordered Set



LIP: Initialization Procedure

Phase A

Phase B

Phase C

Phase D

Phase E

Close

Start The Initialization Procedure

Select Temporary Loop

Master

AL_PA Mapping

Phase

Reporting Phase

Distribute AL_PA Map Phase

LIP

LISM

LIFA, LIPA, LIHA, LISA

LIRP

LILP

Start The Initialization Procedure

FL_Port Wins if Present Otherwise Lowest WWN Wins

Build the AL_PA bit Map in 4 Steps

Collect the AL_PA Position Map

Distribute the AL_PA Position Map


LIP: Phase A

Loop Initialization Primitive Sequence• Transmitted continuously by L_Port until it

receives the same LIP configurationLIP (F7F7) the L_Port is attempting to determine if this is a loop and to acquire an AL_PA

LIP (F8F7) the L_Port has detected a loop failure at its receiver prior to acquiring an AL_PA

LIP (F8) the L_Port (AL_PS) had detected a loop failure at its receiver

LIP (F7) the L_Port (AL_PS) has detected a performance degradation



LIP: Phase B

• Each L_Port will build the LISM with:AL_PA = ’00’ hex if FL_Port

’EF’ hex if NL_Port

D_ID = ‘0000’hex + AL_PA Example (0000EF)

S_ID = “0000’hex + AL_PAPayload = Command + WWN

Current Fill Word = Idle

• Each L_Port will continuously transmit a LISM

• Normal flow control rules are not in effect during initialization


LIP: Phase B (Cont.)

• Each L_Port monitors its receiverWill continue to transmit LISM if Your AL_PA + WWN is less then received AL_PA + WWNOtherwise pass the received LISM

• You are temporary loop masterIf the device receives a LISM identical to the one transmittedFL_Ports always win; If two or more FL _Ports; Lowest WWN wins and the others go non-participatingIf no FL_Port the NL_Port with lowest WWN wins

• Loop masterCurrent fill word would be ARB(F0)When ARB(F0)’s are received, go to phase C



LIP: Phase C

Loop Master Will Form the Initial “Bit” Map as Shown:

000000000000000000000000000000003

000000000000000000000000000000002

000000000000000000000000000000001

0000000000000000000000000000L0000

0781516232431Word

Where L = 1 Requesting F_Login of all NL_Ports

Bit Position = 127 vector corresponding to valid AL_PA’s

Word 0 bit 30 = lowest number ’00’hex

Word 3 bit 0 = high number AL_PA value ‘EF’hex

Set the bit = 1 that corresponds to it’s Fabric Assigned AL_PA

Bit Position

Lowest AL_PA

Highest AL_PA


LIP: Phase C

• Loop master will transmit the following three commands allowing an L_Port to choose a desired AL_PA

LIFA bit map primed with initial value

LIPA bit map primed with results of LIFALIHA bit map primed with results of LIPA

• Loop master will then transmit the LISA commandLISA bit map primed with results of LIFA allowing L_Ports which were unable to obtain their desired AL_PA to get a “soft assigned” AL_PA



LIP: Phase C

• Each NL_Port willReceive, possibly modify and retransmit the four Initialization Command framesSet the Current Fill Word (CFW) = ARB(F0)

• Modify the AL_PA bit map as followsSet one bit of the initialization command AL_PA bit maps based on history of AL_PA assignmentIf the bit map corresponding to a “desired” AL_PA has been set by an up-stream L_Port, this L_Port assumes a soft AL_PA by setting the first “0” bit=1 in the bit map of the LISA frameIf no bit positions were available in the LISA bit map, the L_Port will remain in non-participating modeAt most the bit map of one command will be modified by each L_port


LIP: Phase D

• The loop master will prime the AL_PA position map to:

Byte 0 = ‘01’ hexByte 1 = it’s AL_PABytes 2-127 = ‘FF’ hexThen transmit the LIRP with this position map

• Each NL_Port will:Increment the offset by one and store the offsetStore its AL_PA at the offsetRetransmit the updated LIRP frame

• The loop master will save the resulting loop position map



LIP: Phase E

• The loop master will transmit the LILP command with

Payload = AL_PA position map• Each NL_port will

Save the loop position mapRetransmit the LILP command

• When the loop master receives the LIILP command it will

Transmit a CLS and go to monitoring stateWhen each NL_Port receives a CLS they willRetransmit the CLS and go to monitoring state

Initialization Complete


LIP: Summary

A. LIP starts the initialization procedure

B. Select a temporary loop masterLowest AL_PA | WWN wins

C. Build a AL_PA bit mapEach L_Port indicates the AL_PA it selected in one of 4 requests by the loop master

D. Collect a AL_PA position mapEach L_Port reports its relative position from master and it’s AL_PA

E. Distribute the resulting AL_PA position map to each L_Port



Arbitration

• The process by which L_port request ownership of the loop based on primitive signals

AL_PAAL_PAD20.4K28.5MSB LSB

ARB(x)

Ordered Set


Arbitration

Loop Owner• The current loop owner is responsible for

Seeds the arbitration process with ARB(F0)

Blocks propagation of the received ARB(x) until it relinquishes the loop

• Initiates a new arbitration “window”If ARB(F0) is received by setting current fill word = IDLE

• Fairness variablesAccess

ARB_WON



Arbitration Process

• When a port is arbitrating it enters the arbitrating state

• The CFW is updated to the ports ARB(AL_PA) if the CFW is:

1. IDLE2. ARB(F0)3. ARB(FF)4. Lower-priority ARB (higher value AL_PA)

• Arbitration occurs even if a loop circuit exists between another pair of ports

• Once a port starts arbitrating itMust continue to arbitrate until it winsWithdraw if it knows that another port is arbitrating


Fairness

Access Fairness• Ports with higher-Priority AL_PA values could lock out lower

priority portsWhen they ARB they will always winLower Priority ports might never win Arbitration

• Access fairness limits how often a port can arbitrateThis is done by not arbitrating the loop until all other ports on the loop that are arbitrating have won; This is called a fair port

• Access fairness is based on “access” not “duration of usage”

Does not limit how long a port uses the loop• Fairness is recommended by the standard but not

mandatoryFL_Ports may be unfair but NL_Ports should be fair



Fairness

• The fairness is controlled by the FC-AL fairness algorithm called a fairness window

Window begins when the first port wins arbitration Ends when a port discovers that it was the last arbitrating portIDLE resets the fairness windowThe variables used are

Access = 0 for fairness window openAccess = 1 when NL_Port has won arbitration

• Fair ports can only arbitrate once per windowAfter winning arbitration they wait for the end of the window before arbitrating again

• Unfair ports can arbitrate at anytime


Open

If the Port Requires the Loop when It Wins ARB

• It sends an OPN(yx) or OPN(yy)y=destination port x=source portFull-Duplex establishes a point to point like circuit between the loop portsHalf-duplex restricts open recipient to transmit link control frames only

Cannot transmit device data framesUsed by designs that can not support simultaneous data frames Tx and Rx



Open

Selecting the Destination Port• Is the intended destination port on same loop or

connected via fabric switch?If the upper 16 bits of destination field (D_ID) are all zeros the port is on this private loop

If the upper 16 bits of the source(S_ID)are all zeros then the source port is a private port and can only talk to ports on same loop

If the upper 16 bits of the D_ID are the same as the upper 16 bits of the S_ID then they are both on the same loop or both are public and attached to the same FL_Port

• If none of these are true, the destination port is not on the same loop and must be accessed via FL_Port


Opening a Port on Same Loop

• Open Originator inserts the destination AL_PD in the OPN

• The AL_PD is obtained from the low-order 8 bits of the destination address in the frame header

• This process can be entirely by hardware



Opening a Port Off the Loop

• Originator inserts AL_PD of the FL_Port ’00’ in the AL_PD field of the OPN

• The FL_Port is opened and frames are sent to the FL_Port

• FL_Port and fabric forwards the frames using the destination address field

• FL_Port can send to multiple destination ports on the loop during this OPN


SWITCH FABRIC OPERATION

OPT-2T019899_06_2004_X



Switch Model

Port Port

Port Port

Connection Matrix

Fabric Controller

Connectionless Switch Matrix


Worldwide Names

• Each switch element is assigned a WWN at time of manufacture

• Each switch port is assigned a WWN at the time of manufacture

• During FLOGI the switch identifies the WWN in the service parameters of the accept frame

Fabric port and

Switch element

• These address assignments can then correlate each fabric port with the switch element



Switch Ports

• Four basic types of switch portsF_Port—Uses NOS/LOS to attach to single N_PortFL_Port—Uses LIP to attach 1 to 126 NL_Ports

E_Port—Uses NOS/LOS to interconnect switches (inter-link switch ISL)G_Port—Uses NOS/LOS can be a F or E port


Fabric Addressing

• The 24 bit address is partitioned into 3 fieldsDeviceArea

Domain

• This partitioning helps speed up routing

• Switch element assigns the address to N_Ports

• Address portioning is transparent to N_Ports

DeviceAreaSwitch Domain

8 bits 8 bits 8 bits

Switch Topology Model



Directory Server

• Repository of information regarding the components that make up the FibreChannel network

• Located at address ‘FF FF FC’ (Some readings call this the name server)

• Components can register their characteristics with the directory server

• An N_Port can query the directory server for specific information

Query can be the address identifier, WWN and volume names for all SCSI targets


Directory ServerCommand Requests

• Get objectsGA_NXT—Get all next

GFT_ID—Get FC-4 types

• Register objectsRFT_ID—Register FC-4 types

• Deregister objectsDA_ID—Deregister all

These Are Some of the More Used Commands Used to Query the Directory Server



Fabric Controller

• Each switch has a fabric controller

• Assigned address ‘FF FF FD’Every fabric controller in the fabric has the same addressIt is the N_Port within the switch

Responsible for managing fabric, initialization, routing, setup and teardown of Class-1 connections

• Responsible to receive request and generate responses for the switch fabric

Information must be consistent independent of which fabric controller responds to a request


Extended Link Services

• Extended link services provide a set of protocol functions used by the port to specify a function or service at another port

Usually sent from N_Port to F_port to perform needed requestThe R_CTL field of the first word will be set to 0x22 to indicate an extend link service requestMany ELS services will return a payload in response some have no reply



Extended Link Services

• Some of the more important and most used ELS commands are:

FLOGI F_Port Login PLOGI N_Port Login

FAN Fabric Address Notification

PRLI Process Login PRLO Process Logout

SCN State Change Notification SCR State Change Registration

RSCN Registered State Change Notification


ELS: FLOGI

• FLOGI—Fabric loginIssued by N_Port to destination ‘FF FF FE’ to

Determine if fabric is present

Establish a session with the fabric

Exchange service parameters with the fabricFLOGI assigns N_Ports 24 bit address to N_Port or AL_PA to loop ports



ELS: PLOGI

• PLOGI—N_Port loginEstablished sessions between two N-PortsRequired before upper level protocol operations can begin

N_Port will register to the name server ‘FF FF FC” in fabric with all required login parameters N_Port will then query name server for other N_Ports on the fabric


ELS: PRLI

• PRLI—Process LoginAllows the FC-4 levels to exchange service parameters for communications between each other

Process is protocol specific (type field)

SCSI-3 FCP mapping requires PRLI



ELS: FAN

FAN—Fabric Address Notification• Used in fabric loop attached topology

• Provides mechanism for FL_Port to notify NL_Ports of addresses and names of FL_Ports along with fabric name

• Allows NL_Ports to verify configuration following a loop initialization


ELS: SCN

SCN—State Change Notification• Provides notification to ports of events that may

effect logins or process logins to ports on the fabric• SCN can be sent from

N_Port to N_PortN_Port to fabric controllerFabric controller to N_Ports

• Notification may indicate login session is no longer valid

Loss of signal (NOS, LOS, FLOGI)LIP has occurredSCN sent to fabric controller



ELS: RSCN

RSCN—Registered State Change Notification• Similar to SCN but only sends change notice

to those ports registered

• SCN did not define a registration method


Class_F Service

• Communications between switch elements use Class_F Service

Unique SOF delimiter and normal EOF delimiter

• Used to pass control information within the switch

• Highest priority within switch

• Connectionless service

• Has no meaning outside switch, N_Port will discard if received



Inter-Switch Link

• The interconnection between switches is called the inter-switch link

E_Port to E_Port

• Supports all classes of serviceClass 1, 2, 3, and switch to switch control traffic, class F

• FC-PH permits consecutive frames of a sequence to be routed over different ISL links for maximum throughput


Interswitch Links (ISLs)

• Inter-switch link (ISL) connects switches

• Fabric parameters must match on both switch otherwise link would not come up and fabric will be segmented



Principal Switch Selection

• Only one switch is designated principal switch in a fabric

• The switch with the lowest WWN becomes the principal switch originally

• Principal switch makes sure that no new switch is added to the fabric if it has a domain ID conflict with an existing switch in the fabric

Switch 1

Switch 3

Switch 4 Switch 5

Switch 6

Switch 2


Fabric Configuration Process

• The fabric configuration process enables a switch port to determine its operating mode, exchange operating parameters, and provides for distribution of addresses

• The process is summarized in the following stepsEstablish link parameters and switch port operating mode

Principal switch selectionDomain ID distribution

Path selection



Fabric Configuration Stages

Switch is operations with routes established

Path selection (FSPF) is defined in the next section

Switch has a Domain_ID

Path Selection

Switch has a Domain_IDSwitch requests a Domain_ID from the Domain Address Manager

Domain Address Manager had been selected

Domain ID Acquisition

The Principle Switch is selected

Switch_Names are exchanged over all ISLs to select a Principle Switch, which becomes the Domain Address Manager

BF or RCF SW_ILS transmitted or received

Select Principle Switch

Switch Port mode is known. If a Port is an E port, link parameters have been exchanged & Credit has been initialized.

The Switch Port attempts to discover whether it is an FL, F, or E port.

Switch Port has achieved word synchronization

Establish Link Parameters and Switch Port Operating Mode

Ending

Condition

ProcessStarting

Condition

Operation


Fabric Configuration: PS Selection

• A principal switch shall be selected whenever at least one inter-switch link (A link between two E_Port) is established

• The selection process chooses a principal switch, which is then designated to assign domain identifier to all the switches in the fabric, and any who join later the fabric later on



Fabric Configuration: PS Selection

• The principal switch selection can be triggered by anyone of the following events

Switch boot and EFPBuild Fabric (BF)

Reconfigure Fabric (RCF)


Fabric Build Process

• When the switch first boots up and the first E_Port of a switch becomes operational, the switch starts 2xF_S_TOV timer and then sends out an exchange fabric parameters (EFP) from that port containing its own Destination ID (DoID) in the list trying to become Principle Switch (PS)

• The switch receiving the Exchange Fabric Parameter (EFP) replies with either ACCept or ReJecT after comparing the priority and WWN

0x11 Record Len(0x10) Payload Len

Reserved Priority

Principal Switch WWN (Word 0)

Principal Switch WWN (Word 1)

Domain_ID record 0

Domain_ID record M

A (0) (FF, Aa)

B (0) (FF, Bb) C (0) (FF, Cc)

E_Port E_Port

Domain_id

Priority

WWN

EFP EFP

E_Port

EFP

E_Port

EFP

EFP

D (0) (FF, Dd)

EFP EFPEFP



Fabric Build Process

• If the received information has a lower value, the switch keeps the received information and considers sending switch as potential principal switch and also consider that link to be potential upstream link

• At that point switch generates another EFP for all other links with the updated potential principal switch

• When 2x F_S_TOV expired, all switches in the fabric consider the information collected for the principal switch to be definitive; At that point the principal switch is responsible for assigning the Domain_IDs

A (0) (FF, Aa)

B (0) (128, Aa) C (0) (FF, Aa)

E_Port E_Port

Domain_id

Priority

WWN

SW_RJT

SW_RJT

E_Port

SW_ACC

Potential Upstream port

SW_ACC

E_Port

EFP EFPEFP

Potential Upstream port

D (0) (FF, Bb)


Fabric Configuration Details

• After the principal switch selection, the PS will change its priority to 0x02 (PS Priority) and then assign itself a domain ID and then the process of domain distribution starts

• The principal switch will initialize the process by sending Domain ID Assigned (DIA) SW_REQ out of all its E_Port

• The intermediate switch is actively involved in this process• Each switch will reply back with Request Domain ID (RDI)

To allow each switch to request for one or more domain ID

The neighboring switch receiving RDI will be able to identify its downstream principal ISL

• Each switch can send many RDI but once the principal switch has granted the domain ID to the switch, the following RDI from the switch must request the same set of domain_id



A (1) (XX, Aa)

B (2) (FF, Aa) C (4) (FF, Aa)

E_Port E_Port

DIA

DIA

E_PortUpstream port

E_PortUpstream port

D (3) (FF, Aa)

A B DDIA (SW_REQ)

SW_ACCSW_RJT

RDI (SW_REQ)

SW_ACCSW_RJT DIA (SW_REQ)EFP (SW_REQ)Contains DoID list

SW_ACCSW_RJT

RDI (SW_REQ)

RDI (SW_REQ)

SW_ACCSW_RJT SW_ACC

SW_RJTEFP (SW_REQ)Contains DoID list EFP (SW_REQ)Contains DoID list

Fabric Configuration Flows ID Assignment

SW_ACC

SW_ACC

SW_ACC

BD

A


Fabric Configuration: The PS Battle

• After the principal switch selection and domain id assignment, all switches in the fabric will start two processes

FC_ID assignmentFSPF path selection

• When a new switch is added to the fabric, it will send out an EFP with its local value (I am PS); the fabric rejects that EFP and replies with DIA telling the new switch to send RDI; the RDI is then routed to the current PS

• If the new switch is part of another fabric (it also has a PS) then both fabric sends out an EFP and after comparing the DoID list the fabric enters one of the following states

BF state: If the DoID list does not overlapRCF State: If the DoID list overlapIsolation: No auto-reconfigure state or RCF disabled



Fabric Configuration Disruptive/Non-Disruptive

• One of the following three conditions can trigger BF (non-disruptive) or RCF (disruptive)

Two disjoints fabric are combined togetherA principal ISL fails (upstream or downstream)A switch with Domain_ID request for another Domain_ID

• Whenever a switch receives a BF/RCF, the switch starts F_S_TOV timer and enters the BF/RCF state; It forwards BF/RCF out of all E_ports except the incoming port (only once) and wait for the timer to expire

• When the timer expires, BF/RCF propagation state is left and principal switch selection begins

• BF is not a disruptive process• RCF is a disruptive process


Propagation of BF or RCF Requests

Fabric Configuration Distribution

Switch Starts theReconfig



Fabric Configuration: Reserve ID’s

• N_ports and E_ports get one port ID; F ports don’t get any ID’s; FL ports in public AL gets 0x00 port ID

Fabric F_Port, Fabric Login databaseFEFFFF

N_Port of fabric controllerFDFFFF

Well Known AddressF0-FC,FFFFFF

Reserved00-EFFFFF

Reserved00-FFFD-FEFF

ReservedF0-FFFCFF

N_Port of domain controller. Port ID is the domain ID01-EFFCFF

Reserved00FCFF

Multicast & Broadcast00-FFFBFF

Reserved00-FF00-FAFF

Reserved00-FF00-FFF0-FE

N_Port & E_Port. Port ID=00 for FL port for public devices 255 address00-FF00-FF01-EF

Reserved00-FF01-FF00

ReservedNonAL_PA0000

Private Loop NL_PortAL_PA0000

Used during FLOGI000000

DescriptionPort_IDArea_IDDomain_ID


Fabric Configuration: FSPF

• FSPF stands for fabric shortest path first

• Based on link state protocol

• Begins after domain ID assignment is completed

• Conceptually based on open shortest path first (OSPF) internet routing protocol

• Currently a standard defined in FC-SW-2




• FSPF has four major componentsHello protocolReplicated topology database

A path computation algorithm

Routing table update

• FSPF discovers the paths to switches using Domain—Ids

• Each switch performs its own shortest path calculations



• For FSPF a domain ID identifies a single switchThis limits the max number of switches that can support in the Fabric to 239 when FSPF is supported

• FSPF performs hop-by-hop routing

• FSPF supports hierarchical path selectionProvides the scalable routing tables in large topologies




• Everyone says HELLO to their neighbor, on all initialized ISLs

• The neighbors say HELLO back, unless they are dead

• When the HELLO packet is received with both originator and recipient domain id, the two way communication is done and:

The ISL is active

The ISL may be available as a two-way path for frames


Fabric Configuration: Hellos

• Point to Point Only• Default Hello Int = 20 S• Default HelloDead Int = 80 S

Hello Protocol



Fabric Configuration LinkState Update and Ack

• After a 2-way HELLO is established on a Link, each switch exchanges its entire database with its neighbor using the LSU service

• When the recipient of the LSU has processed the database, it sends back the LSA service

BALSU(DB-A)

LSU(DB-B)

LSA(DB-B)

LSA(DB-A)


Fabric ConfigurationLink State Record

• When the databases are in sync, each switch sends its LSR with the new link included using the LSU service

• The LSU is flooded to the entire fabric

• Each Switch retransmits the LSU by a mechanism called “reliable flooding”

BALSU(LSR-A)

LSU(LSR-B)

LSA(LSR-B)

LSA(LSR-A)



Fabric Configuration

• Link cost is calculated based on baud rate of the link, plus an administratively set factor

• Link cost = S * (1.0625E12/baud rate)S is administrative factor defaults to 1Ex: Link cost of 1G port = 1000

• Path cost is the sum of the traversed link costs

• Lower metric more desirable


Fabric Configuration

FSPF routes are fully functional

Routes are calculated and set up within each switch. Links are maintained by sending HELLOs every Hello_Interval. Link databases are maintained by flooding link updates as appropriate

Initial Database Exchange has been completed

Running State

Link State Databases have been exchanged

LSU SW_ILS frames are exchanged containing the initial database

Two communication has been established

Perform Initial Database Exchange

Two way communication has been established

HLO SW_ILS frames are exchanged on the link until each switch has received a HELLO with a valid neighbor Domain field

The switch sending HELLO has a valid Domain_ID

Perform initial HELLO Exchange

Ending Condition

ProcessStarting Condition

Operation

Path Selection (FSPF) Operation Summary



FSPF Characteristics

• Uses FSPF as the routing algorithm

• FSPF routes traffic based on destination domain ID

• FSPF uses total cost as the metric to determine most efficient path

• Static routes can be applied


FSPF Characteristics

Paths:• Finds the shortest path to each domain, then programs the

hardware routing tablesRoutes:• Dynamically

Round robin• Statically

Administrator can configure the routeAutomatically re-routes upon ISL going away and static routing will again take effect upon ISL return

• Automatic failover• Fault detection 150 ms• Self heals in 500 ms• So, alternate route is live in 650 ms



Routing Software Configurable Parameters

• Link cost

• Static routes

• In Order Delivery (IOD)

• Timers (be careful)


What Is a Route and Path?

• A route is map between the input and output E_port used to reach the next switch

• A path is a map through the topology between a source and destination

FCFC

FCFCFCRoutePathISL



Selecting a Path

• Each inter switch link has a cost metric• The cost of an ISL is related to the bandwidth• The total cost of a path between two switches is the sum

of the cost of all the traversed ISLs• The path to a destination switch is the one with the

minimum total cost• More than one path can be selected (with the same cost)

Cost 500

Cost 250Cost 250

FCFCFC

FCFC


ISL Oversubscription

• Oversubscription occurs when more nodes can contend for the use of one ISL

• Oversubscription ratio is the number of different ports that contend for the use of one ISL

• This a 3:1 over subscription

Multiple Nodes

Switch

Switch

ISL

1G 1G

1G

1G



FC ERROR MANAGEMENT

OPT-2T019899_06_2004_X


Timers

• Four different timers usedReceiver-transmitter time-out (R_T_TOV)Error detect time-out (E_D_TOV)

Resource allocation time-out (R_A_TOV)

Connection request time-out (C_R_TOV)Used in Class 1 You will never see class one



Timers: R_T_TOV Receiver-Transmitter Time-out

• Used to time events at the link levelLoss of synchronizationTimes Responses for link reset protocol

• Generally controlled in hardware for all link configurations

Default value in FC Standard is 100ms


Timers: E_D_TOVError Detect Time-out

• Timers for events and responses at the sequence level

Missing ACK or R_RDY when buffer credit has reached zero

Class 1 or 2 expects response from data frames

N_Port logout

• Timer value is set at fabric login to accommodate the network environment for better scaling according to delivery time of frames

Default is 10 sec



Timers – R_A_TOVResource Allocation Time-out

• Time-out value for how long to hold resources associated with a failed operation

Needed to free shared resources for reuse

• Value to determine how long a port needs to keep responding to a link service request before an error is detected

R_A_TOV is 2 times E_D_TOVDefault setting in Pt to Pt is 20 sec and fabric is 120 seconds


Timers: CR_TOVConnection Request Time-out

• Determines how long the fabric can hold a class-1 request in the queue during connection establishment

• Allows for separation of the time in a stacked queue from the E_D_TOV; This separates queuing time from frame transit time

• Helps in controlling F_BSY issues



Recovery: Class 3

• Errors in class 3 sequence can only be detected by the Sequence recipient because there are no ACKsor rejects in class 3

• Class 3 SR will discard single or multiple frames until the exchange is terminated

• The upper level recovery may retransmit the entire Sequence or at least the sequence following the error detection


Recovery: Class 3

• Errors a class 3 operation can detect:Out of order delivery and potential missing frame based on SEQ_CNT

If missing frame is not Rx’ed within E_D_TOV

Indication of a new sequence when last frame of previous Sequence has not been received (in-order delivery set)

Relative offset not in order with an order delivery set



Abort Sequence: ABTS

• ABTS protocolUsed to terminate sequence or exchangeTransmitted by the sequence initiator

Can be requested by the sequence recipient by setting bits within the F_CTL of the ACK frameSame class of service delimiter as the sequence being aborted is used for ABTS frame


Timers

• Four different timers usedReceiver-transmitter time-out (R_T_TOV)Error detect time-out (E_D_TOV)

Resource allocation time-out (R_A_TOV)

Connection request time-out (C_R_TOV)Used in Class 1 You will never see class one



Timers: R_T_TOV Receiver-Transmitter Time-out

• Used to time events at the link levelLoss of synchronizationTimes Responses for link reset protocol

• Generally controlled in hardware for all link configurations

Default value in FC Standard is 100ms


Timers: E_D_TOVError Detect Time-out

• Timers for events and responses at the sequence level

Missing ACK or R_RDY when buffer credit has reached zero

Class 1 or 2 expects response from data frames

N_Port logout

• Timer value is set at fabric login to accommodate the network environment for better scaling according to delivery time of frames

Default is 10 sec



Timers – R_A_TOVResource Allocation Time-out

• Time-out value for how long to hold resources associated with a failed operation

Needed to free shared resources for reuse

• Value to determine how long a port needs to keep responding to a link service request before an error is detected

R_A_TOV is 2 times E_D_TOVDefault setting in Pt to Pt is 20 sec and fabric is 120 seconds


Timers: CR_TOVConnection Request Time-out

• Determines how long the fabric can hold a class-1 request in the queue during connection establishment

• Allows for separation of the time in a stacked queue from the E_D_TOV; This separates queuing time from frame transit time

• Helps in controlling F_BSY issues



Recovery: Class 3

• Errors in class 3 sequence can only be detected by the Sequence recipient because there are no ACKsor rejects in class 3

• Class 3 SR will discard single or multiple frames until the exchange is terminated

• The upper level recovery may retransmit the entire Sequence or at least the sequence following the error detection


Recovery: Class 3

• Errors a class 3 operation can detect:Out of order delivery and potential missing frame based on SEQ_CNT

If missing frame is not Rx’ed within E_D_TOV

Indication of a new sequence when last frame of previous Sequence has not been received (in-order delivery set)

Relative offset not in order with an order delivery set



Abort Sequence: ABTS

• ABTS can be sent under abnormal conditionsEnd-to-end credits not requiredSequence initiative not required

Open sequence not required

Maximum number of concurrent sequences allowedUnidirectional for class 1 connection

The reply to an ABTS is a Basic_Accept

iSCSIRFC 3720




Session Modules

• What is iSCSI and what is the big picture?

• iSCSI protocol Introduction

• The iSCSI connection

• Security, data integrity and error recovery

• iSCSI protocol details in-depth

• Simple iSCSI connection flows

• Service location protocol for IP storage


What Is iSCSI?

• A SCSI transport protocol that operates on top of TCP

Encapsulates SCSI-3 CDBs (Control Descriptor Blocks) and Data into TCP/IP byte-streams (defined by SAM-2—SCSI Architecture Model 2)

Allows IP hosts to access IP or Fibre Channel-connected SCSI targets

Allows Fibre Channel hosts to access IP SCSI targets

• Standards statusRFC 3720 (assigned May 2004)

Major industry support (Cisco, IBM, EMC, HP, Microsoft)



Storage Technology SCSI Domain

• To be functional, a SCSI Domain needs to contain a SCSI device that contains a target and a SCSI device that contains an Initiator

SCSI DeviceInitiator

Service DeliverySubsystem

SCSI DeviceTarget

Port

Port

Device Service RequestDevice Service Response

Task Request

Task Response TaskManager

LogicalUnit 1DeviceServer

ApplicationClient

ApplicationClient


SAN, NAS, iSCSI Comparison

Adopted from IBM Redbook “IP Storage Networking: IBM NAS & iSCSI Solutions”

DAS

SCSI

Computer System

SCSI Bus Adapter

SCSI Device Driver

Volume Manager

File System

Application

SAN

SAN

FC

Fibre Channel HBA

SCSI Device Driver

Volume Manager

File System

Application

iSCSI Appliance

IP

File System

Application

SCSI Device DriveriSCSI DriverTCP/IP stack

NIC

Volume Manager

NICTCP/IP stackiSCSI layerBus Adapter

iSCSI Gateway

IP

FC switch

File System

Application

SCSI Device DriveriSCSI DriverTCP/IP stack

NIC

Volume Manager

NICTCP/IP stackiSCSI layerBus Adapter

NAS

IP

NICTCP/IP stack

I/O Redirector

File System

Application

NFS/CIFS

NICTCP/IP stackFile System

Device driver

File I/O

Block I/O

Block I/O



IP Storage Networking

• IP storage networking provides solution to carry storage traffic within IP

• Uses TCP: A reliable transport for delivery• Can be used for local data center and long haul applications• Two primary protocols:

IP TCPTCP FCIPFCIP FCFC SCSI Data

IP TCPTCP iSCSIiSCSI SCSI Data

iSCSI—IP-SCSI—Used to Transport SCSI CDBs and Data within TCP/IP Connections

FCIP—IP-SCSI—Used to Transport SCSI CDBs and Data within TCP/IP Connections


Initiator and Target Model for iSCSI

• Initiator—SCSI device which is capable of originating SCSI commands and task management requests

• Target—SCSI device which is capable of executing SCSI commands and task management requests

iSCSIGateway

iSCSIInitiator

iSCSITarget

FCTarget

FC Initiator

iSCSIGateway

FCInitiator

FCTarget

iSCSITarget

iSCSI Initiator

iSCSI Target Mode iSCSI Initiator Mode



iSCSI Components

• iSCSI is an end-to-end protocol

• iSCSI has human readable SCSI device (node) naming

• iSCSI includes the following base components: IPSEC connectivity securityAuthentication for access configuration

Discovery of iSCSI nodesProcess for remote boot

iSCSI MIB standards


iSCSI: Internet SCSI PDU

• The iSCSI layer encapsulates the SCSI CDB into a iSCSI Protocol Data Unit (PDU) and forwards it to the Transmission Control Protocol (TCP) layer

• It also extracts the CDB from an iSCSI PDU received from the TCP layer, and forwards the CDB to the SCSI layer

• iSCSI mapping provides the SCSI-3 command layer with a reliable transport

• The communications between the Initiator and target will occur over one or more TCP connections

• The TCP connections form a session and will carry the iSCSI PDU’s; the sessions are given an ID called a connection ID (CID); there are two parts of the ID, Initiator Session ID (ISID) and Target ID (TSID) and together make up an “I_T nexus”



Data Server HostInitiator iSCSI Target

FC Storage DeviceFC Target

ApplicationClient

requests data from

LUN 1

ApplicationClient

requests data from

LUN 1

Port

Port

Port Device Service Request

Device Service Response

LogicalUnit 1

LogicalUnit 1

DeviceServerDeviceServer

LogicalUnit 2

LogicalUnit 2

DeviceServer

ge2

ge2

fc1

fc1

Target Mapping

LUN 1 = LUN 2LUN 1 = LUN 2

iSCSI Model

SCSI CDB’s Carried in iSCSI PDU’s

SCSI CDB’s carried by Fibre Channel Exchange and Sequences


IPIP

iSCSI Stack

SCSI Applications (File Systems, Databases)SCSI Applications (File Systems, Databases)

SCSI Block CommandsSCSI Block Commands SCSI Stream CommandsSCSI Stream Commands Other SCSI CommandsOther SCSI Commands

iSCSISCSI Over TCP/IP

iSCSISCSI Over TCP/IP

TCPTCP

EthernetEthernetFibre ChannelFibre ChannelParallel SCSI InterfacesParallel SCSI Interfaces

FCP SCSI Over FCFCP SCSI Over FCParallel SCSI TransportParallel SCSI Transport

SCSI Commands, Data, and StatusSCSI Commands, Data, and Status

SCSI Device-Type Commands

SCSI Generic Commands

SCSI Transport Protocols

Layer 3 Network Transport

Layer 2 Network



Sourced Port Destination PortDestination Port

Sequence Number

Acknowledgment Number

Window

Checksum

Offset Reserved U A P R S F

Options and padding

Urgent Pointer

DestinationAddress

SourceAddress Type IP TCP DataData

46–1500 bytes

8 6 6 2

FCS

4 Octet

Preamble

TCP Header

iSCSI encapsulated

iSCSI encapsulated

OpcodeOpcode Opcode Specific FieldsOpcode Specific Fields

Length of Data (after 40Byte header)Length of Data (after 40Byte header)

LUN or Opcode-specific fieldsLUN or Opcode-specific fields

Initiator Task TagInitiator Task Tag

Opcode Specific FieldsOpcode Specific Fields

Data Field …Data Field …

Well-known Ports:21 FTP23 Telnet25 SMTP80 http

3260 iSCSI3260 iSCSI

iSCSIiSCSI Packet


iSCSI Naming and Discovery RFC 3721

• Initiator and target require iSCSI namesName is location independentiSCSI node name = SCSI device name of iSCSI device

Associated with iSCSI nodes, not adapters

Up to 255 byte displayable/human readable string (UTF-8 encoding)

Use SLP, or iSNS, or query target for names (sendtargets)

• Two iSCSI name types:iqn—iSCSI qualified name

eui—Extended Unique Identifier (IEEE EUI-64)



iSCSI Name Structure

iSCSI Name Structure

Type . Unique String

iqn.1987-05.com.cisco.1234abcdef987601267da232.bettyiqn.2001-04.com.acme.storage.tape.sys1.xyz

Type Date OrganizationNaming Authority

Subgroup Naming Authority or String Defined by Organization Naming Authority.. .iqn

eui

Date = yyyy-mm when Domain Acquired Reversed Domain Name

Type EUI-64 Identifier (ASCII Encoded Decimal).eui.02004567a425678d

Host Name


iSCSI Naming and Addressing Terms

• iSCSI host nameName of computer

• iSCSI initiator name (iSCSI Node)Name created at iSCSI driver load time on host system

• Initiator—Target Session ID (SSID)One or more TCP connections between Initiator and target; This session ID is derived from iSCSI host name, iSCSI target name and TSID, ISID




• iSCSI initiator addressIP address on Initiator interface; Initiator can have multiple addresses

• Initiator port—Also known as network portalIP address on initiator no port number assigned, again Initiator can have several network portals

• Target port—Also known as network portalIP address + TCP port number on target interface

There can be more then one target interface



• iSCSI target nameUsed to identify multiple SCSI targets behind a single IP address+port; This name is globally unique

• Initiator session IDThis is an initiator-defined session identifier; It will be the same for all connections within a session; An iSCSI initiator port is uniquely identified by the value pair (iSCSI Initiator Name, ISID)

• Target session IDTarget assigned tag for a session with a specific named initiator that, together with the ISID uniquely identifies a session with that initiator




• iSCSI network entity—Client

• It is a combination of the following:

iSCSI initiator

iSCSI hostiSCSI initiator address

Initiator port (network portal)

• iSCSI network entity—Server

• Is a combination of the following:

iSCSI target name

Target port (network portal)

Initiator—target session (SSID)



• iSCSI NodeiSCSI Initiator or iSCSI Target; There can be one or more iSCSI nodes in a network entity

iSCSI node will equal

iSCSI initiator nameiSCSI target name




• Portal GroupGroups multiple TCP connections across the same session that is is sent across multiple portals

The portal groups are identified by a portal group tag (1-65535)One or more portal groups can provide a path to the same iSCSI node (target node or initiator node)SendTargets requires portal group tag


iSCSI Discovery Methods

• Small networksStatic configuration, initiators and targets‘SendTargets’ command makes configuration easier

• Medium-sized networksService Location Protocol (SLP multicast discovery)

• Large-sized networksiSNS (internet storage name service)

Includes soft zone domains

Includes database for ongoing management



iSCSI Architecture

Network Portal10.1.30.1


iSCSI Node(Initiator)



iSCSI Node(Target)

iSCSI Node(Target)

Network Entity (iSCSI Client)

Network Entity (iSCSI Server)


iSCSI Architecture



iSCSI Session (Target Side)iSCSI Name + TSID=2

iSCSI Target NodeNode(within Network Entity)

Portal Group 1


Portal Group 2

iSCSI Session (Target Side)iSCSI Name + TSID=1

IP Network



iSCSI Session Model

• An iSCSI session exists between a single iSCSI initiator (host) and a single iSCSI target (iSCSI router)

• An iSCSI session consists of one or more iSCSI (TCP) connections• Login phase begins each connection• Deliver SCSI commands in order

iSCSI (TCP) Connection

iSCSI Session

iSCSIStorage Router

iSCSI Routing Instance

TCP/3260TCP/3260TCP/3260


iSCSI Session Images

• Across all connections within a session, an initiator sees one “target image”

• The target image would represent all identifying elements such as LUN’s

• A target also sees one “initiator image” across all connections within a session



3.3.3.3 4.4.4.4 5.5.5.5

Target-1 Target-2 Target-3

Target-5Target-4

iSCSI Target name configured on iSCSI

Device

iSCSI Node

s

Target Port These Network Portals listens for iSCSI connections

on WKP 3260

iSCSI Network Entity-Server

iSCSI Initiator address

Initiator Port

iSCSI Host NameiSCSI Initiator Name

IP TCP Connection

TSID

ISID

iSCSI Driver, Storage NIC

Storage Systems

Configuration will make the

connections between storage

and iSCSI Initiator

SSID

1.1.1.1 2.2.2.2

mike.cisco.com

disk.cisco.com.stor.123

Put It All Together for iSCSI


iSCSI Connections and SCSI Phases

• A SCSI command and its associated data and status phase exchanges must traverse the same TCP connection

• Linked SCSI commands can traverse separate TCP connections for scalability

iSCSI (TCP) Connection 1iSCSI (TCP) Connection 1

iSCSI Session

iSCSI Storage Router

iSCSI RoutingInstance

SCSI Command (1) (Read)SCSI Data (1)

SCSI Status (1)

iSCSI (TCP) Connection 2iSCSI (TCP) Connection 2

Linked SCSI Commands

SCSI Command (1) (Write)SCSI Data (1)

SCSI Status (1)



iSCSI Connection Session

• iSCSI Session has four phasesInitial login phaseSecurity authentication phase

Operational negotiation phase

Full featured phase

Session Can Process SCSI Commands and Data after Login Is Complete


iSCSI Session Establishment

Login Begins with the First Connection• Initial login phase

Initiator sends login with text strings for InitiatorName, TargetName, and authentication options (which are then selected by the target)

• Security authentication phaseAuthentication text exchanges (ID, password, certificates, etc)

• Operational negotiation phaseEach side (initiator and target) negotiate the supported options using Keyword=value, or Keyword=value,value,value

Amount of unsolicited bufferTypes of data deliverySolicited, unsolicited, immediate, etc…

• Full featured phaseCan carry SCSI CDBs/data, task management, and responses



iSCSI Session Key Points

Sessions:• iSCSI Session = a group of TCP connections linking

an initiator with a target (i.e., can be one or more connections)

• NOTE: A TCP connection that is part of an iSCSI session will only be used to carry iSCSI traffic

• The iSCSI initiator and target use the session to communicating iSCSI commands, control messages, parameters, and data to each other

• TCP connections can be added and removed from a session using the iSCSI Login/Logout commands


iSCSI Sessions

• During session establishment, the target identifies the SCSI initiator port (the “I” in the “I_T nexus”) through the value pair (InitiatorName, ISID)

• Any persistent state (e.g., persistent reservations) on the target associated with a SCSI initiator port is identified based on this value pair

• Any state associated with the SCSI target port (the “T” in the “I_T nexus”) is identified externally by the TargetName and portal group tag and internally in an implementation dependent way



iSCSI Connection Allegiance

• For SCSI commands that require data transfer, the data phase and status phase must be sent over the same TCP connection used by the command phase

• Consecutive commands that are part of a SCSI task may use different connections within the session (linked commands)

• Connection allegiance is strictly per-command and not per task

• Multiple connections allow the iSCSI session to be scaled across multiple links/devices


iSCSI Connection Termination

• Session may end with logout or I/O error causing dropped connection

• TCP connections are closed through normal methods i.e. TCP FINs

• Graceful shutdowns can only occur when no outstanding tasks are on the connection and not in full-feature phase

• Termination of connection abnormally may require a recovery method by logout request for all connections; This prevents stale iSCSI PDU’s being received after going down

• Logout can also be issued by the target through asynchronous message PDU



iSCSI Security

• Two types of security IPSec secures TCP/IP nodes; setup at TCP/IP startup—before iSCSI login

Session authentication via IKE (Internet Key Exchange)

Packet by packet authentication (also provides Integrity)Privacy via encryption (also provides Integrity)

See SEC-IPS iSCSI techniques (done/setup during iSCSI Login)

Authentication (ensures nodes are authorized to use the iSCSI target node) may use SRP, Chap, or Kerberos


Challenge Handshake Authentication Protocol

• In-band initiator-target authentication

• IP-SEC is not assumed

• No clear text password accepted

• Compliant iSCSI initiators and targets MUST implement the CHAP (RFC1994)

• Implementations MUST support use of up to 128 bit random CHAP secrets



iSCSI Security

• Various levels of security can fit different topologies

Examples:Secure main floor—No security

Campus LAN—iSCSI authentication and CRC32c (digests)Remote private WAN—IPSec with session/packet authenticationRemote internet WAN—IPSec with privacy encryption


iSCSI Data Integrity

• Basic level of end-to-end data integrity can be reasonably handled by TCP using the standard checksum

• iSCSI CRC32c digest checks for Integrity beyond TCP/IP XOR checksum

a) Header digest

b) Data payload digest



Digests (Checksums)

• Optional header and data digests protect the integrity of the header and data, respectively; The digests, if present, are located, respectively, after the header and PDU-specific data, and cover the proper data and the padding bytes

• The existence and type of digests are negotiated during the login phase

• The separation of the header and data digests is useful in iSCSI routing applications, in which only the header changes when a message is forwarded; In this case, only the header digest should be recalculated

• Digests are not included in data or header length fields• A zero-length data segment also implies a zero-length

data-digest


Error Recovery

Two Considerations for Errors• An iSCSI PDU may fail the digest check and be

dropped, despite being received by the TCP layer; The iSCSI layer must optionally be allowed to recover such dropped PDUs

• A TCP connection may fail at any time during the data transfer; All the active tasks must optionally be allowed to be continued on a different TCP connection within the same session



Error Recover: iSCSI Initiator

A. NOP-OUT to probe sequence numbers of the target

B. Command retry

C. Recovery R2T support

D. Requesting retransmission of status/data/R2T using the SNACK facility

E. Acknowledging the receipt of the data

F. Reassigning the connection allegiance of a task to a different TCP connection

G. Terminating the entire iSCSI session to start fresh


Error Recover: iSCSI Target

A. NOP-IN to probe sequence numbers of the initiator

B. Requesting retransmission of data using the recovery R2T feature

C. SNACK support

D. Requesting that parts of read data be acknowledged

E. Allegiance reassignment support

F. Terminating the entire iSCSI session to force the initiator to start over



Error Recover Classes

• Within a command (i.e., without requiring command restart)

• Within a connection (i.e., without requiring the connection to be rebuilt, but perhaps requiring command restart)

• Connection recovery (i.e., perhaps requiring connections to be rebuilt and commands to be reissued)

• Session recovery


Error Levels

• Level determined during logon text negotiationError recovery level is proposed by an originator in a text negotiation


iSCSI PROTOCOL DETAILS IN-DEPTH



iSCSI Key Points

• Tasks:A linked set of SCSI commandsOne and only one SCSI command at a time can be processed within any given iSCSI task

• Initiator Task Tag (ITT) and Target Transfer Tag (TTT)Initiator tags for all pending commands must be unique initiator-wide

SCSI Data PDUs are matched to their corresponding SCSI commands using tags specified in the protocol

ITT for unsolicited dataTTT for solicited data



iSCSI Key Points

Solicited or unsolicited messages:• Initiator to target

User data or command parameters will be sent as either solicited data or unsolicited dataSolicited data is sent in response to ready to transfer (R2T) PDUsUnsolicited data can be part of an iSCSI command PDU (“Immediatedata”) or an iSCSI data PDUThe maximum size of an individual data PDU or the immediate part of the initial unsolicited burst may be negotiated during login

• Target to initiatorReady to transfer (R2T) message to Initiator, requesting data for a write commandCommand responsesAsynchronous messages (SCSI and iSCSI) describing an unusual or error event


iSCSI Numbering

• iSCSI uses command and status numberingCommand numbering—Session wide and is used for ordered command delivery over multiple connections within a session; It can also be used as a mechanism for command flow control over a sessionStatus numbering—per connection and is used to enable recovery in case of connection failure

• Fields in the iSCSI PDUs communicate the reference numbers between the initiator and target

During periods when traffic on a connection is unidirectional, iSCSI NOP PDUs may be issued to synchronize the command and status ordering counters of the initiator and target



SCSI Command Numbering and Acks within iSCSI

• Initiator and target device have three sequence number registers per session

CmdSN—Current command sequence number; Sent by initator

ExpCmdSN—Expected command by the target; Sent to the initiator by the target to acknowledge CmdSN; Can be used to ACK several sequences

MaxCmdSN—Maximum number target can receive in its queue; Can be sent to Initiator from target to adjust queue size


SCSI Command Numbering and Acks within iSCSI

• iSCSI supports ordered command delivery within the session

• Command-Sequence-Number (CmdSN) is assigned by initiator and carried in the iSCSI PDU

• CmdSN starts at iSCSI login

• CmdSN not assigned to data-out (DataSN used)

• Immediate delivery does not advance CmdSN

• iSCSI must deliver commands to target in order of CmdSN and will not increment until executed state by target



SCSI Status Numbering and Acks within iSCSI

• Status Sequence Number (StatSN) is used to number responses to the Initiator from the target

• ExpStatSN is sent by Initiator to acknowledge status

• Status numbering starts after Login; During login there can be only one outstanding command per connection


Initiator iSCSI OPcodes

0x00 NOP (No operation, used as ping to target gateway)

0x01 SCSI command (Indicates encapsulated iSCSI packet has a SCSI CDB for target device)

0x02 SCSI task management command

0x03 iSCSI login

0x04 text command

0x05 SCSI data-out (Write data to target device)

0x06 iSCSI logout

0x10 SNACK (Request retransmission from target)

0x1c-0x1e Vendor specific codes



Target iSCSI OPcodes

0x20 NOP-In (No operation in, used for ping response from target

0x21 SCSI response (Indicates encapsulated iSCSI packet has status or from target device)

0x22 SCSI task management response

0x23 login response

0x24 text response

0x25 SCSI data-in (Read data from target)

0x26 logout response

0x31 Ready to transfer (Sent to initiator from target to indicate it is ready to receive data)

0x32 async message (Message from target to indicate special conditions)

0x3c-0x3e vendor specific codes

0x3f reject


iSCSI PDU’s

• Several different types of iSCSI PDUs used, each of the different iSCSI Operation Codes (Opcodes) determine what iSCSI PDU to use; Some of the more used PDUs are:

Login and logout PDU

Command and response PDUData-In and data-out PDU



iSCSI Login PDU

If Set to 1 = Recovery from Failed Connection

If Set to 1 Indicates Initiator Is Ready to

Transit to Next Stage

Initiatior May Provide Initial Text Parameters

in This Area

Unique ID for This

Connection

Current Stage/Next Stage0 – Security Negotiation

1 – Login Operational Negotiation- 3 – Full Feature Phase

Initiator ID for This

Connection


iSCSI Login

• Login Phase used to:Enable TCP connection (Target listens on well known port)Authentication (CHAP)

Negotiate session parameters

Open security protocolsMark the TCP connection as a iSCSI sessionand assign IDs



iSCSI Text Mode

During Login Some Sessions or Connection Parameters May Be Negotiated in a Text Format

list = values sent in order of preference

Example of values can be:MaxConnections=<1-65535> T or ISendtargets=all I only

Targetname=<iSCSI-Name> T or I

SessionType=<Discovery|Normal> I onlyOthers—addressed later in slides (see RFC)


iSCSI Full Feature Phase

• iSCSI PDUs can be sent

• PDUs must flow over same connection as login

• Size of PDU is negotiated during login

A Connection Is in Full Feature Mode after a Completed Login



Data Sequencing within iSCSI

• The iSCSI PDUs used for data input and output are the 0x05 iSCSI command and the 0x25 iSCSI command, along with R2T (0x31 ready to transfer)

DataSN is a number field and advances by 1 for each input (read) and output (write)Targets will operate in two modes, solicited (R2T) or unsolicited (non-R2T)

Target operating in R2T mode can only receive solicited data from the initiatorR2TSN advances by one for each received R2T during the data transfer

• The DataSN and R2TSN fields are for the initiator to detect missing data


Data-Out PDU

Final Bit Say This Is the Last PDU of

a Sequence

Data Segment Length Based on Capabilities

Exchange

LUN Number for Data



Data-In PDUFinal Bit say this is the last read of a sequence Acknowledge Bit

used when error recovery level is 1

or higher

Status bit tells that there is meaningful data in the

StatSN, Status, and Residual Count fields

Flags valid when S bit is set tells

how to read Residual Count


iSCSI Read Command Example

1. Initiator sends iSCSI command PDU (CDB=Read)2. Target sends iSCSI data-in PDU(s)3. Target sends iSCSI response PDUNotes:• Solicited data via read command PDU (Initiator requests data

from the target)• Target may satisfy the single read command with multiple iSCSI

data read PDUs (PDUs can be out-of-order)• Command is not complete until all data and status is received

by the initiator• Good status can be sent within the last iSCSI data-in PDU• All iSCSI data-in PDUs and the response PDU will be delivered

on the same TCP connection that the command was sent on• All data-in PDUs will carry the same value in the ITT field



SCSI Command PDULets Target Know if

More Data Is to Follow along with

Expected Data Transfer Length

R=1 If the Command Is Expected to Input Data

W=1 If the Command Is

Expected to Output Data

Task Attributes See RFC for Detailed

Meaning

16 bytes of SCSI CDB,

Some SCSI Commands Have Additional Data and This Field Is Used for the

Accompanied Data

CRC If Capabilities

Required This


SCSI Response PDU

Ox00 = Command Completed at Target0x01 = Target Failure0x08 – 0xff = Reserved for Vendor Response

SCSI Status per SAM2

CRC Check Sums



SCSI Status and Response Fields for iSCSI OpCode 0x21

• The status field of the iSCSI PDU is used to report status of the command back to the initiator

• The specific status codes are documented in the SCSI architectural model for the device

• Response field contains the iSCSI codes that are mapped to the SAM-2 response


Ready to Transfer PDU

• When the initiator has sent a SCSI write command to the target the target can specify the blocks be delivered in a convenient order; This information is passed to the initiator in the R2T PDU

• Allowing an initiator to write data to a target without a R2T is agreed upon during login

• The target may send several R2T PDUs and have several data transfers pending if allowed by the initiator



Task Management

• Functions to provide the initiator a way to control management of the target device

Abort the TASKClear allegiance

Logical reset

Target reset

• Each of these and more are broken down in detail in the iSCSI RFC


SACK, NOP-IN, NOP-OUT

• SACKOptionalUsed to request retransmission of numbered responses, data or R2T PDUs from the target

• NOP-INSent by a target as a response to a NOP-Out, as a“ping” to an initiatorOr a means to carry a changed ExpCmdSN and/or MaxCmdSN if there is no other PDU to carry them for a long time

• NOP-OUTUsed by Initiator as a “ping command”, to verify that a connection/session is still active and all its components are operationalUsed to confirm a changed ExpStatSN if there is no other PDU to carry it for a long time



Message Synchronization and Steering

• Steering of iSCSI out of order TCP segments into pre-allocated buffers instead of temporary buffers

• To decrease reassembly time

• Not needing to rely on message length information

• Provides a synchronization method using fixed interval markers telling where the start of the next iSCSI PDU is in the buffer

• Optional for iSCSI RFC


List of Negotiated Parameters Prior to Going into Full Feature Mode

Header Digest

Data Digest

Max Connections

Send Targets

Target Name

Initiator Name

Target Alias

Initiator Alias

Target Address

Target Portal Group Tag

Initial Ready 2 Transfer

Immediate Data

Max Rec Data Segment Length

Max Burst Length

First Burst Length

Default Time 2 Wait

Default Time 2 Retain

Max Outstanding R2T

Data PDU In-order

Data Sequence In-order

Error Recovery Level

Session Type



Standards: Where to Find Details

• http://www.ietf.org/html.charters/ips-charter.html

• T10 Technical committee—www.t10.orgTechnical committee of the National Committee on Information Technology Standards (NCITS), deals with the storage devices

• T11 Technical committee—www.t11.orgTechnical committee of the NCITS, deals with the physical interface and transport level

SIMPLE ISCSI CONNECTION FLOWSEXAMPLE OF DISCOVERY SESSION WITH CHAP




Establish Initial TCP Session Phase

0X03 Command—LoginKey Values Are Sent, InitiatorName, InitiatorAlias,

SessionType=Discovery, AuthMethod=CHAP/none, HeaderDigest, DataDigest

TCP Port 3260TCP Port 3260

TargetInitiatorTCP port 1026

(Random) Discovery Session

TCP port 1026 (Random) Discovery Session

iSCSI Driver

This Device

Has Already

Initialized Onto the

Fibre Channel

This Device

Has Already

Initialized Onto the

Fibre Channel

iSCSI Flows

0X23 Login ResponseStatus= Accept Login (0X0000), Keyvalues Are Sent,

AuthMethod=CHAP, HeaderDigest=none, DatDigest=none

0X03 Command—LoginKey Values Sent, InitiatorName, InitiatorAlias,

SessionType=Discovery, CHAP_A=5 (CHAP with MD5)

0X23 Login ResponseStatus=Accept Login, KeyValues CHAP_A, CHAP_I & CHAP_C


0X23 Login ResponseFinal PDU in Sequence, Status= Accept login (0X0000)

0X23 Login ResponseStatus=Accept Login, Negotiate Session Parameters

0X03 Command—LoginKey Values Are Sent, InitiatorName, InitiatorAlias,

SessionType=Discovery, CHAP_R, CHAP_N

iSCSI Flows

0X03 Command—LoginKey Values Sent, InitiatorName, InitiatorAlias,

SessionType=Discovery, Negotiate Session Parameters

End of Authentication Phase Start of Parameter Negotiation Phase

for Discovery Session




iSCSI Driver




0X24 Login ResponseFinal PDU in Sequence, KeyValue=TargetName (iqn number

along with target name configured on iSCSI Target)

0X04 Text CommandSendTargets=all




iSCSI Driver


iSCSI Flows

0X03 Command—LoginKey Values sent, InitiatorName, InitiatorAlias,

SessionType=Normal, TargetName, AuthMethod=CHAP,none

0X23 Login ResponseStatus=Accept Login, AuthMethod=CHAP

Establish TCP connection for target

TCP port 1027 (random)

Target Session #1

TCP port 1027 (random)

Target Session #1

Note the Addition of

Another TCP Session

Start of Target Session Authentication and Target Session Parameter Negotiation


0X23 Login ResponseStatus=Accept Login, KeyValues CHAP_A, CHAP_I & CHAP_C

0X03 Command—LoginKey Values are sent, InitiatorName, InitiatorAlias,

SessionType=Normal, CHAP_R, CHAP_N

0X03 Command—LoginKey Values are sent, InitiatorName, InitiatorAlias, SessionType=Normal, TargetName, CHAP_A=5

TargetInitiator

iSCSI Driver


iSCSI Flows

0X23 Login ResponseStatus=Accept Login

TCP Port 1027 (Random)

Target Session #1


Target Session #1

0X03 Command—LoginKey Values sent, InitiatorName, InitiatorAlias,

SessionType=Normal, TargetName, Negotiate session Parameters



TargetInitiator

iSCSI Driver



Target Session #1


Target Session #1

0X01 iSCSI CommandSCSI Inquiry CDB 0X12

iSCSI Flows

0X23 Login ResponseStatus=Accept Login, Negotiate session Parameters

0X25 iSCSI Data-in (read)


FCIP CONCEPTS

OPT-2T019899_06_2004_X



Agenda

• What FCIP Is About

• The StandardsFibre Channel T11 StandardsIETF IPS Working Group Drafts

• Understanding FCIP Protocol

• Relationships to Other SCSI Transport Technologies


FCIP: Fibre Channel over IP

• FCIP provides a standard way of encapsulating FC frames within TCP/IP, allowing islands of FC SANs to be interconnected over an IP-based network

• TCP/IP is used as the underlying transport to provide congestion control and in-order delivery of error-free data

• FC frames are treated the same as datagrams

• It is not iFCP, mFCP, IPFC, iSCSI transports or extended FC fabric



FCIP Design

IP NetworkTunnel SessionTunnel Session

IP ServicesAvailable at Aggregated

FC SAN Level

FC TapeLibrary

FC TapeLibraryFC Server

FCServer FC

ServerFCJBOD

FCJBOD

FC Switch

FC SwitchFC Switch

FC SwitchFC SwitchFC Switch

FC SwitchFC Switch

FSPF RoutingBackbone




Fiber Channel

SAN

Fiber Channel

SAN

FCIPTunnel

FCIPTunnel

FC Server

Tunnel SessionTunnel Session


Four (4) Specifications Define Basic FCIP

• ANSI: http://www.t11.org/index.htmFC-SW-2 describes the operation and interaction of FibreChannel switches, including E_Port, B_Port and fabric operation

FC-BB-2 is a mapping that pertains to the extension of Fibre channel switched networks across a TCP/IP network backbone and defines reference models that support E_Port and B_Port

• IETF IPS working group:Fibre channel over TCP/IP covers the TCP/IP requirements for transporting Fibre Channel frames over an IP network

FC frame encapsulation defines the common Fibre Channel encapsulation format



ANSI: FC-SW-2 Standard

• E_Ports are used at both ends of an Inter Switch Link (ISL)

• E_Ports forward user traffic (storage data) and control information (class F SW_ILS frames containing FSPF, zone exchanges, etc.)

• FC-SW-2 defines fabric merge procedures (Domain_ID assignment, zone transfers, etc.)

• FC-SW-2 also defines FSPF


ANSI: FC-SW-2 Essentials (Recap)

• E_Ports provide switch-to-switch connectivity

• E_Ports negotiate parameters such as:ELP—Exchange Link ParametersESC—Exchange Switch Capabilities

• FSPF is enabled over E_Ports only

• Separate fabrics can be merged over E_Ports

• Zoning information is exchanged over E_Ports



IETF FCIP: Fibre Channel Over IP

• Each interconnection is called a FCIP link and can contain one (1) or more TCP connection(s)

• Each end of a FCIP link is associated to a virtual ISL link (VE_Port or B_Access Portal)

• VE_Ports communicate between themselves just like normally interconnected E_Ports by using SW_ILS: ELP, EFP, ESC, LKA, BF, RCF, FSPF, etc.

• B_Access portals communicate between themselves by using SW_ILS: EBP, LKA

• The result (when all goes well… ) is a fully merged Fibre Channel fabric between FC switch SAN’s


IETF FCIP

• IETF draft standard that allows IP connectivity to link Fibre Channel storage area networks across WANs

Two methods can be used

1) Similar to Cisco STUN—Nailed up tunnel2) Similar to DLSW—Dynamic peering method

We will visit the details of each in later slides

• draft-ietf-ips-fcovertcpipDraft 12 is current, will RFC Jan/Feb 2003



FCIP Architecture Model

FCIP FCIP

FC-2 FC-2

FC-1

FC-0

FC-1

FC-0

TCP TCP

IP IP

LINK LINK

PHY PHY

TCP/IPNetwork

SAN SAN

FCIP Link

Key:

FC-0—Fibre Channel Physical Media Layer

FC-1—Fibre Channel Encode and Decode Layer

FC-2—Fibre Channel Framing and Flow Control Layer

TCP—Transmission Control Protocol

IP—Internet Protocol

LINK—IP Link Layer

PHY—IP Physical Layer


FCIP

• End-station addressing, address resolution, message routing, and other fundamental elements of the network architecture remain unchanged from the Fibre Channel model, with IP introduced exclusively as a transport protocol for an inter-network bridging function

• IP is unaware of the Fibre Channel payload and the fibre channel fabric is unaware of IP

EthernetHeader

EthernetHeader

//

CRCCRCIPChecksum

IPChecksum

TCP FCIPFCIP SCSI Data … FCPFCP

//



FCIP

• FCIP only supports class 2, class 3, class 4, and class F frames

• No FC primitive signals or primitive sequences supported

Physical signal sets used by FC ports to indicate events, i.e. NOS, OLS, LR

• IP transport is transparent to FibreChannel topology


Understanding FCIP Terms

• FC end node—A Fibre Channel device that uses the connection services provided by the FC fabric

• FC entity—The Fibre Channel specific functional component that combines with an FCIP entity to form an interface between an FC fabric and an IP network

• FC fabric—An entity that interconnects various Nx_Ports attached to it, and is capable of routing FC frames using only the destination ID information in a FC frame header

• FC fabric entity—A Fibre Channel specific element containing one or more Interconnect_Ports (see FC-SW-2) and one or more FC/FCIP entity pairs

• FC frame—The basic unit of Fibre Channel data transfer• FC frame receiver portal—The access point through which an FC frame and

time stamp enters an FCIP data engine from the FC entity• FC frame transmitter portal—The access point through which a reconstituted

FC frame and time stamp leaves an FCIP data engine to the FC entity

• FC/FCIP entity pair—The combination of one FC entity and one FCIP entity



Understanding FCIP Terms (Cont.)

• FCIP data engine (FCIP_DE)—The component of an FCIP entity that handles FC frame encapsulation, de-encapsulation, and transmission FCIP frames through a single TCP connection

• FCIP entity—The entity responsible for the FCIP protocol exchanges on the IP network and which encompasses FCIP_LEP(s) and FCIP control and services module

• FCIP frame—An FC frame plus the FC frame encapsulation header, encoded SOF and encoded EOF that contains the FC frame

• FCIP link—One or more TCP connections that connect one FCIP_LEP to another

• FCIP link endpoint (FCIP_LEP)—The component of an FCIP entity that that handles a single FCIP link and contains one or more FCIP_DE’s

• Encapsulated frame receiver portal—The TCP access point through which an FCIP frame is received from the IP network by an FCIP data engine

• Encapsulated frame transmitter portal—The TCP access point through which an FCIP frame is transmitted to the IP network by an FCIP data engine

• FCIP special frame (FSF)—A specially formatted FC frame containing information used by the FCIP protocol


FCIP DiagramFC Entity

TCPPorts WKP = 3225

VE_Port

FCIP_LEP

DE DE

FCIP Entity

TCPPorts WKP = 3225

VE_Port

FCIP_LEP

FCIP Physical Link

TCP Connection FC Frames in TCP/IPClass 3 and Class F Can Beon Separate Ports or Connections

IP Address = 172.16.0.5Ethernet Gigabit/WAN Interface

FCIP Link

Virtual ISL

IP Address = 192.168.1.10Ethernet Gigabit/WAN Interface

FCIP Link End PointFCIP Data Engine(Detail)

FCIP Frame TX RX Portal

FC Frame Receiver Portal

FC/FCIP Entity Pair

TX RX

More than One TCP Connection Is Allowed

Dynamic CONNECTION PORT for FCIP Connections

Non Dynamic Connections

FCIP Data Engine



ANSI Meets IETF E-Port

• FC-BB-2• FCIP


ANSI Meets IETF B-Port

• FC-BB-2

• FCIP



FCIP Standards Stack Details

This Will Be the ISL Connection Either a Bridged connection or

E_Port; Depending on FCIP Implementation selected by Vendor


Additional IETF Drafts

• SLP: Service Location Protocoldraft-ietf-ips-fcip-slp Used for dynamic discovery of FCIP ports

• IPSec for storagedraft-ietf-ips-securityMore details later on this requirement for FCIP

• MIBsdraft-ietf-ips-scsi-mibdraft-ietf-ips-fcmgmt-mibdraft-ietf-ips-fcip-mib

• FC-BBPublished ANSI project being superseded by BB-2



ANSI: FC-BB-2 Essentials(FCIP E-Port)

• Defines a slightly complex model;

• FC-BB-2 covers the FC portion of this model (FC entity and some of above)

• Cisco’s FCIP E_Port implementations will closely follow this model


IETF: FCIP Essentials(FCIP E-Port)

• FCIP follows the model proposed in FC-BB-2;

• FCIP covers the lower portion of this model (FCIP entity and below)

• Cisco’s FCIP E_Port implementation will follow this model



ANSI: FCIP Essentials(FCIP B-Port)

• Again the FC side of the this model follows SW-BB-2 standards

• With B_Port there is no FC switching element so the B_Port device will not be seen as a switch in the fabric but as a passive device


IETF: FCIP Essentials(FCIP B-Port)

• The FCIP part of the B-Port operation is the same as FCIP for the E_Port

• Note in this diagram that implementations of this standard can be any number of ports from 1 to n



About FCIP Links

• The FCIP interface represents both the VE_Port and the FCIP link

• An FCIP link is defined as one or more TCP connections

• FCIP link endpoint (LEP) terminates FCIP links

• FCIP data engine: One per TCP connection

Entity 1

TCPPorts WKP = 3225

VE_Port

FCIP_LEP

DE DE

FCIP Link

Class F Class 3

IP Address = 192.168.1.10TCP/IP Network Interface


About the FC Entity

• FC entity interfaces (internally) with FCIP entity

• FC entity components:Control and ServicesModuleProvides FC frame and timestamp along with synchronization with FCIP entity Correct order delivery of FC framesWorks with FCIP entity for flow controlComputes end-to-end transit timeThrows away expired framesAnswers to authentication of TCP connection request



About the FCIP Entity

• FCIP entity interfaces (internally) with FC entity

• FCIP entity components:

Provide FC frame and timestamp to FC entityTells FC entity about discarded bytesTells FC entity about new and lost TCP connections and reason codesMonitors special frame changesMakes request to FC entity for authentication


FCIP Link Endpoint: Details

• FCIP_LEP is the translation point between an FC entity and an IP network

• LEP coordinates between FC and TCP flow control mechanisms



Error Detection and Recovery

• Data engine uses various methods to detect errors but does not correct errors

• Rather, it inserts EOFa (abort) frame delimiters when possible

• Requests sent up to FC entity to handle recovery


IETF: Fibre Channel Frame Encapsulation Header

• Defines the encapsulation header for Fibre Channel frames

• Not specific to FCIP• Includes timestamp, CRC and provision for special frames



Initialization of Port

B_Port• Link initialization

• Exchange link parameters

• Link reset

E_Port• Link initialization• Exchange link parameters• Reset link• Exchange switch

capabilities• Exchange fabric

parameters• Assign domain IDs• Establish routes• Merge zones if required


Link Initialization Flow

NOS = Not Operational Sequence

OLS = Offline Sequence

LR = Link Reset

LRR =Link Reset Response

AC = Activity State



OL = Offline State

LFLF

LROL

LR

AC

AC

NOS

OLS

LR

Idle

LRR

Idle

E_Port on Switch

B_Port or E_Port on FCIP Device

These Are All Special Ordered Sets of 8B/10B Coding



Link Capture

E_Port on Switch

B_Port or E_Port on FCIP Device

LR & LRR to Initialize Flow

Control Parameters Per

FC-PH

NOS

LR

IDLE

R-RDYIDLE

IDLE

IDLE

R-RDYIDLE

LRR

At this Point B_Port Device Is Up and E_Port to E_Port Exchange Continues


ISL E_Port If It Is an E_Port FCIP Device or If the B_Port Is Now up the Switch to Switch Exchange Continues



ELP Data Bit 15 of flag will be a 1 for

B_Port

RA_TOV is fabric wide timer, ED_TOV

is per Link

Credit value is one to start to allow only one

out standing frame during link start-up

Class 2 & 3 supported

PWWN & WWN,Vendor ID also


E_Port and B_Port Summary

FC SANFC SAN

FC Switch FC Switch

FCIP E-Port FCIP E-Port

IP Network

VE - Port

Exchange Fabric Parameters

Exchange Fabric ParametersExchange Fabric Parameters

Exchange Link Parameters Exchange Link ParametersExchange FCIP-Link Parameters

ESCESCESC

FC SANFC SAN

FC Switch FC Switch

FCIP E-Port FCIP E-Port

IP Network

VB - Port

Exchange Link Parameters Exchange Link ParametersExchange FCIP-Link Parameters

Exchange Fabric Parameters

ESC (Exchange Switch Capabilities) if required

B Port Operation

E Port Operation

FCIP B-Port FCIP B-Port

7200 w/ PA-FC-1G 7200 w/ PA-FC-1G



FCIP: ISL Connection

• The E-Port or B-Port FCIP Connection Will Provide:Simple name service across the IP tunnelFC discovery between SAN islands

FSPF routing services between fabric switches

Management server informationBuffer credits


Comparisons

B-Port and E-Port Differences



FCIP Connection Establishment

• Non-dynamicTCP connection to a specific IP address

• DynamicDiscovery of FCIP entities using SLPv2

• Use of FCIP special frame

• Use of options


Non-Dynamic TCP Connections

• The FCIP entity is informed of a TCP connection is needed (Most likely done by configuration parameters in the device)

• IP address and security features are established (Configured)

• Destination WWN is determined (Configured)

• TCP/IP parameters are set (Configured)

• Quality of service is determined (Configured)

• Connection request is made to Port 3225 or configured port



Dynamic TCP Connections SLPv2

• IP security for SLP determined

• Enter FCIP discovery domain process

• Advertise availability to SLP discovery domain service agent

• Locate FCIP entities in the discovery domain as a SLP user agent

• For each discovered entity follow same process as non-dynamic method to establish connection


FCIP Special FrameTCP Connection Is Established

Sending Side• First frame sent after TCP connection is established

• Sending side waits for FSF echo (90 seconds)

• Echo is match or non-match (Non-match terminates TCP connection)

• Creation of FCIP_LEP and FCIP_DE

• Inform FC Entity of connection and usage flags



FCIP Special FrameTCP Connection Is Established

Receiving Side (Listening)• Listen for connections on WKN port 3225

or configured port• Checks database to allow connection• Checks security features• Wait for FSF frame (90 seconds)• Inspect FSF contents and send echo frame

Connection nonceDestination FC fabric entity world wide nameConnection usage flagsConnection usage code


FCIP Special Frame Details

• Used to exchange WWNs, entity pair identifiers, TCP connection identifiers and to except or reject connection

• Identify what kind of traffic (SOFi3, SOFn3, EOF) is intended; Not enforced

• In conjunction with connection usage flags, connection usage code help FCIP entity apply proper QoS parameters for the connection

• Adjustments to FSF with use of change bits can be made when frame is echoed back

• If two entities are trying to send FSF connection frames simultaneously first to Rx echo wins



• The first frame transmitted in each direction is a special frame used to identify the peers FCIP entities and to synchronize

Connection Nonce (random number)

proto0x01

version0x01

~proto0xFE

~version0xFE

Reserved0x0000

Reserved0xFFFF

ConnUsage flags

0x00 Connectionusage code

proto0x01

version0x01

~proto0xFE

~version0xFE

Flags0x00

Frame Len 0x12

~Flags0x3F

~Frame Len 0x3ED

CRC (Reserved in FCIP)0x00-00-00-00

Source FC Fabric Entity WWN(identify the fabric)

Source FC/FCIP Entity Identifier

Destination FC Fabric Entity WWN

Reserved0x0000

Reserved0xFFFF

0

1

2

3

4/5

6

7

8/9

10/11

12/13

14

15/16

17

Timestamp integer/fraction

~pFlagspFlags 0x00 0x00

1) Special Frame Sent

3) FCIP Tunnel Setup Complete

2)Special Frame echoed

Fibre Channel

IP WAN

Fibre Channel

FC

FC

I Am WWN1, This Is my FC/FCIP

IdentifierAre You Fabric

WWN2?

Ok WWN1, I Am WWN2

Let’s Setup theConnection

FCIP: Tunnel Setup as Proposed in FCIP Draft

FCIP Device


pFlag Breakdown



FCIP Header Format

• FCIP header used after FSF exchange is completed

Ones Compliment for Synchronization and Error Checking


Connection Options

• TCP selective acknowledgement (SACK)Per RFC 2883

• TCP window scale option

• Protection from sequence number wrap (PAWS)

• TCP keepalives (KAD)

• Flow control mapping between TCP and Fibre Channel



FCIP Security Requirements (Per Draft)

To Support IP Network Security FCIP Entities MUST:• Implement cryptographically protected

authentication and cryptographic data integrity keyed to the authentication process, and Implement data confidentiality security features

• FCIP utilizes the IPSec protocol suite to provide data confidentiality and authentication services, and IKE as the key management protocol

• FCIP Security compliant implementations MUST implement ESP and the IPsec protocol suite based cryptographic authentication and data integrity [11], as well as confidentiality using algorithms and transforms as described in this section


FCIP Security Requirements(Per Draft) (Cont.)

• FCIP implementations MUST meet the secure key management requirements of IPsec protocol suite

• FCIP entities MUST implement replay protection against ESP sequence number wrap

• FCIP entities MUST use the results of IKE phase 1 negotiation for initiating an IKE phase 2 “quick mode” exchange and establish new SAs

Note: An External Device May Be Used in Conjunction with the FCIP Implementation to Meet the “Must Implement ESP” Requirement



Important FC and FCIP Timers

• Resource Allocation Timeout Value (R_A_TOV)Timeout value that determines how long a FC frame can be in transit on the Fibre Channel network This is a fabric wide value with a default value usually at 120 sec on switch networks

• Error Detect Timeout Value (E_D_TOV)A value that times events and responses at the link level; Errors at the link level will cause delays of these eventsThis value is defaulted to 10 sec and should be lower then R_A_TOV; Again this is a fabric wide setting

• Keep Alive Timer K_A_TOVA value that is applied to TCP connection and is used when no data is present


Time Stamps and Synchronization

• Clock synchronization is required if timestamps are used

Synchronized to FC servicesSynchronized to IP NTP

• Transit time through IP network is applied via a timestamp Integer

• If no timestamp value is available zero will be used• Fibre channel time values still apply across the ISL

link and are timed-out via lack of RDY coming back• End system devices such as HBA attached hosts

still require normal responses to timers end-to-end (no spoofing)



Timestamps

• TS are the responsibility of the FC entity

• This allows transit through the FCIP entity to be included in the measurement

• This transit time should be well below R_A_TOV


Buffer Credits

• Fibre channel buffer credit methods do not change

• R_Rdy’s will be used to control flow coming from FC switch on a per link basis

• Buffer credit establishment is determined at FLOGI

• Mechanisms to control flow of R_Rdy’s to FC switch based on TCP/IP congestion is per FCIP solution

• FC switches do not require extended credit methods



Error Recovery

• Errors on FC side of local B_Port are not forwarded over the IP network; Issues such as loss of sync or a FC encapsulation error will not be set to the FC entity

• Errors on IP side are handled by TCP and frame is dropped if checksum is in error


Summary

• FCIP is the standards approach to connect FibreChannel ISLs over TCP/IP LAN/WAN connections

• State of draft wording will most likely stay as it is worded today

• Security, network delay and error recovery will be biggest concerns

• No shipping product today conforms to the proposed FCIP draft

• Cisco will have several platforms supporting FCIP solutions



INTERNET FIBRE CHANNEL PROTOCOL

OPT-2T019899_06_2004_X


iFCP

FC-1

FC-0

TCP

IP

LINK

Gateway RegionPHY

iFCP

FC-1

FC-0

TCP

IP

LINK

Gateway RegionPHY

IP Network

FC-4

iFCP Protocol Model

• iFCP replaces the transport layer of Fibre Channel (FC-2) with an IP network but keeps the FC-4 mapping the existing Fibre Channel transport services on TCP/IP

• iFCP processes differently FC-4 frame images (applications), FC-2 frame images (link service request), FC broadcast and iFCP control frames

• Topology within the gateway regions are opaque to the IP network and other gateway regions (they appear just like collection of N_Ports)



IFCPGatewayGateway Region

IFCPGatewayGateway Region

iSNS

N_port-to-N_port session

N_port-to-N_port session

IP Network

iSNS Queries

iSNS Queries

IFCPGateway Gateway Region

IFCPGateway

Gateway Region

iFCP Network Model: iSNS Role

• An iFCP gateway cannot operate without access to an iSNS server

• Client-Server architecture• iSNS functions:

Device Discovery and fabric managementEmulation of the services provided by the FC name server and RSCNDefinition and management of discovery domainsDefinition and management of “logical fabrics”


iFCP Protocol Description: N_Ports Addresses Allocation

• Two different schemes:Address transparent mode (optional): The N_Port FC_IDs are unique across the whole logical fabric

Address translation mode (mandatory): The N_Port FC_IDs are unique only inside the gateway region the N_Port belongs to



Address Transparent Mode

• All the gateways belonging to the same “logical fabric” cooperate to assign addresses that are unique across the gateway regions that form the logical fabric

• No need for address translation

• Not scalable (max 239 gateways)


Address Translation Mode

• iFCP gateways use aliases to map the local representation of addresses of external gateway regions to the real addresses outside the gateway region (comparable to IP NAT)

Requires a rewrite of the FC_IDs in the FC frame header and in the FC payload for some ELS (i.e. ADISC)



TCP/IP IFCPGatewayIFCP

Gateway

FC N

S Req

uest

iSNSGive Me the Remote

Gateway IP Address, N_Port ID,

N_Port WWN

2) iSNS query/

reply

1) The N_Port Issues a NS Query

Remote GW IPDest N_Port ID (y.y.y)

3) The Requesting GW Fills Up the Address Translation Table

Local N_Port alias (z.z.z)Dest N_Port WWN

FC_ID = x.x.x

FC_ID = y.y.y

FC N

S Rep

ly

4) The GatewaySends Back to theN_Port the NS Reply (for FC_ID z.z.z)

iFCP Protocol Description: Address Translation Mechanism


TCP/IP IFCPGatewayIFCP

Gateway

Plogi

I did

z.z.z

sid x.

x.x

1) The N_Port I a

PLOGI toD_ID z.z.z

Remote GW IPDest N_Port ID (y.y.y)

4) The Receiving GW Rewrites the S_ID of the Incoming Request

Local N_Port alias (z.z.z)Dest N_Port WWN

FC_ID = x.x.x

FC_ID = y.y.y

2) The GW Makes a Table Lookup Gets the Remote GW IP Address (to Set Up the iFCP Session) and the Actual Dest N_Port ID ( to Rewrite the D_ID) PLOGI did y.y.y

sid x.x.x

Remote GW IPDest N_Port ID (x.x.x) 3) The receiving GW Fills Up

Its Own Translation TableLocal N_Port alias (w.w.w)Dest N_Port WWN

PLOGI did y.y.ysid w.w.w

iFCP Protocol Description: Address Translation Mechanism (Cont.)

• In case of fabric reconfiguration all the address translation tables need to be recalculated with a consequent loss of every active login session



ISNS AND SLP DISCOVERY PROTOCOLS FOR THE IP-SAN

OPT-2T019899_06_2004_X


Discovery Approach

Deploy and Interoperate in Three Stages:1. Naming and static configuration

Configure both targets and initiators

Use SendTargets to reduce initiator config

2. SLPv2 for multicast and simple discoveryConfigure targets

3. iSNS for centralized managementConfigure central iSNS server



Service Location Protocol (SLP)

• Based on service location protocol v2 (RFC 2608)

• Allows hosts to search for instances of a network service they are interested in:

Example: printers


Basic SLP Discovery Requirements

• Find targets by initiator’s worldwide unique identifier

“Tell me which targets you have that I should see”

• Find targets by target’s worldwide unique identifier“Where is target iscsi.com.acme.foo?”

• Propagate attributes needed before connectingBoot information, authentication information

• Scaling requirementsZero-configuration, no servers in small environmentsReduce or eliminate multicast in medium environmentsInteroperate with LDAP/iSNS in large environments



Service Location Protocol (SLP)

• SA—Service Agent; Services register with SA• UA—User Agent; Queries SA or DA for registered

services• DA—Directory Agent; Proxies for a set of SAs

SA

DA

UA

registerquery/response

query/response servicesservices

servicesregister

Three Components, Two of Which Run in Our Storage Router


Service Location Protocol for IP Storage

• Service Agent (SA) Advertises servicesServices have attributes

• User Agent (UA) Finds servicesZero configuration

• Directory Agent (DA)OptionalPropagate service adverts

• SLP ProtocolUDP or TCPMinimize multicastdevice

host

iSCSITarget

SLPUA

SLPDA

iSCSIInitiator

IP

TCP/IP

Management Code

SLPSA

TCP/IP

Management Code



Implementing SLP for iSCSI

• Targets implement a service agentAnswer multicast requests or register with DA

• Initiators implement a user agentUse multicast or DA to locate targets

• Devices containing targets register:The canonical target or individual targetsAttributes of targets

• Register target at each of its addresses


SLP Summary

• Serverless discovery of targetsOptional, generic DA to scale services

• Zero-configuration of hostsSLP makes careful use of multicast

• Access list and attribute propagation

• Optional message authentication

• Available open source implementations



What Is iSNS

iSNS Facilitates Scalable Configuration and Management of iSCSI, iFCP and Fibre Channel (FCP) Storage Devices in an IP Network, By Providing a Set of Services comparable to that Available in Fibre Channel Networks

http://www.ietf.org/internet-drafts/draft-ietf-ips-isns-22.txt


iSNS Functions

There Are Four Main Functions of the iSNS: 1. A name server providing storage resource

discovery

2. Discovery Domain (DD) and login control service

3. State change notification service

4. Open Mapping of Fibre Channel and iSCSI devices



Basic: How iSNS Works

IP NetworkFibre Channel SAN

iSCSI & iSNS Clients

iSCSI Clients Register with iSNS Server, Done By Adding iSNS IP Address to iSCSI Application Driver

11 1 1

iSCSI Targets Register with iSNS Server2

2

iSNS Clients Query iSNS Server for Storage Location and Name

3 3

iSCSI Client then Selects and Logs into iSCSI Target Using Information from iSNS Server

4

FCFC

FCFC


Internet Storage Name Service (iSNS)

• iSNS server functions:Allows an iSNS client to register/deregister/query with the iSNS server

Provides centralized management for enforcing access control of targets from specific initiatorsProvides a state-change notification mechanism for registered iSNS clients on the change of status of other iSNS clients

• Similar to the functionally provided by the FC name Server, Zone Server and the RSCN mechanism



iSNS Components

• iSNS protocol (iSNSP)A flexible and lightweight protocol that specifies how iSNS clients and servers communicate

• Discovery Domain (DD)A grouping of storage devices much like a zone in the FCP; discovery domains help in control and manage logins and services available to the clients in the domain; Based on the FC-GS standard for fiber channel; Items like default domain are used

• Discovery Domain Set (DDS)A group of one or more discovery domains; A method to store sets of domains within the iSNS database; Multiple DDSs can be active at one time, unlike zonesets in FCP where only one can be active at a time


iSNS Components

• iSNS clientThe iSNS client is located within storage system and talk to theiSNS server using the iSNSP within its configured device domain;client can belong to one or more DDs; iSNS client registers its attributes with the iSNS server and receives notices of changes within the domain

• iSNS databaseThe iSNS database is the information repository for the iSNS server; it maintains information about iSNS clients attributes; a directory-enabled implementation of iSNS may store client attributes in an LDAP directory infrastructure

• iSNS serveriSNS servers respond to iSNS protocol queries and requests, and initiate iSNS protocol state change notifications; properly authenticated information submitted by a registration request isstored in an iSNS database; listens on port 3205



iSNS SCN (State Change Notifications)

• iSNS clients who wish to receive SCN have to explicitly register with iSNS server the events in order to receive the notifications

• Initiator/target/object with add/remove event or to/from discovery domain are the events that can be registered

• iSNS servers generate SCN when either the state of any target device changes or when the target device itself requests an SCN to be generated using SCN event message; iSNS listens to FCNS to registration/deregistration


SCN Types

• Regular registrationsThis type of SCN is used within a DD; The discovery domain will control where the SCN message will go

• Management registrationsUsed by control nodes and can travel outside the DD from which they came

Can be TCP or UDP messaging(Most implementations only using TCP for now)



Services Provided by the Discovery Domain

• Login controlAuthorization and control policies for storage targets can be maintained by iSNS servers only allowing authorized devices to access the targets

Control of what target portals are accessible within the discovery domain

• Fibre Channel to iSCSI device mappingiSNS database learns and stores naming and discovery information about FC storage devices discovery on the iSCSI Gateway and iSCSI devices in the IP network; This database can then be available by FC and IP iSNS clients


High Availability of iSNS Servers

• Can use SLP to discovery other iSNS servers

• Database transfers between servers using iSNSP or SNMP

• Heartbeat mechanism used between active and backup iSNS servers



Internet Storage Name Service (iSNS) for iSCSI

• The iSNS protocol (iSNSP) provides:A mechanism for iSCSI clients to discover other iSCSI targets/initiatorsEnforce access controlNotifications from an iSNS server on changes to the status of a logged in iSCSI device Provide ability to discovery iSCSI target on different IP network

• iSCSI target discovery can happen through:Static configuration of initiatoriSCSI sendTargets commandName server/directory server (via iSNS)


iSNSP Header

iSNSP Version—C the Current Version is 0x0001; All Other Values Are RESERVEDiSNSP Function ID—Defines the Type of iSNS Message and the Operation to Be Executed; iSNSP PDU Length—Specifies the Length of the PDU PAYLOAD Field in bytes; The PDU Payload Contains Attributes for the OperationiSNSP Flags—Indicates Additional Information About the Message and the Type of Network Entity That Generated the Message iSNSP Transaction ID—MUST Be Set to a Unique Value for Each Concurrently Outstanding Request Message; Replies MUST Use the same TRANSACTION ID Value as the Associated iSNS Request MessageiSNSP Sequence ID—The SEQUENCE ID Has a Unique Value for Each PDU Within a Single TransactioniSNSP PDU Payload—The iSNSP PDU PAYLOAD Is Variable Length and Contains Attributes Used for Registration and Query OperationsAuthentication Block—For iSNS Multicast and Broadcast Messages, the iSNSP Provides Authentication Capability; The iSNS Authentication Block Is Identical in Format to the SLP Authentication Block



iSNSP Commands for iSCSI

The Following Are iSNSP Commands Messages Used in Support of iSCSI:


iSNSP Responses for iSCSI

The Following Are iSNSP Response messages Used in Support of iSCSI:



iSNS Queries for iSCSI

• iSNS clients can perform two types of queries:Device attribute query: iSNS server responds with requested attributes of one or more iSNS clients

The iSNS server converts the received query to a FC name server query in the SAN

FC name server will ensure that the resultant set is filtered based on zones

The iSNS server translates each entry returned by the FC name server to the corresponding iSNS clients

Apply filters based on iSCSI access control by removing all statically configured virtual targets the query initiator is notallowed to access

Device get next query: Allows an iterative query of the iSNS server’s iSNS client database


Return Information from iSNS iSCSI Query

iSCSi Name

IP Address of Portal to Log to and Ask for This

target

Name of Port on the IP Gateway

Entity



iSNS for iFCP

• Will work much the same manor as iSCSI just will require other related attributes to be registered and queried

• Is required for iFCP

• Functions much like domain name server and domain ID manager

• Needs to be highly available service for FC devices


iSNSP Commands for iFCPThe Following Are iSNSP Commands Messages Used in Support of iFCP:



iSNSP Responses for iFCPThe Following Are iSNSP Response Messages Used in Support of iFCP:


SLP and iSNS

• SLP used for target discoveryNo configuration required for the simplest networksSmall footprint; no servers required

Just enough discovery for small-to-medium networks

Device-centric access control model

• iSNS adds storage management capabilitiesActive monitoring of initiators and targets

Event propagationPublic key distribution

Centralized access control model



Using Both SLP and iSNS

• Initiators can use both SLP and iSNS to discover targets

• Targets should use SLP only if not configured for iSNS

• Gateways or proxies may provide local SLP discovery of remote iSNS devices


TECHNICAL TOOLS AND SKILLS

OPT-2T019899_06_2004_X



Storage Networking Toolbox

• Test tools for Fibre Channel and IP

• Host based tools

• Network component serviceability tools

• Software debug tools

• Knowledge


Fibre Channel Analyzers

• Most units are based on dedicated hardware, and might be supplied with software tools for performance base lining

Very expensiveOriented to protocol conformance testingRequires 2 GBICs interfaces to be implemented

• Monitoring units might have a retiming mode, to cleanup some of the timing problems on a link, and to separate them from the real problem at layer 1

Statistical software can run on these type unitsCollecting statistics on the status of the line, or other parameters (number of bits, exchanges…)

• Sharing is still a dream in most cases, it is complex to share in the field, so in most cases the portable versions are the most suitable



Fibre Channel Analyzers

• Snooping GBICs or fiber taps; allow to monitor without service interruption; very important for Fibre Channel work in the field

• Traffic probes; used to remotely monitor the state of a network without service interruption

• Trace viewers (free from the vendor websites)Each vendor has its own PC viewer and must be used with each capture tool; these can be found at each of their websites


FC Test Vendors

• Leaders in dedicated hardware tools:Finisar (www.finisar.com)Xyratex (www.xyratex.com)

Aglient (www.agilent.com)

I-Tech (www.I-tech.com)Ancot (http://www.ancot.com/)

Spirent/Netcom systems (www.netcomsystems.com)



SCSI Host-Based Testing

• I/O meterhttp://developer.intel.com/design/servers/devtools/iometer/

• I/O zonehttp://www.iozone.org/

• SCSI toolshttp://scsitools.com/

• Xyratex disk basherhttp://www.xyratex.com/

• Freeshare or software tools for SCSI and I/O analysis, tools for disk manufacturing

• www.ethereal.com• www.wildpackets.com


Windows Tools

• iSCSI Driver debug helpersWindows debug utilities

http://www.osr.com/resources_downloads.shtml

http://www.sysinternals.com/

• Detail uses of O/S disk administrator to verify and check health of target devices



IP: GiGE

• GiGE testers $$$AgilentSniffer

Fluke

Finisar/ShomitiiSCSI decodes just becoming available on most tools

• All your IP toolsIP Ping, trace, etc.Fibre Channel ping available at http://www.teracloud.com/utilities.html


iSCSI Decoding

• Software only analyzers like Ethereal (www.ethereal.com)

• Hardware analyzers

• Can use monitor command on Cisco switches to span the iSCSI GiGE port to a 10/100



Available Certifications

• SNIA (Storage Networking Industry Association)Level 1—Fibre Channel storage networking professionalLevel 2—Fibre Channel storage networking practitioner

• iSCSI training available at many education sourcesInfinity I/O, medusa, solution technology, others

• Other certifications that are vendor specific


ARCHITECTURAL DESIGN OFSTORAGE AREA NETWORKS

OPT-2T019899_06_2004_X



Section Agenda

• Introduction

• Hierarchy

• Modularity

• Architecture Examples

INTRODUCTION




Hierarchy, Modularity and Limited Failure Domains

Why Do This? (Benefits Summary):• Scalable architecture

• Improved performance

• Manage change

• Improve service

• Improved security

• Simplified management and troubleshooting

• Reduced cost of ownership


What Problem Are We Solving?

• Have consistently high performance

• Are reliable, scaleable, and manageable

• Are secure and cost-efficient

• Are service and solution enabling

• Adapt to changing requirements

Applications Must Be Available and Perform Well

Networks that Deliver on this Requirement:



Network Design Goals

Architecture Provides:• Performance

• Reliability, availability, and scalability—RAS

• Cost efficiencies

• Security

• A base to enable services and solutions

To Meet Mission-Critical Business Objectives, Applications Need to Be Consistently Up,

Available, and High-Performing


Architecture:Hierarchy, Modularity, and Domains

Functionally Divides the ProblemCreate Manageable Building BlocksLimits Scope of Potential Failures

HierarchyModularityDomains

Fundamentally, We Break the Network Design Process into Manageable Blocks so that the

Network will Function within the Performance and Scale Limits of Applications, Protocols

and Network Services



Focus of ThisDiscussion

What Does This Mean?

We Build Networks that Have Structure:

Distribution

Backbone

WAN Internet PSTN

Application Servers

Building Blocks

Core

EnterpriseStorage

Access


Applying Design Principles to Storage

• HierarchyPredictable performanceScaleable design

Fault isolation

• ModularityCost-effective

Repeatable

• DomainReliabilitySecurity

Shared Storage

Unified Storage Mgmt

CoreCore


HIERARCHY



Hierarchy: Physical and Logical

• Physical hierarchyPredictable performance

Scaleable design

Fault isolationHigh availability

• Logical hierarchyVirtual SANsZoning

Enhances physical hierarchy

Physical Architecture

Logical Architecture Logical Architecture

Virtual SAN A Virtual SAN B

H1 H2

D1 D2H3

Zone 1 Zone 2

Zone 3H1 H2

D1 D7H7

Zone 1 Zone 2

Zone 4



Hierarchy: PhysicalConsolidated Storage Network

• Cost-effective solutionBenefits of consolidation

• Limited scalabilitySmall to medium business

Expansion can be disruptive

• Single fault redundancyDouble fault would likely result in isolation

iSCSI iSCSI


Hierarchy: Physical Collapsed Core Architecture

• Collapsed coreHigh performanceMultiple unequal paths

• Better scalabilityMedium to large enterprise

ISLs can limit scalability

• RedundantMesh topologyNetwork survives some double faults

iSCSI iSCSI



Hierarchy: PhysicalCore Edge Architecture

• Core—EdgeHigh performanceLoad balancing

Consistent hop count

• Good scalabilityLarge to very large enterprise

Non-disruptive expansion

• Better fault toleranceImproved fault isolationSingle fault within layer okay

iSCSI iSCSI


Hierarchy: PhysicalOversubscription

• To be expected in storage networks

• Typically lower factors than we see in LANs

• Architecture should be flexible to accommodate differing requirements for various hosts and storage subsystems

• Bandwidth can be modified non-disruptively by using port channels between switches

• Take into account any “inherent” over subscription in networking hardware

• Use actual anticipated throughput rather than link speed for calculating bandwidth requirements



Hierarchy: PhysicalInter-Switch Links

• Inter-Switch Link—ISLPhysical FC link between two fabric switches forming a trunkUtilized for FC services and data traffic

• Port ChannelMultiple FC ISLs combined to form a single aggregated trunkAll links in a Port Channel must be directly connected to the same two switchesIndividual link state changes do not cause ISL trunk state changes

ISLPort

Channel

iSCSI iSCSI


Hierarchy: PhysicalScalability

• OversubscriptionHigher OS acceptable for some hostsLower OS for High performance hosts and storage devicesConsider impact of multi-path load balancingDetermine acceptable worst case in various failure scenariosCan be non-disruptively changed by adding/ removing links to port channels

8:1OS

1:1OS

Core

3:1OS

4x2GbISL

8x2GbISL

15:1OS

iSCSI iSCSI



Hierarchy: LogicalVirtual SANs

• VSANs provide a means to build a logical structure on top of a physical SAN

• Similar to how VLANs are used to scale ethernet networks VSANs help scale Fibre Channel networks

• Topology changes are isolated within the VSAN therefore adds, moves, and changes are not disruptive to other VSANs

• VSANs can be utilized to establish administrative domains

• Zoning provides an additional access control mechanism within each VSAN


Hierarchy: LogicalLogical Architecture

• Virtual SANsSimilar to Ethernet VLANs except no inter-VSAN flowsEnhanced ISL provides VSAN trunking (EISL)Complimentary to port channel

• Services scalabilityIndependent Fibre Channel services for each VSANZoning is per VSAN

• Failure domainFaults contained within VSAN

iSCSI iSCSI

EISLPort

Channel



Hierarchy: Logical Maximizing VSAN Architecture

• Isolate multiple paths into separate VSANs

• Independent FC services per VSAN

• Provides complete traffic isolation between redundant paths

• Each VSAN converges independently for faster recovery and improved fault isolation

iSCSI iSCSI


Hierarchy: Combining Physical and Logical

iSCSI iSCSI

• Fabric A provides one set of links and Fibre Channel services A• Fabric B provides an independent set of links and services

B


MODULARITY



Modularity: Key Elements

• The ability to scale the network while maintaining consistent performance

• Building block approach breaks network into smaller chunks that are easier to understand, replicate, and deploy

• Changes and additions can be made non-disruptively

• Provides consistent and limited failure domains

• Modularity can also define administrative boundaries



Modularity: Building Blocks

iSCSI iSCSI

Fiber Channel

CoreFunctional Building Blocks Provide Scalability with Deterministic Performance

StorageModules

ApplicationModules


Modularity: Utilizing VSANs

• Adds, moves, and changes contained within a VSAN are non-disruptive to other VSANs

• Using VSANs facilitates application modeling and testing

• Per VSAN statistics

• Per VSAN traffic engineering

• Per VSAN administration (if desired)

• Eliminates costs associated with separate physical fabrics



Modularity: Benefits of VSANs

• Overlay isolated virtual fabrics on same physical infrastructure

Each VSAN contains zones and separate (replicated) fabric servicesVSAN membership determined by port

• VSANs for availabilityIsolate virtual fabrics from fabric-wide faults/reconfigurations

• SecurityComplete hardware isolation

• ScalabilityReplicated fabric servicesThousands of VSANs per storage network

• ManagementRoll Based Access Control—RBACProvides administrative boundaries

Department/Customer “A”

Shared Storage

Department/Customer “B”

VSAN-EnabledFabric

MgmtVSAN

VSANTrunks


Modularity:Storage Intelligence and VSANs

Dept 3VSAN

Dept 2VSANDept 1

VSAN

• VSANs created to provide isolation of fabric-wide services.

• Virtualization allows physical storage to be in its own VSANs, separate from the host VSANs.

VSANs provide• Secure isolation of physical storage• Easier configuration• Dynamic configuration of fabrics• Role-based access control

Data Center VSANs

Virtualization


ARCHITECTURE EXAMPLES



Architecture: iSCSI

• ScalabilityLess expensive alternative for host not requiring 2GbpsRecommend separate NIC TCP Offload Engine—TOEConsider actual throughput requirements for scalability

• Host ServicesAppears as normal HBACompatible with host based storage utilities—multi-path, load balance, mapping, etc.

TOE

File SystemFile System

Adapter Driver

Adapter Driver

TCP/IPStack

TCP/IPStack

NICDriverNIC

Driver

Block DeviceBlock Device

SCSI GenericSCSI Generic

iSCSIDriveriSCSIDriver

Applications

Network FileSystem

Network FileSystem

NICDriverNIC

Driver

TCP/IPStack

TCP/IPStack

iSCSIDriveriSCSIDriver

TCP/IPStack

TCP/IPStackNIC

DriverNIC

Driver

SCSI AdapterSCSI Adapter



Architecture: iSCSI High-Availability

• Redundant connections to hosts or servers• High-availability iSCSI services• Redundant paths to backend FC SAN

Redundant iSCSI to Fibre Channel Connections

and Services

Storage Array with Redundant Controller

Ports

MultipleEthernetSwitches

Host with Multiple(iSCSI) NICs and Multipathing

Software Installed

iSCSI DriverMultipathingApplicationApplication


Architecture: iSCSI Authentication

• SCSI routing service passes username and MD5-hashed password from initiators to AAA server

• AAA authentication list used to determine which service(s) to use for authentication

iSCSI Hosts

(Initiators)RADIUS TACACS+

RADIUS Server

Local

User1/pwd1User2/pwd2

… / …

CHAP

iSCSI Services

TACACS+ Server


… / …


… / …

Authentication Services List

AAA Authentication ServicesAAA Authentication Services

IP Network FC Fabric


SCSI Routing Instance

iSCSI Storage(Targets)



Architecture: iSCSI Topology

iSCSI Best Practices• Isolate IP storage network

behind application hosts with VLANs

• Minimized potential for bandwidth contention

• Map VLANs to VSANs for manageability

• Dedicated ethernet interfaces on host for attachment to storage network

iSCSI-enabledHosts

Storage Pool

iSCSIServices

Ethernet Switches

Clients

Front-Side IP Network

Front-Side IP Network

FC Attached Hosts with

HBAs

iSCSI iSCSI iSCSI iSCSI

IP StorageNetwork

FCFabric


Architecture: SAN Extension Technology

IPWAN

Technology Choice Requires Matching Storage Application Requirements with Service Availability, Cost, Throughput, and Latency

FCIP FCIP

CWDM

DWDM

SONET/SDH

FC FC



• Both fabrics remain connected if one of the paths fails

• Use of portchannel prevents state change on link failure

Architecture: High Availability for SAN Extension: FC

• Utilize disparate paths and portchannel for high availability

• Utilize VSANs to limit the failure domain in the event of lost connectivity

FabricA

FabricA

FabricB

FabricB

PortChannel

FCFC CWDM

DWDM

SONET/SDH


Architecture: High Availability for SAN Extension: FCIP

• Utilize disparate paths and portchannel for high availability

• Utilize VSANs to limit the failure domain in the event of lost connectivity

• Recommend not using etherchannels

FabricA

FabricA

FabricB

FabricB

PortChannel PortChannel

IPWAN

IPWAN

FCIPFCIP

• Both fabrics remain connected if one of the paths fails

• Use of portchannel prevents state change on link failure



Architecture:Legacy Storage Implementation

• Storage is ‘captive’ behind applications

• Inefficient allocation of storage resources

• Multiple administrative domains

Backbone

ApplicationServers

Captive Storage Blocks

LAN Core

Campus Clients Remote Clients Internet Clients

SANIslands


Architecture: Factors for Determining Architecture

• Current size and anticipated growth for both application servers and storage elements

• Baseline performance requirements for servers and storage

• Business continuance requirements—SAN extension

• Administrative domains

• Migration plans

• Interoperability considerations

• Costs



Architecture: Collapsed Core Architecture

• Servers and storage elements connected to collapsed core

• Some scalability especially with iSCSI

• Redundant paths• Achieves

economical storage consolidation

• VSANs can add scalability and management benefits

ApplicationServers

iSCSI

Shared Storage



Architecture: Large Scale Architecture

• Application servers connect to edge switches

• Storage devices connect to edge switches

• Highly scalable

• Highly redundant• Highly modular

• Multiple equal paths• VSANs limit the size

of any one SAN

ApplicationServers

Shared Storage


iSCSI



Network Design Goals

• Performance Planned hierarchy, managed oversubscription, and modular design

• Reliability, Availability, and Scalability—RASLimited failure domains, leveraged VSANs, and modular design

• Cost efficienciesConsolidated storage, central management, and leveraged resources

• SecurityLimited domains, RBAC management, and consistent architecture

• A base to enable services and solutionsBusiness continuance and disaster recoveryManagement of heterogeneous storage elementsUbiquitous access to storage from anywhereInfrastructure for storage virtualization

Architecture Summary:


Architecture: End-to-End SAN Architecture

Intelligent WorkgroupStorage Networks

Highly ScalableStorage Networks

Optical Network

Resilient OpticalTransport Networks

Synchronous Replication—Optical (FCIP/FC)

SONET Network

Multiprotocol/MultiserviceSONET Network

Asynchronous Replication—FCIP over SONET FCIPRemote StorageAccess

FC FC

FC FC FC

FC

FC

FC

FC FC

FC FC FC

FC FC

FC FC FC

iSCSI-EnabledStorageNetwork

EthernetSwitches

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI

FC

iSCSI


Q & A



Complete Your Online Session Evaluation!

WHAT: Complete an online session evaluation and your name will be entered into a daily drawing

WHY: Win fabulous prizes! Give us your feedback!

WHERE: Go to the Internet stations located throughout the Convention Center

HOW: Winners will be posted on the onsiteNetworkers Website; four winners per day



EXTRAS



FC LOOP OPERATIONS



Single Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

IDLE IDLE

IDLEIDLE

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. The Loop is initially filled with IDLES

2. Each port is in the monitoring state

3. Because of no activity CFW = Idle

4. Rx IDLES are replaced with CFW



Single Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01) IDLE

IDLEIDLE

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Port_01 begins to arbitrate for access to the Loop

2. Port_01 changes its CFW from IDLE to ARB(01)

3. Port_01 transmits ARB(01) when a fill word is required


Single Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01) ARB(01)

IDLEARB(01)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. ARB(01) is Rx by the next port and updates its CFW to ARB(01)

When a Port Discards Rx Fill Words and Transmits the CFW this Allows the Port to Compensate for Clock Differences Between Rx Data Stream and Tx Data Stream

2. Whenever a fill word is required ARB(01) is used; With no other activity on the loop ARB(01) is sent



Single Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(F0)

OPN

ARB(01)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. When Port_01 receives its own ARB(01) it wins arbitration

2. Port_01 sends on OPN to open a loop circuit and changes its CFW to ARB(F0)

3. Port_01 discards any Rx’ed ARB(x)


Single Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

IDLE ARB(F0)

ARB(F0)ARB(F0)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. As each port Rx’s the ARB(F0) it updates its CFW to ARB(F0)

2. Assuming that no other port is arbitrating, ARB(F0) travel the complete loop

3. When ARB(F0) is Rx’ed by Port_01 the CFW in Port_01 is changed to IDLE



Single Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

IDLE IDLE

ARB(F0)IDLE

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Each port receives the IDLE and updates its CFW to IDLE

Discarding the Receiving Arb(x) Prevents Any Other Port from Winning Arbitration

2. Assuming the no other port is arbitrating and the IDLES travel the complete loop

3. As long as Port_01 owns the loop it discards any Rx’ed IDLE or ARB(x) and continues to send its CFW when necessary


Multiple Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01) IDLE

IDLEARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Port_01 begins arbitrating for access to the loop; Done by replacing IDLE and ARB(x) with ARB(01)

2. Port_B2 also begins arbitrating for the loop; It replaces Idle and ARB(x) with ARB(B2)



Multiple Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01) ARB(01)

ARB(B2)ARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. The ARB(01) gets to Port_2A which updates its CFW with ARB(01) and transmits this when the CFW is needed

2. The ARB(B2) also travels to Port_EF which updates its CFW with ARB(B2)


Multiple Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01) ARB(01)

ARB(B2)ARB(01)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. When Port_B2 receives ARB(01) it changes its CFW to ARB(01) because of 01 has higher priority(Lower AL_PA wins)

Because Port_B2’s ARB(B2) Is Replaced with ARB(01) It Will Not Win Arbitration at this Time

2. When Port_01 receives ARB(B2) it is replaced with ARB(01)



Multiple Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(F0) and OPN

ARB(01)

ARB(01)ARB(01)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. ARB(01) is Rx by Port_01 and wins arbitration

2. Port_01 then opens the loop circuit and updates it’s CFW with ARB(F0) when a fill word is required

3. Port_B2 is still arbitrating but is lower priority


Multiple Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(F0) ARB(F0)

ARB(01)ARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Port_2A receives ARB(F0) and updates the CFW to F0

2. Port_B2 replaces the lower-priority ARB(F0) and transmits ARB(B2)



Multiple Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(F0) ARB(F0)

ARB(B2)ARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Port_EF updates its CFW to ARB(B2) and transmits on to Port_01

2. Port_01 Tx’s ARB(F0)

When Port_01 Relinquishes Control of the Loop It Changes Its CFW to ARB(B2) Allowing Port_B2 to Win

3. Port_B2 continues to replace F0 with B2; Port_01 discards all Rx’ed ARB(x) ordered sets


Lower Priority Port ARB

RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

IDLE IDLE

IDLEARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Port_B2 begins to arbitrate for the loop by changing CFW to B2

2. Each Rx’ed IDLE and lower-priority ARB(x) is discarded by Port_B2 and the ARB(B2) is substituted in its place




RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

IDLE IDLE

ARB(B2)ARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. ARB(B2) propagates around the loop to Port_EF

2. Port_EF changes its CFW to ARB(B2) and Tx’s the ARB(B2) whenever a fill word is needed



RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(B2) IDLE

ARB(B2)ARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. The ARB(B2) propagates around the loop to Port_01

2. Port_01 changes its CFW to ARB(B2) and Tx’s ARB(B2) whenever a fill word is needed




RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01) ARB(B2)

ARB(B2)ARB(B2)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Port_01 begins arbitrating after a single ARB(B2) has passed

3. The single ARB(B2) travels around the loop to Port_2A. Port_2A passes the ARB(B2)

2. Port_01 has higher priority than Port_B2 and discards ARB(B2) and replaces it with ARB(01)

4. When ARB(01) is Rx’ed at Port_2A its CFW is changed from B2 to 01

ARB(01)

XX



RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01)

ARB(B2) ARB(F0) and OPN

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. The single ARB(B2) is Rx’ed by Port_B2 which wins arbitration and begins to discard any Rx’d ARB(x)

2. Port_B2 changes its CFW to ARB(F0)

ARB(01)

ARB(B2)




RX TX

Port

RX TX

Port

RX

TX

Port

RX Port

TX

ARB(01) ARB(01)

ARB(F0)ARB(F0)

AL_PA 2A

AL_PA EF

AL_PA B2AL_PA 01

1. Port_EF changes its CFW to ARB(F0) and sends it on to Port_01

2. Port_01 substitutes ARB(01 for every ARB(F0) it receives

3. Port_B2 discards the ARB(01) and sends ARB(F0) as its fill word

4. When Port_B2 relinquishes the loop, it will change its CFW to ARB(01) and allow Port_01 to win the loop

storage area networking protocols and architecture

Documents