troika university 111 technology positioning building a better data center mike dutch 2000-dec-13...
TRANSCRIPT
TROIKA University 111
Technology PositioningBuilding a better Data Center
Mike Dutch 2000-Dec-13
Copyright 2000 Troika Networks, Inc.
TROIKA University 111
Technology PositioningBuilding a better Data Center
This presentation is the property of Troika Networks, Inc. and should not be distributed outside the company
without explicit authorization.
Mike Dutch 2000-Dec-13
Copyright 2000 Troika Networks, Inc.
TROIKA University 111
Networking Standards
OSI Reference Model (ISO 7498:1984)
TCP/IP Protocol Suite (IETF)
Ethernet (IEEE 802.3, ISO 8802-3)
Gigabit Ethernet (802.3z, 802.3ab, 802.3ae)
Fibre Channel (ANSI FC-PI, FC-FS, 10GFC)
InfiniBand (IBTA)
TROIKA University 111
OSI Reference Model
Application
Presentation
Session
Transport
Network
Physical1
2
3
4
5
6
7Layer
Logical Link Control
Medium Access Control
Routing
Reliability
End to end communication
Transformation
Function
Framing
Signaling
OS Interfaces
Dat
a L
ink
Bridges
Repeaters
Routers
Gateways
Devices
Switches
TROIKA University 111
TCP/IP Protocol Suite
Application
Presentation
Session
Transport
Network
Data-Link
Physical
Application
Transport
Internet
NetworkInterface
OSI Model DARPA Model
TCP
IP
802
.1
TCP/IP Protocol Suite
HT
TP
FT
P
Te
lne
t
DN
S
RIP
SN
MP
802
.3
802
.5
X.2
5
FR
AT
M
UDP
IGMPARP
ICMP
Eth
ern
et80
2.3
LAN WAN
TROIKA University 111
Other network technologies
Frame Relay
X.25
TCP/IP
SDLC
Source: The Basic Guide to Frame Relay Networking, published by the Frame Relay Forum (www.frforum.com)
Lightweight WAN
Heavyweight WAN
Internet
IBM SNA
TROIKA University 111
Fibre Channel Standard
Application
Presentation
Session
Transport
Network
Data-Link
Physical
OSI Model
FC-4
FC-3
FC-2
FC-1
FC-0
Functional Levels
Link Trans., Speed Neg., EnDec
Framing, Flow Control, CoS
Physical Components, SerDes
Upper Level Protocol Mappings
Common Services, Link ServicesNod
e
2048 4464244
CRC EOFSOF HDR PayloadHDR
…
Sequence
…
Sequence Exc
hang
e
FC-2 Frame
Channels Networks OtherP
ort
(OX
_ID
, R
X_I
D)
(SEQ_ID)
(SEQ_CNT)
HuntGroups
Signaling Protocol
Transmission Protocol
Interface / Media
ULPs / FC-4 Mappings
TROIKA University 111
FC-2 Frame Fields
A frame is transmitted in the following byte order: A0..A3, B0..B35, C0..C3
Header
Link_ControlLink_Data:- Basic- Extended- FC-4Device_DataVideo_Data
Depends onRouting bits:
- Command- Unsolicited / Solicited- Category
Routing(bits 31-28)
Information(bits 27-24)
OptionalHeaders
fields uniquely identify a frame
Priority bitsClass 2 and 3Preference bit
TROIKA University 111
FC-0 Nomenclature
FC-PH-2
FC-PI
TROIKA University 111
Fibre ChannelClass of Service
Class of Service
Data DeliveryFlow
ControlExample Use
1
1-to-1
Reliable and dedicated(connection oriented)
Node Large files,
Full-bandwidth service
2
n-to-n
Reliable and multiplexed(connectionless)
Node,
Buffer
Clustering, Networking, OLTP
3
n-to-n
Datagram and multiplexed Buffer Storage, Networking, Broadcast, Multicast
4
n-to-n
Reliable and Guaranteed with QoS
Node,
Buffer
Real-time systemsFractional-bandwidth service
6
1-to-n
Reliable and dedicated, Multicast Server(uni-directional)
Node Video Distribution,
Data Acquisition
Source: FCIA, Fibre Channel, Connection to the Future, ISBN 1-878707-45-0, second edition, 1998Not to be confused with Quality of Connection (QoC) classes introduced in a Strategic Research profile.
TROIKA University 111
Gigabit Ethernet
FC-1
FC-0
TCP UDP
IP
Applications
Layer 1
Layer 2
TCP/IP Suite(Upper Layers)
802.3 MAC
802.2 LLC
802.3z
802.3ab
1000Base-T1000Base-CX, SX, LX
WiringCloset
BuildingBackbones
CampusBackbones
25m
100m
220-275m
500-550m
5km
CX
T
SX(62.5u MM)
SX(50u MM)
LX (9u SM)
1000Base- Distance
CRCLengthPreamble LLCDestination Source Data
IEEE 802.3 Frame
46-1500 42668
TROIKA University 111
InfiniBand Architecture
OSILayer
Upper LayerProtocols
Transport
Network
Link Encoding
MAC
1
2
3
4
5-7Consumer Transactions
InfiniBand Messages (QP)
Inter-Subnet Routing (GRH)
SwitchRouter
Subnet Routing (LRH)
Flow Control
Signaling
End IdlesStart Packet
Imm Data ICRCGRH xETH DataBTH VCRCLRH
PhysicalSymbols
QP
Consumer
WQE CQE
Transport
Port
Packet CA
Link
TROIKA University 111
An InfiniBand Subnet
Switch
CPU
CPU
Chipset HCA Switch
TCA
Target
Memory
Processor Node
I/O Unit
Link
Router
SubnetManager
Consumer …
Channel Adapter
Port Port…Channel Adapter
Port Port……
Verbs
ConsumerConsumer
OS Message and Data Service
Processor Node
Infin
iBa
nd
TROIKA University 111
InfiniBand
Link Width Designation
Laser Transmitter wavelength (nm),and connector
Optic Fiber (micron)Distance (meters, feet)
1X-SX 850,
dual LC
62.5/125 μm
multimode
2 – 125,
6.56 – 410.1
50/125 μm
multimode (beige)
2 – 250,
6.56 – 820.2
1X-LX 1310,
dual LC
9/125 μm
singlemode (blue)
2 – 10,000,
6.56 – 32,808
4X-SX 850,
single MPO (MTP)
62.5/125 multimode 2 – 75,
6.56 – 246
50/125 multimode 2 – 125,
6.56 – 410.1
12X-SX 850,
dual MPO (MTP)
62.5/125
multimode
2 – 75,
6.56 – 246
50/125
multimode
2 – 125,
6.56 – 410.1
InfiniBand Optical Links
T R
TROIKA University 111
Application
Presentation
Session
Transport
Network
Data-Link
Physical
OSI Model
ApplicationLayers
IP
ATM
SONET/SDH
DWDM
De-factostandard
IP
ATM
DWDM
IP
DWDM
SONET/SDH
IP over ATM
IP
DWDMIP over Fiber
Potentialarchitectures
Packet over SONET
(POS)
ATM = Asynchronous Transfer ModeSONET = Synchronous Optical NetworkSDH = Synchronous Digital HierarchyDWDM = Dense Wave Division Multiplexing
Wide Area Data Networking
TROIKA University 111
Packet Switching
FrameSwitching
Application
Presentation
Session
Transport
Network
Data-Link
Physical
TCP, UDP
IP, ARP,ICMP, IGMP
802.2 LLC802.3 MAC
IEEE 802.x
HTTP, FTP,DNS SNMP
RPC/XDR
Sockets (1)
SPX, NCP,SPP/PEP
IPX, IDP
QP
GRH
LRH
iSCSI, iFCP,FCIP, SEP
Direct (3)NetBIOS (2),XNS Courier
NFS, XNS, RFS, NW
SONET/SDHDWDM
FC-3FC-2
FCP IPFC FC-VI others
FC
-4
ATM
OSI
DAFS
SCTP, STP
InfiniBand IP Storage
SCSI
FibreChannel
NetworkFile Systems
Internet
CellSwitching
CircuitSwitching
SMB, NFSP,Streams
FC-1FC-0
1X, 4X, 12X
SNADS DIA, DDM
PresentationServices
Data FlowControl
TransmissionControl
Path Control
ANSI FDDITR MAC
FDDI ProtocolFDDI MDI
SNA
Primary Block Stream Changer Multimedia Controller Enclosure Object
SPI SBP FCP SSA-S3P SST SRP
Interconnects(4)
Com
man
d S
ets
Pro
toco
ls
SCSI
7
6
5
4
3
2
1
Summary of Interfaces
TROIKA University 111
Transport Interfaces
Virtual Architecture (VI) Architecture
Sockets and NetBIOS
Windows Sockets Direct Path / SDP
Direct Access (Fast) Sockets
TROIKA University 111
VI Architecture
Application
OS Communication InterfaceSockets, MPI, Cluster, Other
VI User Agent
VI Network Adapter
VI Kernel Agent
User Mode
Kernel Mode
VI Provider
VI Consumer
RECEIVE
SEND
VI
RECEIVE
SEND
VICOMPETION
CQ
Send/Receive/RDMA Read/RDMA Write
Open/Connect/Register Memory
TROIKA University 111
Application Interfaces
TCP
IP IGMPARP
ICMP
Network Interface Layer
UDP
WindowsSockets
NetBT
WinSockApplication
NetBIOSApplication
Session andDatagram Services
Stream andDatagram Sockets
TROIKA University 111
WinSock Direct
TransportDriver
Interface
ServiceProviderInterface
TCP/IP
SAN (VI)
WinSockProvider
Base
WinSockProvider
NDIS
Switch
WinSock
User mode
Kernel mode
TROIKA University 111
Direct Access Sockets (Fast Sockets)
Application
Sockets API
FSockApp
Instance
FSockBroker
VI NIC
OS
Node
Application
Sockets API
FSockApp
Instance
FSockBroker
VI NIC
OS
Node
Data Flow
FSock Flow
App Flow
TROIKA University 111
Storage Protocols
Block-level I/O – SAN Channel Protocols IP Storage
File-level I/O - NAS NFS CIFS DAFS
Object-level I/O
DB/File System
Block Storage
ApplicationSAN
NAS
SAN groups the file systemwith the application
NAS groups the file systemwith the storage
TROIKA University 111
SNIA Storage Model
File/record subsystemFile/record subsystem
Database(dbms)
File system(FS)
Sto
rage
dom
ain
Application
Se
rvic
es
su
bs
yst
emS
erv
ice
s s
ub
sy
stem
Dis
cove
ry,
mon
itorin
gD
isco
very
, m
onito
ring
Res
ourc
e m
gmt,
con
figur
atio
nR
esou
rce
mgm
t, c
onfig
urat
ion
Sec
urity
, bi
lling
Sec
urity
, bi
lling
Red
unda
ncy
man
agem
ent
Red
unda
ncy
man
agem
ent
Hig
h av
aila
bilit
yH
igh
avai
labi
lity
Cap
acity
pla
nnin
gC
apac
ity p
lann
ing
Block subsystemBlock subsystem
Storage devices (disks, …)Storage devices (disks, …)
Block aggregationBlock aggregation
Device-basedblock-aggregation
SAN-basedblock-aggregation
Host-basedblock-aggregation
TROIKA University 111
Sample Architectures
Hos
t. w
ith L
VMan
d so
ftwar
e R
AID
1. Direct-attach
File
/re
cord
su
bs
yst
em
Blo
cks
ub
sy
ste
m
Device block-aggregation
SAN block-aggregation
Host block-aggregation
NASserver
4. NAS serverHost3. NAS head
NAS head
Host
LANH
ost.
with
LVM
Disk array
SAN
2. SAN-attachApplication
TROIKA University 111
Small Computer Systems Interface
TROIKA University 111
SCSI Request/Response
Initiator
ApplicationClient
Target
Logical Unit
TaskManager
DeviceServerDevice Service
Task Management
Source: “iSCSI Technical Overview”, by Mike Mesnier, Intel, March 2001
TROIKA University 111
iSCSI
SCSI Application
iSCSI ProtocolServices
SCSI DeviceServer
iSCSI ProtocolServices
Initiator I/O System Target I/O System
SCSI CDB
iSCSI ProtocolLayer
TCP/IP
EthernetData link +Physical
Data link +Physical
SCSIApplication
Layer
iSCSI PDU
TCP segmentsin IP
datagrams
EthernetFrame
SCSI ApplicationProtocol
iSCSI Protocol
Ethernet
Protocol ServiceInterface
iSCSI TransportInterface
TCP/IPTCP/IPTCP/IP
TCP/IPTCP/IP
TCP/IP TCP/IP Protocol
iSCSI session
Source: “iSCSI Technical Overview”, by Mike Mesnier, Intel, March 2001
Challenges /Proposals
• TCP/IP overhead• Multiple data copies• Out of order delivery
• TCP/IP offload• Large buffers• Stream Markers• RDMA (WARP/SCTP)
TCP Header
4820
iSCSI Header802.3 Header
2014
IP Header iSCSI Data
46
802.3 Trailer
TROIKA University 111
Connecting FC Fabrics
FibreFibreChannelChannel
SANSAN
FibreFibreChannelChannel
SANSAN
FCIPTunnel
FCIPTunnel
IP Network
FCServer
FCJBOD
FCServer
FC TapeLibrary
FCJBOD
FCServer
FCServer
FC TapeLibrary
FCIP creates a single fabric by connecting E_Ports through gateways
IP Tunneling
TROIKA University 111
Connecting Storage Devices
IP Network
iFCPGateway
iFCPGateway
iFCPGateway
iFCPGateway
FCServer
FCServer
FC TapeLibrary
FCServer
FCServer
FC TapeLibrary
iSCSI attaches iSCSI storageiFCP attaches FC storage
iSCSIServer
iSCSIStorage
FCJBOD
FCJBOD
TROIKA University 111
Distributed File Systems
Network File System (NFS)
Server Message Block (SMB) Common Internet File System (CIFS) Samba
Many others (e.g. XNS, RFS, FTP)
TROIKA University 111
Server Message Block
TROIKA University 111
Direct Access File System
A file access and management protocol optimized for local file sharing and clustering
Designed to use memory-to-memory data movement technologies like VI and InfiniBand
TROIKA holds leadership positions
OS InterfaceLibrary
VI ProviderLibrary
Application
Adaptation Library orTransparency Library
DAFS ClientLibrary
File SystemSwitch
DAFS ClientFile System
VI ProviderDriver
OS InterfaceLibrary
Application
Example Deployment ModelsLocal File-Sharing Environment
TROIKA University 111
DAFS Protocol
Session Management• Operation Channel• Back-Control Channel• RDMA Read Channel• Security
File System Operations• Data Transfer
• Inline (Send/Receive)• Direct (RDMA)
• NFSv4-derived operations• Batch write operations• Client read-ahead into server prefetch buffer• Locking (Persist, Autorecover)• Chaining
Failure Recover• “Exactly Once” Semantics• Server Response Cache• Server Failover• File System Migration• Fencing
TROIKA University 111
DAFS Provider Architecture
TROIKA University 111
Positioning Technology
Understand the business goals
Select technologies designed to solve the technical challenges
Simplify to balance risk of deployment with cost of operation
TROIKA University 111
Applications Driving Network Growth
Applications Data Types / Size Traffic Implications
• Internet (public access)• Extranet (partner access)• Intranet (company access)
• Data (e-mail, browsers)• Audio/Video• High transaction rates• Large files to 100 MB
• Large files increase required bandwidth• Low transmission latency• Many data streams
• Scientific Modeling (3D)• Engineering (distr. teams)• Publications (digital printing)• Medical Data Transfer
• Data• 100s of MBs to GBs
• Large files increase required bandwidth
• Data warehousing• Network backup
• Data (queries, reports)• Updated regularly• GBs to terabytes
• Large files increase required bandwidth• Transmitted during fixed and limited time period
• Video conferencing• Interactive whiteboardiing
• Constant data stream•1.5 to 3.5 Mbps at desktop
• Class of service reservation (predicatable latency)• Many data streams
Reference: Gigabit Ethernet Alliance White Paper, 1998
TROIKA University 111
Network Positioning
Fibre Channel Well-suited for Storage Area Networks TROIKA enables converged networks
Gigabit Ethernet Designed for Ethernet compatibility TROIKA monitoring use for IP Storage
InfiniBand Designed for processor communication TROIKA considering product development
TROIKA University 111
Interface Positioning
Virtual Interface (VI) Designed for high-performance I/O Evolving to support InfiniBand
WinSock Direct / Sockets Direct Protocol Designed for transparent sockets acceleration on
Windows 2000 Datacenter Server – SoIB basis Direct Access Sockets (DASockets)
Designed for sockets acceleration on Unix and Windows platforms (not as transparent)
Direct Access File System (DAFS) Designed for high-performance local file sharing
TROIKA University 111
Building a Better Data Center
Storage NetworkCluster NetworkClient Network
Rest of World Server Farms and Appliances Block and File Storage
Block I/O – SCSI (FCP, SRP, iSCSI)File I/O – NFS, CIFS, DAFS, SDP
Intra-server – PCI, PCI-X, IBInter-server – Proprietary, Ethernet, FC, IB
Local – EthernetRemote – ATM, SONET, DWDM
InfiniBand
Fibre Channel
Gigabit Ethernet
Router
PCI-X, IB10/100, GbE
LAN
MANWAN
FC, GbE, IB
Affordable, Manageable, Available, and Secure
TROIKA University 111
Questions & Answers
Name the technologies, interfaces,and protocols we discussed.
Are the technologies converging or balkanizing?
Describe how to deploy these technologies to build a better data center.