pci express (pcie) overview - flash memory summit
TRANSCRIPT
PCI Express (PCIe)
Overview
Flash Memory Summit 2013
Santa Clara, CA
1
Peter Onufryk Sr. Director, Product Development
PMC-Sierra
NVMe and PCIe Layering
NVMe
• Driver interface (registers)
• Queuing interface
• Command set
• Command processing model
PCIe
• Reliable memory read/write transactions
• Discovery and configuration
• Switching & routing
• Physical layer
Flash Memory Summit 2013
Santa Clara, CA
2
PCIe Characteristics
Specification defined by PCI-SIG
• www.pcisig.com
Packet based protocol over serial links
• Software compatible with PCI and PCI-X
• Reliable in-order packet transfer
High performance and scalable from consumer to enterprise
• Scalable link speed (2.5 GT/s, 5.0 GT/s, 8.0 GT/s)
• Scalable link width (x1, x2, x4, …. x32)
Primary application is as an I/O interconnect
• Not a CPU interconnect
• Some multi-host applications (NTB, MR-IOV)
• Some outside the box applications (PCIe cable)
Flash Memory Summit 2013
Santa Clara, CA
3
PCIe and Server Architecture
Flash Memory Summit 2013
Santa Clara, CA
4
*Source - Intel
PCIe is Everywhere
Flash Memory Summit 2013
Santa Clara, CA
5
Blade Server
SERVERS
Mid-range / High End
COMMUNICATIONS
Metro Switch / Router Enterprise Switch / Router Wireless
STORAGE
RAID System SAN Switch NAS
CONSUMER
Small Business
Entertainment Imaging Mobile
PCIe Fabric Topology
Flash Memory Summit 2013
Santa Clara, CA
6
Root Complex
DP DPDP
PCIe Switch
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
Data Transfers
Flash Memory Summit 2013
Santa Clara, CA
7
Root Complex
DP DPDP
PCIe Switch
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UPE
nd
po
int
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
PCIe and NVMe
Flash Memory Summit 2013
Santa Clara, CA
8
NVMe Host
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
NVMe
SSD
Root Complex
DP DPDP
PCIe Switch
NVMe
SSD
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
tUP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
PCIe Layers
Flash Memory Summit 2013
Santa Clara, CA
9
TransactionLayer
Data LinkLayer
PhysicalLayer
TransactionLayer
Data LinkLayer
PhysicalLayer
PCI
Architecture
PCI
Architecture
PCIe Link
Upstream Port
PC
Ie Lane
Downstream Port
PC
Ie Lane
PC
Ie Lane
PC
Ie Lane
PCIe Link
Physical Layer
Scalable Speed
• Gen1 – 2.5 GT/s
• Gen2 – 5.0 GT/s
• Gen3 – 8.0 GT/s
Scalable Width
• x1, x2, x4, x8, x12, x16, x32
Encoding
• 8b10b: 2.5 GT/s & 5.0 GT/s
• 128b/130b: 8 GT/s
Flash Memory Summit 2013
Santa Clara, CA
10
PCIe Performance
Flash Memory Summit 2013
Santa Clara, CA
11
Generation Raw
Bit Rate
Bandwidth
Per Lane
Each Direction
Total x16
Link Bandwidth#
Gen 1 2.5 GT/s ~ 250 MB/s ~ 8 GB/s
Gen 2 5.0 GT/s ~500 MB/s ~16 GB/s
Gen 3 8 GT/s ~ 1 GB/s ~ 32 GB/s
Source – PCI-SIG PCI Express 3.0 FAQ
# Link bandwidth in each direction x 2 (full duplex)
Data Link Layer
Primary Function
• Reliable exchange of Transaction Layer Packets (TLPs)
between the two components of a Link
Other Functions
• Initialization (flow control credits)
• Power management
• Track and report link state to transaction layer (i.e., DL_up &
DL_Down)
Flash Memory Summit 2013
Santa Clara, CA
12
Transaction Layer
Primary Function
• Assembly and disassembly of
Transaction Layer Packets
(TLPs) for read and write
transactions
Other Functions
• Event signaling (e.g., interrupts,
power management, errors)
• Management of TLP credit
based flow control
Flash Memory Summit 2013
Santa Clara, CA
13
MR
d
Cpl
D
Cpl
D ... Cpl
D
Completer
Requester
Address Spaces
Flash Memory Summit 2013
Santa Clara, CA
14
Address Space Function
Memory Data transfer to/from memory mapped locations
- 64-bit memory address space
I/O Data transfer to/from IO-mapped locations
- 32-bit I/O space
Configuration Device Function configuration & setup
- 16-bit configuration space
Message Event signaling & general purpose messaging
Functions and Configuration Space
Function – an addressable entity in configuration
space
• Architecturally visible (i.e., can be discovered and configured)
• Capable of issuing requests and generating completions
Flash Memory Summit 2013
Santa Clara, CA
15
Bus FuncDev
8 5 3
Bus Func
8 8
Alternative Routing-ID Interpretation (ARI) IDTraditional ID
Type 0 and Type 1 Functions
Flash Memory Summit 2013
Santa Clara, CA
16
Type 0
Function
Function 0
Type 0
Function
Function 0
Type 0
Function
Function 1
Type 0
Function
Function 2
Bus Bus
Single Function
DeviceMulti-Function Device
Type 1
FunctionPCI-to-PCI
Bridge
Bus
Bus
Function Config. Space Registers
Flash Memory Summit 2013
Santa Clara, CA
17
PCI Compatible
Configuration Space
Header000h
0FFh
FFFh
PCI
Configuration
Space
PCI Express Extended
Configuration
Space
PCI Express
Extended Capability
MSI
Capability Structure
PCI Express
Capability Structure
PCI Express
Extended Capability
Advanced Error
Reporting
PCIe Switches and I/O Fan-out
Flash Memory Summit 2013
Santa Clara, CA
18
Root Complex
DP DPDP
PCIe Switch
En
dp
oin
t
UP UP
DP DPDP
PCIe Switch
UP
DP DPDP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
En
dp
oin
t
UP
PCIe Switch
Flash Memory Summit 2013
Santa Clara, CA
19
PCIe Switch
UP
DP DPDP Type 1
FunctionPCI-to-PCI
Bridge
Type 1
FunctionPCI-to-PCI
Bridge
Type 1
FunctionPCI-to-PCI
Bridge
Type 1
FunctionPCI-to-PCI
Bridge
PCIe Switch
Physical View
PCIe Switch Logical View
UP
DP DP DP
Virtual PCI Bus
Non-Transparent Bridge (NTB)
Flash Memory Summit 2013
Santa Clara, CA
20
Root
Complex
DP
PCIe Switch
UP
DP DPDP
En
dp
oin
t
UP
En
dp
oin
t
UP
Root
Complex
DP
PCIe Switch
UP
DPDP DP
En
dp
oin
t
UP
En
dp
oin
t
UPType 0
Function
Non-Transparent Bridge
Proprietary
Logic
Type 0
FunctionUPUP
PCIe Multi-Path Usage Model
Flash Memory Summit 2013
Santa Clara, CA
21
PCIe
SSD
PCIe
SSD
PCIe
SSD
PCIe
SSDPCIe
SSD
PCIe
SSD
PCIe Switch
PCIe
SSD
PCIe
SSD
PCIe Switch
Host Host
Inerconnect
PCIe PCIe
PCIe Overview Summary
PCIe is ubiquitous
PCIe provides a scalable interface for SSDs
• Scalable link width and speed
PCIe is not a bottleneck
• Highest performance standard I/O attach point
PCIe switches provide I/O fan-out
• Allows multiple SSDs to be connected to a Root Port
NVMe works within the standard PCIe framework
• Allows use of off-the-shelf Root Complexes and Switches
PCIe may be used to connect multiple hosts to an SSD
Flash Memory Summit 2013
Santa Clara, CA
22