mike krause and renato recio pci sig iov work group co-chairs
TRANSCRIPT
![Page 1: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/1.jpg)
Implementing PCI I/O Virtualization StandardsMike Krause and Renato RecioPCI SIG IOV Work Group Co-chairs
![Page 2: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/2.jpg)
Today’s Approach To IOVVirtualization Intermediaries (VI) and hypervisors are used to safely share IO
Under this approach1 or more System Images share the PCI device through a VI
Virtualization enablers are not needed in the either the Root Complex (RC) or PCIe Device
The VI is involved in all IO transactions and performs all IO Virtualization Functions
VI Based PCI Device Sharing Example
System Images share the adapter through a
VI.MMIO and DMA
operations go through the VI
PCIe RC
PCIe Device
PCIe Port
F
Today’s PCIe Device with one or more
Functions.The Device may not be cognizant at all
that it is being shared
Hypervisor
SI SIVI
![Page 3: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/3.jpg)
Performance Of Today’s IOV
VI based IOV adds path length on every IO operationNative IOV significantly improves performance
For the example above Native IOVdoubles throughput and reduces latency by up to half
Several factors are increasing the virtualization use (e.g., more cores per socket, customer simplification requirements, …)
Making Native IOV even more important in the future
Native IOVVI Based IOV
Native IOVVI Based IOV
Through-put*Latency*
*Source: Self-Virtualized I/O: High Performance, Scalable I/O Virtualization in Multi-core Systems; R. Himanshu, I. Ganev, K. Schwan - Georgia Tech and J. Xenidis - IBM
![Page 4: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/4.jpg)
PCI SIG IOV Overview
PCI SIG is standardizing mechanisms that enable PCIe Devices to be directly shared, with no run-time overheads
Single-Root IOV – Direct sharing between SIs on a single systemMulti-Root IOV – Direct sharing between SIs on multiple systems
PCI-SIG IOV Specification covers “north-side” of the Device“PCI IOV Usage Models and Implementations” session will cover examples of how PCI IOV specifications can be used tovirtualize PCIe Devices
Hypervisor
SI SI
Hypervisor
PCIe Topology
PCIe RC
FC SAN Ethernet LAN
PCIe RC Blade
PCIe Multi-Root IOV
VISI SI
Hypervisor
PCIe RC
PCIe Single-Root IOVVI
PCIe IOV Capable Device
PCIe PortPCIe MR-IOV
Capable FC DevicePCIe MR-IOV
Capable Enet Device
FC SAN
SI SIVI
Blade
![Page 5: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/5.jpg)
TerminologySystem Image (SI)
SW, e.g., a guest OS, to which virtual and physical devices can be assigned
Virtual Intermediary (VI)Performs resource allocation, isolation, management and event handling
PCIM – PCI ManagerControls configuration, management and error handling of PFs and VFsMay be in SW and/or Firmware.May be integrated into a VI
Translation Agent (TA )Uses ATPT to translates PCI Bus Addresses into platform addresses
Address Translation and Protection Table (ATPT)
Validates access rights of incoming PCI memory transactions.Translates PCI Address into platform physical addresses
Processor
Memory
Hypervisor
VI
PCIe Root Complex
ATPTTA
PCIeDevice
F
PCIeDevice
F
PCIeSwitch
VISR-PCIM
PCIe Port PCIe Port
SISI
PCIe Port PCIe Port
![Page 6: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/6.jpg)
Terminology… Continued
Single-Root IOV (SR-IOV) - A PCIe hierarchy with one or more components that support the SR-IOV CapabilityMulti-Root Aware (MRA) - A PCIe component that supports the MR-IOV capabilityMulti-Root IOV (MR-IOV) - A PCIe Topology containing one or more PCIe Virtual Hierarchies
Virtual Hierarchy (VH) - A portion of an MR Topology assigned to a PCIe hierarchy, where each VH has its own PCI Memory, IO, and Configuration Space
PCIe Fabric
PCIe MRADevice
PCIeDevice
PCIe SR-IOV MRA Device
Hypervisor
SI SI
Hypervisor
PCIe RC PCIe RC Blade
PCIe Multi-Root IOV
VISI SI
Hypervisor
PCIe RC
PCIe Single-Root IOVVI
PCIe IOV Capable Device
PCIe Port
SI SIVI
Blade
![Page 7: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/7.jpg)
VH0
VHNVH1
Terminology… ContinuedAddress Translation Cache (ATC)
Cache of recent translations
Physical Function (PF)Function that supports SR-IOVUsed to manage VFs
Virtual Function (VF)Function that supports SR-IOV and shares resources with the PF it associated with
Base Function (BF)Function that supports MR-IOVUsed to manage PFs and VFs on MRA Devices
Internal
Routing
PF0
VF1
VF2
VFN
:
ATC1
Resources1
ATC2
Resources2
ATCN
ResourcesN
Internal
Routing
PF0
VF1
VF2
VFN
:
BF0
PF0
VF1
VF2
VFN
:
….
PCIe Port
PCIe Port
PCIe SR-IOV
Capable Device
PCIe MR-IOV
Capable Device
![Page 8: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/8.jpg)
PCI IOV Related Mechanisms
Function Level Reset (FLR)Alternative Routing ID Interpretation(ARI)Address Translation ServicesSingle-Root IO VirtualizationMulti-Root IO Virtualization
![Page 9: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/9.jpg)
FLR And ARI
FLR - Provides Function level granularity on resetsAll software readable state must be cleared by an FLRAll outstanding transactions associated with the Function referenced by the FLR must be completed when the FLR is returned as completed
ARI - Extends Function number field from 3 to 8 bitsAllows up to 256 Functions or VFsper PCIe Device
VI SI SI
PCIeDevice Routing
Hypervisor
PCIe Root Complex ATPT
PCIe Port
ConfigMgt
Function1Function2Function3
PCIe Topology
FLRs
PCIe DeviceFC HBA
VF1 VF2 VF3
Routing
PCIe Port
PF0 FC Port FC Port
SI
FC SAN FC SAN
![Page 10: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/10.jpg)
Address Translation Services
ATS is used to cache PCIe Memory Address translations in a PCI Device.Consists of three new PCIe transactions
Request Translation Transaction – Used by a PCIe Device to request a translation of an untranslated addressTranslated DMA Transaction – Used to perform a DMA that references a translated addressInvalidate Translation Transaction – Used to invalidatea previously exposed translated address
PCIe Device
VF1 VF2 VF3
PCIe Device with Address Translation
Services
ATC1 ATC2 ATC3
Routing
PCIe Port
PF0
DownstreamPort
Hypervisor
PCIe Root Complex ATPT
PCIe Topology
SISI SIVI
![Page 11: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/11.jpg)
PCI Single-Root IOV
Overview
![Page 12: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/12.jpg)
PCIe SR-IOVCapableDevice
Today’s PCI Device
Function 0 is requiredOverview of Function Attributes
Each Function has a its own configuration and PCIe memory address spaceUp to 8 PCI Functions with unique configuration space / BAR / etc.
ARI Capability enables up to 256 Functions to be supportedSupport INTx, MSI, MSI-X or combination of MSI and MSI-XFunction dependencies through vendor specific mechanismsCannot be directly shared by SIsVendor specific mechanisms to associateFunctions to “South-side” resources
Downstream
PortInterna
l Routin
g
Internal
Routing
F0
F1
F2
FN
:
ATC1
Resources1
ATC2
Resources2
ATCN
ResourcesN
PCIe Port
![Page 13: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/13.jpg)
SR-IOV Device Overview
Function 0 is requiredOverview of VF attributes
Each VF has a its own configuration and PCIe memory address spaceVFs share a contiguous PCIe memory address space
Up to ~216 Virtual FunctionsARI enables up to 256
IOV enables additional Bus Numbers to be associated Function dependencies defined through standard mechanism
Support MSI and MSI-XCan be directly shared by SIs
Vendor specific mechanisms to associate Functions to “South-side” resources
PCIe SR-IOVCapableDevice
Downstream
PortInterna
l Routin
g
Internal
Routing
PF0
VF1
VF2
VFN
:
ATC1
Resources1
ATC2
Resources2
ATCN
ResourcesN
PCIe Port
![Page 14: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/14.jpg)
SR-IOV Device DiscoveryEach PF must have an SR-IOV Extended Capability structure
A Device may have a mix of Functions and PFs
Each Function and PF consumes one Routing Identifier (RID)
For a Device thatisn’t ARI capable, Functions and
PFs must be in the first 8 functions;
is ARI capable, number of Functions and PFs supported is as defined
in the base specification31 20
19 16
15 0
Offset
Next Capability Pointer Cap. Vrsn. Capability ID 00h
SR IOV Capabilities 04h
: :
PCI Root
SI
A
SI
B
SR
PCIM
PF0
PF3
PF1
F2
PCIe Topology
![Page 15: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/15.jpg)
VF Discovery - Part 1The TotalVFs field is used to discover the number of Active VFs that a PF could have.
The InitialVFs field is used to discover the number of Active VFs that a PF initially has.
Note for a Device that isn’t MR capable*: 1 ≤ InitialVFs = TotalVFs
31 20
19
16
15 0
Offset
: :
TotalVFs (RO) InitialVFs (RO) 0Ch
: :
PCI Root
SI
A
SI
B
SR
PCIM
PF0
PF3
PF1
F2
PCIe Topology
![Page 16: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/16.jpg)
BAR DiscoveryEach PF has its own independent set of BARs in its standard configuration space and an MSE bit for those BARsThe VFs share a BAR set and
have an MSE bit that controls the memory space of all the VFs
The BAR set that is shared by all the VFs resides in the PF’s SR-IOV capabilities
31 20
19 16
15 0
Offset
: :VF BAR0 (RW) 20h
: :VF BAR5 (RW) 34h
: :
PCI Root
SI
A
SI
B
SR
PCIM
PF0
PF3
PF1
F2
PCIe Topology
![Page 17: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/17.jpg)
Supported Page SizesSupported Page Sizes is used to discover the page sizes supported by the VFs associated with the PF
When this field is read, the PF must return the page sizes it can supportThis field will be used during the IOV configuration phase to align VF BAR apertures on system page boundaries (more later)31
2019 16
15 0
Offset
: :
Supported Page Sizes (RO) 18h
: :
PCI Root
SI
A
SI
B
SR
PCIM
PF0
PF3
PF1
F2
PCIe Topology
![Page 18: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/18.jpg)
Configuring VFsVFs for a PF are enabled by writing NumVFs and then Setting the “VF Enable” bit
When VF’s are enabled, the PCIe Device must associate NumVFsworth of VFs with the PF
If VF Migration Capable and VF Migration Enabled set, then NumVFs must be
1 ≤ NumVFs ≤ TotalVFsOtherwise NumVFs must be
1 ≤ NumVFs ≤ InitialVFs
31 20
19
16
15 0
Offset
: :SR IOV Capability (RO) 04h
SR IOV Status SR IOV Control (RW) 08hReserved NumVFs (RW) 10h
: :
VF Enable
VF Migration Enable
VF Migration Capable
PCI Root
SI
A
SI
B
SR
PCIM
PF0
PF3
PF1
F2
PCIe Topology
![Page 19: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/19.jpg)
Configuring The VFs’ BARsSystem Page Size defines the page size the system will use
Value of System Page Size must be one of the Supported Page SizesSystem Page Size is used by the PF to align the MMIO aperture defined by each BAR to a system page boundary
VF BARs behave as in PCI 3.0 Spec’s PCI BARs, except that a VF BAR describes the aperture for multiple VFs (see next page)31
2019 16
15 0
Offset
: :System Page Sizes (RW) 1Ch
VF BAR0 (RW) 20h: :
VF BAR5 (RW) 34h: :
PCI Root
SI
A
SI
B
SR
PCIM
PF0
PF3
PF1
F2
PCIe Topology
![Page 20: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/20.jpg)
PF And VF BAR SemanticsThe memory aperture required for each VF BAR can be determined by writing all “1”s and then reading the VF BAR
Behaves as in the base PCI SpecThe address written into each VF BAR is used by the Device to set the starting address for that BAR on the first VFThe differences between VF BARs and PCI 3.0 Spec BARs are
For each VF BAR, the memory space associated with the 2nd and higher VFs is derived from the starting address of the first VF and the memory space apertureThe VF BAR’s MMIO space is not enabled until VF Enable and VF MSE have been set
PCIe MMIO Space
PF BAR0 MMIO Aperture
VF 1 BAR0 MMIO Aperture
VF 2 BAR0 MMIO Aperture
:VFN BAR0 MMIO
Aperture
PF Config Space
:
BAR0 (RW)
:PF IOV Config
Space
:
VF BAR0 (RW)
:
BAR1 VF N SA = BAR1 VF 1 SA + N x (BAR1 VF MA) - 1
where BAR1 VF 1 SA is the address written into VF BAR1and MA is the memory aperture of VF BAR1.
![Page 21: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/21.jpg)
VF Discovery – Part 2The values in NumVFs and VF ARI Enable affect the values for First VF Offset and VF Stride*
First VF Offset and VF Stride are defined by the DeviceFollowing is the equation for determining the RID of VF N:VF N = [PF RID + VF Offset + (N-1) * VF Stride] Modulo 216
where all arithmetic used is 16 bit unsigned dropping all carries
31 20
19 16
15 0
Offset
: :
VF Stride (RO) First VF Offset (RO) 14h
: :
PCI Root
SI
A
SI
B
SR
PCIM
PF0
PF3
PF1
F2
PCIe Topology
*Note: After setting NumVFs and VF ARI Enable, SR-PCIM can read these two fields to determine how many busses will be consumed bt the PF’s VFs
![Page 22: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/22.jpg)
PF And VF BAR SemanticsPF and VF RIDs must not overlap given any valid NumVFs setting across all PFs of a DeviceAs in base, an SR-IOV Device captures Bus # from any Type 0 config request
VFs may reside on a different bus number as associated PF
If the switch above the Devicesupports ARI Forwarding bit, RIDs are interpreted as 8 bit Bus # and 8 bit Device #doesn’t supports ARI, RIDs are interpreted as BDF#
Bus # can be used to assign VFs,but PFs must be on 1st Bus assigned to Device
RID # Func.
: :0200 PF 0
0201 PF 1
0202 VF 0,1
0203 VF 1,1
0204
0205
0206 VF 0,2
0207 VF 1,2
0208
0209
020A VF 0,3
020B VF 1,3
020C
020D
020E VF 0,4
: :0310
0311
0312
VF 0,40
: :
PCIe RID Number
PF 0 IOV Config Space
:
Reserved
NumVFs = 0050x
VF Stride
= 0004x
VF Offset = 0002x
:
PF 0 RID = 0200
PF 1 IOV Config Space
:
Reserved
NumVFs = 0003x
VF Stride
= 0004x
VF Offset = 0002x
:
PF 1 RID = 0201
![Page 23: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/23.jpg)
Reset Mechanisms
Three reset mechanisms are supported
Conventional Reset – Resets all PF and VF state
FLR that targets a VF – Resets a single VF
FLR that targets a PF – Resets a PF and its associated VFs
![Page 24: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/24.jpg)
Conventional Reset
A Fundamental or Hot Reset to an SR-IOV Device shall cause all Functions, PFs, and VFs context to be reset to their original, power-on stateIf a PF has its VFs enabled and a Fundamental or Hot Reset is issued to the Device, the Device must reset all PF and VF state, eg:
The PF must disable its SR-IOV capabilities andreverts back to being a PCI FunctionSettable SR-IOV capabilities (e.g., NumVFs) are reset to default valuesMSE and BME are both offAll BAR values used by the PF and VFs are indeterminate.All interrupts are disabled
![Page 25: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/25.jpg)
FLRs To VFs And PFs
An FLR that targets a VF must be supported
Software may use FLR to reset a VF
An FLR that targets a VF mustNot affect the VFs existence (e.g. it still consumes a RID)
Not affect any address assigned to it. That is, the VF’s BAR registers and MSE are unaffected by FLR
An FLR that targets a PF must be supported
Software may use an FLR to reset a PF
An FLR that targets a PF mustReset the PF’s SR-IOV Extended Capabilities
The VFs are no longer allocated or enabled
![Page 26: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/26.jpg)
PCI Multi-Root IOV
Overview
![Page 27: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/27.jpg)
Today’s PCIe Topology
Strict TreeRoot Ports Connect to Switches
Switches Connect to Devices
Roots Ports can also connect to Devices
Switches can also connect to more Switches
Single Software Management EntityRuns above the Root
Implicit Message RoutingRoute to Root
Broadcast from Root
![Page 28: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/28.jpg)
MR Switch
Single Switch MR Topology
SI
MR Device MR Device MR DevicePCIe Device
Root BRoot A Root C Root D
FC SAN Ethernet LAN
Ethernet LAN
FC SAN
![Page 29: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/29.jpg)
Two Switch MR Topology
RightLeft
MR Device
FC SAN
MR Device MR Device
Root BRoot A Root C Root D
Ethernet LAN
Ethernet LAN
PCIe Device
FC SAN
![Page 30: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/30.jpg)
TLP Labeling
TLP Prefix on MR LinksAfter Seq #, before TLP Hdr
Sent / Resent with TLP
Prefix ContainsVH Number (link local)
VL Number (flow control)
Global Key (error checking)
Added at MR IngressFirst MR Component seen
Dropped at MR EgressLast MR Component seen
Sequence #STP
TLP Prefix
PCIe TLP Header
TLP Data (optional)
ECRC (optional)
LCRC
END
ECRC
LCRC
![Page 31: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/31.jpg)
SR-IOV Device Overview
MR-IOVCapableDevice
PF0
PF0 VFN
PF0 VF1Internal
Routing
PCIe Port
: DownstreamPort
Internal
Routing
![Page 32: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/32.jpg)
MR-IOVCapableDevice
MR-IOV Device Overview
Each Virtual Hierarchy (VH) has a full PCIe address spaceConfiguration, Memory and I/O Space
Up to 28 Virtual HierarchiesVH0 is used to Manage MR-IOV features of the Device
BF in VH0 manages associated PFs and VFsOther Functions optional in VH0
VH1 to VHmax have same PFs at the same Function #sEach PF in each VH has it’s own values for InitialVFs, TotalVFs, NumVFs, VF Stride and VF Offset
VH1 PF0
VH1 PF0 VFN
VH1 PF0 VF1Internal
Routing
PCIe Port
: DownstreamPort
Internal
Routing
VHK PF0 VF1
VHK PF0 VFM
:
VHK PF0VH0 BF0
![Page 33: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/33.jpg)
MR-IOV Device Configuration
Initially, only VH0 is enabled
Within VH0, MR-PCIM enumerates Config SpaceLocate all BFs by looking for MR-IOV Capability
For each Device located…In BF0
Determine MaxVH, set NumVHEnable additional VLs (if available)Configure VL arbitration (optional)Configure per VH mapping of VCID to VLConfigure per VH Global Keys
In every BFIn each VH, provision VC Resources (optional)If hardware present, configure Initial VF mapping
![Page 34: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/34.jpg)
PCI Specification Schedule
ATSSR-IOVMR-IOV
![Page 35: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/35.jpg)
Schedules
ATS Specification released 3/8/2007SR-IOV Specification
Following PCI SIG Specification ProcessDraft 0.7 CompletedDraft 0.9 May, 2007Version 1.0 early 3Q/2007
MR-IOV SpecificationFollowing PCI SIG Specification Process.Draft 0.5 CompletedDraft 0.7 2Q/2007Draft 0.9 late 2Q/2007Version 1.0 late 3Q/2007
![Page 36: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/36.jpg)
Call To Action
Please participate in the PCI-SIG Specification Development Process
For more information please go to www.pcisig.com
Thank you for attending
![Page 37: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/37.jpg)
Workgroup MembersAMD
Broadcom
Emulex
HP
IBM
IDT
Intel
LSI Logic
Microsoft
NextIO
Neterion
Nvidia
PLX
Qlogic
Stargen
Sun
VMWare
![Page 38: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/38.jpg)
Questions
![Page 39: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/39.jpg)
MR Topics
Back-ups
![Page 40: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/40.jpg)
Switch TLP Routing
Three step routing decision:MR Input Map
{ Input Port, Input VH# } { VS, VS Input Port }MR-PCIM Manages
VS PCIe Map{ VS Input Port, TLP Hdr } { VS Output Port }VH Software Manages using PCIe rules
MR Output Map{ VS, VS Output Port } { Output Port, Output VH# }MR-PCIM Manages
![Page 41: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/41.jpg)
Hot Reset
PCIe Hot Reset is broadcast downstream
Switch Upstream port all Downstream portsUses Phy Training SequenceHandshake to detect link partner received
MR Reset is per VHVS Upstream port all VS Downstream portsUses Reset DLLPHandshake to detect link partner receivedLink stays up, other VHs unaffected
![Page 42: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/42.jpg)
Flow Control
PCIe uses Virtual Channels (VC) for QoS, traffic isolation, etc.MR Extends this to Virtual Links (VL)MR adds per VH, VL flow control
Traffic GatePCIe: sufficient VC Credits to send TLPMR: sufficient VL Credits to send TLPand sufficient VH, VL Credits to send TLP
![Page 43: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/43.jpg)
Management / Authorization
Devices are managed using VH0
Switches are managed using the Upstream port of any Authorized VS
Multiple VS may be simultaneously authorized
Supports MR-PCIM failover
At failover, new MR-PCIM remaps VH0 so it can mange MR Devices
![Page 44: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/44.jpg)
Virtual Hot Plug
Virtual Hot Plug is implemented in each Downstream Port of each VS
Software in VH sees PCIe Slot Control Regs
MR-PCIM controls virtual “buttons & lights”
e.g. pushing the virtual Attention Button, detecting virtual slot power state, …
Physical Hot-Plug is implemented in each Physical Port
Managed by MR-PCIM
Optional (same as PCIe)
![Page 45: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/45.jpg)
Power Management
Each PF has a power stateif all PFs in low power state, link can go to low power state (like PCIe multi-function rules)
PM_PME Messages sometimes trigger WAKE#
e.g., some Root powered itself off but a shared device below it was only virtually powered off (Device still being used by other VHs)
![Page 46: Mike Krause and Renato Recio PCI SIG IOV Work Group Co-chairs](https://reader036.vdocuments.mx/reader036/viewer/2022062515/56649cf45503460f949c2968/html5/thumbnails/46.jpg)
© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this
presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.