pci express port bus driver support for linux · pdf file2 • pci express port bus driver...

PCI Express Port Bus Driver Support for Linux

Tom Long Nguyen, Dely L. Sy, & Steven CarbonariIntel R© Corporation∗

{tom.l.nguyen, dely.l.sy, steven.carbonari}@intel.com

Abstract

PCI ExpressR©1 is a high performance gen-eral purpose I/O Interconnect defined for awide variety of computing and communicationplatforms. It defines PCI Express Ports andswitches to provide a fabric based point-to-point topology. PCI Express categorizes PCIExpress Ports into three types: the Root Ports,the Switch Upstream Ports, and the SwitchDownstream Ports. Each PCI Express Portcan provide up to four distinct services: na-tive hot-plug, power management, advanced er-ror reporting, and virtual channels[1][3]. Tofit within the existing LinuxR©2 PCI DriverModel but provide a clean and modular solu-tion, in which each service driver can be builtand loaded independently, requires the PCI Ex-press Port Bus Driver architecture. The PCI Ex-press Port Bus Driver initializes all services anddistributes them to their corresponding servicedrivers. This paper is targeted toward kerneldevelopers and architects interested in the de-tails of enabling service drivers for PCI ExpressPorts. The i386 Linux implementation will beused as a reference model to provide insightinto the implementation of the PCI Express

∗Intel is a trademark or registered trademark of IntelCorporation in the United States, other countries, or both.This work represents the view of the authors and does notnecessarily represent the view of Intel.

1PCI Express is a trademark of the Peripheral Com-ponent Interchange Special Interest Group (PCI-SIG)

2Linux is a registered trademark of Linus Torvalds

Port Bus Driver and specific service drivers likethe advanced error reporting root service driverand the native hot-plug root service driver.

1 Introduction

The Linux PCI Driver Model restricts a deviceto a single driver. Drivers in Linux are loadedbased off the PCI Device ID and function. Oncea driver is loaded, no other drivers for that de-vice can be loaded[2]. Referring to Figure 1,if the Root Port hot-plug driver is loaded first,it claims the Root Port device. The Linux PCIDriver Model therefore prevents the support ofmultiple services per PCI Express Port usingindividual service drivers.

Figure 1: Service Drivers under the Linux PCIDriver Model

A PCI Express Port may have multiple distinctservices operating independently. A PCI Ex-press Port is not required to support all ser-vices, so some PCI Express Ports within a PCIExpress hierarchy may support none, some orall the services. A possible solution is to im-plement a single driver to handle all services

• 1 •

2 • PCI Express Port Bus Driver Support for Linux

per PCI Express Port. However, this solutionwould lack the ability to have each service builtand loaded independently from each other, pre-venting extensibility for addition of future ser-vices and the ability to have a service driverloaded on more than one PCI Express Port.Separate service drivers are required to supportaddition of new features and loading of servicesbased on the PCI Express Port capabilities.

To support multiple drivers per device withoutchanging the existing Linux PCI Driver Modelrequires a new architecture that fits withinthe existing Linux PCI Driver Model but pro-vides the flexibility required to support multi-ple service drivers per PCI Express Port. Asshown in Figure 2, the PCI Express Port BusDriver (PBD)[5] fits into the existing Linux PCIDriver Model while providing an interface toallow multiple independent service drivers tobe loaded for a single PCI Express Root Port.The PBD acts as a service manager that ownsall services implemented by the Ports. Each ofthese services is then distributed and handledby a unique service driver. The PBD achievesthe following key advantages:

• Allows multiple service drivers to runsimultaneously and independently fromeach other and to service more than onePCI Express Port.

• Allows service drivers to be designed andimplemented in a modular fashion.

• Centralizes management and distributionof resources of the PCI Express Port de-vices to requested service drivers.

This paper describes the PCI Express Port BusDriver architecture. Following the port busdriver architecture are two examples of servicedrivers. The first example is the advanced er-ror reporting service driver that was designed to

Figure 2: Service Drivers under the PBD

work with the port bus architecture. The secondexample is the hot-plug service driver that wasoriginally designed as an independent driverthen converted to a service driver to operatewith the Port Bus Driver. Lastly, an overviewof the impact to device drivers and future ser-vice drivers is outlined.

2 PCI Express Port Bus Driver

2.1 PCI Express Port Topology

To understand the Port Bus Driver architecture,it helps to begin with the basics of PCI ExpressPort topology. Figure 3 illustrates two types ofPCI Express Port devices: the Root Port andthe Switch Port. The Root Port originates aPCI Express Link from a PCI Express RootComplex. The Switch Port, which has its sec-ondary bus representing switch internal rout-ing logic, is called the Switch Upstream Port.The Switch Port which is bridging from switchinternal routing buses to the bus representingthe downstream PCI Express Link is called theSwitch Downstream Port[1].

Each PCI Express Port device can be imple-mented to support up to four distinct services:native hot plug (HP), power management event(PME), advanced error reporting (AER), virtualchannels (VC). The PCI Express services dis-cussed are optional, so in any given PCI Ex-press hierarchy a port may support none, some,or all of the services.

2005 Linux Symposium • 3

Figure 3: PCI Express Port Topology

2.2 PCI Express Port Bus Driver Architec-ture

The design of the PCI Express Port Bus Driverachieves a clean and modular solution in whicheach service driver can be built and loaded in-dependently from each other. The PCI ExpressPort Bus Driver serves as a service managerthat loads and unloads the service drivers ac-cordingly, as illustrated in Figure 4.

Figure 4: PCI Express Port Bus Driver SystemView

The PCI Express Port Bus Driver is a PCI-PCI Bridge device driver, which attaches to PCIExpress Port devices. For each PCI ExpressPort device, the PCI Express Port Bus Driversearches for all possible services, such as na-tive HP, PME, AER, and VC, implemented byPCI Express Port device. For each service

found, the PCI Express Port Bus Driver createsa corresponding service device, named pcieXYwhere X indicates the PCI Express Port typeand Y indicates the PCI Express service typeas described in Table 1, and then registers thisservice device into a system device hierarchy.Figure 5 shows an example of how the PCI Ex-press Port Bus Driver creates service deviceson a system populated with two Root Port de-vices, one Switch Upstream Port device, andtwo Switch Downstream Port devices.

Port Service Service Entity Description (pcieXY)Type Type(X) (Y)0 0 PME service on PCI Express Root Port (PMErs)0 1 AER service on PCI Express Root Port (AERrs)0 2 HP service on PCI Express Root Port (HPrs)0 3 VC service on PCI Express Root Port (VCrs)1 0 PME service on PCI Express Switch Upstream Port (PMEus)1 1 AER service on PCI Express Switch Upstream Port (AERus)1 2 Not a supported PCI Express configuration1 3 VC service on PCI Express Switch Upstream Port (VCus)2 0 PME service on PCI Express Switch Downstream Port (PMEds)2 1 AER service on PCI Express Switch Downstream Port (AERds)2 2 HP service on PCI Express Switch Downstream Port (HPds)2 3 VC service on PCI Express Switch Downstream Port (VCds)

Table 1: Service Entity Description

Figure 5: Service Devices in a PCI Express PortBus Driver Architecture

Once service devices are discovered and addedin the system device hierarchy, a service driveris loaded accordingly if it registers its ser-vice with the PCI Express Port Bus Driver.The PCI Express Port Bus Driver providesan interface, namedpcie_port_service_register , to allow a service driver to registerits service[4]. The registration enables the user


to configure services during kernel configura-tion regardless of HW support. It enables de-bugging and adding of new services in a modu-lar fashion. When a service driver callspcie_port_service_register , the PCI Ex-press Port Bus Driver loads a service driverby invoking the PCI subsystem, which walksthrough a system device hierarchy for a servicematch. If the port bus finds a match, it loads aservice driver for that service device.

In addition, the PCI Express Port BusDriver provides pcie_port_service_

unregister , to undo the effects of callingfunction pcie_port_service_register

[4]. Note that a service driver should alwayscall pcie_port_service_unregister

when a service driver is unloading.

2.3 The Service Driver

To maintain modularity in the PCI Express PortBus Driver design, individual service driversare required. In some cases a driver may al-ready exist for a PCI Express Port. In these in-stances the driver must be ported to the servicedriver to allow other service drivers to load onthe PCI Express Port. To port drivers to servicedrivers, the following three basic steps are re-quired. Refer to Sections 3.1.1 to 3.1.3 for aspecific example.

• Specify service ID. The PCI Express PortBus Driver defines the data structure ofservice ID similar to the data structureof pci_device_id except with twoadditional fields: theport_type andservice_type fields as described inTable 1. Note that failure to specify a cor-rect service ID will prevent the port busfrom loading a service driver.

• Initialize service driver. The PCI ExpressPort Bus Driver defines the data struc-ture of service driver similar to thepci_

driver data structure. The pointer to thepci_dev data structure is replaced with apointer to thepcie_device data struc-ture in each callback function.

• Call pcie_port_service_register

insteadpci_register_driver .

Once a service driver is loaded, a service drivershould always configure and initialize its owncapability structure and required IOs to oper-ate normally without any support from the PCIExpress Port Bus Driver. However, a servicedriver is prohibited from doing the following:

• Switch the interrupt mode on a device.The interrupt mode can be INTx legacy,MSI or MSI-X. A service driver should al-ways use the assigned service IRQ to callrequest_irq to have its software in-terrupt service routine hookup. Note thatthe assigned service IRQ may be sharedamong service drivers; therefore, a servicedriver should always treat this assignedservice IRQ as shared interrupt.

• Access resources that are not directly re-quired by the service. For example, theadvanced error reporting service driver isprohibited from accessing any configura-tion registers other than the Advanced Er-ror Reporting Capability structure. A ser-vice driver uses theport pointer, a mem-ber of thepcie_device data structuredefined by PBD, to access PCI configura-tion and memory mapped IO space.

• Call pci_enable_device and pci_set_master functions. This is nolonger necessary because these functionsnow get called by the PCI Express PortBus Driver.


2.4 Resource Allocation and Distribution

Service drivers must adhere to the guidelines inthis document to deal with resource allocationand distribution. Since all service drivers of aPCI Express Port device are allowed to run si-multaneously, a decision of which driver (PortBus Driver vs. service driver) owns which re-source is described in Sections 2.4.1 to 2.4.3.These resources include the MSI capabilitystructure, the MSI-X capability structure, andPCI IO resources.

2.4.1 The MSI Capability Structure

The MSI capability structure enables a devicesoftware driver to callpci_enable_msi torequest an MSI based interrupt. Once MSI isenabled on a device, it stays in this mode un-til a device driver callspci_disable_msito return to INTx emulation. Since each ser-vice driver runs independently from each other,changing the interrupt mode on the PCI Ex-press Port by an individual service driver mayresult in unpredictable behavior. Each ser-vice driver is therefore prohibited from call-ing these APIs. The PCI Express Port BusDriver is responsible for determining the in-terrupt mode and assigning the service IRQto each service device accordingly. A servicedriver must use its service vector when callingrequest_irq /free_irq .

2.4.2 The MSI-X Capability Structure

Similar to MSI a device driver for an MSI-X ca-pable device can callpci_enable_msix torequest MSI-X interrupts. The key differenceis that the MSI-X capability structure enablesa PCI Express Port device to generate multi-ple messages. Managing multiple MSI-X vec-tors is handled by the PCI Express Port Bus

driver. The PCI Express Port Bus Driver is re-sponsible for determining the interrupt modetransparent to the service drivers. A servicedriver must use its service vector when callingrequest_irq /free_irq .

If a PCI Express Port device supports MSI-X,the PCI Express Port Bus Driver will requestthe number of MSI-X messages equal to thenumber of supported services for the device.This allows each service to have it own hard-ware interrupt resource independently gener-ated from other services.

2.4.3 PCI IO Resources

PCI IO resources include PCI memory/IOranges and PCI configuration registers are as-signed by BIOS during boot. For PCI mem-ory/IO ranges, the service driver is responsiblefor initializing its PCI memory/IO maps duringservice startup. There is possibly where the PCImemory/IO ranges are shared. If this occurs,each service driver is responsible for mappingits PCI memory/IO regions without overstep-ping on resources of others. The PCI ExpressPort Bus Driver does not arbitrate access to theregions and assumes service drivers to be wellbehaved.

For PCI configuration registers, each serviceruns PCI configuration operation on its own ca-pability structure except the PCI Express ca-pability structure, in which the Device Con-trol register and the Root Control register haveunique control bits assigned to AER serviceand PME service. A read-modify-write shouldalways be handled by the AER/PME servicedriver. Again this paper assumes that all servicedrivers are responsible for not overstepping onresources of others.


3 PCI Express Advanced Error Re-porting Root Service Driver

PCI Express error signaling can occur on thePCI Express link itself or on behalf of trans-actions initiated on the link. PCI Express de-fines the Advanced Error Reporting capability,which is implemented with the PCI ExpressAdvanced Error Reporting Extended Capabil-ity Structure, to allow a PCI Express compo-nent (agent) to send an error reporting messageto the Root Port. The Root Port, a host receiverof all error messages associated with its hierar-chy, decodes an error message into an error typeand an agent ID and then logs these into its PCIExpress Advanced Error Reporting ExtendedCapability Structure. Depending on whether anerror reporting message is enabled in the RootError Command Register, the Root Port devicegenerates an interrupt if an error is detected[1].The PCI Express advanced error reporting ser-vice driver is implemented to service AER in-terrupts generated by the Root Ports[6].

Once the PCI Express advanced error report-ing service driver is loaded, it claims all AERRoot service devices in a system device hierar-chy, as shown in Figure 6. For each AERrs ser-vice device, the advanced error reporting ser-vice driver configures its service device to gen-erate an interrupt when an error is detected. Foreach detected error, the advanced error report-ing service driver performs the followings[6]:

• Gather comprehensive error information.

• Guide error recovery associated with thehierarchy in question based on the com-prehensive error information gathered.

• Report error to user in a format with moreprecise what error type and severity.

Figure 6: AER Root Service Driver

3.1 Register AER Service

The advanced error reporting service driver isimplemented based on the service driver frame-work as defined in Section 2.3. Sections 3.1.1to 3.1.3 below illustrate how the advanced er-ror reporting service driver follows three basicsteps as required.

3.1.1 Specify AER Service ID

Since the PCI Express advanced error reportingservice driver is implemented to serve only theRoot Ports, the data structure of AER serviceID is defined below[7]:

static struct pcie_port_service_id aer_id[]={{.vendor = PCI_ANY_ID,.device = PCI_ANY_ID,.port_type = PCIE_RC_PORT,.service_type = PCIE_PORT_SERVICE_AER,}, {}

};

3.1.2 Initialize AER Service Driver

Once the AER service ID is defined, the ad-vanced error reporting service driver initial-izes the service callbacks as defined in thepcie_port_service_driver data struc-ture. The data structure of service callbacks isdefined below[7]:


static struct pcie_port_service_driver aerdrv={.name = "aer",.id_table = &aer_id[0],

.probe = aer_probe,

.remove = aer_remove,

.suspend = aer_suspend,

.resume = aer_resume,};

3.1.3 Calling pcie_port_service_register

The final step in initialization of the advancederror reporting service driver is calling func-tion pcie_port_service_register to reg-ister AER service with the PBD. During driverinitialization, the module routine is called forinitialization when the kernel calls the ad-vanced error reporting service driver. Call-ing pcie_port_service_register /pcie_

port_service_unregister should alwaysbe done inmodule_init /module_exit asdepicted below[7]:

static int __init aer_service_init(void){

return pcie_port_service_register(&aerdrv);}

static void __exit aer_service_exit(void){

pcie_port_service_unregister(&aerdrv);}

module_init(aer_service_init);module_exit(aer_service_exit);

Figure 7 depicts the state diagram once the ad-vanced error reporting service driver’s moduleroutine is called.

4 PCI Express Native Hot-PlugService Driver

The PCI Express Hot-Plug standard usagemodel is derived from the standard usage

Figure 7: State Diagram of Registering AERService with PBD

model defined in the PCI Standard Hot-PlugController and Subsystem Specification, Rev.1.0[8].

4.1 PCI Express Native Hot Plug Features

PCI Express Native Hot-Plug features are:

• Root ports and downstream ports ofswitches are hot-pluggable ports in PCIExpress hierarchy.

• Interrupt driven hot plug events will resultin hot-plug interrupts.

• Hot plug registers are part of the PCI Ex-press Capability register set; hot-plug op-erations are invoked by writing to theseregisters.

• Based on SHPC usage model, but not thebus centric SHPC register set.

4.2 Porting the PCI Express Hot-PlugDriver to a Service Driver

As mentioned in Section 2.2, the PCI ExpressPort Bus Driver provides a mechanism for a


service driver to register its service. If the re-quested service is found in a service devicehierarchy, the service driver can successfullyload. This section focuses on showing what thechanges are required to port the PCI Expressnative hot-plug driver to a service driver.

4.2.1 Registering the Hot Plug ServiceDriver

The pciehp driver calls pcie_port_

service_register (struct pcie_

port_service_driver *driver ) toregister its hot-plug service with the PBD.The pciehp driver is responsible for set-ting up the data structures before callingpcie_port_service_register . Belowshows the difference in the data structures usedwhen the driver is used as a standard driver oras a service driver[9].

+ static struct pcie_port_service_id+ port_pci_ids[] = {{+ .vendor = PCI_ANY_ID,+ .device = PCI_ANY_ID,+ .port_type = PCIE_ANY_PORT,+ .service_type = PCIE_PORT_SERVICE_HP,+ .driver_data = 0,+ }, { /* end: all zeroes */ }+ };

- static struct pci_device_id pcied_pci_tbl[]={- {- .class = ((PCI_CLASS_BRIDGE_PCI << 8) |- 0x00),- .class_mask = ~0,- .vendor = PCI_ANY_ID,- .device = PCI_ANY_ID,- .subvendor = PCI_ANY_ID,- .subdevice = PCI_ANY_ID,- }, { /* end: all zeroes */ }- };

4.2.2 Initialize the Hot-Plug Service Driver

Once the HP service ID is defined, the ser-vice driver initializes the service callbacks asdefined in thepcie_port_service_driver

data structure. The following shows the

changes that need to be made in porting the PCIExpress hot-plug driver to a service driver[9].

+ static struct pcie_port_service_driver+ hpdriver_portdrv = {+ .name = "hpdriver",+ .id_table = &port_pci_ids[0],+ .probe = pciehp_probe,+ .remove = pciehp_remove,+ .suspend = pciehp_suspend,+ .resume = pciehp_resume,+ };

- static struct pci_driver pcie_driver = {- .name = "pciehp",- .id_table = pcied_pci_tbl,- .probe = pcie_probe,- .remove = pcie_remove,- };

4.2.3 Calling pcie_port_service_register API

The final step in initialization of the HP ser-vice driver is callingpcie_port_service_

register to register HP service with the PBD.The following shows the changes that need tobe made in the standalone driver to port it to aservice driver[9].

static int __init pcied_init(void){

:+ retval = pcie_port_service_register(+ &hpdriver_portdrv);- retval = pci_register_driver(- &pcie_driver);

:}

static void __exit pcied_cleanup(void){

:+ pcie_port_service_unregister(+ &hpdriver_portdrv);- pci_unregister_driver(&pcie_driver);

:}

Figure 8 depicts the state diagram once HP ser-vice driver’s module routine is called.


Figure 8: State Diagram of Registering HP Ser-vice with PBD

4.2.4 Available Resources

As a service driver, dev->irq is provided by thePCI Express Port Bus Driver and is passed tothe pciehp driver. Whether dev->irq is a reg-ular system interrupt, MSI or MSI-X, the PCIExpress Port Bus Driver assigns the value toit. The pciehp service driver does not needto call pci_enable_msi to request use ofMSI/MSI-X if the OS supports that.

5 Impacts to PCI Express Drivers

The Port Bus Driver design does not directlyimpact existing PCI Express endpoint devicedrivers. However, a service driver may impacta PCI Express endpoint driver. Additional PCIExpress services may require endpoint driverchanges to take full advantage of the new func-tionality. For example, to take full advantage ofAER error recovery will require drivers to sup-port the AER callback API. Driver writers forPCI Express components should be well versedwith this architecture and evaluate driver im-pacts as new services (VC or PME) becomeavailable.

The Port Bus perspective impacts devicedrivers for PCI Express Switch components.

The PCI Express Port Bus Driver claims all PCIExpress Ports in a system device hierarchy, in-cluding ports in a PCI Express switch. Switchservice drivers must follow the port bus driverframework. Switch vendors can use existingroot service drivers as a reference while writ-ing their own service drivers.

When developing a switch service driver theusage model at each level in the PCI Expresshierarchy needs to be considered. A servicedriver for a downstream switch port may berequired to provide different functionality thana similar root port service driver. For exam-ple, the AER Root service driver cannot bereusedas-is . The usage model is differ-ent. AER Switch service driver should pro-vide error-handling callbacks and AER initial-ization of the switch, while the AER Root ser-vice driver provides the primary mechanism tohandle the error recovery process. However, inthe case of the hot-plug driver, the same servicedriver may be used for both the Root Ports andthe Switch Downstream Ports because the hot-plug usage model is identical.

6 Conclusion

The design of the PCI Express Port Bus Driverdelivers a clean and modular solution to sup-port the multiple features of PCI Express whileremaining compatible with the Linux DriverModel. Each feature can have its own soft-ware service driver that can be built and loadedas a separate module. In addition when/if fu-ture PCI Express features come available or areadded to future specification revisions, the PCIExpress Port Bus architecture is extensible tosupport those additions. The PCI Express PortBus Driver and changes to port the native PCIExpress hot-plug driver has been incorporatedLinux kernel version 2.6.11. The advanced er-ror reporting service driver is currently underreview on the LKML.


7 Acknowledgements

Special thanks to Greg Kroah-Hartman for hiscontributions to the architecture design of PCIExpress Port Bus driver.

References

[1] PCI Express Base Specification Revision1.1. March 28, 2005.http://www.pcisig.com/specifications/pciexpress/ .

[2] Linux Device Drivers, 3rd Edition.Publisher: O’Reilly & Associates; 3edition (February 10, 2005) by JonathanCorbet, Alessandro Rubini, GregKroah-Hartman.

[3] Renato John Recio. Promises andReality: Server I/O networks, past,present, and future. In Proceedings of theACM SIGCOMM Workshop onNetwork-I/O Convergence: Experience,Lessons, Implications. Pages 163-178,Karlsruhe, Germany, August 2003.

[4] PCIEBUS-HOWTO.txt. Available from:2.6.11/Documentation .

[5] PCI Express Port Bus Driver code.Available from:2.6.11/drivers/pci/pcie .

[6] PCIEAER-HOWTO.txt, under review. Ifbeing accepted:2.6.x/Documentation .

[7] PCI Express advanced error reportingdriver code, under review. If accepted:2.6.x/drivers/pci/pcie/aer.

[8] PCI Standard Hot-Plug Controller andSubsystem Specification Revision 1.0.June 20, 2001.

http://www.pcisig.com/

specifications/conventional/

pci_hot_plug/SHPC_10/ .

[9] PCI Express hot-plug driver code.Available from:2.6.11/drivers/pci/hotplug .

Proceedings of theLinux Symposium

Volume Two

July 20nd–23th, 2005Ottawa, Ontario

Canada

Conference Organizers

Andrew J. Hutton,Steamballoon, Inc.C. Craig Ross,Linux SymposiumStephanie Donovan,Linux Symposium

Review Committee

Gerrit Huizenga,IBMMatthew Wilcox,HPDirk Hohndel,IntelVal Henson,Sun MicrosystemsJamal Hadi Salimi,ZnyxMatt Domsch,DellAndrew Hutton,Steamballoon, Inc.

Proceedings Formatting Team

John W. Lockhart,Red Hat, Inc.

Authors retain copyright to all submitted papers, but have granted unlimited redistribution rightsto all as a condition of submission.

pci express port bus driver support for linux · pdf file2 • pci express port bus driver...

Documents