thunderboltdevice driver programming guide by apple

8/18/2019 ThunderboltDevice Driver Programming Guide by Apple

1/29

Thunderbolt Device

Driver ProgrammingGuide


2/29

Contents

About the Thunderbolt Technology 4

At a Glance 4

Prerequisites 4

See Also 5

Reference Material 5

Thunderbolt Technology Overview 6

The Beginning: The Thunderbolt Controller Chip 6

Using Thunderbolt Communication over Ports 8

Working with Thunderbolt Technology 10

Discovering Thunderbolt Devices 10

Using PCI Resources Wisely 10

Tolerating PCI Latency 11

Using a Registered Vendor ID 11

Enabling PCI Device Drivers for Thunderbolt Operation 11

Verifying Thunderbolt Connections in PCI Devices 13

Handling and Routing Interrupts 14

Building Only MSI-Capable Thunderbolt Devices 14

Handling Hardware Exceptions for MSI Support 15

Enabling MSI 16

Using Hot Plug Operation with PCI Devices 17

Supporting PCIe Pause 19

Returning I/O Operations as Errors 23

Debugging Thunderbolt Drivers 24

Debugging VT-d I/O MMU Virtualization 24Enabling VT-d Panics 25

Disabling VT-d 26

Debugging PCIe Pause 26

Avoiding Memory Leaks 26

Document Revision History 28

2013-10-22 | Copyright © 2013 Apple Inc. All Rights Reserved.

2


3/29

Figures and Listings

Thunderbolt Technology Overview 6

Figure 1-1 Intel 82524EF controller chip (host mode) 6

Figure 1-2 Intel 82524EF controller chip (endpoint mode) 7

Figure 1-3 Expansion chassis utilizing PCI paths 8

Figure 1-4 PCI fanout (two disk controllers) 9

Figure 1-5 PCI direct routing (two disk controllers) 9

Working with Thunderbolt Technology 10

Listing 2-1 PCI Driver Key (Thunderbolt Operation) 12Listing 2-2 PCI Driver Personality (Thunderbolt Key Set) 12

Listing 2-3 Verifying Thunderbolt Connections 13

Listing 2-4 PCI Driver: External Storage Key Set 13

Handling and Routing Interrupts 14

Listing 3-1 Enabling MSIs 16

Listing 3-2 Single Bottleneck Routine (MMIO Read/Write) 18

Debugging Thunderbolt Drivers 24

Listing 4-1 Searching for Memory Leaks 27


3


4/29

TheThunderbolt interface is a newI/O technologythat supports high-resolution displays andhigh-performance

data devices through a single expansion interface.

At a GlanceThe Thunderbolt interface is made up of an Intel controller chip designed to tunnel DisplayPort (DP) version

1.1 and PCI Express (PCIe) information. Its dual-link cables are designed to carry 10 gigabits per second (Gbps)

of data bidirectionally on each link, for a total of 40 Gbps per cable. Thunderbolt ports may be hot-plugged,

daisy chained with up to six devices in depth, and can be connected host-to-host.

Thunderbolt ports use a Mini DisplayPort (mDP) connector and provide 10 W of power to downstream devices.

Thunderboltcablesmay be connectedby eitherend(both ends areexactly the same). Thunderbolt technology

is a host-centered and host-managed system (like USB), but also allows for host-to-host connections.

A Thunderbolt port or cable provides two 10 Gbps bidirectional links, but these two links cannot be bonded

into a single channel. Host software must assign specific paths (for example, DP, PCIe, native) to each link to

balance the load. This functionality is included in EFI and in OS X. For example, if there are two Thunderbolt

enabled displays attached, it can choose to route the DisplayPort (DP) over each link and use the remainingbandwidth for PCI. If there is only one Thunderbolt display, software can route the DisplayPort traffic over the

optimal link. The Thunderbolt interface has no requirement for fixed routing.

It is possible to boot directly from Thunderbolt devices. Apple provides a Unified Target Disk Mode (UTDM),

which supports FireWireandThunderbolt connections. Users may boot from UTDM over FireWireor Thunderbolt

connections as well.

PrerequisitesDrivers must be built as Universal and be able to support systems with greater than 2 GB of RAM. Refer to the

section entitled “Kernel Extensions and Drivers” in the 64-Bit Transition Guide for more details. In particular,

all drivers must use IODMACommand for scatter-gather list support.


4

About the Thunderbolt Technology

http://developer.apple.com/library/mac/#documentation/Darwin/Reference/KernelIOKitFramework/IODMACommand_h/Classes/IODMACommand/index.htmlhttp://developer.apple.com/library/mac/#documentation/Darwin/Reference/KernelIOKitFramework/IODMACommand_h/Classes/IODMACommand/index.html


5/29

See AlsoApple provides the following reference material. For more information please refer to these documents.

Reference Material ● I/O Kit Fundamentals

This document provides a broad, conceptual description of the I/O Kit and device-driver development on

OS X. This document is useful for both the developer who is creating a device driver that is resident in the

kernel and the developer who is using an I/O Kit device interface to communicate with the hardware.

● 64-Bit Transition Guide

This document assists you in the transition to Snow Leopard and Lion where the kernel is using a 64-bit

environment on some hardware.

About the Thunderbolt Technology

See Also


5


6/29

This chapter describes the basic components and functions of the Thunderbolt technology and how these are

used with DisplayPort (DP) and PCI Express (PCIe) devices.

The Beginning: The Thunderbolt Controller ChipThe Thunderbolt interface is a revolutionary I/O technology that supports high-resolution displays and

high-performance data devices through a single, compact port. It sets new standards for speed, flexibility, and

simplicity. At the core of this technology is the Intel Thunderbolt controller chip.

For example, the first generation of controller chip is the 82524EF controller as shown in Figure 1-1.

Figure 1-1 Intel 82524EF controller chip (host mode)

The Intel controller chip can be used in both host mode and endpoint mode. In host mode, the controller has

a Gen2 x4 uplink to the system PCI Express Root Complex and one or more DisplayPort (DP) inputs (depending

on the graphics capabilities of the system). Additionally, there is a PCI switch and a collection of DMA engines,

referred to as the Native Host Interface (NHI). The PCI switch enables PCI uplink for downstream devices and

the NHI is used for software protocols and device discovery. The controller chip includes four Thunderbolt

ports as outputs.


6

Thunderbolt Technology Overview


7/29

Note: Each open circle in the diagram indicates an adapter, that translates the native transport (NHI,

PCI, DP) for transmission over Thunderbolt ports and cables.

In endpoint mode, the controller chip provides a Gen2 x4 downlink (supports x1, x2, x4 at Gen2 speeds) or

Gen1 4x1 downlinks to support multiple devices. Figure 1-2 provides an example of the controller chip as an

endpoint with both a x4 Gen2 PCI and a DisplayPort (DP) output option.

Figure 1-2 Intel 82524EF controller chip (endpoint mode)


The Beginning: The Thunderbolt Controller Chip


7


8/29

Note: The PCI switch gets its PCI input from the upstream Thunderbolt connection through the

Thunderbolt switch. Similarly, the DisplayPort (DP) out must be connected through the Thunderbolt

topology to a DP in adapter.

Using Thunderbolt Communication over PortsCommunication over Thunderbolt ports is achieved by establishing paths between two adapters. A practical

example of a PCI expansion chassis as shown in Figure 1-3.

Figure 1-3 Expansion chassis utilizing PCI paths

In Figure 1-3, a PCI path has been established by routing the PCI upstream connection from the PCI switch in

the Machost through the Thunderbolt switch andacross the cable to theThunderbolt switchin thePCIchassis.

From the Thunderbolt switch inside the PCI chassis, the path is routed up to the PCI switch in the controller

chip. The PCI fanout from the switch provides 4x1 connectivity.


Using Thunderbolt Communication over Ports


8


9/29

PCI paths can utilize fanout or can connect directly to PCI adapters in the Mac host. The host software controls

how the paths are established. Wherever possible, OS X and EFI attempt to use the same algorithms for path

selection to provide a similar user experience. Devices can also provide hints to software, indicating how much

and the type of bandwidth the device consumes in order for the software to make a more optimal selection.

An example of fanout is shown in Figure 1-4.

Figure 1-4 PCI fanout (two disk controllers)

The disk labeled Disk #1 has a PCI switch inside the controller chip used for fanout. In Figure 1-5, the host

software can connect Disk #2 directly to the host, instead of using fanout.

Figure 1-5 PCI direct routing (two disk controllers)

Only four PCI adapters are available through the Mac host. The host determines the optimal routes for all of

the attached devices.


Using Thunderbolt Communication over Ports


9


10/29

This chapter provides guidelines for working with the Thunderbolt technology. Keep these guidelines in mind

when creating compatible devices.

Discovering Thunderbolt DevicesSystem software is used to discover Thunderbolt devices and to determine when a new Thunderbolt device

has been attached. The software discovers all DisplayPort and PCI devices, and creates Thunderbolt paths to

those devices using any available resources. Once the paths arecreated, the applicable OS driver(s) are notified

to the presence of the devices.

In the case of DisplayPort, graphics drivers may light up the screen. The screen is only lit up based on the

capabilities of the GPU.

In the case of PCI, the I/O PCI family (IOPCIFamily) within the I/O Kit is responsible for enumerating the PCI

bridge in the controller chip and any downstream bridges or devices.

Using PCI Resources WiselyPCIresources (bus numbers, memory apertures)arescarce andprogrammers anddevicemanufacturers should

do their best to present the true requirements of the device.

For example, consider a PCI device that has multiple functions, where only function 0 will be used in the

application. PCI functions other than function 0 should be hidden from the OS if possible, by returning

0xFFFFFFFF for a Configuration Read at offset 0 for that function. Another example is a device that requires

only 2 KB of aperture for Memory Mapped I/O (MMIO) space, but instead reports 64 KB of aperture during

enumeration. These devices should report accurate aperture values.

I/O space cannot be counted on when designing for the Thunderbolt interface. PCI limitations require an I/O

space allocation of 4 KB per PCI Bridge, and the total system-wide I/O space is only 64 KB. Therefore, use MMIO

instead of relying on I/O space allocations for devices. Using MMIO allows the operating system to load the

device driver, even if I/O space could not be properly allocated for the device.


10

Working with Thunderbolt Technology


11/29

Tolerating PCI LatencyDevices on a Thunderbolt bus exhibit higher latency than devices on internal slots—about 1.5 microseconds

of round-trip latency per hop. This means that putting a device at the end of a Thunderbolt chain can add up

to 9 microseconds of round-trip latency. Most devicesand drivers handle this additional communication latency

with no particulardifficulty. In some cases, however, your driver or devicefirmware must be tweaked to toleratethat latency.

For example:

● If your driver uses timers to detect device failures, you may need to lengthen the timeout value.

● If your driver needs to perform an operation the moment an interrupt occurs, you may haveto find another

way to schedule the operation, such as telling the device to delay the action until the next interrupt or

using a software-defined phase-locked loop.

● If your device expects the driver to send some data and then wait for the device to acknowledge it before

sending more data, then the device needs to be more tolerant of delays than it otherwise might need to

be, or else the driver needs to handle failure gracefully.

● If your driver tells an isochronous endpoint to stop sending data, it may continue to send data longer than

usual.

And so on. The details are highly dependent on the driver.

Note: Interrupt latency on Thunderbolt can be slightly more problematic if your device does not

useMSI-based interrupts. Because legacy PCI interrupts aresharedbus-wide, other devices can cause

serious performance problems. For this reason, Thunderbolt devices should always use MSI-based

interrupts.

Using a Registered Vendor IDEach Thunderbolt device must have a Device ROM (DROM) that contains information about the vendor and

the device, as well as a 64-bit UID, which uniquely identifies the device. Each vendor must have a registered

vendor ID with the Thunderbolt naming authority (Intel).

Enabling PCI Device Drivers for Thunderbolt OperationTo declare support for Thunderbolt technology, the following key must be added to each of the personalities

in your driver’s Info.plist file.


Tolerating PCI Latency


11


12/29

Listing 2-1 PCI Driver Key (Thunderbolt Operation)

IOPCITunnelCompatible

You must include the key in the IOKitPersonalities section of the Info.plist file or the IOPCIFamily will

not load the drivers for Thunderbolt connected PCI devices. This opt-in key protects consumers against older

drivers that have not yet been updated to support Thunderbolt technology. This method also makes it easy

for third party developers, who have properly updated their driver, to signify they have done so. (See example

driver Listing 2-2 (page 12).)

Note: A PCI device driver must not ship with this key set, unless it fully supports the requirements

set forth in this document. This opt-inkey is specific to a personality for the driver, but the personality

can contain additional information in the form of key/value pairs in order to provide additionalcontext to the device driver should the IOPCITunnelCompatible key not be enough.

An example driver that includes this key in one of its personalities is AppleAHCIPort.kext, which is shown

in the PCI Driver Personality (Thunderbolt Key Set).

Listing 2-2 PCI Driver Personality (Thunderbolt Key Set)

GenericAHCI

CFBundleIdentifier

com.apple.driver.AppleAHCIPort

Chipset Name

AHCI Standard Controller

IOClass

AppleAHCI

IOPCIClassMatch

0x01060100&0xffffff00

IOProbeScore

800

IOProviderClass

IOPCIDevice

Vendor Name

Unknown


Enabling PCI Device Drivers for Thunderbolt Operation


12


13/29

key>IOPCITunnelCompatible/key>

Verifying Thunderbolt Connections in PCI DevicesPCI device drivers can determine if a Thunderbolt device is connected by recursively searching over parents

in the I/O Registry for the key IOPCITunnelled. (See code example Listing 2-3 (page 13) .)

Listing 2-3 Verifying Thunderbolt Connections

#include < IOKit/pci/IOPCIDevice.h>

bool tb = false;

if ( getProperty ( kIOPCITunnelledKey, gIOServicePlane, kIORegistryIterateRecursively

| kIORegistryIterateParents ) )

tb = true;

Additionally, PCI device drivers that present storage to the operating system must ensure the “Physical

Interconnect Location” key is set to indicate that the device is externally connected. (See physical interconnect

location key example Listing 2-4 (page 13) .)

Listing 2-4 PCI Driver: External Storage Key Set

setProperty ( kIOPropertyPhysicalInterconnectLocationKey,

kIOPropertyExternalKey );

Note: This key must be set before any devices are instantiated (that is, early in the driver’s start()

routine or when it loads based on its IOKitPersonality in the Info.plist file).


Verifying Thunderbolt Connections in PCI Devices


13


14/29

This chapter provides guidelines to help you support Thunderbolt devices using Message Signaled Interrupts

(MSI) and to understand how to respond to hardware devices that do not support MSI. There is also a section

on hot plug operations with PCI Devices that describes how to change PCI drivers so that they are able to deal

with unplanned disconnections.

Most modern PCI devices support flexibility when dealing with interrupt routing. With the advent of PCI-X (PCI

eXtended) and PCIe (PCI Express), Message Signaled Interrupts were introduced as an in-band mechanism for

asserting interrupts.

Building Only MSI-Capable Thunderbolt DevicesApple strongly recommends building only MSI-capable Thunderbolt devices. Ensure your OS X drivers enable

MSI when supporting Thunderbolt devices. To ease development, the Mac Pro computers have PCIe slots that

support MSI.

Note: MSI are edge-triggered interrupts while legacy interrupts are typically level-triggered. The

hardware may behave differently depending on the interrupt type used. Ensure your software iswritten so that it can handle the different types of interrupts properly. Whenever possible, Apple

recommends using MSI.

OS X services all interrupts on CPU 0 and uses the secondary interrupt context threads to defer work to task-level

interrupts rather than primary-interrupt levels. This method ensures that multiple drivers can run in parallel

on the available CPUs, that the OS can schedule real-time threads with accuracy, and that the system remains

responsive to user interaction. The best practice methods, described in I/O Kit Fundamentals , encourage the

use of the IOWorkLoop abstraction model for device drivers in order to defer work to the IOWorkLoop thread.

Theremay be a small number of devices that require work to be done in the primary interrupt context, however,drivers should spend the least amount of time possible in the primary-interrupt context.


14

Handling and Routing Interrupts


15/29

Note: Since Message Signaled Interrupts are never shared, most device drivers can use the MSI

vector with an IOInterruptEventSource instanceandavoidspendingtime in theprimary interrupt

context.

Handling Hardware Exceptions for MSI SupportThe hardware should support MSI, but in the unlikely event the hardware device does not support MSI, the

following three scenarios are most likely to occur. These three scenarios usually occur in the driver’s filter

routine and it is important to understand the ramifications of the results returned from the filter routine:

1. The filter is called, but the device has not signaled an interrupt.

This may be due to another device sharing the same interrupt pin. The filter routine should return false.

2. The filter is called and the device has signaled an interrupt.

The driver cannot mask the interrupt source, nor can it handle the interrupt at this level for locking or

performance reasons. The filter routine should return true.

Note: The I/O Kit interprets this true as needing to disable the entire interrupt pin at the

interrupt controller in the computer. This disables the interrupts for all devices that share the

same interrupt pinand induces additional latency in servicing each device’s respective interrupts.

3. The filter is called and the device has signaled an interrupt.

a. The driver can handle the interrupt at this level and return the device to a non interrupting state. If

the interrupt action needs to be run, the filter routine should call signalInterrupt().

Note: The filter may be called again either before or during the execution of the interrupt

action. The filter routine should return false.

b. The driver cannot handle the interrupt at this level, but it can prevent the device from signaling

another interrupt (for example, by manipulating an internal mask).

The filter routine should prevent the device from signaling another interrupt and then call

signalInterrupt() to cause the interrupt action to be run. The interrupt action should handle the

interrupt and return the device to a state where it can again signal an interrupt. The filter routine

should return false.


Handling Hardware Exceptions for MSI Support


15


16/29

Note: The MMIO read cycles and PCI Configuration cycles of any type (read or write) are non-posted

transactions from the CPU, which turn into PCIe transactions and may take many thousands of clock

cycles to complete . These transactions should be minimized whenever possible.

Apple provides developers with tools that can track primary interrupt times and help developers to minimize

the time needed for primary interrupts.

Enabling MSI

To enable MSI, a device driver should do the following, assuming that the provider is an IOPCIDevice instance:

Note: The device driver should use source as the last argument when creating an

IOInterruptEventSource or IOFilterInterruptEventSource.

Listing 3-1 Enabling MSIs

int index = 0;

int source = 0;

for ( index = 0; ; index++ )

{

IOReturn result = kIOReturnSuccess;

int interruptType = 0;

result = provider->getInterruptType ( index, &interruptType );

if ( result != kIOReturnSuccess )

break;

if ( interruptType & kIOInterruptTypePCIMessaged )

{

source = index;

break;

}

}


Handling Hardware Exceptions for MSI Support


16


17/29

Using Hot Plug Operation with PCI DevicesPCI device drivers are typically developed with the expectation that the device will not be removed from the

PCI bus during its operation. However, Thunderbolt technology allows PCI data to be tunneled through a

Thunderbolt connection, and the Thunderbolt cables may be unplugged from the host or device at any time.

Therefore, the system must be able to cope with the removal of PCI devices by the user at any time.

Warning: To prevent data loss, storage devices should not be unplugged unless the user unmounts

the storage device first.

The user may freely unplug Thunderbolt devices at any time, with theexception of storage devices, andshould

be able to sleep and wake the system with devices attached without causing any problems. For all devices,

the user must not be able to hang the system (computer) by unplugging a device or cable.

The PCI device drivers used with Thunderbolt devices may need to be updated in order to handle surprise or

unplanned removal. In particular, MMIO cycles and PCI Configuration accesses require special attention. When

a PCI device that is connected to a Thunderbolt port is detached from the system, the PCIe Root Port must

time out any outstanding transactions sent to thedevice, terminate thetransaction as though an Unsupported

Request occurred on the bus, and return a value of 0xFFFFFFFF. The Root Port has a completion timeout

value that is many milliseconds long and varies depending on the system layout. Real-time scheduling,

particularly for audio and video threads, can be affected by transactions that must be timed out.

Note: The surprise removal of PCIdevices mentioned in the previous paragraphs is similar to removal

of ExpressCard devices in the previous generation of MacBook Pro laptop products.

As a basic guideline, developers should modify their drivers to handle a return value of 0xFFFFFFFF. If any

thread, callback, interrupt filter, or code path in a driver receives 0xFFFFFFFF indicating the device has been

unplugged, then all threads, callbacks, interrupt filters, interrupt handlers, and other code paths in that driver

must cease MMIO reads and writes immediately and prepare for termination.

Note: Use an instance variable to enforce modifying the drivers to return a value of 0xFFFFFFFF

and prevent the driver from issuing additional MMIO or PCI Configuration cycles, which may cause

additional transaction timeouts.

If 0xFFFFFFFF is a legal value for a particular register offset, one additional read of a different register, which

is known tonever return 0xFFFFFFFF is thepreferredmechanismfor determining if thedevice is still connected.

Finally, if I/O Kit has already performed termination and called the driver’s willTerminate() method, no

further accesses should be performed.


Using Hot Plug Operation with PCI Devices


17


18/29

Once it has been determined that a device is no longer connected, do not try to clean up or reset the hardware

as attempts to communicate with the hardware may lead to further delays.

Apple recommends auditing usage of MMIO writes when no further access should be performed. MMIO writes

are posted transactions and it is possible for device drivers to queue up multiple writes in a row.

Note: It is best to prevent large numbers (i.e. > 2) of writes without an intervening read access even

though writes are preferred over reads in PCI.

A typical way for a developer to solve this problem is to provide a single bottleneck routine for all MMIO reads

and have that routine check the status of the device before beginning the actual transaction. An example of

such a routine using little endian fields for its memory mapped apertures follows:

Listing 3-2 Single Bottleneck Routine (MMIO Read/Write)

class AppleSamplePCI

{

...

bool fDeviceRemoved;

volatile unit8_t * fBaseAddressRegister;

...

unit32_t ReadRegister ( uint32_t offset );

virtual bool willTerminate ( IOService * provider, IOOptionBits options);

};

unit32_t

AppleSamplePCI::ReadRegister ( unint32_t offset )

{

unint32_t result = 0xFFFFFFFF;

if ( !fDeviceRemoved )

{

result = OSReadLittleInt32 ( fBaseAddressRegister, offset );

if ( result == 0xFFFFFFFF )

fDeviceRemoved = true;




18


19/29

}

return result;

}

bool

AppleSamplePCI::willTerminate ( IOService * provider, IOOptionBits options )

{

fDeviceRemoved = true;

return super::willTerminate ( provider, options );

}

Similar routines can be written for PCI Configuration cycle transactions, which may receive a return value of

0xFFFFFFFF and MMIO reads of smaller sizes.

Note: Be sure to initialize the variables fBaseAddressRegister and fDeviceRemoved before

use.

Supporting PCIe Pause

Because Thunderbolt allows theaddition andremoval of arbitrary numbersof peripherals connected in arbitrary

topologies, the task of dividing up the PCI tree’s address space can be challenging. Sometimes, particularly

when large numbers of devices are attached, it is possible to exhaust portions of that address space. Whenthis happens, a new device cannot be enabled without moving existing devices.

To solve this problem, OS X v10.9 supports PCIe Pause—a special power management state in which all driver

and device operations are temporarily suspended. Whenever address space exhaustion occurs, OS X may ask

drivers to pause operations. After the drivers are paused, OS X changes the address space layout of the paused

devices to make room for new devices, and then tells the drivers to resume normal operation.

To support pause, you must add an additional Info.plist key and a new power management state

(kIOPCIDevicePausedState ) to your driver, as described in the following sections.

Declare Support for Pausing In Your Info.plist File

To declare support for pausing, add the following key to each of the personalities in your driver’s Info.plist

file:

IOPCIPauseCompatible




19


20/29

Important: If your driver does not explicitly declare support for pausing, your driver will never receive

pause requests. As a result, devices may fail to appearwhen theuser plugs in additional devices, particularly

on hardware with multiple Thunderbolt ports. For this reason, you are strongly encouraged to support this

functionality as soon as possible.

Add Support for the kIOPCIDevicePausedState Power State

When a driver for a IOPCIDevice provider registers for power management, it must provide a set of power

state definitions indicating which of its power states should be used for each power state of the IOPCIDevice

object itself. The inputPowerRequirement field of these power states is matched against masks of the PCI

power state’s outputPowerCharacter field.

If you do not add any power states to your driver, your driver is put into its off state during pause, because the

pause state does not include the kIOPMPowerOn flag. Any clients of the driver in the power management

hierarchy then must change their states to match.

Modified drivers can support pause explicitly by adding a power state with the kIOPMConfigRetained flag

set in the inputPowerRequirement field, causing that state to be selected by the power management system

when the device needs to enter a paused state. The outputPowerCharacter value of thedriver’s newpower

state then dictates what the driver’s power management clients see. Depending on this flag, clients of your

driver may or may not change states.

For example, your power state table might look like this:

static const IOPMPowerState powerStates[kYourDriverPowerStateCount] = {

// version, capabilityFlags, outputPowerCharacter,

// inputPowerRequirement, staticPower, unbudgetedPower

// powerToAttain timeToAttain settleUpTime

// timeToLower settleDownTime powerDomainBudget

// Device off; inputPowerRequirement is 0,

// which matches the flags for kIOPCIDeviceOffState.

{ kIOPMPowerStateVersion1, 0, 0,

0, 0, 0,

0, 0, 0,




20


21/29

0 , 0 , 0 } ,

// Sleep mode; inputPowerRequirement has kIOPMSoftSleep flag set,

// which matches the flags for kIOPCIDeviceDozeState.

{ kIOPMPowerStateVersion1, 0, kIOPMSoftSleep,

kIOPMSoftSleep, 0, 0,

0, 0, 0,

0 , 0 , 0 } ,

// Device paused; inputPowerRequirement has kIOPMConfigRetained flag set,

// which matches the flags for kIOPCIDevicePausedState.

{ kIOPMPowerStateVersion1, kIOPMConfigRetained, kIOPMConfigRetained,

kIOPMConfigRetained, 0, 0,

0, 0, 0,

0 , 0 , 0 } ,

// Device active; inputPowerRequirement has kIOPMPowerOn flag set,

// which matches the flags for kIOPCIDeviceOnState.

{ kIOPMPowerStateVersion1, kIOPMPowerOn | kIOPMUsable, kIOPMPowerOn,

kIOPMPowerOn, 0, 0,

0, 0, 0,

0 , 0 , 0 }

};

To avoid disrupting service, write your code in a way that makes entering and exiting the pause state as fast

as possible. In particular, you do not need to run all of the code that you would use for a

kIOPCIDeviceOffState/kIOPCIDeviceOnState transition, because the device remains powered on

through the state transition, making a full reinitialization unnecessary.

However, when OS X tells your driver to pause, your driver should still do many of the things that it would do

when the computer goes into safe sleep—that is, it should tell the device to stop issuing additional transactionsand then wait until all outstanding transactions have finished before telling OS X that the device’s power state

has changed.

While the driver is paused:

● The driver should not access the device using memory-mapped I/O or configuration space transactions

● The device should not generate any interrupts, whether MSI or pin-based interrupts




21


22/29

● The device should not generate any DMA requests

● The device should not be the target of any DMA requests

Note: This quiesced (inactive) behavior is similar to what your driver should do in all of the otherinactive power states (that is, all power states other than kIOPCIDeviceOnState), so the driver

state should be very similar to the off and doze states. However, because the system is not actually

being powered down or going to sleep, the device does not receive any PCI broadcast messages,

such as PME_turn_off.

When a driver is resumed after a pause, the driver should act as though the computer just woke from safe

sleep, but without performing anyunnecessaryhardware initialization (becausethedevice remainedpowered).

In particular, it must determine whether the device has changed addresses, and if it has, it must use the new

physical addresses for all future communication with the device. The following values may have changed: ● The device’s base address registers (BARs)

● The device’s bus number

● Registry properties reflecting these values: "ranges", "assigned-addresses", and "reg"

● The device’s MSI capability register blockvalues for address and value, but not the number of MSIs allocated

The following values will not change:

● The PCI power management configuration block registers of the device—that is, the device will not be

put into a device sleep state (runtime D3)

● The virtual addresses of the BARs

Any mappings previously created by thedriver with IOPCIDevice::mapDeviceMemoryWithRegister

or IOPCIDevice::mapDeviceMemoryWithIndex for memory-mapped I/O access to hardware are

automatically remapped with the same virtual address and the new physical address

● Any other configuration registers

● The set of available BAR resources

Any BAR resource present at pause is guaranteed to be reallocated—that is, a device will never lose

resources across reconfiguration.

● The registry hierarchy of the device

● The number or kind of interrupt assignments (shared versus MSI)




22


23/29

Most drivers have few or no dependencies on the items above being changed. If they do, while entering the

state kIOPCIDeviceOnState (from any other state), these dependencies should be updated to the current

configuration of the device.

Returning I/O Operations as ErrorsDrivers should ensure any I/Os that are in flight at the time of surprise removal are properly returned as errors

to the upper layers that issued the I/O requests. The exact manner in which this is done is I/O family specific.




23


24/29

This chapter contains debuggingtips that may be useful when tracking down problems in Thunderbolt drivers.

Debugging VT-d I/O MMU VirtualizationNewer Thunderbolt-capable Macs provide support for I/O MMU virtualization (VT-d). This technology allows

virtual machines to have direct access to hardware. As a consequence of this support, when your device

performs DMA operations, the I/O addresses it uses may be different from the physical addresses as seen by

the OS X kernel.

The complete spec can be found here: Intel® Virtualization Technology for Directed I/O Specification.

IvyBridgesystems running OSX v10.8.2 and later enable the Intel VT-d unit as a DMA remapper. This functionality

is supported by the current APIs in much the same way as the DART controller on PowerMac G5 computers.

(To learn how this works at a high level, read “Supporting DMA on 64-Bit System Architectures” in I/O Kit

Fundamentals ).

When VT-d is enabled, some of the changes you'll see as a results of this are:

● If you are writing DMA programs, although physical segments returned by IOMemoryDescriptor orIODMACommand are contiguous ranges of I/O addresses, the I/O addresses for the segments may be

different than the CPU physical addresses, and the ranges are no longer necessarily contiguous in physical

memory from perspective of the CPU.

As a result, segments will usually be the same size as the full IOMemoryDescriptor size (for a virtually

contiguous IOMemoryDescriptor object with a single source buffer).

Note: This assumes you are using the default behavior for IODMACommand (mappingOptions

= kMapped) or getPhysicalSegment (without the kIOMapperNone option set). Specifying

nondefault options produces different behavior.

● By default, requests for physically contiguous memory return memory that is contiguous only from the

perspective of I/O devices, and may not be physically contiguous in RAM.


24

Debugging Thunderbolt Drivers

https://www-ssl.intel.com/content/www/us/en/intelligent-systems/intel-technology/vt-directed-io-spec.htmlhttps://www-ssl.intel.com/content/www/us/en/intelligent-systems/intel-technology/vt-directed-io-spec.html


25/29

If the memory must be contiguous to the CPU as well, you must pass the

kIOMemoryHostPhysicallyContiguous option to IOBufferMemoryDescriptor when you request

the memory. As before, such requests can fail if no physically contiguous blocks are available.

● Transactions now fail if memory descriptors or DMA commands are not properly prepared for reading or

writing.The prepare method adds a mapping entry into the I/O address translation table. You must call prepare

(and specify the correct direction) before you tell the device to read data from or write data to that region

of memory.

You must also balance each prepare call with a matching call to the complete method. If you do not,

your driver will eventually fill up the mapping table, and you will get a kernel panic the next time a driver

tries to add a mapping table entry.

Important: The I/O direction matters. If you prepare a descriptor or DMA command for reading and

your device then tries to write to that memory, the transaction will fail.

● The prepare and complete methods have additional overhead, though this impact is partially

counterbalanced by shorter scatter-gather lists.

If you ask for unmapped physical addresses—by calling an IOMemoryDescriptor object’s

getPhysicalSegment method with the option kIOMapperNone or an IODMACommand object’s

initWithSpecification method with mappingOptions set to something other than kMapped—and try

to use them for DMA, your driver will break.

Enabling VT-d Panics

By default, VT-d faults are logged to /var/log/system.log. A VT-d fault looks like the following:

vtd[0] fault: device 13:0:0 reason 0x6 R:0x3dff000

In the example above, 13:0:0 is the bus, device, and function of the device that generated the fault. You can

determine if your device produced the fault by looking at the pci-debug field in the output of the ioreg or

in the IORegistryExplorer application and comparing the values.

For a list of reason codes, see the Intel® Virtualization Technology for Directed I/O Specification.

The R indicates that a read operation triggered the fault. (A W indicates a write.) The final value is the address

that the device was trying to read or write (expressed as an I/O-space address).

When debugging faults, it can be useful to configure your kernel to panic whenever a fault occurs. To do this,

add the following flag in your kernel boot args:


Debugging VT-d I/O MMU Virtualization


25

https://www-ssl.intel.com/content/www/us/en/intelligent-systems/intel-technology/vt-directed-io-spec.htmlhttps://www-ssl.intel.com/content/www/us/en/intelligent-systems/intel-technology/vt-directed-io-spec.html


26/29

pci=0x100

Disabling VT-d

When debugging PCIe device drivers, it is often useful to temporarily disable VT-d so that I/O addresses are

the same as the corresponding physical addresses. To disable VT-d, add the following to your kernel boot args:

dart=0x0

If the problem you are debugging goes away in this mode, it usually indicates one of the following mistakes:

● Some part of your code is incorrectly passing a physical address in RAM to your device for DMA purposes

instead of an I/O address.

● Your code failed to call prepare or called prepare incorrectly on an IOMemoryDescriptor object

before using it to perform DMA.

Debugging PCIe PauseBy default, the OS X kernel spreads outPCIaddress allocations. As a result, PCIe pauseevents occur infrequently,

which makes these events challenging to debug. You can make debugging easier by disabling this allocation

spreading behavior. With spreading disabled, nearly every hot plug event triggers a pause event.

To disable allocation spreading, add the following to your kernel boot-args string:

pci=0x200

Avoiding Memory LeaksDrivers should ensure release of any and all resources acquired over the lifecycle of the driver. In particular,

memory mapped I/O ranges, memory, and objects should all be freed and double-checked for any leaks. Use

of tools such as ioclasscount can help identify some of these types of leaks. A common form of leaks is

introduced by references (that is, retaincounts). Often, it is difficult to determinethesourceof these references.

One way to determine the source of these references is to override the taggedRetain() method and obtain

a backtrace of the caller, which you can then send to printf(), kprintf(), or IOTimeStampConstant().


Debugging PCIe Pause


26

http://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man8/ioclasscount.8.htmlhttp://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man8/ioclasscount.8.html


27/29

Listing 4-1 Searching for Memory Leaks

void

AppleSamplePCI::taggedRetain ( const void * tag ) const

{

void * bt[16] = { 0 };

OSBacktrace ( &bt[0], sizeof ( bt ) / sizeof ( bt[0] ) );

super::taggedRetain ( tag );

}

Using standard symbolication tools allowsyouto determine which functions or methods causedthereference(s)

to be taken. Furthermore, you can override taggedRelease() and match the retains and releases to find

calls to taggedRetain(), which have no corresponding call to taggedRelease().

Note: The const keyword at the end of the method signatures is required to properly override

these methods.


Avoiding Memory Leaks


27


28/29

This table describes the changes to Thunderbolt Device Driver Programming Guide .

NotesDate

Updated with information about PCIe pause and tips on debugging

memory leaks.

2013-10-22

First version of a document that explains the Thunderbolt technology for

developers.

2011-12-21


28

Document Revision History


29/29

Apple Inc.

Copyright © 2013 Apple Inc.

All rights reserved.

No part of this publication may be reproduced,

storedin a retrievalsystem, or transmitted, in any

form or by any means, mechanical, electronic,photocopying, recording, or otherwise, without

prior written permission of Apple Inc., with the

following exceptions: Any person is herebyauthorized to store documentation on a single

computer for personal use only and to print

copies of documentation for personal useprovided that the documentation contains

Apple’s copyright notice.

No licenses, express or implied, are granted with

respect to anyof thetechnology describedin this

document. Apple retains all intellectual propertyrights associated with the technology described

in this document. This document is intended to

assist application developers to develop

applications only for Apple-labeled computers.

Apple Inc.1 Infinite Loop

Cupertino, CA 95014

408-996-1010

Apple, the Apple logo, FireWire, Leopard, Mac,

Mac Pro, MacBook, OS X, and Snow Leopard are

trademarks of Apple Inc., registered in the U.S.and other countries.

Intel and Intel Core are registered trademarks of

Intel Corporation or its subsidiaries in the United

States and other countries.

Even though Apple has reviewed this document,APPLE MAKES NO WARRANTY OR REPRESENTATION,EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THISDOCUMENT, ITS QUALITY, ACCURACY,MERCHANTABILITY, OR FITNESS FOR A PARTICULARPURPOSE. AS A RESULT,THIS DOCUMENT IS PROVIDED“AS IS,” AND YOU, THE READER, ARE ASSUMING THEENTIRE RISK AS TO ITS QUALITY AND ACCURACY.

IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT,INDIRECT, SPECIAL,INCIDENTAL,OR CONSEQUENTIALDAMAGES RESULTING FROM ANY DEFECT ORINACCURACY IN THIS DOCUMENT, even if advised of the possibility of such damages.

THE WARRANTY AND REMEDIES SET FORTH ABOVEARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORALOR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer,agent, or employee is authorized to make anymodification, extension, or addition to this warranty.

Some states do not allow the exclusion or l imitationof implied warranties or liability for incidental orconsequential damages, so the above limitation orexclusion may not apply to you. This warranty givesyou specific legal rights, andyou mayalsohaveotherrights which vary from state to state.

thunderboltdevice driver programming guide by apple

Documents