PCI Bus & Interrupts

PCI Bus

A bus is made up of both an electrical interface and a

programming interface

PCI (Peripheral Component Interconnect)

A set of specifications of how parts of a computer should

interconnect

A replacement for the ISA standard (bare metal kind of

bus)

Goals

Better performance

Platform independence

Simplify adding and removing peripherals to the system

PCI Bus

PCI Bus

Better performance

Higher clock rate (than ISA)

66 MHz/133 MHz

32-bit data bus

Platform independence

Supports auto detection of interface boards

Jumper less

Automatically configured at boot time

• Device driver then access the configuration information to complete

initialization

• Without the need to perform probing

PCI Addressing

Each PCI peripheral is identified by a 16-bit address <a

8-bit bus number, a 5-bit device number, and a 3-bit

function number>

Sometimes a 32-bit address (prefix with a 16-bit domain number)

Linux uses pci_dev to specify PCI devices to hide the

16-bit address

Workstations feature at least two PCI buses

A bridge is a PCI peripheral to join two buses

Overall layout of a PCI system is a tree

• Each bus is connected to an upper-layer bus, up to bus 0 at the root

of the tree

PCI Addressing

PCI Addressing

A PCI device can be addressed in three ways

Memory locations (shared by all)

• 32-bit or 64-bit

• Can be mapped at boot time to avoid collisions

I/O ports (shared by all)

• 32-bit

Configuration registers

• Uses geographical addressing

• Never collide

• A PCI driver can access its devices without probing

– Just read from the configuration space

» 256 bytes for each device function

» 4 bytes holds a unique function ID

Boot Time

At power on

A PCI device remains inactive

• Responds only to configuration transactions

• No memory and no I/O ports mapped

• Interrupt disabled

A PCI motherboard firmware (BIOS, NVRAM, PROM) performs

configuration transactions with each PCI device

• Allocates non-overlapping memory region

Configuration Registers and Initialization

Each PCI device features at least a 256-byte address

space

First 64 bytes standardized

PCI registers are always little-endian

• Need to watch out for byte ordering

• Use macros defined in <asm/byteorder.h>

Configuration Registers and Initialization

Configuration Registers and Initialization

vendorID (16-bit register)

Identifies a hardware manufacturer

• E.g., 0x8086 for Intel

• A global registry maintained by the PCI Special Interest Group

deviceID (16-bit register)

decided by the manufacturer

A device driver signature = <vendorID, deviceID>

class (16-bit value)

Top 8 bits identify the base class (group)

• E.g., network group contains Ethernet and token ring classes

Configuration Registers and Initialization

A PCI driver tells the kernel what kind of device it

supports via a data structure#include <linux/mod_devicetable>

struct pic_dev_id {

__u32 vendor, device;

__u32 subvendor, subdevice;

__u32 class, class_mask;

kernel_ulong_t driver_data;

};

Registering a PCI Driver

To register, create struct pci_driver (see

<linux/pci.h>)

Some important fields/* need to be unique */

/* normally the same as the module name of the driver

displayed in /sys/bus/pci/drivers/ */

const char *name;

/* pointer to the pci_device_id table declared earlier */

const struct pci_device_id *id_table;

Registering a PCI Driver

/* pointer to a probe function in the PCI driver */

/* if the PCI driver claims the PCI device, return 0 */

/* else return a negative error value */

int (*probe) (struct pci_dev *dev,

const struct pci_device_id *id);

/* called when the PCI device is removed from the system */

void (*remove) (struct pci_dev *dev);

/* called when the PCI device is suspended */

int (*suspend) (struct pci_dev *dev, u32 state);

/* called to resume from the suspended state */

int (*resume) (struct pci_dev *dev);

Registering a PCI Driver

Creating a struct pci_driver requires initializing

four fieldsstatic struct pci_driver pci_driver = {

.name = "pci_skel",

.id_table = ids,

.probe = probe,

.remove = remove,

};

Registering a PCI Driver

To register, call pci_register_driver

Returns 0 on success

Returns a negative error number on failure

static int __init pci_skel_init(void) {

return pci_register_driver(&pci_driver);

}

To unload a PCI driver, call pci_unregister_driver

Calls the remove function before it returns

static void __exit pci_skel_exit(void) {

return pci_unregister_driver(&pci_driver);

}

Enabling the PCI Device

In the probe function, the driver must call

pci_enable_deviceint pci_enable_device(struct pci_dev *dev);

Wakes up the device

Assigns its interrupt line and I/O regions

Accessing the I/O and Memory Spaces

A PCI device implements up to six I/O address regions

A region is a generic I/O address space that is either memory-

mapped or port-mapped

Size and the current location of I/O regions are reported

via 32-bit configuration registers

Symbolic names PCI_BASE_ADDRESS_0 to

PCI_BASE_ADDRESS_5

Accessing the I/O and Memory Spaces

I/O regions of PCI devices have been integrated into the

generic resource management

Can use the following functions

/* returns the first address (memory address/IO port)

associated with one of the six PCI IO regions */

/* set bar to 0 to 5 to select the region */

unsigned long pci_resource_start(struct pci_dev *dev,

int bar);

Accessing the I/O and Memory Spaces

/* returns the last usable address of the I/O region number

bar */

unsigned long pci_resource_end(struct pci_dev *dev,

int bar);

/* if associated I/O regions exist, return IORESOUCE_IO or

IORESOURCE_MEM in the flags */

/* returns IORESOURCE_PREFETCH in the flags to indicate

whether compiler optimizations need to be disabled */

/* returns IORESOURCE_READONLY in the flags to indicate

whether a memory region is write protected */

unsigned long pci_resource_flags(struct pci_dev *dev,

int bar);

Interrupt-Driven Digital Control

+Controller Plant

Sensor

ControlSignal

Command

Digital Computer

D/A

A/D

T

Sampler

Interrupt

Main ProgramInterruptServiceRoutine

History of IBM PC

-Wikipedia

8088(IBM PC/XT) Interrupt

8088 Interrupt

PC Interrupt

AD Converter Timer

IRQ

PCI busAD board

Timer

AD Converter

IRQ

AD conversion start

AD conversion end

Interrupt Request

T(Sampling Period)

Interrupt Request by A/D board

Interrupt Descriptor Table

IDT Gate Descriptors

Calling the IRQ Handler

External Interrupts

Events triggered by devices connected to the system

Network packet arrivals, disk operation completion, timer

updates, etc

Can be mapped to any IRQ vector above the exceptions

• (IRQs 32-255)

External because they happen outside the CPU

External logic signals CPU and notifies it which handler to

execute

Managed by Interrupt Controller

• Special device included in south bridge

Interrupt Controllers

Translate device IRQ signals to CPU IRQ vectors

Each device has only a single pin

• High = IRQ pending, Low = no IRQ

Interrupt controller maps devices to vectors

Two x86 controller classes

Legacy: 8259 PIC

• Connected to a set of default I/O ports on CPU

Modern: APIC + IOAPIC

• Memory mapped into each CPUs physical memory

– (How?)

• Next generation APIC (x2PIC) accessed via MSRs

– Model specific registers – control registers accessed via special

instructions

» WRMSR, RDMSR

8259 PIC

• Allows the mapping of 8 IRQ pins (from devices) to 8

separate vectors (to CPU)

• Assumes continuous numbering

• Assign the PIC a vector offset,

• Each pin index is added to that offset to calculate the vector

1 PIC only supports 8 device lines

Often more than 8 devices in a system

Solution: Add more PICs

• But x86 CPUs only have 1 INTR input pin

X86 Solution:

Chain the PICs together (master and slave)

• Slave PIC is attached to the 2nd IRQ pin of the master

• CPU interfaces directly with master

80286 IBM PC/AT

IRQ

IRQ INT Hardware Device

0 32 Timer

1 33 Keyboard

2 34 PIC Cascading

3 35 Second serial port

4 36 First serial port

6 38 Floppy Disk

8 40 System Clock

10 42 Network Interface

11 43 USB port, sound card

12 44 PS/2 Mouse

13 45 Math Coprocessor

14 46 EIDE first controller

15 47 EIDE second controller

APIC

Problem: PICs don’t support multiple CPUs

Only one INT signal, so only one CPU can receive interrupts

SMP required a new solution

APIC + IOAPIC

Idea: Separate the responsibility of the PIC into two components

• APIC = Interfaces with CPU

• IOAPIC = Interfaces with devices

PIC

Advanced PIC(APIC)

APIC

Each CPU has its own local APIC

In charge of keeping track of interrupts bound for its assigned

CPU

Since Pentium Pro, the APIC has been implemented in the CPU

die

APIC interfaces with CPUs interrupt pins to invoke

correct IDT vector

This is its primary responsibility

ICC Bus

• APICs and IOAPICs share a common communication

bus

• ICC bus: Interrupt Controller Communication Bus

• Handles routing of interrupts to the correct APIC

Interrupt Vectors

Vector Range Use

0-19 Nonmaskable interrupts and exceptions.

20-31 Intel-reserved

32-127 External interrupts (IRQs)

128 System Call exception

129-238 External interrupts (IRQs)

239 Local APIC timer interrupt

240 Local APIC thermal interrupt

241-250 Reserved by Linux for future use

251-253 Interprocessor interrupts

254 Local APIC error interrupt

255 Local APIC suprious interrupt

IRQ Handling

1. Monitor IRQ lines for raised signals.If multiple IRQs raised, select lowest # IRQ.

2. If raised signal detected1. Converts raised signal into vector (0-255).

2. Stores vector in I/O port, allowing CPU to read.

3. Sends raised signal to CPU INTR pin.

4. Waits for CPU to acknowledge interrupt.

5. Kernel runs do_IRQ().

6. Clears INTR line.

3. Goto step 1.

Slide #43

do_IRQ

1. Kernel jumps to entry point in entry.S.

2. Entry point saves registers, calls do_IRQ().

3. Finds IRQ number in saved %EAX register.

4. Looks up IRQ descriptor using IRQ #.

5. Acknowledges receipt of interrupt.

6. Disables interrupt delivery on line.

7. Calls handle_IRQ_event() to run handlers.

8. Cleans up and returns.

9. Jumps to ret_from_intr().

Slide #44

Interrupt Handlers

Function kernel runs in response to interrupt.

More than one handler can exist per IRQ.

Must run quickly.

Resume execution of interrupted code.

How to deal with high work interrupts?

Ex: network, hard disk

Slide #45

Registering a Handler

request_irq()

Register an interrupt handler on a given line.

free_irq()

Unregister a given interrupt handler.

Disable interrupt line if all handlers unregistered.

Slide #46

Registering a Handler

int request_irq(unsigned int irq,

irqreturn_t (*handler)(int, void *, stru

ct pt_regs *),

unsigned long irqflags,

const char * devname,

void *dev_id)

Slide #47

irqflaqs = SA_INTERRUPT

| SA_SAMPLE_RANDOM

| SA_SHIRQ

Writing an Interrupt Handler

Differentiating between devices

Pre-2.0: irq

Current: dev_id

Registers

Pointer to registers before interrupt occurred.

Return Values

IRQ_NONE: Interrupt not for handler.

IRQ_HANDLED: Interrupted handled.

Slide #48

irqreturn_t ih(int irq,void *devid,struct pt_regs *r)