writing device drivers under linux

Advanced Linux Programming M. Barbeau

Writing Device Drivers Under Linux Outline:

1. Introduction 2. User-mode drivers

o Access to I/O map devices o Access to memory map devices o Finding devices o Data link access with Linux PF_PACKET

3. Kernel-mode drivers o Block/char device o Polling/interrupt mode o Major/minor number o Registration of a device driver o Interrupt handling

1. Introduction What is a device driver? A software abstraction of a device (peripheral or chip)! Types of drivers: user-mode / kernel-mode. 2. User-mode drivers Direct access to devices within a user application The x86 architecture has two memory maps: one for memory and one for I/O. Access to I/O Map devices Opening the I/O map Access to the I/O map under Linux: root user, system calls ioperm(2) and iopl(2).

April 1, 2002 1


ioperm(): opens a block of consecutive addresses in the I/O map, parameters are a base address and a length, in bytes, works for ports located from 0x0000 to 0x03ff. iopl(): opens the entire I/O map (note: I/O mapped PCI devices are located above 0x03ff.) Reading/Writing the I/O map System calls: inb(), inw(), and inl(), for byte, word and long reading; outb(), outw(), and outl(), for byte, word, long writing. (see sys/io.h) Access to memory-map devices Abstraction is device /dev/mem and system call mmap(). Opening the memory map Open /dev/mem, a file descriptor is returned. Call mmap() with the file descriptor, offset and size (in the file). An address mapped to the physical address space is returned. Address can be cast to a data type and used like a normal pointer. Where to find the address of a device? From manufacturer's documentation! From directory /proc/bus! Example: more /proc/bus/pci/device 0070 10d90531 b 00002001 4c000000 00000000 00000000 00000000 00000000 00000000 00000100 00000100 00000000 00000000 00000000 00000000 00040000 tulip Vendor ID: 10d9 (managed by PCI SIG), Device ID: 0531 (vender's choice), also printed on the card. Configured Base Address Register zero (BAR0): 00002001

April 1, 2002 2


Address of BAR0 is 0x2000 least significant bit 1 (0) indicates I/O (memory) map. Pitfalls: Multiple access must be thread safe. Disastrous effects of writing wrong I/Os. Improper access may freeze the computer. Data link access with Linux PF_PACKET This section contains code extracted from a user level Wireless (Ethernet) LAN driver. Only the data link access is addressed. The complete example available at: http://www.scs.carleton.ca/~barbeau/Courses/SETP/index.html Data: // wireless interface configuration struct Ifconfig { int sockid; // socket descriptor int ifindex; // index of the interface WLANAddr hwaddr; // MAC address int mtu; // maximum transmission unit }; // label of device, e.g. "eth0" char device[MAX_NAME_LEN]; // interface configuration Ifconfig ifconfig; // mutex to prevent concurrent transmission pthread_mutex_t transmutex; // ID of the receive thread pthread_t threadID; // constructor WLANProtocol::WLANProtocol(char * device, Outcome & retval) { // undefined the socket descriptor ifconfig.sockid = -1; // copy the device name strcpy(this->device, device); // initialize the mutex pthread_mutex_init(&transmutex, NULL);

April 1, 2002 3


retval = init(); } // wrapper for an invocation of the Receive() operation static void * startReceive(void * arg) { // make the thread asynchronously cancellable pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL); ((WLANProtocol *)arg)->Receive(); pthread_exit(NULL); // this statement should never be executed } Outcome WLANProtocol::init() { // (1) create device level socket // - PF_PACKET : low level packet interface // - SOCK_RAW : raw packets including link level header // - ETH_P_ALL : all frames will be received if ((ifconfig.sockid = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL))) == -1) { aLog.log("WLANProtocol, cannot open socket: %s", strerror(errno)); return NOK; } // (2) fetch the interface index struct ifreq ifr; strcpy(ifr.ifr_name, device); if (ioctl(ifconfig.sockid, SIOGIFINDEX, &ifr) < 0) { aLog.log("WLANProtocol, failed to fetch ifindex: %s", strerror(errno)); return NOK; } ifconfig.ifindex = ifr.ifr_ifindex; aLog.log("WLANProtocol, ifindex: %d", ifconfig.ifindex); // (3) fetch the hardware address if (ioctl(ifconfig.sockid, SIOCGIFHWADDR, &ifr) == -1) { aLog.log("WLANProtocol, failed to fetch hardware address: %s", strerror(errno)); return NOK; } memcpy(&ifconfig.hwaddr.data, &ifr.ifr_hwaddr.sa_data, WLAN_ADDR_LEN); aLog.log("WLANProtocol, hwaddr: %s", ifconfig.hwaddr.wlan2asc()); // (4) fetch the MTU if (ioctl(ifconfig.sockid, SIOCGIFMTU, &ifr) == -1) { aLog.log("WLANProtocol, failed to the MTU: %s", strerror(errno)); return NOK; }

April 1, 2002 4


ifconfig.mtu = ifr.ifr_mtu; aLog.log("WLANProtocol, MTU: %d", ifconfig.mtu); // (5) add the promiscuous mode struct packet_mreq mr; memset(&mr,0,sizeof(mr)); mr.mr_ifindex = ifconfig.ifindex; mr.mr_type = PACKET_MR_PROMISC; if (setsockopt(ifconfig.sockid, SOL_PACKET, PACKET_ADD_MEMBERSHIP, (char *)&mr, sizeof(mr)) < 0) { aLog.log("WLANProtocol, failed to add the promiscuous mode: %d", strerror(errno)); return NOK; } // (6) bind the socket to the interface (device) struct sockaddr_ll sll; memset(&sll, 0, sizeof(sll)); sll.sll_family = AF_PACKET; sll.sll_ifindex = ifconfig.ifindex; sll.sll_protocol = htons(ETH_P_ALL); if (bind(ifconfig.sockid, (struct sockaddr*)&sll, sizeof(sll)) < 0) { aLog.log("WLANProtocol, failed to bind the socket: %d", strerror(errno)); return NOK; } // (7) create attribute of a received thread (detached) pthread_attr_t attr; if (pthread_attr_init(&attr) != 0) { aLog.log("WLANProtocol, failed to create thread attribute!"); return NOK; } if (pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED)!=0) { aLog.log("WLANProtocol, failed to set thread attribute!"); return NOK; } // (8) create a receive thread int error = pthread_create(&threadID, &attr, startReceive, (void *) this); if (error) { aLog.log("WLANProtocol, failed to create thread %s!", strerror(error)); return NOK; } return OK; } // receive data over a socket void WLANProtocol::Receive()

April 1, 2002 5


{ Message * msg; // received message Outcome result; unsigned char * buff; // pointer to received data unsigned int i; // frame length struct sockaddr_ll from; // source address of the message socklen_t fromlen = sizeof(struct sockaddr_ll); int error; // (1) initialisation #define THREADPERMSG #ifdef THREADPERMSG // (1.1) thread per message aLog.log("WLANProtocol, thread per message model"); // actualize the DemuxArg class typedef DemuxArg<ProtocolUI, Message> DemuxArgClass; // create the attribute of a msg thread (detached) pthread_attr_t attr; if (pthread_attr_init(&attr) != 0) { aLog.log("WLANProtocol, failed to create thread attribute!"); assert(NULL); // fail } if (pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED)!=0) { aLog.log("WLANProtocol, failed to set thread attribute!"); assert(NULL); // fail } #else // (1.2) single thread model aLog.log("WLANProtocol, single thread model"); // only one message created and reused msg = new(nothrow) Message(ifconfig.mtu, result); if (msg == NULL || result == NOK) { aLog.log("WLANProtocol, failed to create msg!"); assert(NULL); // fail } #endif // (2) infinite loop while (true) { #ifdef THREADPERMSG // (3.1) thread per message, one mssg is created per rcvd frame msg = new(nothrow) Message(ifconfig.mtu, result); if (msg == NULL || result == NOK) { aLog.log("WLANProtocol, failed to create msg!"); // sleep for 10 milliseconds before re-trying usleep(10000); continue; } #else

April 1, 2002 6


// (3.2) clear the contents of the message msg->Reset(); #endif // (4) get a pointer to the message buffer buff = msg->Push(ifconfig.mtu); assert(buff); // (5) loop until a non-empty frame has been received on "device" while (true) { // (6) wait and receive a frame fromlen = sizeof(from); i = recvfrom(ifconfig.sockid, buff, ifconfig.mtu, 0, (struct sockaddr *) &from, &fromlen); if (i == -1) { aLog.log("WLANProtocol, cannot receive data: %s", strerror(errno)); // sleep for 10 milliseconds before re-trying usleep(10000); } break; // exit the loop } aLog.log("WLANProtocol, frame received"); // (7) a non-empty frame received, truncate non-used tail msg->Truncate(ifconfig.mtu-i); // (8) pass the message to the high-level protocol #ifdef THREADPERMSG // (8.1) thread per message, a thread for each mssg is created // build the arguments of a thread DemuxArgClass *aDemuxArg = new(nothrow) DemuxArgClass(this, msg); if (aDemuxArg == NULL) { aLog.log("WLANProtocol, failed to create argument"); delete msg; } // create a thread // the thread is reponsible for deleting "aDemuxArg" and "msg" pthread_t msg_tid; // identifier of the thread error = pthread_create( &msg_tid, &attr, demux<DemuxArgClass, ProtocolUI>, (void *)aDemuxArg); if (error) { aLog.log("WLANProtocol, failed to create thread: %s", error); delete msg; delete aDemuxArg; } #else // (8.2) single thread model xDemux(NULL, msg); #endif

April 1, 2002 7


} } // destructor WLANProtocol::~WLANProtocol() { aLog.log("WLANProtocol, shutdown"); // cancel the receive thread int error = pthread_cancel(threadID); if (error != 0) { aLog.log("WLANProtocol, failed to cancel thread"); } // close the socket if (ifconfig.sockid != -1) close(ifconfig.sockid); }

Sending a frame: // sends a frame Outcome WLANSession::xPush(Message *msg) { assert(msg); if (ifconfig->sockid == -1) { aLog.log("WLANSession::xPush(), sockid is NULL"); return NOK; } // (1) prepend the header onto the message unsigned char *buff = msg->Push(WLAN_HEADER_LEN); memmove(buff, &hdr, WLAN_HEADER_LEN); // (2) set the from address struct sockaddr_ll from; int fromlen = sizeof(from); from.sll_family = AF_PACKET; from.sll_ifindex = ifconfig->ifindex; // (3) lock the mutex on transmission pthread_mutex_lock(transmutex); // (4) send a frame int sentlen = sendto( ifconfig->sockid, buff, msg->Size(), 0, (sockaddr *) &from, fromlen);

April 1, 2002 8


// (5) lock the mutex on transmission pthread_mutex_unlock(transmutex); if (sentlen == -1 ) { aLog.log("WLANSession::xPush(), sendto failed"); return NOK; } return OK; } 3. Kernel-mode drivers

Block device

A block device is something that can host a file system such as a disk. A block device can only be accessed as multiples of a block, where a block is usually one kilobyte of data.

Buffering is used.

Character device

A character device is one that can be accessed like a file, and a char driver is in charge of implementing this behavior. This driver implements the open, close, read and write system calls. The console and parallel ports are examples of char devices.

No buffering.

April 1, 2002 9


Polling mode I/O

A process performs a system call (enters kernel-mode), device driver starts the device, periodic inspection of the device by the device driver, then the system call returns.

Polling mode I/O

Advantages: No other processes are running while a critical operation is performed, no interrupt management overhead, and no interrupt support required from the hardware. Interrupt mode A device driver, an IRQ, interrupt service routine (ISR), and a device bottom half are involved. A process performs a system call and blocks in state TASK_INTERRUPTIBLE. The device driver starts the device and performs the I/O, an IRQ is issued, ISR is executed, bottom half is marked for execution or a function is put in the task queue. The ret_from_sys_call code block is executed (runs after any system call), the process state is switched to TASK_RUNNING, and scheduler is invoked. Role of the ISR: completion of the I/O (i.e. check the device status, data transfer from device to primary memory) and marks bottom half for later execution (with the mark_bh() kernel function).

April 1, 2002 10


Role of the bottom half: postponed interrupt handling processing (initialized with the init_bh() kernel function).

Interrupt mode

Advantage: While an I/O is being performed, the CPU can be allocated to other processes. Major/minor number Major number: standard identity of a device driver (e.g. 2 for floppy, 3 for IDE hard disk, 6 for a parallel interface, see file /usr/include/linux/major.h). Minor number: identity of physical device, a driver can drive several physical devices (e.g. 0 for 1st disk, 1 for 2nd disk). Registration of device driver

Device drivers in Linux are known as modules and can be loaded dynamically into the kernel using the insmod command.

A single module can be compiled and linked to the kernel.

When a module is loaded or when a machine is booted, a device driver is registered with the OS (using a kernel function). A major number, a device name, and a list of functions supported by the driver (an instance of struct file_operations). Examples of supported functions: open(), read(), write(), and ioctl().

Names and operations are stored in kernel tables (chrdevs[]/blkdevs[] for character/block driver).

April 1, 2002 11


Interrupt handling The request_irq() kernel function associates an ISR with an IRQ. Fails if an ISR is already associated to the IRQ. Demultiplexing of IRQs: several devices on the same IRQ, a master ISR de-multiplexes the IRQ to several ISRs (hardware is polled to determine the cause of the IRQ and the ISR is selected). Organization of a device driver APIs to character device drivers and block device drivers are slightly different. Each device is represented by a special file in /dev. Created by the kernel a boot time or with: mknod /dev/<dev_name> <type> <major_number> <minor_number>

<type> is c for character device and b for block device. Interface to a device driver (file /usr/include/linux/fs.h): struct file_operations { struct module *owner; loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char *, size_t, loff_t *); ssize_t (*write) (struct file *, const char *, size_t, loff_t *); int (*readdir) (struct file *, void *, filldir_t); unsigned int (*poll) (struct file *, struct poll_table_struct *); int (*ioctl) (struct inode *, struct file *, unsigned int, unsigned long); int (*mmap) (struct file *, struct vm_area_struct *); int (*open) (struct inode *, struct file *); int (*flush) (struct file *); int (*release) (struct inode *, struct file *); int (*fsync) (struct file *, struct dentry *, int datasync); int (*fasync) (int, struct file *, int); int (*lock) (struct file *, int, struct file_lock *); ssize_t (*readv) (struct file *, const struct iovec *, unsigned long, loff_t *); ssize_t (*writev) (struct file *, const struct iovec *, unsigned long, loff_t *); ssize_t (*writepage) (struct file *, struct page *, int, size_t, loff_t*, int); };

April 1, 2002 12


Look the same as an interface to the file system! A device driver may support only a subset of the operations. Example: Disk driver (pseudo code, full code in drivers/block/hd.c) static struct file_operations hd_fops = { NULL, /* lseek - default */ block_read, /* read - general block-dev read */ block_write, /* write - general block-dev write */ NULL, /* readdir */ NULL, /* select */ hd_ioctl, /* ioctl */ NULL, /* mmap */ hd_open, /* open */ hd_release, /* release */ block_fsync /* fsync */ } Initialization function: called in the kernel initialization code (e.g. tty_init()) or when the module is loaded (i.e. init_module()). Kernel initialization code: Function chr_dev_init() in file drivers/char/mem.c, for character devices, or function blk_dev_init() in file drivers/block/ll_rw_bl.c, for block devices. Example: Disk driver (pseudo code) int hd_init(void) { /* registration of the interface with the kernel */ if (register_blkdev(MAJOR_NR, "hd", &hd_fops) { printk("hd: unable to get major %d for harddisk\n", MAJOR_NR); return -1; } blk_dev[MAJOR_NR].request_fn = DEVICE_REQUEST; read_ahead[MAJOR_NR] = 8; /*8 sectors (4kB) read-ahead*/ hd_gendisk.next = gendisk_head; gendisk_head = &hd_gendisk; timer_table[HD_TIMER].fn = hd_times_out; return 0; }

April 1, 2002 13


April 1, 2002 14

Implementation of operations: static int hd_ioctl(struct inode * inode, struct file *file, unsigned int cmd, unsigned long arg) { ... } The hd_open() function: called when a user-space process calls open() for device /dev/hdi. static int hd_open(struct inode * inode, struct file *filp) { int target; /* get minor number of device */ target = DEVICE_NR(inode->i_rev); /* check if number is acceptable */ if (target >= NR_HD) { return -ENODEV; } /* sleep while the target is busy */ while (busy[target]) { sleep_on(&busy_wait); } /* increase the reference count */ access_count[target]++; return 0; } static void hd_release(struct inode, struct file * file) { ... } Further Reading A good overview of user-mode device drivers can be found in:

• B. Nakatani, User-Mode Device Drivers, Embedded Linux Journal, March/April 2002, pp. 47-48.

A good overview of kernel-mode device drivers can be found in: • G. Nutt, Kernel Projects for Linux, Addison Wesley, 2001.

A detailed treatment of drivers for Linux can be found in: • A. Rubini, Linux Device Drivers, O'Reilly, 2001.

writing device drivers under linux

Documents