Preemptibility-Aware Responsive Multi-Core Scheduling

Jupyung Lee, Geunsik Lim and Sang-bum SuhSoftware Lab, Future IT Research Center

Samsung Advanced Institute of Technology, Yongin, Korea{jupyung.lee, geunsik.lim, sbuk.suh}@samsung.com

ABSTRACTWe propose a novel responsive scheduling technique to min-imize the scheduling latency of a real-time process in themulti-core architecture, called the preemptibility-aware schedul-ing (PAS). Modern complex operating systems contain nu-merous long interrupt-disabled and non-preemptible sections,and consequently these sections obstruct the immediate han-dling of urgent interrupts and the rapid scheduling of interrupt-driven real-time tasks, causing significant latency betweeninterrupt arrival and process scheduling. The proposed PASguarantees that before an urgent interrupt occurs, at leastone among multiple CPU cores is always in both interrupt-enabled and preemptible sections, so that the incoming ur-gent interrupt can always be handled with no significantdelay by such CPU core. Experimental results show thatthe worst-case latency can be reduced by 27-96%.

1. INTRODUCTIONIn this paper, we propose a novel technique to minimize

the scheduling latency of a real-time process in the multi-core architecture. Among various factors affecting the la-tency in the GPOS, interrupt-disabled and non-preemptiblesections are the most dominant factors because the interrupt-disabled sections impede the immediate handling of an ur-gent interrupt, and the non-preemptible sections delay thescheduling of a real-time process.

The fundamental problem basically originates from thefact that disabling the interrupt and/or kernel preemptionis allowed everywhere in the operating system as long as thecurrent process is in the kernel mode. Although the kerneldevelopers usually take the utmost care to make both theinterrupt-disabled and non-preemptible sections as short aspossible, that is, to re-enable the interrupt and/or kernelpreemption as quickly as possible, it comes with no guaran-tee that the worst-case execution time of these ‘harmful’ sec-tions will not exceed a certain threshold value. This meansthat a poorly designed device driver that makes one non-preemptible section extremely long can significantly degrade

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SAC’11 March 21-25, 2011, TaiChung, Taiwan.Copyright 2011 ACM 978-1-4503-0113-8/11/03 ...$10.00.

the real-time performance of the overall system.In this study, we attempt to resolve this problem by tak-

ing advantage of the characteristics of the multi-core archi-tecture. In the multi-core architecture, each CPU core re-peats the operation of entering/leaving interrupt-disabledand non-preemptible sections independently to each other.Thus, although it is impossible for one fixed CPU core tostay in the interrupt-enabled and preemptible section all thetime before an urgent interrupt occurs, it is possible that atleast one CPU core among them is always allowed to be inthe interrupt-enabled and preemptible section by managingthe entering/leaving operation in each CPU core in a coor-dinated way. Afterwards, if an urgent interrupt occurs, itis sent to one of the CPU cores that are in the interrupt-enabled and preemptible section, so that it can always behandled with no significant delay by the CPU core. Puttingthe interrupt-disabled and non-preemptible section managerand urgent interrupt dispatcher together, this technique isnamed the preemptibility-aware scheduling (PAS) becauseit considers the preemptibility of each CPU core in deter-mining whether the entrance to an interrupt-disabled or anon-preemptible section will be allowed or not, and to whichCPU core the incoming urgent interrupt will be delivered.

2. PROBLEM DEFINITIONThe interrupt-disabled section is a kernel-level code section

in which it is temporarily disabled to handle the incoming in-terrupt, while the interrupt-enabled section has the oppositemeaning. Each will be called DI section and EN section,respectively. The non-preemptible section is a kernel-levelcode section in which kernel preemption is temporarily dis-abled in the preemptive kernel, while the preemptible sectionhas the opposite meaning. Each will be called NP sectionand P section, respectively. The OS latency is definedas the time elapsed between the arrival of an interrupt andthe reactivation of a process that was waiting for the inter-rupt[2]. An urgent interrupt is defined as an interruptthat will wake up a real-time process.

If an urgent interrupt occurs in the middle of either DI orNP section, the OS latency with respect to the interrupt willbe increased; if an urgent interrupt occurs in DI section, itcannot be handled until interrupt is re-enabled, causing thedelay between the urgent interrupt and interrupt handler.If an urgent interrupt occurs in NP section, the real-timeprocess awakened by the interrupt cannot preempt the cur-rently running process until kernel preemption is re-enabled,causing the delay between the interrupt handler and processswitch. Both cases will increase the OS latency accordingly.

748

CPU1CPU1 CPU2CPU2 CPU3CPU3 CPU4CPU4PP NPNP NPNPNPNP

Interrupt DispatcherInterrupt DispatcherUrgent InterruptENEN DIDI DIDIDIDI Preemptible Core

DI/NP DI/NP Section Section ManagerManagerFigure 1: The basic concept of the preemptibility-aware scheduling (PAS)

If an urgent interrupt occurs in both EN and P sections, itcan be handled rapidly without causing these latencies. ACPU core that is running in both EN and P sections will becalled the preemptible core.

To simplify the problem, let us assume in this paper thatthere is only one high-priority real-time process that needsto be scheduled immediately after an urgent interrupt oc-curs. Then the problem to be solved in this paper is howto guarantee that each urgent interrupt occurs in the pre-emptible core among multiple CPU cores.

3. PREEMPTIBILITY-AWARE SCHEDULINGThe basic concept of the preemptibility-aware scheduling

(PAS) is illustrated in Fig. 1. The PAS includes the DI/NPsection manager and the urgent interrupt dispatcher. TheDI/NP section manager monitors the operation of enteringand leaving DI and NP sections and if necessary, it blocksthe entrance to the section in order to guarantee that thereis always at least one preemptible core among the multipleCPU cores before an urgent interrupt occurs. Afterwards,if an urgent interrupt actually occurs, the urgent interruptdispatcher sends it to one of the preemptible cores. By doingso, the increase of the OS latency caused by DI or NP sec-tion can be avoided and the problem defined in the previoussection can be resolved.

Urgent Interrupt Dispatcher: Since the preemptible coreset is dynamically changed while waiting for the urgent in-terrupt, the urgent interrupt dispatcher should be able todispatch the interrupt to the appropriate CPU core. Thereare two ways of doing this: (i) modifying the interrupt dis-patch table dynamically, or (ii) using interrupt broadcasting.

DI/NP Section Manager: For the operation, it is requiredto identify each entrance to DI and NP section in the OS.In the case of Linux, for example, Linux 2.6 is known tohave approximately 11,000 NP sections, and thus findingeach entrance to those sections seems to be difficult. For-tunately, since each DI and NP section is clearly identifiedby its entry and exit calls, for example, preempt disable()and preempt enable(), the DI/NP section manager can beimplemented with only a minimal modification of the Linuxcode, simply by extending its entry and exit calls.

If the DI/NP section manager intervenes in the entranceto each and every DI and NP section, the performance degra-dation might be significant depending on the frequency of

05001000150020002500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Number of Trialmaximum latency (usec) dw/o PAS w/ PAS

Figure 2: OS latency under logout stress (pdf)

access to DI and NP sections. In order to alleviate the degra-dation, it is more appropriate to apply the DI/NP sectionmanager only to a set of DI and NP sections whose worst-case execution time exceeds a certain threshold value. Forexample, if a real-time process can endure 10 usec of addi-tional latency, then it is safe to access DI and NP sectionswhose worst-case execution time is less than 10 usec withoutthe intervention of the manager. According to an experimen-tal result in [1], among non-preemptible sections that variousapplications access, only 10% are longer than 10 usec. Thus,it is preferable to find a set of long DI and NP sections andapply the proposed technique only to them. As mentionedabove, since each DI and NP section is clearly identified byits entry and exit calls, the execution time of each sectioncan be easily measured. In practice, Linux 2.6.29 includes ameasurement module that allows us to find the longest NPand DI section during a certain period[1].

4. EXPERIMENTAL RESULTSThe prototype of PAS was implemented based on the

Linux 2.6.29 kernel source code. We first added Ingo Mol-nar’s real-time patch, which is regarded as the state-of-the-art implementation in terms of improving the real-time per-formance of the Linux kernel. We next extended the en-try/exit calls to DI and NP sections and added a few func-tions to implement the proposed technique.

The OS latency is measured with and without PAS underGUI logout stress, which is known to significantly increasethe OS latency. The experimental result is shown in the Fig.2. The GUI logout is repeated 20 times and at each trial, theOS latency is measured from remote shell. When the PASis not applied, the maximum OS latency stands at 63-2177usec. Conversely, when the PAS is applied, the maximumOS latency stands at 55-83 usec (96% reduction).

5. REFERENCES[1] J. Lee and K. Park. Prediction-based micro-scheduler:

Toward responsive scheduling of general-purposeoperating system. IEEE Trans. Computers,58(5):648–661, May 2009.

[2] J. Lee and K. Park. Interrupt handler migration anddirect interrupt scheduling for rapid scheduling ofinterrupt-driven tasks. ACM Trans. EmbeddedComputing Systems, 9(4), March 2010.

749