Oregon Health & Science UniversityOHSU Digital Commons

CSETech

January 2002

Supporting time-sensitive applications on general-purpose operating systemsAshvin Goel

Luca Abeni

Jim Snow

Charles Krasic

Jonathan Walpole

Follow this and additional works at: http://digitalcommons.ohsu.edu/csetech

This Article is brought to you for free and open access by OHSU Digital Commons. It has been accepted for inclusion in CSETech by an authorizedadministrator of OHSU Digital Commons. For more information, please contact champieu@ohsu.edu.

Recommended CitationGoel, Ashvin; Abeni, Luca; Snow, Jim; Krasic, Charles; and Walpole, Jonathan, "Supporting time-sensitive applications on general-purpose operating systems" (2002). CSETech. 53.http://digitalcommons.ohsu.edu/csetech/53

Supporting Time-Sensitive Applications on General-Purpose OperatingSystems

Ashvin Goel, Luca Abeni, Jim Snow, Charles Krasic, Jonathan WalpoleDepartment of Computer Science and Engineering

Oregon Graduate Institute, Portland�ashvin,luca,jsnow,krasic,walpole � @cse.ogi.edu

Abstract

General-purpose operating systems are increasingly be-ing used for serving time-sensitive applications. Sup-porting these applications requires low-latency responsefrom the kernel and from other system-level services.This paper explores various operating systems tech-niques needed to support time-sensitive applications anddescribes the design of a time-sensitive, general-purposeLinux system. We show that a high-precision timing fa-cility together with a well-designed preemptible kernelcan be the basis for a low-latency response system andsuch a system can have low overhead. We evaluate thebehavior of realistic time-sensitive user- and kernel-levelapplications on our system and show that, in practice, it ispossible to properly support time-sensitive applicationsin a general-purpose operating system without compro-mising the performance of throughput-oriented applica-tions.

1 Introduction

Multimedia applications, and soft real-time applicationsin general, are driven by real-world demands and arecharacterized by timing constraints that must be satisfiedfor correct operation; for this reason, we call these appli-cations time-sensitive. Time-sensitive applications mayrequire, for example, periodic execution with low jitter,or response in a short period of time to external eventssuch as the arrival of network packets.

To support time-sensitive applications, a general-purpose operating system must respect the application’stiming constraints. For these constraints to be satisfied,

This work was partially supported by DARPA/ITO under the In-formation Technology Expeditions, Ubiquitous Computing, Quorum,and PCES programs and by Intel.

resources must be allocated to the application at the ap-propriate times, hence resource allocation must be accu-rate.

As we will show in Section 2, there are some impor-tant requirements for achieving a correct resource allo-cation; each of these requirements have been addressedin the past with specific mechanisms, but unfortunatelygeneral-purpose operating systems, such as Linux, oftendo not support or integrate these mechanisms.

This paper focuses on three specific techniques thatcan be integrated to satisfy the constraints of time-sensitive applications. First, we present firm timers, anefficient high-resolution timer mechanism.

Firm timers incorporate the benefits of periodic timers,one-shot timers available on modern hardware [8] andsoft timers [5] and provide accurate timing with low over-head. Second, we use a preemptible kernel design fora responsive kernel. Finally, we use both priority andreservation-based CPU scheduling mechanisms for sup-porting various types of time-sensitive applications. Wehave integrated these techniques in our extended versionof the Linux kernel, which we call Time-Sensitive Linux(TSL).

Currently, general-purpose systems provide coarse-grained resource allocation with the goal of maximiz-ing system throughput. Such a policy conflicts with theneeds of time-sensitive applications which require moreprecise allocation. Thus, recently several approacheshave been proposed to improve the timing response ofa general-purpose system such as Linux [17, 24]. Theseapproaches include improved kernel preemptibility and amore generic scheduling framework. Since their focus ishard real-time, they do not evaluate the performance ofnon-real time applications.

In contrast TSL focuses on integrating an efficient sup-port for time-sensitive applications in a general-purpose

1

OS without degrading the performance of traditional ap-plications. One of the main contribution of this pa-per is to show through experimental evaluation that us-ing the above techniques it is possible to provide goodperformance to time-sensitive applications as well as tothroughput-oriented applications.

The rest of the paper is organized as follows. Section 2investigates the factors that contribute to poor temporalresponse in general-purpose systems. Section 3 describesthe techniques that we have used to implement our time-sensitive Linux system. Section 4 evaluates the behav-ior of several timing-sensitive applications and presentsoverheads of our time-sensitive Linux system. Finally, inSection 5 we state our conclusions.

1.1 Related Work

The scheduling problem has been extensively studied bythe real-time community [11, 14, 20]. However, most ofthe scheduling analysis is based on an abstract mathemat-ical model that ignores practical systems issues such askernel non-preemptibility and interrupt processing over-head. Recently, many different real-time algorithms havebeen implemented in Linux and in other general pur-pose kernels. For example, Linux/RK [17] implementsResource Reservations in the Linux kernel, and REDLinux [24] provides a generic scheduling framework forimplementing different real-time scheduling algorithms.These kernels tackle the practical systems issues men-tioned above with techniques similar to the techniquespresented in this paper. For example, RED Linux insertspreemption points in the kernel, and Timesys Linux/RT(based on RK technology) uses kernel preemptibility forreducing kernel latency. Kernel preemptibility is alsoused by MontaVista Linux [1]. However, while these ker-nels work well for time-sensitive applications, their per-formance overhead on throughput-oriented applicationsis not clear.

A different approach for providing real-time perfor-mance is used by other systems, such as RTLinux [6],RTAI [13], and KURT[21], which decrease the unpre-dictability of the system by running Linux as a back-ground process over a small real-time executive. In thiscase, real-time tasks are not Linux processes, but run onthe lower-level real-time executive, and the Linux kernelruns as a non real-time task. This solution provides goodreal-time performance, but does not provide it to Linuxapplications. Linux processes are still non real-time,and cannot support time-sensitive applications. Also, na-

tive real-time threads use a completely different, and lessevolved, ABI compared to the Linux one, and do not haveaccess to Linux device drivers.

An accurate timing mechanism is crucial for support-ing time-sensitive applications. Thus most of the exist-ing real-time kernels or real-time extensions to Linuxprovide high resolution timers. The high resolutiontimers concept was proposed by RT-Mach [19] and hassubsequently been used by several other systems suchas Rialto [9]. In a general-purpose operating system,the overhead of such timers can affect the performanceof throughput-oriented applications. This overhead iscaused by the increased number of interrupts generatedby the timing mechanism and can be mitigated by thesoft-timer mechanism [5]. Thus, our firm-timer imple-mentation uses soft timers.

Finally, the Nemesis operating system [10] is de-signed for multimedia and other time-sensitive applica-tions. However, its structure and API is very differentfrom the standard programming environment providedby general-purpose operating systems such as Linux.Our goal is to minimize changes to the programming en-vironment to encourage the use of time-sensitive appli-cations in a general-purpose environment.

2 Time-Sensitive Requirements

As said, to respect applications’ temporal constraints re-sources must be allocated at the appropriate times, wherethe appropriate times are determined by events of interestto the application, such as readiness of a video frame fordisplay. Hence, we can view the timeline of the applica-tion as a sequence of such events and the correspondingapplication’s activations. For example, Figure 1 showsan event and the corresponding activation. As the figureshows, there is a latency between the event and the ac-tivation. This latency has three components called timerresolution latency, non-preemptible section latency andscheduling latency as shown in the figure, which depictsthe execution sequence in a system after a wall-clocktime event. A time-sensitive system needs low total la-tency.

There are three requirements for providing allocationswith low latency: 1) an accurate timing mechanism, 2) aresponsive kernel and 3) an appropriate CPU schedulingalgorithm. These requirements are described below.

Timing Mechanism: An accurate timing mechanism iscrucial for reducing latency as timer resolution is

2

Another applicationscheduled

InterruptHandler

Section LatencyNon−Preemptible

LatencyTimer Resolution Scheduling

Latency

Wall−clock timeevent

Time

Scheduler Application scheduledTimer Interrupt(activation)

Figure 1: Execution sequence after wall-clock time expi-ration event.

the most common source of latency in an operat-ing system such as Linux [4]. Such a mechanismcan be implemented efficiently by using two tech-niques, one-shot timers available on most commonmodern hardware and soft timers [5]. These tech-niques are complementary and can be combinedtogether. One-shot timers can provide high accu-racy, but, unlike periodic timers, they require repro-gramming at each activation. On an x86 machine,one-shot timers can be generated using the on-chipCPU Advanced Programmable Interrupt Controller(APIC). This timer has very high resolution and canbe reprogrammed in a few CPU cycles.1

Soft timers check and run expired timers at strate-gic points in the kernel thus reducing the number ofhardware generated timer interrupts and the num-ber of user-kernel context switches. We call thecombination of these two mechanisms, firm timers.In Section 4.3, we show that the overhead of firmtimers is not significant.

Responsive Kernel: An accurate timing mechanism isnot sufficient for reducing latency. For example,even if a timer interrupt is generated by the hard-ware at the correct time, the activation could oc-cur much later because the kernel is unable to in-terrupt its current activity. This problem occurs be-cause either the interrupt might be disabled or thekernel is in a non-preemptible section. In traditionalkernels, a thread entering the kernel becomes non-preemptible and must either yield the CPU or exitthe kernel before an activation can occur. The so-lution is to reduce the size of such non-preemptiblesections as described in Section 3.2.

CPU Scheduling Algorithm: The scheduling problem

1In contrast, reprogramming the standard programmable intervaltimer (PIT) on an x86 is very expensive because it requires severalslow out instructions on the ISA bus.

for providing precise allocations has been exten-sively studied in the literature but most of the workrelies on some strict assumptions such as full pre-emptibility of tasks. A responsive kernel with anaccurate timing mechanism enables implementationof such CPU scheduling strategies because it makesthe assumptions more realistic and improves the ac-curacy of scheduling analysis. In this paper, weuse two different real-time scheduling algorithms:a proportion-period scheduler and a priority-basedscheduler.

The proportion-period scheduler provides tempo-ral protection to tasks. With proportion-period, wemodel application behavior by identifying a char-acteristic delay that the application can tolerate andallocating a fixed proportion of the CPU within aperiod equal to the delay to each task in the applica-tion. Alternatively, we assign priorities to all tasksin some application-specific order for use with thepriority scheduler.

While each of these requirements have been addressedin the past, they have generally been applied to specificproblems in limited environments. When applied in iso-lation in the context of a general-purpose system, theyfail to provide good time-sensitive performance. For ex-ample, a high resolution timer mechanism is not use-ful to user-level applications without a responsive kernel.This probably explains why soft timers [5] did not exporttheir functionality to the user level through the standardPOSIX API. Conversely, a responsive kernel without ac-curate timing has only a few applications. For example,the low latency Linux kernel [16] provides low latencyonly when an external interrupt source such as an audiocard is used.

Similarly, a scheduler that provides good theoreticalguarantees is not effective when the kernel is not respon-sive or its timers are not accurate. Conversely, a respon-sive kernel with an accurate timing mechanism is un-able to handle a large class of time-sensitive applicationswithout an effective scheduler. Unfortunately, these so-lutions have generally not been integrated: on one hand,real-time research has developed good schedulers and an-alyzed them from a mathematical point of view, and onthe other hand, there are real systems that provide a re-sponsive kernel but provide simplistic schedulers that areonly designed to be fast and efficient [1]. Real-time oper-ating systems integrate these solutions for time-sensitivetasks but tend to ignore the performance overhead of their

3

solutions on throughput-oriented applications [17]. Ourgoal is to support both types of applications well.

It is worth noting that latency due to system services,such as the X11 server for graphical display [3] on aLinux system, has the same components as kernel latencydescribed above. In fact, the simple scheduling modelpresented above assumes that tasks are independent. Ina real application, tasks can be interdependent which cancause priority inversion problems. For example, in Sec-tion 4.2.1 we show that a multimedia player that uses theX11 server for display can perform sub-optimally due topriority inversion, even if the kernel allocates resourcescorrectly. The X11 server operates on client requests inan event-driven manner and the handling of each eventis non-preemptible and generally in FIFO order. As aresult, time-sensitive clients expecting service from theserver observe latencies that depend on the time to ser-vice previous client requests: the performance of time-sensitive applications depends on not just kernel supportfor such applications but also the design of other systemservices. Thus in Section 3.3 we enhance the priorityscheduler with techniques that solve priority inversion.

In the next section, we present the components of TSL,providing accurate timers, a responsive kernel, and time-sensitive scheduling algorithms to support the require-ments highlighted above.

3 Implementing Time-Sensitive Sys-tems

We propose three techniques, firm timers, kernel pre-emptibility and proportion-period CPU scheduling thataim to reduce the total latency. We have integrated thesetechniques in the 2.4.16 Linux kernel to implement TSL.

3.1 Firm Timers

Firm timers provide an accurate timing mechanism withlow overhead by combining three approaches for imple-menting timers: one-shot timers, periodic timers and softtimers. One-shot timers provide high accuracy while pe-riodic timers and soft timers provide low overhead tim-ing.

Periodic timers are normally implemented with pe-riodic tick interrupts. For example, on x86 machines,this interrupt is generated by the Programmable IntervalTimer (PIT), and on Linux, the period of the interrupt is10 ms. As a result, the maximum timer resolution latency

is 10 ms. This latency can be reduced by reducing thetick period but this solution increases system overheadbecause more tick interrupts are generated. To reducethe overhead of timers, we have to move from a periodictimer interrupt model to a one-shot timer interrupt modelwhere interrupts are generated only when needed. Thefollowing example explains the benefits of one-shot in-terrupts. Consider two tasks with periods 5 and 7 ms.With periodic interrupts, the tick period must be 1 ms toeliminate timer resolution latency. Hence in 35 ms, therewould be 35 interrupts generated. With one-shot inter-rupts, interrupts will be generated at 5 ms, 7 ms, 10 ms,etc., and the total number of interrupts is 11. One-shottimers have to be reprogrammed for the next timer eventat each activation (unlike periodic timers) but avoid theflood of unwanted interrupts that are necessary to main-tain good accuracy with a periodic timer and an aperiodicinterval.

One-shot timers improve timer accuracy but have theirown sources of overhead. These overheads occur due tothree reasons: 1) reprogramming at each activation, 2)maintenance of timer data structures, and 3) increasedinterrupts with fine-grained timing. The data structuresfor one-shot timers are less efficient than for periodictimers. For instance, periodic timers can be implementedusing calendar queues [7] which operate in ������� time,while one-shot timers require priority heaps which re-quire ������ �������� time, where � is the number of activetimers. This difference exists because periodic timershave a natural bucket width (in time) that is the tick pe-riod of the timer interrupt. Calendar queues need thisbucket width and derive their efficiency by providing noordering to timers within a bucket. One-shot fine-grainedtimers have no corresponding bucket width. Firm timersare implemented using one-shot timers. Below, we de-scribe the methods used by firm timers to reduce eachsource of overhead.

Timer reprogramming: Fortunately, timer reprogram-ming is inexpensive on modern hardware. Forexample, our firm-timers implementation uses theAPIC one-shot timer present in modern x86 ma-chines. The APIC is set by writing a value into aregister which is decremented at each memory buscycle until it reaches zero and generates an interrupt.Given a 100 mhz memory bus, a one-shot timerhas a theoretical accuracy of 10 nanoseconds.2This

2In practice, the interrupt handler is much slower than 10 ns andis the limiting factor for timer accuracy.

4

timer resides on-chip an can be reprogrammed in afew cycles without noticeable performance penalty.

Data structures: Firm timers maintain a timer queuefor each processor. When the APIC timer expires,the interrupt handler checks the timer queue and ex-ecutes the callback function associated with eachexpired timer in the queue. Expired timers are re-moved while periodic timers are re-enqueued aftertheir expiration field is incremented by the value intheir period field. The APIC timer is then repro-grammed to generate an interrupt at the next timerevent.

The firm timer expiration times are specified as CPUclock cycle values. We use two performance opti-mizations in the firm timers implementation. First,the expiration time is specified as a 32 bit quantityalthough the current time in CPU cycles is storedin a 64 bit register in an x86 processor. The 32 bitexpiration value avoids expensive 64 bit time con-versions from CPU cycles to memory cycles neededfor programming the APIC timer. This choice of a(signed) 32 bit value limits the use of firm timers toa maximum one second timeout on a modern twoGhz processor due to time roll over. Fortunately,one-shot timers are not needed for long timeouts.Instead, and second, firm timers combine periodictimers together with one-shot timers for long time-outs. A firm timer for a long timeout uses a peri-odic timer to wake up at the last period before thetimer expiration and then sets the one-shot APICtimer. Hence, our firm timers approach only has ac-tive one-shot timers within one tick period. Sincethe number of such timers, � , is decreased, the datastructure implementation is more efficient.

Increased interrupts: The increased timer accuracy ofone-shot timers comes at the cost of increased over-head because every timer interrupt leads to a user-kernel context switch that can stall the CPU pipelineand can result in cache pollution. To solve this prob-lem, firm timers use soft timers [5]. The APIC timercan be set to always overshoot by a fixed amount oftime called timer latency. Soft timers add checks forexpired timers at strategic points in the kernel suchas system call, interrupt, and exception return paths.In some cases, an interrupt, system call, or excep-tion may happen after a timer has expired but beforethe APIC timer generates an interrupt. At this point,

the timer expiration is handled and the APIC timeris reprogrammed for the next timer event. This ap-proach avoids the extra user-kernel context switchoverhead of the timer interrupt at the cost of sometimer accuracy.

Soft timer checks are normally placed at kernel exitpoints, which end critical sections in the kernel andwhere the scheduler function can be invoked. A pre-emptible kernel design, as described in Section 3.2,reduces the granularity of critical sections in the ker-nel and thus allows more frequent soft timer checksand hence can provide better timing accuracy.

The timer overshoot or latency value allows makinga tradeoff between accuracy and overhead. A smallvalue of timer latency provides high timer resolu-tion while a large value decreases the timer over-head. This latency value can be changed dynami-cally. With a zero value, we obtain hard timers andwith a large value, we obtain soft timers. A choicein between leads to a hybrid approach that is used byour firm timers. This choice depends on the timingaccuracy needed by the application. Our current im-plementation uses a single global latency value butit is easy to extend this implementation so that eachtimer can specify its desired latency.

We want to provide the benefits of the accurate timingmechanism to standard user-level applications. These ap-plications use the standard POSIX interface calls suchas nanosleep(), pause(), setitimer(), se-lect() and poll(). We have modified the imple-mentation of these system calls to use firm timers withoutchanging their interface. As a result, unmodified appli-cations automatically get increased timer accuracy in oursystem.

3.2 Kernel Responsiveness

A kernel is responsive when its non-preemptible sections,which keep the scheduler from being invoked to sched-ule a task, are small. For example, if the timer inter-rupt in Figure 1 is disabled, the task can only enter theready queue when the interrupt is re-enabled. In addi-tion, the task upon entering the ready queue may still notbe scheduled if another task is running in the kernel in anon-preemptible section.

The length of non-preemptible sections in a kernel de-pends on the strategy that the kernel uses to guarantee the

5

consistency of its internal structures and on the internalorganization of the kernel. Traditional general purposekernels allow at most one execution flow in the kernel atany given time by disabling preemption when an execu-tion flow enters the kernel, i.e., when an interrupt fires orwhen a system call is invoked. Thus the non-preemptiblesection latency is equal to the maximum length of a sys-tem call plus the processing time of all the interrupts thatfire before returning to user mode. Unfortunately, thisvalue can be as large as 100 ms [4].

One approach that reduces non-preemptible section la-tency is explicit insertion of preemption points at strate-gic points inside the kernel so that a thread in the ker-nel explicitly yields the CPU to some other thread af-ter it has executed for some time. In this way, the sizeof non-preemptible sections is reduced. The choice ofpreemption points depends on system call paths and hasto be manually placed after careful auditing of systemcode. This approach is used by some real-time versionsof Linux, such as RED Linux [17] and by Andrew Mor-ton’s low-latency project [16]. Non-preemptible sectionlatency in such a kernel decreases to the maximum timebetween two preemption points.

Another approach, used in most real-time systems, re-moves the constraint of a single execution flow inside thekernel. Thus it is not necessary to disable preemptionwhen an execution flow enters the kernel. To supportthis level of kernel preemptibility, kernel data must be ex-plicitly protected using mutexes or spinlocks. The Linuxpreemptible kernel project [12] uses this approach anddisables kernel preemption only when a spinlock is held.In a preemptible kernel, the non-preemptible section la-tency is determined by the maximum amount of time forwhich a spinlock is held inside the kernel.

Our previous evaluation [4] shows that these ap-proaches work fairly well and should be incorporated inthe design of any time-sensitive kernel. Our experimentswith real applications in Section 4.2 show that a respon-sive kernel complements an accurate timing mechanismto help improve time-sensitive application performance.

3.3 CPU Scheduling

The CPU scheduling algorithm should ensure that time-sensitive tasks obtain their correct allocation with low la-tency. We use the proportion-period and priority modelsas described Section 2 to schedule time-sensitive appli-cations. The proportion-period model provides temporalprotection to applications and allows balancing the needs

of time-sensitive applications with non-real time applica-tions but requires specification of proportion and periodscheduling parameters of each task. The priority modelhas a simpler programming interface but assumes that thetiming needs of tasks are well-behaved.

3.3.1 Proportion-Period CPU Scheduling

For a single independent task, the simplest scheduling so-lution is to assign the highest priority to the task. How-ever, with this solution, a misbehaving task that does notyield the CPU can starve all other tasks in the system.A time-sensitive general purpose system should providetemporal protection to tasks so that misbehaved tasks thatconsume “too much” execution time do not affect theschedule of other tasks. The temporal protection prop-erty is similar to memory protection in standard operatingsystems that provides memory isolation to each applica-tion.

Our proportion-period allocation model automaticallyprovides temporal protection because each task is allo-cated a fixed proportion every period. The period of atask is related to some application-level delay require-ment of the application, such as the period of a periodictask, or it can be derived from the jitter requirements ofa time-sensitive task. The proportion is the amount ofCPU allocation required every period for correct task ex-ecution. The proportion-period model can be effectivelyimplemented using well known results from real-timescheduling research[15]. In this implementation, classi-cal real-time scheduling techniques (EDF or RM priorityassignment) are used and a task is allowed to execute asa real-time task for a time � equal to the product of itsproportion � and its period � and then blocking the task(or scheduling it as a non real-time task) until its next pe-riod. In this way, a task is reshaped so that it behaves likea periodic real-time task with parameters ��������� and canbe properly scheduled by a classical real-time scheduler.A similar technique is used in networks by traffic shaperssuch as leaky buckets or token buckets.

We have implemented a proportion-period CPU sched-uler in Linux to provide temporal protection to tasks.This scheduler uses an EDF scheduling mechanism toobtain full processor utilization. In addition, when twotasks have the same deadline, the one with the small-est remaining capacity is scheduled to reduce the averagefinishing time.

6

3.3.2 Priority CPU Scheduling

In the priority model, real-time priorities are assigned totime-sensitive tasks based on application needs [11]. Onekey problem with the priority model is priority inversionwhich occurs when an application is composed of mul-tiple tasks that are interdependent. The classic priorityinversion problem occurs with three tasks. For example,consider a simple example of a video application consist-ing of a client and an X11 server. Let us assume that theclient has been assigned the highest priority because itis time-sensitive. It displays graphics by requesting ser-vices from the X11 server. When it needs to display avideo frame, it sends the frame to the server and thenblocks waiting for the display to complete. If the X11server has a priority lower than the client’s priority, thenit can be preempted by another task with a medium prior-ity. Hence the medium priority task is delaying the serverand thus delaying the high-priority client task.

We use a variant of the priority ceiling protocol [20]called the highest locking priority (HLP) protocol to copewith priority inversion. The HLP protocol works as fol-lows: when a task acquires a resource, it automaticallygets the highest priority of any task that can acquire thisresource. In the example above, the display is the sharedresource and thus the X11 server gets the highest priorityamong all time-sensitive clients accessing it. Hence, theX11 server gets the priority of the client task and is notpreempted by the medium priority task.

The HLP protocol is very general and works withacross multiple servers. Interestingly, this protocol han-dles the FIFO ordering problem in server queues men-tioned in Section 3. Since servers have the highest prior-ity among all their potential clients, they are able to serveeach request immediately after it is enqueued and thusthe queue size is never more than one and the queuingstrategy is not relevant. After servicing the request, thenext highest-priority client is scheduled and the latencycaused by the server is minimized.

4 Evaluation

This section describes the results of experiments we per-formed to evaluate the behavior of time-sensitive appli-cations running on this system and the overheads of thissystem. Our experiments focus on evaluating the behav-ior of realistic time-sensitive applications running on aloaded general-purpose environment, and were run on a

1.5 Ghz Pentium-4 Intel processor with 512 MB of mem-ory.

4.1 Micro Benchmarks

Before evaluating the impact of the latency reductiontechniques used in TSL on a real application, we per-formed some micro-benchmarks for evaluating the ker-nel latency as defined in Section 2.

As previously shown, the kernel latency is composedby the timer resolution latency, the non-preemptible la-tency, and the scheduling latency. We evaluated the firsttwo components in isolation by running a time-sensitiveprocess that requires to sleep for a specified amount oftime (using the nanosleep() system call) and mea-sures the time that it actually sleeps. In a first set ofexperiments, we evaluated the timer resolution latency,showing that it is ������� in standard Linux, and that firmtimers can reduce it to few microseconds.

After that, we evaluated the non-preemptible latencywhen a number of different system loads are ran in back-ground. The first interesting result was that on standardLinux the worst case non-preemptible latency (occurringwhen the kernel is copying large amounts of data be-tween kernel and user space) can arrive to ��� ����� , butthat in most common cases the non-preemptible latencyis less than ������� , and it is hidden by the timer reso-lution latency. However, when firm timers are used thenon-preemptible latency becomes more visible, and it ispossible to see that it is easy to obtain latencies biggerthan !"��� . Using appropriate kernel preemptibility tech-niques, the latency can be greatly reduced, and TSL pro-vides a maximum kernel latency of less than �#��� on ourtest machine.

The full details of the experiments and more resultsare presented in our previous paper [4], that we brieflysummarized here for the reader’s convenience.

4.2 Latency of Real Applications

After evaluating the kernel latency in isolation throughmicro-benchmarks, we performed experiments on tworeal applications, mplayer and the proportion-periodscheduler which is a kernel-level application. We chooseaudio/video synchronization skew as the latency metricfor mplayer. The latency metric for the proportion-periodscheduler is maximum error in the allocation and periodboundary.

7

4.2.1 Mplayer

Mplayer [2] is an audio/video player that can handle sev-eral different media formats. Mplayer synchronizes au-dio and video streams by using timestamps that are asso-ciated with the audio and video frames. The audio card isused as a timing source, i.e., audio samples are put in theaudio card buffer, and when a video frame is decoded,its timestamp is compared with the timestamp of the cur-rently played audio sample. If the video timestamp issmaller than the audio timestamp then the program is late(i.e., a video deadline has been missed) and the video isimmediately displayed. Otherwise, the system sleeps un-til the video and audio timestamps are equal and thendisplays the video.

On a responsive kernel and with sufficient avail-able CPU capacity, audio/video synchronization can beachieved by simply sleeping for the correct amount oftime. Thus, mplayer uses the Linux nanosleep() callfor synchronization. Unfortunately, if the kernel is unre-sponsive, mplayer will not be able to sleep for the correctamount of time leading to poor audio/video synchroniza-tion and high jitter in the inter-frame display times. Syn-chronization skew and display jitter are correlated andthus this paper presents results only for audio/video syn-chronization skew.

We compare the audio/video skew of mplayer on stan-dard Linux and on our time-sensitive Linux under threecompeting loads: 1) non-kernel CPU load, 2) kernel CPUload, and 3) file system load. For non-kernel load, a user-level CPU stress test is run in the background. For ker-nel CPU load, a large memory buffer is copied to a file,where the kernel uses CPU to move the data from theuser to the kernel space. Standard Linux does this activ-ity in a non-preemptible section. This load spends 90%of its execution time in kernel mode. For the file systemload, a large directory is copied recursively and the filesystem is flushed multiple times to create heavy file sys-tem activity. In each of these tests, mplayer is run for 90seconds at real-time priority.

Non-kernel CPU load Figure 2 shows the audio/videoskew in mplayer on a Linux and a time-sensitive Linuxkernel when a CPU stress test is the competing load. Thiscompeting load runs an infinite loop consuming as muchCPU as possible. Figure 2(a) shows that for standardLinux the maximum skew is large and close to 50 mswhen the X11 server is run at a non-real time priority.For most data points, the skew lies between -5 ms to

5 ms because of Linux’s 10 ms timer resolution. Fig-ure 2(b) shows that the skew for time-sensitive Linux,when X11 is run at a non-real time priority, is still large(up to $ !"��� ) but does not show oscillations between -5 ms and 5 ms. Finally, Figure 2(c) shows that the skewfor time-sensitive Linux improves considerably and isless than 250 us when the X11 server runs at real-timepriority. The real-time priority value of X11 is the sameas the priority assigned to mplayer.

These figures show that time-sensitive Linux workswell on a non-kernel CPU load as long as the HLP proto-col, described in Section 3.3.2, is used to assign prioritiesto time-sensitive tasks and to server tasks with the sharedresources. Linux with the X11 server at real-time prior-ity still has a skew between -5 ms to 5 ms because of thetimer resolution.

As a result of this experiment, the rest of the exper-iments in this section are run with mplayer and X11 atreal-time priority to avoid any user-level priority inver-sion effects.

Kernel CPU Load The second experiment comparesthe audio/video skew in mplayer between Linux andtime-sensitive Linux when the background load copies alarge 8 MB memory buffer to a file with a single writesystem call. Figure 3(a) shows the audio/video skew isas large as 90 ms for Linux. In this case, the kernelmoves the data from the user to the kernel space in a non-preemptible section. Figure 3(b) shows that the maxi-mum skew is less than 400 us for time-sensitive Linux.This improvement occurs as a result of improved ker-nel preemptibility for large write calls in time-sensitiveLinux.

File System Load The third experiment compares theaudio/video skew in mplayer between Linux and time-sensitive Linux when the background load repeatedlycopies a compiled Linux kernel sources directory recur-sively and then flushes the file system. This directory has13000 files and 180 MB of data and is stored on the Linuxext2 file system. The kernel uses DMA for transferringdisk data. Figure 4(a) shows that the skew under Linuxcan be as high as 120 ms while Figure 4(b) shows thatskew is less than 200 us under time-sensitive Linux. Thisresult shows that time-sensitive Linux can provide lowlatencies even under heavy file-system and disk load.

8

-10000

0

10000

20000

30000

40000

50000

60000

0 500 1000 1500 2000 2500 3000

Diff

eren

ce B

etw

een

Vid

eo a

nd A

udio

Tim

esta

mps

(us

ec)

Video Frame Number

Audio/Video Synchronization

(a) Linux, X server non-real time

0

5000

10000

15000

20000

25000

30000

35000

40000

0 500 1000 1500 2000 2500 3000

Diff

eren

ce B

etw

een

Vid

eo a

nd A

udio

Tim

esta

mps

(us

ec)

Video Frame Number

Audio/Video Synchronization

(b) Time-sensitive Linux, X servernon-real time

0

50

100

150

200

250

0 500 1000 1500 2000 2500 3000

Diff

eren

ce B

etw

een

Vid

eo a

nd A

udio

Tim

esta

mps

(us

ec)

Video Frame Number

Audio/Video Synchronization

(c) Time-sensitive Linux, X serverreal-time

Figure 2: Audio/Video Skew on Linux and time-sensitive Linux. Background load is a CPU stress test that run anempty loop. Note that the three figures have different scales, and that the maximum skew in Figure (c) is muchsmaller than the maximum skew in the other two cases.

-10000

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

0 500 1000 1500 2000 2500 3000

Diff

eren

ce B

etw

een

Vid

eo a

nd A

udio

Tim

esta

mps

(us

ec)

Video Frame Number

Audio/Video Synchronization

Firm Timers

(a) Linux

0

50

100

150

200

250

300

350

400

0 500 1000 1500 2000 2500 3000

Diff

eren

ce B

etw

een

Vid

eo a

nd A

udio

Tim

esta

mps

(us

ec)

Video Frame Number

Audio/Video Synchronization

Firm Timers

(b) Time-sensitive Linux

Figure 3: Audio/Video Skew on Linux and time-sensitive Linux. Background load copies a 8 MB buffer from userlevel to a file with a single write call. Note that the two figures have different scales, and that the maximum skewin Figure (b) is much smaller than in Figure (a)

4.2.2 Proportion-Period Scheduler

Our original motivation for implementing a time-sensitive Linux system was to accurately implement aproportion-period scheduler. This scheduler is usedto provide a reservation mechanism for a higher-levelfeedback-based real-rate scheduler[22]. The real-ratescheduler uses an application-specific progress rate met-ric in time-sensitive tasks to automatically assign correctallocations to such tasks. For example, the progress of aproducer or consumer of a bounded buffer can be inferred

by measuring the fill-level of the bounded buffer. If thebuffer is full, the consumer is falling behind and needsmore resources while the producer needs to be sloweddown.

The feedback allocation accuracy depends (amongother factors) on the accuracy of actuating proportions.There are three sources of inaccuracy in our proportion-period scheduler implementation on Standard Linux: 1)the period boundaries are quantized to multiples of thetimer resolution or 10 ms, 2) the policing of proportionsis also limited to the same value because timers have to

9

-20000

0

20000

40000

60000

80000

100000

120000

0 500 1000 1500 2000 2500 3000

Diff

eren

ce B

etw

een

Vid

eo a

nd A

udio

Tim

esta

mps

(us

ec)

Video Frame Number

Audio/Video Synchronization

(a) Linux

0

50

100

150

200

250

300

350

400

450

500

0 500 1000 1500 2000 2500 3000

Diff

eren

ce B

etw

een

Vid

eo a

nd A

udio

Tim

esta

mps

(us

ec)

Video Frame Number

Audio/Video Synchronization

(b) Time-sensitive Linux

Figure 4: Audio/Video Skew on Linux and time-sensitive Linux. Background load repeatedly copies a compiledLinux kernel sources directory recursively and then flushes the file system. Note that the two figures have differentscales, and that the maximum skew in Figure (b) is much smaller than in Figure (a).

be used to implement policing, and 3) heavy loads causelong non-preemptible paths and thus large jitter in periodboundaries and proportion policing. These inaccuraciesintroduce noise in the system that can cause large alloca-tion fluctuations even when the input progress signal canbe captured perfectly and the controller is well-tuned.

Our time-sensitive Linux’s proportion-period sched-uler uses firm-timers for implementing period boundariesand proportion policing. To evaluate the accuracy of thisscheduler, we ran two processes with proportions of 40%and 20% and periods of 8192 us and 512 us respectively.3

These processes were run first on an unloaded system toverify the correctness of the scheduler. Then, we eval-uated the scheduler behavior when the same processeswere run with competing file system load (described inSection 4.2.1). In this experiment each process runs atight loop that repeatedly invokes the gettimeofday sys-tem call to measure the current time and stores this valuein an array. The scheduler behavior is inferred at the user-level by simply measuring the time difference betweensuccessive elements of the array. A similar technique isused by Hourglass [18].

Table 1 shows the maximum deviation in the propor-tion allocated and the period boundary for each of the two

3The current proportion-period scheduler allows task periods thatare multiples of 512 us. While this period alignment restriction isnot needed for a proportion-period scheduler, it simplifies feedback-based adjustment of task proportions.

processes. This table shows that the proportion-periodscheduler allocates resources with a very low deviationof less than % !"&�� on a lightly loaded system. Under highfile system load the results show larger deviations. Thesedeviations occur because execution time is “stolen” bythe kernel interrupt handling code which runs at a higherpriority than user-level processes in Linux.

One way to improve the proportion-period schedulingperformance in the presence of heavy file system load isto defer certain parts of interrupt processing after real-time processes. We are currently investigating this solu-tion.

An alternative method for evaluating the scheduler be-havior is to use a kernel tracer, such as LTT[23], that canregister the occurrence of certain key events in the ker-nel. These events can be analyzed later after programexecution. Kernel tracers are often used in real-time sys-tems for verifying the temporal correctness of the ker-nel and of time-sensitive applications. We ported LTTto time-sensitive Linux, and Figure 5 shows a samplesession analyzing the schedule generated in the previousexperiments. The trace visualizer application shows theschedule, visualizing processes execution (an executingprocess is shown in black): the two proportion-periodprocesses, marked as “unnamed child” by the trace vi-sualizer and characterized by PIDs 2493 and 2492, areeasily recognizable, because they use most of the CPU

10

No Load File System LoadMax Proportion Max Period Max Proportion Max Period

Deviation Deviation Deviation DeviationTask 1

Proportion: 40%, 3276.8 us 0.3% ( ' 25us) 5us 6% ( ' 490us) 534usPeriod: 8192 us

Task 2Proportion: 20%, 102.4 us 0.7% ( ' 3us) 10us 4% ( ' 20us) 97us

Period: 512 us

Table 1: Deviation in proportion and period for two processes running on the proportion-period scheduler on time-sensitive Linux.

Figure 5: Linux kernel tracer (LTT) helps to visualize the schedule generated by two proportion-period processes.

time. Also note that their execution is regular, and co-incides with the one expected for two processes withproportion-periods ��()�+*,�.-/�10 %)� and �2%��/�.!/�1%)� . All thenon time-sensitive processes can execute when processes2492 and 2493 exhausted their proportions: for example,a small execution of the lyx editor is visible after the firstperiod of process 2493.

4.3 System Overhead

High resolution timers in a general purpose OS can po-tentially have high overheads. To mitigate this overhead,our firm timers implementation combines hard and softtimers. In this section, we present experiments to high-light the advantages of firm timers as compared to purehard timers and show that the firm-timer overhead is ac-ceptable.

In all these experiments, we measure the performance

11

26

27

28

29

30

31

32

33

hard firm 0 firm 50 firm 100 firm 500 soft

Fin

ishi

ng T

imes

povray with 1000 ms period timer

1000

Figure 7: Comparison between Hard and Firm timerswith different accuracy.

of a throughput-oriented application when one or moretime-sensitive processes are run in the background tostress the firm timers mechanism. For the throughputapplication, we selected povray, a ray-tracing appli-cation and used it to render a standard benchmark imagecalled skyvase.

Figure 6 shows the performance overhead of firmtimers as compared to standard Linux timers when multi-ple periodic time-sensitive processes are running simul-taneously. We conducted experiments with 20 and 50timers running with 10 ms and 100 ms period. Theoverhead is defined as the ratio of the time needed bypovray to render the image in time-sensitive Linux andthe time needed to render the same image in standardLinux. The figures show the overhead for time-sensitiveLinux with pure hard timers, firm timers with accuracy500 us, and pure soft timers. These figures show thatpure soft timers do not have any overhead as comparedto standard Linux timers (except when 50 time-sensitiveprocesses with period 10 ms are run), hard timers have aslightly greater overhead and firm timers, in this case, donot provide much improvement.

We also performed the same experiment but with pe-riodic processes running at higher frequencies. Figure7 shows the time needed to render the image when 20periodic processes are run with a period of 1 ms. We donot compare these results with Linux because Linux doesnot support 1 ms timer accuracy. The benefit of the firmtimers mechanism for improving throughput becomesobvious when the process periods are made shorter.4

4Note that the low execution time with pure soft timers is because

The previous experiments show that pure hard timershave lower overhead in some cases and firm timers havelower overhead in other cases. This result can be ex-plained by the fact that there is a cost associated withchecking whether a soft timer has expired. Thus, the softtimers mechanism is effective in reducing overhead whenenough of these checks result in the firing of a soft timer.Otherwise the firm-timer overhead as compared to purehard timers will be higher.

More formally, the previous behavior can be explainedas follows. Let � be the total number of timers that mustfire in a given interval of time, 3 the number of hardtimers that fire, 4 be the number of soft timers that fire(hence, �65 38794 ) and : be the number of checks forsoft timers expirations. Let :<; be the cost for firing ahard timer, :>= be the cost of firing a soft timer, and :<? bethe cost of checking if some soft timer has expired. Thetotal cost of firing firm timers is : ? :@79:A; 3B79: = 4 . Ifpure hard timers are used then the cost is :C; � . Hence,firm timers reduce the system overhead if

:A?.:D7D:A; 387D:E=F4HG@:A; �JI

:A?.:KG@:A; �L�NM 3 �OM :E=P4 I:A?.:KG � :A; M :E= � 4 I4RQ :TS@: ? Q � :A; M : = �

Thus when the ratio of the number of the soft timersthat fire to the number of soft timer checks is suffi-ciently large (i.e., it is larger than :C?.Q � :A; M :E= � ), thenfirm timers are effective in reducing the high-resolutiontimers overhead. In our experiments, we measured :U; 5-"&�� , :E= 5 �#&V� , and :A? 5 �/WX�1!/��Y YZ&V� , hence thefirm timers mechanism becomes effective when 4[Q :\S�/WX�1!/��Y Y Q �2-MJ���U5]�/W^�)%/�1_ - % , that is to say when morethan %`WX��* of the soft timer checks result in a soft timerto be fired.

Note that the number of checks : depends on theamount of interrupts and system calls that happen in themachine, whereas the amount of soft timers that fire 4depends on how the checks and the timers’ deadlines aredistributed in time. The original work on soft timers [5]studied these distributions for a number of workloads.Their results show that for many workloads the distribu-tions are such that checks often occur close to deadlines(thus increasing 4RQ : ), although how close is very work-load dependent. Firm timers have the benefit of assuring

not all timers fired at the correct times. Thus this value should not beconsidered in the comparison.

12

0.9

0.95

1

1.05

1.1

No timers 20 timers, 100 ms 50 timers, 100 ms 20 timers, 10 ms 50 timers, 10 ms

Nor

mal

ized

Fin

ishi

ng T

ime

povray with hard timers

hard

(a) Hard Timers

0.9

0.95

1

1.05

1.1

No timers 20 timers, 100 ms 50 timers, 100 ms 20 timers, 10 ms 50 timers, 10 ms

Nor

mal

ized

Fin

ishi

ng T

ime

povray with firm500 timers

firm500

(b) Firm Timers (accuracy a.bFbPc d )

0.9

0.95

1

1.05

1.1

No timers 20 timers, 100 ms 50 timers, 100 ms 20 timers, 10 ms 50 timers, 10 ms

Nor

mal

ized

Fin

ishi

ng T

ime

povray with soft timers

soft

(c) Soft Timers

Figure 6: Overhead of firm timers in time-sensitive Linux as compared to standard Linux.

response even for workloads with poor distributions, yetretaining the performance benefits of soft timers whenthe workload permits.

5 Conclusions

This paper describes the design and implementation ofa time-sensitive Linux system that can support applica-tions requiring fine-grained resource allocation and low-latency response. The three key techniques that wehave investigated in the context of our time-sensitiveLinux system are firm timers for accurate timing, ker-nel preemptibility for improving kernel responsivenessand proportion-period scheduling for providing preciseallocations to tasks. Our experiments show that integrat-ing these techniques helps provide allocations to time-sensitive tasks with a variation of less than 400 us evenunder heavy CPU, disk and file system load. We showthat the overhead of time-sensitive Linux on throughput-oriented applications is low and thus such a systemcan be used effectively for time-sensitive and general-purpose applications.

Although the first results presented in this paper arepromising, TSL still need further investigation, sincethere are open issues related, for example, to interruptservice, to network latencies, and to firm timers perfor-mance. For firm timers in particular, we interested ininvestigating whether real workloads commonly lead tothe 4RQ :eSgf condition under which firm timers are ef-fective.

References

[1] Montavista software - powering the embedded rev-olution. http://www.mvista.com/.

[2] Mplayer - movie player for linux. http://www.mplayerhq.hu.

[3] The X window system. www.x.org.

[4] Luca Abeni, Ashvin Goel, Charles Krasic, JimSnow, and Jonathan Walpole. A Measurement-Based Analysis of the Real-Time Performance ofthe Linux Kernel. In submission to the Real TimeTechnology and Applications Symposium (RTAS),March 2002.

[5] Mohin Aron and Peter Druschel. Soft Timers: Effi-cient Microsecond Software Timer Support for Net-work Processing. ACM Transactions on ComputerSystems, August 2000.

[6] Michael Barabanov and Victor Yodaiken. Real-time linux. Linux Journal, March 1996.

[7] Randy Brown. Calendar queues: A fast 0(1) prior-ity queue implementation for the simulation eventset problem. CACM, 31(10):1220–1227, October1988.

[8] Intel Corporation, editor. Pentium Pro Family De-veloper’s Manual, chapter 7.4.15. Intel, December1995.

[9] Michael B. Jones, Joseph S. Barrera III, AlessandroForin, Paul J. Leach, Daniela Rosu, and Marcel-Catalin Rosu. An overview of the rialto real-time

13

architecture. In In Proceedings of the Seventh ACMSIGOPS European Workshop, Connemara, Ireland,September 1996.

[10] Ian M. Leslie, Derek McAuley, Richard Black,Timothy Roscoe, Paul T. Barham, David Evers,Robin Fairbairns, and Eoin Hyden. The design andimplementation of an operating system to supportdistributed multimedia applications. IEEE Journalof Selected Areas in Communications, 14(7):1280–1297, 1996.

[11] C. L. Liu and J. Layland. Scheduling algorithm formultiprogramming in a hard real-time environment.Journal of the ACM, 20(1):46–61, Jan 1973.

[12] Robert Love. The Linux Kernel PreemptionProject. http://kpreempt.sourceforge.net.

[13] P. Mantegazza, E. Bianchi, L. Dozio, and S. Pa-pacharalambous. RTAI: Real time application in-terface. Linux Journal, 72, 2000.

[14] C. W. Mercer, S. Savage, and H. Tokuda. Pro-cessor Capacity Reserves: Operating System Sup-port for Multimedia Applications. In Proceedingsof the IEEE International Conference on Multime-dia Computing and Systems, May 1994. To appear.(This is a condensed version of tech report CMU-CS-93-157.).

[15] Clifford W. Mercer and Hideyuki Tokuda. Pre-emptibility in real-time operating systems. In InProceedings of the 13th IEEE Real-Time SystemsSymposium, December 1992.

[16] Andrew Morton. Linux scheduling latency.http://www.zip.com.au/˜akpm/linux/schedlat.html.

[17] Shui Oikawa and Raj Rajkumar. Linux/RK: Aportable resource kernel in Linux. In Proceedingsof the IEEE Real-Time Systems Symposium Work-In-Progress, Madrid, December 1998.

[18] John Regehr. Inferring scheduling behavior withhourglass. In Proceedings of the Freenix Track ofthe 2002 USENIX Annual Technical Conference,Monterey, CA, June 2002.

[19] S. Savage and H. Tokuda. Rt-mach timers: Export-ing time to the user. In In Proceedings of USENIX3rd Mach Symposium, April 1993.

[20] Lui Sha, Raghunathan Rajkumar, and JohnLehoczky. Priority inheritance protocols: An ap-proach to real-time synchronization. IEEE Trans-actions on Computers, 39(9):1175–1184, Septem-ber 1990.

[21] B. Srinivasan, S. Pather, R. Hill, F. Ansari, andD. Niehaus. A firm real-time system imple-mentation using commercial off-the-shelf hardwareand free software. In Proceedings of the IEEEReal-Time Technology and Applications Sympo-sium, 1998.

[22] David Steere, Ashvin Goel, Joshua Gruenberg, Dy-lan McNamee, Calton Pu, and Jonathan Walpole.A Feedback-driven Proportion Allocator for Real-Rate Scheduling. In Proceedings of the ThirdUSENIX Symposium on Operating Systems Designand Implementation. USENIX, February 1999.

[23] Karim Yaghmour and Michel R. Dagenais. Mea-suring and characterizing system behavior usingkernel-level event logging. In USENIX AnnualConference, San Diego, CA, June 2000.

[24] Yu-Chung and Kwei-Jay Lin. Enhancing the Real-Time Capability of the Linux Kernel. In Proceed-ings of the IEEE Real Time Computing Systems andApplications, Hiroshima, Japan, October 1998.

14