ee514 – real-time computing 1 real-time linux department of electrical engineering the hong kong...
TRANSCRIPT
EE514 – Real-Time Computing 1
EE514 – Real-Time Computing
Real-Time Linux
Department of Electrical Engineering
The Hong Kong Polytechnic University
EE514 – Real-Time Computing
Real Time Linux Operating Systems
Swaminathan SivasubramanianDependable Computing & Networking Laboratory
Iowa State University
[email protected]@iastate.edu
EE514 – Real-Time Computing
Linux as a Real-Time OS
Traditional RT Systems used custom built systems – which were not extensible I.e. tough to develop new applications
However, as technology improved, generic real-time OS became acceptable
In OS suited for extensible development Linux looks more appealing
3
EE514 – Real-Time Computing
•Why Linux?• Linux (and its Real Time versions) are
free!!• Linux (and its Real Time versions) are
Open Source!!• Easy for developing RT applications
4
EE514 – Real-Time Computing
Why not Linux?
Linux didn’t have any corporate support until now
Linux, is a very good general purpose operating system, but not so for real-time OS (Why??)
Because, the design motive of a conventional OS and RTOS is different
5
EE514 – Real-Time Computing
• Linux as your real-time solution?
• Could increase priority for “real-time” tasks and assume they get scheduled
• Problem – Linux optimizes average case whereas an RTOS should work under worst case assumptions
6
EE514 – Real-Time Computing
Linux – A Simplified View
7
EE514 – Real-Time Computing
Linux – conflicts with RT constraints Coarse grained synchronization – long intervals when a
task has exclusive use of data ( as fine grained – leads to lot of overhead reducing the average case performance)
Linux batch all operations for efficient use of H/W E.g freeing a list of pages when memory is full
reducing the worst case performance Linux doesn't preempt low-priority task during system call Linux will make high priority tasks wait for low priority to
release resources
8
EE514 – Real-Time Computing
Real Time Linux approaches
Modify the current Linux kernel to guarantee RT constraints Used by KURT
Make the standard Linux kernel run as a task of the real-time kernel Used by RT-Linux, RTAI
9
EE514 – Real-Time Computing
Modifying Linux kernel
Advantages Most problems, such as interrupt handling,
already solved Less initial labor
Disadvantages No guaranteed performance RT tasks don’t always have precedence over
non-RT tasks.
10
EE514 – Real-Time Computing
Running Linux as a process of a second RT kernel Advantages
Can make hard real time guarantees Easy to implement a new scheduler
Disadvantages Initial port difficult, must know a tremendous
amount about underlying hardware Running a small real-time executive is not a
substitute for a full fledged RTOS
11
EE514 – Real-Time Computing
RTAI Overview
Based on Real-Time Hardware Abstraction Layer (HAL) (also used in Windows NT)
HAL exports Linux data and functions related to hardware
HAL defines an interface between RTAI and Linux
The Windows NT hardware abstraction layer (HAL) refers to a layer of software that deals directly with your computer hardware.
12
EE514 – Real-Time Computing
RTAI – A Simplified View
13
EE514 – Real-Time Computing
RTAI Overview (continued)
Software architecture Interface to Linux hardware management
(HAL) 3 basic components: dispatcher, scheduler,
fifos 1 interface used in user tasks to initialize and
start the components RTAI is basically an interrupt dispatcher
(reroutes interrupts to Linux if necessary) e.g: Disk interrupt
14
EE514 – Real-Time Computing
RT-Linux Overview
Open source Linux project Supports x86, PowerPC, Alpha Patch of the regular Linux kernel (simply
install the patch and recompile the kernel) Provides an RT API for developers Runs Linux kernel as lowest priority
process
15
EE514 – Real-Time Computing
•RT-Linux Task Structure
16
EE514 – Real-Time Computing
•RT-Linux Interrupt Dispatcher
17
EE514 – Real-Time Computing
RT-Linux Overview (continued)
RT tasks are coded as modules Modules are inserted and removed at
users discretion Extremely good at handling periodic tasks Communicates with non-RT kernel and
other RT tasks via fifo queues Tools are provided for graphical analysis of
RT execution
18
EE514 – Real-Time Computing
Problems with RT-Linux
Currently no support for aperiodic tasks Not very useful for complex RT systems Currently limited to simple problems
19
EE514 – Real-Time Computing
Montavista – A Faster Response Linux Kernel Montavista Inc. provides a linux solution for
embedded systems The solution’s aim is to make the Linux
kernel fully preemptable It identifies the points where priority
inversion occurs in Linux and makes those points fully preemptable
A good embedded solution not a complete RT solution.
20
EE514 – Real-Time Computing 21
Real-Time Linux A class of Linux systems which could meet the real-time
requirements.
There are a number of such real-time Linux systems available:
RTLinux – hard real-time
RTAI – hard real-time
MontaVista – soft real-time
and many more …
Many of them are developed as research projects and were released to public freely.
Some of them are commercialized though.
EE514 – Real-Time Computing 22
RTLinux or RTCore The first real-time Linux system developed as a research
project over a decade ago.
Subsequently commercialized to become a real-time core (RTCore) of a commercial Linux called Wind River Linux.
RTLinux or RTCore is a hard real-time microkernel that runs the entire Linux operating system as a fully preemptable process.
The technologies developed to enable the real-time operation are patented and can’t be used freely.
EE514 – Real-Time Computing 23
RTAI RTAI stands for Real-Time Application Interface and is a
Linux extension to hard real-time application.
It is initially developed as a research project in Italy for enhancing the RTLinux functionalities at that time.
Subsequently, new technologies were invented to replace all the RTLinux code adopted from the initial RTLinux prototype to overcome any patent problems.
The new technologies is collectively referred as ADEOS.
Current version is 3.8.1 released in May 2010.
Still actively under development and maintained by the original developers in Italy and the Internal community.
EE514 – Real-Time Computing 24
Beginning of Linux – as a UNIX clone
EE514 – Real-Time Computing 25
Linux System Linux kernel
In Sept 1991, Linus Torvalds, a second year student of Computer Science at the University of Helsinki, developed the preliminary kernel of Linux, known as Linux version 0.0.1, and released to the Internet.
Version 1.0, 2.0, 2.2, 2.4, and 2.6 were released in 1994, 1996, 1999, 2001, and 2003, respectively.
Current version is 2.6.37.2 released on 24 Feb 2011.
Linux system Software from the GNU project + Linux kernel = Linux system
Redhat, Fedora, CentOS, Ubuntu, etc. are popular variants of Linux distributions.
EE514 – Real-Time Computing 26
Evolution of Linux Early Linux is Not Designed for Real-Time Processing
Early Linux (1.x Kernel) installations on retired Windows PCs Old/Obsolete hardware useful under Linux due to efficiency of O/S
Linux outperformed Windows in reliability and uptime (still does)
Linux Design: Fairness, Throughput and Resource-Sharing Basic Unix development design principles applied in Kernel
Heavily (over)-loaded systems continue to make progress
Does not drop network connections or starve users / applications
Fairness- and Resource-Sharing Design is Linux's Strength contributed to make Linux competitive and popular in the enterprise-
server and development-application environments
Gave rise to RedHat and others.
Essential to the evolution of Linux, endemic of UNIX legacy
EE514 – Real-Time Computing 27
Why Linux in Real-Time Systems? Not because of the Kernel’s Real-Time Performance!
UNIX-legacy Operating Systems were designed with operating principles focused on throughput and progress
User tasks should not stall under heavy load System resources must be shared fairly between users
Fairness, progress and resource-sharing conflict with the requirements of time-critical applications
VIP vs. General Admission
UNIX systems (and Linux) are historically not Real-Time OS
Linux has lagged many commercial Unix's in Real-Time performance-enhancement and Real-Time capabilities
Solaris, LynxOS, QNX SCO
EE514 – Real-Time Computing
28
Real time computing requirements
Scheduling Hardware Interface Communication and
Synchronization Memory Management Task Management
Timing Configuration External
Communications Error Reporting Embedded Features File system Reliability
EE514 – Real-Time Computing 29
Why Real-Time in Linux Systems (Embedded)
Most Important Factors Inhibiting Linux Adoption
Data from VDC, “Linux’s Future in the Embedded Systems Market”, June 2004
EE514 – Real-Time Computing 30
Real-Time in Handheld & Embedded Systems
Cost / Performance / Power / Weight Compromise Competitive, High-Volume, Low-margin Markets
Maximum Feature-set, Add-ons, Responsive UI feel
Device specs: minimal CPU & Memory & Battery Powered
Minimal CPU = High CPU utilization
High CPU load + Time-Critical functionality RT specs
Real-time Requirements will never be alleviated by Improvements in Hardware Performance / Efficiency
Software utilizing latest hardware technologies easily keep up with, and usually out-paces, advances in hardware technology
If you don't believe that, go shopping (for a mobile phone)
EE514 – Real-Time Computing 31
Real-Time Linux 2.6 Enablers Pro-Audio Performance Requirements
Audio Community Involved in Kernel-Preemption since 2.2 Audio Community strongly Endorsing RT technology
Embedded Application Domain Single-Chip, Mobile Applications (Wireless / Cellular Handsets) Predictable OS performance eliminates HW design uncertainty
Reliable Prototyping and Improved Product Scheduling
Multimedia Carrier (QOS) Application Domain Telephony, Audio / Video / Multimedia / Home Entertainment
Fine-Granular Preemption improves SMP scalability Mainstreaming of SMP Technology
Dual / Quad / Octa - Core Intel, AMD, PPC, Arm
EE514 – Real-Time Computing 32
Real-Time and Linux Kernel Evolution Gradual Kernel Optimizations over Time
SMP Critical sections (Linux 2.x) Low-Latency Patches (Linux 2.2) Preemption Points / Kernel Tuning (Linux 2.2 / 2.4) Preemptible Kernel Patches (Linux 2.4) Fixed-time “O(1)” Scheduler (Linux 2.6) Voluntary Preemption (Linux 2.6)
In 2003-04 Linux 2.6 RT Technology Regressed Early Linux 2.6 Real-Time Performance was worse than 2.4
Kernel Performance Audio Community and others balked at moving to 2.6 Kernel
Base
EE514 – Real-Time Computing 33
Real-Time Inhibitor: Critical Section Locking
Linux 2.6 Kernel Critical Sections are Non-Preemptible Critical sections protect shared resources, e.g. hardware
registers, I/O ports, and data in RAM Critical sections are shared by Processes, Interrupts and CPUs. Effective protection is provided by the Spin-Lock Subsystem Critical sections must be locked and unlocked Locked critical sections are not preemptible Linux 2.6 Kernel has 11,000 critical sections Exhaustive Kernel testing to identify worst-case code paths Labour-intensive cleanup of critical sections No control over 3rd party drivers Worst-case after cleanup still not acceptable Maintenance, community education, policing / regression testing
EE514 – Real-Time Computing 34
Real-Time Inhibitor: Interrupt Handlers Linux 2.6 Kernel: Unbounded IRQ subsystem latencies
Task-Preemption latency increases with hardware-interrupt load
Interrupts cannot be preempted
No Priorities for Interrupts IRQ Subsystem always preempts tasks unconditionally
Unbounded SoftIRQ subsystem (“Bottom Half Processing”) Activated by HW IRQs (Timers, SCSI, Network)
SoftIRQs re-activate, iterate
Driver-level adaptations Network Driver NAPI adaption reduces denial of service (D.o.S.)
effects of high packet loads
EE514 – Real-Time Computing 35
Real-Time Inhibitor: Legacy Locking Existing Locking Subsystems are not Priority-Aware
System semaphore Counting semaphore used to wake multiple waiting tasks No support for priority inheritance No priority ordering of waiters
Big Kernel Lock (BKL) Originally non-preemptible, now preemptible using system
semaphore Can be released by blocking tasks, re-acquired upon wake-up No priority-awareness, or priority inheritance for contending tasks
RCU (Read-Copy-Update) Locks in Network subsystem Read-optimized cached locking requiring race-free invalidation
Read – Write Locks Classical blocking / starvation issues with no priority awareness
EE514 – Real-Time Computing 36
The Fully Preemptible Linux Kernel Dramatic Reduction in 2.6 Preemption Latencies
Multiple Concurrent Tasks in Independent Critical Sections Generally Fully Preemptible “No Delays”
Non-preemptible: Interrupt off paths and lowest-level interrupt management Non-preemptible: Scheduling and context switching code
Design Flexibility Provides Full Access to Kernel Resources to RT Tasks Supports existing driver and application code User-space Real-Time
Optimization Flexibility RT Tasks designed to use Kernel-resources in managed ways
can reduce or eliminate Priority-Inheritance delays
Adequate Instrumentation Latency timing, latency triggers & stack tracing, histograms
EE514 – Real-Time Computing 37
Kernel Evolution: Preemptible Code
Kernels 2.2-2.4
PreemptibleKernel 2.4Kernel 2.6
Preemptible Non-Preemptible
Kernel 2.0
Real-Time Kernel 2.6
EE514 – Real-Time Computing 38
Real-Time Linux 2.6 Performance Real-Time Linux 2.6 Kernel Performance
Far exceeds most stringent Audio performance requirements
Enables sub-millisecond control-loop response
Enables Hard Real Time for qualified RT-aware Applications
SMP Kernel Performance
SMP-safe code is by definition preemptible
Any code that allows concurrent execution by multiple CPUs, also allows context switching and therefore preemption
Increased preemptible code surface in the Kernel also increases SMP throughput / efficiency
EE514 – Real-Time Computing 39
Real-Time Linux 2.6 Performance (Cont)
No Preemption Preemptible
Real-Time Preemption
MontaVista Linux 4.0 Target machine:
Intel® Celeron® 800 MHz CPU utilization during test:
100% most of the time Test Duration:
20 hours
EE514 – Real-Time Computing 40
RTLinux / RTAI Architecture
Hardware abstraction layer real-time interrupt dispatcher, real-time scheduler Inter processes communication services
Real-time plugin
System libraries
Linux kernelDrivers
I/O
I/O
Hardware interrupts
Software interrupts
Real Time task
User’s task
Real-time Scheduler
HARDWARE
EE514 – Real-Time Computing 41
Interrupt Latency The time stamp is frozen when the interrupt is generated The current time is read when the interrupt is
acknowledged The difference gives the interrupt latency
Dual-Xeon 3.2GHz
Dual-core Opteron 2GHz
EE514 – Real-Time Computing
Freescale – dual-kernel, real-time solutions (BSP) for Linux (Case studies) Solutions provide sub-20 microsecond precision with
hard, real-time scheduling guarantees but at the expense of increased programming effort and memory protection
These techniques offer less than 250 µsec latency performance for greater than 99.5 percent of all scheduled task under demanding system test loads
Under real-world conditions, performance is near the 99.5 percent, 100 µsec mark
BSP : board support service
42
EE514 – Real-Time Computing
Linux kernel improvements for real time
Features and performance enhancement for upgrading standard Linux kernel to challenge real time products and applications: Enhanced schedulers Virtual memory Shared memory Portable operating system interface X (POSIX) timers Real-time signals POSIX asynchronous I/O POSIX threads Quality of service capabilities Low latency/preemptable kernel modifications
43
EE514 – Real-Time Computing
Quality of sercice
As for quality of service, Linux v2.4 offers a sophisticated component for bandwidth management called Traffic Control. The component supports methods for classifying, prioritizing and limiting both incoming and outgoing traffic. Linux can perform: Limit bandwidth for certain computers Help fairly share bandwidth Protect the Internet from abuses Restrict access and do routing based on user ID, MAC
address, source IP address and so on
44
EE514 – Real-Time Computing
Linux network subsystem (Traffic control)
45
EE514 – Real-Time Computing
Queueing discipline- traffic control
Queueing discipline provides a method to enqueue a packet
46
EE514 – Real-Time Computing
Low-Latency patches for Linux
The kernel latency is a quantity used to measure the difference between the theoretical schedule and the actual one
In a standard Linux kernel, the maximum latency is equal to the maximum length of a system call plus the processing time of all the interrupts that fire before returning to user mode. This value can be as large as 100 ms
47
EE514 – Real-Time Computing
Low-Latency Linux (Red-Linux, some Real-time Linux) This approach corrects the monolithic structure by
inserting explicit rescheduling points inside the kernel. In this approach, when a thread is executing inside the kernel it can explicitly decide to yield the CPU to some other thread
In this way, the size of non-preemptable sections is reduced, thus decreasing the latency
The consistency of kernel data is enforced by suing cooperative scheduling
Since the low-latency patch has been carefully hand-tuned for quite a long time, it performs surprisingly well
48
EE514 – Real-Time Computing
Preemptable Linux (used in most real-time systems) Removes the constraint of a single execution flow
inside the kernel. It is not necessary to disable preemption when an execution flow enters the kernel
To support full kernel preemptability, kernel data must be explicitly protected using mutexes or spinlocks
The Linux preemptable kernel pathc, maintained by Robert Love and sponsored by MontaVista
49
EE514 – Real-Time Computing
Evaluation
50
EE514 – Real-Time Computing
Probability density function of the latency in the low-latency kernel
51
EE514 – Real-Time Computing
Summary of the Freescale case studies Linux is rapidly evolving to include more and more
features that have explicit value to the embedded developer
Key enhancements to the Linux kernel, which perform identically for all practical purposes
The developer’s choice regarding which should be implemented can be based on independent criteria including the ability to update/modify existing device drivers for compatibility with the real-time implementation, and the availability of each real-time technique for the selected architecture and kernel version
52
EE514 – Real-Time Computing
References
RT-Linux : http://www.rtlinux.org RTAI :
http://www.aero.polimi.it/projects/rtai/contrib.htm
Montavista: http://www.mvista.com Linux as a real-time operating system –
Freescale semiconductor, David Beal, Nov/2005
53
EE514 – Real-Time Computing Real Time Linux 54
EE514 – Real-Time Computing Real Time Linux 55