Adaptive QoS Control ���for Real-Time Systems

Chenyang Lu CSE 520S

Challenges for Real-Time Systems

Ø  Classical real-time scheduling theory relies on accurate knowledge about workload and platform.

New challenges under uncertainties Ø  Maintain robust real-time properties in face of

q  unknown and varying workload

q  system failure q  system upgrade

Ø  Certification & testing of real-time properties of adaptive systems

2  

Challenge 1: Workload Uncertainties

Ø  Task execution times q  Heavily influenced by sensor data or user input

q  Unknown and time-varying

Ø  Disturbances q  Aperiodic events

q  Resource contention from subsystems

q  Denial of Service attacks

Ø  Examples: SCADA for power grid management, total ship computing environment

3  

Challenge 2: System Failure

Ø  Only maintaining functional reliability is not sufficient. " Must also maintain robust real-time properties!

4  

1.  Norbert fails. 2.  Move its tasks to other

processors. "   hermione & harry are

overloaded!

Challenge 3: System Upgrade

Ø  Goal: Portable application across HW/OS platforms q  Same application “works” on multiple platforms

Ø  Existing real-time middleware ü  Support functional portability

û  Lack QoS portability: must manually reconfigure applications on different platforms to achieve desired QoS •  Profile execution times •  Determine/implement allocation and task rate •  Test/analyze schedulability

"   Time-consuming and expensive!

5  

Example: nORB Middleware

6  

Timer thread Worker

thread

Server

…

… Conn. thread

…

…

… …

…

… Priority queues nO

RB

* Ap

plic

atio

n" CORBA Objects

Client

T1: 2 Hz T2: 12 Hz

Manually set offline Conn.

thread

Operation Request Lanes

Challenge 4: Certification

Ø  Uncertainties call for adaptive solutions. But…

Ø  Adaptation can make things worse. Ø  Adaptive systems are difficult to test and certify

7  

0

0.2

0.4

0.6

0.8

1

0 100 200 300

Time (sampling period)

CPU

util

izat

ion

P1 P2 Set Point

An unstable adaptive system

Adaptive QoS Control

Ø  Develop software feedback control in middleware

q  Achieve robust real-time properties for many applications

Ø  Apply control theory to design and analyze control algorithms

q  Facilitate certification of embedded software

8  

Available resources? HW failure?!

Drivers/OS/HW?!

Applications!

Sensor/human input? Disturbance?!

Adaptive QoS Control Middleware! Maintain QoS guarantees •  w/o accurate knowledge about workload/platform •  w/o hand tuning

Adaptive QoS Control Middleware

Ø  FCS/nORB: Single server control Ø  FC-ORB: Distributed systems with end-to-end tasks

9  

Feedback Control Real-Time Scheduling

Ø  Developers specify q  Performance specs

•  CPU utilization = 70%; Deadline miss ratio = 1%.

q  Tunable parameters •  Range of task rate: digital control loop, video/data display •  Quality levels: image quality, filters •  Admission control

Ø  FCS guarantees specs by tuning parameters based on feedbacks online

q  Automatic: No need for hand tuning

q  Transparent from developers à  Performance Portability!

10  

A Feedback Control Loop

11  

FC-U!

Monitor!

HW?!

Drivers/OS?!

Application?!

Sensors, Inputs!

Specs!Us = 70%!!!Parameters!R1: [1, 5] Hz!R2: [10, 20] Hz!

Middleware!

Actuator!Controller!

U(k)!

{Ri(k+1)}!

The FC-U Algorithm

Us: utilization reference

Ku: control parameter Ri(0): initial rate

1.  Get utilization U(k) from Utilization Monitor.

2.  Utilization Controller: ���B(k+1) = B(k)+ Ku*(Us–U(k)) /* Integral Controller */

3.  Rate Actuator adjusts task rates

Ri(k+1) = (B(k+1)/B(0))Ri(0) 4.  Inform clients of new task rates.

12  

The Family of FCS Algorithms

Ø  FC-U controls utilization q  Performance spec: U(k) = Us ü  Meet all deadlines if Us ≤ schedulable utilization bound û  Relatively low utilization if utilization bound is pessimistic

Ø  FC-M controls miss ratio q  Performance spec: M(k) = Ms ü  High utilization ü  Does not require utilization bound to be known a priori

û  Small but non-zero deadline miss ratio: M(k) > 0

Ø  FC-UM combines FC-U and FC-M q  Performance specs: Us, Ms ü  Allow higher utilization than FC-U ü  No deadline misses in “nominal” case q  Performance bounded by FC-M

13  

Control Analysis

Ø  Rigorously designed based on feedback control theory Ø  Analytic guarantees on

q  Stability

q  Steady state performance q  Transient state: settling time and overshoot

q  Robustness against variation in execution time

Ø  Do not assume accurate knowledge of execution time

Ø  Lu, Stankovic, Tao, and Son, Feedback Control Real-Time Scheduling: Framework, Modeling, and Algorithms, Real-Time Systems, 23(1/2), July/September 2002.

14  

Dynamic Response

15  

Settling time!

Controlled"variable"

Time "

Reference"""""""

Steady State!Transient State!

Steady state error!

Stability!

FCS/nORB Architecture

16  

Timer thread worker

thread

conn. thread

Server

…

…

util monitor

miss monitor

controller

rate assigner

conn. thread

…

…

rate modulator

feedback lane"… …

…

Operation Request Lanes"

… Priority Queues FC

S/nO

RB  

Applica:

on   CORBA  Objects  

Client

Implementation

Ø  Running on top of COTS Linux

Ø  Deadline Miss Monitor q  Instrument operation request lanes

q  Time-stamp operation request and response on each lane

Ø  CPU Utilization Monitor q  Interface with Linux /proc/stat file

q  Count idle time: “Coarse” granularity: jiffy (10 ms)

Ø  Only controls server delay

17  

Offline or Online?

Ø  Offline q  FCS executed in testing phase on a new platform

q  Turned off after entering steady state ü  No run-time overhead

û  Cannot deal with varying workload

Ø  Online û  Run-time overhead (actually small…)

ü  Robustness in face of changing execution times

18  

Set-up

Ø  OS: Redhat Linux Ø  Hardware platform

q  Server A: 1.8GHz Celeron, 512 MB RAM q  Server B: 1.99GHz Pentium 4, 256 MB RAM q  Same client q  Connected via 100 Mbps LAN

Ø  Experiment 1.  Overhead 2.  Steady execution time (offline case) 3.  Varying execution time (on-line case)

19  

Server Overhead

20  

• Overhead:  FC-­‐UM  >  FC-­‐M  >  FC-­‐U  • FC-­‐UM  increases  CPU  u:liza:on  by  <1%  for  a  4s  sampling  period.  

Server Overhead per Sampling Period

0510152025303540

FC-U FC-M FC-UM

Ove

rhea

d (m

s)

Sampling Period = 4 sec

Performance Portability���Steady Execution Time

21  

0.000.100.200.300.400.500.600.700.800.901.00

0 25 50 75 100 125 150 175 200

Time (4 sec)

U(k)

B(k)

M(k)

FC-U on Server A!1.8GHz Celeron, 512 MB RAM"

0.000.100.200.300.400.500.600.700.800.901.00

0 25 50 75 100 125 150 175 200

Time (4 sec)

U(k)

B(k)

M(k)

•  Same  CPU  u:liza:on  (and  no  deadline  miss)  on  different  plaTorms  w/o  hand-­‐tuning!  

FC-U on Server B!1.99GHz Pentium 4, 256 MB RAM"

Us = 70%

Steady-state Deadline Miss Ratio ���Server A

Average Deadline Miss Ratio in Steay State

1.49

0.00

0.50

1.00

1.50

2.00

FC-U FC-M FC-UM

%

22  

Ms = 1.5%

•  FC-­‐M  enforces  miss  ra:o  spec  •  FC-­‐U,  FC-­‐UM  causes  no  deadline  misses  

Steady-State CPU Utilization ���Server A

Average CPU Utilization in Steady State

70.01

98.93

74.97

0

20

40

60

80

100

FC-U FC-M FC-UM

%

23  

Us = 70% Us = 75%

•  FC-­‐U,  FC-­‐UM  enforces  u:liza:on  spec  •  FC-­‐M  achieves  higher  u:liza:on  

Robust Guarantees���Varying Execution Time

24  

0.000.100.200.300.400.500.600.700.800.901.00

0 50 100 150 200 250 300 350 400

Time (4 sec)

U(k)

B(k)

M(k)

Same  CPU  u:liza:on  and  no  deadline  miss  in  steady  state  despite  changes  in  execu:on  :mes!  

Tolerance to Load Increase

Ø  Surprise: server crashes under FC-M when execution time increases q  FCS/nORB threads run at real-time priority q  Kernel starvation when CPU utilization reaches 100%

Ø  Tolerance margin of load increase q  FC-U, FC-UM: margin = 1/Us-1

•  Us=70% à Server can tolerate (1/0.7-1)=43% increase in execution time q  FC-M: small and “unknown” margin

•  Inappropriate middleware-level service when execution time can increase unexpectedly

25  

Summary of Experimental Results

Ø  FCS algorithms enforces specified CPU utilization or miss ratio in steady state q  Experimental validation of control design and analysis of FCS

Ø  Performance Portability: FCS/nORB achieves the same performance guarantee when q  platform changes

q  execution time changes (within tolerance margin)

Ø  Overhead acceptable à FCS can be used online

26  

Summary: FCS/nORB

Ø  FCS/nORB supports robust, performance-portable real-time software q  Program application once à runs on multiple platforms with robust

performance guarantees!

q  FCS/nORB 1.0 release: http://deuce.doc.wustl.edu/FCS_nORB

Ø  Next: FC-ORB q  Handle end-to-end tasks q  Fault tolerance

27  

References Ø  C. Lu, X. Wang and C.D. Gill, Feedback Control Real-Time Scheduling in

ORB Middleware, IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), May 2003.

Ø  C. Lu, J.A. Stankovic, G. Tao, and S.H. Son, Feedback Control Real-Time Scheduling: Framework, Modeling, and Algorithms, Real-Time Systems, Special Issue on Control-theoretical Approaches to Real-Time Computing, 23(1/2): 85-126, July/September 2002.

28