Flexible Scheduling in Middleware for Distributed Rate-Based Real-Time

Applications

Christopher D. GillDissertation Supervisors: Dr. Ron K. Cytron, Dr. Douglas C. Schmidt

Department of Computer ScienceWashington University, St. Louis, MO

cdgill@cs.wustl.edu

Tuesday, December 18, 2001

Historical Challenges

Motivation for Studying Adaptive MiddlewareTrends

•Many mission-critical distributed applications require real-time QoS guarantees– e.g., combat systems, online trading, telecom

•Building QoS-enabled applications manually is tedious, error-prone, & expensive

•Conventional middleware does not support real-time QoS requirements effectively

•Building distributed systems is hard•Building them on-time & under budget is even harder

•Hardware keeps getting smaller, faster, & cheaper

1 1Proxy

service

Service

service

AbstractService

service

Client

•Software keeps getting larger, slower, & more expensive

New Challenges

Overview of Research AreasTechnical Challenge

Research Approach

ResearchImpact

Efficient and Safe Systems

Customizable Middleware

Nimble Adaptation

Performance Awareness

Inclusive Systems Approach

These technical challenges raise crucial systems issues in both theoretical and empirical dimensions

These technical challenges raise crucial systems issues in both theoretical and empirical dimensions

Motivating Applications

Boeing Bold Stroke Middleware Infrastructure Platform

•Operations well defined

•Event-mediated middleware solution

•Previous-generation systems static

•Next-generation systems dynamic and adaptive

Domain CompanyAvionics mission computing Boeing, RaytheonMass storage devices SUTMYN, StorTekMedical Information Systems Siemens, GESatellite Control LMCO COMSATTelecommunications Motorola, Lucent, Nortel,

Cisco, SiemensMissile & Radar Systems LMCO Sanders, RaytheonSteel Manufacturing Siemens ATDBeverage Bottling Automation Krones AG

Limitations With Existing Approaches

Historically, each application has solved scheduling on its own

• Tedious, error-prone• Costly over system lifecycles• Single-paradigm approaches

Current middleware lacks hooks for key domain-specific features, e.g.:• Optimized integration with higher level managers• Hybrid static-dynamic scheduling strategies• Strategies built from primitive elements• Adaptive domain-specific optimizations

Research Contributions

Systems Architecture and Framework• Extends open-source middleware

Patterns for Adaptive Scheduling• Capture design experience and solutions

Empirical Evaluation of Adaptive Scheduling• Practical benefits for real-world applications

Research Approach: the Kokyu Flexible Middleware Scheduling/Dispatching Framework

Application specifies characteristics• e.g., criticality, periods, dependencies

Application specifies characteristics• e.g., criticality, periods, dependencies

Dispatcher is (re)configurable• Multiple priority lanes• Queue, thread, timers per lane• Starts repetitive timers once• Looks up lane on each arrival

Dispatcher is (re)configurable• Multiple priority lanes• Queue, thread, timers per lane• Starts repetitive timers once• Looks up lane on each arrival

Scheduler

sub-graph

ratetuples

WCET propagation

selectedrates

rate propagation

propagatedrates

tuplevisitor

operationvisitors Rate and priority

assignment policy

DispatcherDispatching configuration

RMS

LLF

mandatory

optionallaxity

static

static

timers

Scheduler assigns rates & priorities per topology, scheduling policy

• Defines necessary dispatch configuration

Scheduler assigns rates & priorities per topology, scheduling policy

• Defines necessary dispatch configuration

Implicit projection• Of specific scheduling policy into

generic dispatch infrastructure

Implicit projection• Of specific scheduling policy into

generic dispatch infrastructure

Greater Utilization with Criticality Isolation

Technical Challenge

Research Approach

ResearchImpact

Efficient and Safe Systems

Arbitrary strategies that hybridize static/dynamic scheduling/dispatching

Increased utilization, critical operations still meet their deadlines

Customizable Middleware

Nimble Adaptation

Performance Awareness

Inclusive Systems Approach

Safety: Meet Critical Deadlines

Static scheduling cannot isolate critical operation deadlines in overload

Dynamic Static

same point in execution

Solution: Support for Hybridizing Static & Dynamic Scheduling Heuristics

Abstract mapping: operation characteristics OS & middleware primitives

• Generalizes RMS, EDF, LLF, MUF, RMS+LLF, RMS+EDF, …

• Arbitrary composition of primitives• Drives factory-driven dispatching

module (re)configuration at run-time (work in progress)

DispatcherDispatching configuration

laxity

laxity

critical non-critical

MUF

No One Strategy is Optimal

Technical Challenge

Research Approach

ResearchImpact

Efficient and Safe Systems

Arbitrary strategies that hybridize static/dynamic scheduling/dispatching

Increased utilization, critical operations still meet their deadlines

Customizable Middleware Dispatching composed from primitive elements

Supports tailored “fit” of scheduling/dispatching

Nimble Adaptation

Performance Awareness

Inclusive Systems Approach

Case for Multi-Paradigm Scheduling

Dynamic1Dynamic2Static

)

Solution: Compose of Scheduling Heuristics from Dispatching Primitives

Dispatcher

laxity

laxity

mandatory optional

Gives Fine Grain Control over Feasibility / Performance Trade-Off

laxity

staticstatic

static

staticfeasibility

performance

MUF RMS+LLF

Improving the Bound on Adaptive Rescheduling

Technical Challenge

Research Approach

ResearchImpact

Efficient and Safe Systems

Arbitrary strategies that hybridize static/dynamic scheduling/dispatching

Increased utilization, critical operations still meet their deadlines

Customizable Middleware Dispatching composed from primitive elements

Supports tailored “fit” of scheduling/dispatching

Nimble Adaptation Integrated rate/priority selection mechanisms

O(n2)/O(n log n)

O(n log n)/O(n) bound on adaptation

Performance Awareness

Inclusive Systems Approach

Benefits of Resource Adaptation in Middleware

Adaptation Preserves Feasibility in Changing Environment• Skillfully done, it also improves performance

Middleware Offers Portable, Robust Solutions• Can add new managers to a system, E.g., QuO, RTARM

However, a new Manager (e.g., RTARM) may be a Black Box• Determines the number of scheduler calls• Determines the inputs to each call

Need Optimizations to Ensure Nimble Adaptation

Adaptation Time (RTARM as a Black Box)

n = # target region operationsO(log n) or O(n) fits measured time / callNumber of calls is O(n)Time to adapt thus O(n log n) or O(n2)Either way, we can do better

Solution: Integrated Rate & Priority Selection

Skillfully integrate rate and priority mechanisms

At least as good overall: O(log n) per operation: comparison sort

Better in special cases: O(1) per operation: radix sort

Retains Modular Policy while Supporting Optimization

Scheduler

sub-graph

ratetuples

WCET propagation

selectedrates

rate propagation

propagatedrates

tuplevisitor

operationvisitors Rate and priority

assignment policy

Unobtrusive Monitoring and Control FeedbackTechnical Challenge

Research Approach

ResearchImpact

Efficient and Safe Systems

Arbitrary strategies that hybridize static/dynamic scheduling/dispatching

Increased utilization, critical operations still meet their deadlines

Customizable Middleware Dispatching composed from primitive elements

Supports tailored “fit” of scheduling/dispatching

Nimble Adaptation Integrated rate/priority selection mechanisms

O(n2)/O(n log n)

O(n log n)/O(n) bound on adaptation

Performance Awareness Time and space efficient data collection and storage framework

Provides run-time observable info for control, post-analysis

Inclusive Systems Approach

Solution: Kokyu Real-Time Metrics Monitoring Framework

EM

BE

DD

ED

BO

AR

DS

RE

MO

TE

WO

RK

ST

AT

ION

SH

AR

ED

ME

MO

RY

METRICS CACHE

REMOTE LOGGER

METRICS MONITOR

RTARM

DOVEBrowser (Java)

STO

RA

GE

QuO

OPERATIONS

PRO

BE

S

DISPATCHER

PROBES

FRAME MANAGER

Feedback to Higher-LevelResource Managers

Logging and Visualization

Customized Data Collection

Consistent View of Time

Co-Scheduling Control Elements with ApplicationTechnical Challenge

Research Approach

ResearchImpact

Efficient and Safe Systems

Arbitrary strategies that hybridize static/dynamic scheduling/dispatching

Increased utilization, critical operations still meet their deadlines

Customizable Middleware Dispatching composed from primitive elements

Supports tailored “fit” of scheduling/dispatching

Nimble Adaptation Integrated rate/priority selection mechanisms

O(n2)/O(n log n)

O(n log n)/O(n) bound on adaptation

Performance Awareness Time and space efficient data collection and storage framework

Provides run-time observable info for control, post-analysis

Inclusive Systems Approach

Decision lattice joining a priori analysis with empirical measurement

Towards run-time reflective and adaptive policy selection

Inclusive Systems Approach

Dispatcher

low

medium

high

App! App?Control! Control?

Goal: co-scheduling of control elements with the application

Problem: separate treatment of control elements and application

Solution: Use Empirical & A Priori Information to Co-Schedule Resource Mgrs & Applications

Preserve Assurances butOptimize Performance

DispatcherLLF {App!, Control!,?, App?}

RMS {App!, Control!,?, App?}

RMS {App!, Control!,?}LLF {App?l}

RMS {App!, Control!}LLF {Control?, App?}

RMS {App!}LLF {Control!,?, App?}

LLF {App!, Control!,?}LLF {App?}

LLF {App!, Control!}LLF {Control?, App?}

LLF {App!}LLF {Control!,?, App?}

feasibility

performance

feas

ibil

ity

per

form

ance

feas

ibil

ity

per

form

ance

feas

ibil

ity

per

form

ance

feas

ibil

ity

per

form

ance

feas

ibil

ity

per

form

ance

feas

ibil

ity

per

form

ance

feasibility

performance

feasibility

performance

feasibility

performance

Towards an Adaptive Control Decision Lattice

Experimental Test-BedApplication• Research Version of Operational Flight Program for AV-8B Aircraft• Added navigation route computations to ramp non-critical load• Added critical and non-critical computations to inject execution time jitter

Middleware• The ACE ORB (TAO)• TAO Real-Time Event Channel• Kokyu Framework: Scheduling, Dispatching, and Metrics

Operating System and Hardware• VxWorks RTOS on the PPC boards• 200 MHz Motorola PPC 604 card• two 100 MHz Dy4-177 PPC 603 cards• Dy4-783 memory mapped display processor• Commercial VME-64 chassis with all boards• Switched Ethernet, MIL-STD-1553 MUX Bus on one Dy4-177 card

Popular Scheduling Strategies

Rate Monotonic Scheduling (RMS)• Assigns thread priorities by rate• Operations at each priority handled in FIFO order

RMS + Minimum Laxity First (MLF)• Critical operations managed as in RMS• Non-critical operations managed in single lowest priority• Non-critical operations handled in minimum laxity (slack time) order

Maximum Urgency First (MUF)• Thread priority per criticality level• Operations in each priority level handled in laxity order

Experimental Benchmarks

Measure Performance of Heuristics over Execution Time Jitter & Load

• Each numbered operating region is stable• Transitions between operating regions: changes in SRT load, HRT+SRT jitter • Performed using realistic hardware, OS, middleware, OFP application

Region 7 Performance (MUF does Better)

Region 8 Performance (RMS+MLF does Better)

Adaptation is Beneficial: Best Strategy Differs

L ML MHH L ML MHH L ML MHH

Mining Adaptation Clues

Refining Adaptation Clues

Adaptation over Scheduling Heuristics

A Map of the Best Performing Heuristic over Execution Jitter & Load• RMS performs best if system is under-loaded (theory predicts this)• RMS+MLF performs best in overload if jitter is very high or very low• MUF performs best if jitter is moderate

A Basis for Adaptive Control• Run-time observable measure correlates with performance: operation latency• A simple automaton could be constructed

My Contribution: A Unified Middleware Approach

Real QoS problems require both theoretical and empirical perspectives• Scheduling theory generalized over OS/middleware primitives heuristic space• Empirical study of specific heuristic (sub-)spaces is crucial• Analogy: theory/studies of Ethernet behavior: bin-exp-backoff vs. congestion collapse

Technical Challenge Research Approach

ResearchImpact

Efficient and Safe Systems

Arbitrary strategies that hybridize static/dynamic scheduling/dispatching

Increased utilization, critical operations still meet their deadlines

Customizable Middleware Dispatching composed from primitive elements

Supports tailored “fit” of scheduling/dispatching

Nimble Adaptation Integrated rate/priority selection mechanisms

O(n2)/O(n log n) O(n log n)/O(n) bound on adaptation

Performance Awareness Time and space efficient data collection and storage framework

Provides run-time observable info for control, post-analysis

Inclusive Systems Approach

Decision lattice joining a priori analysis with empirical measurement

Towards run-time reflective and adaptive policy selection

Research Impacts and Collaborations

Topics Publications, Systems, & Middleware

Kokyu

Middleware Framework

• Journal of Real-Time Systems, 2001• IEEE Proceedings special issue (submission in progress)• Boeing: ASTD, ASFD, WSOA, Bold Stroke (SEC, MoBIES)• Distinct open-source framework (Kokyu) – early 2002

Integrated Middleware Resource Management

• With Boeing, Honeywell: DASC, 1999• With Boeing, BBN: ICDCS, 2001• With Boeing: DASC, 2001• WORDS 2002• Boeing: ASTD, WSOA, Bold Stroke

Real-Time Metrics & Visualization Infrastructure

• DASC, 1999• DASC, 2000• Boeing: ASFD, WSOA, ASTD II, Bold Stroke (in progress)

Future Research ObjectivesTopics Problems, Approaches, & Investigations

Extremely Small Footprint DRE Firm/Soft/

Middleware

• Dispatch/comm/addressing in micro-niches (downward scalability)• Interesting design tensions between time/space/power/…• “Just enough” middleware: e.g., from Jini-like backbone to an ORB• DARPA ITO NEST Program: OEP middleware

Advanced Techniques for QoS Mechanism Instantiation

• Composing middleware scheduling/dispatching points end-to-end• Heterogeneous: multiple layers and paths• Multi-dimensional resource management: memory, network, CPU• Discover/apply good heuristics and domain-specific optimizations• Empirical/theoretical study/construction of adaptive decision lattices• AOP, domain-specific type systems

Coordinated Multi-Layer Multi-Agent QoS Management

• Integration, cooperation and co-design• Resource managers, schedulers, dispatchers, feature sets• Across hardware, firmware, OS, middleware, application layers• Toward generalized techniques, patterns, and a “complete” theory and practice of QoS composition for real-world systems

• E.g., real-time + anytime + adaptive control + decision-aiding + power-awareness + footprint-awareness + …

Concluding Remarks

Empirical Evaluation• Validates adaptive/hybrid scheduling approach• Quantifies costs/benefits of discrete alternatives• Powerful when combined with theoretical view

–“Mining” technique for problems & properties

Composable Scheduling/Dispatching• Enables domain-specific optimizations, especially when design decisions are aided by empirical data

Heuristic Space Experiments• Offer a quantitative blueprint for adaptation

Open-Source Code• All software described here that is uniquely a part of my research will be made available in the ACE_wrappers distribution

–Kokyu framework (early 2002)–Dispatching for new TAO Event Channel

ThanksMentors•Dr. Douglas C. Schmidt, Dr. David L. Levine, and Dr. Ron K. Cytron

Colleagues and Collaborators•Faculty and Staff of WU CS and CoE•Dr. Douglas Niehaus•Mr. David Sharp, Mr. Bryan Doerr, Mr. Don Winter, Dr. David Corman, Dr. Doug Stuart, Mr. Brian Mendel, Mr. Greg Holtmeyer, Mr. Pat Goertzen, Ms. Jeanna Gossett, Ms. Amy Wright, Mr. Jim Urness, Mr. Tom Venturella, Mr. Russ Wolter

•Mr. Kenneth Littlejohn (AFRL), Dr. Gary Koob (DARPA ITO)•Dr. Rakesh Jha, Mr. John Shackleton, Mr. Nigel Birch•Dr. Joseph Loyall, Dr. Richard Schantz, Dr. John Zinky, Dr. David Bakken•Dr. Ebrahim Moshiri, Mr. Malcolm Spence, Mr. Kevin Stanley

Friends and Family•Members of the DOC Group•WU CS Graduate Students•My wife Barb and son Paul•My Mom Dr. Helen Gill, Dad Mr. David Gill and sister Ms. Sarah E. Gill•My parents and siblings-in-law

Additional Questions ?These slides:

http://www.cs.wustl.edu/~cdgill/cdgill_defense.ppt http://www.cs.wustl.edu/~cdgill/cdgill_defense.pdf

Contact Information:

http://www.cs.wustl.edu/~cdgill

cdgill@cs.wustl.edu

Backup Slides

Number of Calls is Linear

Time in Each Call Appears Linear

Problem: Limitations with Existing Approaches to Time Frame Management

Previously ad hoc• Used different time types

–I.e., gethrtime ( ) vs. gettimeofday ( )

• Fortunately mapped to same clock–Brittle & non-portable solution

• No logical time support end-to-end• No common point of reference for time frames across subsystems

Application

DispatcherUpcall

Adapter

QuO RT-ARM

???

10 32

0 1

01 Hz

2 Hz

4 Hz

Harmonic Frame Rates

Start EndId

Solution: Distinct Manager for Physical & Logical Time Frames

FRAME MANAGER

EMBEDDED BOARD

DISPATCHER

START END IDRATE

0 50 020HZ

0 100 010HZ

0 200 05HZ

50 100 120HZ

0 100 010HZ

0 200 05HZ

100 150 220HZ

100 200 110HZ

0 200 05HZ

150 200 320HZ

100 200 110HZ

0 200 05HZ

200 250 420HZ

200 300 210HZ

200 400 15HZ

250 300 520HZ

200 300 210HZ

200 400 15HZ

300 350 620HZ

300 400 310HZ

200 400 15HZ

350 400 720HZ

300 400 310HZ

200 400 15HZ

400 450 820HZ

400 500 410HZ

400 600 25HZ

ADAPTER RTARM QuO

20HZ45

0

20H

Z45

0 50010H

Z

6005HZ

WSOA:

Singleton Frame Manager• Consistent low-level service

• Register rates with Frame Manager

• Frame start, end, id for each rate

• Timer-based call to update frames

• Dispatcher gets deadline for period–I.e., for laxity, deadline queues

• Upcall adapter gets deadline for cancellation, metrics decisions

• Higher-level resource managers can also use Frame Manager for adaptation decisions

–E.g., RT-ARM, QuO• IDL interface allows remote calls

20HZ

10HZ

5HZ

Problem: Limitations with Existing Adaptive Feedback Collection Approaches

Solutions originally ad hoc• Tedious, error prone, brittle

Lack of support for embedded, RT constraints

• Stringent memory limitations• Narrow windows of harvest time• Shared memory (e.g., VME) is attractive

• However, C++ semantics can cause problems

–Virtual functions–Native pointers–Heap allocation –Address space mapping

SH

AR

ED

ME

MO

RY

METRICS CACHE

SH

AR

ED

ME

MO

RY

METRICS CACHE

Process A Process B

Solution: Metrics Collection Within Embedded & Real-Time Constraints

WSOA:Shared Memory Capability• Offset-based smart pointers, • Strategized allocators• Supports run-time cache sizing and use from any address space

• System instrumented with C++ inline probe macros: write to a singleton metrics cache

Cache is Usable from Process Owned Memory

• Upcall Adapter• Metrics Monitor, Logger DOVE

SH

AR

ED

ME

MO

RY

METRICS MONITOR

METRICS CACHE

DIS

PA

TC

HE

R

20H

z D

eque

ue

20Hz E

nqueue

BEGINSUSPENDRESUME

END

Process Tile

BEGINEND

EMBEDDED BOARDS

UPCALL ADAPTER

OPERATION

BE

GIN

B

SUSP

EN

D

S

RE

SUM

E

R

EN

D

EBEGIN

B

SUSPEND

S

RESUME

R

END

E

B

EN

D

BE

GINE

20Hz EnQ

20Hz DQ

Proc Tile