RT-­‐Xen:  Real-­‐Time  Virtualiza2on  from  embedded  to  cloud  compu2ng  

Chenyang  Lu  Cyber-­‐Physical  Systems  Laboratory  Department  of  Computer  Science  and  Engineering  

Real-­‐Time  Virtualiza2on  for  Cars  Ø  Consolidate 100 ECUs à 10 multicore processors. Ø  Integrate multiple systems on a common platform à virtualization.

q  Infotainment on Linux or Android q  Safety-critical control on AUTOSAR q  Example: COQOS, Integrity Multivisor, Xen automotive branch

Ø  Must preserve real-time guarantees on a virtualized platform!

5/27/14   1  

LeB:  hDp://www.edn.com/design/automoGve/4399434/MulGcore-­‐and-­‐virtualizaGon-­‐in-­‐automoGve-­‐environments  Right:  hDp://community.arm.com/docs/DOC-­‐4610  

Real-­‐Time  Cloud  

Ø  Internet of Things à Cyber-Physical Systems q  Smart manufacturing, smart transportation, smart grid.

q  Internet-scale sensing and control à real-time cloud computing.

Ø  Example: Intelligent Transportation Systems q  Data center collects data from cameras and roadside detectors.

q  Control the traffic signals and message signs in real-time.

q  Transportation information feed to drivers.

q  SCATS @ Sydney: controlling 3,400 signals at 1s round-trip latency.

Ø  Latency-sensitive applications, e.g., cloud gaming q  Xbox One: cloud offloading computation of environmental elements

q  Sony acquired Gaikai, an open cloud gaming platform.

5/27/14   2  

Ø  Existing hypervisors provide no guarantee on latency q  Xen: credit scheduler, [credit, cap]

q  VMware ESXi: [reservation, share, limitation]

q  Microsoft Hyper-V: [reserve, weight, limit]

Ø  Public clouds lack service level agreement on latency q  EC2, Compute Engine, Azure: #VCPUs

5/27/14   3  

Virtualiza2on  is  not  real-­‐2me  today  

Current platforms provision resources, not latency!

Challenges  

Ø  Support real-time applications in a virtualized environment. q  Latency guarantees to tasks running in virtual machines (VMs).

q  Real-time performance isolation between VMs.

Ø Multi-level real-time performance provisioning q  Virtualization within a host

q  Communication and I/O

q  Cloud resource management

5/27/14   4  

RT-­‐Xen  

Ø  Real-time hypervisor based on Xen q  Real-time VM scheduling

q  Real-time communication

Ø  Build on compositional scheduling theory q  VMs specify resource interfaces

q  Real-time guarantees to tasks in VMs

Ø Open source q  Xen patch in progress

Ø  RT-OpenStack: cloud management based on RT-Xen

5/27/14 5

Xen  Virtualiza2on  Architecture  

Ø Xen: type-1, baremetal hypervisor q  Domain-0: drivers, tool stack to control VMs.

q  Guest Domain: para-virtualized or fully virtualized OS.

Ø Xen scheduler q  Guest OS runs on VCPUs.

q  Xen schedules VCPUs on PCPUs.

q  Credit scheduler: round-robin with proportional share.

5/27/14   6  

PCPUs

OS Sched

Xen Scheduler

OS Sched OS Sched

VCPU  Real-­‐Time  Task  

Composi2onal  Scheduling  

Ø Analytical real-time guarantees to tasks running in VMs. Ø  VM resource interfaces

q  Hides task-specific information q  Multicore: <period, budget, #VCPU> q  Computed based on compositional scheduling analysis

5/27/14   7  

Resource Interface Resource Interface

Resource Interface

Hypervisor

Virtual Machines

Workload Workload

Scheduler

Scheduler

Scheduler

Real-­‐Time  Scheduling  Policies  

Ø Global scheduling q  Shared global run queue

q  Allow VCPU migration across cores

q  Work conserving

q  Migration overhead and cache penalty

Ø  Partitioned scheduling q  Assign and bind VCPUs to cores

q  Schedule VCPUs on each core independently

q  Not work conserving: cores may idle when others have work pending

q  No migration overhead or cache penalty

Ø  Priority scheme q  Static priority: Rate Monotonic

q  Dynamic priority: Earliest Deadline First (EDF)

5/27/14   8  

Scheduling  VCPU  as  “Server”  

5/27/14   9  

Gme  

T1  (5,  3)  

T2  (5,  3)  

0   5   10   15  

0   5   10   15  

0   5   10   15  

Gme  

Gme  

Periodic  Server  (5,3)  

Deferrable  Server  (5,3)  

Budget  

Budget  back-­‐to-­‐back  execuGon    

Not  work  conserving  

Run  Queues  

Ø  rt-global: all cores share one run queue with a spinlock Ø  rt-partition: one run queue per core

Ø A run queue q  Holds VCPUs that are runnable (not idle) q  Divided into two parts: with budget; without budget

q  Sorted by priority (DM or EDF) within each part

5/27/14 10

Scheduling  Func2ons  

Ø  do_schedule q  triggered every 1 ms

q  consume current VCPU’s budget

q  replenish a VCPU’s budget if it is the end of its period

q  return runnable VCPU with the highest priority

Ø  context_switch

Ø  context_saved q  global scheduler only

q  insert VCPU back to run queue after context switch

Ø wake_up q  triggered by timer or I/O

q  call do_schedule if necessary

5/27/14 11

RT-­‐Xen:  Real-­‐Time  Scheduling  in  Xen  

5/27/14   12  

Fixed  Priority  (RM)  

ParGGoned  Scheduling  

Dynamic  Priority  (EDF)  

Periodic   Deferrable  

Periodic   Deferrable  

Deferrable   Periodic  

Deferrable   Periodic  

Sporadic  

Polling  Work  

Conserving  Periodic  

Capacity  Reclaiming  Periodic  

Global    Scheduling  

Ø  Single-core RT-Xen 1.0 Ø  Single-core enhanced RT-Xen 1.1

Ø Multi-core scheduling RT-Xen 2.0 q  RT-global

q  RT-partition

Experimental  Setup  

Ø Hardware: Intel i7 processor, 6 cores, 3.33 GHz q  Allocate 1 VCPU for Domain-0, pinned to PCPU 0

q  All guest VMs use the remaining cores

q Software q  Xen 4.3 patched with RT-Xen

q  Guest OS: Linux patched with LITMUS

Ø Workload q  Period tasks: synthetic, ARINC 653 avionics workload (RT-Xen 1.1)

q  Allocate tasks à VMs

5/27/14   13

RT-­‐Xen  2.0:  Credit  Scheduler    

5/27/14   14  

•  Credit misses deadlines at 22% of CPU capacity.

•  RT-Xen delivers real-time performance at 78% of CPU capacity.

RT-­‐Xen  2.0:  Scheduling  Overhead  

5/27/14   15

•  Credit has poor max overhead due to load balancing.

•  rt-global has extra overhead due to global lock.

RT-­‐Xen  2.0:  Theory  vs.  Experiments  

5/27/14   16  

•  gEDF > pEDF empirically, thanks to work-conserving global scheduling.

•  gEDF < pEDF theoretically due to pessimistic analysis.

1.1 1.5 1.9 2.3 2.7 3.1 3.5 3.9 4.3 4.7 0

0.2

0.4

0.6

0.8

1

Task Set Utilization

Frac

tion

of s

ched

ulab

le ta

skse

ts

gEDF (RT−Xen)pEDF (RT−Xen)pEDF (CSA Theory)gEDF (CSA Theory)

RT-­‐Xen  2.0:  Deferrable  vs.  Periodic  

5/27/14   17  

Work-conserving wins empirically! •  Deferable Server (DS) > Periodic Server.

•  gEDF+DS à best real-time performance.

RT-­‐Xen  2.0:  How  about  Cache?  

5/27/14   18  

•  gEDF > pEDF for cache intensive workload.

•  Benefit of global scheduling dominates migration cost.

•  Shared cache mitigates cache penalty due to migration.

pEDF in guest OS

Real-­‐Time  Inter-­‐VM  Communica2on  

Ø  Real-time communication between high priority VMs under contention from other VMs

Ø  Local communication between VMs on a same host q  ~20 VMs on a single host is not rare.

q  Co-locate VMs to reduce communication cost.

Ø Network communication (on-going)

5/27/14   19  

Is  real-­‐2me  VM  scheduling  enough?  

5/27/14   20  

VMM  Scheduler  

C  5  C  0   C  1   C  2   C  4  C  3  

Dom  3  

Dom  4  

Dom  5  

Dom  0   Dom  1   Dom  2  

0 5000 10000 150000

0.5

1

Micro Seconds

CDF

Plot

RT−Xen, Original Dom 0

…  

…  

…  5K  data  points  

Xen  Communica2on  Architecture  

21  

C   D  

A  

nearont  

Domain  1   Domain  0  

…   B  

nearont  

Domain  2  

…  neGf   neGf  

TX   RX  netback  

netback[0]  {      rx_ac2on();      tx_ac2on();  }  

nearont  

Domain  m  

…  

nearont  

Domain  n  

…  

…   …  neGf  

neGf   neGf  

neGf  

soBnet_data  

Significant priority inversion in Domain 0 ✘  Packets are fetched in a round-robin order. ✘  Sharing one queue in softnet_data.

RTCA:  Real-­‐Time  Communica2on  Architecture  

22  

Preserve prioritization in Domain 0 •  Packets are fetched by priority, up to batch size •  Queues are separated by priority in softnet_data

A  

nearont  

Domain  1   Domain  0  

…   A  

nearont  

Domain  2  

…  neGf   neGf  

TX   RX  netback  

netback[0]  {      rx_ac2on();      tx_ac2on();  }  

…   …  

soBnet_data  B  

nearont  

Domain  m  

…  

neGf  

neGf  nearont  

Domain  n  

…  

…  

neGf  

neGf  

B  

Real-­‐Time  Inter-­‐VM  Communica2on  Median   75th  percen2le   95th  percen2le  

Domain-­‐0   Original   RTCA   Original   RTCA   Original   RTCA  

Base   68   70   69   72   71   74  

Light   5183   60   5803   61   6610   66  

Medium   9621   61   9780   63   11954   68  

Heavy   9872   69   10095   71   11085   76  

RTCA maintains low latency for high-priority VMs despite contention from low-priority VMs

23

Conclusion  

Ø Diverse applications demand real-time virtualization. q  Embedded real-time systems.

q  Internet-scale cyber-physical systems.

q  Latency-sensitive cloud applications.

Ø  RT-Xen provides real-time performance q  Efficient implementation of diverse real-time scheduling policies.

q  Leverage compositional scheduling theory à analytical guarantee.

q  Resource interface à resource allocation for latency bounds.

q  RTCA supports real-time inter-VM communication.

Ø On-going q  RT-Xen patch for Xen core distribution.

q  RT-OpenStack: integration with OpenStack on the way.

5/27/14   24  

Check  out  RT-­‐Xen  

5/27/14  

Team

Ø  Sisu Xi, Justin Wilson, Chong Li, Chris Gill, Washington University Ø  Meng Xu, Jaewoo Lee, Insup Lee, Lihn Phan, Oleg Solosky, University of Pennsylvania

RT-­‐Xen  

hDps://sites.google.com/site/realGmexen/    

25