xpds14 - vnuma in xen - wei liu, citrix
DESCRIPTION
vNUMA (virtual NUMA) is a memory optimisation technology that makes virtual machine aware of the NUMA topology of the underlying physical server topology, which is very important for some specific kind of workload like HPC. Some significant work on vNUMA on PV has been done by Elena Ufimtseva, vNUMA on HVM was also posted by other developers, but none of them was merged upstreamed. This talk will cover the history, design and implementation of vNUMA, and possibly with some number to back up the importance of this feature.TRANSCRIPT
vNUMA in Xen
Wei Liu & Elena Ufimtseva
Chicago – August 18, 2014
Agenda Motives History and status Design Problems Benchmark Future Work
Agenda
I Motives
I History and status
I Design
I Problems
I Preliminary benchmark results
I Future work
Chicago – August 18, 2014 vNUMA in Xen 2 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Motives
I Cross NUMA node memory access is expensiveI Need to avoid cross node memory access
I Xen is NUMA awareI NUMA aware schedulingI NUMA aware guest memory placement
I Operating system like Linux is NUMA awareI NUMA aware schedulingI NUMA aware memory allocation / migration
I The missing bitsI Memory layout informationI CPU topology
Chicago – August 18, 2014 vNUMA in Xen 3 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
History and status
I PV vNUMA presented in Xen Summit 2010 by Dulloor Raohttp://slidesha.re/1AXsFbu
I HVM vNUMA patches posted by Andre Przywara circa 2010
I Elena Ufimtseva has been working on upstreamable PVvNUMA since 2013
Chicago – August 18, 2014 vNUMA in Xen 4 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Design: PV and PVH
I Toolstack puts enlightenment information in hypervisor
I Guest memory allocation in accordance with enlightenmentinformation
I Guest retrieves enlightenment information via hypercall duringboot up
Chicago – August 18, 2014 vNUMA in Xen 5 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Design: PV and PVH
Enlightenment information structure
struct vnuma_info
{
nr_vnodes;
vdistance[nr_vnodes * nr_vnodes];
vcpu_to_vnode[nr_vnodes];
vnode_to_pnode[nr_vnodes];
vmemrange[nr_vnodes];
}
Chicago – August 18, 2014 vNUMA in Xen 6 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Design: HVM
I Toolstack puts enlightenment information in hypervisor
I Toolstack arranges ACPI tables
I Guest memory allocation in accordance with enlightenmentinformation
I Guest retrieves layout information via ACPI tables during bootup
Chicago – August 18, 2014 vNUMA in Xen 7 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Problems: vNUMA and other features
PV PVH HVMBallooning Y Y* NPoD N/A ? N
Chicago – August 18, 2014 vNUMA in Xen 8 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Problems: CPU topology
Chicago – August 18, 2014 vNUMA in Xen 9 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark
I HostI 2 sockets, 12 PCPUs, HT disabledI 36GB RAM, 2 NUMA nodesI NUMA balancing enabled
I GuestI 12 VCPUsI 16GB RAM, 2 virtual NUMA nodesI vnodes mapped to different pnodes, vcpu pinned to pnodeI NUMA balancing enabled
I Benchmarks to runI AutonumaI SPECJBBI STREAM
Chicago – August 18, 2014 vNUMA in Xen 10 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Autonuma
Chicago – August 18, 2014 vNUMA in Xen 11 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Autonuma
Chicago – August 18, 2014 vNUMA in Xen 12 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Autonuma
Chicago – August 18, 2014 vNUMA in Xen 13 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: SPECJBB
Chicago – August 18, 2014 vNUMA in Xen 14 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: SPECJBB
Chicago – August 18, 2014 vNUMA in Xen 15 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: STREAM
Chicago – August 18, 2014 vNUMA in Xen 16 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Benchmark: Conclusion
I vNUMA improves performance for PV guest
I vNUMA has weird result in SPECJBB for PVH guest, butother two benchmarks have good results
Chicago – August 18, 2014 vNUMA in Xen 17 / 19
Agenda Motives History and status Design Problems Benchmark Future Work
Future Work
I basic vNUMA support for all guest types
I Dom0 vNUMA
I address vNUMA compatibility issues with PoD and ballooningfor HVM guest
I address performance issue for PVH
Chicago – August 18, 2014 vNUMA in Xen 18 / 19
Thank you!
Chicago – August 18, 2014 vNUMA in Xen 19 / 19