© 2007 ibm corporation ibm t. j. watson research center xen summit 2007 | april 17, 2007 xensocket...

© 2007 IBM Corporation

IBM T. J. Watson Research Center

Xen Summit 2007 | April 17, 2007

XenSocketSuzanne McIntosh Security, Privacy and Extensible Technologies IBM T. J. Watson Research Center [email protected]

Interdomain transport for VMs

2

McIntosh - IBM T. J. Watson Research Center

XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation

IBM T. J. Watson Research CenterXiaolan ZhangPankaj RohatgiSuzanne McIntosh

BAE SystemsJohn Linwood Griffin

We would like to thank the following people:Ronald Perez Douglas Lee Schales Anthony LiguoriRyan Harper Muli Ben-Yehuda Eric Van HensbergenReiner Sailer Stefan Berger Wesley Most

XenSocket Team

3



XenSocket Project – Background

4



Use Xen to enhance security of System S, a stream processing system described as follows:

High-throughput, large-scale, distributed stream processing system Security architecture being developed by IBM Research Extracts important information by analyzing voluminous amounts of

unstructured and mostly irrelevant data Example applications

– Analyze audio, video and data feeds to support trading activities in financial institutions

– Disaster response through analysis of vehicular movements, traffic and other sensors, news reports, etc.

XenSocket Project – The Objective

5



Enhance security with minimal performance hit– Throughput is key metric of goodness for our purposes

– Achieve throughput approaching that of UNIX domain socket

– Especially targeting message sizes in 10 KB – 100 KB range

Facilitate porting of code to the Xen environment– Implement socket-based interface

XenSocket Project – The Requirements

6



Unfortunately, interdomain communication using the Xen virtual network fell short of throughput required for System S.

Inefficient same-system networking performance is a well-known problem wrt VMs [1] [2]

We speculate that the inefficiency can be attributed to

– Overhead incurred by TCP/IP stack

– Repeated hypercalls to invoke Xen page flipping

XenSocket Project – The Problem

7



Proof-of-concept built and tested in Xen 3.0.2.

Bypass use of TCP stack Replace page-flipping with copy Statically allocate memory buffers to be shared between two domains Use sockets-based interface to shared-memory-based transport No modification to Xen or OS required XenSocket compiles into a kernel module

XenSocket Project – The Solution

8



XenSocket Design – Overview

Sockets-based interface to shared memory buffers for domain-to-domain communication

Provides one-way tunnel between sender and receiver domains– Conserves memory in the event only one-way communication

needed

Two types of memory pages shared by each endpoint– Descriptor page (4KB) for storing state and control information

– Buffer pages (multiple 4KB pages) form the circular buffer for writing and reading data

9



XenSocket Architecture

DataPlane

128KB Shared Circular Buffer

(32 4KB pages)

ControlPlane

Shared Descriptor Page

(4KB page)

xen_recvmsg()xen_sendmsg()

Tx Data Rx Data

available_bytes

send_offset

available_bytes

recv_offset

Stream in Stream outSenderApp

ReceiverApp

10



XenSocket Implementation – Receiver

Calls socket() API to create socket

Calls bind() API– Bind the socket to an address

– Allocates physical memory for descriptor page and shared circular buffer

– Returns grant table reference of descriptor page

– Uses sender domain ID to allocate event channel for communication with sender

Calls read() or recv() for receiving data– Blocks until data is detected in circular buffer

11



XenSocket Implementation – Sender

Calls socket() to create a socket

Calls connect()– Uses supplied receiver domain ID and grant table reference of

shared descriptor page

– Maps physical pages of shared circular buffer into the virtual address space of sender

– Establishes other end of event channel to communicate events

Calls send() or write() to transmit data– Blocks if buffer full

12



XenSocket Implementation – Data Transfer Algorithm

Core piece of implementation is an efficient data transfer algorithm

Send and receive algorithms use one shared control variable, available_bytes– Indicates number of bytes available for write in circular buffer

Sender and receiver maintain local read/write offsets into circular buffer– Offsets are not shared

13



XenSocket Performance

XenSocket evaluation environment

IBM HS20 blade with dual 2.8GHz Pentium Xeon processors and 4GB RAM

Used netperf version 2.4.2 as our primary benchmark

All data reported was run on Xen 3.0.2 and Linux version 2.6.16.18

Each test was run 3 times, with the average reported

All experiments were run in single CPU mode with hyper-threading disabled to minimize performance variation

14



XenSocket Performance – Common Message Sizes

Throughput Comparison of XenSocket vs. Unix Domain Socket and TCP for message sizes between 512 Bytes and 16 KB. XenSocket achieves up to 72 times the throughput of standard TCP stream at message size of 16 KB.

15



XenSocket Performance – Larger Message Sizes

Throughput Comparison of XenSocket vs. Unix Domain Socket and TCP for large message sizes. Both XenSocket and Unix Domain Socket see a large drop-off when the message size reaches 512 KB and then stabilize around 5-6 Mb/s. The performance curves invert at message size of 512 KB where XenSocket outperforms Unix Domain Socket.

16



What about performance of TCP under Windows and other OSs?

How does XenSocket perform in current Xen release?

Our design of a XenSocket is a one-way communication pipe between two domains. – Traditional view of a socket is a two-way mechanism

– Alternate design would include variable-size circular buffers with logic capable of adapting the buffer reservation size to actual usage of the buffer

Next Steps

17



1. A. Menon, A. L. Cox, and W. Zwaenepoel. Optimizing network virtualization in Xen. In 2006 USENIX Annual Technical Conference, pages 15–28, Boston, Massachusetts, USA, June 2006.

2. A. Menon, J. R. Santos, Y. Turner, G. J. Janakiraman, and W. Zwaenepoel. Diagnosing performance overheads in the Xen virtual machine environment. In VEE’05: First International Conference on Virtual Execution Environments, pages 13–23, Chicago, Illinois, USA, June 2005.

References

18



Questions?

Suzanne McIntoshSecurity, Privacy and Extensible Technologies

IBM T. J. Watson Research [email protected]

© 2007 ibm corporation ibm t. j. watson research center xen summit 2007 | april 17, 2007 xensocket...

Documents