extending globus to support multicast transmission karl jeacle [email protected]

Extending Globus to support Multicast Transmission

Karl [email protected]

All Hands Meeting 2004

Rationale

Would like to achieve high-speed bulk data delivery to multiple sites

Multicasting would make sense Existing multicast research has focused on

sending to a large number of receivers But Grid is an applied example where sending to

a moderate number of receivers would be extremely beneficial


Multicast availability

Deployment is a problem!Protocols have been defined and implementedValid concerns about scalability; much FUD “chicken & egg” means limited coverage

Clouds of native multicastBut can’t reach all destinations via multicastSo applications abandon in favour of unicast

What if we could multicast when possible……but fall back to unicast when necessary?


Multicast TCP?

TCP “single reliable stream between two hosts”

Multicast TCP “multiple reliable streams from one to n hosts”

May seem a little odd, but there is precedent…TCP-XMO – Liang & CheritonM-TCP – Mysore & VargheseM/TCP – Visoottiviseth et alPRMP – Barcellos et alSCE – Talpade & Ammar


ACK implosion


Building Multicast TCP

Want to test multicast/unicast TCP approachBut new protocol == kernel changeWidespread test deployment difficult

Build new TCP-like engineEncapsulate packets in UDPRun in userspacePerformance is sacrificed……but widespread testing now possible


TCP/IP/UDP/IP

SendingApplication

TCP

IP

UDP

IP

ReceivingApplication

TCP

IP

UDP

IP

If natively implemented

test deployment


TCP engine

Where does initial TCP come from? Could use BSD or Linux

Extracting from kernel could be problematic

More compact alternative lwIP = Lightweight IPSmall but fully RFC-compliant TCP/IP stack

lwIP + multicast extensions = “TCP-XM”


TCP-XM overview

Primarily aimed at push applications Sender initiated – advance knowledge of receivers Opens sessions to n destination hosts simultaneously Unicast is used when multicast not available Options headers used to exchange multicast info API changes

Sender incorporates multiple destination and group addresses Receiver requires no changes

TCP friendly, by definition


TCP

SYN

SYNACK

ACK

FIN

FIN

ACK

ACK

ACK

DATASender Receiver


TCP-XM

Sender

Receiver 1

Receiver 2

Receiver 3


TCP-XM connection

ConnectionUser connects to multiple unicast destinationsMultiple TCP PCBs created Independent 3-way handshakes take placeSSM or random ASM group address allocated

(if not specified in advance by user/application)

Group address sent as TCP optionAbility to multicast depends on TCP option


TCP-XM transmission

Data transferData replicated/enqueued on all send queuesPCB variables dictate transmission modeData packets are multicast (if possible)Retransmissions are unicastAuto fall back/forward to unicast/multicast

CloseConnections closed as per unicast TCP


Fall back / fall forward

TCP-XM principle “Multicast if possible, unicast when necessary”

Initial transmission mode is group unicastEnsures successful initial data transfer

Fall forward to multicast on positive feedbackTypically after ~75K unicast data

Fall back to unicast on repeated mcast failure


TCP-XM reception

ReceiverNo API-level changesNormal TCP listenAuto-IGMP join on TCP-XM connectAccepts data on both unicast/multicast ports tcp_input() accepts:

packets addressed to existing unicast destination……but now also those addressed to multicast group

Tracks how last n segs received (u/m)


Grid multicast?

How can multicast be used in Grid environment?

TCP-XM is new multicast-capable protocol

Globus is de-facto Grid middleware

Would like TCP-XM support in Globus…


Globus XIO

eXtensible Input Output libraryAllows “i/o plugins” to Globus

APISingle POSIX-like API / set of semanticsSimple open/close/read/write API

Driver abstractionHides protocol details / Allows for extensibilityStack of 1 transport & n transform driversDrivers can be selected at runtime


XIO architecture


XIO implementation


XIO/XM driver specifics

Two important XIO data structures

1. Handle Returned to user when XIO framework ready Used for all open/close/read/write calls lwIP netconn connection structure used

2. Attribute Used to set XIO driver-specific parameters… … and TCP-XM protocol-specific options List of destination addresses


XIO code example// init stackglobus_xio_stack_init(&stack, NULL);

// load drivers onto stackglobus_xio_driver_load("tcpxm", &txdriver);globus_xio_stack_push_driver(stack, txdriver);

// init attributesglobus_xio_attr_init(&attr);globus_xio_attr_cntl(attr, txdriver, GLOBUS_XIO_TCPXM_SET_REMOTE_HOSTS,

hosts, numhosts);

// create handleglobus_xio_handle_create(&handle, stack);

// send dataglobus_xio_open(&handle, NULL, target);globus_xio_write(handle, "hello\n", 6, 1, &nbytes, NULL);globus_xio_close(handle, NULL);


One-to-many issues

Stack assumes one-to-one connectionsXIO user interface requires modificationNeeds support for one-to-many protocolsMinimal user API changesFramework changes more significant

GSI is one-to-oneAuthentication with peer on connection setupBut cannot authenticate with n peersNeed some form of “GSI-M”


LAN/WAN tests

LAN Computer Laboratory Mix of CPU speeds Linux All multicast

Throughput to each host is typically at least 10Mb/s, depending on CPU and network load of both sender and receiver.

WAN UK eScience Network Mix of CPU speeds FreeBSD/Linux/Solaris Some multicast

Throughput varies (Mb/s) Imperial 25 Cardiff

13 Manchester 11 Southampton 10 Belfast 5


LAN speed


LAN efficiency


WAN efficiency


Driver availability

Multicast transport driver for Globus XIORequires Globus 3.2 or later

Source code onlineSample clientSample serverDriver installation instructions

http://www.cl.cam.ac.uk/~kj234/xio/


mcp & mcpd

Multicast file transfer application using TCP-XM

‘mcpd &’ on servers ‘mcp file host1 host2… hostN’ on client

http://www.cl.cam.ac.uk/~kj234/mcp/Full source code onlineFreeBSD, Linux, Solaris


Future

Protocol workParallel unicast / multicast transmissionMulticast look ahead & multiple groups

DeliverablesUpdates to mcp/mcpd

ExperimentationMore detailed testing requiredCurrently limited to LAN and UK eScienceWill extend to global Intel PlanetLab


eScience volunteers…

1. In place Cambridge Cardiff Imperial Manchester Newcastle Oxford Southampton UCL

2. Firewall issues Belfast Daresbury Glasgow Rutherford

3. Not possible Edinburgh


All done!

Thanks for listening!

Questions?

extending globus to support multicast transmission karl jeacle [email protected]

Documents