xrootdUpdate

Andrew HanushevskyStanford Linear Accelerator Center

15-Feb-05

http://xrootd.slac.stanford.edu

15-Feb-05 2: xrootd

Goals

High Performance File-Based Access Scalable, extensible, usable

Fault tolerance Servers may be dynamically added and removed

Flexible Security Allowing use of almost any protocol

Simplicity Can run xrootd out of the box

No config file needed for non-complicated/small installationsGenerality Can configure xrootd for ultimate performance

Meant for intermediate to large-scale sitesRootd Compatibility

15-Feb-05 3: xrootd

How Is high performance achieved?

Rich but efficient server protocol Combines file serving with P2P elements Allows client hints for improved performance

Pre-read, prepare, client access & processing hints, Multiplexed request stream

Multiple parallel requests allowed per client

An extensible base architecture Heavily multi-threaded

Clients are dedicated threads whenever possible Extensive use of OS I/O features

Async I/O, memory mapped I/O, device polling, etc. Load adaptive reconfiguration.

15-Feb-05 4: xrootd

How performant is it?

Can deliver data at disk speeds (streaming mode) Assuming good network, proper TCP buffer size, and

asynchronous I/O

Low CPU overhead 75% less CPU than NFS for same data load

General requirements Middling speed machine NIC matched

Rule of thumb: 1MHz/1Mb + 20% (minimum) The more CPU’s the better

1GB of RAM

15-Feb-05 5: xrootd

Measured performance

15-Feb-05 6: xrootd

Xrootd vs. NFS (2/9/2005 - Gregory Schott, DESY)

15-Feb-05 7: xrootd

xrootd Server Architecture

Protocol LayerProtocol Layer

Filesystem Logical LayerFilesystem Logical Layer

Filesystem Physical LayerFilesystem Physical Layer

Filesystem ImplementationFilesystem Implementation

Protocol & Thread ManagerProtocol & Thread Manager

(included in distribution)

p2p heart

15-Feb-05 8: xrootd

Acronyms, Entities & Relationships

datadataxrootd

olbdxrootd

olbd

Data Clients

Redirectors

Data Servers

MM

SS

ctlctl

olbdolbd Control Network

Managers & Servers(resource info, file location)

xrootdxrootd Data Network

(redirectors steer clients to dataData servers provide data)

15-Feb-05 9: xrootd

Closer Look Into Now & The Future

Adaptive Reconfiguration

Load Balancing

Fault Tolerance

Proxy Services

Security & Privacy

Monitoring & Client Tuning

File Services (MSS, SRM, & Grid)

15-Feb-05 10: xrootd

Adaptive Reconfiguration

Server dynamically adjusts configuration Number of threads Count and sizes of buffers and objects

Finally got the algorithms right 400 Clients can be accommodated in about 80MB RAM

High latency connection rescheduling Affects how async I/O is done

Future:Future: Make algorithms CPU-speed sensitive Appropriate time/space trade-off

15-Feb-05 11: xrootd

Load Balancing

xrootd scales in multiple dimensions Can run multiple load balanced xrootd’s

Architected as self-configuring structured peer-to-peer (SP2) data servers

Servers can be added & removed at any time Client (XTNetFile) understands SP2 configurations

xrootd informs client when running in this mode Client has more recovery options in the event of failure

15-Feb-05 12: xrootd

Example: SLAC Configuration

client machinesclient machines

kan01 kan02 kan03 kan04 kanxx

bbr-olb03 bbr-olb04 kanolb-a

15-Feb-05 13: xrootd

Next Load Balancing Step

Fully deploy scalable p216p architecture

Accommodate swarms of up to 64,000 servers Perhaps more….

Self-organizing (no config file changes needed) Servers self-organize into cells of 64 Cells cluster into a minimum spanning tree

Testing alpha code now Some message semantics still need work

15-Feb-05 14: xrootd

Fault Tolerance

Server and resources may come and go Uses load balancing to effect recovery

New servers can be added at any time for any reason Files can be moved around in real-time

Client simply adjust to the new configuration XTNetFile object handles recovery protocol

Future:Future: Provide Protocol UnResp interface Can be used to perform reactive client scheduling

15-Feb-05 15: xrootd

Bottom Heavy System

The olbd/xrootd architecture is bottom heavy Allows super-rapid client dispersal Ideal scaling characteristics

15-Feb-05 16: xrootd

Current Proxy Support

client machinesclient machines

data01 data02 data03 data04

proxy xrootd

IN2P3

INDRA22

3322

FirewallFirewall

olbd

15-Feb-05 17: xrootd

Future:Future: Comprehensive Proxies

Current support addresses external access Need solution when clients also behind firewall

Allow proxy/proxy connections Will need direct connections for performance

Local Proxy xrootd

Remote Proxy xrootd

client

data02

data03

data04

15-Feb-05 18: xrootd

Flexible SecurityNegotiated Security Protocol Allows client/server to agree on protocol

Can be easily extended Multi-protocol authentication support

Protocols implemented as dynamic plug-insFuture:Future: Add GSI and encryption interfaces Foundation for distributed proxy security Accommodates data encryption for privacy

Geri Ganis, CERN, working on this

15-Feb-05 19: xrootd

Scalable Proxy Security

SLAC PROXY RAL PROXY

3

2 2

1

11 Authenticate & develop session key22 Distribute session key to authenticated subscribers

33 Data servers can log into each other using session key

Data ServersData Servers

15-Feb-05 20: xrootd

Monitoring

Today xrootd can provide event traces Login, file open, close, disconnect Application markers and client information All read/write requests

Highly scalable architecture usedFuture:Future: Complete toolkit for utilizing data Critical for tuning applications

Jacek Becla, SLAC, working on this

15-Feb-05 21: xrootd

MSS SupportLightweight agnostic interfaces provided oss.mssgwcmd command

Invoked for each create, dirlist, mv, rm, stat oss.stagecmd |command

Long running command, request stream protocol Used to populate disk cache (i.e., “stage-in”)

xrootd(oss layer)

mssgwcmd

MSSstagecmd

15-Feb-05 22: xrootd

SRM DirectionMSS Interface ideal spot for SRM hook Can simply use existing hooks

mssgwcmd & stagecmd Define new long running hook

oss.srm |command Process all external disk cache management requests

xrootd(oss layer)

srm

MSS

15-Feb-05 23: xrootd

Future:Future: SRM

Great interest in SRM Provides external data management interface

MSS and Grid Unfortunately, SRM interface in flux

Heavy vs light protocol Two sites looking at providing SRM layer

BNL IN2P3

Will work with LLNL team to speed effort

15-Feb-05 24: xrootd

Conclusion

xrootd provides high performance file access It appears to be one of the best in it’s class

Unique performance, usability, scalability, security, compatibility, and recoverability characteristics

One server can easily support over 600 parallel clients New software architecture

Challenges Maintain scaling while interfacing to external systems

Opportunities Ability to provide data access at the LHC scale.