StorkCloud Data Transfer Scheduling and

Optimization as a Service Presented by: Brandon Ross

Contents 1.  Introduction 2.  Components 3.  Optimization 4.  Conclusion

2 of 23

StorkCloud? ü Multi-protocol data transfer scheduler ü Remote metadata retrieval and

caching service ü Dynamic, protocol-agnostic transfer

optimization to improve speed ü All in the cloud – accessible through

thin client GUIs and public REST API

3 of 23

Why StorkCloud? ü  Storage and computation are both in the

cloud – cloud transfer is the missing link ü  Data transfer is usually inconvenient –  Many different protocols –  Many different applications

ü  Transferring large files requires monitoring whole process – StorkCloud is “fire-and-forget”

4 of 23

Why optimize? ü  Transfers are usually suboptimal

–  Inadequacies of underlying protocols –  End-system misconfiguration

ü  Not designed for high-speed networks ü  Some applications can be specially

configured, but it’s mostly guesswork ü  Network environments can vary – dynamic

optimization is important ü  StorkCloud aims to solve these problems

5 of 23

Who? ü  Scientists in data-oriented fields

(astrophysics, genomics, climatology, biochemistry, etc.)

ü Data centers looking to outsource replication and data placement

ü Application developers who might want to offload data transfer tasks

ü Anyone with a lot of data to move

6 of 23

Similar Work ü Only similar service we know of is

Globus Online – Mature, popular service

•  Over 18.3 petabytes transferred! – Designed to support FTP and GridFTP –  Statically optimized transfers – No prefetching/caching available for

directory listings

7 of 23

Contents 1.  Introduction 2.  Components 3.  Optimization 4.  Conclusion

8 of 23

9 of 23

Stork Data Scheduler ü Accepts file transfer jobs – source,

destination, and other options ü Places job in queue for processing ü Passes jobs off to transfer module ü Job status can be queried by clients ü Currently first come, first served

10 of 23

Directory Listing Service ü Conceptually: unified metadata

interface to many file systems ü Retrieves file and directory metadata

from remote file systems ü Returns results as JSON ü Uses caching and prefetching to

improve responsiveness

11 of 23

12 of 23

DLS Performance

Transfer Modules ü Pluggable transfer modules perform

transfers for specific protocols ü Communicate progress and

messages back to scheduler ü Either Java bytecode or external

executable – can be any language!

13 of 23

Client Interfaces ü  Thin clients communicate with server REST API

using JSON –  Starting or canceling transfers, or querying transfer

status –  Browsing remote directories –  Initializing credentials (e.g. GSI proxies)

ü  Currently have Android and web applications, and command line tools

ü  Our GUI applications can browse remote files and check transfer progress

14 of 23

Contents 1.  Introduction 2.  Components 3.  Optimization 4.  Conclusion

15 of 23

Optimization in StorkCloud ü  Each pluggable optimizer is an

implementation of a different algorithm ü  Each targets a set of file transfer

parameters ü  Feedback loop; optimizer and TM work

together to optimize transfer ü  Optimizers are protocol-agnostic – only

care about whether transfers support targeted parameters

16 of 23

Parameters and Techniques ü Pipelining – “queuing up” transfer

commands at a remote system ü Parallelism – transferring file data

over multiple connections ü Concurrency – transferring multiple

files at a time

17 of 23

18 of 23

Parameters Visualized

Algorithms ü Optimal parallelism prediction: – Samples points, performs regression

analysis to predict optimal parallelism – 2nd order and c-order analysis

ü Parallelism-Concurrency-Pipelining – Uses historical database and clustering

19 of 23

Algorithms ü  Single/Multi Chunk Concurrency

–  Newest algorithms, designed for multi-file transfers with mixed sizes

–  Partition file sets into “chunks” based on file size and transfer concurrently on multiple channels

–  Each channel is configured heuristically –  SCC: transfer chunks one at a time, split across all

concurrent channels –  MCC: each chunk gets a dedicated channel;

reduces effect of small files

20 of 23

Contents 1.  Introduction 2.  Components 3.  Optimization 4.  Conclusion

21 of 23

22 of 23

SCC/MCC Performance

0 500

1000 1500 2000 2500 3000 3500 4000 4500 5000

1 2 4 6 8 10

Thro

ughp

ut (M

bps)

Concurrency

(a) (XSEDE)Globus-Online

SCCMCC

SCC_bufferMCC_buffer

0

100

200

300

400

500

600

700

800

900

1 2 4 6 8 10

Thro

ughp

ut (M

bps)

Concurrency

(b) (LONI)SCCMCC

XSEDE LONI

23 of 23

Effects of Parameters

Future Work ü  Additional scheduling algorithms and priority ü  Date-based scheduling and routine jobs ü  Direct file upload/download through client

interfaces – currently only 3rd party transfers ü  SSL/TLS for secure communications (HTTPS) ü  Additional protocols (SFTP, HTTP, AFP, etc.) ü  Temporary file parking – storage on server in

case of destination issues ü  Historical performance database

24 of 23

The End Thank you! Any questions?

26

27