globus online for research data management

globus online

Globus Online for Research Data Management

Rachana Ananthakrishnan Great Plains Network Annual Meeting 2013

We started with technology proven in many large-scale grids

GridFTP GRAM

MyProxy GSI-OpenSSH

…

1.2 PB of climate data delivered to 23,000 users

Typical of large, well funded research projects using GT

1.2 PB of climate data delivered to 23,000 users

GT provides robust infrastructure for the 1%

What about the 99%?

GT provides robust infrastructure for the 1%

What about the 99%?

BIG SCIENCE. Small labs

GT provides robust infrastructure for the 1%

globus online

Managing data should be easy …

Registry  

Staging  Store  

Ingest  Store  

Analysis  Store  

Community  Store  

Archive   Mirror  

Ingest  Store  

Analysis  Store  

Community  Store  

Archive   Mirror  

Registry  

… but it’s hard and frustrating!

Registry  

Staging  Store  

Ingest  Store  

Analysis  Store  

Community  Store  

Archive   Mirror  

Ingest  Store  

Analysis  Store  

Community  Store  

Archive   Mirror  

Registry  

Quota exceeded

!

Expired credentials

!

Network failed. Retry.

!

Permission denied

!

What is Globus Online?

Transfer and sharing of large data sets…

…with dropbox-like characteristics…

…directly from your own storage systems

We adopted SaaS approaches to transform the user experience

… for both researchers and resource owners/system

administrators

We started with reliable, secure, high-performance file transfer …

Data Source

Data Destination

User initiates transfer request

1

Globus Online moves and syncs files

2

Globus Online notifies user

3

… and then made it simple to share big data off existing storage systems

Data Source

User A selects file(s) to share, selects user or group, and sets permissions

1

Globus Online tracks shared files; no need to move files to cloud storage!

2

User B logs in to Globus Online and accesses

shared file

3

Log into Globus Online

Alternate Logins

Login using InCommon

InCommon Login

Source endpoint

Destination endpoint

Activation

Transfer data

Share data

Set permissions

Manage Groups

Interactive login to command line interface:

Running commands remotely:

Using CLI with gsissh:

Globus Online CLI

$ ssh tuecke@cli.globusonline.org

$ ssh tuecke@cli.globusonline.org <command>

$ gsissh tuecke@cli.globusonline.org <command>

$ ssh tuecke@cli.globusonline.org scp –r –s 3 -D \ nersc#dtn:~/myfile* mylaptop:~/projects/p1 Task ID: 4a3c471e-edef-11df-aa30-1231350018b1 $ _

Usage is accelerating

Early Adopters

•  What is GCMU? –  Globus Connect version for easily creating (sharable) endpoints

on multi-user storage servers –  Packages a GridFTP server and MyProxy CA authentication

server, pre-configured for use with Globus Online

•  Why GCMU? –  Create transfer endpoints in minutes –  Avoid complex GridFTP install

•  To download: www.globusonline.org/gcmu

Globus Connect Multiuser (GCMU)

29

“We  used  GCMU  to  form  a  campus-­‐wide  GSI  authenAcaAon  service  spanning  mulAple  servers.  Now  my  users  have  a  fast,  easy  way  to  get  their  data  wherever  it  needs  to  go,  and  the  setup  process  was  trivial."    -­‐-­‐University  of  Michigan  

“As  a  resource  admin,  I've  found  GCMU  an  exceedingly  useful  tool....  With  GCMU,  seHng  up  a  GridFTP  server  and  handling  authenAcaAon  for  mulAple  users  is  easy."    -­‐-­‐Oak  Ridge  Na8onal  Lab  

We are a non-profit service provider to the non-profit

research community

Our challenge:

Sustainability

We are a non-profit service provider to the non-profit

research community

Globus Online Provider Plans

Support ongoing operations

Offer value-added capabilities Engage more closely with users

•  Endpoint operations management •  Branded web sites •  Alternate identity provider •  Usage reporting •  MSS optimizations •  Multiple GridFTP servers per endpoint

Provider Plans offer…

Starting at $20k per year

End User Plans

•  Basic: Free – File transfer and synchronization to/from

servers – Personal endpoints with Globus Connect – Access to shared endpoints created by others

•  Plus: $7/month (or $70/year) – Create and manage shared endpoints – Peer-to-peer transfer and sharing

Globus Platform-as-a-Service

Globus Nexus (Identity, Group, Profile)

…

Sharing Service

Transfer Service

Dataset Services

Globus Toolkit

Glo

bus

Onl

ine

API

s

Glo

bus

Con

nect

Our research is supported by:

U.S. DEPARTMENT OF

ENERGY

Questions?

Contact: support@globusonline.org

Providers: globusonline.org/provider-plans

www.globusonline.org