how to shift large amounts of data · why use b2stage? research challenges are getting larger and...

B2STAGE

How to shift large amounts of data

Version 3

June 2014

1

www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training

http://www.eudat.eu/

http://www.eudat.eu/b2stage

B2STAGE is part of EUDAT...

a pan-European initiative building a sustainable

cross-disciplinary and cross-national data

infrastructure providing a set of shared services for

accessing and preserving research data

supporting multiple research

communities by working closely

with them to deliver these

technical services as part of the

EUDAT Collaborative Data

Infrastructure (CDI) www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training



A truly pan-European Infrastructure

general data centres

community centres

representing all the associated

community data centres

Research Communities

National Data centres

Technology providers

Offering permanence,

persistence, reliability

and long term

solutions

www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training

http://www.kit.edu/english/

http://en.wikipedia.org/wiki/File:Epcc_logo.jpg

http://www.man.poznan.pl/online/en/

http://www.snic.vr.se/

http://www.sara.nl/index_eng.html

http://www.cines.fr/

http://www.umweltbundesamt.at/



Where is B2SHARE in the EUDAT suite?

B2STAGE represents an extension of the B2SHARE and offers communities a light approach to ingest and replicate data. Data ingested through B2STAGE is registered with a Persistent Identifier (PID) using the same mechanism adopted by B2SAFE www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training



A reliable, efficient, lightweight and easy-to-use service to ship large amounts of research data between EUDAT storage resources and workspace areas of high-performance computing systems.

5

B2STAGE is... B2STAGE does...

B2Stage can be used to simply ingest community data onto EUDAT resources using a high performance protocol, like GridFTP.

www.eudat.eu www.eudat.eu/b2stage

Why use B2STAGE?

Research challenges are getting larger and more complex : full-Earth climate simulation, coupled simulations of multiple organs in the human body, seismic analyses of earthquakes at continental scale and

Researchers’ data and compute demands are rising fast

Efficient shipping of data to high performance computing (HPC) workspaces is essential especially in distributed computing, where resources are geographically dispersed

6 www.eudat.eu www.eudat.eu/b2stage

Why use B2STAGE?

Facilitate transfer of large data collections from EUDAT storage resources to external HPC facilities.

Offers reliable, efficient, easy-to-use tools to manage data transfers.

Provides the means to re-ingest computational results back into the EUDAT infrastructure.

Ingests data sets onto EUDAT resources for long-term preservation.


Who can use B2STAGE?

Researchers can transfer large data collections from EUDAT storage resources to HPC facilities for processing.

Community Managers can replicate community data through a lightweight service and ingest data sets to EUDAT storage resources for long term preservation.


Why is B2STAGE unique?

The DSS is the only tool handling data transfer using PIDs.

Easy, reliable and fast solution for data ingestion and transfer onto and from EUDAT resources.


How can you use B2STAGE?

10

For more information please email: [email protected]

EUDAT offers B2STAGE to all registered researchers and interested communities enabling them to make use of the service to stage data out of EUDAT, and ingest computational results back.

Access to remote HPC facilities should be negotiated and

arranged by individual users in parallel.

To help researchers to use the B2STAGE service, EUDAT offers documentation, educational material and a service helpdesk.


mailto:[email protected]



B2STAGE User communities

VPH Community to ingest data onto EUDAT resources

Approximately 12TB will be ingested thought this service

NeuGRID and INCF are considering its adoption to replicate data

Collaboration with other e-infrastructures

VPH to transfer data across EUDAT, PRACE, EGI


B2STAGE currently...

The current version of B2STAGE offers:

data staging functionalities to easily and efficiently ship data across EUDAT storage resources and HPC facilities;

a powerful mechanism to ingest data onto EUDAT resources;

a script to facilitate the staging, the ingestion and the retrieving of PID information of transferred data.


Where does B2STAGE fit within EUDAT?

13

B2STAGE represents an extension of the B2SHARE and offers communities a light approach to ingest and replicate data. Data ingested through B2STAGE is registered with a Persistent Identifier (PID) using the same mechanism adopted by B2SAFE


Future features...

Optimization of transfers on the basis of data location within the EUDAT infrastructure (under evaluation).

Improvement of user experience with the Data Staging Script (i.e. data path autocompletion, multi-pid parallel handling, etc.).

Foster the collaboration with EGI and PRACE to develop cross-infrastructure usage: the B2STAGE will be the main service to enable the

interoperability of these infrastructures.


Thanks

For more info: www.eudat.eu/b2stage www.eudat.eu | http://www.eudat.eu/b2stage B2STAGE Training



how to shift large amounts of data · why use b2stage? research challenges are getting larger and...

Documents