BioHPC Cloud Storage & Backup

1

[web] portal.biohpc.swmed.edu

[email] biohpc-help@utsouthwestern.edu

Updated for 2015-03-09

Overview

2/32

What is the cloud service/storage?

Services we provide to BioHPC user

– BioHPC cluster

– BioHPC Lamella Storage System

– BioHPC File Exchange

How can I use BioHPC cloud storage?

– Basic File Management

– File Synchronization

– File Transfer

– File Sharing

– Lab/core manager account

Backup

–What is backed up?

–How does backup count against usage?

Slides and guide online at:

https://portal.biohpc.swmed.edu

- Training -> Slides & Handouts

- Guides -> BioHPC Cloud Storage

3/32

Cloud Service ( Cloud Computing and Cloud Storage )

Image retrieved from http://giannisgiataganas.blogspot.com/2010/06/lets-talk-about-cloud.html, http://www.nec.com/en/global/solutions/cloud/portfolio/storage.html

Cloud storage (one of the fundamental components of cloud computing) is a service model in which data is maintained, managed and backed up remotely and made available to users over a network (typically the Internet).

Cloud computing refers to the delivery of computing resources over the network.

Security

Storage space

Other Limitations of Web-based Storage Systems

4/32

Why We Need the BioHPC Cloud Service

UTSouthwestern General Cloud Policy: UTSW information must never be stored in unapproved online file storage systems.

OneDrive for Business – 1TB Free for UTSW users, but max file size is 10GB, slow.vs

BioHPC Cloud: 150GB + access to BioHPC storage systems >2.5PB

• Max upload/download speed ~ 10 MB/s• Web browsers do not support folder uploading• Upload/download limit for single file (size)

50 GB

BioHPC Storage – Standard Users

5/32

As a BioHPC user, you will have access to:

• BioHPC Cluster/home2/username 50 GB / user/work/department/group/username 5 TB / per user/project/department/group/username 5 TB / per group*

• BioHPC File Exchange (web-interface)https://cloud.biohpc.swmed.edu 50 GB / user, local storage

• BioHPC Lamella Cloud Storage (web-interface), on campus only, private cloudhttps://lamella.biohpc.swmed.edu 100 GB / user, local storageGateway to BioHPC Cluster (via FTP, SAMBA or WebDAV*)

* Can be increased on PI request with Dept. Chair approval.

BioHPC Storage – Core Users

6/32

Some core facilities provide access to BioHPC for their users to transfer data etc.

The core decides the amount and type of storage to provide to their users, e.g.

TIBIR WBMF Core:

/project/TIBIR/WBMF_Core/<username> 250GB / core user

This is also your home directoryNo separate home2 or work space

Storage allocation and usage is at the discretion of the core, not BioHPC.

BioHPC Storage Backup

7

Since January we perform mirror back up on:

/home2Twice weekly (Mon & Wed)2 copies – home2 usage counts 3x against lab storage allocation

/workWeekly (Fri->Sat)1 copy – work usage counts 2x against lab storage allocationExcludes some large users

Mirror backupsCopy of your files at the time the backup runs. No old versions.

Email biohpc-help@utsouthwestern.edu to recover files.

BioHPC Storage Backup

8

/project

Backup will begin this month

Only requested files – not all of /project !

PI needs to request backup to biohpc-help@utsouthwestern.eduLet us know which directories to backup, and how often to run the backup.

Weekly incremental backup by default.Old versions of files are kept available.

Backup usage counts against lab storage allocation.

Self-service access to backup to retrieve old files etc. from /project1

BioHPC Storage Cheat Sheet

9

https://portal.biohpc.swmed.edu- Guides -> Storage Cheat Sheet

Summary of storage available

How to check quotas / usage

Optimizing large files on /project for parallel cluster jobs (advanced HPC users)

BioHPC Cluster Storage & Cloud Storage

10/32

50 GBBioHPC File Exchange

BioHPC Cluster

Compute nodesLamella Cloud Storage/Gateway

UTSW private cloud

1 way access2 way access

SSH Web SAMBA FTP

BioHPC Cluster Yes No No No

Lamella Cloud Storage No Yes No No

Lamella Cloud Gateway No Yes Yes* Yes*

BioHPC File Exchange No Yes No No

* Do not have access to lamella local cloud storage

Initial Setup for Lamella Gateway

11/32

* Mount External Storage

— Setting up Lamella to access BioHPC cluster (home, project, and work space)

lysosome username password homeproject

For home leave blank

For private project space:department/lab/user

For lab shared project space:department/lab/shared

BioHPC Endosome/Lysosome

What can we do with BioHPC cloud storage system

12/32

• Basic File Managementview, move, rename, download, upload, delete your files and etc.

• File Synchronizationowncloud client

• File TransferClusterSambaFTPWebDAV

• File SharingShared FolderBioHPC File Exchange (https://cloud.biohpc.swmed.edu)BioHPC Lamella Cloud Storage (https://lamella.biohpc.swmed.edu)

• BioHPC manager accountA lab manager can share data to non-BioHPC users

Navigation bar:

– Allows navigation between different parts

New/Upload button:

– Allows you to create new files or upload existing ones from your device

– you can also drop files from Explorer or Finder onto the file view.

– With New button, you can create a text file, folder or download a file from the provided URL.

Search/Settings:

– Search allows you to look for files and directories

New/Upload button:

– Clicking on small thumbnails or icons of the files/directory

– Select multiple files can e deleted or downloaded as ZIP

Viewing Files

– Can display text files, PDFs, and image files

– If displaying of a file is not possible, a download process will start

Moving files

– Dragging and dropping files/directory

Basic File Management

9/32

File Synchronization

14/32

BioHPC Cluster(work or project folder)

Lamella Cloud Storage

< 100 GB

> 100 GB

15/32

File Synchronization – know your data

Before setting up file synchronization:

1) Estimate the total size of your dataif the total size of your data is always < 100GB, you may directly

synchronize your data to lamella, otherwise use your BioHPC project directory.

2) Estimate the total free space on your local diskif the total size of your data is always < free space on your local disk,

you may synchronize everything from lamella, otherwise chose what to sync.

File Synchronization – use synchronization software

16/32

download and install owncloud client from https://portal.biohpc.swmed.edu/content/software/

connect to lamella.biohpc.swmed.edu -> username/password

* File Synchronization

File Synchronization

17/32

Chose What to Sync

total data size < local disk space

total data size > local disk space

Total data size = actual file stored in lamella locally (up to 100 GB) + mounted folders (up to 10 TB)

Anything your deleted from your PC will be removed from BioHPC cloud storage server as well.

* File Synchronization

18/32

If your are using a BioHPC thin client or workstation you have direct access to BioHPC storage.

File Transfer

Separate training session – check calendar

Accessing BioHPC Storage Directly from thin client, workstation

19/32

Computer -> Map Network Drive

Folder is: \\lamella.biohpc.swmed.edu\username home\\lamella.biohpc.swmed.edu\project\department\group project\\lamella.biohpc.swmed.edu\work\department\group work

Check ‘Connect using different credentials’Enter your BioHPC username and password when prompted.

File Transfer

Accessing BioHPC Storage Directly from Windows via Samba

Accessing BioHPC Storage Directly from Mac OSX via Samba

20/32

Finder -> Go -> Connect to Server

Folder is: smb://lamella.biohpc.swmed.edu/username homesmb://lamella.biohpc.swmed.edu/project/department/group projectsmb://lamella.biohpc.swmed.edu/work/department/group work

Enter your BioHPC username and password when prompted.

File Transfer

File Transfer

21/32

Host: lamella.biohpc.swmed.eduUsername: Your BioHPC usernamePassword: Your BioHPC passwordPort: Empty

Transfer data to/from BioHPC cloud storage via FTP

22/32

Accessing your Files from Windows via WebDAV

1) Increase the maximum size of WebDav downloads to 4GB

regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\FileSizelimitinBytes

By default Windows limits the maximum size of WebDav downloads to ~50 MB

File Transfer

Not recommended but still useful when you need direct access to the local

storage of BioHPC Lamella or File Exchange system from your PC

23/32

2) Mount your lamella storage to PC

Start -> Computer -> Map Network Drive

* You could mount BioHPC File Exchange storage to your PC in a similar way .

(https//cloud.biohpc.swmed.edu/remote.php/webdav)

File Transfer

Accessing your Files from Windows via WebDAV

24/32

File Transfer

Cyberduck – A Client support FTP, WebDAV for both Mac and Windows

• Download and install Cyberduck from https://portal.biohpc.swmed.edu/content/software/

• Once installed into your 'Applications' folder, start Cyberduck, choose 'Open Connection' and configure as in the image below:

Accessing your Files from Mac OSX via WebDAV

25/32

Uploading / Downloading Speed

Max Uploading andDownloading Speed*

Lamella Cloud Storage(Samba)

400MB/s (10GbE)100MB/s (1GbE)

BioHPC File-Exchange(webDav)

~ 10 MB /s

Lamella Cloud Storage(webDav)

~ 100 MB /s

Lamella Cloud Storage(FTP )

300 MB - 1.4 GB/s

• Upload and download speed depends on your actual network connection• We don’t set a limitation on max concurrent sessions

File Transfer

File Sharing

26/32

• Sharing data with all BioHPC user/work/shared /project/shared

• Sharing data inside your department/work/department_name/shared /project/department_name/shared

• Sharing data inside your group/work/department_name/group_name/shared /project/department_name/group_name/shared

If you have a workstation connected to our BioHPC cloud or you know how to use

command line to connect to BioHPC cloud, you can share your data on cluster directly.

Shared folders had already been set up at different levels

27/32

Sharing data with UTSW collaborators

File Sharing

BioHPC Lamella Cloud Storage

https://lamella.biohpc.swmed.edu

Share with linkShare with BioHPC user or group(Email notification)

28/32

Sharing data with non UTSW collaborators

File Sharing

BioHPC File Exchange

https://cloud.biohpc.swmed.edu

Share with link(Email notification, setup password & expiration date)

29/32

A lab manager ( BioHPC manager account ) sharing data to non-BioHPC users

Lysosome

Lab Manager Lab manger account : only provide storage space under

/project/department/group/manager_name

and this is the home directory of manager account

Lab Facilities, e.g.: scanning EM

images

non-BioHPC user A

non-BioHPC user B

non-BioHPC user C

directly saved to manager’s home directory

30/32

A lab manager ( BioHPC user ) sharing data to non-BioHPC users

Lysosome

Lamella Cloud Storage

Lab Manager

non-BioHPC user A

non-BioHPC user B

non-BioHPC user C

Share data for user to download

A lab manager sharing data to non-BioHPC users

31/32

Step 1: mount manager’s home directory to your lab facility to save data

Support PC, Mac and Linux/Unix

A lab manager sharing data to non-BioHPC users

32/32

Step 2: Sharing data with non BioHPC users with a linkpassword protection (recommended) set up an expiration date (recommended) email notification

A lab manager sharing data to non-BioHPC users

33/32

Step 3: non BioHPC users download data by providing correct password

For More Information

• Read related articles from https://portal.biohpc.swmed.eduLamella - Internal Cloud StorageBioHPC File Exchange - External Cloud Storage

• Email the ticket system: biohpc-help@utsouthwestern.edu