storage services let the data flow! nordunet 2008,.fi, 9 april 2008 jan meijer

Storage Services

Let the data flow!NorduNet 2008, .fi, 9 April 2008

Jan Meijer

2

Storage importance increasing

3

Something is brewing...

Storage as (a) utility service(s) within the

European higher ecudational and

research community

4

Spot the differences

high capacity network as a utility

compute cycles as a utility

data storage as a utility

5

Run that buzz by me again, please?

Storage as (a) utility service(s)

institutional / national / international scale

combining centralization, distribution, federation

creating a location and administrative domain agnostic ’make it so’ storage layer

6

So where does this fit in?

7

Drivers ’stuff’ as a service, the $500M data centers technology: because we can storing is cheaper then re-acquiring the

data. Sometimes reacquiring is impossible data explosion enabling new applications with old data limited pool of experts: institutional,

national, within Europe need to deal with an ever increasing

complexity and array of technologies increased reliability demands Do more with less money Storage as an enabler

8

the storage mix

Tiered storage HSM Tapes don’t spin..

9

Storage mix: Centralize, distribute, specialize

10

Revisiting that ’utility service’ buzz

’Half’ services: sites doing end-user frontends, end-user support

Block storage: iSCSI, FC (..), InfiniBand?

File systems: CIFS, (p)NFS, GPFS, ....

Archiving, data curation Full services: disaster

recovery, off site backup, storage frontends, ..?

11

Relevant tech trends

SSDs! Fast, high throughput, reliable

Improved understanding of storage reliability.

Storage middlewares, server virtualization

iSCSI Increasing demands for data

security Increasing pressure on availability

of data, services

12

Lucky us! Fast networks!

1 PetaByte = 8,000,000,000,000,000 bits

10 Gbit = 10,000,000,000 bits 1 day is 86400 sec

And this matters why? NorStore has 2x600TB netto in phase 1 1 minute of JPEG2000 video = 6GB LHC > 1 PB/year raw data production Data is kept!

13

Other challenges

proof of data ownership single name spacing, tight integration

between disparate storage elements what with storing data across borders? predictability (measurements) reliability of our storage archiving & data curation! AAI-enabling of storage access methods energy consumption

14

What are *we* doing?

Investigating what makes sense for our community

Coordinating, stimulating collaboration between sites (network!)

Learning...and putting that to use in

NorStore Experimenting

15

File Exchange

16

File exchange + EduGain + NorStore

17

Digital notary experiment

Investigating: Proof of data ownership Protecting data integrity Deployment options

18

What are folks doing?

.hu: national storage service, iSCSI

.pl: national storage service project

national research data storage NREN storage consolidation LHC PetaFlops = PetaBytes ...

19

Where is this going?

People want to share their data People want to collaborate across

organizational borders People want to have access to

their data The technology has to support this

Virtualizing storage into a service across institutional boundaries is

needed

20

To close with

http://www.terena.org/activities/tf-storage/

http://www.norstore.no/

Thank you

jan dot meijer at uninett dot no

BoF?