2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Dealing with Data:Choosing a Good Storage Technology
for Your Application
Rick WagnerHPC Systems Manager
July 1st, 2014
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Application Focus
Storage choices should be driven byapplication need, not just what’s available.
But, applications need to adaptas they scale.
Writing a few small files to anNFS server is fine…
writing 1000’s simultaneously willwipe out the server.
If you use binary files,don’t invent your own format.
Consider HDF5.
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Storage Technologies
File Systems Devices Services
memory
block
Cloud
MySQL
CouchDB
ext4
NFS
Lustre
PVFS
FUSE
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Storage Technologies
File Systems Devices Services
memory
block
Cloud
MySQL
CouchDB
ext4
NFS
Lustre
PVFS
FUSE
Each has its own performance characteristics
Not all are available everywhere
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
File Systems
Classic access, POSIX, Windows
Most relevant:• Local• Remote
• NFS, CIFS• Parallel (Lustre, GPFS)
Local file systems are good for small and temporary files
Network file systems very convenient for sharing databetween systems
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Parallel File Systems
STATUS
PSU 1
PSU 2
FAN
RST
3433
3231
3635
2827
2625
3029
2221
2019
2423
1615
1413
1817
109
87
1211
43
21
65
IS 5030CONSOLEMGT
0
1
00
1
0 0
1
00
1
0 0
1
00
1
0 0
1
00
1
0
0
1
00
1
0 0
1
00
1
0 0
1
00
1
0 0
1
00
1
0
STATUS
PSU 1
PSU 2
FAN
RST
3433
3231
3635
2827
2625
3029
2221
2019
2423
1615
1413
1817
109
87
1211
43
21
65
IS 5030CONSOLEMGT
Rail 0
Rail 1
0
1
00
1
0 0
1
00
1
0 0
1
00
1
0 0
1
00
1
0
0
1
00
1
0 0
1
00
1
0 0
1
00
1
0 0
1
00
1
0
STATUS
PSU 1
PSU 2
FAN
RST
3433
3231
3635
2827
2625
3029
2221
2019
2423
1615
1413
1817
109
87
1211
43
21
65
IS 5030CONSOLEMGT
STATUS
PSU 1
PSU 2
FAN
RST
3433
3231
3635
2827
2625
3029
2221
2019
2423
1615
1413
1817
109
87
1211
43
21
65
IS 5030CONSOLEMGT
16 Compute Nodes
16 Compute NodesFlash I/O Node
Flash I/O Node
Each switch connected to its 6neighbors via 3 QDR links
LustreFilesystem
Dual 10GbE
Dual 10GbE
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Parallel File Systems
OSS72TB
32 OSS (Object Storage Servers) Provide 100GB/s Performance and >4PB Raw Capacity
Arista 750810G
Arista 750810G
Redundant Switches for Reliability and Performance
3 Distinct Network Architectures
OSS72TB
OSS72TB
OSS72TB
64 Lustre LNET Routers100 GB/s
Mellanox 5020 Bridge12 GB/s
MDS
MDS
Myrinet 10G Switch25 GB/s
MDS
GORDONIB cluster
TRITONMyrinet cluster
TRESTLES IB cluster
Metadata Servers
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
A Cautionary Tale
http://www.youtube.com/watch?v=gDfLXAtRJfY&feature=youtu.be
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Devices
Raw block device (/dev/sdb) or RAM FS (/dev/shm)
Useful in specific cases, like fast scratch
Can be very good for small I/O
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Services
Things accessed programmatically
Frequents the last thought for HPCapplications: A MISTAKE
DatabasesCloud storage (Amazon S3)Document storage (MongoDB, CouchDB)
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Know What You Need
http://www.youtube.com/watch?v=F4OIDszDA9E
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Order of Magnitude Guide
Storage file/directory file sizes BW IOPs
Local HDD 1000s GB 100 MB/s 100
Local SSD 1000s GB GB/s 10000
RAM FS 10000s GB GB/s 10000
NFS 100s GB 100 MB/s 100
Lustre/GPFS 100s TB 100 GB/s 1000
Cloud Infinite TB 10 GB/s 0
DB N/A N/A N/A 10000
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Choosing
My application needs to:
I should consider:
Write a checkpoint dump from memory from a large parallel simulation.
A parallel file system and a binary file formatlike HDF5.
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Choosing
My application needs to:
I should consider:
Run analysis on remote systems and return the results to a web portal for users.
Cloud storage for results and input, and local scratch space for the job.
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Choosing
My application needs to:
I should consider:
Randomly access many small files, or read and write small blocks from large files.
A database, RAM FS, or local scratch space.
2013 Summer Institute: Discover Big Data, August 5-9, San Diego, California
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Many Boxes Make a Sad Panda
http://www.youtube.com/watch?v=N2zK3sAtr-4
Database logos courtesy of RRZEiconshttp://commons.wikimedia.org/