Upload File

architectureofa)) nextgenerationparallel)filesystem) · pdf filewhyparallelfile)system?...

  • Home
  • Documents
  • Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •
46
Architecture of a NextGeneration Parallel File System

Upload: hangoc

Post on 16-Mar-2018

222 views

Category:

Documents


2 download

Report

TRANSCRIPT

Page 1: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Architecture  of  a    Next-­‐Generation  Parallel  File  System  

Page 2: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

-­‐  Introduction  -­‐  Whats  in  the  code  now  -­‐  futures  

Agenda  

Page 3: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

An  introduction  

Page 4: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

What  is  OrangeFS?  

•  OrangeFS  is  a  next  generation  Parallel  File  System  •  Based  on  PVFS  •  Distributes  file  data  across  multiple  file  servers  

leveraging  any  block  level  file  system.  •  Distributed  Meta  Data  across  1  to  all  storage  servers    •  Supports  simultaneous  access  by  multiple  clients,  

including  Windows  using  the  PVFS  protocol  Directly  •  Works  w/  standard  kernel  releases  and  does  not  require  

custom  kernel  patches  •  Easy  to  install  and  maintain  •  Happy  member  of  the  Linux  Foundation  

Page 5: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Why  Parallel  File  System?  

HPC  –  Data  Intensive   Parallel  (PVFS)  Protocol  

•  Large  datasets  

•  Checkpointing  

•  Visualization  

•  Video  

•  BigData  

 

Unstructured  Data  Silos   Interfaces  to  Match  Problems  

•  Unify  Dispersed  File  Systems  

•  Simplify  Storage  Leveling  

§  Multidimensional  arrays  

§   typed  data    §  portable  formats  

Page 6: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Original  Design  Goals  §  Scalable  §  Configurable  file  striping  §  Non-­‐contiguous  I/O  patterns  §  Eliminates  bottlenecks  in  I/O  path  §  Does  not  require  locks  for  metadata  ops  §  Does  not  need  locks  for  non-­‐conflicting  applications  

§  Usability  §  Very  easy  to  install,  small  VFS  kernel  driver  §  Modular  design  for  disk,  network,  etc  §  Easy  to  extend  -­‐>  Hundreds  of  Research  Projects  have  

used  it,  including  dissertations,  thesis,  etc…  

Page 7: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

OrangeFS  Philosophy    •  Focus  on  a  Broader  Set  of  Applications  •  Customer  &  Community  Focused  

•  (~300  Member  Strong  Community  &  Growing)  

•  Completely  Open  Source  •  (no  closed  source  add-­‐ons)  

•  Commercially  Viable  •  Enable  Research    

Page 8: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Configurability  

Performance  

Consistency  Reliability  

Page 9: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

System  Architecture  

•  OrangeFS  servers  manage  objects  •  Objects  map  to  a  specific  server  •  Objects  store  data  or  metadata  •  Request  protocol  specifies  operations  on  one  or  

more  objects    

•  OrangeFS  object  implementation  •  Distributed  DB  for  indexing  key/value  data  •  Local  block  file  system  for  data  stream  of  bytes  

Page 10: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Current  Architecture  

Client   Server  

Page 11: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

In  the  Code  now  

Page 12: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Server  to  Server  Communications  (2.8.5)    

Traditional  Metadata  Operation  

 Create  request  causes  client  to  communicate  with  all  servers  O(p)  

Scalable  Metadata  Operation  

 Create  request  communicates  with  a  single  server  which  in  turn  communicates  with  other  servers  using  a  tree-­‐based  protocol  O(log  p)  

Mid  

Client  

Serv  

App  

Mid  Mid  

Client  

Serv  

Client  

Serv  

App  

Network  

App  

Mid  

Client  

Serv  

App  

Mid  Mid  

Client  

Serv  

Client  

Serv  

App  

Network  

App  

Page 13: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Recent  Additions  (2.8.5)  

SSD  Metadata  Storage   Replicate  on  Immutable  (file  based)  

Windows  Client  

Supports  Windows  32/64  bit  

Server  2008,  R2,  Vista,  7  

Page 14: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Direct  Access  Interface    (2.8.6)  

•  Implements:  •  POSIX  system  calls  •  Stdio  library  calls  

•  Parallel  extensions  •  Noncontiguous  I/O  •  Non-­‐blocking  I/O  

•  MPI-­‐IO  library  •  Found  more  boundary  

conditions  fixed  in  upcoming  2.8.7  

App  

Kernel  PVFS  lib  

Client  Core  

Direct  lib  

PVFS  lib  Kernel  

App  

IB  

TCP  

Page 15: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

File  System  File  System  File  System  

Direct  Interface  Client  Caching    (2.8.6)    

•  Direct  Interface  enables  Multi-­‐Process  Coherent  Client  Caching  for  a  single  client  

File  System  File  System  

Client  Application  

Direct  Interface    

Client  Cache  

File  System  

Page 16: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

WebDAV  (2.8.6  webpack)  

PVFS  Protocol  

Orang

eFS  

Apache  

•  Supports  DAV  protocol  and  tested  with  the  Litmus  DAV  test  suite  

•  Supports  DAV  cooperative  locking  in  metadata    

Page 17: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

S3  (2.8.6  webpack)  

PVFS  Protocol  

Orang

eFS  

Apache  

•  Tested  using  s3cmd  client  •  Files  accessible  via  other  access  methods  

•  Containers  are  Directories  

•  Accounting  Pieces  not  implimented  

Page 18: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Summary  -­‐  Recently  Added  to  OrangeFS  

•  In  2.8.3  •  Server-­‐to-­‐Server  Communication  •  SSD  Metadata  Storage  •  Replicate  on  Immutable  

•  2.8.4,  2.8.5  (fixes,  support  for  newer  kernels)  •  Windows  Client  •  2.8.6  –  Performance,  Fixes,  IB  updates  •  Direct  Access  Libraries  (initial  release)  •  preload  library  for  applications,  Including  Optional  Client  

Cache  •  Webpack  •  WebDAV  (with  file  locking),  S3  

Page 19: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Coming  Soon…  

Page 20: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Hadoop  JNI  Interface  (2.8.x)  

•  OrangeFS  Java  Native  Interface  

•  Extension  of  Hadoop  File  System  Class  –>JNI  

•  Replicate  on  Immutable  

•  Buffering  

•  Distribution  •  Based  on  work  by  

 Wittawat  Tantisiriroj,      Swapnil  Patil,    Garth  Gibson,  CMU    

PVFS  Protocol  

Page 21: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Distributed  Directory  Metadata  (2.9.0)  

DirEnt1  

DirEnt2  

DirEnt3  

DirEnt4  

DirEnt5  

DirEnt6  

DirEnt1  DirEnt5  

DirEnt3  

DirEnt2  

DirEnt6  

DirEnt4  

Server0  

Server1  

Server2  

Server3  

Extensible Hashing  

u  State  Management  based  on  Giga+  u  Swapnil  Patil  and  Garth  Gibson    CMU  

u  Improves  access  times  for  directories  with  a  very  large  number  of  entries  

u  Potential  Extension  to  First  Stripe  of  Data  (10x+)  

Page 22: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Cert  or  Credential  

Signed  Capability  I/O  

Signed  Capability  

Signed  Capability  I/O  

Signed  Capability  I/O  

•  Untrusted  Client  support  at  the  pvfs  protocol  level  •  Working  on  how  to  simplify  certificate  integration  

•  Make  it  so  people  will  want  to  use  

OpenSSL  

Page 23: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

File  Data  Replication  

•  Forked  Flow  •  small  processing  overhead  •  Bandwidth  

•  Configurable  •  Number  of  replicas  (O  ..  N)  •  Update  modes    •  Immutable  (since  2.8.3)  •  Real  Time  (2.9.x)  

Page 24: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Configurable  Locking    (2.8.x)  

•  File  Locking  already  in  WebDAV  •  Bringing  Locking  concept  into  Client  &  KM  •  Fcntl  /  flock  •  Locks  maintained  across  server  in  MD  

•  Initially  support  client  and  NFS  locks  through  KM  

•  Eventually  integrate  with  WebDAV  locks        (if  semantics  work  out)  

Page 25: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

OrangeFS  on  the  Cloud  

•  Build  a  Scale  out  Shared  File  System  on  top  of  Cloud  Resources  

•  Cloud  Formation  Templates    •  AWS,  EC2+EBS    •  OpenStack  –  Heat  •  Others  as  requested  

•  Spin  up  n  number  of  storage  servers,  each  with  8  or  so  EBS  volumes  and  create  a  single  large  file  system  in  Minutes.  

 

Page 26: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

On  The  Fence  (2.9.x  or  3.x)  

Page 27: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

•  Modular  infrastructure  to  easily  build  background  parallel  processes  for  the  file  system  Used  for:  •    Gathering  Stats  for  Monitoring  •  Usage  Calculation  (can  be  leveraged  for  Directory  

Space  Restrictions,  chargebacks)  •  Background  safe  FSCK  processing  (can  mark  bad  

items  in  MD)  •  Background  Checksum  comparisons    •  Parallel  Cross  File  System  rsync  •  Etc…  

 

Background  Parallel  Processing  Infrastructure  (2.9.x)  

Page 28: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Admin  REST  interface  /  Admin  UI    

PVFS  Protocol  

Orang

eFS  

Apache  

REST

 

(2.x.x?)  

Page 29: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Attribute  Based  Metadata  Search    

•  Client  tags  files  with  Keys/Values    •   Keys/Values  indexed  on  Metadata  Servers  •   Clients  query  for  files  based  on  Keys/Values  •   Returns  file  handles  with  options  for  filename  

and  path  

Key/Value  Parallel  Query  

Data  

Data  

File  Access  

OrangeFS  NEXT  

Page 30: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

OrangeFS    3  

Page 31: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

OrangeFS  3    •  Goals  •  Scalability    •  to  tens  of  thousands  of  storage  nodes  

•  Security  •  support  systems  with  untrusted  root  as  clients  

•  Fault  Accepting  Architecture  •  Diverse  Access  Methods  •  Monitoring  and  Management  •  Storage  Tiers  •  Wide  Area  Capabilities  

Page 32: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Replication  /  Redundancy    •  Redundant  Metadata  

•  seamless  recovery  after  a  failure  •  Redundant  objects  from  root  directory  down  •  Configurable  •  Distributed  Master  Objects  

•  Fully  Redundant  Data    •  Extension  of  2.9.x  work  •  Configurable  •  Number  of  replicas  (O  ..  N)  •  Update  modes  (continuous,  on  close,  on  immutable,  none)  

•  Emphasis  on  continuous  operation  •  Fault  Accepting  Architecture  

OrangeFS    3  

Page 33: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Policy  Based  Location  

•  User  defined  attributes  for  servers  and  clients  •  Stored  in  SID  cache  

•  Policy  is  used  for  data  location,  replication  location  and  multi-­‐tenant  support    

•  Completely  Flexible  •  Rack  •  Row  •  App  •  Region    

OrangeFS  3  

Page 34: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

•  In  an  Exascale  environment    •  thousands  of  I/O  servers  •  Servers  cannot  know  about  all  other  servers.  

•  Servers  will  discover    a  subset  of  their  neighbors    •  At  Startup  or  Cached  •  Servers  will  learn  about  unknown  servers  on  an  as  

needed  basis  and  cache  them.    •   Similar  to  DNS  query  mechanisms  (root  servers,  

authoritative  domain  servers).  

•  SID  Cache,  in  memory  db  to  store  server  attributes  

Server  Location  /  SID  Mgt  OrangeFS  3  

Page 35: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

•  An  OID  (object  identifier)    •  128-­‐bit  UUID  that  is  unique  to  the  data-­‐space  

•  An  SID  (server  identifier)    •  128-­‐bit  UUID  that  is  unique  to  each  server.  

•  No  more  than  one  copy  of  a  given  data-­‐space  can  exist  on  any  server  

•  The  (OID,  SID)  tuple  is  unique  within  the  file  system.  

•  (OID,  SID1),    (OID,  SID2),  (OID,  SID3)  are  copies  of  the  object  identifier  on  different  servers.  

Handles  -­‐>  UUIDs  OrangeFS  3  

Page 36: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Data  Migration  /  Mgt  

•  Built  on  Redundancy  &  DBG  processes  •  Migrate  objects  between  servers  

•  De-­‐populate  a  server  going  out  of  service  •  Populate  a  newly  activated  server  (HW  lifecycle)  •  Moving  computation  to  data  

•  Hierarchical  storage  •  Use  existing  metadata  services  

•  Possible  -­‐  Directory  Hierarchy  Cloning  •   Copy  on  Write  (Dev,  QA,  Prod  environments  with  high  %  data  

overlap)  

OrangeFS  3  

Page 37: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Hierarchical  Data  Management  

Archive  Intermediate  Storage  NFS  

Remote  Systems  exceed,  OSG,  Lustre,  GPFS,    Ceph,  Gluster  

HPC  OrangeFS  

Metadata  OrangeFS  

Users  

OrangeFS  3.x  

Page 38: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Beyond  OrangeFS  3  

Page 39: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Extend  Capability  based  security  

•  Enable  certificate  level  access  (in  process)  •  Federated  access  capable  •  Can  be  integrated  with  rules  based  access  

control  •  Department  x  in  company  y  can  share  with  

Department  q  in  company  z    •  rules  and  roles  establish  the  relationship  •  Each  company  manages  their  own  control  of  who  is  in  

the  company  and  in  department  

Page 40: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

SDN  -­‐  OpenFlow  

•  Working  with  OpenFlow  researchers  at  Clemson  

•  OF  separates  the  control  plane  from  delivery,  gives  ability  to  control  network  with  SW  

•  Looking  at  bandwidth  optimization  leveraging  OF  and  OrangeFS  

Page 41: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

ParalleX  

ParalleX  is  a  new  parallel  execution  model  •  Key  components  are:  •  Asynchronous  Global  Address  Space  (AGAS)  •  Threads  •  Parcels  (message  driven  instead  of  message  passing)  •  Locality    •  Percolation    •  Synchronization  primitives  •  High  Performance  ParalleX  (HPX)  library  

implementation  written  in  C++  

Page 42: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

PXFS  

•  Parallel  I/O  for  ParalleX  based  on  PVFS  •  Common  themes  with  OrangeFS  3  

•  Primary  objective:  unification  of  ParalleX  and  storage  name  spaces.  •  Integration  of  AGAS  and  storage  metadata  subsystems  •  Persistent  object  model  

•  Extends  ParalleX  with  a  number  of  IO  concepts  •  Replication  •  Metadata  

•  Extending  IO  with  ParalleX  concepts  •  Moving  work  to  data  •  Local  synchronization  

•  Effort  with  LSU,  Clemson,  and  Indiana  U.  •  Walt  Ligon,  Thomas  Sterling  

Page 43: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Community  

Page 44: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Learning  More  

•  www.orangefs.org  web  site  •  Releases  •  Documentation  •  Wiki  

•  Dell  OrangeFS  Reference  Architecture  Whitepaper  

•  pvfs2-­‐users@beowulf-­‐underground.org  •  Support  for  users  

•  pvfs2-­‐developers@beowulf-­‐underground.org  •  Support  for  developers  

Page 45: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Support  &  Development  Services  

•  www.orangefs.com  &  www.omnibond.com  •  Professional  Support  &  Development  team  •  Buy  into  the  project    

Page 46: Architectureofa)) NextGenerationParallel)FileSystem) · PDF fileWhyParallelFile)System? HPC)–)Data)Intensive) Parallel)(PVFS))Protocol) • Large+datasets+ • Checkpointing+ •

Speculative Memory Checkpointing

Checkpointing strategies for parallel jobs

Armazenamento distribuído de dados e checkpointing de

Checkpointing HPC Applications - JLESC › downloads › 5th-workshop › jlesc...Checkpointing HPC Applications Thomas Ropars [email protected] Universit e Grenoble Alpes 2016

Checkpointing 2.0 Compiler-Assisted Checkpointing Uncoordinated Checkpointing

Evaluating Overheads of Integrated Multilevel Checkpointing

IBM Streams V4.1 and Incremental Checkpointing

By Ali Alskaykha PARALLEL VIRTUAL FILE SYSTEM PVFS PVFS Distributed File System:

Implement Checkpointing for Android (ELCE2012)

FAULT TOLERANT SYSTEMS Chapter 6 – Checkpointing I

Checkpointing Transaction-based Distributed Shared Memory ...static.cs.brown.edu/research/pubs/theses/masters/1999/chen.pdf · Checkpointing Transaction-based Distributed Shared Memory

Distributed Algorithms for Message-Passing Systemsdisplexity/Angers/Distr-algo-presentation.pdf · 8.2.3 On Distributed Checkpointing Algorithms .....198 8.3 Checkpointing Algorithms

Stabilizers: A Modular Checkpointing Abstraction for ... · PDF fileStabilizers: A Modular Checkpointing Abstraction for Concurrent Functional Programs LukaszZiarek Philip Schatz Suresh

Diskless Checkpointing

Checkpointing and Rollback-Recovery for Distributed Systems

Hypervisor-Assisted Application Checkpointing for High Availability

Non-Blocking Checkpointing for Optimistic Parallel ...pdfs.semanticscholar.org/cb65/2fa4c5a21af558b2ade... · Non-blocking checkpointing has been widely explored in the context of

Coordinated Checkpointing Presented by Sarah Arnold 1

Checkpointing 2.0

Privacy-preserving Virtual Machine Checkpointing Mikhail I

Pooja Shetty Usha B Gowda. Network File Systems (NFS) Drawbacks of NFS Parallel Virtual File Systems (PVFS) PVFS components PVFS application

Checkpointing & Rollback Recovery

Koo-Toueg coordinated checkpointing algorithm A

Checkpointing the Un-checkpointable: the Split-Process

Checkpointing With Minimal Recovery In

Online Checkpointing with Improved Worst-Case Guarantees

PROTOCOLO DE CHECKPOINTING - TCC - CAUAN KASPER

The Next Evolution of PVFS

A Look at PVFS, a Parallel File System for Linux

Rebound: Scalable Checkpointing for Coherent Shared Memory

PVFS (parallel Virtual file system)

Fault Tolerance and Checkpointingcds.iisc.ac.in/wp-content/uploads/FaultTolerance.pdf · Fault Tolerance and Checkpointing - Sathish Vadhiyar. Introduction Checkpointing? storing

Node Failure Resiliency for Uintah Without Checkpointing

Diskless Checkpointing on Super-scale Architectures

Uncoordinated Checkpointing Without Domino Effect for Send