rakuten leofs - distributed file system
DESCRIPTION
LeoFS has High Performance, High Reliability, High Scalability, file system developed by Rakuten.TRANSCRIPT
![Page 1: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/1.jpg)
1
S3 Compatible Storage “LeoFS”
Rakuten. Inc, RIT Yosuke Hara 25/05/2012
![Page 2: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/2.jpg)
2
Table of Contents
1. Motivation
2. Overview
3. Inside of LeoFS
4. WEB Console
![Page 3: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/3.jpg)
3
1. Motivation
![Page 4: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/4.jpg)
4
Which is suitable storage for storing the media files?
? Low ROI Possibility of SPOF Storage Expansion is difficult during increasing data
Why NFS?
![Page 5: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/5.jpg)
5
![Page 6: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/6.jpg)
6
Object Storage Farm for IaaS
![Page 7: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/7.jpg)
7
2. Overview
![Page 8: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/8.jpg)
8
Storage
Gateway
Manager
Overview
![Page 9: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/9.jpg)
9
LeoFS-Manager
LeoFS-Gateway w/Cache Server
LeoFS-Storage
REST over HTTP RPC
Request from Web Application(s)
META Object Store
Storage Engine/Router
META Object Store META Object Store
RPC
Storage Engine/Router Storage Engine/Router
Load Balancer
S3-API
SNMP
WEB Console
System Layout
![Page 10: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/10.jpg)
10
LeoFS-Manager
LeoFS-Gateway w/Cache Server
LeoFS-Storage
REST over HTTP RPC
Request from Web Application(s)
META Object Store
Storage Engine/Router
META Object Store META Object Store
RPC
Storage Engine/Router Storage Engine/Router
Load Balancer
S3-API
SNMP
GUI Console
Gateway HTTP
Request/Response Handling +
w/Object Cache (AWS S3-API)
Manager Cluster
Management
Ring Watcher Node Watcher
Storage Object Storage, Meta data Storage
+ Replicator/Recoverer
System Layout
![Page 11: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/11.jpg)
11
3. Inside of LeoFS
![Page 12: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/12.jpg)
12
Manager Cluster
Storage Cluster (multi master)
Gateway (stateless)
Process Monitor
Erlang RPC
HTTP
Erlang RPC
Erlang RPC
Architecture
![Page 13: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/13.jpg)
13
LeoFS Gateway REST over HTTP (S3-API)
membership (fault-detection)
redundant-manager replicator
queue
Storage Engine
Object Storage Metadata Storage
LeoFS-Storage
read-repairer
redundant-manager membership (fault-detection)
get put delete head
Architecture - Gateway/Storage
RPC
RPC RPC
Cacher
Architecture
![Page 14: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/14.jpg)
14
Erlang Mnesia
Architecture - Manager Cluster
Gateway / Storage Cluster
RING Member / Cluster State
Auth / ACL
Process Monitor
Architecture
![Page 15: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/15.jpg)
15
Architecture
![Page 16: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/16.jpg)
16
High Cost Performance
Monolithic Storage System
Storage Engine For Unstructured Files
Traffic Restrain Mechanism
> File Cache System (Gateway Plugin)
Three “HIGH”
![Page 17: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/17.jpg)
17
Three “HIGH”
High Reliability
NO SPOF
Split Brain Measure
“Erlang OTP” > Nine Nines (99.9999999%)
![Page 18: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/18.jpg)
18
Three “HIGH”
High Scalability
Elastic Storage System
> Able to dynamic attach/detach nodes
> Able to over 100-nodes
> NOT Mesh-connected Mutual Servers
![Page 19: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/19.jpg)
19
Cache Mechanism
![Page 20: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/20.jpg)
20
High I/O efficiency
Low Latency
Gateway Buffer Pool
Gateway
from Client
Slab Alloc Skiplist
request
{$filename, $etag}
response
match: {ok, match} NOT match: {ok, $metadata, $body,}
Cache Mechanism
![Page 21: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/21.jpg)
21
Storage Engine
![Page 22: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/22.jpg)
22
Metadata + Object Storage
Metadata : Keeps an in-memory index of all data. Object Storage : Log structured (append-only) object store.
Object’s Attribute Storage (metadata) Object Storage
LeoFS-Storage
Storage Engine / Replicator / Recoverer
Storage Engine
![Page 23: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/23.jpg)
23
< META DATA >IDFilenameOffsetSizeChecksum
Metadata
Data
Object Container
Retrieve an object from Storage
Header
File
Footer
< META DATA >IDFilenameOffsetSizeChecksum
Storage Engine
Storage Engine
![Page 24: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/24.jpg)
24
Meta Data Server
Append a File
Insert an object into Storage
Data
Add a Metadata
Storage Engine
Storage Engine
![Page 25: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/25.jpg)
25
Reduce unnecessary files
Compaction
Storage Engine
![Page 26: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/26.jpg)
26
4. WEB Console
![Page 27: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/27.jpg)
27
File Manager
Node Stats
Log Search
WEB Console
Web Console
![Page 28: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/28.jpg)
28
Web Console
Log Analyzer / Searcher
Node State Monitor
![Page 29: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/29.jpg)
29
Storage Gateway Manager
Logstatsh
or
Producers and Admins
GUI Console
Web Console System Layout
Web Console
![Page 30: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/30.jpg)
30
Wrap Up
![Page 31: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/31.jpg)
31
Wrap Up
High Cost Performance High Reliability High Scalability
![Page 32: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/32.jpg)
32
![Page 33: Rakuten LeoFs - distributed file system](https://reader031.vdocuments.mx/reader031/viewer/2022020718/54c2ec754a795950248b45f7/html5/thumbnails/33.jpg)
33
Thank you for your time