![Page 1: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/1.jpg)
The Next Step
Yosuke Hara - @yosukehara
A Researcher of R.I.T. and Tech Lead LeoFSwith Paras Patel, An engineer of Rakuten DU
1
![Page 2: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/2.jpg)
OverviewBrief Benchmark Reportv1.0 -Multi Data Center Replicationv1.1 - NFS SupportLeoFS Administration at RakutenFuture Plan
2
![Page 3: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/3.jpg)
Overview
3
![Page 4: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/4.jpg)
Terabyte Petabyte Exabyte2014 2020
LeoFS - Background
2010 20122008 2016 2018
4
![Page 5: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/5.jpg)
� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �
� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �
� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �
The Lion of Storage Systems
HIGH Availability
HIGH Cost Performance Ratio
HIGH Scalability
LeoFS Non Stop
Velocity: Low LatencyMinimum Resources
Volume: Petabyte / ExabyteVariety: Photo, Movie, Unstructured-data
3 Vs in 3 HIGHs
5
![Page 6: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/6.jpg)
Metadata Object Storage
Storage Engine/Router
Monitor
GUI Console
( Erlang RPC)
LeoFS - Architecture
Storage(storage cluster)
Manager
( Erlang RPC)
Gateway
( TCP/IP,SNMP )
S3-API / NFS
- Software Defined Storage- Satisfied the 3 HIGH- Easy Administration
Metadata Object Storage
Storage Engine/Router
Metadata Object Storage
Storage Engine/Router
Clients
.....Monitoring
System
Huge Capacity Storage
6
![Page 7: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/7.jpg)
LeoFS Gateway
7
![Page 8: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/8.jpg)
LeoFS Overview - Gateway
Stateless Proxy + Object Cache
REST-API / S3-API
Use Consistent Hashingfor decision of a primary node
[ Memory Cache, Disc Cache ]
Storage C
lusterG
ateway(s)
Clients
HTTP Request and ResponseBuilt in Object Cache Mechanism
Storage Cluster
Fast HTTP Server - CowboyAPI HandlerObject Cache Mechanism
8
![Page 9: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/9.jpg)
LeoFS Storage
9
![Page 10: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/10.jpg)
Storage (S
torage Cluster)
Gatew
ay
LeoFS Overview - Storage
Use "Consistent Hashing"for Data Operation
in the Storage Cluster
Choosing Replica Target Node(s)
RING2 ^ 128 (MD5)
# of replicas = 3
KEY = “bucket/leofs.key”Hash = md5(Filename)
Secondary-1
Secondary-2
Primary Node
"P2P"
WRITE: Auto ReplicationREAD : Auto Repair of an Inconsistent Object with Async
10
![Page 11: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/11.jpg)
Request From Gateway
LeoFS Overview - Storage
...
LeoFS Storage
ReplicatorRecoverer
...
Storage Engine
Storage E
ngine, Metadata + O
bject Storage
Gatew
ay
Storage consists of Object Storage and Metadata StorageIncludes Replicator and Recoverer for the eventual consistency
MetadataStorage Object
Storage
11
![Page 12: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/12.jpg)
LeoFS Overview - Storage - Data Structure
Metadata
Storage
Object S
torage
Robust andHigh PerformanceNecessary for GC
Offset Version Time-stamp Key
<Metadata>
Checksum
for Sync
KeySize CustomMeta Size File Size
for retrieving an object
Footer (8B)
Checksum KeySize DataSize Offset Version Time-stamp Key User-Meta Footer
Header (Metadata - Fixed length) Body (Variable Length)
User-MetaSize
ActualFile
<Needle>
Supe
r-bl
ock
Nee
dle-
1
Nee
dle-
2
Nee
dle-
3
<Object Container>N
eedl
e-4
Nee
dle-
5
Log Structure File Format
12
![Page 13: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/13.jpg)
LeoFS Manager
13
![Page 14: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/14.jpg)
Storage Cluster
LeoFS Overview - Manager
Monitor
Operate
RING, Node State
status, suspend,resume, detach, whereis, ...
Gateway(s)
Storage C
lusterG
ateway(s)
Manager(s)
Operates LeoFS - Gateway and Storage Cluster"RING Monitor" and "NodeState Monitor"
14
![Page 15: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/15.jpg)
Brief BenchmarkReport
15
![Page 16: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/16.jpg)
Brief Benchmark Report
1st Case: Group of Value Ranges Storage:5, Gateway:1, Manager:2 R:W = 9:1
2nd Case: Group of Value Ranges Storage:5, Gateway:1, Manager:2 R:W = 8:2
source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140605/tests/1m_r9w1_240min
source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140605/tests/1m_r8w2_120min
16
![Page 17: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/17.jpg)
Brief Benchmark Report
CPU Intel(R) Xeon(R) CPU X5650 @ 2.67GHz * 2 (12 cores / 24 threads)
Memory 96GBDisk HDD - 240GB RAID0
Network 10G-Ether
Server Spec - Gateway:
CPU Intel(R) Xeon(R) CPU X5650 @ 2.67GHz * 2 (12 cores / 24 threads)
Memory 96GB
DiskHDD - 240GB RAID0 (System)
DiskHDD - 2TB RAID0 (Data)
Network 10G-Ether
Server Spec - Storage x5:
17
![Page 18: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/18.jpg)
Network 10GbpsOS CentOS release 6.5 (Final)
Erlang OTP R16B03-1LeoFS v1.0.2
Environment:
System Consistency Level: [ N:3, W:2, R:1, D:2 ]
Duration 4.0hR:W 9:1
# of Concurrent Processes 64
# of Keys 100,000
Value Size
Benchmark Configuration:
Range (byte)Range (byte) Percentage
1,024 10,240 24%
10,241 102,400 30%
102,401 819,200 30%
819,201 1,572,864 16%
Brief Benchmark Report - 1st Case (R:W=9:1)
18
![Page 19: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/19.jpg)
Range (byte)Range (byte) Percentage
1,024 10,240 24%
10,241 102,400 30%
102,401 819,200 30%
819,201 1,572,864 16%
Brief Benchmark Report - 1st Case (R:W=9:1)
The range of the data size of 80%is between 1KB and 1.5MB
19
![Page 20: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/20.jpg)
source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140601/tests/1m_r9w1_240min
50ms
Brief Benchmark Report - 1st Case (R:W=9:1)
50ms
1,500ops
No Errors
OPS
Latency
20
![Page 21: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/21.jpg)
Brief Benchmark Report - 1st Case / Network Traffic
0
150,000
300,000
450,000
600,000
750,000
900,000
1,050,000
1,200,000
1,350,000
1,500,000
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
7500s
8000s
8500s
9000s
9500s
10000s
10500s
11000s
11500s
12000s
12500s
13000s
13500s
14000s
gateway rxbyt/s gateway txbyt/sstorage-1 rxbyt/s storage-1 txbyt/sstorage-2 rxbyt/s storage-2 txbyt/sstorage-3 rxbyt/s storage-3 txbyt/sstorage-4 rxbyt/s storage-4 txbyt/sstorage-5 rxbyt/s storage-5 txbyt/s
10.0Gbps
7.0Gbps
5.0Gbps
6.0Gbps
Storage
Gatew
ay
60%
21
![Page 22: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/22.jpg)
Brief Benchmark Report - 1st Case / Memory and CPU
0
0.1
0.3
0.4
0.6
0.7
0.9
1.0
1.1
1.3
1.4
1.6
1.7
1.9
2.0
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
7500s
8000s
8500s
9000s
9500s
10000s
10500s
11000s
11500s
12000s
12500s
13000s
13500s
14000s
Memory Usage
CPU Load 5min
1.0
0
10
20
30
40
50
60
70
80
90
100
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
7500s
8000s
8500s
9000s
9500s
10000s
10500s
11000s
11500s
12000s
12500s
13000s
13500s
14000s
gateway storage-1storage-2 storage-3storage-4 storage-5
22
![Page 23: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/23.jpg)
Network 10GbpsOS CentOS release 6.5 (Final)
Erlang OTP R16B03-1LeoFS v1.0.2
Environment:
System Consistency Level: [ N:3, W:2, R:1, D:2 ]
Duration 2.0hR:W 8:2
# of Concurrent Processes 64
# of Keys 100,000
Value Size
Benchmark Configuration:
Brief Benchmark Report - 2nd Case (R:W=8:2)
Range (byte)Range (byte) Percentage
1,024 10,240 24%
10,241 102,400 30%
102,401 819,200 30%
819,201 1,572,864 16%
23
![Page 24: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/24.jpg)
Brief Benchmark Report - 2nd Case (R:W=8:2)
65ms 85ms
1,000ops
No Errors
OPS
Latency
24
![Page 25: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/25.jpg)
Compare 1st case with 2nd case
25
![Page 26: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/26.jpg)
Brief Benchmark Report
0
150,000
300,000
450,000
600,000
750,000
900,000
1,050,000
1,200,000
1,350,000
1,500,000
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000sgateway rxbyt/s gateway txbyt/sstorage-1 rxbyt/s storage-1 txbyt/sstorage-2 rxbyt/s storage-2 txbyt/sstorage-3 rxbyt/s storage-3 txbyt/sstorage-4 rxbyt/s storage-4 txbyt/sstorage-5 rxbyt/s storage-5 txbyt/s
0
300,000
600,000
900,000
1,200,000
1,500,000
1,800,000
2,100,000
2,400,000
2,700,000
3,000,000
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
6.0Gbps7.0Gbps
6.0Gbps7.0Gbps
minus 0.7Gbps
1st Case - Network Traffic
2nd Case - Network Traffic
26
![Page 27: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/27.jpg)
Brief Benchmark Report
0
50.0
100.0
150.0
200.0
250.0
300.0
350.0
400.0
450.0
500.0
550.0
600.0
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
0
50.0
100.0
150.0
200.0
250.0
300.0
350.0
400.0
450.0
500.0
550.0
600.0
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
storage-1 storage-2 storage-3storage-4 storage-5
100
100
2nd Case - Disk util%
200
200
1st Case - Disk util%
1.8x high
27
![Page 28: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/28.jpg)
Brief Benchmark Report
00.20.40.60.81.01.21.41.61.82.02.22.42.62.83.0
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
gatewaystorage-1storage-2storage-3storage-4storage-5
00.20.40.60.81.01.21.41.61.82.02.22.42.62.83.0
0s 500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
1.00
1.00
1.6x high2nd Case - CPU Load 5min
1st Case - CPU Load 5min
28
![Page 29: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/29.jpg)
LeoFS kept in a stable performance through the benchmark
Brief Benchmark Report
Bottleneck is Disk I/O
The cache mechanism contributed to reduce network traffic between Gateway and Storage
Conclusion:
29
![Page 30: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/30.jpg)
LeoFS v1.0Multi Data CenterReplication
30
![Page 31: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/31.jpg)
LeoFS - For Disaster Recovery
Storage cluster
Manager cluster
For Disaster Recovery= High Scalability + High Availability
w/o SPOF and Performance Degradation
31
![Page 32: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/32.jpg)
1. Easy Operation to build multi clusters.
2. Asynchronous data replication between clusters
Stacked data is transferred to remote cluster(s)
3. Eventual consistency
Multi Data Center Replication
Designes it as simple as possible
32
![Page 33: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/33.jpg)
DC-3DC-2
Storage cluster
Manager cluster
Client
DC-1
Monitors and Replicates each “RING” and “System Configuration”
"Leo Storage Platform"
[# of replicas:1] [# of replicas:1][# of replicas:3]
"join cluster DC-2 and DC-3"
leo_rpcleo_rpc
Multi Data Center Replication
Executing “Join Cluster” on Manager Console
Preparing the MDC Replication
33
![Page 34: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/34.jpg)
DC-3DC-2
Storage cluster
Manager cluster
Client
Monitors and Replicates each “RING” and “System Configuration”
"Leo Storage Platform"
[# of replicas:1] [# of replicas:1]
Request tothe Target Region
Application(s)
DC-1
[# of replicas:3]
Temporally Stacking objects- One container's capacity is *32MB- When capacity is full,
send it to remote cluster(s)* 32MB: default capacity - able to set optional value
leo_rpcleo_rpc
Multi Data Center Replication
Stacking objects
34
![Page 35: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/35.jpg)
DC-3DC-2
Storage cluster
Manager cluster
Client
Monitors and Replicates each “RING” and “System Configuration”
"Leo Storage Platform"
DC-1
Stacked an object with a metadata
Compress it with LZ4
Replicated an object
Request tothe Target Region
Application(s)
leo_rpc
leo_rpcleo_rpc
Multi Data Center Replication
Transferring stacked objects
Stacked objects
35
![Page 36: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/36.jpg)
DC-3DC-2
Storage cluster
Manager cluster
Client
Monitor and Replicate each “RING” and “System Configuration”
"Leo Storage Platform"
Request tothe Target Region
Application(s)
DC-1
1) Receive metadata of stored objects2) Compare them at the local cluster3) Fix inconsistent objects
leo_rpcleo_rpc
leo_rpcleo_rpc
Multi Data Center Replication
Investigating stored objects
36
![Page 37: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/37.jpg)
LeoFS v1.1NFS Support
37
![Page 38: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/38.jpg)
PaaS / IaaS
BigData Analysis
Log / Event / Sensor data
BigDataAnalysis Platform
Document / Contents
Photo / Movie
Various Kind and Huge Amount Data
Various Kind Data
Any Services
Log / Event / Sensor data
Offline - NFS / S3-API Online - REST / S3-API
LeoFS v1.1 - NFS Support
38
![Page 39: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/39.jpg)
LeoFS Administrationat Rakuten
Presented by Paras Patel An engineer of Rakuten DU
39
![Page 40: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/40.jpg)
Storage PlatformFile Sharing ServiceEnd-User Servers Portal Site
Photo Storage
Background Storage of OpenStack
LeoFS Administration at Rakuten
40
![Page 41: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/41.jpg)
41
Rakuten Services are using LeoFS
41
![Page 42: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/42.jpg)
Project FiFo is an open-source Cloud Management and Orchestration system for SmartOS virtualization environments.
The components of FiFo are written entirely in Erlang which gives the suite excellent stability and fault recovery as it continues maturing to a production quality release.
Project FiFo uses LeoFS packages for SmartOS in their repository.
External Users
42
![Page 43: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/43.jpg)
Storage Platform
43
![Page 44: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/44.jpg)
Storage Platform - Scaling the Storage Platform
(Movie)
Reduce CostsHigh ReliabilityEasy to ScaleS3-API
44
![Page 45: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/45.jpg)
Using Various Services Total Usage: 450TB/600TB
# of Files: 600Million Daily Growth: 100GB Daily Reqs: 13Million
Storage Platform - Scaling the Storage Platform
E-Commerce
Blog
Insurance Calendar
Recruiting
Review Photoshare
Portal &Contents
Bookmark
B
Storage Platform
(Movie)
45
![Page 46: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/46.jpg)
Monitor
GUI Console
( Erlang RPC)
( Erlang RPC) ( TCP/IP,SNMP )
Gatew
ay x 4Storage x 14
Manager x 2
Requests fromWeb Applications / Browsers
w/HTTP over S3-API
Load Balancer / Cache Servers
Storage Platform - System LayoutTotal disk space: 600TBNumber of Files: 600MillionAccess Stats: 800Mbps (MAX) 400Mbps (AVG)
46
![Page 47: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/47.jpg)
Monitor
GUI Console
( Erlang RPC)
( Erlang RPC) ( TCP/IP,SNMP )
Gatew
ay x 4Storage x 14
Manager x 2
Storage Platform - Monitor
Send Mail AlertGanglia Agent
Status Collection (Ganglia)Status Check (Nagios)Port + Threshold Check
47
![Page 48: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/48.jpg)
Storage Platform - Spreading Globally
Covering All Services with Multi DC Replication
48
![Page 49: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/49.jpg)
+https://owncloud.com/ http://roma-kvs.org/
+=
49
![Page 50: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/50.jpg)
+
File Sharing Service - Required Targets
Reduce CostsHandle Confidential Files
Store Large FilesScale Easily
50
![Page 51: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/51.jpg)
+
Share Docs and Videos with Group CompaniesOver 20 Companies, Over 10 Countries
Over 10,000 Users, Over 4,000 Teams
File Sharing Service - Usage
51
![Page 52: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/52.jpg)
LDAP
Monitor
GUI Console
( Erlang RPC)
( Erlang RPC) ( TCP/IP,SNMP )
Manager x 2
Authenticate Users
Manage Configurations
ManageLogin Session(KVS)
File Sharing Service - System Layout
Web GUI File Browser
52
![Page 53: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/53.jpg)
LeoFS Client Test: https://github.com/leo-project/leofs_client_tests
LeoFS Continuous Automated Testing
AWS SDK for PHPAWS SDK for RubyAWS SDK for JavaBoto(Python)erlcloud(Erlang)s3cmds3fuse
Continuous Integration with Jenkins
53
![Page 54: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/54.jpg)
Cover 25 Countries/RegionsOver 20,000 Users
+
File Sharing Service - Future Plans
54
![Page 55: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/55.jpg)
Enhancing the Servicesand Empowering the Users through the Cloud Storage
55
![Page 56: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/56.jpg)
LeoFS v2.0
Beyond Storage System,Toward Cloud Storage
56
![Page 57: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/57.jpg)
LeoFS v2.0 - Cloud Storage
Autonomic OperationAutomatic RebalanceAutomatic Data Compaction
QoSStatistics of internal LeoFSWatchdog
Request / Traffic Control
Always keeps best condition of LeoFS
57
![Page 58: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/58.jpg)
PaaS / IaaS
BigData Analysis
Log / Event / Sensor data
BigDataAnalysis Platform
Document / Contents
Photo / Movie
Various Kind and Huge Amount Data
Various Kind Data
Any Services
Log / Event / Sensor data
LeoFS v2.0 - Cloud Storage
Hybrid StorageCentralizes Huge Amount and Various Kind data in LeoFSSupports Online and Offline Access
58
![Page 59: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/59.jpg)
PaaS / IaaS
BigData Analysis
Log / Event / Sensor data
BigDataAnalysis Platform
Document / Contents
Photo / Movie
Various Kind and Huge Amount Data
Various Kind Data
Any Services
Log / Event / Sensor data
REST-API (JSON)
Operate LeoFS
Notify a message of over # of req threshold
NewDB Projfor LeoFS Insight
LeoFS v2.0 - Cloud Storage
+Retrieve m
etrics and stats from NewDB Proj's Agents
- Autonomic operation- NFS Support- Erasure Code- NewDB Integration- OpenStack Integrationand more...
59
![Page 60: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/60.jpg)
IntroducingNewDB Project
60
![Page 61: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/61.jpg)
NewDB Proj - Background
We need to get "Value in Data" in realtime
Transaction Analysis Cache
Typical System
NewDB Proj is a Hybrid DBHandle Atomic Data
and Analyze Data in realtime
+Database Statistics / ML
NewDB Proj
Typical Systems still realise"Semi-realtime Data Analysis"
61
![Page 62: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/62.jpg)
NewDB Proj - Concept
Analyzing Data in RealtimeRetrieving "Value" in Exponential Data
High Realiability High Scalability
Ad-hoc QueryPluggable QLODBC/JDBC, REST, CLI,...
NewDB Proj
ML-libIntegration
Pluggable ML/Statistics Mechanism
62
![Page 63: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/63.jpg)
Analyzing DataOnline
TransactionStatistics, ML
Handle Event, Sensorand Metrics Data
Clustering DataFinding Patterns in Data
Calculate and Generate DataThen Grow Value of Data
Pluggable QLEase of Operation
and The QL Express Intuition Directly
NewDB Proj - Concept
SQL
Snapshot, BackupRestore
Visualizing Data
63
![Page 64: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/64.jpg)
Early 2015
64
![Page 65: [RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project](https://reader033.vdocuments.mx/reader033/viewer/2022060122/5595a0501a28ab0a448b4649/html5/thumbnails/65.jpg)
Website: leo-project.netTwitter: @LeoFastStorage
LeoProject
Set Sail for the new Leo land
65