qct ceph solution - design consideration and reference architecture
TRANSCRIPT
QCT Ceph Solution – Design
Consideration and Reference
ArchitectureGary LeeAVP, QCT
2
• Industry Trend and Customer Needs
• Ceph Architecture
• Technology
• Ceph Reference Architecture and QCT Solution
• Test Result
• QCT/Red Hat Ceph Whitepaper
AGENDA
3
Industry Trendand
Customer Needs
4
5
• Structured Data -> Unstructured/Structured Data
• Data -> Big Data, Fast Data
• Data Processing -> Data Modeling -> Data Science
• IT -> DT
• Monolithic -> Microservice
Industry Trend
6
• Scalable Size• Variable Type• Longivity Time• Distributed Location• Versatile Workload
• Affordable Price• Available Service• Continuous Innovation• Consistent
Management• Neutral Vendor
Customer Needs
7
Ceph Architecture
8
Ceph Storage Cluster
Cluster NetworkCeph
LinuxCPU
Memory
SSDHDDNIC
Ceph
Linux
CPUMemor
ySSDHDDNIC
Ceph
Linux
CPUMemor
ySSDHDDNIC
Ceph
Linux
CPUMemor
ySSDHDDNIC
Object Block File
UnifiedStorage
Scale-outCluster
Open Source Software
Open Commodity Hardware
…..
9
Block I/O
Ceph Client
RBD RADOSGW
Ceph FS
Object I/O
File I/O
RADOS/Cluster Network
OSD
File System
I/O
Disk I/O
Public N
etwork
End-to-end Data Path
App Service
10
Ceph Software Architecture
Public Network (ex. 10GbE or 40GbE)
Cluster Network (ex. 10GbE or 40GbE)
Ceph Monitor
…...
RCT or RCCNx Ceph OSD Nodes
Ceph OSD Node
Clients
Ceph OSD Node Ceph OSD Node
Ceph Hardware Architecture
12
Technology
13
• 2x Intel E5-2600 CPU• 16x DDR4 Memory• 12x 3.5” SAS/SATA HDD• 4x SATA SSD + PCIe M.2• 1x SATADOM• 1x 1G/10G NIC• BMC with 1G NIC• 1x PCIe x8 Mezz Card• 1x PCIe x8 SAS Controller• 1U
QCT Ceph Storage ServerD51PH-1ULH
14
• Mono/Dual Node • 2x Intel E5-2600 CPU• 16x DDR4 Memory• 78x or 2x 35x SSD/HDD• 1x 1G/10G NIC• BMC with 1G NIC• 1x PCIe x8 SAS Controller• 1x PCIe x8 HHLH Card• 1x PCIe x16 FHHL Card• 4U
QCT Ceph Storage ServerT21P-4U
15
• 1x Intel Xeon D SoC CPU• 4x DDR4 Memory• 12x SAS/SATA HDD• 4x SATA SSD• 2x SATA SSD for OS• 1x 1G/10G NIC• BMC with 1G NIC• 1x PCIe x8 Mezz Card• 1x PCIe x8 SAS Controller• 1U
QCT Ceph Storage ServerSD1Q-1ULH
16
• Standalone, without EC
• Standalone, with EC
• Hyper-converged, without EC
• High Core vs. High Frequency
• 1x OSD ~ (0.3-0.5)x Core + 2G RAM
CPU/Memory
17
• SSD:– Journal– Tier– File System Cache– Client Cache
• Journal– HDD: SSD (SATA/SAS): 4~5– HDD: NVMe: 12~18
SSD/NVMe
18
• 2x NVMe ~40Gb
• 4x NVMe ~100Gb
• 2x SATA SSD ~10Gb
• 1x SAS SSD ~10Gb
• (20~25)x HDD ~10Gb
• ~100x HDD ~40Gb
NIC10G/40G -> 25G/100G
19
• CPU Offload through RDMA/iWARP
• Erasure Coding Offload
• Allocate computing on different silicon areas
NICI/O Offloading
20
• Object Replication– 1 Primary + 2 Replica (or more) – CRUSH Allocation Ruleset
• Erasure Coding– [k+m], e.g. 4+2, 8+3– Better Data Efficiency
• k/(k+m) vs. 1/(1+replication)
Erasure Coding vs. Replication
21
Size/Workload Small Medium Large
Throughput Transfer BandwidthSequential R/W
Capacity Cost/capacityScalability
IOPS IOPS/ per 4k BlockRandom R/W
Hyper-converged ?
Desktop Virtualization
LatencyRandom R/W
Hadoop ?
Workload and Configuration
22
Red Hat Ceph
23
• Intel ISA-L
• Intel SPDK
• Intel CAS
• Mellanox Accelio Library
Vendor-specific Value-added Software
24
Ceph Reference Architecture and
QCT Solution
• Trade-off among Technologies
• Scalable in Architecture
• Optimized for Workload
• Affordable as Expected
Design Principle
26
1. Needs for scale-out storage
2. Target workload
3. Access method
4. Storage capacity
5. Data protection methods
6. Fault domain risk tolerance
Design Considerations
27
Transaction
DataWarehou
se Big Data
ScientificBlock Transf
erAudio Video
IOP
S
MB/sec
OLTP
OLAP
HPC
Streaming
DB
Storage Workload
SMALL (500TB*) MEDIUM (>1PB*) LARGE (>2PB*)Throughput optimized
QxStor RCT-200 16x D51PH-1ULH (16U)• 12x 8TB HDDs• 3x SSDs• 1x dual port 10GbE• 3x replica
QxStor RCT-400 6x T21P-4U/Dual (24U)• 2x 35x 8TB HDDs• 2x 2x PCIe SSDs• 2x single port 40GbE• 3x replica
QxStor RCT-400 11x T21P-4U/Dual (44U)• 2x 35x 8TB HDDs• 2x 2x PCIe SSDs• 2x single port 40GbE• 3x replica
Cost/Capacity optimized
IOPS optimized Future direction Future direction NA
* Usable storage capacity
QxStor RCC-400Nx T21P-4U/Dual • 2x 35x 8TB HDDs• 0x SSDs• 2x dual port 10GbE• Erasure Coding 4:2
QCT QxStor Red Hat Ceph Storage Edition PortfolioWorkload-driven Integrated Software/Hardware Solution
• Densest 1U Ceph building block • Best reliability with smaller
failure domain
• Scale at high scale 2x 280TB• At once obtain best throughput
and density
• Block or object storage • 3x replication • Video, audio, image repositories, and streaming media
• Highest density 560TB raw capacity per chassis with greatest price/performance
• Typically object storage• Erasure coding common
for maximizing usable capacity • Object archive
Throughput-Optimized RCC-400RCT-200 RCT-400
Cost/Capacity-Optimized
USE
CAS
E
QCT QxStor Red Hat Ceph Storage Edition Co-engineered with Red Hat Storage team to provide Optimized Ceph Solution
30
Ceph Solution DeploymentUsing QCT QPT Bare Metal Privision Tool
31
Ceph Solution DeploymentUsing QCT QPT Bare Metal Privision Tool
32
QCT Solution Value Proposition
• Workload-driven
• Hardware/software pre-validated, pre-optimized and
pre-integrated
• Up and running in minutes
• Balance between production (stable) and innovation
(up-streaming)
33
Test Result
Client 1S2B
Client 2S2B
Client 3S2B
Ceph 1S2PH
Ceph 2S2PH
Ceph 3S2PH
Ceph 5S2PH
Ceph 4S2PH
Client 8S2B
Client 9S2B
Client 10S2B
10Gb
10Gb
Public Network
Cluster Network
General Configuration
• 5 Ceph nodes (S2PH) with each 2 x 10Gb link.• 10 Client nodes (S2B) with each 2 x 10Gb link.
• Public network : Balanced bandwidth between Client nodes and Ceph nodes.• Cluster network : Offload the traffic from public network to improve performance.
Option 1 (w/o SSD) a. 12 OSD per Ceph storage node b. S2PH (E5-2660) x2 c. RAM : 128 GB
Option 2 : (w/ SSD) a. 12 OSD / 3 SSD per Ceph storage node b. S2PH (E5-2660) x2 c. RAM : 12 (OSD) x 2GB = 24 GB
Testing Configuration (Throughput-Optimized)
Client 1S2S
Client 2S2S
Client 3S2S
Ceph 1S2P
Ceph 2S2P
Client 6S2S
Client 7S2S
Client 8S2S
10Gb
Public Network
40Gb 40Gb
General Configuration
• 2 Ceph nodes (S2P) with each 2 x 10Gb link.• 8 Client nodes (S2S) with each 2 x 10Gb link.
• Public network : Balanced bandwidth between Client nodes and Ceph nodes.• Cluster network : Offload the traffic from public network to improve performance.
Option 1 (w/o SSD) a. 35 OSD per Ceph storage node b. S2P (E5-2660) x2 c. RAM : 128 GB
Option 2 : (w/ SSD) a. 35 OSD / 2 PCI-SSD per Ceph storage node b. S2P (E5-2660) x2 c. RAM : 128 GB
Testing Configuration (Capacity-Optimized)
36
Level Component Test SuiteRaw I/O Disk FIONetwork I/O Network iperfObject API I/O librados radosbenchObject I/O RGW CosbenchBlock I/O RBD librbdfio
CBT (Ceph Benchmarking Tool)
37
Linear Scale Out
38
Linear Scale Up
39
Price, in terms of Performance
40
Price, in terms of Capacity
41
Protection Scheme
42
Cluster Network
43
QCT/Red Hat Ceph
Whitepaper
44
http://www.qct.io/account/download/download?order_download_id=1022&dtype=Reference%20Architecture
QCT/Red Hat Ceph Solution Brief
https://www.redhat.com/en/files/resources/st-performance-sizing-guide-ceph-qct-inc0347490.pdf
http://www.qct.io/Solution/Software-Defined-Infrastructure/Storage-Virtualization/QCT-and-Red-Hat-Ceph-Storage-p365c225c226c230
QCT/Red Hat Ceph Reference Architecture
46
• The Red Hat Ceph Storage Test Drive lab in QCT Solution Center provides you a free hands-on experience. You'll be able to explore the features and simplicity of the product in real-time.
• Concepts:Ceph feature and functional test
• Lab Exercises:Ceph BasicsCeph Management - Calamari/CLICeph Object/Block Access
QCT Offer TryCeph (Test Drive) Later
47
Remote access to QCT cloud solution centers• Easy to test. Anytime and anywhere.
• No facilities and logistic needed
• Configurations• RCT-200 and newest QCT solutions
QCT Offer TryCeph (Test Drive) Later
48
• Ceph is Open Architecture• QCT, Red Hat and Intel collaborate to provide
– Workload-driven,– Pre-integrated, – Comprehensive-tested and – Well-optimized solution
• Red Hat – Open Software/Support PioneerIntel – Open Silicon/Technology InnovatorQCT – Open System/Solution Provider
• Together We Provide the Best
CONCLUSION
www.QuantaQCT.com
Thank you!
50
www.QCT.io
QCT CONFIDENTIAL
Looking forinnovative cloud solution?
Come to QCT, who else?