PERFORMANCE AND SIZINGSOFTWARE DEFINED STORAGEKyle BaderRed Hat Storage Day, Los AngelesAugust 2016

$> whois kyle bader

Senior Solution ArchitectRed Hat

DATA CHALLENGES

Exponential growth in digital content increases pressure on capacity, scalability, and cost.

The need for access to data from anywhere, anytime, on any device requires unprecedented agility.

Modern services require the flexibility to store data on-premises or in the cloud.

Growing content requires advanced data protection that ensures integrity & high availability at very large scale.

1

2

3

4

Traditional StorageComplex proprietary silos

Open, Software-Defined StorageStandardized, unified, open platforms

Custom GUI

Proprietary Software

ProprietaryHardware

StandardComputersand DisksSt

anda

rdHa

rdwa

reOp

en S

ourc

eSo

ftwar

e

Ceph Gluster +++

Control Plane (API, GUI)

ADMIN USER

THE FUTURE OF STORAGE

ADMIN

USER

ADMIN

USER

ADMIN

USER

Custom GUI

Proprietary Software

ProprietaryHardware

Custom GUI

Proprietary Software

ProprietaryHardware

FLEXIBILITY IS CRUCIAL

Server-based storage uses software and standard hardware to provide services traditionally provided by single-purpose storage

appliances, providing increased agility and efficiency.

DISTRIBUTED CLUSTER OF SERVERS

MEDIA

MEDIA

MEDIA

MEDIA

MEDIA

MEDIA

MEDIA

APPLIANCE

MEDIA

MEDIA

APPLIANCE

MEDIA

MEDIA

APPLIANCE

MEDIA

MEDIA

USER USER USER

SERVER-BASED STORAGE

USER USER USER

STORAGE APPLIANCE

COMPUTE NODE + + + +

+

STORAGE NODE

COMPUTE NODE + + + +

+ + + +

+

+

+

+

+

+

+

+

VIRTUALIZED STORAGE SCALES BETTER

STANDARD SAN/NAS IS ON THE DECLINE

Server-based storage is “will account for over 60% of shipments long term.”

“By 2016, server-based storage solutions will lower storage hardware costs by 50% or

more.”Gartner: “IT Leaders Can Benefit From Disruptive Innovation in the Storage Industry”

Credit Suisse Storage Update, September 3, 2015

Changing workloads drive the need for flexible, economical server-based

storage.

WW DEPLOYED CAPACITY (TB)

2010

2011

2012

2013

2014

2015

(est)

2016

(est)

Source: IDC

0%

60%

20%

40%

80%

100%

Internal Capacity

External Capacity

STORAGE ORCHESTRATION

Storage orchestration is the ability to provision, grow, shrink, and decommission storage resources on-demand and programmatically,

providing increased control and integration of storage into a software-defined data center.

WEB CONSOLEA browser interface designed for managing distributed

storage

APIA full API for automation and integration with outside

systems

COMMAND LINEA robust, scriptable command-

line interface for expert operators

PROVISION INSTALL CONFIGUR

E TUNE MONITOR

Full lifecycle management for distributed, software-defined data services

A RISING TIDE

“By 2020, between 70-80% of unstructured data will be held on lower-cost storage managed by SDS”

“By 2019, 70% of existing storage array productswill also be available as software only versions”

Innovation Insight: Separating Hype From Hope for Software-Defined Storage

Innovation Insight: Separating Hype From Hope for Software-Defined Storage 2013

2014

2015

2016

2017

2018

2019

$1,349M

$1,195M

$1,029M

$859M

$706M

$592M

SDS-P MARKET SIZE BY SEGMENT

$457M

Block StorageFile StorageObject StorageHyperconverged

Source: IDC

Software-Defined Storage is leading a shift in the global storage

industry, with far-reaching effects.

THE BALANCE

InflexibleExpensive at large scale

DurableConvenient

FlexibleEconomical at large scale

DurablePowerful

Appliances are suitable for small-scale, workloads, but they do not

scale economically.

Software-defined storage has a learning curve, but bring

performance and economy at petabyte scale.

THE ROBUSTNESS OF SOFTWARE

Software is more flexible than hardwareSoftware can do things hardware appliances can’t. SDS brings the flexibility of software to the enterprise storage world.• Can be deployed on bare metal, inside containers, inside

VMs, or in the public cloud.• Can deploy on a single server, or thousands, and can be

upgraded and reconfigured on the fly.• Grows and shrinks programmatically to meet changing

demands

BUILDING ON PROVEN HARDWARE

Hardware is hard, and we got you coveredTested software defined storage solutions, for repeatable success.• Ceph Hardware Configuration Guide• Ceph Hardware Selection Guide• Ceph Performance and Sizing Guide - Supermicro• Ceph Performance and Sizing Guide - Quanta QCT

OPTIMIZATION CRITERIA

IOPS Optimized

ThroughputOptimized

CapacityOptimized

• Lowest cost per IO• Highest IOPS• Meet minimum fault domain

requirement

• Lowest cost per unit of throughput• Highest throughput• Highest throughput per watt/BTU• Meet minimum fault domain

requirement

• Lowest cost per TB• Lowest watt/BTU per TB• Meet minimum fault domain

requirement

• Typically block storage• Replication• MySQL for OpenStack tenants

• Block and object storage• Replication or erasure coded• Active performance storage for

video, audio, and images• Streaming media

• Typically object storage• Erasure coding dominant• Media archives• Data lake

• Shared, elastic storage pool• Dynamic DB placement• Flexible volume resizing• Live instance migration• Backup to object pool• Read replicas via copy-on-write

snapshots

MySQL ON CEPH STORAGE CLOUDOPS EFFICIENCY

MYSQL-ON-CEPH PRIVATE CLOUDFIDELITY TO A MYSQL-ON-AWS EXPERIENCE

• Hybrid cloud requires public/private cloud commonalities

• Developers want DevOps consistency• Elastic block storage, Ceph RBD vs. AWS EBS• Elastic object storage, Ceph RGW vs. AWS S3• Users want deterministic performance

HEAD-TO-HEADPERFORMANCE

30 IOPS/GB: AWS EBS P-IOPS TARGET

HEAD-TO-HEAD LABTEST ENVIRONMENTS

• EC2 r3.2xlarge and m4.4xlarge• EBS Provisioned IOPS and GPSSD• Percona Server

• Supermicro servers• Red Hat Ceph Storage RBD• Percona Server

OSD Storage Server Systems5x SuperStorage SSG-6028R-OSDXXX

Dual Intel Xeon E5-2650v3 (10x core)32GB SDRAM DDR32x 80GB boot drives 4x 800GB Intel DC P3700 (hot-swap U.2 NVMe)1x dual port 10GbE network adaptors AOC-STGN-i2S 8x Seagate 6TB 7200 RPM SAS (unused in this lab)Mellanox 40GbE network adaptor(unused in this lab)

MySQL Client Systems12x Super Server 2UTwin2 nodes

Dual Intel Xeon E5-2670v2 (cpuset limited to 8 or 16 vCPUs)64GB SDRAM DDR3

Storage Server Software:Red Hat Ceph Storage 1.3.2Red Hat Enterprise Linux 7.2Percona Server

5x OSD Nodes 12x Client Nodes

Shared 10G SFP+ Networking

Monitor Nodes

SUPERMICRO CEPHLAB ENVIRONMENT

P-IOPS _x000d_m4.4xl P-IOPS_x000d_ r3.2xl GP-SSD _x000d_r3.2xl0

1000

2000

3000

4000

5000

6000

7000

8000

9000

7996 7956

950

1680 1687

267

100% Read100% Write

SYSBENCH BASELINE ON AWS EC2 + EBS

01000020000300004000050000600007000080000

7996

67144

40031

1680 5677 1258

20053

4752

100% Read

100% write

70/30 RW

SYSBENCH REQUESTS PER MYSQL INSTANCE

CONVERTING SYSBENCH REQUESTS TO IOPS READ PATH

X% FROM INNODB BUFFER POOL

IOPS = (READ REQUESTS – X%)

SYSBENCH READ

CONVERTING SYSBENCH REQUESTS TO IOPS WRITE PATH

SYSBENCH WRITE

1X READ

X% FROM INNODB BUFFER POOL

IOPS = (READ REQ – X%)

LOG, DOUBLE WRITE BUFFER

IOPS = (WRITE REQ * 2.3)

1X WRITE

P-IOPS _x000d_m4.4xl P-IOPS _x000d_r3.2xl GP-SSD _x000d_r3.2xl0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

30.0 29.8

3.6

25.6 25.7

4.1

100% Read100% Write

AWS IOPS/GB BASELINE: ~ AS ADVERTISED!

IOPS/GB PER MYSQL INSTANCE

0

50

100

150

200

250

300

30

252

150

26

78

19

MySQL IOPS/GB ReadsMySQL IOPS/GB Writes

FOCUSING ON WRITE IOPS/GBAWS THROTTLE WATERMARK FOR DETERMINISTIC PERFORMANCE

0102030405060708090

26

78

19

A NOTE ON WRITE AMPLIFICATIONMYSQL ON CEPH – WRITE PATH

INNODB DOUBLEWRITE BUFFER

CEPH REPLICATION

OSD JOURNALING

MYSQL INSERT

X2

X2

X2

EFFECT OF CEPH CLUSTER LOADING ON IOPS/GB

Ceph c

luster

_x00

0d_(1

4% ca

pacit

y)

Ceph c

luster

_x00

0d_(3

6% ca

pacit

y)

Ceph c

luster

_x00

0d_(7

2% ca

pacit

y)

Ceph c

luster

_x00

0d_(8

7% ca

pacit

y)0

20406080

100120140160

78

3725 19

134

72

37 36

100% Write70/30 RW

IOPS

/GB

05

10152025303540

18 18 19

6

34 34 36

8

100% Write70/30 RW

IOPS

/GB

CONSIDERING CORE-TO-FLASH RATIO

HEAD-TO-HEADPERFORMANCE

30 IOPS/GB: AWS EBS P-IOPS TARGET

25 IOPS/GB: CEPH 72% CLUSTER CAPACITY (WRITES)78 IOPS/GB: CEPH 14% CLUSTER CAPACITY (WRITES)

HEAD-TO-HEADPRICE/PERFORMANCE

$2.50: TARGET AWS EBS P-IOPS STORAGE PER IOP

IOPS/GB ON VARIOUS CONFIGS

-

10

20

30

40

50

60

70

80

90

31

18 18

78

AWS EBS Provisioned-IOPSCeph on Supermicro FatTwin 72% CapacityCeph on Supermicro MicroCloud 87% CapacityCeph on Supermicro MicroCloud 14% Capacity

IOPS

/GB

(Sys

benc

h W

rite)

$/STORAGE-IOP ON THE SAME CONFIGS

$-

$0.50

$1.00

$1.50

$2.00

$2.50

$3.00

$2.40

$0.80 $0.78 $1.06

AWS EBS Provisioned-IOPSCeph on Supermicro FatTwin 72% CapacityCeph on Supermicro MicroCloud 87% CapacityCeph on Supermicro MicroCloud 14% Capacity

Stor

age

$/IO

P (S

ysbe

nch

Writ

e)

HEAD-TO-HEADPRICE/PERFORMANCE

$2.50: TARGET AWS P-IOPS $/IOP (EBS ONLY)$0.78: CEPH ON SUPERMICRO MICRO CLOUD CLUSTER

8x Nodes in 3U chassisModel: SYS-5038MR-OSDXXXP

Per Node Configuration:CPU: Single Intel Xeon E5-2630 v4Memory: 32GB NVMe Storage: Single 800GB Intel P3700 Networking: 1x dual-port 10G SFP+

+ +

1x CPU + 1x NVMe + 1x SFP

SUPERMICRO MICRO CLOUDCEPH MYSQL PERFORMANCE SKU