hedvig distributed storage platform overview · 2016. 9. 22. · hedvig distributed storage...

Hedvig Distributed Storage Platform Overview Scalable, Hybrid, Multi-site

Tarmo Katmuk

Senior Systems Engineer

Introduction to Hedvig DSP

3 Copyright 2016 Hedvig Inc.

Standard Servers with HDDs, SSDs

Software Hedvig Distributed Storage Platform

+ =

Software decoupled from commodity

hardware

1 Distributed systems

engine with app-focused storage policies

2

Automated and API-driven

3

Hedvig is software-defined storage

4 Copyright 2016 Hedvig Inc.

Hedvig is a hyperscale approach to software-defined storage

Application servers

Hedvig Distributed Storage Platform

VM VM App App

§  Unique two-tier architecture: Enables independent scaling at application and storage tiers

§  Advanced set of storage features: Tunes storage and HA capabilities for each VM, container, or app

§  Modern programmable infrastructure: Automates and integrates with all common virtualization & cloud tools

Datacenter #1 Datacenter #2 Public cloud

5 Copyright 2016 Hedvig Inc.

Hedvig components

Storage Service •  Forms elastic storage cluster on

servers and/or cloud instances •  Hosts metadata and provides

primary storage services

APIs •  Gives developers access to

all storage functions •  Provides object storage via

S3 or Swift •  Enables seamless provisioning

of storage from applications

Storage Proxy •  Presents block and file

storage interface to app hosts •  Directs traffic to and from

storage cluster •  Provides local flash caching

and client-side dedupe

6 Copyright 2016 Hedvig Inc.

Deployment Options

Apps with proxies and storage service on same nodes

Hyperconverged

App nodes with storage proxies

Storage service nodes

Hyperscale

App nodes

Storage service nodes

Hyperscale

Storage proxy nodes

7 Copyright 2016 Hedvig Inc.

Scalable abstraction for granular provisioning of volumes with enterprise storage functions

The Hedvig Virtual Disk

§  Size §  Disk type §  Residence §  Replication factor §  DR policy §  Compression §  Deduplication §  Caching

8 Copyright 2016 Hedvig Inc.

Block File Object

Cloud DC2 DC 1

1 Create and present Virtual Disks to the application tier

2 Capture and direct I/O to storage cluster

3 Distribute and replicate data across nodes

4 Auto-tier and auto-balance across nodes & racks

5 Replicate for DR across datacenters and (or) clouds storage cluster

application tier

VM VM

VM VM

VM VM

VM VM

How a distributed storage system works

9 Copyright 2016 Hedvig Inc.

Node/Rack/Data Center

App host

Replication factor (RF) One to six copies per-Virtual Disk

Residence Store replicas across racks, datacenters, or clouds

Optimized for multi-site and multi-region replication

Data distribution Chunks distributed across servers and containers in the storage cluster

Node/Rack/Data Center Node/Rack/Data Center

Granular data distribution and replication

10 Copyright 2016 Hedvig Inc.

Storage efficiency

Client-side dedupe

App hosts §  Thin provisioning default for all Virtual Disks

§  Inline compression

§  Inline global deduplication w  System-wide – all attached server nodes

participate w Average 75% data reduction – rates vary

based on data type w Dedupe cache can reside on Storage Proxy

SSD/flash in application server w Duplicate I/O eliminated from network to

dramatically lower latency and increase IOPS!

11 Copyright 2016 Hedvig Inc.

Client-side read cache “Pin to flash” dedicated primary storage volume

Read/write cache on storage nodes

Three-ways to use SSD and PCIe flash

12 Copyright 2016 Hedvig Inc.

Cluster Wide Data Distribution and Rebuilding •  The full power of the cluster is used to rebuild any failure •  Data recreated from replicas across cluster •  Quick rebuilds allow for use of largest disks •  Average 4TB disk rebuild time is under 20 minutes •  Easily support 6TB, 8TB, and 10TB drives •  Many to many rebuild: as the cluster grows, recovery times shrink

X

Management and Integration

14 Copyright 2016 Hedvig Inc.

The Hedvig GUI: Dashboard view

15 Copyright 2016 Hedvig Inc.

The Hedvig GUI: Virtual Disk provisioning

16 Copyright 2016 Hedvig Inc.

The Hedvig GUI: Virtual Disk management

AlertsandNo-fica-ons

Block,FileorObject

Agnos-c,RackorDC

1–6copies

17 Copyright 2016 Hedvig Inc.

The Hedvig GUI: Metrics

•  Real-time insight in performance •  Metrics per Virtual Disk or Proxy •  Switch between IOPS, Latency and

Throughput

18 Copyright 2016 Hedvig Inc.

The Hedvig GUI: HTML-5 Support Yes you can manage it from a mobile device. More importantly you can fully manage the system from any locked down host, without needing Java / Flash / etc. installed

19 Copyright 2016 Hedvig Inc.

Management and Provisioning

§ Graphical User Interface – Customizable, HTML-5, Batch provisioning

§ Command Line Interface – All commands

§  RESTful API – Extensive implementation

§  SNMP (v3) – Provides centralised notifications

20 Copyright 2016 Hedvig Inc.

VMware vCenter Plugin

Provision, Manage, Snapshot and clone Hedvig vDisks from

the vSphere Web Client

21 Copyright 2016 Hedvig Inc.

Hedvig and Containers Hedvig has the ability to deliver persistent storage for containers using the Docker volume plugin or Flocker Daemon

Why Hedvig for containers? •  Enterprises can reduce the capital and

operational costs of Docker storage •  Use iSCSI and/or NFS •  Simplifies workflows and increases Docker

admin productivity •  Spin up or down on demand •  Define virtual disk to fit the service •  Persistent storage for containers •  Portability of containers •  Native multi-site & hybrid cloud support

22 Copyright 2016 Hedvig Inc.

Modern Storage for OpenStack

Hedvig Delivers: •  Block, file, and object all from the same platform via native Cinder

and Swift integration. •  The ability to set granular, per-volume (Cinder) or per-container

(Swift) policies for capabilities like compression, dedupe, snaps, and clones.

•  A distributed platform that’s been optimized for multi-site and multi-region replication. You can set a replication factor of one to six and determine a destination datacenter or clouds for each replica.

•  Ecosystem integration simplifies rollout •  Plugins and API integration (e.g. Mirantis, ClusterHQ, Vmware,

Docker, etc..)

Automation, elasticity, and Flexibility for cloud builders

23 Copyright 2016 Hedvig Inc.

Feature review

•  Hyper-converged | Hyper-scale •  Auto-balance | auto-tier | auto-cache •  Hardware and OS agnostic •  No limit to # of scale-out nodes •  Grow and shrink dynamically

•  Block – iSCSI (+Cinder) •  File – NFS 3 and 4 •  Object (+S3/Swift)

•  Multiple copies of data acknowledged •  Checksums for consistency •  Snapshots and clones •  Non-disruptive upgrades •  No planned downtime •  Wide-stripe disk rebuilds (fast!!!)

•  Convert random writes to sequential in memory •  Stream detection (random vs sequential) •  Set custom block size (4KB to 64KB) •  Virtual Disk granular feature/DR deployment per server •  REST, CLI, GUI, OpenStack (CINDER/SWIFT) and Docker

•  Thin provisioning •  Inline compression •  Inline deduplication •  Client side read-cache •  Client side dedupe-cache

Elastic Architecture

Unified Protocol Access

Data Protection & Availability

Base Functionality

Data Efficiency & Performance

•  Rack, DC and cloud aware DR Policies •  Multi-site DR replications •  Sync and semi-sync simultaneously •  No limit in # DCs, all active sites

Disaster Recovery

THANK YOU!