hedvig distributed storage platform overview · 2016. 9. 22. · hedvig distributed storage...
TRANSCRIPT
Hedvig Distributed Storage Platform Overview Scalable, Hybrid, Multi-site
Tarmo Katmuk
Senior Systems Engineer
Introduction to Hedvig DSP
3 Copyright 2016 Hedvig Inc.
Standard Servers with HDDs, SSDs
Software Hedvig Distributed Storage Platform
+ =
Software decoupled from commodity
hardware
1 Distributed systems
engine with app-focused storage policies
2
Automated and API-driven
3
Hedvig is software-defined storage
4 Copyright 2016 Hedvig Inc.
Hedvig is a hyperscale approach to software-defined storage
Application servers
Hedvig Distributed Storage Platform
VM VM App App
§ Unique two-tier architecture: Enables independent scaling at application and storage tiers
§ Advanced set of storage features: Tunes storage and HA capabilities for each VM, container, or app
§ Modern programmable infrastructure: Automates and integrates with all common virtualization & cloud tools
Datacenter #1 Datacenter #2 Public cloud
5 Copyright 2016 Hedvig Inc.
Hedvig components
Storage Service • Forms elastic storage cluster on
servers and/or cloud instances • Hosts metadata and provides
primary storage services
APIs • Gives developers access to
all storage functions • Provides object storage via
S3 or Swift • Enables seamless provisioning
of storage from applications
Storage Proxy • Presents block and file
storage interface to app hosts • Directs traffic to and from
storage cluster • Provides local flash caching
and client-side dedupe
6 Copyright 2016 Hedvig Inc.
Deployment Options
Apps with proxies and storage service on same nodes
Hyperconverged
App nodes with storage proxies
Storage service nodes
Hyperscale
App nodes
Storage service nodes
Hyperscale
Storage proxy nodes
7 Copyright 2016 Hedvig Inc.
Scalable abstraction for granular provisioning of volumes with enterprise storage functions
The Hedvig Virtual Disk
§ Size § Disk type § Residence § Replication factor § DR policy § Compression § Deduplication § Caching
8 Copyright 2016 Hedvig Inc.
Block File Object
Cloud DC2 DC 1
1 Create and present Virtual Disks to the application tier
2 Capture and direct I/O to storage cluster
3 Distribute and replicate data across nodes
4 Auto-tier and auto-balance across nodes & racks
5 Replicate for DR across datacenters and (or) clouds storage cluster
application tier
VM VM
VM VM
VM VM
VM VM
How a distributed storage system works
9 Copyright 2016 Hedvig Inc.
Node/Rack/Data Center
App host
Replication factor (RF) One to six copies per-Virtual Disk
Residence Store replicas across racks, datacenters, or clouds
Optimized for multi-site and multi-region replication
Data distribution Chunks distributed across servers and containers in the storage cluster
Node/Rack/Data Center Node/Rack/Data Center
Granular data distribution and replication
10 Copyright 2016 Hedvig Inc.
Storage efficiency
Client-side dedupe
App hosts § Thin provisioning default for all Virtual Disks
§ Inline compression
§ Inline global deduplication w System-wide – all attached server nodes
participate w Average 75% data reduction – rates vary
based on data type w Dedupe cache can reside on Storage Proxy
SSD/flash in application server w Duplicate I/O eliminated from network to
dramatically lower latency and increase IOPS!
11 Copyright 2016 Hedvig Inc.
Client-side read cache “Pin to flash” dedicated primary storage volume
Read/write cache on storage nodes
Three-ways to use SSD and PCIe flash
12 Copyright 2016 Hedvig Inc.
Cluster Wide Data Distribution and Rebuilding • The full power of the cluster is used to rebuild any failure • Data recreated from replicas across cluster • Quick rebuilds allow for use of largest disks • Average 4TB disk rebuild time is under 20 minutes • Easily support 6TB, 8TB, and 10TB drives • Many to many rebuild: as the cluster grows, recovery times shrink
X
Management and Integration
14 Copyright 2016 Hedvig Inc.
The Hedvig GUI: Dashboard view
15 Copyright 2016 Hedvig Inc.
The Hedvig GUI: Virtual Disk provisioning
16 Copyright 2016 Hedvig Inc.
The Hedvig GUI: Virtual Disk management
AlertsandNo-fica-ons
Block,FileorObject
Agnos-c,RackorDC
1–6copies
17 Copyright 2016 Hedvig Inc.
The Hedvig GUI: Metrics
• Real-time insight in performance • Metrics per Virtual Disk or Proxy • Switch between IOPS, Latency and
Throughput
18 Copyright 2016 Hedvig Inc.
The Hedvig GUI: HTML-5 Support Yes you can manage it from a mobile device. More importantly you can fully manage the system from any locked down host, without needing Java / Flash / etc. installed
19 Copyright 2016 Hedvig Inc.
Management and Provisioning
§ Graphical User Interface – Customizable, HTML-5, Batch provisioning
§ Command Line Interface – All commands
§ RESTful API – Extensive implementation
§ SNMP (v3) – Provides centralised notifications
20 Copyright 2016 Hedvig Inc.
VMware vCenter Plugin
Provision, Manage, Snapshot and clone Hedvig vDisks from
the vSphere Web Client
21 Copyright 2016 Hedvig Inc.
Hedvig and Containers Hedvig has the ability to deliver persistent storage for containers using the Docker volume plugin or Flocker Daemon
Why Hedvig for containers? • Enterprises can reduce the capital and
operational costs of Docker storage • Use iSCSI and/or NFS • Simplifies workflows and increases Docker
admin productivity • Spin up or down on demand • Define virtual disk to fit the service • Persistent storage for containers • Portability of containers • Native multi-site & hybrid cloud support
22 Copyright 2016 Hedvig Inc.
Modern Storage for OpenStack
Hedvig Delivers: • Block, file, and object all from the same platform via native Cinder
and Swift integration. • The ability to set granular, per-volume (Cinder) or per-container
(Swift) policies for capabilities like compression, dedupe, snaps, and clones.
• A distributed platform that’s been optimized for multi-site and multi-region replication. You can set a replication factor of one to six and determine a destination datacenter or clouds for each replica.
• Ecosystem integration simplifies rollout • Plugins and API integration (e.g. Mirantis, ClusterHQ, Vmware,
Docker, etc..)
Automation, elasticity, and Flexibility for cloud builders
23 Copyright 2016 Hedvig Inc.
Feature review
• Hyper-converged | Hyper-scale • Auto-balance | auto-tier | auto-cache • Hardware and OS agnostic • No limit to # of scale-out nodes • Grow and shrink dynamically
• Block – iSCSI (+Cinder) • File – NFS 3 and 4 • Object (+S3/Swift)
• Multiple copies of data acknowledged • Checksums for consistency • Snapshots and clones • Non-disruptive upgrades • No planned downtime • Wide-stripe disk rebuilds (fast!!!)
• Convert random writes to sequential in memory • Stream detection (random vs sequential) • Set custom block size (4KB to 64KB) • Virtual Disk granular feature/DR deployment per server • REST, CLI, GUI, OpenStack (CINDER/SWIFT) and Docker
• Thin provisioning • Inline compression • Inline deduplication • Client side read-cache • Client side dedupe-cache
Elastic Architecture
Unified Protocol Access
Data Protection & Availability
Base Functionality
Data Efficiency & Performance
• Rack, DC and cloud aware DR Policies • Multi-site DR replications • Sync and semi-sync simultaneously • No limit in # DCs, all active sites
Disaster Recovery
THANK YOU!