Application-awarestorage

@LaurentDenel

Co founder & CEOat OpenIO

@GuilDelaporte Co founder & Product Managerat OpenIO

Agenda

2.

OpenIOWebUI

Storage Tiering

3.1.

What is OpenIO

4.

Kinetic Drives

5.

Hybrid Cloud with Backblaze B2

1. What isOpenIO

2006

Idea & 1st concept

2007

Designdev starts

2009

1st massiveproduction above 1PB

2012

Opensourced

2014

10+ PBmanaged in asingle cluster

About

OpenIOfork

2015

20 60employees millions end-users

San Francisco | Lille (FR) | Tokyo | Montreal

Use cases

> Email Platforms > Video Streaming > Object and file storage

• Storage-as-a-Service • Compute + Storage Platform • On-prem Data

> High frequency modifications

most

88 % and growing fast

iops

capacity

> Low frequency modifications & immutable data

Storage market evolution (80/20 rule)

IOPS Realm Hyper-Scale Capacity Realm

x86 servers Software-Defined Storage

Hyper ScalableStorage and Compute Platform

The OpenIO Solution

+ =

OpenIO transforms a rack of x86 servers into a large storage and compute pool

OpenIO Democratizes Large Data Platforms

Internet giants have initiated the wave and proved the technology

OpenIO offers the same model “on-premise “

> Simple > Elastic > Flexible > Cost effective > On-demand

OpenIO Approach

Object Storage

Scale and store 1000+ PBs of data, Billions of objects

Open Source softwareCommodity hardware

Reduce cost and TCO

Application-aware Grid for Apps

Run applications on the storage infrastructure

1. 2. 3.

HOW

…

……

Multi tenancy

Flat structure

Track containers, not objects

Directory with indirections

Containers Objects

grid://namespace/account/container/object

>

Namespace

Account

Container

Object

New nodes are automatically discovered and immediately available

Scaling

No consistent hashing algorithm > no recalculation of the key space

Grid of nodes with no consistent hashingNever rebalance>

1. Collects metrics from the services of each node

2. Computes a score for each service

3. Distributes scores to every node and client

4. On the fly best match making for each request

The score is computed with a configurable formula usually based on: capacity, io performance, CPU

ConscienceRealtime load balancing for optimal data placement

>

Grid for AppsData usage at the heart of the datacenter >

A data processing framework integrated inside OpenIO’s Grid

Scale-out application back-ends can be built on the storage platform itself

Avoid wasted resources and simplify load balancing for storage and processing

APIs and Specific App ConnectorsData at the heart of the datacenter >

Gateway layer

Optimized native APIs

• C • Python • Java • Go

Command line interface

Specific App Connectors

• Enterprise Storage • Email • Video streaming

REST APIs

• Amazon S3 • Openstack Swift

What makes us unique?

3.2.1.Ease of use

Easy to test, deploy in production, manage and use

0 TB > 1000+ PB

Start small and grow with your needs (from one engine for small config to very large ones)

Grid for Apps

Unifying Compute and Storage on a single platform

1. Grid for Apps

Ingest Search

Store

Full Text Index

Video workflow for user generated contents

Ingest Stream

Store

Transcode

Store

Spam sample long-term archiving and search

Scale-out real life use cases

2. From 0TB to 1000+PB

Start small! No compromise on 3 object storage characteristics

Fast

> Simple and standard APIs

> Data protection with replication or erasure coding on multiple nodes

> Scale-out capabilities to hyper-scale

> A 3 nodes cluster can be deployed in 5 minutes and be production ready

3. Ease of use

Full Operational Control

CLI available

Ubiquitous Management via Web GUI

2. OpenIO Web UI

What you can do in 5’

Drink an espresso

Stretch your body

Deploy an OpenIO cluster

3.Storage Tiering

OpenIO tiering is obvious

1.Built-in feature

Across storage devices managed by OIO or connected to an external pool

2.Multicriteria settings

Name, date, size, patterns, type, container name, etc.

3.Transparent

At upload, or with background jobs

OIO chunk Layer

OIO directory

Client side applications

OpenIO standard architecture

R R R R R R R R

Storage Array

x86 Commodity Servers

EthernetTCP/IP drives

1995 - 2005 2006 - 2016 2017 - …

Large capacity storage evolution - drive presentation cost cutting

4. Kinetic Drives

OpenIO & Kinetic

Thanks Seagate for the Ember system!

KineticOpen SourceProject Member

Participation to the Plugfest in RaleighApril ‘16

Working proof of concept in our lab

Ready to get certified during the next Plugfest

github.com/open-io/oio-kinetic

OpenIO standard architecture

OIO chunk Layer

OIO directory

Client side applications

R R R R R R R R

OpenIO Kinetic architecture NO CHANGE!

OIO Kinetic Layer

K K K K K K K K

OIO directory

Client side applications

5. Hybrid Cloud

OpenIO with Backblaze

June 2016 Gleb Budman, CEO

WHEN WHO

$5/Month = 30GB

“Backblaze Undercuts All Cloud Storage Competitors”

Key Metrics

30.000 developers & IT people signed up

200+ PB stored

Profitable Hundreds of PB in the B2 pipeline

IT struggling with archiving data explosion

1. Explosion of Data Volume

2. Complexity

3. Infrastructure cost

Current solution: Silos

Hot data stored on disk

Cold data stored on LTO

Archiving softwareMove cold data to cold storage

But the current solution has lots of problems

Backblaze B2 & OpenIO take up the challenge

Data is siloed LTO has very slow performance

High cost of having multiple solutions

1. 2. 3.

A hybrid cloud storage solution to rule them all

OpenIO single namespace & tiering

Backblaze B2public cloud No more "cold" storage

Backblaze B2 offered through OpenIO solution

Data is siloed LTO has very slow performance

High cost of having multiple solutions

1. 2. 3.

1. OpenIO single namespace & tiering

OIO chunk Layer

OIO directory

Client side applications

R R R R R

Blackblaze B2 objects

HybridCloud

GW

2. Backblaze B2 public cloud / No more "cold" storage

2.

High performance

Unlimited storage

3.1.

Data is instantly available

4.

Pay only for what you use

3. Backblaze B2 offered through OpenIO solution

OpenIO (on premise)

Backblaze B2 (cloud)

hybrid

Push your datacenter’s walls!

2. Fast Performance

3. 99.999999% durability

Backblaze B2 Fully integrated in OpenIO

1. Cost Effective

Only $.005/GB/month

No Capex Pay for what you use

All data are stored on in-production. Always online disks. Parallelization achieves fast throughput.

All data checked ever 30 days for degradation and rebuilt when necessary.

3 years TCO for 1PB

Legacy OpenIO + Backblaze B2

Storage array

3 years hosting

Tape gear

Backup software

Human admin

OIO subscription

x86 servers

3 years hostingHuman adminB2

(>40% archived in Backblaze B2)

do you still want to live in the ’80s?

Ease of Use

3.

5 minutes deployment

Kinetic Drives

4.

Ready for the next storage hardware

2.

Tiering

Built-in, multi criteria and transparent

Grid for Apps

1.

scale-out backend for applications

Wrapping up

Backblaze B2

5.

Unique hybrid cloud storage approach

Application-AwareStorage