big data @ storage memory convergence · native data structures into block i/o conventional i/o...

18
Fusion-io Confidential—Copyright © 2013 Fusion-io, Inc. All rights reserved. Copyright © 2013 Fusion-io, Inc. All rights reserved. Big Data @ Storage Memory Convergence Challenges & Opportunities Pankaj Mehra, SVP & WW CTO

Upload: others

Post on 09-Jun-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Fusion-io Confidential—Copyright © 2013 Fusion-io, Inc. All rights reserved. Copyright © 2013 Fusion-io, Inc. All rights reserved.

Big Data @ Storage Memory Convergence

Challenges & Opportunities

Pankaj Mehra, SVP & WW CTO

Page 2: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Big Data trends

May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 2

▸ Almost universally about figuring out the larger context of human actors

▸ Sharpening relevance in moments of truth ▸ Through rapid contextualization of situational signal ▸ With slow and pervasive integration (space, time) of

other dimensions of human existence ▸ By context engines capable of identifying 10 or so

useful items from perhaps 1012 digital breadcrumbs

Mobile Social Sensed Contextual

Page 3: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 3

Page 4: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 4

Page 5: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

The Changing Face of IT IT ▸ Mountains of unprocessed

information ▸ Capacity- and cost-oriented

IT operations ▸ People-heavy information

processes and workflows ▸  IT-imposed look and feel in

information systems ▸  Information must be found

IT

▸ Extracted, correlated, and contextualized information

▸ Quality oriented optimizations ▸ Agency, automation and

affordances ▸ Contextualized remapping

into end user world view ▸  Information finds context

Page 6: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Where the rubber meets the road …

5/22/13 6

▸ As is •  85h to contextualize! •  Unforgiving infrastr.

▸ To be •  Log everything! (1014) •  Complex MDM (<10 s) •  Many enrichment algos •  UX = ability to put any

piece of right data into the customer’s hands at a moment’s notice, often before the query

•  Think Google Now, FB Taste, realtime MBRs

Business value of

information

Time since ingest

Expectedness

Raw, new information

Contextualized information

Net gain from contextualization

“right time”

Page 7: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Lessons

May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 7

▸ What I didn’t mention? •  SQL vs NoSQL •  Optimizing DBs, SEs •  20:20 analytics

▸ Why? •  Moving bits around to

optimize for disk overheads is no longer the game

•  New era, new primitives ▸ Sparsity, atomicity

▸ Custom apps, behind the firewall infrastr!

▸ How to scale out when you must •  Processing in store path

and in query path •  Can’t add servers just to

get more DRAM •  Graceful degradation of

CPU overhead @ large, low latency and low cost persistent memory

Page 8: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Comparative architecture models

IT’s tools ▸  Search engines &

databases

IT’s tools ▸ foraging loop (scent)

▸ sensemaking loop (intent)

▸ contextual delivery

Q Q,Q’,… D,D’,…

D

extraction, correlation

D Q Q Q profiles & contexts

D R,R’,.. R(D), R’(D), …

T

Page 9: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Convergence: An Evolving Notion

May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 9

2002-2006 2006-2013

WHAT Moving Intelligence closer to Storage

Moving Persistence closer to Intelligence

WHY Storage media unsuitable for bringing into CPU-memory complex

Solid state media increasingly suitable for

memory like access

HOW

Storage grid •  Offload content

mining & metadata Data grid •  Offload indexing &

query processing RD

MA

-acc

essi

ble

pers

iste

nt m

emor

y Scale out server-attached NVM

•  I/O Memory (PCIe) •  Flash as ACM/ExtMem

•  Storage-class memory

Page 10: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

The ideas in play

May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 10

▸ Memory-semantic access

▸ Familiar storage management metaphor

▸ Media wear leveling

▸ Disk-era protocols ▸ Disk-era optimizations

Looking good Questionable

Page 11: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

⌃ 2006

⌃ 2007

⌃ 2008

⌃ 2009

⌃ 2010

⌃ 2011

⌃ 2012

Mission to consolidate memory and storage

ioMemory technology unveiled

First products launched

1 million IOPS IBM Quicksilver

Dell strategic investment

HP OEMs products

IBM OEMs products

Samsung strategic investment

Dell OEMs products

VSL introduced

IPO on NYSE ioTurbine acquired

ioDrive2 announced

50+ Petabytes shipped

Supermicro OEMs products

1 Billion IOPS 2,500 customers

>120 channel and alliance partners

ioFX announced ioMemory SDK introduced

Cisco OEMs products

Fusion-io First Mover Milestones

May 22, 2013 11

Page 12: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

ioMemory Portfolio

12

Up to 1.2TB Up to 2.4TB Up to 10.24TB maximize performance for large data sets

420GB of workstation acceleration

160GB to 640GB of Fusion’s industry-leading ioMemory

160GB to 1.28TB of Fusion’s industry-leading ioMemory

3.2TB to 12.8TB for Web & Cloud Environments

Page 13: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Evolution of Enterprise Flash

FLASH + DISK FLASH AS DISK FLASH BEYOND DISK FLASH AS MEMORY

Page 14: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Evolution of Enterprise Flash FLASH AS DISK

Application

Application source code converts native data structures into block I/O

Conventional I/O Access

Block I/O

Proprietary Storage OS

F L A S H B E Y O N D D I S K

Application

Application source code does I/O with native data structures

Native: Enhanced I/O

Atomic I/O Transaction

Key-Value Transaction

User-Defined Object

Transaction

Open Interface Layer

F L A S H A S M E M O R Y

Application

Application source code manipulates native data structures

directly in persistent memory

Native: Persistent Memory

High-speed Logging

Memory Transaction

Checkpointed Memory

Open Interface Layer

Page 15: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Hybrid Memory-Storage Hierarchy

May 22, 2013 15

L1, L2, L3 CPU Caches DRAM Persistent Memories Flash Hard Drive

M i c r o s e c o n d s N a n o s e c o n d s

CYCLES TO WAIT

Hybrid Memory Storage Hierarchy

Main Memory System High Performance Disk System

Accessed Like Memory and Managed Like Storage

M i l l i s e c o n d s

ACCESS DELAY

2 c y c l e s 4 m i l l i o n c y c l e s

Page 16: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Fusion-io Accelerates

16

ANALYTICS SEARCH

ORACLE Text

MESSAGING

MQ

DATABASES

INFORMIX

VIRTUALIZATION

KVM

HPC

GPFS

BIG DATA

SECURITY/LOGGING

COLLABORATION

Lotus

DEVELOPMENT WEB

LAMP

CACHING WORKSTATION

Page 17: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

Comprehensive Customer Success

17 30+ case studies at http://fusionio.com/casestudies

F I N A N C I A L S M A N U F A C T U R I N G / G O V E R N M E N T

W E B T E C H N O L O G Y R E T A I L

FASTER DATA WAREHOUSE QUERIES 40x QUERY

PROCESSING THROUGHPUT 15x

The world’s leading Q&A site

®

FASTER DATABASE REPLICATION 30x FASTER DATA

ANALYSIS 5x FASTER QUERIES 15x

Page 18: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source

▸ THANK YOU