big data @ storage memory convergence · native data structures into block i/o conventional i/o...
TRANSCRIPT
![Page 1: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/1.jpg)
Fusion-io Confidential—Copyright © 2013 Fusion-io, Inc. All rights reserved. Copyright © 2013 Fusion-io, Inc. All rights reserved.
Big Data @ Storage Memory Convergence
Challenges & Opportunities
Pankaj Mehra, SVP & WW CTO
![Page 2: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/2.jpg)
Big Data trends
May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 2
▸ Almost universally about figuring out the larger context of human actors
▸ Sharpening relevance in moments of truth ▸ Through rapid contextualization of situational signal ▸ With slow and pervasive integration (space, time) of
other dimensions of human existence ▸ By context engines capable of identifying 10 or so
useful items from perhaps 1012 digital breadcrumbs
Mobile Social Sensed Contextual
![Page 3: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/3.jpg)
May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 3
![Page 4: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/4.jpg)
May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 4
![Page 5: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/5.jpg)
The Changing Face of IT IT ▸ Mountains of unprocessed
information ▸ Capacity- and cost-oriented
IT operations ▸ People-heavy information
processes and workflows ▸ IT-imposed look and feel in
information systems ▸ Information must be found
IT
▸ Extracted, correlated, and contextualized information
▸ Quality oriented optimizations ▸ Agency, automation and
affordances ▸ Contextualized remapping
into end user world view ▸ Information finds context
![Page 6: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/6.jpg)
Where the rubber meets the road …
5/22/13 6
▸ As is • 85h to contextualize! • Unforgiving infrastr.
▸ To be • Log everything! (1014) • Complex MDM (<10 s) • Many enrichment algos • UX = ability to put any
piece of right data into the customer’s hands at a moment’s notice, often before the query
• Think Google Now, FB Taste, realtime MBRs
Business value of
information
Time since ingest
Expectedness
Raw, new information
Contextualized information
Net gain from contextualization
“right time”
![Page 7: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/7.jpg)
Lessons
May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 7
▸ What I didn’t mention? • SQL vs NoSQL • Optimizing DBs, SEs • 20:20 analytics
▸ Why? • Moving bits around to
optimize for disk overheads is no longer the game
• New era, new primitives ▸ Sparsity, atomicity
▸ Custom apps, behind the firewall infrastr!
▸ How to scale out when you must • Processing in store path
and in query path • Can’t add servers just to
get more DRAM • Graceful degradation of
CPU overhead @ large, low latency and low cost persistent memory
![Page 8: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/8.jpg)
Comparative architecture models
IT’s tools ▸ Search engines &
databases
IT’s tools ▸ foraging loop (scent)
▸ sensemaking loop (intent)
▸ contextual delivery
Q Q,Q’,… D,D’,…
D
extraction, correlation
D Q Q Q profiles & contexts
D R,R’,.. R(D), R’(D), …
T
![Page 9: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/9.jpg)
Convergence: An Evolving Notion
May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 9
2002-2006 2006-2013
WHAT Moving Intelligence closer to Storage
Moving Persistence closer to Intelligence
WHY Storage media unsuitable for bringing into CPU-memory complex
Solid state media increasingly suitable for
memory like access
HOW
Storage grid • Offload content
mining & metadata Data grid • Offload indexing &
query processing RD
MA
-acc
essi
ble
pers
iste
nt m
emor
y Scale out server-attached NVM
• I/O Memory (PCIe) • Flash as ACM/ExtMem
• Storage-class memory
![Page 10: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/10.jpg)
The ideas in play
May 22, 2013 Copyright © 2013 Fusion-io, Inc. All rights reserved. 10
▸ Memory-semantic access
▸ Familiar storage management metaphor
▸ Media wear leveling
▸ Disk-era protocols ▸ Disk-era optimizations
Looking good Questionable
![Page 11: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/11.jpg)
⌃ 2006
⌃ 2007
⌃ 2008
⌃ 2009
⌃ 2010
⌃ 2011
⌃ 2012
Mission to consolidate memory and storage
ioMemory technology unveiled
First products launched
1 million IOPS IBM Quicksilver
Dell strategic investment
HP OEMs products
IBM OEMs products
Samsung strategic investment
Dell OEMs products
VSL introduced
IPO on NYSE ioTurbine acquired
ioDrive2 announced
50+ Petabytes shipped
Supermicro OEMs products
1 Billion IOPS 2,500 customers
>120 channel and alliance partners
ioFX announced ioMemory SDK introduced
Cisco OEMs products
Fusion-io First Mover Milestones
May 22, 2013 11
![Page 12: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/12.jpg)
ioMemory Portfolio
12
Up to 1.2TB Up to 2.4TB Up to 10.24TB maximize performance for large data sets
420GB of workstation acceleration
160GB to 640GB of Fusion’s industry-leading ioMemory
160GB to 1.28TB of Fusion’s industry-leading ioMemory
3.2TB to 12.8TB for Web & Cloud Environments
![Page 13: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/13.jpg)
Evolution of Enterprise Flash
FLASH + DISK FLASH AS DISK FLASH BEYOND DISK FLASH AS MEMORY
![Page 14: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/14.jpg)
Evolution of Enterprise Flash FLASH AS DISK
Application
Application source code converts native data structures into block I/O
Conventional I/O Access
Block I/O
Proprietary Storage OS
F L A S H B E Y O N D D I S K
Application
Application source code does I/O with native data structures
Native: Enhanced I/O
Atomic I/O Transaction
Key-Value Transaction
User-Defined Object
Transaction
Open Interface Layer
F L A S H A S M E M O R Y
Application
Application source code manipulates native data structures
directly in persistent memory
Native: Persistent Memory
High-speed Logging
Memory Transaction
Checkpointed Memory
Open Interface Layer
![Page 15: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/15.jpg)
Hybrid Memory-Storage Hierarchy
May 22, 2013 15
L1, L2, L3 CPU Caches DRAM Persistent Memories Flash Hard Drive
M i c r o s e c o n d s N a n o s e c o n d s
CYCLES TO WAIT
Hybrid Memory Storage Hierarchy
Main Memory System High Performance Disk System
Accessed Like Memory and Managed Like Storage
M i l l i s e c o n d s
ACCESS DELAY
2 c y c l e s 4 m i l l i o n c y c l e s
![Page 16: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/16.jpg)
Fusion-io Accelerates
16
ANALYTICS SEARCH
ORACLE Text
MESSAGING
MQ
DATABASES
INFORMIX
VIRTUALIZATION
KVM
HPC
GPFS
BIG DATA
SECURITY/LOGGING
COLLABORATION
Lotus
DEVELOPMENT WEB
LAMP
CACHING WORKSTATION
![Page 17: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/17.jpg)
Comprehensive Customer Success
17 30+ case studies at http://fusionio.com/casestudies
F I N A N C I A L S M A N U F A C T U R I N G / G O V E R N M E N T
W E B T E C H N O L O G Y R E T A I L
FASTER DATA WAREHOUSE QUERIES 40x QUERY
PROCESSING THROUGHPUT 15x
The world’s leading Q&A site
®
FASTER DATABASE REPLICATION 30x FASTER DATA
ANALYSIS 5x FASTER QUERIES 15x
![Page 18: Big Data @ Storage Memory Convergence · native data structures into block I/O Conventional I/O Access Block I/O Proprietary Storage OS FLASH BEYOND DISK Application Application source](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f046b007e708231d40dde90/html5/thumbnails/18.jpg)
▸ THANK YOU