introduction to cloud storage
TRANSCRIPT
2
5 trends in data center for 2016 you have to know
http://tecno.americaeconomia.com/articulos/5-tendencias-en-los-centros-de-datos-para-2016-que-debes-conocer?utm_source=dlvr.it&utm_medium=twitter
• Cloud deployment is complexing
• Architecture vs technology
• Common lenguage, common interface
• Safety social responsability
• Distributed Data Access
AETECNO
Jose Maria Alvarez PalletePresidente Telefonica@jmalvpa
3
Álvarez-Pallete: “Data is the OIL of XXI Century”
http://www.iese.edu/es/conoce-iese/prensa-noticias/noticias/2015/julio/alvarez-pallete-datos-son-petroleo-siglo-xxi/
• We have an important latent potential
• This is the TRUE Technology Disruption
• Simplification and Digitalization go together
• In 2019, at least, Four devices per person.
IESE Business School
Jose Maria Alvarez PalletePresidente Telefonica@jmalvpa
4
My Files have been ciphered and they’re bribering me to de-ciphered them!!!!.. 5 Advises to avoid a ‘Cryptolocker’
http://www.elladodelmal.com/2015/01/me-han-cifrado-los-archivos-y-me-piden.html
• Ransomware (malicious data-ciphered)
• How to avoid it????
• Keep your software update!!!!!!
• Fortify your system
• Do a back-up!!!!!!
BLOG El Lado del Mal
Chema AlonsoCDO Telefonica@chemaalonso
5
WHERE WE ARE
WHAT PROBLEMS DO WE HAVE
STaaS: STORAGE ABSTRACTION AND AUTOMATIZATION
CREATING STaaS (SDS) MODEL FOR OUR IT
APP VISION vs BYTE VISION
WHAT’S NEXT – DATA SERVICES (HDFS) AND HYBRID CLOUD (COMMODITY)
1
2
3
4
5
6
@julitojul101
6
WHERE WE ARE
WHAT PROBLEMS DO WE HAVE
STaaS: STORAGE ABSTRACTION AND AUTOMATIZATION
CREATING STaaS (SDS) MODEL FOR OUR IT
APP VISION vs BYTE VISION
WHAT’S NEXT – DATA SERVICES (HDFS) AND HYBRID CLOUD (COMMODITY)
1
2
3
4
5
6
@julitojul101
7
3ª PLATAFORMA
MILES DE MILLONES DE USUARIOS
MILLONES DE APPS
LAN/Internet Cliente/Servidor
PC
2ª PLATAFORMA
CIENTOS DE MILLONES DE USUARIOS DECENAS DE MILES DE APPS
Mainframe, Mini Computer
Terminales
1ST PLATFORM
MILLONES DE USUARIOS MILES DE APPS
2012-2016 Current Approach
Móvil Cloud Big Data Social
Dispositivos Móviles
8
ADVANCE DATA SERVICES
BASE DATA SERVICES
Different Apps
TransactionProcessing
OnlineArchive
VM Infrastructure
DecisionSupport
Full Motion Video
HighPerformanceComputing
WebApps
ContentManagement
ITApplications
HomeDirectories
ApplicationDevelopment
TechApps
9
HIGHEND
MIDTIER
SCALEOUT
ALL-FLASH
DATA Protection(BACK-UP)
Different Storage Requeriments…..
ScaleIO VSAN
CommodityHW x86
10
… and different Architecture Requeriments
11
What could Cloud Storage provide?ABSTRACTIÓN and AUTOMATIZATIÓN
With the performance and availability Requeriments
DEPLOY
IMPROVE
AVAILABILITY
Protect de Data, its continuity and its integrity
The use of the existing and new capacity
• Flash Technology• SAN, NAS, Object• Open
• Back-up• Data center Replication
• Resource optimization
12
Market OpportunityWhat are the Experts Saying about Cloud Storage?
• Total market size of ~$6B in 2015 with a CAGR of 47%*
o 80% ($4.8B) Raw storage services such as PaaS or IaaS
o 20% ($1.2B) Backup/Archiving 50-50 split
*Source: Cloud Computing, Sector Focus: Storage-as-a-Service, 451 Research Market Monitor, March 2012
80%
10%
10%
13
WHERE WE ARE
WHAT PROBLEMS DO WE HAVE
STaaS: STORAGE ABSTRACTION AND AUTOMATIZATION
CREATING STaaS (SDS) MODEL FOR OUR IT
APP VISION vs BYTE VISION
WHAT’S NEXT – DATA SERVICES (HDFS) AND HYBRID CLOUD (COMMODITY)
1
2
3
4
5
6
@julitojul101
14
InsufficientStaff
Lacking technical expertise
(ONE) Enterprise IT ChallengesWhy Do Organizations Stall?
IncreasingComplexity
Inconsistent, undocumented
processes
UnpredictableCosts
Cost/risk related to IT transformation
15
(TWO) Daily Challenges in IT –REAL
How long does it take IT stuff to deploy a Virtual Machine (VM) in production?
My Architecture stuff doesn’t exist… it’s in the operating department
Automatize provisioning?, HOW? I have a SIMO in my DC
ONE Management POINT?, what’s that?, I have SIX
Incidents kill us, and fit them more…
Daily Storage tickets fill IT stuff time
WE CANNOT
INNOVATE
DAILY TASKS KILL US
16
HOWEVER, business is moving much FASTER
MEASURE SUCCESS IN
YEARS
TRADITIONALIT TEAM
MEASURE SUCCESS IN
WEEKS
ENTREPRENURIALBUSINESS TEAM
17
… SO, The SHADOW IT shows up
Shadow IT
AWS is easy to deploy
AWS is cheaper
AWS is flexible(expand/reduce Storage)
Yes, but not at all.What Happen if in the
future do I want to move data?
May be some months, but it’s necessary to do a TCO
for 3 or 4 years
They are orphan services.Availability Services?Replication Services?
18
AWS services on/off premise
Amazon S3 Amazon EBSAWS Storage
Gateway
Amazon Glacier Amazon EC2 Amazon EMR
SDS Object SDS
Cloudstorage /Activearchive
Coldarchive
Cloudgateway
Blockstorage
IaaS Dataanalytics
Elastic Beanstalk
PaaS
Container
19
At this point, Can we do anything?Cloud Storage Delivers Business Agility
Cloud StorageBroker & Builder
New Business ModelNew Technology Architecture
New Operation models and roles
CULTURE
Shadow IT
AgileApps
Big Data
TECH
DevOpsMobile
BUSINESS
Apps
Customer Data
Speed
20
WHERE WE ARE
WHAT PROBLEMS DO WE HAVE
STaaS: STORAGE ABSTRACTION AND AUTOMATIZATION
CREATING STaaS (SDS) MODEL FOR OUR IT
APP VISION vs BYTE VISION
WHAT’S NEXT – DATA SERVICES (HDFS) AND HYBRID CLOUD (COMMODITY)
1
2
3
4
5
6
@julitojul101
21
How do we define STaaS?
• IT Architecture Model
• Business oriented
• Optimize costs
ABSTRACTING
AUTOMATE
22
ABSTRACTING and AUTOMATE
EMC
DIFFERENT APPROACHES EXIST
ALL IT STORAGE
DELL – HITACHI- NETAPP
23
•Cre
ate
a L
UN
•Cre
at
the Z
onin
g•
LU
N M
askin
g im
ple
menta
tion
•Cre
at
Sto
rage G
roup
•Assig
n it
to t
he s
erv
er
ABSTRACTING and AUTOMATE
THE PROVISION OF THAT STORAGE
Client 2
Client 3
Client 1 Server A
Application A
Server B
Application B
Server C
Application C
Disks for Server A
Disks for Server B
Disks for Server C
SANLocal Area
Network
24
What the Host Sees
A
One Logical VolumeMultiple Logical Volumes
ABC
D
Disco Fisico
25
• Storage Systems Architecture
Data Organization: Strips and Stripes
Stripe 1
Stripe 2
Stripe 3
Strips
26
ABSTRACTING and AUTOMATE
There are different models on different suppliers
DIFFERENT APPROACHES EXIST
ALL IT STORAGE
27
ABSTRACTING physical array technology
Intelligent Storage System
Cache
Front End Back End
Cache
Physical Disks
Host Connectivity
28
… like DELL EMC VMAX3
100K 400K200K
1 to 2 Engines
48 Cores
2TB Cache
1,440 2.5” Drives
500TB, 64 Ports
1 to 4 Engines
128 Cores
8TB Cache
2,880 2.5” Drives
2PB, 128 Ports
1 to 8 Engines
384 Cores
16TB Cache
5,760 2.5” Drives
4PB, 256 Ports
29
But STaaS has also to ABSTRACTs Functionality. AUTOTIERING
Active blocks (data) the best response time
blocks low/no use
SDDflash
HDDHigh
Capacity
30
What is AUTOTIERING?Place the right DATA, in the right TIER, in the right TIME
FC
Flash
SATA
Traditional FAST FAST VP
SAS
Flash
NL-SAS
31
Or Like – ALL-ROAD ARRAY (Unified)
VNX5200* VNX5400 VNX5600 VNX5800VNX7600
VNX-FVNX8000
Max FAST Cache 600 GB 1 TB 2 TB 3 TB 4.2 TB 4.2 TB
Max drives 125 250 500 750 1000 1500**
Drive types Flash, SAS, NL-SAS
File System modules
X-Blades 1 or 2 1 or 2 1 or 2 2 or 3 2 to 4 2 to 8
I/O slots per X-blade 3 3 3 4 4 5
Memory per X-blade 6 GB 6 GB 12 GB 12 GB 24 GB 24 GB
Protocols CIFS, NFS, pNFS
Storage Pool
modules
Storage Processors 2 2 2 2 2 2
I/O ports per SP 2 2 2 2 2 2
I/O slots per SP 3 4 5 5 5 11
Memory per SP 16 GB 16 GB 24 GB 32 GB 64 GB 128 GB
Cores per SP 4 4 4 6 8 16
Protocols Fibre Channel, Fibre Channel over Ethernet, iSCSI
32
MIDTIER – Unified Storage (All-Road Array)
Virtual Blended Storage Pool
Near-Line SAS
15K SASFLASH
Block Storage Processor
Data Mover File Storage Pool
Data Mover File Services
CIFS or NFS
LUNLUN
LUNLUN
33
… And NOW it’s on fashion, the AFA
$0,10
$1,00
$10,00
$100,00
2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020
$5.00
$0.50
$50.00
TLC
$/G
B
34
Híbridas optimizadasFlash• Controladora dual• Escalabilidad limitada• Cargas de trabajo simples
• Algunos Data Services
Gen 2
Arrays Scale Up todo Flash• Controladora dual• Escalabilidad limitada• Cargas simples (VDI)• Data Services? Quizás….• Elegir entre
• Rendimiento• Eficiencia
Gen 3
Arrays Scale Out todoFlash• Multi Controladora (16-XtremIO)
• Escala hasta PBs• Cargas de trabajo mixtas• In-Line Data Services
• TCO mínimo• Rendimiento máximo
• DataCenter All-Flash-Array
Gen 4
Gen
1 Flash
Sys
tems
Leg
acy
Disk A
rray
s
Gen 1
… And NOW it’s on fashion, the AFA
35
WHERE WE ARE
WHAT PROBLEMS DO WE HAVE
STaaS: STORAGE ABSTRACTION AND AUTOMATIZATION
CREATING STaaS (SDS) MODEL FOR OUR IT
APP VISION vs BYTE VISION
WHAT’S NEXT – DATA SERVICES (HDFS) AND HYBRID CLOUD (COMMODITY)
1
2
3
4
5
6
@julitojul101
36
Cloud Storge Model -Software-Defined Storage
MID TIER
SCALEOUT
3rd PartyHIGHEND
Commodity
SDS Platform
ProvisioningSelf-
ServiceReporting Automation
STaaSData Services
STaaSController
37
STaaS = ¿Virtualización?
Storage Federation
VPLEXSVC
vSANScaleIO
Software Define Storage
Storage Abstraction by SOFTWARE
Physical and Virtual ARRAYS
38
IF STaaS doesn’t exist……
x86
Virtualized
EMC
VM11111001010001101101100101
01110
10
00
10
00
NetApp
V
Scripts
V
Otherscripts???
39
1111100101000110110110010111110000101001010101
… If STaaS Exists
x86
Virtualized
EMC
SDSVM
ControllerMONITOR
CO
NT
RO
L
11111001010001101101100101
01110
10
00
10
00
NetApp
ONE API
MONITOR
CO
NT
RO
L
01110
10
00
10
00
40
Virtualized Storage with SDS
x86
Virtualized
EMC
SDSVM
ControllerMONITOR
CO
NT
RO
L
11111001010001101101100101
01110
10
00
10
00
SDSData Services
ONE API
41
3 Simple Steps to storage provisioning the easy way
Automate Storage
Discover and register arrays– Identify physical storage and fabrics
– Scan and map
Configure virtual arrays– Partition physical infrastructure
Configure virtual storage pools– Define policies and service levels
42
Without compromiseStorage Virtualization (I)
VMAX Hitachi NetApp
RUBY virtual SAN
GOLD virtual SAN
SILVER virtual SAN
Tenant 1
Tenant 2
Tenant 3
vPOOL-R1 vPOOL-G1 vPOOL-S1
vPOOL-S2vPOOL-G2
vPOOL-R2 vPOOL-G3
43
Without CompromiseStorage Virtualization (II)
Abstraction of physical arrays– Abstract Control plane from the data plane
– Divide storage into virtual storage arrays managed by policies
Use the full value of underlying physical arrays– Be able to use advanced array capabilities
(e.g. replication, tiering…)
Create virtual storage pools– Define storage services in software
STaaS
44
WHERE WE ARE
WHAT PROBLEMS DO WE HAVE
STaaS: STORAGE ABSTRACTION AND AUTOMATIZATION
CREATING STaaS (SDS) MODEL FOR OUR IT
APP VISION vs BYTE VISION
WHAT’S NEXT – DATA SERVICES (HDFS) AND HYBRID CLOUD (COMMODITY)
1
2
3
4
5
6
@julitojul101
45
Complete Automatization of Service
Connect to Virtualization
DEPLOYED
Provision Compute
Hand-off Validation
Provision Storage Hand-off
Validation
Provision NetworkTRADITIONAL PROCESS Hand-off
Validation
• Server Admin
– Allocate Cisco UCS blades
– Create logical service profiles
• Allocate logical identifiers
– MAC addresses, UUIDs (compute)
– WWNN, WWPN (storage)
• Storage Admin
– Configure the zones on the SAN
– Provision block storage, masking resources
• Network Admin
– Connect resources to networks (inside of the Vblock platform, not connecting Vblock platforms to external)
• VMware Admin
– Install VMware ESX
– Format data stores
– Create VMware ESX cluster in a particular VMware vCenter Server
– Configure DRS HA Clusters
– Register VMware ESX host and data stores with the VMware vCenter Server cluster
• Unified PRovisioning
• Compute, Storage, Network,
Virtualization
DEPLOYED
• Class of Services
• Automatized
• Fast Delivery
46
THE COMPLEXITY OF A COMPLETE AUTOMATIZATION
BIG ORCHESTRATOR
BMC-CLMvRealize
ABSTRACTING
STORAGE
VIRTUALIZE ALL IT
STORAGE
AUTOMATE
Storage
IaaS, VMs at Tiered Service
Levels
47
GENERIC Service Workflow…
Std.Offering?
ServiceRequest
Portal
No
ServiceCatalog
CustomAllowed?
No
CMDB
Service
Broker
Commence
Provisioning
Workflow
Yes OrchestrationEngine
Compute Domain
Network Domain
Storage Domain
Services Domain
HypervisorOSApp
L2L3
Tier/CapacityBCDR
L4/SLBSecurity
Billing
ServiceCompilation
Start
End
ITSMDomainMonitoringComplianceGovernance
47
48
BMC-CLM vision of IaaS(with STaaS)
Std.Offering?
ServiceRequest
Portal
No
ServiceCatalog
CustomAllowed?
No
CMDB
Service
Broker
Commence
Provisioning
Workflow
Yes OrchestrationEngine
Compute Domain
Network Domain
Storage Domain
Services Domain
HypervisorOSApp
L2L3
Tier/CapacityBCDR
L4/SLBSecurity
Billing
ServiceCompilation
Start
End
ITSMDomainMonitoringComplianceGovernance
BMC CLM
BMC CLM
BMC CLM BMC CLM
IaaS (STaaS embedded)
48
49
Vmware vision of IaaS (with STaaS)
VIRTUALIZED Infrastructure
Traditional Storage
ApplicationSelf-service
Big Data AppSelf-service
Encryption Services
Disaster Recovery-as-a-Service
Backup-as-a-Service
IaaS
vRealize Business
FinancialTransparency
vRealize Operations
Operational Management
Self-Serviceand Automation
vRealize Automation
Converged Infrastructure
STaaS
50
STaaS Provides them….
Unified storage views across the Data Center
Heterogeneous storage array management
Policy based management of storage pools abstracted from
hardware
Leverage the capabilities of the underlying hardware platforms
Storage network management
New interface support
51
STaaS has to be integrated with …
vSpherevSAN / vVol
vCloud Automation Center
vCenter OpsManager
SDSController
SDSData Services
3rd Party CommodityMIDTIER FILER
HIGHEND
HDFS
STaaS
52
STaaS has to be opened
SDSData Services
SDSController
We can choose (Vmware, HyperV, OpenStack)
3rd Party CommodityMIDTIER FILER
HIGHEND
HDFS
STaaS
53
What is OpenStack?OpenStack Cloud Operating System
Connects to Apps via API
Creates Pools of Resources Automates the Network
Self-Service Portals for Users
Swift
Glance
Nova
Cinder
Neutron
Trove Heat
Keystone Horizon
Celiometer
54
OpenStack/STaaS RelationshipCloud Storage can be a part of an OpenStack cloud
• Cloud Storgeimplements the Swift REST API, replacing the open source object store
• Glance may use Swift or S3 APIs in STaaS for backing store, so support is indirect
• Cinder plug-in provides adapter from OpenStack to API calls for block storage
Nova (Compute)
Cinder(Block Volume)
Message Bus
Swift(Object)
OpenStack users/tenants/admins
OpenStack/EC2 API
Neutron(Network)
Horizon (GUI)
Keystone (IAM)
Glance(Image)
Cloud Storage
55
… SO, in summary, everything FIT with STaaS
BIG ORCHESTRATOR
BMC-CLMvRealize
ABSTRACTING
STORAGE
VIRTUALIZE ALL IT
STORAGE
AUTOMATE
Storage
IaaS, VMs at Tiered Service
Levels
56
WHERE WE ARE
WHAT PROBLEMS DO WE HAVE
STaaS: STORAGE ABSTRACTION AND AUTOMATIZATION
CREATING STaaS (SDS) MODEL FOR OUR IT
APP VISION vs BYTE VISION
WHAT’S NEXT – DATA SERVICES (HDFS) AND HYBRID CLOUD (COMMODITY)
1
2
3
4
5
6
@julitojul101
57
UNSTRUCTURED DATA GROWTH
2015
71 EB
Total Capacity Shipped, Worldwide
% of Unstructured Data
75%
78%
80%
2016
106 EB
2017
133 EB
58
STaaS with Commodity Hardware
© Copyright 2014 EMC Corporation. All rights reserved.
Servers
Network
StorageConverged Architecture200,000 IOPS
20K IOPS20K IOPS20K IOPS20K IOPS20K IOPS20K IOPS 20K IOPS20K IOPS20K IOPS20K IOPS
Flash | SSD | HDD | RAID Cache | RAM
Bare Metal | KVM | VMware | Hyper-V
59
STaaS: Scalability
Scale from 3 nodes to 1000s of nodes
Add storage devices and servers on the fly to increase capacity and performance
Storage growth always automatically aligned with application needs
60
For all ‘aaS’ models we decide
Bare Metal
Add, remove, re-allocate, on the fly, without stopping IO
HypervisorsOperating System PlatformsPCIe FLash
SSDs
HDDs
Media
61
NAS
SAN CLOUD
TAPE
DAS
OBJECT
61
NEXT-GEN WORKLOADSTRADITIONAL WORKLOADS
HPC
Backup/Archive
Analytics
Mobile
File Shares
Cloud Apps
62
TAPE
NAS DAS
CLOUDSAN
OBJECT
Data LakeFoundation
62
NEXT-GEN WORKLOADSTRADITIONAL WORKLOADS
HPC
Backup/Archive
Analytics
Mobile
File Shares
Cloud Apps
63
NEXT-GEN ACCESS METHODS
FILE
FILE
63
HPC
Backup/Archive
Analytics
Mobile
File Shares
Cloud Apps
64
SCALE-OUT NAS ARCHITECTURE
Clients and Applications
RESTful APIGET PUT POST DELETE
Gig-e10 Gig-eNetwork
OneFS Operating Environment
Multi-ProtocolClient/Application
LayerEthernet Layer
Protocols
SMBNFS
FTPHTTP
HDFSfor
Hadoop
RESTfor
Object
Intra-cluster Communication
65
USE CASE: HADOOP
66
FEDERATION ENTERPRISE HYBRID CLOUDREDUCE COST AND INCREASE AGILITY IN IT
© Copyright 2015 EMC Corporation. All rights reserved.
FEW WEEKS TO BEUP & RUNNING
75% FASTER Infrastructure Deployment
95% FASTER IaaS, PaaS, AaaS, DBaaS DEPLOYMENT
SIMPLE
AGILE SECURE
67© Copyright 2015 EMC Corporation. All rights reserved.
Virtualize Infrastructure
BYO
vRealize Business
FinancialTransparency
vRealize Operations
Operational Management
Self-Serviceand Automation
vRealize Automation
ENTERPRISE HYBRID CLOUD, from STaaS to ITaaS
CLOUD Services Broker
Q4
Accelerated DeploymentConverged Infrastructure
Business Continuity
VPLEXRecoverPoint
Backup andRecovery
Data DomainAvamar
Service Catalog
EMC Cloud Service Providers
STaaS
68
CONCLUSIONESNo CLOUD PLEASE, We’re Bankers!!!
http://www.bankingtech.com/406232/bank-it-cloud/#.Vl7OGmIChEQ.twitter
• Since 2009, only 1% of worldwide banks have
their CORE in CLOUD
• Email (67%), CRM (52%), Storage (35%)
• Move budget from maintenance to Transformation
• However, It seems that there will be a change in
2016
BANKING TECH
Pepe OlallaBBVA@PepeOlalla
69© Copyright 2015 EMC Corporation. All rights reserved.
We have a lot of things to do and we have a lot of business to coverCONCLUSIONS
WHERE WE AREA LOT OF CUSTOMER DOESN’T KNOW HOW TO BEGIN WITH THEIR TRANSFORMATION
Cloud Storage is the BASELINE for the Transformation NOT always but the DATA has to be inside the Data Center.
HOW TO BUILT ITSTaaS ABSTRACTING AND AUTOMATE and also Manage al our IT Stuff
BMC, Vmware, MicrosoftThey have a different vision, but it has to FIT with STaaS
COMMODITY + DATA LAKE + CLOUD HIBRIDAWhat is on fashion Now????
@julitojul101
70
CONCLUSIONSHow important is this slide, how important!!!!!
@W
IKIB
on
GRACIAShttp://www.xataka.com/otros/lo-ultimo-de-amazon-es-un-camion-para-llevar-tus-datos-hasta-su-nube-snowmobile-no-es-una-broma