a new platform for a new era emc
DESCRIPTION
TRANSCRIPT
A NEW PLATFORM FOR A NEW ERA
O
Shahaf Azriely
Sr. Field Engineer, Israel Pre-Sale Manager SEMEA
© Copyright 2013 Pivotal. All rights reserved.© Copyright 2013 Pivotal. All rights reserved.
Who is Pivotal?
© Copyright 2013 Pivotal. All rights reserved.© Copyright 2013 Pivotal. All rights reserved.
Introducing Pivotal
Led by CEO, Paul Maritz, former CEO of VMware
Redefining Enterprise Platform-as-a-Service
Enabling a new class of applications, leveraging big & fast data, with the power of cloud independence
© Copyright 2013 Pivotal. All rights reserved.
Integrating EMC and VMWare Assets
Cloud Storage
Virtualization
Pivotal DataFabric
Pivotal CloudFabric
Data-DrivenApplication
Development
Pivotal Data Science Labs
...ETC
© Copyright 2013 Pivotal. All rights reserved.
Pivotal Data Fabric
Cloud Storage
Virtualization
Pivotal CloudFabric
Data-DrivenApplication
Development
Pivotal Data Science Labs
...ETC
Pivotal DataFabric
© Copyright 2013 Pivotal. All rights reserved.
Enterprise Data Architecture
AnalyticData Marts
MPP Database
OperationalIntelligence
In-Memory DB
Run-TimeApplications
In-Memory Object
Enterprise Data WarehouseRDBMS
Data StagingPlatformData
IngestionSystem
Stream/CEP
© Copyright 2013 Pivotal. All rights reserved.
AnalyticData Marts
OperationalIntelligence
Run-TimeApplications
Enterprise Data Warehouse
Data StagingPlatformData Ingestion
System
Pivotal Data Portfolio Today
© Copyright 2013 Pivotal. All rights reserved.
Multi-Target Deployment Model
depl
oy
Portable
Elastic
Promotable
HW abstracted
Manageable
Public Cloud
Private Cloud
On Premise
© Copyright 2013 Pivotal. All rights reserved.© Copyright 2013 Pivotal. All rights reserved.
PIVOTAL HDThe Foundation for Change
© Copyright 2013 Pivotal. All rights reserved.
Our Big Bets for the Future
1. HDFS becomes the data substrate for the next generation of data infrastructures
2. A set of integrated, enterprise-scale services will evolve on top of HDFS – stream ingestion, analytical processing, and transactional serving
3. Provisioning flexibility and elasticity become critical capabilities for this data infrastructure
© Copyright 2013 Pivotal. All rights reserved.
Did You Know?
Our HD distribution has been scale-tested on our unique, 1,000-node Analytics Work Bench
Our distribution is the first to bundle VMWare’s Hadoop Virtualization Extensions (HVE)
We are backed by EMC’s global, 24x7 support infrastructure
Available as a software-only or appliance-based solution
© Copyright 2013 Pivotal. All rights reserved.
Hadoop Pain Points
• No Integrated Hadoop Stack• Hadoop, Pig, Hive, HBase, Zookeeper, Oozie, Mahout…Integrated Product Suite
• No Industry standard ETL and BI Stack Integration• Informatica, Microstrategy, Business Objects …Interoperability
• Poor Job and Application Monitoring Solution• Non-existent Performance MonitoringMonitoring
• Complex System Configuration and Manageability• No Data Format Interoperability & Storage Abstractions
Operability and Manageability
• Poor Dimensional Lookup Performance• Very poor Random Access and Serving PerformancePerformance
© Copyright 2013 Pivotal. All rights reserved.
The Pivotal Position on Hadoop
Hadoop fits Pivotal’s strategy based on open source innovation for Big Data analytics
– Hadoop and Pivotal are complementary technologies
Hadoop needs to become mission-critical and easier to use and manage for enterprise customers
– Lacks operational interfaces and high-level tooling for big data analysis
– Pivotal HD addresses these challenges offering robust operational tools and with Advanced Database Services powered by HAWQ
– HAWQ is the first true SQL processing engine that runs on Hadoop
Why Hadoop?
© Copyright 2013 Pivotal. All rights reserved.
Pivotal HD Enterprise1.0
Commercially supported distribution of Apache Hadoop 2.0 – HDFS, MapReduce 2.0, YARN, Pig, Hive, HBase,
Mahout, Zookeeper, Flume, Sqoop, Hadoop Virtualization Extensions (HVE)
– Spring Hadoop integrates the Spring Framework into Hadoop
▪ Create and run Hadoop MapReduce, Hive and Pig jobs▪ Work with HDFS and HBase
Open Source Apache Stack
© Copyright 2013 Pivotal. All rights reserved.
Pivotal HD Open Source Components
•Hadoop Distributed File System HDFS•Processing framework for writing scalable data applicationsMapReduce•Procedural language that abstracts lower level MapReducePig•Highly reliable distributed coordinationZookeeper•System for querying data on top of HDFS (SQL-like query)Hive•Database for random, real time read/write accessHBase•Scalable machine learning librariesMahout
© Copyright 2013 Pivotal. All rights reserved.
Pivotal HD Components
•Cluster installation, upgrade and expansion tools ICM
•Visual interface to monitor jobs, cluster health, system metricsCommand Center
•Supports virtual node awareness HVE
•Virtual resource partitioning and performance monitoringMore-VRP
•Enterprise grade NAS-based storage option for HadoopIsilon Integration
•SQL query processor based on GPDB running on HDFSHAWQ
•Extension Framework component of HAWQ to create external tablesGPXF
© Copyright 2013 Pivotal. All rights reserved.
Command
Center
&
More-VRP
ICM Deployment&
Configuration
DataLoader
XtensionFramework
CatalogServices
QueryPlanner
Dynamic Pipelining
HAWQ
HDFSHadoop Virtualization
HBase
Pig, Hive & Mahout
Map Reduce
Sqoop Flume
Resource Management & Workflow
Yarn
Zookeeper
Chorus
Partner Tools and Applications
Spring
Spring Data Framework
ANSI SQL + Analytics
Collaboration & Orchestration
Applications
Apache Pivotal HD Added Value Pivotal Partners
Pivotal HD
Cetas
MoreVRP
Pivotal HD Architecture
© Copyright 2013 Pivotal. All rights reserved.
Powerful Partner Ecosystem
© Copyright 2013 Pivotal. All rights reserved.
Powerful Partner Ecosystem
© Copyright 2013 Pivotal. All rights reserved.
Use Cases
Pivotal HD & HAWQ GA will come available by end of 05/13.
1. Retail – leavreging for an enterprise data lake. All data will flow into PivHD HDFS. Some will be loaded into HAWQ.
2. Telco – petabytes of data with network/cell phone tower data will be stored in PivHD and HAWQ for faster analytics.
3. Financial – migration from GPDB to leverage GPXF to allow interconnection to Hbase.
More information Under NDA
© Copyright 2013 Pivotal. All rights reserved.© Copyright 2013 Pivotal. All rights reserved.
L E A R N M O R E
goPivotal.com F O L L O W U S
@gopivotal
Shahaf [email protected]