hortonworks presentation at big data london

© Hortonworks Inc. 2013

Hortonworks Enterprise Apache Hadoop

March 5, 2013


Hortonworks

•  Who is Hortonworks

•  Our Approach

•  Customer Use Cases


Housekeeping Items

•  Restrooms on 2nd and 4th Floors

•  Hadoop Summit –  March 20-21 in Amsterdam –  PreConference Training on March 18-19

–  Discount Code Amst13Spon20

•  Download SandBox –  QR Code at postcode on table


A Brief History of Apache Hadoop

2013

Focus on INNOVATION 2005: Yahoo! creates

team under E14 to work on Hadoop

Focus on OPERATIONS 2008: Yahoo team extends focus to

operations to support multiple projects & growing clusters

Yahoo! begins to Operate at scale

Enterprise Hadoop

Apache Project Established

Hortonworks Data Platform

2004 2008 2010 2012 2006

STABILITY 2011: Hortonworks created to focus

on “Enterprise Hadoop“. Starts with 24 key Hadoop engineers from Yahoo


Hortonworks Snapshot

•  We distribute the only 100% Open Source Enterprise Hadoop Distribution: Hortonworks Data Platform

•  We engineer, test & certify HDP for enterprise usage

•  We employ the core architects, builders and operators of Apache Hadoop

•  We drive innovation within Apache Software Foundation projects

•  We are uniquely positioned to deliver the highest quality of Hadoop support

•  We enable the ecosystem to work better with Hadoop

Develop Distribute Support

We develop, distribute and support the ONLY 100% open source Enterprise Hadoop distribution

Endorsed by Strategic Partners

Headquarters: Palo Alto, CA Employees: 180+ and growing Investors: Benchmark, Index, Yahoo


Hortonworks

•  Who is Hortonworks •  Our approach

–  Leading Open Source Hadoop innovation –  Addressing “Enterprise Hadoop” Requirements –  Enabling Interoperability of the Ecosystem –  Ensuring No Lock-In: 100% Open Source

•  Patterns of Use


Apache Software Foundation Guiding Principles •  Release early & often •  Transparency, respect, meritocracy

Key Roles held by Hortonworkers •  VP & PMC Members

–  Arun Murthy (Hadoop), Daniel Dai (Pig), Mahadev Konar (Zookeeper)

•  Release Managers –  Matt Foley (Hadoop 1.x), Arun Murthy

(Hadoop 2.x), Ashutosh Chauhan (Hive), Daniel Dai (Pig), Alan Gates (HCatalog), Mahadev Konar (Ambari)

•  Committers –  54 across all Hadoop-related projects

Apache Hadoop

Test & Patch

Design & Develop

Release

Apache Pig

Apache HCatalog

Apache HBase

Other Apache Projects

Apache Hive

Apache Ambari

“We have noticed more activity over the last year from Hortonworks’ engineers on building out Apache Hadoop’s more innovative features. These include YARN, Ambari and HCatalog..”

- Jeff Kelly: Wikibon

Apache Community Leadership


Leadership that Starts at the Core

• Driving next generation Hadoop – YARN, MapReduce2, HDFS2, High

Availability, Disaster Recovery

•  420k+ lines authored since 2006 – More than twice nearest contributor

• Deeply integrating w/ecosystem

– Enabling new deployment platforms –  (ex. Windows & Azure, Linux & VMware HA)

– Creating deeply engineered solutions –  (ex. Teradata big data appliance)

• All Apache, NO holdbacks –  100% of code contributed to Apache


Driving Enterprise Hadoop Innovation

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

AMBARI

HBASE

HCATALOG

HIVE

PIG

HADOOP CORE

Lines Of Code By Company Source: Apache Software Foundation

Hortonworks Yahoo!

Cloudera Other

Hortonworks Committers

Cloudera Committers

19 9

5 1

1 0

5 0

3 7

14 0


Hortonworks Process for Enterprise Hadoop

Upstream Community Projects Downstream Enterprise Product

Hortonworks Data Platform

Design & Develop

Distribute

Integrate & Test

Package & Certify

Apache HCatalog

Apache Pig

Apache HBase

Other Apache Projects

Apache Hive

Apache Ambari

Apache Hadoop

Test & Patch

Design & Develop

Release

No Lock-in: Integrated, tested & certified distribution lowers risk by ensuring close alignment with Apache projects

Virtuous cycle when development & fixed issues done upstream & stable project releases flow downstream

Stable Project Releases

Fixed Issues


Hortonworks


–  Leading Open Source Hadoop Innovation –  Addressing “Enterprise Hadoop” Requirements –  Enabling Interoperability of the Ecosystem –  Ensuring NO LOCK-IN: 100% Open Source

•  Patterns of use


Enhancing the Core of Apache Hadoop

Deliver high-scale storage & processing with enterprise-ready platform services

Unique Focus Areas: •  Bigger, faster, more flexible

Continued focus on speed & scale and enabling near-real-time apps

•  Tested & certified at scale Run ~1300 system tests on large Yahoo clusters for every release

•  Enterprise-ready services High availability, disaster recovery, snapshots, security, …

HADOOP CORE

Hortonworkers are the architects, operators, and builders of core Hadoop

Distributed Storage & Processing

PLATFORM SERVICES Enterprise Readiness


HADOOP CORE

DATA SERVICES

Provide data services to store, process & access data in many ways

Unique Focus Areas: •  Apache HCatalog

Metadata services for consistent table access to Hadoop data

•  Apache Hive Explore & process Hadoop data via SQL & ODBC-compliant BI tools


Hortonworks enables Hadoop data to be accessed via existing tools & systems

Store, Process and Access Data


Data Services for Full Data Lifecycle


Operational Services for Ease of Use

OPERATIONAL SERVICES

Include complete operational services for productive operations & management

Unique Focus Area: •  Apache Ambari:

Provision, manage & monitor a cluster; complete REST APIs to integrate with existing operational tools; job & task visualizer to diagnose issues

Only Hortonworks provides a complete open source Hadoop management tool

Manage & Operate at

Scale

DATA SERVICES


HADOOP CORE Distributed Storage & Processing



OS Cloud VM Appliance

PLATFORM SERVICES

HADOOP CORE

DATA SERVICES


Manage & Operate at

Scale


Enterprise Readiness

Only Hortonworks allows you to deploy seamlessly across any deployment option

•  Linux & Windows •  Azure, Rackspace & other clouds •  Virtual platforms •  Big data appliances

HORTONWORKS DATA PLATFORM (HDP)


Deployable Across a Range of Options



HDP: Enterprise Hadoop Distribution

PLATFORM SERVICES

HADOOP CORE

DATA SERVICES


Manage & Operate at

Scale




Hortonworks Data Platform (HDP) Enterprise Hadoop

•  The ONLY 100% open source and complete distribution

•  Enterprise grade, proven and tested at scale

•  Ecosystem endorsed to ensure interoperability

Enterprise Readiness



HDP 1.2: Data Services Improvements

PLATFORM SERVICES

HADOOP CORE

Enterprise Readiness High Availability, Disaster Recovery, Snapshots, Security, etc…



DATA SERVICES

HCATALOG

HIVE PIG HBASE

OOZIE

AMBARI

HDFS YARN (in 2.0)

WEBHDFS MAP REDUCE

Hortonworks Data Platform (HDP) Enterprise Hadoop

•  The ONLY 100% open source and complete distribution

•  Enterprise grade, proven and tested at scale

•  Ecosystem endorsed to ensure interoperability

SQOOP

FLUME


Latest Hortonworks Announcements

Two releases in January 2013

Hortonworks Data Platform 1.2 Hortonworks Brings Enterprise Manageability to 100% Open Source Apache Hadoop Distribution

Hortonworks Sandbox Hortonworks accelerates Hadoop skills development with an easy-to-use, flexible and extensible platform to learn, evaluate and use Apache Hadoop

JANUARY

15

JANUARY

22


Latest Hortonworks Announcements

February 2013

Hortonworks : New Apache projects Hortonworks fuel the Open Source by releasing three new projects : KNOX / TEZ / STINGER

HDP available on Microsoft Windows To help the Hadoop adoption, Hortonworks release HDP on Microsoft Windows

February

20

February

25


Hortonworks


–  Leading Open Source Hadoop Innovation –  Addressing “Enterprise Hadoop” Requirements –  Enabling Interoperability of the Ecosystem –  Ensuring No Lock-in: 100% Open Source



Existing Data Architecture

APPLICAT

IONS

DATA

SYSTEMS

TRADITIONAL REPOS RDBMS EDW MPP

DATA

SOURC

ES

OLTP, POS SYSTEMS

OPERATIONAL TOOLS

MANAGE & MONITOR

TradiLonal Sources (RDBMS, OLTP, OLAP)

DEV & DATA TOOLS

BUILD & TEST

Business AnalyLcs

Custom ApplicaLons

Enterprise ApplicaLons


An Emerging Data Architecture

APPLICAT

IONS

DATA

SYSTEMS


DATA

SOURC

ES

MOBILE DATA

OLTP, POS SYSTEMS

OPERATIONAL TOOLS

MANAGE & MONITOR


New Sources (web logs, email, sensor data, social media)

DEV & DATA TOOLS

BUILD & TEST

Business AnalyLcs

Custom ApplicaLons


HORTONWORKS DATA PLATFORM


Interoperating With Your Tools

APPLICAT

IONS

DATA

SYSTEMS

TRADITIONAL REPOS

DEV & DATA TOOLS

OPERATIONAL TOOLS

Viewpoint

Microsoft Applications


DATA

SOURC

ES

MOBILE DATA

OLTP, POS SYSTEMS




Hortonworks


–  Leading Open Source Hadoop Innovation –  Addressing “Enterprise Hadoop” Requirements –  Enabling Interoperability of the Ecosystem –  Ensuring No Lock-In: 100% Open Source



Hortonworks

•  Who is Hortonworks •  Our approach •  Patterns of use


Operational Data Refinery

DATA

SYSTEMS

DATA

SOURC

ES

1

3 1 Capture

Capture all data

Process Parse, cleanse, apply structure & transform

Exchange Push to existing data warehouse for use with existing analytic tools

2

3

Refine Explore Enrich

2

APPLICAT

IONS

Collect data and apply a known algorithm to it in trusted operational process



Business AnalyLcs

Custom ApplicaLons





Big Data Exploration & Visualization

DATA

SYSTEMS

DATA

SOURC

ES


APPLICAT

IONS

1 Capture Capture all data


Exchange Explore and visualize with analytics tools supporting Hadoop

2

3

Collect data and perform iterative investigation for value

3

2 TRADITIONAL REPOS

RDBMS EDW MPP

1


Business AnalyLcs



Custom ApplicaLons



Application Enrichment

DATA

SYSTEMS

DATA

SOURC

ES


APPLICAT

IONS

1 Capture Capture all data


Exchange Incorporate data directly into applications

2

3

Collect data, analyze and present salient results for online apps

3

1

2 TRADITIONAL REPOS

RDBMS EDW MPP



Custom ApplicaLons


NOSQL



Key 2013 “Enterprise Hadoop” Initiatives

Invest In:

– Platform Services – DR, Snapshot, …

– Data Services –  In support of Refine,

Explore, Enrich

– Operational Services – Manageability,

Security, …

Tez / “Stinger” Interactive Query

“Gateway” Secure Access

“Continuum” Biz Continuity


PLATFORM SERVICES

HADOOP CORE

DATA SERVICES


Ambari Manage & Operate

“Herd” Data Integration

HBase Online Data


Stinger: Make Hive Best for All Needs

Interac4ve Batch

•  Parameterized Reports

•  Drilldown •  Visualiza4on •  Explora4on

•  Opera4onal batch processing

•  Enterprise Reports •  Data Mining

Data Size

5s – 1m 1m – 1h 1h+

Non-‐Interac4ve

•  Data prepara4on •  Incremental batch processing

•  Dashboards / Scorecards

Improve Latency & Throughput •  Query engine improvements •  New “Optimized RCFile” column store •  Next-gen runtime (elim’s M/R latency)

Extend Deep Analytical Ability •  Analytics functions •  Improved SQL coverage •  Continued focus on core Hive use cases


Flexible Support Subscription Programs

Leverage Hortonworks Expertise: Subscription and Support delivered and backed by Hadoop experts; subscriptions based on nodes or storage

Developer Support “How to” guidance for developers and archs

Essential Support* Operations support for small research clusters

Standard Support Operations support for dev & test clusters

12 x 5 Web only

12 x 5 Web only

All Sev: 1 business day

All Sev: 1 business day

12 x 5 Web only

Application Design Advice Code Review

Cluster Design, Install, Maintain, Performance


All Sev: 1 business day 1 seat

3 Contacts

3 Contacts

Patches & Updates

Patches & Updates

* Limited in size and no expansion

Enterprise Support Operations support for critical clusters

24 x 7 Phone &

Web

Sev 1: 1 Hour Sev 2: 4 Bus Hour


5 Contacts

Patches & Updates

Additional Options


Hortonworks: Best In Class Hadoop Support

•  Experienced enterprise support team – Experience supporting enterprise clients in production – Core engineers have real operational

experience: built and supported 44+K nodes in production – Extensive experience in commercial big data offerings

including HDP, MapR, Karmasphere

•  Global 24x7 operation – support based in Sunnyvale, UK & India

•  Stringent case management processes ensures high quality customer service & responsiveness


Transferring Our Hadoop Expertise to You

The expert source for Apache Hadoop training & certification

• World class training programs designed to

help you learn fast – Role-based hands on classes with 50% lab time

• Expert consulting services – Programs designed to transfer knowledge

•  Industry leading Hadoop Sandbox program

– Fastest way to learn Apache Hadoop – Multi-level tutorials for wide applicability – Customizable and updateable


Summary

• Leading the Innovation in Core Hadoop • Addressing the requirements for Enterprise usage • Enabling interoperability of the ecosystem • No lock-in. 100% Open Source.

• Best in industry support with flexible pricing model

• Find out more – www.hortonworks.com

– http://hortonworks.com/hadoop-training/

hortonworks presentation at big data london

Technology

core of apache hadoop

enterprise hadoop requirements

hortonworks yahoo

hortonworks process

hortonworks engineers

hortonworks establishedoperate

hortonworks snapshot

open source hadoop innovation