denodo datafest 2016: modern, agile data architecture for iot/ streaming analytics

O C T O B E R 1 8 , 2 0 1 6 S A N F R A N C I S C O B A Y A R E A , C A

#DenodoDataFest

RAPID, AGILE DATA STRATEGIESFor Accelerating Analytics, Cloud, and Big Data Initiatives.

Big Data and IoTAnna Yong

Director of Product Marketing: Data-in-Motion, Hortonworks

4ZBDATA

44ZBDATA

TOMORROW

INTERNETOF

ANYTHING

The New World of Business

Pre-transaction insights

Create new sources of revenue

New customer engagement models

Gain actionable intelligence fromall of the data

Business has access to more data

Accelerate Time to Value

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

EMBRACE AN OPEN APPROACH

MASTER THE VALUE OF DATA

EVERY BUSINESS IS A DATA BUSINESS

MachineLearning

C L O U D

Edge Analytics

The Connected Data Platforms Reference Architecture

Deep HistoricalAnalysis

D ATA C E N T E R

Stream Analytics

Data in Motion

Data in Motion

Data atRest

Data atRest

EdgeData

EdgeData

Data is Transforming the Enterprise…Now!

Billions of connected devices

Consumers access everything instantly

Data has become a valuable corporate asset

Adoption is Accelerating

FinancialServices

Retail

Telco

Automotive

55% of theUS F100

75% of theUS F100

8 of the top 9 in

North America

5 of the top 10

Automakers

Adoption

Imagine this Future

All data

Link customer, product, and supply chain

Real time analytics at the edge

Data is everyone’s product

Data is your future.

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Data Drives the Connected Car

Insurance Premiums

Warranties

Recalls

Pricing Models

DesignInnovation

Autonomous Driving

Connected City Infotainment

Sensors

Scheduled Maintenance

Predictive Maintenance

RouteOptimization

INSURANCECOMPANIES

GOVERNMENTAGENCIES

INFOTAINMENTPROVIDERS

SOFTWARECOMPANIES

AUTOMAKERS

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Big data from the Internet of Things incorporates both data in motion and data at rest.

The Connected Car is just one example of IoT

DATA IN MOTION DATA AT REST

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

HDPHORTONWORKS

DATA PLATFORMPowered by Apache Hadoop

The Internet of Things

IoT

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

…….Becomes the Internet of Anything

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Constrained High-latency Localized context

Hybrid – cloud / on-premises Low-latency Global context

CoreInfrastructure

Big Data from the Internet of Things

RegionalInfrastructureSources

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

IoT Drives New Requirements for Big Data

Perishable Insights

ConnectivityPrioritization

Security

AdaptabilityExtensibility

Scalability

Provenance

Real-Time

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Hortonworks DeliversConnected Data Platforms

DATA AT RESTDATA IN MOTION

Modern Data Applications

Hortonworks DataFlow

Hortonworks Data Platform

INTERNETOF

ANYTHING

PERISHABLE INSIGHTS

HISTORICAL INSIGHTS

ACTIONABLEINTELLIGENCE

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Secure

Real-time

Streaming

Integrated

Hortonworks DataFlow for Data in MotionPowered by Apache NiFi, Kafka, and Storm

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Hortonworks Data Platform for Data at RestPowered by Open Enterprise Hadoop

Open

Interoperable

Ready

Central

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

HDPHORTONWORKS

DATA PLATFORMPowered by Apache Hadoop

HDF Enables IoT Applications

HDPHORTONWORKS

DATA PLATFORMHDPHORTONWORKS

DATA PLATFORMPowered by Apache Hadoop

Before HDF After HDF

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

From the Device into the Core for Real-Time Analysis and Insights

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

HDF IoT Use CasesConnected Car

https://community.hortonworks.com/content/kbentry/32605/running-nifi-on-raspberry-pi-best-practices.html

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Schlumberger Dataflow

Slideshare: http://www.slideshare.net/HadoopSummit/from-zero-to-data-flow-in-hours-with-apache-nifi-64032731LinkedIn: www.linkedin.com/hp/update/6171029401820479488Twitter: https://twitter.com/aldrinpiri/status/747865822228422656

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Open Energi Uses HDF for Electricity Demand Response

As a result of its investment in Hortonworks Dataflow, Open Energi is Already:

• Reducing costs thanks to 10-15% less data being transmitted across a mobile network

• Creating a full transparent trail for data provenance that Open Energi can sharewith customers

• Enabling line of business teams to contribute to building dataflow rules and processes

• Standardizing the output of data across various end point devices

Open Energi: hortonworks.com/blog/data-fuel-open-energi-virtual-power-station-hortonworks-dataflow/

The Journey to Safer Global Travel

25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Prescient’s Journey

Prescient Needed to Rapidly Ingest Security Data from Tens of Thousands of Sources and Analyze it Quickly to Protect the Safety of Global Travelers

Prescient is a risk management company with its roots in the US federal sector

As it developed commercial products for the civilian market, Prescient chose Apache NiFi to ingest data sources that indicate traveler risk down to the street level

Prescient is currently ingesting data from 49,500 unstructured sources like RSS feeds, news feeds, social media, forums and blogs, using humans to identify false positives

Initially, the geospatial analysts could produce one assessment every 3-4 days

Prescient needed a cost-effective way to store the raw data for subsequent post-collection analytics that would further strengthen their product portfolio

26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Prescient’s Journey

Provenance Metadata

Threat-Proximity Mobile Alerts

Global Threat Map

Streaming Threat Ingest

Threat Assessments

Rapid Ingest & Machine Learning Deliver Accurate, Local Traveler Warnings

HDF ingests 49,500 data sources

Rich data ingest, real-time alerts and deep historical analysis increase analyst output by 700%

HDF connects Prescient’s on-prem/cloud hybrid architecture

HDP stores 5 petabytes of data interconnected to EMC tech

Spark machine-learning algorithms promise additional efficiencies

Innovate

Renovate Safe Global Travel

PREDICTIVEANALYTICS

E T LO N B O A R D

D A T AD I S C O V E R Y

E T LO N B O A R D

A C T I V EA R C H I V E

S I N G L EV I E W

“We know that when we define a high-threat area in a given area of the world, that it is underpinned by very specific data sources. It’s data-driven, and we can point to those sources—if ever asked—and say, ‘Here’s why.’”

Mike Bishop, Chief Systems Architect, Prescient

Security Threat Archive

D A T AE N R I C H M E N T

27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Prescient Traveler’s Assessment of Santa Clara Yesterday

28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Virtualization, Big Data and IoT

30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Thank You