Enabling Next Gen Analytics with Azure Data Lake

Microsoft Azure

Microsoft Cloud

Global Trusted Hybrid

Big Data Definition

Big data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.

– Gartner, Big Data Definition*

* Gartner, Big Data (Stamford, CT.: Gartner, 2016), URL: http://www.gartner.com/it-glossary/big-data/

Big Data as a Cornerstone of Cortana Intelligence

Action

People

Automated Systems

Apps

Web

Mobile

Bots

Intelligence

Dashboards & Visualizations

Cortana

Bot Framework

Cognitive Services

Power BI

Information Management

Event Hubs

Data Catalog

Data Factory

Machine Learning and Analytics

HDInsight (Hadoop and Spark)

Stream Analytics

Intelligence

Data Lake Analytics

Machine Learning

Big Data Stores

SQL Data Warehouse

Data Lake Store

Data Sources

Apps

Sensors and devices

Data

However, there are challenges to Big Data…

Obtaining skills and capabilities

Determining howto get value

Integrating with existing IT

investments*Gartner: Survey Analysis – Hadoop Adoption Drivers and Challenges (Stamford, CT.: Gartner, 2015)

Azure HDInsightA Cloud Spark and Hadoop service for the Enterprise

Reliable with an industry leading SLA

Enterprise-grade security and monitoring

Productive platform for developers and scientists

Cost effective cloud scale

Integration with leading ISV applications

Easy for administrators to manage

63% lower TCO than deploy your own Hadoop on-premises*

*IDC study “The Business Value and TCO Advantage of Apache Hadoop in the Cloud with Microsoft Azure HDInsight”

• One-click deploy experience for installing apps.

• Fully managed PaaS offering.

• Access to entire cluster and secure by default.

• Install apps on new or existing clusters.

• Ease of authoring and deployment.

• Certified partners only.

HDInsight Application Platform

Hybrid cloud, a reality today

74%

Enterprises believe a hybrid cloud will enable

business growth1

82%

Enterprises have a hybrid cloud strategy, up from 74

percent a year ago2

Workload requirements

Regulation

Sensitive data

Customization

Latency

Legacy support

Introduction to StreamSets for Microsoft Azure

Who is StreamSets?

Enterprise Data DNA

StreamSets Mission

Top-tier Investors Commercial Customers Across Verticals

150,000 downloads⅓ of the Fortune 100

Empower enterprises to harness their data in motion.

ProductsStreamSets Dataflow Performance Manager™ (DPM)StreamSets Data Collector™ (open source)

Strong Partner Ecosystem Open Source Success

StreamSets Solution

Desired Business Outcomes

● Developer & operator efficiency

● On-time delivery

● Data trust & governance

Data in motion middleware that ensures data trust.

StreamSets Dataflow Performance Manager (DPM)

StreamSets ProductsStreamSets

Data Collector (SDC)

Open source tooling and engine to build complex any-to-any dataflows.

Cloud Service to map, measure and master

dataflow operations.

DATAFLOW LIFECYCLE

DEVELOP OPERATE

EVOLVE (Proactive)

REMEDIATE (Reactive)

● Developers● Scientists● Architects

● Operators● Stewards● Architects

StreamSets Deployment Models

Install on Local Machine

Install on Azure VM

StreamSets Deployment Models

StreamSets and Microsoft Azure in Use in a Major Bank

The Customer

● Forbes Global 500 financial services company.

● Adopting and moving into cloud at rapid phase.

● Growing rapidly both via acquisitions and organic growth.

Key Challenges Related to Data Movement

● Number of legacy tools both customer and vendor built.

● Security policy changes very hard to manage.

● Lack of security governance due to fragmentation of tools and lack of standardization.

● Difficulty onboarding new data sources as soon as the are created (technology change).

● Data drift (unexpected changes) very hard to manage at scale.

Key Factors for the Customer to Consider Streamsets

● KPIs

● Delivery guarantees

● Multiple types of origins and destinations using a single tool.

● Works natively with Microsoft Azure as part of HDInsight or Azure Virtual Machine or deployed on premise.

● Visualization of actual data transfers.

● Define security boundaries, actors etc.

● Repeating pattern

Customer’s Business Objectives

● Short Compute and Long Storage (ADLS,Azure Blob) in turn fine-grained cost control.

● Ability to build microanalytics framework. For instance, instead of taking entire dataset, build same micro datasets and build microanalytics framework and derive results faster (faster iteration).

● Move away from traditional Data Lake to Azure Data Lake to manage cost and scale.

Use Cases for StreamSetsUse Cases

1. Data Movement from On-Premise to Azure Data Lake

2. Consolidating Migration tools into single tool

3. Building DR for HDInsight Kafka workloads.

Resources / Q & A

StreamSets Data Collector @ Azure Marketplacehttps://azure.microsoft.com/en-us/marketplace/partners/streamsets/streamsets-data-collector/

Ingest Data into Microsoft Azure Data Lake (YouTube)https://www.youtube.com/watch?v=c1dVnOK7Luw

StreamSets Communityhttps://streamsets.com/community/

StreamSets Dataflow Performance Manager Product Information https://streamsets.com/products/dpm/

Thanks!