role of data in digital transformation

The Role of Data in Your Digital TransformationLes KleinField CTO DataPivotal

@LesKlein #PivotalForum #DigitalTransformation

Great software companies leverage Big Data

to fundamentally change the consumer experience and pioneer entirely new business models

4© 2016 Pivotal Software, Inc. All rights reserved.

$4BN

Financial Services

$26BN

Hospitality

$50BN

Transportation

$54BN

Entertainment

$30BN

Automotive

$3.2BN

Industrial Products

CLOUD NATIVE SOFTWARE IS CHANGING INDUSTRIES

Data is Fueling Software

5© Copyright 2015 Pivotal. All rights reserved.

Hundreds of thousands of “trip” events each day

400+ billion of viewing-related events per day

Five billion training data

points for Price Tip feature

Disruptors Use a LOT of Data


“We’ve found that when a host selects a price that’s

within 5% of their tip, they’re nearly 4 times

more likely to get booked”

“The importance of accuracy and efficiency […], will continue to

rise as we expand and improve products like uberPOOL and

beyond.”

“Over 75% of what people watch come

from our recommendations”

Data manifests as features in an app


(Data)Microservices

Loosely coupled services architecture, bounded by context

Cloud-NativePlatforms

Enabling continuous delivery & automated

operations

Open SourceDatabase

Innovation

Extreme scale & performance advantages,

built for the cloud

Machine Learning

Use of predictive analytics to build

smart apps

How are they accomplishing this?


These companies…

Release new features in minutes, multiple times a day

Support a micro-services architecture

Consume a wide range of data sources and protocols

Store and Analyze all their data

Update algorithms and predictive models daily

Continuously ask lots of questions of their data

Modify data pipelines and add processing steps daily

They are Data Driven, are you?


“Companies need to learn how to catch

people or things in the act of doingsomething and affect the outcome“

PAUL MARITZExecutive Chairman, Pivotal

Real-time andPersonalised Informationin Context is what Wins!


A lot of reports do not make you data

drivenA lot of

dashboards do not make you data

driven A lot of alerts do not make you data

driven


Converging Trends

Innovation

New Data New Processes New Insights

The Journey to the Data-Driven Enterprise

Data Scienceand Machine

LearningBig Data

Internet of Things &

Social Media


HDFS

Data Lake

Ingest Store Analytics

Hard to changeLabor intensive

Inefficient

Coding basedNo real-time informationBased on expensive ETL

Migrating from a Reactive, Static and Constrained Model…


HDFSData LakeExpert System /

Machine Learning

In-Memory Real-Time Data

Continuous LearningContinuous Improvement

Continuous Adapting

Data Stream Pipeline

Multiple Data SourcesReal-Time ProcessingStore Everything

To Pro-Active, Self-Improving, Machine Learning Systems


What does “Customer” mean to you?


The new normal: “an audience of one”DATA DEVICES

Media

Banks

DeliveryServices

Marketers

Government

Individuals Employers

AnalyticServices

Advertising

CatalogCo-ops

ListBrokers

Websites

InformationBrokers

CreditBureausMedia

Archives

GOVERNMENTGOVERNMENT

PHONE/TV

PHONE/TV

INTERNETINTERNETAD

AGENCYAD

AGENCY

RETAILRETAIL


Your customers are a new breed


Evolving from digital natives to data natives


“A data native is someone who expects their world to not just be digital,

but to be smart and to adjust immediately to their taste and habits.”

Beyond tech savvy …..


Digital natives program their thermostat


Data natives expect the thermostat to program itself


Do you know the full value of your data and how to leverage it?


µs ms s hour day month yr+

“Fast Data” “Big Data”

year

TraditionalSystems

Pivotal Data Science Labs

Value ofData ($)

Time

Value of data over time…


Analytics - the journey so far (BI)

Complexity

Value ofAnalytics ($)

Descriptive Analytics

DiagnosticAnalytics

Whathappened?

Why did ithappen?

Hindsight

Insight


Analytics – where you need to be today



DiagnosticAnalytics

PredictiveAnalytics

PrescriptiveAnalytics

Whathappened?

Why did ithappen?

What willhappen?

How can wemake it happen?

Hindsight

Insight

Foresight

Complexity


Interesting questions are found here



DiagnosticAnalytics

PredictiveAnalytics

PrescriptiveAnalytics

Whathappened?

Why did ithappen?

What willhappen?

How can wemake it happen?

Hindsight

Insight

Foresight

Complexity

Data science

28© 2016 Pivotal Software, Inc. All rights reserved.Data Science at Pivotal

Theory-driven approach

‘start with the problemand work towards the data’

‘start with the data and work towards the problem’

Data-driven approach

29© 2016 Pivotal Software, Inc. All rights reserved.Data Science at Pivotal

‘start with the data and work towards the problem’

Data-driven approach


User Feedback

Scoping Data Review

Model Building

Feature Review

Feature Engineering

Model Evaluation

Operationalization

Our tried and tested approach helps extract value from data

“THE MAGIC HAPPENS WHEN YOU MARRY THE

TRADITIONAL ENGINEERING APPROACH WITH THE DATA SCIENCE ENABLED BY THE DATA LAKE. IT OPENS UP A WHOLE NEW WORLD OF POSSIBLE

‘WHAT IF’ QUESTIONS.

”DAVE BARTLETT, GE AVIATION


IT’S ALL ABOUTHOW YOU

OPERATIONALISE YOUR INSIGHTS


Scale-out analytic

database

Model API Cloud Native Application

Platform

Data Sources

0 5

Smart Apps: Models Manifesting as Microservices


OSSSmart/Data-driven App Technology Needs

Fast Ingest/Pipelining

• Pipelines to consume streaming and batch data from various endpoints

Speed/Serving Layer

• In-memory data grids for real-time and aggregated data sets

• Low latency

Big Data Analytics Platform

• Scale-out Storage• Ability to process large structured and unstructured data• Advanced SQL query capabilities• Ability to run machine learning at scale

Platform for ScalableWeb/Mobile Apps

• Infrastructure Automation

• Develop, run and manage Web applications without the complexity of building and maintaining the infrastructure


Fast Ingest/Pipelining Speed/Serving Layer

Big Data Analytics Platform

Platform for ScalableWeb/Mobile Apps

Pivotal’s Answer for Technology Needs

Spring Cloud Data Flow GemFire

GreenplumDatabase

Pivotal HDB

Pivotal Cloud Foundry

Spring

OSS


● Silo’d and aging database systems

● Spaghetti data pipelines

● Expensive, proprietary data management systems

● Lack of structured platforms for continuous software delivery

● Monolithic application architectures

● Batch-oriented data integration

● Limited operationalization of analytics

● Proprietary systems

Today’s Enterprise Challenge


Data Programming Model

Cloud-Native Platform

Microservices FrameworkPlatform Runtime

Stream Data Platform

Hadoop Spark DW

Apps & Microservices

DBMS

IMDG

K/V Store

Relational DB

Big Data & Machine Learning

Modern Cloud-Native Data Architecture


1. Modernize your data platforms:

• Improve performance and scalability for mission-critical data workloads

• Lower overall TCO by adopting industry-leading open source database technologies on modern cloud platforms

2. Prove out a specific business use-case:

• Build a smart app end-to-end

• Focus on a specific analytical model / data-microservice

• Automate ingestion and data pipelines

Pivotal can help you to:


Pivotal Offerings

Big Data Suite:

• Best-in-class open source data management software

• GemFire: In-Memory Data Grid

• Greenplum: Data Warehouse

• Pivotal HDB (Apache HAWQ): Hadoop-Native SQL

Pivotal Cloud Foundry

• Industry’s Leading Cloud-Native Platform

Spring Boot, Cloud and Data Flow

• Modern-Java micro-services framework

Pivotal Labs & Data Science

• Agile software development and machine learning

Let’s build something

MEANINGFULLet’s build something

MEANINGFUL


AP

PA Real-World Example


AppDevelopment

Data analytics

Cloud-nativeApp platform

Data Science & Model building

DataMicro-service

AP

PA Real-World Example


AppDevelopment

Data analytics

Cloud-nativeApp platform

Data Science & Model building

DataMicro-service

AP

P Must support scale-out query processing

Must support scale-out query processing

Must deliver as an API Must deliver as an API

Must embrace agile development, focus on outcomes

Must embrace agile development, focus on outcomes

Must support microservices, agile dev, and connect to big data analytics

Must support microservices, agile dev, and connect to big data analytics

A Real-World Example


Major Credit Card Issuer Builds Infrastructure Determining Fraud Markers Across 30 Disparate Sources

Challenge:• Desire to identify potential fraudulent activity within Fortune

100 credit card issuer with 100K employees• Identifying markers across 30 internal data sources caused

lags in reporting• Establish Infrastructure to support Palantir, Platfora and RSA

reporting frameworks

Solution:• Provided ability to store 3 months of data at 60-70TB, interest

in moving to 3-year retention period• Build large-scale security analytics platform to scale with data

needs, analytics and reporting requirements• Support organizational desire to evolve from pure BI to

advanced, predictive analytics

Pivotal Solution includes: GPDB, Hadoop, HAWQ


Leading Swiss Bank Re-shapes Business to Better Monetize and Regulate Fraudulent Activity with Big Data

Challenge:• Needed to monetize data around privilege user access and

assure security around fraudulent activity • Reduce the cost of monitoring privileged user access for

regulatory requirements. (currently taking 45 people to do this)• Extend across the enterprise to solve business pains around

faster data as opposed to just longer looks in “Hadoop”

Solution:• Built on top of Big Data Suite to operationalize, automate and

extend across the enterprise • What took 6 hours should now take less than 1 hour with real

time decision-making• Capability of monitoring relationship manager’s activity with

analytics to ensure regulatory compliance

Pivotal Solution includes: Big Data Suite and Data Labs


Pivotal Data Innovation Across Industries

RETAIL / TRANSPORT

MEDIA / SERVICES

TELCO / ISP

FINANCIAL SERVICES

HIGH TECH / ELECTRONICS

EDUCATION

GOVERNMENT

OTHER

INSURANCE

AUTO / INDUSTRIAL

HEALTHCARE

MANUFACTURING