big data munich – decision automation and big data

A U T O M AT E D D E C I S I O N M A K I N G

L A R S T R I E L O F F – B L U E Y O N D E R – I ’ M @ T R I E L O F F O N T W I T T E R

T H E F R O N T I E RW H A T N O B O D Y T E L L S Y O U A B O U T

W H O I S U S I N G B I G D ATA T O D AY

W H E R E B I G D ATA I S U S E D

Effective Use

Marketing

Finance

Everyone Else

S O U R C E : O W N , P O T E N T I A L LY B I A S E D O B S E R VAT I O N S

Potential Use

Marketing

Finance

Everyone Else

S O U R C E : O W N , P O T E N T I A L LY B I A S E D O B S E R VAT I O N S

W H E R E B I G D ATA S H O U L D B E U S E D

T H R E E A P P R O A C H E S

Faster DataMore Data Better Decisions

M O R E D ATAD I G I TA L M A R K E T I N G ’ S A P P R O A C H :

FA S T E R D ATAF I N A N C E ’ S A P P R O A C H

B U T W H AT A B O U T B E T T E R D E C I S I O N S ?

Physiological

Safety

Love/Belonging

Esteem

Self-Actualization

Collection

Storage

Analysis

Prediction

Decision

Collection

Storage

Analysis

Prediction

Decision

Physiological

Safety

Love/Belonging

Esteem

Self-Actualization

“… all others bring data.”

— W. E D W A R D S D E M I N G

H O W D ATA - D R I V E N D E C I S I O N S S H O U L D W O R K

C O M P U T E R C O L L E C T S

C O M P U T E R S T O R E S

H U M A N A N A LY Z E S

H U M A N P R E D I C T S

H U M A N D E C I D E S

H O W D ATA - D R I V E N D E C I S I O N S R E A L LY W O R K

C O M P U T E R C O L L E C T S

C O M P U T E R S T O R E S

H U M A N A N A LY Z E S

C O M M U N I C AT I O N B R E A K D O W N

H U M A N D E C I D E S

C O M M U N I C AT I O N B R E A K D O W N

Communication Breakdown, It's always the same, I'm having a nervous breakdown, Drive me insane!

— L E D Z E P P E L I N

• Drill-down analysis … misunderstood or distorted• Metrics dashboards … contradictory and confusing• Monthly reports … ignored after two iterations• In-house analyst teams … overworked and powerless

H O W D ATA - D R I V E N D E C I S I O N S R E A L LY W O R K

H T T P : / / D I L B E R T. C O M / S T R I P S / C O M I C / 2 0 0 7 - 0 5 - 1 6 /

H O W D E C I S I O N S R E A L LY S H O U L D W O R K

C O M P U T E R C O L L E C T S

C O M P U T E R S T O R E S

C O M P U T E R A N A LY Z E S

C O M P U T E R P R E D I C T S

C O M P U T E R D E C I D E S

99.9% of all business decisions can be automated

H O W D E C I S I O N S A R E B E I N G M A D E

H O W D E C I S I O N S A R E B E I N G M A D E

9 0 % N O D E C I S I O N I S M A D E

9 0 % N O D E C I S I O N I S M A D E

9 % D E C I S I O N F O L L O W S R U L E

9 % D E C I S I O N F O L L O W S R U L E

1 % H U M A N D E C I S I O N M A K I N G

1 % H U M A N D E C I S I O N M A K I N G

– R O B I N S H A R M A

“Making no decision is a decision. To do nothing. And nothing always brings you nowhere..”

B U S I N E S S R U L E S F O R B E G I N N E R S

Not doing anything is the simplest business rule in the world – and also the most popular

A D VA N C E D B U S I N E S S R U L E S

Computers are machines following rules. This means business rules are programs.

B U S I N E S S R U L E S A R E P R O G R A M S , J U S T N O T V E R Y G O O D P R O G R A M S .

• Business rules are like programs – written by non-programmers

• Business rules can be contradictory, incomplete, and complex beyond comprehension

• Business rules have no built-in feedback mechanism: “It is the rule, because it is the rule”

H U M A N D E C I S I O N M A K I N G

• • System 1: Fast, automatic, frequent, emotional, stereotypic, subconscious

• • System 2: Slow, effortful, infrequent, logical, calculating, conscious

D A N I E L K A H N E M A N N , T H I N K I N G FA S T A N D S L O W

H O W T O D E C I D E FA S TF R E Q U E N T D E C I S I O N M A K I N G M E A N S FA S T D E C I S I O N M A K I N G , M E A N S U S I N G H E U R I S T I C S O R C O G N I T I V E B I A S E S

Anchoring effect

IKEA effect

Confirmation bias

Bandwagon effect

Substitution

Availability heuristic Texas Sharpshooter Fallacy

Rhyme as reason effect

Over-justification effect

Zero-risk bias

Framing effect

Illusory correlationSunk cost fallacy

Overconfidence

Outcome bias

Inattentional Blindness

Benjamin Franklin effect

Hindsight bias

Gambler’s fallacy

Anecdotal evidenceNegativity bias

Loss aversion

Backfire effect

• Abraham Lincoln and John F. Kennedy were both presidents of the United States, elected 100 years apart. 

• Both were shot and killed by assassins who were known by three names with 15 letters, John Wilkes Booth and Lee Harvey Oswald, and neither killer would make it to trial.

• Lincoln had a secretary named Kennedy, and Kennedy had a secretary named Lincoln.

• They were both killed on a Friday while sitting next to their wives, Lincoln in the Ford Theater, Kennedy in a Lincoln made by Ford.

• Abraham Lincoln and John F. Kennedy were both presidents of the United States, elected 100 years apart. 

• Both were shot and killed by assassins who were known by three names with 15 letters, John Wilkes Booth and Lee Harvey Oswald, and neither killer would make it to trial.

• Lincoln had a secretary named Kennedy, and Kennedy had a secretary named Lincoln.

• They were both killed on a Friday while sitting next to their wives, Lincoln in the Ford Theater, Kennedy in a Lincoln made by Ford.

H O W C O M P U T E R S D E C I D E FA S TM A C H I N E L E A R N I N G O F F E R S A N A LT E R N A T I V E T O H U M A N C O G N I T I V E B I A S E S A N D C A N B E M A D E FA S T T H R O U G H B I G D A TA

K-Means Clustering

Naive BayesSupport Vector Machines Affinity Propagation

Least Angle Regression

Nearest Neighbors

Decision Trees

Markov Chain Monte Carlo

Spectral clustering

Restricted Bolzmann Machines

Logistic Regression

M A C H I N E L E A R N I N G M E A N S P R O G R A M S T H AT W R I T E P R O G R A M S

• A machine learning algorithm is a system that derives a set of rules based on a set of data

• It is based on systematic observation, double-checking and cross-validation

• There is no magic, just data – and without data there no magic either

Better decisions through predictive applications

H O W P R E D I C T I V E A P P L I C AT I O N S W O R K

C O L L E C T & S T O R E

A N A LY Z E C O R R E L AT I O N S

B U I L D D E C I S I O N M O D E L

D E C I D E & T E S T O P T I M I Z E

W H Y T E S T ?

– R A N D A L L M U N R O E

“Correlation doesn’t imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look

over there’”

S T O R Y T I M E(Not safe for vegetarians)

T H E G R O U N D B E E F D I L E M M A

T H E G R O U N D B E E F D I L E M M A

Yesterday Today Tomorrow Next Delivery Next Day

In Stock Demand

T H E G R O U N D B E E F D I L E M M A

• Order too much and you will have to throw meat away when it goes bad. You lose money and cows die in vain

• Order too little and you won’t serve all your potential customers. You lose money and customers stay hungry.

A C C U R AT E LY P R E D I C T D E M A N DC H A L L E N G E # 1

A U T O M AT E R E P L E N I S H M E N TC H A L L E N G E # 2

C O L L E C T S T O C K A N D

S A L E S

P R E D I C T D E M A N D

T R A D E O F F C O S T S

C R E AT E O R D E R S I N E R P

S Y S T E M

O P T I M I Z E & R E P E AT

Collect Data Predict Outcomes Drive Decisions

Optimize Returns

Trend Estimation Classification Event Prediction

Reduce risk and cost, increase profit and revenue

Batch and streaming data ingestion, batch and streaming decision delivery (with

realtime option)

powered byBlue Yonder Platform

Secure Micro Cloud Infrastructure

Multi-Tenant Runtime Environment

Flex Storage Auto APIs Data ServicesJob Control &ML Toolkit

Application RuntimeRelational, Flat File

and In-Memory Storage Service

RESTful APIs for easy integration and data

accessPluggable machine learning pipelines

HTML5 based Web UI and API Builder

Public data prepared for machine learning

@ B YA N A LY T I C S _ E N @ T R I E L O F F