© 2012 raphael saulus. data mining for fun & profit russ blake principle architect, runge...

Data Mining

© 2012 Raphael Saulus

…not as hard as it sounds!

Data Mining for Fun & ProfitRuss BlakePrinciple Architect, Runge Limited

Kevin ClarkeManager Software DevelopmentEB Games Australia, NZ

Darien NagleSolution SpecialistApplication PlatformMicrosoft Corporation

DBI226

Who Is Russ Blake?

Who Is Russ Blake?

or…

Who Is Russ Blake?

or… …would you

buy a PerformanceMonitor from

this dude?

Who is Russ Blake?

Manager of Performance: Windows NTInventor & Author of PerfmonWrote NT Resource Kit book “Optimizing Windows NT”

Who is Russ Blake?

Manager of Performance: Windows NTInventor & Author of PerfmonWrote NT Resource Kit book “Optimizing Windows NT”

Holds 3 US patents and one Chinese patent

Who is Russ Blake?

Manager of Performance: Windows NTInventor & Author of PerfmonWrote NT Resource Kit book “Optimizing Windows NT”

Holds 3 US patents and one Chinese patent But, what have you done for me lately?

Who is Russ Blake?

Manager of Performance: Windows NTInventor & Author of PerfmonWrote NT Resource Kit book “Optimizing Windows NT”

Holds 3 US patents and one Chinese patent But, what have you done for me lately?

Fundamental physics. 2 recent papers:The effect of particle creation on space

Explains why things fall

Who is Russ Blake?

Manager of Performance: Windows NTInventor & Author of PerfmonWrote NT Resource Kit book “Optimizing Windows NT”

Holds 3 US patents and one Chinese patent But, what have you done for me lately?

Fundamental physics. 2 recent papers:The effect of particle creation on space

Explains why things fall

The architecture of nuclear binding energy7 times more accurate than the next best model of the nucleus

Runge Ltd

Planning, Scheduling and Forecasting

Mine Planning Consultancy

Why is Russ at Runge?World’s leading mine planning software

Firm roots in applied mathematical modelling

Really smart people

Firm commitment to innovation

Focused on planning and forecasting

“Predicting is Hard…

Forecasting:

“Predicting is Hard…

…especially about the future!”

--Yogi Berra

Forecasting:

Data Mining to the rescue!

© 2011 Microsoft Corporation

The future will be like the past…

Data Mining to the rescue!

© 2011 Microsoft Corporation

The future will be like the past…

because…

Data Mining to the rescue!

© 2011 Microsoft Corporation

The future will be like the past…

because…

in the past…

Data Mining to the rescue!

© 2011 Microsoft Corporation

The future will be like the past…

because…

in the past…

the future was like the past!-- Gerald M. Weinberg, An Introduction to General Systems Thinking

Why are we here?

“Data Mining is the top technology to have a

major impact across a wide range of industries in Australia

within the next 5 years…

©2012 Gartner Group Advanced Technology Research Note

Why are we here?

“Data Mining is the top technology to have a

major impact across a wide range of industries in Australia

within the next 5 years…

…and has the greatest skills gap!”

©2012 Gartner Group Advanced Technology Research Note

What it is…

Data Mining finds patterns in data

What it is…

Data Mining finds patterns in data

Uses these patterns to make predictions

© 2012 http://www.holdemreview.com

What it is…

Data Mining finds patterns in data

Uses these patterns to make predictions

Using Machine Learning Algorithms© 2012 http://www.holdemreview.com

What it is…

Data Mining finds patterns in data

Uses these patterns to make predictions

Using Machine Learning Algorithms

Don’t worry: the hard yards are done© 2012 http://www.holdemreview.com

What it is…

Data Mining finds patterns in data

Uses these patterns to make predictions

Using Machine Learning Algorithms

Don’t worry: the hard yards are done

A lot at Microsoft Research

© 2012 http://www.holdemreview.com

How We Do It @

Market Basket Analysis

How We Do It @

Market Basket AnalysisCustomer Loyalty Program – Links purchases to the individual

How We Do It @

Market Basket AnalysisCustomer Loyalty Program – Links purchases to the individual

Customer Characteristics

How We Do It @

Market Basket AnalysisCustomer Loyalty Program – Links purchases to the individual

Customer CharacteristicsPurchasing patterns – Enables direct marketing

How We Do It @

Market Basket AnalysisCustomer Loyalty Program – Links purchases to the individual

Customer CharacteristicsPurchasing patterns – Enables direct marketing

Forecasting

How We Do It @

Market Basket AnalysisCustomer Loyalty Program – Links purchases to the individual

Customer CharacteristicsPurchasing patterns – Enables direct marketing

ForecastingYear on year sales analysis – Allows for more realistic comps sales

How We Do It @

Market Basket AnalysisCustomer Loyalty Program – Links purchases to the individual

Customer CharacteristicsPurchasing patterns – Enables direct marketing

ForecastingYear on year sales analysis – Allows for more realistic comps sales

Clickstream Analysis

How We Do It @

Market Basket AnalysisCustomer Loyalty Program – Links purchases to the individual

Customer CharacteristicsPurchasing patterns – Enables direct marketing

ForecastingYear on year sales analysis – Allows for more realistic comps sales

Clickstream AnalysisAlmost real time e-commerce sales reporting

Data Mining is Self-sufficient

Data Mining does not need a Cube!

What it’s not

SSAS≠

Cube

What it’s not

SSAS≠

Cube

© blog.viXra.org

What it’s not

SSAS≠

Cube© recultured.com

Look, Ma, No Cube!

© 2012 onlyHDwallPapers.com

Cube:

Look, Ma, No Cube!

Prettyhigh barrierto entry

© 2012 onlyHDwallPapers.com

Cube:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube

Prettyhigh barrierto entry

© 2012 onlyHDwallPapers.com

Cube:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX

Prettyhigh barrierto entry

© 2012 onlyHDwallPapers.com

Cube:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX Construct Analyses

Prettyhigh barrierto entry

© 2012 onlyHDwallPapers.com

Cube:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX Construct Analyses

Prettyhigh barrierto entry

© 2012 onlyHDwallPapers.com© 2012 Microsoft Corporation

Cube: Data Mining:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX Construct Analyses

Prettyhigh barrierto entry

Prettylowbarrierto entry

© 2012 onlyHDwallPapers.com© 2012 Microsoft Corporation

Cube: Data Mining:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX Construct Analyses

Data Mining:Build Structure

Prettyhigh barrierto entry

Prettylowbarrierto entry

© 2012 onlyHDwallPapers.com© 2012 Microsoft Corporation

Cube: Data Mining:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX Construct Analyses

Data Mining:Build Structure Configure Model

Prettyhigh barrierto entry

Prettylowbarrierto entry

© 2012 onlyHDwallPapers.com© 2012 Microsoft Corporation

Cube: Data Mining:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX Construct Analyses

Data Mining:Build Structure Configure Model Make Predictions

Prettyhigh barrierto entry

Prettylowbarrierto entry

© 2012 onlyHDwallPapers.com© 2012 Microsoft Corporation

Cube: Data Mining:

Look, Ma, No Cube!

Dimensional Modelling:Build a Cube Learn MDX Construct Analyses

…of the PAST

Data Mining:Build Structure Configure Model Make Predictions

…about the Future

Prettyhigh barrierto entry

Prettylowbarrierto entry

© 2012 onlyHDwallPapers.com© 2012 Microsoft Corporation

Cube: Data Mining:

Why no Cube?

Data mining finds patterns in data

Why no Cube?

Data mining finds patterns in dataCubes abstract much of the interesting information

Why no Cube?

Data mining finds patterns in dataCubes abstract much of the interesting information Data Mine directly on your Data Warehouse

Why no Cube?

Data mining find patterns in dataCubes abstract much of the interesting information Data Mine directly on your Data Warehouse

(or [“shudder”] on your operational database)

Why no Cube?

Data mining find patterns in dataCubes abstract much of the interesting information Data Mine directly on your Data Warehouse

(or [“shudder”] on your operational database)…but now we do have read-only mirrors!

When to Data Mine a Cube

Complex calculations determine outcome

When to Data Mine a Cube

Complex calculations determine outcome

Feed results in as new Cube data

When to Data Mine a Cube

Complex calculations determine outcome

Feed results in as new Cube data

(Caveat: Cannot feed data into original Cube)

Can it really be this easy?

Excel Data Mining Add-in

Contrasting Time Series Example

Caveat: Correlation ≠ Causation!

© 2011 xkcd.com

Caveat: Correlation ≠ Causation!

© 2011 xkcd.com

Caveat: Correlation ≠ Causation!

© 2011 xkcd.com

Caveat: Beware the Black Swan!

Caveat: Beware the Black Swan!The Black Swan

The Impact of the Highly Improbable

Nassim Nicholas Taleb

Caveat: Beware the Black Swan!The Black Swan

The Impact of the Highly Improbable

Nassim Nicholas Taleb Central Thesis:All significant eventsare unpredictable!

SQL 2008 Data Mining Videos …msdn

http://msdn.microsoft.com/en-us/library/dd776389%28v=SQL.100%29.aspx

Tutorial:

Logical Architecture

Demo

Your one-stop-shop for data mining

Data Mining Designer

But is it Respectable?Is it all just smoke and mirrors?

But is it Respectable?Is it all just smoke and mirrors?Or…

But is it Respectable?Is it all just smoke and mirrors?Or…Was Data Mining invented just to make Astrology look respectable?

How it works inside

Public Domain (Wikipedia Commons)

Decision Tree Algorithm

Correlation Tree Node

Decision Tree Algorithm

Correlation Tree Node

Cluster & Association Algorithms

Cluster & Association Algorithms

Naïve Bayes Algorithm

• Simple, fast, surprisingly accurate

Naïve Bayes Algorithm

• Simple, fast, surprisingly accurate• “Naïve”: attributes assumed to be independent of each

other

Naïve Bayes Algorithm

• Simple, fast, surprisingly accurate• “Naïve”: attributes assumed to be independent of each

other• Pervasive use throughout Data Mining

Naïve Bayes Algorithm

• Simple, fast, surprisingly accurate• “Naïve”: attributes assumed to be independent of each

other• Pervasive use throughout Data Mining

Naïve Bayes Algorithm

• Simple, fast, surprisingly accurate• “Naïve”: attributes assumed to be independent of each

other• Pervasive use throughout Data Mining• Uses Bayes Law:

P(Result | Data) =

P(Data | Result) * P(Result) / P(Data)

Naïve Bayes Algorithm

P(Girl | Trousers) = ?

Naïve Bayes Algorithm

P(Girl | Trousers) = ?

P(Trousers | Girl) = 20 / 40

Naïve Bayes Algorithm

P(Girl | Trousers) = ?

P(Trousers | Girl) = 20 / 40

P(Girl) = 40 / 100

Naïve Bayes Algorithm

P(Girl | Trousers) = ?

P(Trousers | Girl) = 20 / 40

P(Girl) = 40 / 100

P(Trousers) = 80 / 100

Naïve Bayes Algorithm

P(Girl | Trousers) = ?

P(Trousers | Girl) = 20 / 40

P(Girl) = 40 / 100

P(Trousers) = 80 / 100

P(Girl | Trousers) = P(Trousers | Girl) P(Girl) / P(Trousers)

Naïve Bayes Algorithm

P(Girl | Trousers) = ?

P(Trousers | Girl) = 20 / 40

P(Girl) = 40 / 100

P(Trousers) = 80 / 100

P(Girl | Trousers) = P(Trousers | Girl) P(Girl) / P(Trousers)

= (20 / 40) (40 / 100) / (80 / 100) = 20 / 80 = 0.25

Naïve Bayes Algorithm

Neural Network Algorithm

LocWeight2

Sex

Weight3

Age

Weight

Buy

No

Input NeuronsHidden Neurons

Output Neurons

W

W

W

W

W

W

W

W

WW

W

W

W

W

W

Neural Network Algorithm

LocWeight2

Sex

Weight3

Age

Weight

Buy

No

Input NeuronsHidden Neurons

Output Neurons

W

W

W

W

W

W

W

W

WW

W

W

W

W

W

• Multilayer Perceptron Network

Neural Network Algorithm

LocWeight2

Sex

Weight3

Age

Weight

Buy

No

Input NeuronsHidden Neurons

Output Neurons

W

W

W

W

W

W

W

W

WW

W

W

W

W

W

• Multilayer Perceptron Network akaBack-Propagated Delta Rule Network

Comparing Algorithms

Bike Buyers

Population Random: 50%

TargetedData Mining: 85%

Ideal: 100%

Lift Chart Operation

Time Series

Combines 2 algorithms

Time Series

Combines 2 algorithmsARTxp

Short-term prediction

Time Series

Combines 2 algorithmsARTxp

Short-term prediction

ARIMALong-term prediction

Auto-Regressive Time x Periods

Auto-Regressive Including Moving Averages

Handles dependencies

ARIMA

Handles shocks

ARIMA

Detect periodicity:

Fourier Transform

Detect periodicity:

⁼+

ARTxp and ARIMA Blended

Default PREDICTION_SMOOTHING = 0.5

ARTxp and ARIMA Blended

PREDICTION_SMOOTHING = 0.2

Take-Aways

Off-the-shelf toolkit

Take-Aways

Off-the-shelf toolkitNo Cube required

Take-Aways

Off-the-shelf toolkitNo Cube requiredNo code required

Take-Aways

Off-the-shelf toolkitNo Cube requiredNo code requiredGood default parameters

Take-Aways

Off-the-shelf toolkitNo Cube requiredNo code requiredGood default parameters Easily explored models

Take-Aways

Off-the-shelf toolkitNo Cube requiredNo code requiredGood default parameters Easily explored models

Change parameters, filter input, compare lift

Take-Aways

Off-the-shelf toolkitNo Cube requiredNo code requiredGood default parameters Easily explored models

Change parameters, filter input, compare liftExcel Add-In

Caveats:

Correlation ≠Causation

Caveats:

Correlation ≠Causation

Beware the Black Swan

References

Data Mining Add-inshttp://office.microsoft.com/en-us/excel-help/data-mining-add-ins-HA010342915.aspx#_Toc257717762

Analysis Services - Data Mining Videoshttp://msdn.microsoft.com/en-us/library/dd776389(v=SQL.100).aspx

SQL Server Data Mining Homehttp://www.sqlserverdatamining.com/ssdm/

 Microsoft Contoso BI Demo Dataset for Retail Industryhttp://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=868662dc-187a-4a85-b611-b7df7dc909fc

 What Every IT Manager Should Know About Business Users’ Real Needs for BIhttp://docs.media.bitpipe.com/io_25x/io_25515/item_392177/Tableau_S_MktgLtr_BI_IT.pdf

 An Introduction to Data Mining : Discovering hidden value in your data warehousehttp://www.thearling.com/text/dmwhite/dmwhite.htm

Related Content

Database and Business Intelligence Track: All Sessions

Exam 467 (new) or 460 (upgrade) to MCSE Business Intelligence

Find Me Later At the Friday 11AM Meet and Greet

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to

be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS

PRESENTATION.