running with elephants: predictive analytics with hdinsight

Running with Elephants

Predictive Analytics with Mahout & HDInsight

Introduction

Chris Price Senior BI Consultant with Pragmatic Works

AuthorRegular SpeakerData Geek & Super Dad!

@BluewaterSQL http://bluewatersql.wordpress.com/ cprice@pragmaticworks.com

You are the demo….

SQL Brewhaushttp://sqlbrewhaus.azurewebsites.net

Create an Account…

Rate some beers…

Don’t worry your infowill only be sold to the HIGHEST bidder

Agenda

• Business Case for Recommendations• How a Recommendation Engine Works• Recommendation Implementation & Integration• Evaluating Recommendations• Challenges of Implementing Recommendations

Making the Business Case

ObjectiveIncreaseRevenue

Increase # of Orders

Increase Items per

Order

Increase Average

Item PriceUp-Sell Website

Navigational

Inefficiency

Cross-Sell

Business Case Example

Up- Sell

Increase

Unit Pric

e

Cross-Sell

Increase Unit Qty

IncreasedRevenue

Recommendation Engines

• Take observation data and use data mining/machine learning algorithms to predict outcomes

• Assumptions:• People with similar interest have common preferences• Sufficiently large number of preferences available

Recommendation Options

• Collaborative Filtering (Mahout)• User-Based• Item-Based

• Content-Based (Mahout Clustering)• Data Mining (SSAS)

• Association• Clustering

Technology

• A scalable machine learning library• Fast, Efficient & Pragmatic• Many of the algorithms can be run on Hadoop

HDInsight• Hadoop on Windows• HDInsight on Windows Azure (Seamlessly scale in the

cloud)• HortonWorks Data Platform/HDP (On-Premise Solution)

Generating Recommendations

1. Sources of Data2. Clean & Prepare Data3. Generate Recommendations• Build User/Item matrix• Calculate User Similarity• Form Neighborhoods• Generate Recommendations

Sources of Data

• Implicit• Ratings• Feedback• Demographics• Psychographics (Personality/Lifestyle/Attitude),• Ephemeral Need (Need for a moment)

• Explicit• Purchase History• Click/Browse History

• Product/Item• Taxonomy• Attributes• Descriptions

Our focus for today

Data Preparation

• Clean-Up:• Remove Outliers (Z-Score)• Remove frequent buyers (Skew)• Normalize Data (Unity-Based)

• Format Data into CSV input file:<User ID>, <Item ID>, <Rating>

How it Works?

• Build a User/Item Matrix

Item

s

Users

1 2 3 4 5 6 7 8 9 10 … n

1 1 1 1 1

2 1 1 1

3 1 1 1 1 1

4 1 1 1

… 1 1

N

Neighborhood Formation

U2

U1

U5

U3

U6

U7

U4

Neighborhood Formation

• Requires some experimentation• Similarity Metrics

• Pearson Correlation• Euclidean Distance• Spearman Correlation• Cosine• Tanimoto Coefficient• Log-Likelihood

How it Works?

• Find users similar to U5

• Use a similarity metric (kNN)

• U1 & U7 are identified as most similar to U5

Item

s

Users

1 2 3 4 5 6 7 8 9 10 … n

1 1 1 1 1 1

2 1 1 1

3 1 1 1 1 1

4 1 1 1

… 1 1

N

How it Works?

• Generate Recommendations:• Find items that have not been reviewed (I1 and I6)

• Predict rating by taking weighted sum

Item

s

Users

1 2 3 4 5 6 7 8 9 10 … n

1 1 1 1 0.5 1 1

2 1 1 1

3 1 1 1 1 1

4 1 1 1

5 1 1

6 0.7 1

Pseudo-Code Implementation

for each item i that u has no preferencefor each user v that has a preference for i

compute similarity s between u and vcalculate running average of v‘s

preference for i, weighted by s

return top ranked (weighted average) i

Restrict to Neighborhood

Mahout Implementation

• Real-Time Recommendations• Write Java Code and host in JVM Instance• Limited scalability• Requires Training Data• Integration typically handled through web services

• Batch-Based Recommendations• Uses MapReduce jobs on Hadoop• Offline, Slow, yet scalable• Out-of-the-box recommender jobs

Mahout MapReduce Implementation1 – Generate List of ItemIDs2 – Create Preference Vector3 – Count Unique Users4 – Transpose Preference Vectors5 – Row Similarity

• Compute Weights• Computer Similarities• Similarity Matrix

6 – Pre-Partial Multiply, Similarity Matrix7 – Pre-Partial Multiply, Preferences8 – Partial Multiple (Steps 6 & 7)9 – Filter Items10 – Aggregate & Recommend

Integrating Mahout

• Real-Time• Requires Java coding• Web Service• Process:• Load training data (memory pressure)• Generate recommendations

• Batch• ETL from source• Generate input file (UserID, ItemID, Rating)• Load to HDFS

• Process with Mahout/Hadoop• ETL output from HDFS/Hadoop

• 7 [1:4.5,2:4.5,3:4.5,4:4.5,5:4.5,6:4.5,7:4.5]• UserID [ItemID:Estimate Rating, ………]

Handling Recommendations

Storing Recommendations:• Hive• Data Warehouse system for Hadoop• Hive ODBC Driver

• MongoDB• Leading NOSQL database• JSON-like storage with flexible schema• C#/.Net MongoDB Driver

• HBase• Open-source distributed, column-oriented database modeled

after Google’s BigTable• Use Pig/MapReduce to process output files and load HBase

table• Java API for easy reading

• Source System (SQL Server, etc)

Evaluating the Recommendations

• How good are your recommendations?• How do you evaluate the recommendation

engine?• Two options both split data into test & training

data sets:• Average Difference• Root-Mean Square

• How it works?I1 I2 I3

Estimated Review 3.5 4.0 1.5

Actual Review 4.0 2.0 2.0

Absolute Difference 0.5 2.0 0.5

Average Difference = (0.5 + 2.0 + 0.5) / 3 = 1.0

Root-Mean-Square = √((0.52 + 2.02 + 0.52) / 3) = 1.23

Evaluating the Recommendations

DataModel model = new FileDataModel(new File(“ratings.csv”));

RecommenderEvaluator eval = new AverageAbsoluteDifferenceRecommenderEvaluator();

RecommenderBuilder bldr = new RecommenderBuilder(){@Overridepublic Recommender buildRecommender(DataModel model) throws TasteException{

//Use the Pearson Correlation to calculate similarityUserSimilarity similarity = new PearsonCorrelationSimilarity(model);//Generate neighborhoods of approx. 10 usersUserNeighborhood hood = new NearestUserNeighborhood(10, similarity,

model);return new GenericUserBasedRecommender(model, hood, similarity);

}};

//Use 70% of the data to train the model and 30% to testdouble score = eval.evaluate(bldr, model, 0.7, 1.0);

Challenges

1. Context2. Cold Start3. Data Scarsity4. Popularity Bias5. Curse of Dimensionality

Context Challenges

???January

20 degrees & Snowing…..

Other Challenges

• Cold Start• Occurs when either a new item or new user is introduced• Can be handled by:• Can substitute average item/user profile• Use another recommendation generation technique

(Content-Based)

• Data Sparsity• Too many items/user make finding intersections difficult

• Popularity Bias• Skewed towards popular items, people with “unique”

taste are left out

• Curse of Dimensionality• More items/user leads to more noise and greater error

Resources

Mahout in ActionSean Owen, Robin Anil, Ted Dunning, Ellen Friedman

Hadoop: The Definitive GuideTom White

Thank You!

@BluewaterSQL http://bluewatersql.wordpress.com/ cprice@pragmaticworks.com

QUESTIONS???