recsys 2015 - unifying the problem of search and recommendations at opentable
TRANSCRIPT
![Page 1: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/1.jpg)
Unifying the Problem of Search and Recommendations at OpenTable
Jeremy Schiff, Ph.D.RecSys 201509/20/2015
![Page 2: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/2.jpg)
![Page 3: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/3.jpg)
BEFORE DURING AFTER
DIN
ERS
REST
AURA
NTS
Understanding& Evolving
Attracting &Planning
OpenTable: Deliver great experiences at every step, based on who you are
Proprietary 3
![Page 4: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/4.jpg)
OpenTable in Numbers•Our network connects diners with more than 32,000 restaurants worldwide.•Our diners have spent approximately $35 billion at our partner restaurants.•OpenTable seats more than 17 million diners each month.• Every month, OpenTable diners write more than 475,000 restaurant reviews
4
![Page 5: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/5.jpg)
OpenTable Data Ecosystem
Search(Context & Intent)
Restaurant Profile(Decision Confidence)
Reservation History (Verifying the Loop)
Reviews(Verifying the Loop)
User’s LocationSearch LocationDate, TimeQuery
Reviews, Ratings (Overall, Food, Noise Level, etc)
Seating Logs
Photos, Reviews, Ratings, Menus
![Page 6: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/6.jpg)
OpenTable Data Ecosystem
Search(Context & Intent)
Restaurant Profile(Decision Confidence)
Reservation History (Verifying the Loop)
Reviews(Verifying the Loop)
User’s LocationSearch LocationDate, TimeQuery
Reviews, Ratings (Overall, Food, Noise Level, etc)
Seating Logs
Photos, Reviews, Ratings, MenusUser
Interaction Logs
![Page 7: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/7.jpg)
So what are recommendations?
![Page 8: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/8.jpg)
So what are recommendations?
![Page 9: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/9.jpg)
What’s the Goal
Minimizing Engineering Time to Improve The Metric that Matters
•Make it Easy to Measure•Make it Easy to Iterate• Reduce Iteration Cycle Times
9
![Page 10: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/10.jpg)
Pick Your Business MetricRevenue, Conversions •OpenTable • Amazon
Retention, Engagement•Netflix• Pandora• Spotify
10
![Page 11: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/11.jpg)
Importance of A/B Testing• If you don’t measure it, you can’t improve it
•Metrics Drive Behavior
• Continued Forward Progress
11
![Page 12: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/12.jpg)
The Optimization Loops
Introspect Offline Learning
OnlineLearning
Hours Days Weeks
12
![Page 13: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/13.jpg)
13
The ingredients of a spectaculardining experience…
![Page 14: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/14.jpg)
14
… and a spectacularly bad one
![Page 15: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/15.jpg)
Examples of Topics (using MF)
15
![Page 16: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/16.jpg)
Edit via the Header & Footer menu in PowerPoint
1616
![Page 17: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/17.jpg)
Lead
Tim
eLe
ad T
ime
Distance
Distance
New York
Dallas
Lead Time
95%
95%
![Page 18: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/18.jpg)
Query Logs• Effective mechanism for understanding what users are trying to do
• Reducing 0 result queries
- Anecdote: should we support zipcodes next?
![Page 19: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/19.jpg)
Search to Recommendations Continuum• Common Themes
- Ranking always tries to move key metric (like conversion)
- Always leverage implicit signals (time of day, day of week, location, etc)
- User Control vs. Paradox of ChoiceAdvantage Example Stage Item Count
Search User Control $$, French, Takes Credit Card
Retrieval Many
Browse Use Case Control Great View / Romantic
Ranking Many
Recommend Data-Driven Flexibility
Best around me Ranking Few
![Page 20: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/20.jpg)
Differences in Recommender Usage
Right now vs. Planning
Cost of Being Wrong
Search vs. Recommendations
20
![Page 21: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/21.jpg)
Search vs. RecommendationsCollaborative Filtering Models• Personalized•Without Context
Search • Leverage Context•Using CF as One of Many Inputs
![Page 22: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/22.jpg)
Search & Recommendation Stack
Query Interpretation
Retrieval
Ranking – Item & Explanation
Index Building
Context for Query & User
Model Building
Explanation Content
Visualization
CollaborativeFilters
Item / User Metadata
22
![Page 23: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/23.jpg)
Using Context, Frequency & Sentiment• Context
- Implicit: Location, Time, Mobile/Web
- Explicit: Query
• High End Restaurant for Dinner
- Low Frequency, High Sentiment
• Fast, Mediocre Sushi for Lunch
- High Frequency, Moderate Sentiment
23
![Page 24: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/24.jpg)
Offline Models with Limited Data• Minimize Confusing User Experience
• Little to No Data- Heuristics
Encoding Product Expectations• Eg: Romantic Dates are not $. Sushi is not good for
Breakfast• Limited Data
- Data-Informed Eg: Analyze what Cuisines Users Click on when they
Query for Lunch
![Page 25: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/25.jpg)
Offline Models with Significant Data• Compensate for Sparseness• As Signals Improve, Popular -> Personalized•OpenTable Example
- Context: User Location, Searched Location, Query, etc.
• Learning to Rank- E [ Revenue | Query, Position, Item, User ]- E [ Engagement | Query, Position, Item, User ]- Regression, RankSVM, LambdaMart…
![Page 26: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/26.jpg)
The Metric Gap
26
Training Test
Training Error
Generalization Error
RMSE Precision @ K
Stage
Example
![Page 27: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/27.jpg)
The Metric Gap
27
Training Error
Generalization Error
RMSE Precision @ K
Stage
Example
Generalization Gap
Training Test
![Page 28: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/28.jpg)
The Metric Gap
28
Training Error
Generalization Error
RMSE Precision @ K
Stage
Example
Learning to Rank
Training Test
![Page 29: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/29.jpg)
The Metric Gap
Training Error
Generalization Error A/B Metric
RMSE Precision @ K Conversion
Learning to Rank
Stage
Example
Offline (Hours) Online (Weeks)
![Page 30: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/30.jpg)
The Metric Gap
30
Training Error
Generalization Error A/B Metric
RMSE Precision @ K Conversion
Learning to Rank
Stage
Example
Offline (Hours) Online (Weeks)
Offline -> Online Gap
![Page 31: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/31.jpg)
Online Learning – Overview• Naïve Online Learning is A/B testing
- Try different sets of parameters, pick the winner
•Multi-Arm Bandit- Exploiting the parameter
sets that do well- Exploring parameters that
we don’t understand well yet (high variance)
![Page 32: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/32.jpg)
Online Learning – Implementation• Iteration Loop
- Add Sets of Parameters - Explore vs. Exploit
Current Parameters• Validate Online Learning with A/B testing•Note: Tradeoff in Time to Statistical Significance
![Page 33: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/33.jpg)
Example – Start with 1 arm
Parameter
Met
ric
![Page 34: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/34.jpg)
Example – Resample arm
Parameter
Met
ric
![Page 35: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/35.jpg)
Example – Determine 2nd Arm
Parameter
Met
ric
![Page 36: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/36.jpg)
Example – Select Arm
Parameter
Met
ric
![Page 37: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/37.jpg)
Example – Improve Arm’s Estimation
Parameter
Met
ric
![Page 38: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/38.jpg)
Example – Select Arm
Parameter
Met
ric
![Page 39: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/39.jpg)
Example – Learn from Arm
Parameter
Met
ric
![Page 40: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/40.jpg)
Example – Determine New Arm
Parameter
Met
ric
![Page 41: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/41.jpg)
Training DataFlow
Collaborative Filter Service
(Realtime)
Collaborative Filter HyperParameter Tuning
(Batch with Spark)
Collaborative Filter Training
(Batch with Spark)
![Page 42: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/42.jpg)
Training DataFlow
Collaborative Filter Service
(Realtime)
Collaborative Filter HyperParameter Tuning
(Batch with Spark)
Collaborative Filter Training
(Batch with Spark)
Search Service(Realtime)
Search HyperParameter Tuning
(Batch with Spark)
Search Training(Batch with Spark)
![Page 43: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/43.jpg)
Training DataFlow
Collaborative Filter Service
(Realtime)
Collaborative Filter HyperParameter Tuning
(Batch with Spark)
Collaborative Filter Training
(Batch with Spark)
Search Service(Realtime)
Search HyperParameter Tuning
(Batch with Spark)
Search Training(Batch with Spark)
User Interaction Logs(Kafka)
Frontends & Backend Services
![Page 44: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/44.jpg)
Training DataFlow
Collaborative Filter Service
(Realtime)
Collaborative Filter HyperParameter Tuning
(Batch with Spark)
Collaborative Filter Training
(Batch with Spark)
Search Service(Realtime)
Search HyperParameter Tuning
(Batch with Spark)
Search Training(Batch with Spark)
User Interaction Logs(Kafka)
Online Learning
Frontends & Backend Services
![Page 45: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/45.jpg)
Training DataFlow
Collaborative Filter Service
(Realtime)
Collaborative Filter HyperParameter Tuning
(Batch with Spark)
Collaborative Filter Training
(Batch with Spark)
Search Service(Realtime)
Search HyperParameter Tuning
(Batch with Spark)
Search Training(Batch with Spark)
User Interaction Logs(Kafka)
Online Learning
Frontends & Backend Services
A/B Validation
![Page 46: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/46.jpg)
Compelling Recommendations
46
![Page 47: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/47.jpg)
Recommendation Explanations• Amazon
• Ness
• Netflix
• Ness - Social
47
![Page 48: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/48.jpg)
Summarizing Content
• Essential for Mobile• Balance Utility With Trust?
- Summarize, but surface raw data
• Example: - Initially, read every review- Later, use average star rating
48
![Page 49: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/49.jpg)
Summarizing Restaurant Attributes
49
![Page 50: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/50.jpg)
Dish Recommendation•What to try once I have arrived?
50
![Page 51: RecSys 2015 - Unifying the Problem of Search and Recommendations at OpenTable](https://reader035.vdocuments.mx/reader035/viewer/2022062503/58ef0cfb1a28ab23368b459d/html5/thumbnails/51.jpg)
Thanks!
Jeremy Schiff, [email protected]
Other OpenTable Members @ RecSys:Sudeep Das & Pablo Delgado