recommending the world's knowledge
TRANSCRIPT
![Page 1: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/1.jpg)
Recommending the World’s KnowledgeApplication of Recommender Systems at Quora
Lei Yang ([email protected])
![Page 2: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/2.jpg)
Quora: Share and grow the world’s knowledge
Millions of questions & answers, millions of users, thousands of topics
![Page 3: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/3.jpg)
Data and Scale
![Page 4: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/4.jpg)
Topic Question
User
Answer
Actions
![Page 5: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/5.jpg)
Lots of data relations
![Page 6: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/6.jpg)
Complex network propagation effects
![Page 7: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/7.jpg)
Complex topics network & semantics
![Page 8: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/8.jpg)
RecommendationProblems
![Page 9: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/9.jpg)
Recommending the digest email
Recommend 10 most interesting answers in an
email to get users to re-engage on Quora
● Requirements
○ Relevance
○ Freshness
○ Quality
● Challenges
○ Many candidates
○ Personalized frequency and send times
● Personalized LTR model
○ Predict probability of click on answers
shown
![Page 10: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/10.jpg)
Recommending the digest email
● Candidates
○ User/topic follows
○ Globally popular/viral content
○ Collaborative filtering suggestions
■ Relevant writers
■ Relevant topics
■ Relevant answers
● Features
○ Text features
○ User follow graph
○ Historic data
○ ...
![Page 11: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/11.jpg)
Recommending writers for a question
Recommend experts who are best qualified
to answer a question
● Ask to Answer (A2A)
● Given a questions and a viewer, predict
how “well-suited” a candidate user is for an
A2A suggestion
● Well-suited
○ viewer send a request &
○ candidate write a good answer
![Page 12: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/12.jpg)
Recommending writers for a question
● Candidates
○ Users who have written at least an
answer on one of the topics of the
question
○ A number of filters
● Features
○ Candidate history features
○ Viewer history features
○ Candidate-viewer relevance
○ ...
![Page 13: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/13.jpg)
Recommending the home feed
Present most interesting stories for a user at
a given time
● Stories: questions + answers
● Relevance-ordered vs time-ordered = big
gains in engagement
● Challenges
○ Many candidate stories
○ Real-time ranking
○ Objective function
![Page 14: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/14.jpg)
Recommending the home feed
● Candidates
○ Different temporal windows
○ Multi-stage solution with different
“streams”
● Features
○ Quality of question/answer
○ Topics the user is interested in
or knows about
○ Users the user is following
○ What is trending/popular
○ ...
![Page 15: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/15.jpg)
Recommending topics and users
Recommend new topics and users for
the user to follow, based on
● Topics you already follow
● Users you already follow
● Interactions with
questions/answers/users
● Topic-related features
● Social graph
● …..
![Page 16: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/16.jpg)
Many more problems…
Related Questions Sessions
Trending
Topic feed
![Page 17: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/17.jpg)
What We’ve Learned
![Page 18: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/18.jpg)
Learning to recommend
● ML can often solve recommendation
problems
○ Select candidates
○ ML model to score each candidate
○ Top scored candidates are returned
● Workflow
○ Build data pipeline
○ Define training label
○ Choose your model and metric
○ Create features
![Page 19: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/19.jpg)
ML models used at Quora
● Logistic Regression
● Elastic Nets
● Gradient Boosted Decision Trees
● Random Forests
● (Deep) Neural Networks
● LambdaMART
● Matrix Factorization
● LDA
● ...
![Page 20: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/20.jpg)
Open source project -- QMF
Quora Matrix Factorization
https://github.com/quora/qmf
● Currently BPR and WALS
● Multithreaded implementation in
C++14
![Page 21: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/21.jpg)
Challenges
● Personalization
● Explicit v.s. implicit user feedback
● Explore v.s. exploit and the filter bubble
● Business logic in your system
● Trade-offs between different optimization goals
● Explain the recommendations
● ...
![Page 22: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/22.jpg)
Conclusions
![Page 23: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/23.jpg)
Conclusions
● At Quora we have not only big, but also “rich” data
● Many interesting recommendation problems and unsolved challenges
● We use ML to solve many of our recommendation problems
● Our algorithms need to understand and optimize complex aspects such
as quality, relevance, or user expertise
![Page 24: Recommending the world's knowledge](https://reader031.vdocuments.mx/reader031/viewer/2022022202/587e14921a28abbc2e8b4f25/html5/thumbnails/24.jpg)
We are hiring! www.quora.com/careers