sudarshan gaikaiwari - lucene @ yelp
DESCRIPTION
This talk describes how the Yelp uses Lucene to provide search services. It includes * Statistics of Yelp search usage * Overview of Yelp search architecture: Yelp uses different services to provide searches for different types of data. Some are based on Lucene and some on SOLR * Deeper dive into business and review search. This is the most important search service at Yelp.TRANSCRIPT
Lucene @ Yelp
Sudarshan Gaikaiwari
Bio
1. Over a decade of experience in information retrieval2. Used IR techniques at Symantec's DLP group3. Search Engineer at Yelp
Outline
1. Overview of search services at Yelp2. Federation Motivation3. Lucy Indexing4. Lucy Searching5. Efficiently Retrieving top k hits
The services we provide
Lucy: business search
Lucy also powers phone search
Cathy: she 'talks' a lot
Listsearch: it searches lists....
Reviewsearch: it searches reviews....
DYM: did you really mean that?
Suggest: auto completion
Federation Motivation
Problem
Search is too slow
Hard Disk Seek LatencyDisk seek 10,000,000 ns
Source Software Engineering Advice from Building Large-Scale Distributed SystemsJeffery Dean
RAM read latency
Main memory reference100 ns
Problem
Index is too large fit in memory on a single machine
Geographical sharding
Geographical Sharding drawbacks
1. Cumbersome manual process to determine shard boundary2. No guarantee that a boundary can be found.
Federation
1. �Split index across multiple machines2. Shard on business id3. TF-IDF scores from different machines should be
comparable
Mapping businesses to shards
1. Assigning businesses to shards
shard = shardlist[hash(business_id) % len(shardlist)]
Problems 1. Involves re-indexing all the businesses if we want to add a new shard
Virtual Nodes
Advantages
1. Flexibility (move vbuckets from one shard to another)2. Split hot spot shards
Lucy Master Slave Architecture
Separate indexing (masters)A master for each shard of a service
Searching (slaves)A slave for every replica of a service
Lucy Indexing
Lucy Searching
Federator: Combining results across shards1. Once we distribute an index across shards we need a
component which will search all these shards and combine their results.
2. Written in Python (runs inside a python web process).3. Uses Tornado IO loop to send requests to all shards.4. The transfer protocol for the requests in JSON RPC
Lucy Server
Tokens to Business Attributes
Executing queries
1. Gather the top results for a query2. Collect attribute statitics for attributes like places, categories
Lucene
1. Efficiently executes queries over the index2. Provides how relevant the business is to the words in the
query (word score)3. Upgrading lucene to 2.9/3.1 is WIP
Successive geobounds relaxation
Successive geobounds relaxation
Federation
Efficiently Retrieving top k hits
1. When user moves through multiple pages the number of hits to be returned increases
num hits = start + count
2. So if we need to retrieve 500 hits the naive way would be to retrieve 500 hits from each shard and then sort them
Distribution of hits in shards
Probability a hit is in a shard
Binomial DistributionProbability (r of top k hits) are in a particular shard
Mean
Variance
Formula
Std Deviation
Formula
Simulation
Formula Hits selected from each shard k = 100p = 0.2
Results Missed (%)
24 0.017
32 0.0001407
44 0.00000
Simulation Graph
Results
1. ~ 50% savings over 100 hits (44 hits requested from each shard)
2. 77% savings over 1000 hits (228 hits requested from each shard)
Future work
1. In memory index2. Move towards real time search
Come Join Us!
