nice docs finish first - designing search ranking for fairness at etsy: presented by fiona condon,...
TRANSCRIPT
![Page 1: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/1.jpg)
O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
![Page 2: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/2.jpg)
Nice Docs Finish First: Designing Search Ranking for Fairness at Etsy
Fiona Condon Senior Software Engineer, Etsy
![Page 3: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/3.jpg)
3
02
![Page 4: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/4.jpg)
4
02
![Page 5: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/5.jpg)
5
02
![Page 6: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/6.jpg)
6
02
![Page 7: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/7.jpg)
7
02Etsy by the Numbers
• 1.5 million active sellers • 21.7 million active buyers • 32 million listings for sale • Solr for listing search
![Page 8: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/8.jpg)
8
02Etsy by the Numbers
![Page 9: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/9.jpg)
9
02Search at Etsy
![Page 10: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/10.jpg)
10
02Search at Etsy
![Page 11: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/11.jpg)
11
01Search at Etsy
![Page 12: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/12.jpg)
12
01Search at Etsy
![Page 13: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/13.jpg)
13
01Our challenge
To return the best listings
![Page 14: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/14.jpg)
14
01Our challenge
To return the most informative, honest, high-quality listings
![Page 15: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/15.jpg)
15
01Our challenge
To return a diverse, fresh mix of the most informative, honest, high-quality listings
![Page 16: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/16.jpg)
16
02
![Page 17: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/17.jpg)
17
01Etsy Values
We are a mindful, transparent, and
humane business.
We plan and build for the long term.
We value craftsmanship in
all we make.
We believe fun should be part of everything we do.
We keep it real, always.
![Page 18: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/18.jpg)
18
01
Don’t hate the player, change the game
We keep it real, always.
![Page 19: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/19.jpg)
19
01Don’t hate the player, change the game
![Page 20: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/20.jpg)
20
01Don’t hate the player, change the game
![Page 21: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/21.jpg)
21
01Don’t hate the player, change the game
The count of the term in the document
The inverse of the count of the term across all documents
![Page 22: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/22.jpg)
22
01Don’t hate the player, change the game
![Page 23: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/23.jpg)
23
01Don’t hate the player, change the game
![Page 24: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/24.jpg)
24
01Don’t hate the player, change the game
![Page 25: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/25.jpg)
25
01Don’t hate the player, change the game
![Page 26: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/26.jpg)
26
01Don’t hate the player, change the game
@Overridepublic float tf(float freq) {
return (freq > 0) ? 1.0f : 0.0f; }
![Page 27: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/27.jpg)
27
01Don’t hate the player, change the game
• Set TF to 1
![Page 28: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/28.jpg)
28
01Don’t hate the player, change the game
• Set TF to 1 • Choose the right fields to index
![Page 29: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/29.jpg)
29
01Don’t hate the player, change the game
![Page 30: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/30.jpg)
30
01Don’t hate the player, change the game
![Page 31: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/31.jpg)
31
01Don’t hate the player, change the game
• Set TF to 1 • Choose the right fields to index
![Page 32: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/32.jpg)
32
01
Crafting a quality signal
We value craftsmanship in all we make
![Page 33: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/33.jpg)
33
01Crafting a quality signal
![Page 34: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/34.jpg)
34
01Crafting a quality signal
![Page 35: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/35.jpg)
35
01Crafting a quality signal
![Page 36: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/36.jpg)
36
01Crafting a quality signal
![Page 37: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/37.jpg)
37
01Crafting a quality signal
![Page 38: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/38.jpg)
38
01Crafting a quality signal
• Avoid presentation bias
![Page 39: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/39.jpg)
39
01Crafting a quality signal
![Page 40: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/40.jpg)
40
01Crafting a quality signal
![Page 41: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/41.jpg)
41
01Crafting a quality signal
• Avoid presentation bias • Store outside the index
![Page 42: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/42.jpg)
42
01Crafting a quality signal
<fieldType name="listing_quality_file" keyField=“listing_id” defVal="0.5" stored="true" indexed="true" class="solr.ExternalFileField" valType=“float" />
![Page 43: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/43.jpg)
43
01Crafting a quality signal
• Avoid presentation bias • Store outside the index • Bootstrap
![Page 44: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/44.jpg)
44
01Crafting a quality signal
![Page 45: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/45.jpg)
45
01
Freshness is fun
We believe fun should be part of everything we do.
![Page 46: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/46.jpg)
46
01Freshness is fun
![Page 47: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/47.jpg)
47
01Freshness is fun
• Diversify by seller
![Page 48: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/48.jpg)
48
01Freshness is funpublic SearchResults diversify(SearchResults results) { SearchResults diversifiedResults; int nextWindow = diversityOptions.getWindow(); do { diversityOptions.window = nextWindow; diversifiedResults = shopDiversifier.diversify(results); DiversityStats shopDiversity = diversifiedResults.docs.stream() .collect(DiversityStatsCalculator.collector(ListingDoc::getShopId)).getStats(); // if the results are sufficiently diverse, we're done if (shopDiversity.getDiversityIndex() <= diversityOptions.progressive.getTargetDiversityIndex()) { break; } // otherwise, broaden the window and re-‐try nextWindow = Math.min( diversifiedResults.totalCount, Math.min(diversityOptions.getMaxWindow(), diversityOptions.window * 2) ); } while (diversityOptions.window < nextWindow); return diversifiedResults; }
![Page 49: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/49.jpg)
49
01Freshness is fun
. . .
// if the list is sufficiently diverse, we're done if (shopDiversity.getDiversityIndex() <= diversityOptions.progressive.getTargetDiversityIndex()) { break; } // otherwise, broaden the window and re-‐try nextWindow = Math.min( diversifiedResults.totalCount, Math.min(diversityOptions.getMaxWindow(), diversityOptions.window * 2) );
. . .
![Page 50: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/50.jpg)
50
01Freshness is fun
![Page 51: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/51.jpg)
51
01Freshness is fun
![Page 52: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/52.jpg)
52
01Freshness is fun
• Diversify by seller • Recency boost
![Page 53: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/53.jpg)
53
01Freshness is fun
The inverse of the time elapsed between now and listing creation
![Page 54: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/54.jpg)
54
01Freshness is fun
![Page 55: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/55.jpg)
55
01
Evaluating for stability
We plan and build for the long term.
![Page 56: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/56.jpg)
56
01Evaluating for stability
• Replayer • CL tool • Uses sampled request logs to “replay” real traffic • Accepts target hosts, duration, query rate • Programmatically filters or alters requests • Provides realistic stats on average/worst-case impact
![Page 57: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/57.jpg)
57
01Evaluating for stability
![Page 58: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/58.jpg)
58
01Evaluating for stability
• RankDelta • Web UI • Allows user to specify query set, hosts and thrift params
in PHP code • Provides high-level statistics about the results • Plus full result set deep-dive
![Page 59: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/59.jpg)
59
01Evaluating for stability
![Page 60: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/60.jpg)
60
01Evaluating for stability
• In an ideal world…
![Page 61: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/61.jpg)
61
01Evaluating for stability
• In an ideal world… • Making trade-offs
![Page 62: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/62.jpg)
62
01
Communicating clearly
We are a mindful, transparent, and
humane business.
![Page 63: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/63.jpg)
63
01Communicating clearly
![Page 64: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/64.jpg)
64
01Communicating clearly
• Focus on the constants
![Page 65: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/65.jpg)
65
01Communicating clearly
• Focus on the constants • Provide a feedback loop
![Page 66: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/66.jpg)
66
01Communicating clearly
![Page 67: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/67.jpg)
67
01Takeaways
• Minor changes to the default scoring can be powerful • Handle quality contextually • Conscious diversity serves both searcher & searchee • Invest in a feedback loop on ranking changes • Be honest but keep it consistent
![Page 68: Nice Docs Finish First - Designing Search Ranking for Fairness at Etsy: Presented by Fiona Condon, Etsy](https://reader034.vdocuments.mx/reader034/viewer/2022051709/587067b41a28ab48378b54a3/html5/thumbnails/68.jpg)
68
02