leveraging bloom › bloom-presentation.pdftuning bloom filters •false positive probability:...
TRANSCRIPT
![Page 1: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/1.jpg)
Leveraging bloom filters on Redis
![Page 3: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/3.jpg)
Stream processing at Scopely
![Page 4: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/4.jpg)
Stream processing at Scopely
![Page 5: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/5.jpg)
![Page 6: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/6.jpg)
![Page 7: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/7.jpg)
![Page 8: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/8.jpg)
![Page 9: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/9.jpg)
![Page 10: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/10.jpg)
![Page 11: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/11.jpg)
![Page 12: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/12.jpg)
![Page 13: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/13.jpg)
![Page 14: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/14.jpg)
![Page 15: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/15.jpg)
![Page 16: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/16.jpg)
Idempotence
![Page 17: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/17.jpg)
An operation is said to be idempotent when applying it multiple times has the same
effect.
![Page 18: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/18.jpg)
Simplest approach to idempotence
![Page 19: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/19.jpg)
Idempotence with Redis sets
![Page 20: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/20.jpg)
Idempotence with Redis sets
![Page 21: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/21.jpg)
Idempotence with Redis sets
![Page 22: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/22.jpg)
Idempotence with Redis sets
![Page 23: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/23.jpg)
Memory usage per idempotence store320 million records/day ≈ 70GB of memory
![Page 24: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/24.jpg)
Is there a better way?
![Page 25: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/25.jpg)
Is there a better way?• Space-efficient
![Page 26: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/26.jpg)
Is there a better way?• Space-efficient
• Cost-effective
![Page 27: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/27.jpg)
Is there a better way?• Space-efficient
• Cost-effective
• More performant
![Page 28: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/28.jpg)
Is there a better way?• Space-efficient
• Cost-effective
• More performant
• Awesome
![Page 29: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/29.jpg)
Enter bloom filtersProbabilistic data structure to
check for item membership
![Page 30: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/30.jpg)
Enter bloom filtersProbabilistic data structure to
check for item membership
![Page 31: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/31.jpg)
Bloom filters query
![Page 32: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/32.jpg)
Bloom filters query• Definitely not in the set
![Page 33: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/33.jpg)
Bloom filters query• Definitely not in the set
• Probably in the set
![Page 34: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/34.jpg)
Bloom filters query• Definitely not in the set
• Probably in the set
• Configurable error rate
![Page 35: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/35.jpg)
Bloom fiters space efficiencyGiven 10.000.000 UUIDs...
![Page 36: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/36.jpg)
Bloom fiters space efficiencyGiven 10.000.000 UUIDs...
• Redis set: 1GB
![Page 37: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/37.jpg)
Bloom fiters space efficiencyGiven 10.000.000 UUIDs...
• Redis set: 1GB
• Plain text: ~300 MB
![Page 38: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/38.jpg)
Bloom fiters space efficiencyGiven 10.000.000 UUIDs...
• Redis set: 1GB
• Plain text: ~300 MB
• gzip: ~150 MB
![Page 39: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/39.jpg)
Bloom fiters space efficiencyGiven 10.000.000 UUIDs...
• Redis set: 1GB
• Plain text: ~300 MB
• gzip: ~150 MB
• Bloom filter with 1e-05 error rate: ~30MB(i.e., 1 in a million)
![Page 40: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/40.jpg)
Bloom fiters space efficiencyGiven 10.000.000 UUIDs...
• Redis set: 1GB
• Plain text: ~300 MB
• gzip: ~150 MB
• Bloom filter with 1e-05 error rate: ~30MB(i.e., 1 in a million)
• Bloom filter with 1e-11 error rate: ~60MB(i.e., 1 in a million million)
![Page 41: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/41.jpg)
Memory usage comparisonSets 70GB vs Bloom Filters 7GB
![Page 42: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/42.jpg)
Latency comparison
Redis sets Bloom filters
![Page 43: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/43.jpg)
Bloom filters example
![Page 44: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/44.jpg)
![Page 45: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/45.jpg)
![Page 46: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/46.jpg)
![Page 47: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/47.jpg)
![Page 48: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/48.jpg)
![Page 49: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/49.jpg)
![Page 50: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/50.jpg)
![Page 51: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/51.jpg)
![Page 52: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/52.jpg)
![Page 53: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/53.jpg)
![Page 54: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/54.jpg)
![Page 55: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/55.jpg)
![Page 56: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/56.jpg)
![Page 57: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/57.jpg)
![Page 58: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/58.jpg)
![Page 59: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/59.jpg)
![Page 60: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/60.jpg)
False positive == dropped data
![Page 61: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/61.jpg)
Bloom filters characteristics
• Capacity
• Error rate probability
![Page 62: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/62.jpg)
Scaling bloom filters
![Page 63: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/63.jpg)
Scaling bloom filters
![Page 64: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/64.jpg)
Scaling bloom filters
![Page 65: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/65.jpg)
Scaling bloom filters
![Page 66: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/66.jpg)
Scaling bloom filters
![Page 67: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/67.jpg)
Scaling bloom filters
![Page 68: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/68.jpg)
Scaling bloom filters
![Page 69: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/69.jpg)
Scaling bloom filters
![Page 70: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/70.jpg)
Tuning bloom filtersSize depends on capacity/error
probability
![Page 71: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/71.jpg)
Tuning bloom filters
![Page 72: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/72.jpg)
Tuning bloom filters
• False positive probability:
• Depends on your use case
![Page 73: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/73.jpg)
Tuning bloom filters
• False positive probability:
• Depends on your use case
• Initial capacity:
• Can't be too generous
• Can't be too conservative
![Page 74: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/74.jpg)
First attempt: LUA scripts
![Page 75: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/75.jpg)
Second attempt: bloomd
github.com/armon/bloomd
![Page 76: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/76.jpg)
bloomd drawbacks
![Page 77: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/77.jpg)
bloomd drawbacks• Lack of High Availability
![Page 78: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/78.jpg)
bloomd drawbacks• Lack of High Availability
• No clustering support
![Page 79: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/79.jpg)
bloomd drawbacks• Lack of High Availability
• No clustering support
• Maintenance
![Page 80: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/80.jpg)
bloomd drawbacks• Lack of High Availability
• No clustering support
• Maintenance
• Rigid API
![Page 81: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/81.jpg)
bloomd drawbacks• Lack of High Availability
• No clustering support
• Maintenance
• Rigid API
• Feels like abandonware
![Page 82: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/82.jpg)
ReBloomBloom filters as a Redis module
![Page 83: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/83.jpg)
ReBloom example> BF.RESERVE your_filter 0.00001 50000000OK
> BF.ADD your_filter foo1
> BF.EXISTS your_filter foo1
> BF.EXISTS your_filter bar0
![Page 84: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/84.jpg)
ReBloom
![Page 85: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/85.jpg)
ReBloom• Clustering
![Page 86: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/86.jpg)
ReBloom• Clustering
• Redundancy/replication
![Page 87: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/87.jpg)
ReBloom• Clustering
• Redundancy/replication
• Lower cognitive overhead
![Page 88: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/88.jpg)
ReBloom• Clustering
• Redundancy/replication
• Lower cognitive overhead
• Powerful API
![Page 89: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/89.jpg)
ReBloom• Clustering
• Redundancy/replication
• Lower cognitive overhead
• Powerful API
• No maintainance
![Page 90: Leveraging bloom › bloom-presentation.pdfTuning bloom filters •False positive probability: •Depends on your use case •Initial capacity: •Can't be too generous •Can't be](https://reader036.vdocuments.mx/reader036/viewer/2022063000/5f0f38e47e708231d44317cd/html5/thumbnails/90.jpg)
Summary
• Bloom filters significantly reduce memory usage and latency • Redis modules allows your custom data structures to scale
github.com/casidiablocristian.io