design tradeoffs in the redis project - antirezantirez.com/misc/zenconf-redis-antirez.pdf · design...
TRANSCRIPT
![Page 1: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/1.jpg)
Design tradeoffs in the Redis project
Salvatore Sanfilippo @antirez — Pivotal
#zenantirez
![Page 2: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/2.jpg)
Before: some history…
• Software is about humans, not just engineering
• The history of an OSS project changes its design
#zenantirez
![Page 3: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/3.jpg)
Too much time in my hands
• Creating a startup as a side project is hard.
• OSS is no exception, it is like a startup.
• You need a lot of time and focus.
• Good moments? After you graduate, or when your previous thing provided a money buffer.
#zenantirez
![Page 4: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/4.jpg)
To be alone…
• It’s hard to find a co-founder in an “pure” OSS project.
• To be alone is hard: accountability, self-motivation.
• To be alone is fun: single-minded design.
• To be alone narrows your POV: use the community.
#zenantirez
![Page 5: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/5.jpg)
Design to scale yourself• Complex software is rarely a good idea…
• Far worse if you want to maintain, often alone, a non trivial project.
• Sometimes complexity is intrinsically required: too simple is also not optimal.
• It is critical (and hard) to find a good balance.
#zenantirez
![Page 6: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/6.jpg)
Important contribs? Mostly from payed people
• Peter Noordhuis (VMware)
• Matt Stancliff (Pivotal)
• Important fixes / debugging / contribs from RedisLabs, Weibo, Pinterest, Citrusbye.
• There are exceptions, of course (example: IPv6 support, high quality support in maling list).
#zenantirez
![Page 7: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/7.jpg)
Stay motivated• Rule 0: Do things you want to see existing.
• Rule 1: Evolve by sub-projects that are interesting to you.
• Rule 2: Create an economically sustainable model.
• Rule 3: Build a successful OSS, or abandon and try again.
#zenantirez
![Page 8: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/8.jpg)
Design tradeoffsTo design means to sacrifice something
#zenantirez
![Page 9: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/9.jpg)
Copy on write based persistenceSacrifice memory
for predictability and design freedom
#zenantirez
![Page 10: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/10.jpg)
Append-only disk access
Sacrifice incremental updatesto avoid disk seeks and corruptions
#zenantirez
![Page 11: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/11.jpg)
Single thread modelSacrifice single process scalabilityfor big wins in simplicity and reliability(spoiler: you need to scale cross-process, anyway…)
#zenantirez
![Page 12: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/12.jpg)
Asynchronous replicationSacrifice safety
for two order of magnitude more performances
(but it’s not a black-or-white tradeoff: we now have async ACKs from slaves)
#zenantirez
![Page 13: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/13.jpg)
No nested data structures
Sacrifice expressivenessfor data model simplicity and simpler sharding
#zenantirez
![Page 14: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/14.jpg)
Approximated LRUSacrifice theoretical correctness
to save memory
#zenantirez
![Page 15: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/15.jpg)
We managed to improve it lately…
#zenantirez
![Page 16: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/16.jpg)
No stored proceduresSacrifice least surprise principlefor a simpler to manage system
#zenantirez
![Page 17: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/17.jpg)
Redis Cluster non trivial client role
Sacrifice separation of concernsfor better latency and simplicity
#zenantirez
![Page 18: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/18.jpg)
Redis Cluster consistency model
Sacrifice safetyfor data model freedom
(but limiting the danger when shit happens)
#zenantirez
![Page 19: Design tradeoffs in the Redis project - antirezantirez.com/misc/Zenconf-Redis-antirez.pdf · Design tradeoffs in the Redis project Salvatore Sanfilippo @antirez — Pivotal ... •](https://reader030.vdocuments.mx/reader030/viewer/2022040519/5e7752cbef096a295508fa0d/html5/thumbnails/19.jpg)
Ask me anything!Now or later…
I’m @antirez on Twitter.
#zenantirez