![Page 1: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/1.jpg)
Taming Pythons with ZooKeeper
PyCon Finland 2012
![Page 2: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/2.jpg)
@nailor
![Page 3: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/3.jpg)
Content Squad
![Page 4: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/4.jpg)
• Consume boatloads of XML
• Build database out ofit
• Transcode audio
• Enrich data
• Build indexes
• Ship indexes
Day to day operations
![Page 5: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/5.jpg)
Shipping
![Page 6: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/6.jpg)
Pre-defined order
![Page 7: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/7.jpg)
• Central orchestrator
• Assume all machines are running
• Run remote commands
• We need to know all the quirks
• Fail fast
The naïve solution
![Page 8: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/8.jpg)
Guess what? It breaks
![Page 9: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/9.jpg)
...and replacing isn't simple
![Page 10: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/10.jpg)
• No central orchestrator
• Assume some machines are always down
• Run locally
• Shift responsibility to system owners
• Fail gracefully
New tool is needed
![Page 11: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/11.jpg)
CAP Theorem
![Page 12: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/12.jpg)
The CAP Theorem
1.Consistency
2.Availability
3.Partition tolerance
![Page 13: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/13.jpg)
Pick two. Any two will do.
![Page 14: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/14.jpg)
We went for CA
![Page 15: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/15.jpg)
• Users are mostly contained in single DC
• Inside a single DC connections are quite robust
• Remember the order? We need consistency
• Availability is everything
Why?
![Page 16: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/16.jpg)
How?
![Page 17: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/17.jpg)
![Page 18: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/18.jpg)
Apache ZooKeeper
![Page 19: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/19.jpg)
• A distributed tree-like data structure
• Simple primitives
• Automatic leader elections
• Guaranteed hard consistency
• Ephemeral nodes
What?
![Page 20: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/20.jpg)
Tree-like structure
/
/dir/
/
/subdir/
/dir2/
![Page 21: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/21.jpg)
Simple primitives
● Guaranteed atomic operations● Counters● Change notifications
![Page 22: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/22.jpg)
Automatic leader election
● Nodes know who is the most up to date● If no leader can be picked, ZooKeeper refuses to work
![Page 23: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/23.jpg)
Guaranteed hard consistency
● Every change is sent to every node!● Quorum for all operations is always required
![Page 24: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/24.jpg)
Ephemeral nodes
● Node is present only if the client is alive
![Page 25: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/25.jpg)
Library: zkPython
![Page 26: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/26.jpg)
The Good:
● Thin● Comes with ZooKeeper● Maintained by the Apache ZooKeeper project
![Page 27: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/27.jpg)
The bad:
● Thin● C bindings only, no PyPy for you
![Page 28: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/28.jpg)
The ugly:
● No documentation :(
![Page 29: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/29.jpg)
There are others: Kazoo
![Page 30: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/30.jpg)
The Good:
● Pure Python● Recipes implemented● Used by many (Quora, Mozilla, reddit, Zope)
![Page 31: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/31.jpg)
The bad:
● Not much recipes done● Not owned by the mainline
![Page 32: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/32.jpg)
The ugly:
● Own implementation of the protocol
![Page 33: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/33.jpg)
Dos and Don'ts
![Page 34: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/34.jpg)
Don't ship large chunks
![Page 35: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/35.jpg)
Monitor the ZooKeeper
![Page 36: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/36.jpg)
Don't write there all the time
![Page 37: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/37.jpg)
Stay in one DC
![Page 38: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/38.jpg)
Spotify & ZooKeeper
![Page 39: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/39.jpg)
Summary time!
![Page 40: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/40.jpg)
Concurrency == hard
![Page 41: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/41.jpg)
Distributed consistency == hard
![Page 42: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/42.jpg)
No partitions? Go ZooKeeper!
![Page 43: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/43.jpg)
Pick your weapon library
![Page 44: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/44.jpg)
Remember tradeoffs
![Page 45: Taming Pythons with ZooKeeper (Pyconfi edition)](https://reader034.vdocuments.mx/reader034/viewer/2022051413/553af3714a79593d798b461b/html5/thumbnails/45.jpg)
Thank you