zeromq zookeeper and flatbuffers
TRANSCRIPT
![Page 1: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/1.jpg)
ZeroMQ, ZooKeeper & FlatBuffers
Ravi Okade @ Code Camp NYC
![Page 2: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/2.jpg)
![Page 3: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/3.jpg)
Agenda:ZeroMQ Pub-SubZooKeeper Leader ElectionFlatBuffers SerializationDownload sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing
![Page 4: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/4.jpg)
Demo Weather Application- Publishes weather info via ZeroMQ
- Publisher is a Console app. WPF and Console Subscribers.
- Starts with a simple string serialization
- Then switch to FlatBuffers for serialization
- Then add ZooKeeper:- Publishers use ZK to ensure only one publisher at a time
- Subscribers listen to ZK for any change in the publisher and re-initialize ZeroMQ
- All code runs on a single machine for the demo
- The demo is to demonstrate concepts and not to measure performance
- Code is available on Git here: TODO
![Page 5: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/5.jpg)
ZeroMQ
http://www.slideshare.net/pieterh/overview-of-zeromqhttps://www.infoq.com/author/Pieter-Hintjenshttp://zguide.zeromq.org/page:all#Why-We-Needed-ZeroMQ
![Page 6: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/6.jpg)
I am confused with all these products ending with MQ!- ZeroMQ, ActiveMQ, RabbitMQ, QPID, Kafka..
Some good references:
- https://www.predic8.com/activemq-hornetq-rabbitmq-apollo-qpid-comparison.htm
- http://bipinkunjumon.blogspot.com/2013/05/zeromq-and-activemq-comparison.html
- https://tomasz.janczuk.org/2015/09/from-kafka-to-zeromq-for-log-aggregation.html
![Page 7: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/7.jpg)
ZeroMQ
http://www.slideshare.net/pieterh/overview-of-zeromq
![Page 8: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/8.jpg)
Publisher (publishes weather data by zipcode)
![Page 9: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/9.jpg)
Subscriber (subscribes to weather data by zipcode)
![Page 10: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/10.jpg)
Features- Handles slow subscriber
- Resilient to publisher aborts
- Multiple language support (.NET, Java, Python..)
- Subscription filters
- Fast!
![Page 11: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/11.jpg)
NetMQ https://github.com/zeromq/netmq- .NET port of zeromq
![Page 12: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/12.jpg)
FlatBuffers Intro https://google.github.io/flatbuffers/https://google.github.io/flatbuffers/md__internals.html
![Page 13: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/13.jpg)
FlatBuffers- Create Schema:
namespace codecamp;table Weather{
zipCode:string;temperature:int;humidity:int;
}
root_type Weather;
- Compile: flatc -n weather.schema
![Page 14: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/14.jpg)
Compare FlatBuffers with:
http://real-logic.github.io/simple-binary-encodinghttps://capnproto.org/
![Page 15: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/15.jpg)
ZooKeeperZooKeeper is a distributed, open-source coordination service for distributed applications.
ZooKeeper is very popular for leader election implementation and is widely used in the Hadoop eco-system.
![Page 16: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/16.jpg)
What else is ZooKeeper used for?- Service discovery
- Leader election (this demo)
- Configuration repository, group membership
- Barriers
- Queue, Priority Queue
- Locks
- Two phase commit
![Page 17: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/17.jpg)
ZooKeeper Leader Election
Publishers
Leader
ZooKeeper Service
Subscribers
Co-ordinating who is the leader
Notifying when leader changes
![Page 18: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/18.jpg)
ZooKeeper leader election strategy- Like a DMV token system
- Zookeeper assigns an id to each publisher
- The process with lowest id is the leader
- If any process dies, Zookeeper sends a notification; the process with next lowest id becomes the leader
- Note - ZooKeeper is an ordered system and guarantees consistency; hence each id is assigned only once
![Page 19: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/19.jpg)
ZooKeeper is also used for:- Service Discovery
- Distributed Key-value store (database)
- Configuration store
- Many more: Barriers/Locks, Queues, 2phase Commit: https://zookeeper.apache.org/doc/trunk/recipes.html
![Page 20: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/20.jpg)
Compare ZooKeeper with:https://www.consul.io/
https://coreos.com/etcd/
Apache Curator - a framework over Zookeeper http://curator.apache.org/
Eureka (by Netflix)https://github.com/Netflix/eureka/wiki/
![Page 21: ZeroMq ZooKeeper and FlatBuffers](https://reader036.vdocuments.mx/reader036/viewer/2022081419/58f385a01a28aba6398b456d/html5/thumbnails/21.jpg)
Referenceshttps://github.com/ewhauser/zookeeperhttps://github.com/zeromq/clrzmq (.NET Client for 0MQ)https://github.com/zeromq/clrzmq4 (newer .NET Client for 0MQ)Download sample code here:https://drive.google.com/file/d/0B3URL-I_ojg7UlQ2RzNoU0tYcG8/view?usp=sharing