building distributed system with celery on docker swarm - pycon jp 2016
TRANSCRIPT
![Page 1: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/1.jpg)
Building Distributed Systemwith Celery on Docker Swarm
Wei Lin
PyCon JP 2016
2016/09/22
![Page 2: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/2.jpg)
2
About Me
• Wei Lin@[email protected]://github.com/Wei1234c
• Mostly worked in the fields of:– marketing– strategy-planning– project-management
• Materials of this talk:https://github.com/Wei1234c/PyCon-JP-2016-talk
![Page 3: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/3.jpg)
3
Parallel computinghttps://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/
![Page 4: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/4.jpg)
4
Distributed System
![Page 5: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/5.jpg)
5
Distributed system
![Page 6: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/6.jpg)
6
Distributed system (II)
![Page 7: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/7.jpg)
7
Neural network
h1 hh2 h3
![Page 8: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/8.jpg)
8
Communicate via Celery
![Page 9: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/9.jpg)
Celery
![Page 10: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/10.jpg)
10
Celery flowchart
![Page 11: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/11.jpg)
Send message to a specific queue
![Page 12: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/12.jpg)
12
Send messagehttp://docs.celeryproject.org/en/latest/userguide/calling.html
kick.apply_async([‘neuron_x’])
kick.apply_async([‘neuron_x’], routing_key = ‘neuron_h1’)
![Page 13: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/13.jpg)
13
Configuration file
![Page 14: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/14.jpg)
14
Task Message Examplehttp://docs.celeryproject.org/en/latest/internals/protocol.html
![Page 15: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/15.jpg)
15
Routing Taskshttp://docs.celeryproject.org/en/latest/userguide/routing.html
![Page 16: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/16.jpg)
Assign an unique worker for a specific queue
![Page 17: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/17.jpg)
17
Starting the Workerhttp://docs.celeryproject.org/en/latest/userguide/workers.html
https://github.com/celery/celery/issues/3065
$ celery -A IoT worker -n neuron_h1 \-Q neuron_h1 --concurrency=1
![Page 18: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/18.jpg)
Docker
![Page 19: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/19.jpg)
19
Host and containershttps://denibertovic.com/talks/supercharge-development-env-using-docker/#/10
![Page 20: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/20.jpg)
20
Run worker container
![Page 21: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/21.jpg)
21
Deploy containers
![Page 22: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/22.jpg)
22
Celery worker containersas neurons
hh1 h2 h3
kick.apply_async( routing_key = ‘neuron_h1’)
getData.apply_async(routing_key = ‘neuron_h1’) .get()
![Page 23: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/23.jpg)
23
“Celery + Docker-Swarm”
So ?
I think…
![Page 24: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/24.jpg)
24
Docker file/image/containerhttps://denibertovic.com/talks/supercharge-development-env-using-docker/#/11
![Page 25: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/25.jpg)
25
Think OO
• “From” to Inherit• Dockerfile to Encapsulate• Docker Image as Class• Docker Container as Object
![Page 26: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/26.jpg)
26
Docker Swarm & Celery
doSomething.apply_async (routing_key = ‘host_NY’)
Tokyo Taipei New York
getData.apply_async(routing_key = ‘host_NY’) .get()
![Page 27: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/27.jpg)
27
Distributed System
![Page 28: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/28.jpg)
28
OOADStation C Station B Station A
![Page 29: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/29.jpg)
29
Bound services
![Page 30: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/30.jpg)
30
Parallel computinghttps://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/
![Page 31: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/31.jpg)
31
parallel computing⊆⊆⊆⊆⊆⊆⊆⊆ distributed computing
![Page 32: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/32.jpg)
32
How to do Parallel Computing
1. Run and deploy Docker containers across DockerSwarm cluster, with a Celery worker process running in each container.
2. When initiate worker processes, don’t assign “Queue”parameter, so ALL the workers will listen to the default queue in broker.
3. Sending bunch of messages to broker without setting routing_key parameter, the messages will be routed to the default queue.
4. In a parallel fashion, all workers will pick up messages from the default queue and do the tasks.
![Page 33: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/33.jpg)
33
Deploy parallel computing
• Deploy containers over Docker Swarm– deploy containers
• docker-compose up –d• docker-compose scale celery= n (for example)
• Send task messages to worker containersresults = [ doSomething.apply_async([data]) for data in dataset ]
• But, Docker Swarm needsroot password-less SSH
![Page 34: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/34.jpg)
34
SETI@home
![Page 35: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/35.jpg)
35
One for all, All for onehttps://github.com/Wei1234c/OneForAll_AllForOne/blob/master/celery_projects/OneForAll_AllForOne.md
• run a Docker image to join distributed parallel computing cluster, that’s all.docker run –d wei1234c/musketeer (for example)
![Page 36: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/36.jpg)
36
Summary
• Easy to use– doSomething.apply_async ([xxx], routing_key = ‘host_NY’)– getData.apply_async(routing_key = ‘host_NY’) .get()
• Adventages of a distributed system– Decoupling, DI/IC (dependency injection / inversion of control )
– Distributed = Shared– Load-Balancing
• Caution:– See ref: Distributed systems theory for the distributed systems engineer
• Take a look of Canvas in Celery– Chain– Group
![Page 37: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/37.jpg)
37
Source Code
• Celery on Docker Swarm– using “Word Count” as an example of parallel computing.– https://github.com/Wei1234c/CeleryOnDockerSwarm/tree/master
/celery_projects• IoT as Brain
– simulate an artificial neural-network of 6 neurons.– https://github.com/Wei1234c/IOTasBrain/tree/master/celery_proj
ects• One for all, all for one
– follow the SETI@home paradigm.– on each machine, running a single Docker container to join the
cluster.– https://github.com/Wei1234c/OneForAll_AllForOne/tree/master/c
elery_projects
![Page 38: Building Distributed System with Celery on Docker Swarm - PyCon JP 2016](https://reader030.vdocuments.mx/reader030/viewer/2022013004/58eb8db21a28abde148b4729/html5/thumbnails/38.jpg)
38
References
References :• Distributed systems theory for the distributed systems engineer
– http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems-engineer/
• Celery user guide– http://docs.celeryproject.org/en/latest/userguide/
• Docker document– https://docs.docker.com/
• MQTT Message Type and Flows– http://blog.maxkit.com.tw/2014/02/mqttmessage-type-and-flows.html
• Celery on Docker Swarm– https://github.com/Wei1234c/PyCon-JP-2016-
talk/blob/master/Celery%20on%20Docker%20Swarm%20-%20PyCon%20JP%202016.md
• IoT as Brain– https://github.com/Wei1234c/PyCon-JP-2016-
talk/blob/master/IoT%20as%20Brain%20-%20PyCon%20JP%202016.md