building distributed system with celery on docker swarm
TRANSCRIPT
![Page 1: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/1.jpg)
Building Distributed Systemwith Celery on Docker Swarm
Wei Lin
2016/06/03
![Page 2: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/2.jpg)
2
About Me
• Wei Lin• @Wei_1144• [email protected]
• Mostly worked in the fields of:– marketing– strategy-planning
– project-management
![Page 3: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/3.jpg)
3
Parallel computinghttps://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/
![Page 4: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/4.jpg)
4
Monitor and Control
![Page 5: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/5.jpg)
5
Distributed system
![Page 6: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/6.jpg)
6
Distributed system (II)
![Page 7: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/7.jpg)
7
Neural network
h1 hh2 h3
![Page 8: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/8.jpg)
8
Communicate via Celery
![Page 9: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/9.jpg)
Celery
![Page 10: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/10.jpg)
10
Celery flowchart
![Page 11: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/11.jpg)
Send message to a specific queue
![Page 12: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/12.jpg)
12
Send messagehttp://docs.celeryproject.org/en/latest/userguide/calling.html
kick.apply_async([‘neuron_x’])
kick.apply_async([‘neuron_x’], routing_key = ‘neuron_h1’)
![Page 13: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/13.jpg)
13
Configuration file
![Page 14: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/14.jpg)
14
Task Message Examplehttp://docs.celeryproject.org/en/latest/internals/protocol.html
![Page 15: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/15.jpg)
15
Routing Taskshttp://docs.celeryproject.org/en/latest/userguide/routing.html
![Page 16: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/16.jpg)
Assign an unique worker for a specific queue
![Page 17: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/17.jpg)
17
Starting the Workerhttp://docs.celeryproject.org/en/latest/userguide/workers.html
https://github.com/celery/celery/issues/3065
$ celery -A IoT worker -n neuron_h1 \-Q neuron_h1 --concurrency=1
![Page 18: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/18.jpg)
Docker
![Page 19: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/19.jpg)
19
Host and containershttps://denibertovic.com/talks/supercharge-development-env-using-docker/#/10
![Page 20: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/20.jpg)
20
Run worker container
![Page 21: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/21.jpg)
21
Deploy containers
![Page 22: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/22.jpg)
22
Celery worker containersas neurons
hh1 h2 h3
kick.apply_async( routing_key = ‘neuron_h1’)
getData.apply_async(routing_key = ‘neuron_h1’) .get()
![Page 23: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/23.jpg)
23
“Celery + Docker-Swarm”
So ?
![Page 24: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/24.jpg)
I think…
![Page 25: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/25.jpg)
25
Docker file/image/containerhttps://denibertovic.com/talks/supercharge-development-env-using-docker/#/11
![Page 26: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/26.jpg)
26
Think OO
• “From” to Inherit• Dockerfile to Encapsulate• Docker Image as Class• Docker Container as Object
![Page 27: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/27.jpg)
27
Docker Swarm & Celery
doSomething.apply_async (routing_key = ‘host_NY’)
Taipei Tokyo New York
getData.apply_async(routing_key = ‘neuron_h1’) .get()
![Page 28: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/28.jpg)
28
Monitor and Control
![Page 29: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/29.jpg)
29
OOADStation C Station B Station A
wait.apply_async ([xxx], routing_key = ‘station_B’)
wait.apply_async ([xxx], routing_key = ‘station_A’)
![Page 30: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/30.jpg)
30
Bound services
![Page 31: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/31.jpg)
31
Effects
• easier to convert a system into its distributed version.
– Decoupling, DI/IC (dependency injection / inversion of control )
– Distributed = Shared– Load-Balancing
doSomething.apply_async(routing_key = ‘host@remote’)
getData.apply_async(routing_key = ‘neuron_h1’) .get()
![Page 32: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/32.jpg)
32
Load-Balancingcontroller model database
addItem.apply_async ([order], routing_key = ‘model’)
addItem.apply_async ([‘order’], routing_key = ‘database’)
![Page 33: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/33.jpg)
33
Parallel computinghttps://resin.io/blog/what-would-you-do-with-a-120-raspberry-pi-cluster/
![Page 34: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/34.jpg)
34
Deploy parallel computing
• Deploy containers over Docker Swarm– Scale up to 480 containers
• docker-compose up –d• docker-compose scale celery=480
• Send task messages to worker containersresults = [ doSomething.apply_async([data]) for data in dataset ]
• But, needs root password-less SSH
![Page 35: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/35.jpg)
35
SETI@home
![Page 36: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/36.jpg)
36
One for all, All for onehttps://github.com/Wei1234c/OneForAll_AllForOne/blob/master/celery_projects/OneForAll_AllForOne.md
• run a Docker image to join distributed parallel computing cluster, that’s all.docker run –d wei1234c/musketeer (for example)
• Works on RPi and amd64 PC
One for all, All for one
![Page 37: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/37.jpg)
37
Summary
• Caution:– Distributed systems theory for the distributed systems engineer
• Easy to usedoSomething.apply_async ([xxx], routing_key = ‘host_NY’)
getData.apply_async(routing_key = ‘neuron_h1’) .get()
• Canvas– Chain– Group
![Page 38: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/38.jpg)
Q & A
![Page 39: Building Distributed System with Celery on Docker Swarm](https://reader030.vdocuments.mx/reader030/viewer/2022012923/58eb8db21a28abde148b4727/html5/thumbnails/39.jpg)
39
References
References :• Distributed systems theory for the distributed systems engineer• Celery user guide• Docker document• MQTT Message Type and Flows• Celery on Docker Swarm• IoT as Brain• One for all, All for one