an introduction to celery
DESCRIPTION
Given to PyWeb-IL 8 on 29th Sept 2009TRANSCRIPT
![Page 1: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/1.jpg)
Idan Gazit
PyWeb-IL 8 / 29th September 2009
CeleryA Distributed Task Queue
![Page 2: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/2.jpg)
What is Celery?
![Page 3: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/3.jpg)
DistributedAsynchronous
Task QueueFor Django
Celery is a...
![Page 4: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/4.jpg)
DistributedAsynchronous
Task QueueFor Django
Celery is a...
![Page 5: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/5.jpg)
DistributedAsynchronous
Task QueueFor Django
Celery is a...
![Page 6: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/6.jpg)
DistributedAsynchronous
Task QueueFor Django
Celery is a...
![Page 7: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/7.jpg)
DistributedAsynchronous
Task QueueFor Django
Celery is a...
since0.8
![Page 8: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/8.jpg)
What can I use it for?
http://www.flickr.com/photos/jabzg/2145312172/
![Page 9: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/9.jpg)
Potential Uses» Anything that needs to run
asynchronously, e.g. outside of the request-response cycle.
» Background computation of ‘expensive queries’ (ex. denormalized counts)
» Interactions with external API’s(ex. Twitter)
» Periodic tasks (instead of cron & scripts)
» Long-running actions with results displayed via AJAX.
![Page 10: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/10.jpg)
How does it work?
http://www.flickr.com/photos/tomypelluz/14638999/
![Page 11: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/11.jpg)
Celery Architecture
user AMQPbroker
celeryworkers
task resultstore
![Page 12: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/12.jpg)
Celery Architecture
user
submit:taskstask setsperiodic tasksretryable tasks
![Page 13: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/13.jpg)
Celery Architecture
AMQPbroker
celeryworkers
broker pushestasks to worker(s)
![Page 14: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/14.jpg)
Celery Architecture
celeryworkers
workers executetasks in parallel(multiprocessing)
![Page 15: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/15.jpg)
Celery Architecture
celeryworkers
task resultstore
task result (tombstone)is written to task store:‣RDBMS‣memcached‣Tokyo Tyrant‣MongoDB‣AMQP (new in 0.8)
![Page 16: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/16.jpg)
Celery Architecture
user task resultstore
read task result
![Page 17: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/17.jpg)
Celery Architecture
Celery
Carrot
AMQP Broker
uses...
to talk to...
![Page 18: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/18.jpg)
Celery Architecture
Celery
Carrot
AMQP Broker
pip install celery
(dependency of celery)
![Page 19: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/19.jpg)
Celery Architecture
Celery
Carrot
RabbitMQ
pip install celery
(dependency of celery)
http://www.rabbitmq.com
![Page 20: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/20.jpg)
AMQP is... Complex.
![Page 21: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/21.jpg)
AMQP is Complex
» VHost
» Exchanges
» Direct
» Fanout
» Topic
» Routing Keys
» Bindings
» Queues
» Durable
» Temporary
» Auto-Delete
![Page 23: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/23.jpg)
I Can Haz Celery?
![Page 24: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/24.jpg)
Adding Celery
1. get & install an AMQP broker(pay attention to vhosts, permissions)
2. add Celery to INSTALLED_APPS
3. add a few settings:AMQP_SERVER = "localhost"AMQP_PORT = 5672AMQP_USER = "myuser"AMQP_PASSWORD = "mypassword"AMQP_VHOST = "myvhost"
4. manage.py syncdb
![Page 25: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/25.jpg)
Celery Workers
» Run at least 1 celery worker server
» manage.py celeryd (--detatch for production)
» Can be on different machines
» Celery guarantees that tasks are only executed once
![Page 26: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/26.jpg)
Tasks
![Page 27: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/27.jpg)
Tasks
» Define tasks in your app
» app_name/tasks.py
» register & autodiscovery(like admin.py)
![Page 28: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/28.jpg)
Task
from celery.task import Taskfrom celery.registry import tasks
class FetchUserInfoTask(Task): def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = twitter.users.show(id=screen_name) logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError: logger.error("Unable to fetch {0}: {1}".format( screen_name, TwitterError)) raise return user
tasks.register(FetchUserInfoTask)
![Page 29: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/29.jpg)
Run It!
>>> from myapp.tasks import FetchUserInfoTask>>> result = FetchUserInfoTask.delay('idangazit')
![Page 30: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/30.jpg)
Task Result»result.ready()
true if task has finished
»result.resultthe return value of the task or exceptioninstance if the task failed
»result.get()blocks until the task is complete then returns result or exception
»result.successful()returns True/False of task success
![Page 31: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/31.jpg)
Why even check results?
![Page 32: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/32.jpg)
Chained Tasksfrom celery.task import Taskfrom celery.registry import tasks
class FetchUserInfoTask(Task): def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = twitter.users.show(id=screen_name) logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError: logger.error("Unable to fetch {0}: {1}".format( screen_name, TwitterError)) raise else: ProcessUserTask.delay(user) return user
tasks.register(FetchUserInfoTask)
![Page 33: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/33.jpg)
Task Retries
![Page 34: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/34.jpg)
Task Retriesfrom celery.task import Taskfrom celery.registry import tasks
class FetchUserInfoTask(Task): default_retry_delay = 5 * 60 # retry in 5 minutes max_retries = 5 def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = twitter.users.show(id=screen_name) logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError, exc: self.retry(args=[screen_name,], kwargs=**kwargs, exc) else: ProcessUserTask.delay(user) return user
tasks.register(FetchUserInfoTask)
![Page 35: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/35.jpg)
Periodic Tasks
![Page 36: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/36.jpg)
Periodic Tasks
from celery.task import PeriodicTaskfrom celery.registry import tasksfrom datetime import timedelta
class FetchMentionsTask(Task): run_every = timedelta(seconds=60) def run(self, **kwargs): logger = self.get_logger(**kwargs) mentions = twitter.statuses.mentions() for m in mentions: ProcessMentionTask.delay(m) return len(mentions) tasks.register(FetchMentionsTask)
![Page 37: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/37.jpg)
Task Sets
![Page 38: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/38.jpg)
Task Sets
>>> from myapp.tasks import FetchUserInfoTask>>> from celery.task import TaskSet>>> ts = TaskSet(FetchUserInfoTask, args=( ['ahikman'], {}, ['idangazit'], {}, ['daonb'], {}, ['dibau_naum_h'], {}))>>> ts_result = ts.run()>>> list_of_return_values = ts.join()
![Page 39: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/39.jpg)
MOAR SELRY!
![Page 40: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/40.jpg)
Celery.Views
![Page 41: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/41.jpg)
Celery.Views
» Celery ships with some django views for launching /getting the status of tasks.
» JSON views perfect for use in your AJAX (err, AJAJ) calls.
» celery.views.apply(request, task_name, *args)
» celery.views.is_task_done(request, task_id)
» celery.views.task_status(request, task_id)
![Page 42: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/42.jpg)
Routable Tasks
![Page 43: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/43.jpg)
Routable Tasks
» "I want tasks of type X to only execute on this specific server"
» Some extra settings in settings.py:CELERY_AMQP_EXCHANGE = "tasks"CELERY_AMQP_PUBLISHER_ROUTING_KEY = "task.regular"CELERY_AMQP_EXCHANGE_TYPE = "topic"CELERY_AMQP_CONSUMER_QUEUE = "foo_tasks"CELERY_AMQP_CONSUMER_ROUTING_KEY = "foo.#"
» set the task's routing key:class MyRoutableTask(Task): routing_key = 'foo.bars'
![Page 44: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/44.jpg)
like django, it's just python.
![Page 45: An Introduction to Celery](https://reader033.vdocuments.mx/reader033/viewer/2022052310/554f5fdbb4c905bb178b464d/html5/thumbnails/45.jpg)
Support#celery on freenode
http://groups.google.com/group/celery-users/
AskSol (the author) is friendly & helpful