the current state of asynchronous processing with ruby
DESCRIPTION
A small overview of current technologiesTRANSCRIPT
The Current State of Asynchronous
Processing in Ruby
Mathias Meyer, Peritor GmbH
self
• Software-Developer for Peritor GmbH in Berlin
• Code-Reviews, Scaling, Performance-Reviews and -Tuning, Refactorings
• Maintainer: acts_as_solr, run_later, heaps of stuff for Capistrano/Webistrano
What?
• Asynchronous Processing
• Just a fancy word for...
“Dude, this request is getting too long!”
“Okay, let’s move stuff into the background.”
In other words: What?
• A process puts a job into some queue
• Another process picks up the job and processes it
Why?
• Requests are taking too long
• Image uploads to S3
• Full-text search updates
• Generating PDFs, reports, etc.
• Sending emails (newsletters, etc.)
Why?
• Your app requires longer running tasks
• Compute daily statistics data
• Create reports
• Data crunching
Why?
• You need to run tasks at a certain time
• Scheduled tasks like invoicing, billing
• Sending out reminder emails
Why?
• Longer tasks
• Are harder to debug and monitor
• Block user and the application
Example
Rails
Client
Upload to S3
Response
!
Example - Much Better
RailsClient
Upload to S3
Response
Worker
The simplest Thing that could possibly work
Thread.newdoAccountMailer.deliver_signup(@user)end
Done.
Not so fast...
• Problems
• Not easy on resources
• Unreliable
• Not reproducible
A little better...
run_laterdoAccountMailer.deliver_signup(@user)end
run_later
• Borrowed from Merb, available as a Rails-Plugin
• Uses worker thread and a queue
• Simple solution for simple tasks
What you really want
• Reliable messaging
• Durability
• Scheduling
• Scalable processing
• Not necessarily all at the same time
Options, options
• Messaging Queues
• Polling
• Schedulers
• Oh my!
Protocols, oh my!
• AMQP
• STOMP
• JMS
• XMPP
• RestMS
Message Queues
Message Queues
• Publish/subscribe mechanism
• Usually require more than one new component in your infrastructure
• Broker middleware
• Subscribers, message listeners
Message Queues
• ActiveMQ
• RabbitMQ
• ActiveMessaging
• Amazon SQS
• Usually more code involved
Pollers
• Polling a database for new jobs
• Simple and domain-specific
• Only one new component
• Usually requires more effort to make them less prone to errors
Pollers
• Roll Your Own w/ or w/o daemons gem
• delayed_job - adds some infrastructure
• background_job
Pollers - RYO
create_table:jobsdo|t|t.string:klazz,:methodt.string:obj_idend
Pollers - RYOclassJob<ActiveRecord::Basedefself.schedule(obj,method)create(:klazz=>obj.class.name,:obj_id=>obj.id,:method=>method.to_s)enddefrunobj=klazz.constantize.find(obj_id)obj.send(self[:method])endend
Job.schedule(@user,:notify_signup)
Pollers - RYO
task:pollerdoJobPoller.new.runend
classJobPollerdefrunloopdojob=Job.find(:first,:lock=>true)nextunlessjobjob.runjob.deleteendendend
Pollers - delayed_job
User.find(params[:id]).send_later(:notify_signup)
rakejobs:work
Pollers - delayed_job
classSignupNotifier<Struct.new(:name)defperformuser=User.find_by_name(name)user.notify_signupendend
Delayed::Job.enqueueSignupNotifier.new("david")
Message Queues, Pollers
• Usually don’t directly support scheduling
Schedulers
Schedulers
• cron, cron, cron (oh, and rake)
• rufus-scheduler
• BackgrounDRb
• Quartz with JRuby (if you’re into that sort of thing)
• Roll Your Own
rufus-schedulerscheduler=Rufus::Scheduler.start_new
scheduler.cron'022**1‐5'doJob.new.runend
scheduler.in'20m'doputs"Getaflatwhite"end
Playing with the Big Guys
• Nanite - Jack of all trades
• Uses RabbitMQ, Erlang-based messaging server, AMQP implementation
• Distributes work across a network of workers
Nanite
• A self-assembling fabric of Ruby daemons
• Uses mappers and agents
• Mappers route message requests
• Agents handle messages
Nanite
Agent
Agent
Agent
Agent
RabbitMQ
Mapper
Mapper
Nanite
classReactorincludeNanite::Actorexpose:reactdefreact(payload)"reactingtomessagewithpayload:#{payload}"endend
Nanite
Nanite.request('/reactor/react','goodactingisreacting')do|r|prend
Nanite
• Agents register themselves, constantly pinging the mappers
• Mappers remove timed-out agents
• Work distributed based on agent load
• Unfortunately: Poor documentation
Other Stuff• BackgrounDRb
• AP4R
• job_fu
• spawn
• Starling/Workling
• Daemons
• beanstalkd
Careful now! Pitfalls
Race Conditions
Race Conditions
• Workers try accessing the same data
• Workers update data that is updated heavily from other layers
Race Conditions
• Make jobs repeatable
• Reduce proneness to errors coming from the database
• Let your broker take care of that
Queue Congestion
Queue Congestion
http://www.flickr.com/photos/lasgalletas/263909727/
Queue Congestion
• Workers can’t keep up with queue
• More work coming in than being processed
Queue Congestion
• Make it easy to add more workers
• Ensure they don’t steal each other’s work or do it twice (row-level locking, lock on fields)
Stalled Workers
http://www.flickr.com/photos/freeurmind/2941677586/
Stalled Workers
• Workers stopped processing jobs
• Got stuck on an a particular task
• Choked on an error
Stalled Workers
• Monitor your workers and queues
• Build in error reporting, report exceptions like in the rest of your code
• Make jobs repeatable
• Use Nanite (self-healing)
Recommendations
• Simple jobs
• Roll Your Own
• delayed_job
Recommendations
• Distributed
• Amazon SQS w/ ActiveMessaging or custom worker
• Works best on EC2
Recommendations
• Heavy load, scalable
• Nanite/RabbitMQ
The End
• Questions?
• @roidrage
• http://www.paperplanes.de
• http://github.com/mattmatt