ResqueBackground processing

System administration on a multi-instances setup

Experience feedback

Plugins

Nicolas BlancoNovapost

My project : bookandgolf.com

bookandgolf.comAt launch :

• more than 60 golfs to synchronise• 4 different APIs• More than 350 000 slots (starts) to synchronize every 2 hours

At first we tried to do the synchronization with cron tasks... but...

:web

:app

:dbredis-server

nginx resque-web

worker 1

worker 2

worker 3

worker 4

Currentarchitecture

Resque job class

class MyKikoololSleepJob  @queue = :medium

  def self.perform(sleep_duration)    sleep sleep_duration  endend

Resque.enqueue(MyKikoololSleepJob, 10)

Resque workers

rake environment resque:work QUEUE=medium

rails@www:~$ ps aux | grep resque

rails  8950  sh -c cd /home/rails/www/socianalytics/current && rake environment resque:work QUEUE=high,medium,low...rails  8951  resque-1.9.10: Forked 9105 at 1302095560                                             rails  9105  resque-1.9.10: Processing medium since 1302095560                        

Resque workers Unix signals

• QUIT - Wait for child to finish processing then exit

• TERM / INT - Immediately kill child then exit

• USR1 - Immediately kill child but don't exit

• USR2 - Don't start to process any new jobs

• CONT - Start to process new jobs again after a USR2

Resque-web (Sinatra webapp)

God setup - my way!

Versioned god config files by instance name, ie :

• god / app.god• god / web.god• god / db.god• ...

God init.d file found on gist.github.com :)

god.conf :GOD_CONFIG=/home/rails/www/bookgolf/current/god/db.godGOD_COMMAND="sudo -u rails /usr/local/bin/god"GOD_LOG=/home/rails/log/god.log

God config

# ResqueGod.watch do |w|  w.env = { 'RAILS_ROOT' => rails_root,              'RAILS_ENV'  => rails_env }

  w.name          = "resque-worker"  w.interval      = 30.seconds  w.start         = "cd #{rails_root} && rake environment resque:work QUEUE=high,medium,low"  w.start_grace   = 10.seconds    ...

Workers graceful restartRake task to restart workers, the "graceful" way

namespace :resque do  task :restart_workers => :environment do    pids = Array.new        Resque.workers.each do |worker|      pids << worker.to_s.split(/:/).second if worker.to_s.include?(Settings.resque.localhost_name)    end        if pids.size > 0      system("kill -QUIT #{pids.join(' ')}")    end        system("rm /home/rails/.god/pids/resque-worker*.pid")  endend

Workers graceful restart

namespace :resque do  task :restart_workers, :roles => [:app, :db] do    rake "resque:restart_workers"  endend

Resque-web - Final setup

• Launched externally with nginx reverse proxy (easy to add http basic auth)

• External Ruby config file to change Redis setup, load plugins, etc.

     resque-web config/resque-web.rb

Resque-web God monitoring%w{5678}.each do |port|  God.watch do |w|

    w.env = { 'RAILS_ROOT' => rails_root,              'RAILS_ENV'  => rails_env }

    w.uid       = "rails"    w.name      = "resque-web"    w.interval  = 30.seconds    w.start     = "cd #{rails_root} && resque-web config/resque-web.rb"

    w.start_grace   = 15.seconds

    w.start_if do |start|      start.condition(:process_running) do |c|          c.interval = 5.seconds          c.running = false      end    end  endend

ActiveRecord - timeout / stale :(

Fast workaround...

   ...

   ActiveRecord::Base.connection.reconnect!

   ...

Resque plugins

• defunkt / resque-lock

Prevent two workers from working on the same Job class with same arguments

Resque plugins

• jayniz / resque-loner

TIP :

Do not forget to :  require "resque-loner"in your Resque-web config.rb file! or... FAIL (cleared jobs won't be relaunched again!)

BAD :

Need inheritance in job classes :'(... But a fork exists that doesn't need it.

Other plugins...

Look @ github.com/defunkt/resque Wiki!

And remember that with a goodsystem architecture, Resque is just...