starling + workling: simple distributed background jobs ...assets.en.oreilly.com/1/event/13/starling...

Starling + Workling: simple distributed background jobs with Twitter’s queuing system

PLAY/TYPE

morning, hi, my name is rany keddo, i run a little startup in frankfurt called play/type.

• sudo gem install gitjour

• gitjour list

• gitjour clone cows_not_kittens

git clone \ git://github.com/purzelrakete/cows-not-kittens.git

OR...

PLAY/TYPE

you might want to start by grabbing the demo project. i’m serving it.

WTF?

PLAY/TYPE

this talk is about running code asynchronously in your rails application. this means removing long running or side effect code from your request cycle.

for some reason cows started to creep into the slides while i was working on them. hoping to start a trend *away* from cats in tech presentations... you’ll see this reflected in the example project. make sure it’s working by rake db:migrate && starting irb in the project, then you type

>> CowSubsystem.moo

PLAY/TYPE

Cows not Kittens

PLAY/TYPE

this is an example app to demonstrate why you need background work, and how you can do this. also, you can milk cows with this application.

1 class CowsController < ApplicationController 2 resource_this 3 4 # milking has the side effect of causing 5 # the cow to moo. we don't want to 6 # wait for this while milking, though, 7 # it would be a terrible waste ouf our time. 8 def milk 9 @cow = Cow.find(params[:id]) 10 @cow.milk 11 end 12 end

PLAY/TYPE

1 class Cow < ActiveRecord::Base 2 3 # TODO: SAP integration 4 def milk 5 moo 6 end 7 8 # Bothersome side-effect 9 def moo 10 CowSubsystem.moo 11 end 12 end

PLAY/TYPE

Milk it

PLAY/TYPE

show the application

Real Examples

PLAY/TYPE

lets look at some real examples!

1 AnalyticsHit.create \ 2 :potential_user_id => potential_user_id, 3 :event => "converted" 4

PLAY/TYPE

thinking: doesnt really belong in the request cycle: statistics.

1 class PageView < ActiveRecord::Base 2 belongs_to :viewer 3 belongs_to :viewable, :polymorphic => true 4 end

PLAY/TYPE

nor does this: stuff that does not have to have an immediate effect on the page you’re rendering.

1 CommentMailer.deliver_created(comment) 2

PLAY/TYPE

or this, thinking: this should be put in the background, really.

1 Blackbook.get \ 2 :username => "moo@cheapmail.com", 3 :password => "milky"

PLAY/TYPE

and especially this sort of long running process - scraping contacts from webmailer.

Wherefore art thou, Rails?

PLAY/TYPE

no active* way of handling this... something consistent that works for almost everybody. instead: too many options. that’s why people come to this sort of talk.

You are a snowflake again.

which solution will you tie yourself to?decide now, because doing background stuff last is like deciding to write your tests *after* the code is done.

Trust nobody!

PLAY/TYPE

my solution to this: remain independent of all these background technologies, by building a little worker framework with providers, like active record.

WorklingPLAY/TYPE

wrote workling. aims of workling are...

Workling

• Easy plugging of new Job Runners

PLAY/TYPE

wrote workling. aims of workling are...

Workling

• Easy plugging of new Job Runners

• Nice Rails integration

PLAY/TYPE

wrote workling. aims of workling are...

Workling

• Easy plugging of new Job Runners

• Nice Rails integration

• Plays nicely with tests

PLAY/TYPE

wrote workling. aims of workling are...

Workling

• Easy plugging of new Job Runners

• Nice Rails integration

• Plays nicely with tests

• Lightweight and hackable

PLAY/TYPE

wrote workling. aims of workling are...

1 script/plugin install \ 2 git://github.com/purzelrakete/workling.git 3 4 script/plugin install \ 5 git://github.com/tra/spawn.git

PLAY/TYPE

workling will automatically use spawn if it is installed.

PLAY/TYPE

create a worker class in app/workers

1 # 2 # handle asynchronous mooing. 3 # 4 class CowWorker < Workling::Base 5 6 # let the moo-ings begin! 7 def moo(options = {}) 8 cow = Cow.find(options[:id]) 9 cow.moo 10 end 11 end

PLAY/TYPE

subclass workling:base, add a method. you need to have an options argument.

1 class Cow < ActiveRecord::Base 2 3 # TODO: SAP integration 4 def milk 5 CowWorker.async_moo(:id => id) 6 end 7 8 # bothersome side-effect 9 def moo 10 CowSubsystem.moo 11 end 12 end

PLAY/TYPE

now make the asynch call in your milk method.

Milk it!

PLAY/TYPE

What’s Spawn?

1 script/plugin install \ 2 git://github.com/tra/spawn.git

PLAY/TYPE

explain what’s going on here... we’ve used spawn as a runner for workling. what’s spawn?

1 spawn do 2 logger.info("I feel sleepy...") 3 sleep 11 4 logger.info("Time to wake up!") 5 end

PLAY/TYPE

by itself you can run it like this. it will fork the process....

1 >> fork { sleep 100 } 2 => 1060

PLAY/TYPE

like this, basically, but with all rails fixes and tweaks in place. above: drops to unix, the OS copies the process & creates a child process. try this in your console and use top to look at the processes.

PLAY/TYPE

workling + spawn inherits these traits.

• Fast. Happens at OS level

PLAY/TYPE

workling + spawn inherits these traits.

• Fast. Happens at OS level

• Rails copy can be big. Irb says ~35MB

PLAY/TYPE

workling + spawn inherits these traits.

• Fast. Happens at OS level

• Rails copy can be big. Irb says ~35MB

• Local. Happening on same Machine

PLAY/TYPE

workling + spawn inherits these traits.

• Fast. Happens at OS level

• Rails copy can be big. Irb says ~35MB

• Local. Happening on same Machine

• Kill scenario - no persistence, job lost

PLAY/TYPE

workling + spawn inherits these traits.

...If you just want to fire and forget a local process as you say, I think Spawn is pretty good.

“Twitter’s Evan Weaver and nesting friend.

PLAY/TYPE

before i started on workling, i asked evan weaver of chow fame (twitter now) what he thought. this is his what he said about spawn.

BackgroundJob

PLAY/TYPE

new kid on the block. very nice take on things.

1 script/plugin install \ 2 git://github.com/purzelrakete/workling.git 3 4 ./script/plugin install \ 5 http://codeforpeople.rubyforge.org/svn/rails/plugins/bj 6 7 ./script/bj setup

PLAY/TYPE

lets start over, workling + bj. don’t need to do anything else, since bj is automatically detected.

1 Workling::Remote.dispatcher = 2 Workling::Remote::Runners::BackgroundjobRunner.new 3

PLAY/TYPE

however, the workling runner can also be set manually like this, inside of environment.rb or under config/initializers. this is being done automatically for you.

Milk it!

PLAY/TYPE

Why the lag?

PLAY/TYPE

i will explain... first of all, what is backgroundjob.

PLAY/TYPE

next slide: installing. already did this.

• Written by Ara T. Howard (codeforpeople)

PLAY/TYPE

next slide: installing. already did this.

• Written by Ara T. Howard (codeforpeople)

• Sponsored by Engineyard

PLAY/TYPE

next slide: installing. already did this.

• Written by Ara T. Howard (codeforpeople)

• Sponsored by Engineyard

• Lightweight, persistent.

PLAY/TYPE

next slide: installing. already did this.

1 ./script/plugin install \ 2 http://codeforpeople.rubyforge.org/svn/rails/plugins/bj 3 4 ./script/bj setup

PLAY/TYPE

1 create_table :bj_config do |t| 2 t.column "command" , :text 3 t.column "state" , :text 4 t.column "priority" , :integer 5 t.column "tag" , :text 6 t.column "is_restartable" , :integer 7 t.column "submitter" , :text 8 t.column "runner" , :text 9 t.column "pid" , :integer 10 t.column "submitted_at" , :datetime 11 t.column "started_at" , :datetime 12 t.column "finished_at" , :datetime 13 t.column "env" , :text 14 t.column "stdin" , :text 15 t.column "stdout" , :text 16 t.column "stderr" , :text 17 t.column "exit_status" , :integer 18 end

setup is running this migration.

1 job = Bj.submit 'cat /etc/password' 2 Bj.table.job.find(:all) # jobs table

PLAY/TYPE

t.column "pid" , :integer t.column "finished_at" , :datetime t.column "stdin" , :text t.column "stdout" , :text t.column "stderr" , :text t.column "exit_status" , :integer

1 if(job.finished) ...

PLAY/TYPE

If you want something back... these are some useful columns in the db. they are available on the job object, too.

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: load Rails 1x / Request

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: load Rails 1x / Request

• Memory: copy of Rails / Request. No leaks.

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: load Rails 1x / Request

• Memory: copy of Rails / Request. No leaks.

• Kill scenario - Persistent over DB

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: load Rails 1x / Request

• Memory: copy of Rails / Request. No leaks.

• Kill scenario - Persistent over DB

• Jobs runner process manages itself

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: load Rails 1x / Request

• Memory: copy of Rails / Request. No leaks.

• Kill scenario - Persistent over DB

• Jobs runner process manages itself

• Runner can be on another machine

PLAY/TYPE

workling + bj inherits these traits.

PLAY/TYPE

howz it work? this is why the moo came later than with spawn.

• Starts a thread for each job

PLAY/TYPE

howz it work? this is why the moo came later than with spawn.

• Starts a thread for each job

• The thread invokes a new OS process

PLAY/TYPE

howz it work? this is why the moo came later than with spawn.

• Starts a thread for each job

• The thread invokes a new OS process

• ./script/runner loads rails

PLAY/TYPE

howz it work? this is why the moo came later than with spawn.

• Starts a thread for each job

• The thread invokes a new OS process

• ./script/runner loads rails

• Results written to DB

PLAY/TYPE

howz it work? this is why the moo came later than with spawn.

• Starts a thread for each job

• The thread invokes a new OS process

• ./script/runner loads rails

• Results written to DB

• Client side gets results from DB

PLAY/TYPE

howz it work? this is why the moo came later than with spawn.

Added Bj Runner to Workling like this...

PLAY/TYPE

Added the BJ runner yesterday. here’s how it was done...

1 require 'workling/remote/runners/base' 3 module Workling 4 module Remote 5 module Runners 6 class BackgroundjobRunner < Workling::Remote::Runners::Base 7 cattr_accessor :routing 8 9 def initialize 10 BackgroundjobRunner.routing = 11 Workling::Starling::Routing::ClassAndMethodRouting.new 12 end 13 14 def run(clazz, method, options = {}) 15 stdin = @@routing.queue_for(clazz, method) + 16 " " + 17 options.to_xml(:indent => 0, :skip_instruct => true) 18 19 Bj.submit "./script/runner ./script/bj_invoker.rb", 20 :stdin => stdin 21 22 return nil # that means nothing! 23 end 24 end 25 end 26 end 27 endexplain what’s going on.

1 @routing = Workling::Starling::Routing::ClassAndMethodRouting.new 2 unnormalized = REXML::Text::unnormalize(STDIN.read) 3 message, command, args = *unnormalized.match(/(^[^ ]*) (.*)/) 4 options = Hash.from_xml(args)["hash"] 5 6 if workling = @routing[command] 7 workling.send @routing.method_name(command), options.symbolize_keys 8 end

Starling

PLAY/TYPE

1 gem sources -a http://gems.github.com/ 2 sudo gem install starling-starling 3 sudo gem install fiveruns-memcache-client 4 5 script/plugin install \ 6 git://github.com/purzelrakete/workling.git

PLAY/TYPE

add github to your sources if you havent already done so. explain fiveruns client.

1 mkdir /var/spool/starling 2 sudo starling -d 3 script/workling_starling_client start

PLAY/TYPE

need 2 processes running. 1: starling. 2: workling starling client.

1 Workling::Remote.dispatcher = 2 Workling::Remote::Runners::StarlingRunner.new

PLAY/TYPE

Milk it already...

PLAY/TYPE

StarlingPLAY/TYPE

lightweight queue that speaks memcached. developed at twitter by blaine cook 2 make twitter arch more msg-oriented.

4 # Put messages onto a queue: 5 require 'memcache' 6 starling = MemCache.new('localhost:22122') 7 starling.set('my_queue', 1) 8 9 # Get messages from the queue: 10 require 'memcache' 11 starling = MemCache.new('localhost:22122') 12 loop { puts starling.get('my_queue') } 13

PLAY/TYPE

Memcache ClientPLAY/TYPE

Memcache Client

• Errors in Memcache Client (Robot Co-Op 1.5.0)

PLAY/TYPE

Memcache Client

• Errors in Memcache Client (Robot Co-Op 1.5.0)

• Solution: http://github.com/fiveruns/memcache-client/tree/mastermemcache-client/tree/master

PLAY/TYPE

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: very fast.

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: very fast.

• Memory low, unless you’re leaking. Use God to monitor / restart your workers.

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: very fast.

• Memory low, unless you’re leaking. Use God to monitor / restart your workers.

• Kill scenario - Persistent over Starling

PLAY/TYPE

workling + bj inherits these traits.

• Warmup speed: very fast.

• Memory low, unless you’re leaking. Use God to monitor / restart your workers.

• Kill scenario - Persistent over Starling

• Need to manage processes

PLAY/TYPE

workling + bj inherits these traits.

... The main things lacking in Starling are non-destructive reads (transactions), and speed.

“PLAY/TYPE

twitter moving away from starling. putting msgs back onto queue not possible after kill/crash.

... The main things lacking in Starling are non-destructive reads (transactions), and speed.

“• Transactions. Imagine Starling is killed just

after reading a msg off a queue... not reliable. Doesnt map nicely onto memcache

PLAY/TYPE

twitter moving away from starling. putting msgs back onto queue not possible after kill/crash.

... The main things lacking in Starling are non-destructive reads (transactions), and speed.

“• Transactions. Imagine Starling is killed just

after reading a msg off a queue... not reliable. Doesnt map nicely onto memcache

• It can take 20 minutes to play back a Starling journal after a crash on a very powerful machine. In production, this is about 19.5 minutes too many.

PLAY/TYPE

twitter moving away from starling. putting msgs back onto queue not possible after kill/crash.

apparently stable, millions of messages / day with workling + starling. we are using starling at play/type and for us, it’s fine. but if replay for huge traffic / destructive reads are an issue, starling isn’t for you.

TODOs

PLAY/TYPE

workling is up on github. fork it! here’s what needs to be done, come join the project.

MemcachelikeRunner

PLAY/TYPE

MemcachelikeRunnerPLAY/TYPE

take the StarlingRunner and refactor it to be generic for all Queue Systems that imitate the memcache api. once this is done, we’ll be able to plug in the following... sparrow + workling running out there, no code unfortunately.

MemcachelikeRunner

• Sparrow (“a really fast lightweight queue written in Ruby that speaks memcache. “)

PLAY/TYPE

take the StarlingRunner and refactor it to be generic for all Queue Systems that imitate the memcache api. once this is done, we’ll be able to plug in the following... sparrow + workling running out there, no code unfortunately.

MemcachelikeRunner

• Sparrow (“a really fast lightweight queue written in Ruby that speaks memcache. “)

• RudeQ (DB based, no process for queue)

PLAY/TYPE

take the StarlingRunner and refactor it to be generic for all Queue Systems that imitate the memcache api. once this is done, we’ll be able to plug in the following... sparrow + workling running out there, no code unfortunately.

BeanstalkdRunnerPLAY/TYPE

might be possible to run this with a MemcachelikeRunner.

BeanstalkdRunner

• Fast non persistent Queue written in C.

PLAY/TYPE

might be possible to run this with a MemcachelikeRunner.

BeanstalkdRunner

• Fast non persistent Queue written in C.

• Written for “Causes” on Facebook

PLAY/TYPE

might be possible to run this with a MemcachelikeRunner.

AMPQRunner

PLAY/TYPE

BackgroudndRB

PLAY/TYPE

heavyweight of backgrounding, oldest solution. lots of people using this.

I wish, people will check their facts before making any claims, I am kinda getting tired of fighting this FUD within community. There are few outstanding issues, but BackgrounDRb supports many features that other similar alternatives doesn’t offer. And I am working on it.

“

- Hemant

FUD?PLAY/TYPE

backgroundrb comes with emotional baggage, for me. who’s running backgroundrb in the room, hands up? who has problems with it? who has NO problems?

BackgroundRB

PLAY/TYPE

BackgroundRB

• As of version1.0.3 - complete rewrite with Packet, no DRB code in there anymore.

PLAY/TYPE

Packet is a network programming library in the spirit of EventMachine and yet it has nice functionality of letting you attach callbacks to workers running in separate process. It can even let you invoke callbacks running on worker in different machine and stuff like that. When I took over project it was based on DRb, but since then I have removed DRb and BackgrounDRb is 100% based on evented model of network programming.

“

- Hemant

PLAY/TYPE

my personal impression: still heavy. waiting for somebody to try integrating it into workling, no personal need.

Okay, but what about Workling Status and

Return?

PLAY/TYPE

have a real world examle. old school, circa Feb. 2008 social network imports over gmail scraping... need this out of the request, but the response has to be shown, too.

1 class NetworkWorker < Workling::Base 2 def search(options) 3 accounts = options[:accounts] 4 uid = options[:uid] 5 6 accounts.map do |network| 7 Blackbook.get \ 8 :username => network[:username], 9 :password => network[:password]) 10 end 11 12 Workling::Return::Store.set(uid, accounts) 13 end 14 end

PLAY/TYPE

explain how this works - scraping gmail. return store: again, using memcache api.

1 def poll 2 @results = Workling::Return::Store.get \ 3 params[:workling_uid] 4 5 # TODO: handle no results, results 6 # and results with errors 7 end

PLAY/TYPE

Questions?Lunch!

Rany Keddorany@playtype.net

PLAY/TYPE