Asynchronous Service Servers

Based on AMQP for Ruby

Versapay

• Point of sales

• eCommerce payment processing

• Partner with JPChase

• No.1 Growth Company in Canada (Profit50)

Prelude: ConcurrencyOr, ouch, it hurts…

Nietzsche on Concurrency

…if you gaze for long into an abyss, the abyss gazes also into you.

Spectrum of Grainedness

• Parallelism– The Multicore Future™

– Code structuring (e.g. block/wait)

• Concurrency– Simultaneous tasks

• Distributed Services

• Internet/REST– REST (all its constraints: stateless, layered,

cacheable, uniform interace, …)

Lazy Scalability

• Worry Later?

– Very hard to fix a wrong architecture

• Get it right, right now!

– Architecture (constraints) makes scaling possible

– Actual scaling when you need it

• Path from concurrency to SOA

– Actor Model (e.g. Erlang)

– Key-value store (no join)

Actor Model

The Actor model is characterized by inherent concurrency of computation within and among actors, dynamic creation of actors, inclusion of actor addresses in messages, and interaction only through direct asynchronous message passing with no restriction on message arrival order.

N.B. From wikipedia

Actor Model

• Identified by address– Mobile (doesn’t matter which machine)

– Transient (doesn’t matter which process)

– Decouples the “who” of computation

• Message passing– Ask, don’t tell

– Decouples the “what” of computation

• Asynchrony– Event-based

– Decouples the “when” of computation

Erlang-Style Concurrency

• Creating and destroying processes is very fast.• Sending messages between processes is very fast.• Processess behave the same way on all operating

systems.• We can have very large numbers of processes.• Processes share no memory and are completely

independent.• The only way for processes to interact is through

message passing.– Joe Armstrong

More on Erlang

• Message passing is asynchronous.

• Processes can monitor each other.

• It is possible to selectively receive messages.

• Remote processes appear largely the same as local processes.

– Ulf Wiger

ASS kicking with RabbitMQ

Stealing Ideas from Erlang

AMQP

• Just a protocol specification

– Open, interoperable

– Binary

– Extensible

– Dynamic creation of entities

• For enterprise message middleware

• RabbitMQ is a free implementation

– Qpid is an Apache project

AMQP(<1.0) Details

• AMQP Entities– Exchange (publish messages here)

– Queue (route messages here)

– Binding (establishes routing between exchange and queue)

• AMQP Error Model– The sound of success is silence

– An error just closes connection

• QOS, prefetch window

ASS

• Inspired by Erlang gen_server

• Hide concurrency behind generic server

• Parameterize generic server with modules

– Familiar sequential programming

• Pass around Ruby objects by marshalling

– Switch to JSON?

• Full access to AMQP if you need it

A Simple ASS Actor

require 'rubygems'

require 'ass'

ASS.start do

ASS.actor("demo") {

def echo(o)

o

end

}

end

Demo

• Server/Client

• Load distribution

• Store and forward

– Zero downtime

• Future & RPC

– Wait in batches

• Failure recovery with :ack => true

RPC is an Illusion

• ASS is fundamentally asynchronous

– Async/event-based programming is unfamiliar

• RPC client does blocking wait to simulate sequential programming

– “Future” gives some asynchrony back

• Timeout as generic exception

– Usually just give up and die

RPC is EVIL when…

• Hides too much

– Random failures

• Doesn’t hide enough

– Exposing the implementation

– Leaky abstraction

See:

• http://steve.vinoski.net/blog/2008/07/01/convenience-over-correctness/

Experiences So Far

Programming Murphy’s Computer

• Anything can fail at any time…

– Think in critical sections

• Retry is the bedrock of reliability

– :ack => true

• FSM is a good mental aid

– Mark progress with a database (ACID state)

• It’s ok to die… process is transient, data is forever

API

• Message verification

– Pattern Matching

– http://github.com/hayeah/matchmaker

• Fail Fast and Hard at the API

– Avoid error spreading through the services

• Share the API spec among services

• Weak API

Processes, Not Threads

• Avoid threads, avoid horrors

• The stack is opaque, you don’t know if anything you use might cause blocking

• Let OS and DB do the hard work

– Process isolation

– Anything wrong? Easy, kill it!

• IPC cost, but you don’t care (most of the time)

Testing Concurrent App

• No established practice…

• Test the real thing, not mocks

– Pretty easy with ASS, hard with web-services

• Test in a single process, deploy to multiple processes, then hosts, then networks

• Invariants are super important

– Some kind of property testing

– Must test with enough load

Future Work

Like, TODO, right now!

System Health

• Monitoring

• Event based

• Metrics

• Logging

• Self-management

– Supervisor hierarchy

– Zookeeper?

Better Testing

• Stress Test

• Testing with generated message sequences

– Some kind of proxy to force message order

• Some kind of code injection technique

• System invariant checker

• Test failures (connection dying, db dying, etc)

• Continuous integration

What Datastore?

• We are using ActiveRecord at the moment

– Because it’s there…

• Some kind of schemaless key-value store

– Like Friendsfeed

Conclusion

Three Legged Stool

• Scalability gives,

• Distributability gives,

• Reliability gives scalability

• http://github.com/hayeah/ASS

– ZERO documentation

– Quirks

– It’s not Rails