wishbone documentation - media.readthedocs.org · wishbone.module.stdoutmodule instances which...

Wishbone DocumentationRelease 0.4.8

Jelle Smet

September 14, 2015

Contents

1 Bootstrapping 31.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.3 Actor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61.4 Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.5 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131.6 Bootstraping on CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191.7 Patterns and best practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2 Indices and tables 25

i

Wishbone Documentation, Release 0.4.8

https://github.com/smetj/wishbone

A Python library to build and CLI tool to manage asynchronous coroutine based event pipeline servers with minimaleffort.

Works on python 2.6+, 2.7+ including Python 3 and PyPy 2.0.2

>>> from wishbone.router import Default>>> from wishbone.module import TestEvent>>> from wishbone.module import RoundRobin>>> from wishbone.module import STDOUT>>>>>> router=Default()>>> router.register(TestEvent, "input")>>> router.register(RoundRobin, "mixing")>>> router.register(STDOUT, "output1", prefix="I am number one: ")>>> router.register(STDOUT, "output2", prefix="I am number two: ")>>>

Contents 1



>>> router.connect("input.outbox", "mixing.inbox")>>> router.connect("mixing.one", "output1.inbox")>>> router.connect("mixing.two", "output2.inbox")>>>>>> router.start()>>> router.block()I am number one: testI am number two: testI am number one: testI am number two: testI am number one: testI am number two: testI am number one: testI am number two: testI am number one: testI am number two: test

This example we initialize the wishbone.router.Default router to create a simple setup in which weconnect the TestEvent input module, which does nothing more than generating the word “test” every sec-ond, to the wishbone.module.RoundRobin module which on its turn roundrobins the incoming events to 2wishbone.module.STDOUT module instances which print all incoming events to STDOUT.

2 Contents

CHAPTER 1

Bootstrapping

Wishbone comes with a CLI tool to easily bootstrap a server using a YAML formatted config file. Following filecreates exactly the same environment as the above example:

---modules:

input:module: wishbone.builtin.input.testevent

mixing:module: wishbone.builtin.flow.roundrobin

output1:module: wishbone.builtin.output.stdoutarguments:

prefix: "I am number one: "


prefix: "I am number two: "

routingtable:- input.outbox -> mixing.inbox- mixing.one -> output1.inbox- mixing.two -> output2.inbox

...

Bootstrapping the environment is just a matter of invoking the wishbone executable with the –config parameter point-ing to the bootstrap file.

[smetj@indigo ~]$ wishbone debug --config test.yaml2013-08-09T23:13:39 pid-13797 informational Router: Queue one does not exist in module mixing. Autocreate queue.2013-08-09T23:13:39 pid-13797 informational Router: Queue two does not exist in module mixing. Autocreate queue.2013-08-09T23:13:39 pid-13797 informational Router: Starting.2013-08-09T23:13:39 pid-13797 informational loglevelfilter: Initiated2013-08-09T23:13:39 pid-13797 informational loglevelfilter: Created module queue named inbox with max_size 0.2013-08-09T23:13:39 pid-13797 informational loglevelfilter: Created module queue named outbox with max_size 0.... snip ...2013-08-09T23:13:39 pid-13797 informational input: StartedI am number two: testI am number one: testI am number two: testI am number one: test

3


I am number two: testI am number one: testI am number two: testI am number one: test^C2013-08-09T23:13:47 pid-13797 informational Router: Received SIGINT. Shutting down.2013-08-09T23:13:47 pid-13797 informational Router: Stopping.2013-08-09T23:13:47 pid-13797 informational output2: Shutdown2013-08-09T23:13:48 pid-13797 warning output2: Queue <wishbone.tools.wishbonequeue.WishboneQueue instance at 0x2101a28> locked.2013-08-09T23:13:48 pid-13797 informational output1: Shutdown2013-08-09T23:13:48 pid-13797 warning output1: Queue <wishbone.tools.wishbonequeue.WishboneQueue instance at 0x2101680> locked.2013-08-09T23:13:48 pid-13797 informational mixing: Shutdown2013-08-09T23:13:48 pid-13797 warning mixing: Queue <wishbone.tools.wishbonequeue.WishboneQueue instance at 0x2101098> locked.2013-08-09T23:13:49 pid-13797 informational input: Shutdown[smetj@indigo ~]$

Contents:

1.1 Installation

1.1.1 Pypi

You can install the latest stable version of Wishbone from https://pypi.python.org/pypi/wishbone/ by usingeasy_install:

$ easy_install wishbone

All dependencies should be resolved automatically.

1.1.2 From source

You can install the latest stable or development version from https://github.com/smetj/wishbone

development

To install the latest development release you have to checkout the develop branch and build from there:

$ git clone https://github.com/smetj/wishbone.git$ cd wishbone$ git checkout develop$ sudo python setup.py install

stable

To install the latest stable release you to checkout the master branch and build from there:

$ git clone https://github.com/smetj/wishbone.git$ cd wishbone$ git checkout master$ sudo python setup.py install

4 Chapter 1. Bootstrapping

https://pypi.python.org/pypi/wishbone/



1.2 Introduction

Wishbone is a Python library to create IO driven event processing servers by defining a pipeline of inputs and outputswith a number of intermediate processing stages in between through which events travel.

It also provides the tools to quickly bootstrap and control event pipeline setups from CLI and have them running aspermanent solutions.

1.2.1 Modules and Queues

Modules are isolated blocks of code (greenlets) each with their own specific functionality. They are created by sim-ply inheriting wishbone.Actor as a baseclass. Modules cannot directly invoke each others functionality. Theironly means of interaction is by passing messages or events to each other’s wishbone.tools.WishboneQueuequeues. Modules typically have, but are not limited to, an inbox and outbox queue.

Queues always live inside a wishbone.tools.QueuePool which, besides offering some convenience functions,is nothing more than a container to centralize all the queues of the module. Typically, modules consume each evententering the inbox queue using the wishbone.Actor.consume() function where the event can be processed andafter which it is submitted to the module’s outbox queue from where it can be forwarded again to another module’squeue.

Besides a selection of builtin modules which deliver core Wishbone functionality, there is also a collection of moduleswhich are maintained as a separate project. The modules can be found on https://github.com/smetj/wishboneModules.

1.2.2 Router

The wishbone.router.Default router is used to initialize the modules and to organize the event stream betweenthem.

Modules are registered using wishbone.router.Default.register(). The router takes care of the properstartup wishbone.router.Default.start() and shutdown wishbone.router.Default.start()sequence of all the modules.

Queues are connected to each other using wishbone.router.Default.connect(). Queues can only havea “1 to 1” relationship. If you require a “1 to N” or similar scenario you might have to use one of the builtin flowmodules.

The router also takes care of the logs and metrics produced by the modules. By register-ing Wishbone modules using wishbone.router.Default.registerLogModule() andwishbone.router.Default.registerMetricModule() we can pretty much do what we want withthem.

1.2.3 Events

Wishbone events are simple data structures which need to have following format:

{ "header":{}, "data": object }

Input modules are responsible to create events with the correct format from the incoming data. Events which do notcomply with this format are discarded.

1.2. Introduction 5

https://github.com/smetj/wishboneModules


1.2.4 Gevent

Wishbone is build using Gevent. The modules are running as cooperatively scheduled greenlets while taking advantageof the cooperative socket support for network IO.

1.3 Actor

1.3.1 Introduction

Modules are isolated blocks of code (greenlets) each with their own specific functionality. They are created by sim-ply inheriting wishbone.Actor as a baseclass. Modules cannot directly invoke each others functionality. Theironly means of interaction is by passing messages or events to each other’s wishbone.tools.WishboneQueuequeues. Modules typically have, but are not limited to, an inbox and outbox queue.

Queues always live inside a wishbone.tools.QueuePool which, besides offering some convenience functions,is nothing more than a container to centralize all the queues of the module. Typically, modules consume each evententering the inbox queue using the wishbone.Actor.consume() function where the event can be processed andafter which it is submitted to the module’s outbox queue from where it can be forwarded again to another module’squeue.

1.3.2 Example

As an example, let’s make a module which reverses the content of incoming events and optionally converts the firstletter into a capital.

1 from wishbone import Actor2

3

4 class ReverseData(Actor):5 '''**Sample module which reverses incoming events.**6

7 Parameters:8

9 - name (str): The instance name.10

11 - capitalize(bool): When True capitalizes the12 first letter.13 Default: True14

15 Queues:16

17 - inbox: Incoming events.18 - outbox: Outgoing events.19 '''20

21 def __init__(self, name, capitalize=False):22 Actor.__init__(self, name, setupbasic=True)23 self.capitalize=capitalize24 self.logging.info("Initialized")25

26 def consume(self, event):27

28 data = event["data"]29



30 data = [::-1]31

32 if self.capitalize == True:33 data = data.title()34 event["data"]=data35

36 self.queuepool.outbox.put(event)

So let’s break down what we are seeing here:

We inherit the Actor base class [4] and initialize it [21]. We pass it the a name and the setupbasic pa-rameter [21]. The setupbasic parameter does nothing than creating an inbox and outbox queue and registringthe consume() function as the function which is going to process the events arriving to inbox by calling thewishbone.Actor.createQueue() and wishbone.Actor.registerConsumer() in the background.

When setupbasic is False you will have to create the required queues and register the required consume yourself.

Wishbone modules can produce log messages [24] using the functions from wishbone.tools.QLogging. Logsare collected by the router and can be forwarded to other modules for further processing, so we don’t need to worryany further about that.

Our module itself accepts 1 parameter [21] which we can use to modify its behaviour and which we can use whenregistering the module in the router using wishbone.router.Default.register().

In the background, the Actor module takes care of consuming the events arriving to the inbox and invoking thewishbone.Actor.consume() [26] on each one of them. Events always have following format:


So we change the content of data [28-34] according to the modules behaviour.

Last but not least, our modified event needs to go to another module. Remember modules don’t communicate directlyto each other? That’s why we put them into the outbox queue [36] for another module to consume.

And that’s it. Save your module and make sure you can import it as a regular Python module and that you can importit an entrypoint.

class wishbone.Actor(name, setupbasic=True)Bases: wishbone.base.BaseActor, wishbone.tools.consumer.Consumer

This baseclass provides the functionality required to build a Wishbone module

Parameters:

•name(string): The name of the module.

•setupbasic(bool): When True, does some assumptions on how to setup the module. Default: True

•context_switch(int): Execute a context switch every <context_switch> messages consumed from aqueue. Default: 100

block()Convenience function which blocks untill the actor is in stopped state.

consume(event)Raises error when user didn’t define this function in his module.

createQueue(name, max_size=0)Creates a queue in <self.queuepool> named <name> with a size of <max_size>

1.3. Actor 7


deleteQueue(name)Deletes the <name> queue from <self.queuepool>.

disableConsuming()Sets a flag which makes the router stop executing consume().

The module will not process further any events at this point until enableConsuming() is called.

enableConsuming()Sets a flag which makes the router start executing consume().

The module will starts/continues to excete the consume() function.

loop()Convenience function which returns True until stop() has be been called. A word of caution. Since we’redealing with eventloops, if you use a loop which doesn’t have any gevent aware code in it then you’ll blockthe event loop.

putEvent(event, destination)Convenience function submits <event> into <destination> queue. When this fails due to QueueFull orQueueLocked, the function will block untill the error state disappeard and will resubmit the event afterwhich it will exit.

Should ideally be used by input modules.

registerConsumer(fc, q)Registers <fc> as a consuming function for the given queue <q>.

shutdown()Stops each module by making <self.loop> return False and which unblocks <self.block>

start()Starts to execute all the modules registered <self.consume> functions.

stop()Stops each module by making <self.loop> return False and which unblocks <self.block>

class wishbone.tools.QueuePool

dump(name)Convenience function to self.<name>.dump

getQueueInstances()Return a dict of queue instances from the queuepool.

listQueues()return the list of queue names from the queuepool.

messagesLeft()Checks each queue whether there are any messages left.

shutdown()Closes all queues in preparation of actor shutdown.

class wishbone.tools.WishboneQueue(max_size=0)A queue used to organize communication messaging between Wishbone Actors.

Parameters:

•max_size (int): The max number of elements in the queue. 0 is unlimited Default: 0



clear()Clears the queue.

dump()Dumps and returns the queue in tuple format.

empty()Returns True when queue and unacknowledged is empty otherwise False.

get()Gets an element from the queue.

Blocks when empty until an element is returned.

getLock()Locks getting data from queue.

getUnlock()Unlocks getting data from queue.

isLocked()Returns whether the queue is in locked or unlocked state.

True means locked, False means unlocked.

lock()Sets queue in locked state.

When in locked state, elements can not be added or consumed.

putLock()Locks putting data in queue.

putUnlock()Unlocks putting data in queue.

rescue(data)Puts data to the beginning of the queue overriding the lock.

size()Returns a tuple of the queue and unacknowledged the size of the queue.

stats()Returns statistics of the queue.

unlock()Sets queue in unlocked state.

When in unlocked state, elements can be added or consumed.

waitUntilData()Blocks untill data is available unless queue is locked.

waitUntilFreePlace()Blocks until we have at least 1 slot free in the queue.

waitUntilGetAllowed()Blocks until getting data from the queue is allowed again.

waitUntilPutAllowed()Blocks until writing data into the queue is allowed again.

1.3. Actor 9


class wishbone.tools.QLogging(name)Generates Wishbone formatted log messages following the Syslog priority definition.

Generated logs are stored in a local queue <self.logs>. It is up to an external process the consume this queue.

alert(message)Generates a log message with priority alert(1).

crit(message)Generates a log message with priority critical(2).

critical(message)Generates a log message with priority critical(2).

debug(message)Generates a log message with priority debug(7).

emerg(message)Generates a log message with priority emergency(0).

emergency(message)Generates a log message with priority emergency(0).

err(message)Generates a log message with priority error(3).

error(message)Generates a log message with priority error(3).

info(message)Generates a log message with priority informational(6).

informational(message)Generates a log message with priority informational(6).

notice(message)Generates a log message with priority notice(5).

warn(message)Generates a log message with priority warning(4).

warning(message)Generates a log message with priority warning(4).

1.4 Router

The wishbone.router.Default router is used to initialize the modules and to organize the event stream betweenthem.

Modules are registered using wishbone.router.Default.register(). The router takes care of the properstartup wishbone.router.Default.start() and shutdown wishbone.router.Default.start()sequence of all the modules.

Queues are connected to each other using wishbone.router.Default.connect(). Queues can only havea “1 to 1” relationship. If you require a “1 to N” or similar scenario you might have to use one of the builtin flowmodules.

The router also takes care of the logs and metrics produced by the modules. By register-ing Wishbone modules using the wishbone.router.Default.registerLogModule() andwishbone.router.Default.registerMetricModule() we can pretty much do what we want withthem.



1.4.1 Throttling

The default router contains basic support for throttling. When throttling is enabled by setting the the throttle variable toTrue, a greenthread scans periodically all initialized modules to detect instances which have more events queued thandefined in the throttle_threshold variable. Once such a module is found, all upstream parent modules are identifiedand verified which one is the source of overflowing the children. When the parent module is identified, the router callsthe wishbone.Actor.enableThrottling() and wishbone.Actor.disableThrottling() accord-ingly.

It is up to the author of the module to override these functions and have them make correct measurements.

1.4.2 Logging

Modules produce logs using wishbone.tools.QLogging. These logs are automatically collected by the routerand are put into a queue within the router instance. By using wishbone.router.registerLogModule() wecan connect whatever to process the logstream in whatever way.

Typically the wishbone.module.LogLevelFilter is registered with wishbone.router.Default.registerLogModule()to filter out the interesting loglevels before sending the logstream to any other modules such aswishbone.module.Syslog or wishbone.module.STDOUT.

1.4.3 Metrics

Modules produce 2 kinds of metrics:

• Queue related metrics which are generated by wishbone.tools.WishboneQueue.stats().

• Function related metrics can optionally be generated by decorating a function withwishbone.tools.Measure.runTime()

The router instance is responsible for grabbing the metrics at the wishbone.router.Default.interval. Thegathered metrics are stored in the router’s metrics queue from which they can be consume by the module which hasbeen registered using wishbone.router.Default.registerMetricModule()

Typically the wishbone.module.Graphite is registered as metricmodule, which formats the metrics into aGraphite format. That module can be connected on its turn to the TCP client module in order to forward the metricdata to Graphite.

class wishbone.router.Default(interval=1, context_switch=100, rescue=False, uuid=False, throt-tle=True, throttle_threshold=15000)

The default Wishbone router.

A router is responsible to:

•Forward the events from one queue to the other.

1.4. Router 11


•Forward the logs from all modules to the logging module

•Forward the metrics from all modules to the metrics module.

SIGINT is intercepted and initiates a proper shutdown sequence.

Parameters:

•interval(int): The interval metrics are polled from each module

•context_switch(int): When looping over blocking code a context switch is done every X iterations.Default: 100

•rescue(bool): Whether to extract any events stuck in one of the queues and write that to a cache file.Next startup the events are read from the cache and inserted into the same queue again.

•uuid(bool): If True, adds a uuid4 value in the header of each event if not present when forwardedfrom one queue to the other. (default False)

•throttle(bool): If True, scans every second for modules which overflow child modules with messagesand thus have to be throttled. default: False

•throttle_threshold(int): When <throttle> is True, start to throttle start to throttle when a module hasmore than <throttle_threshold> messages. default: 5000

block()A convenience function which blocks untill all registered modules are in a stopped state.

Becomes unblocked when stop() is called and finisehd.

connect(producer, consumer)Connects a producing queue to a consuming queue.

A separate greenthread consumes the events from the consumer queue and submits these to the producerqueue. When a non existing queue is defined, it is autocreated.

The syntax of queue names is:

module_instance_name.queuename

Parameters:

•producer(str): The name of the producing module queue.

•consumer(str): The name of the consuming module queue.

doRescue()Runs over each queue to extract any left behind messages.

loop()Convenience funtion which returns a bool indicating the router is in running or stop state.

moduleMonitor(module, child_name, child_size)A moduleMonitor is a greenthread which monitors the state of a module and triggers the expected throttlingfunctions. It exits when no more throttling is required.

register(module, name, *args, **kwargs)Registers a Wishbone actor into the router.

Parameters:

module(module) A wishbone module. name(string) The name to assign to the module insance.args(list) Positional arguments to pass to the module. kwargs(dict) Named arguments to pass tothe module.



registerLogModule(module, name, *args, **kwargs)Registers and connects the module to the router’s log queue.

If this method is not called (no module connected to it) the queue is automatically connected to a Nullmodule.

Parameters:


registerMetricModule(module, name, *args, **kwargs)Registers and connects the module to the router’s log queue.

If this method is not called (no module connected to it) the queue is automatically connected to a Nullmodule.

Parameters:


start()Starts the router and all registerd modules.

Executes following steps:

•Starts the registered logging module.

•Starts the registered metrics module.

•Calls each registered module’s start() function.

stop()Stops the router and all registered modules.

It stops all the modules except the modules connected to the logs or metrics endpoint to ensure these eventstreams survive untill the end. All other modules are shutdown in the order they have been registered.

throttleMonitor()Sweeps over all registered modules and tries to detect modules with problematic flow rates. Once found amoduleMonitor greenthread is spawned which is reponsible for enabling and disabling throttling.

class wishbone.tools.Measure

classmethod runTime(fn)Decorator function which keeps track of the number of calls and the time it takes to execute the function.

1.5 Modules

Wishbone comes with a set of builtin modules. Modules are isolated blocks of functionality which are connected toeach other within a router instance.

There are different types of modules:

• input modules

• output modules

1.5. Modules 13


• flow modules

• function modules

Besides the builtin modules there is also a list modules which are developed and maintained apart from Wishbone.They include tcp/udp servers and clients, amqp client, etc ...


1.5.1 Input modules

Input modules take input from the outside world into the Wishbone framework. They often have only 1wishbone.tools.WishboneQueue called outbox in which incoming events go. Input modules are also re-sponsible to format any incoming data into the expected Wishbone internal event format.

TestEvent

class wishbone.module.TestEvent(name, interval=1)A WishBone input module which generates a test event at the chosen interval.

This module is only available for testing purposes and has further hardly any use.

Events have following format:

{ “header”:{}, “data”:”test” }

Parameters:

•name (str): The instance name when initiated.

•interval (float): The interval in seconds between each generated event. Should have a value > 0. de-fault: 1

Queues:

•outbox: Contains the generated events.

1.5.2 Output modules

Output modules take input from another module registered in the Wishbone router and submit these events to theoutside world. They often have only 1 wishbone.tools.WishboneQueue called inbox in which events arrivedestined to go out. They typically act as TCP, AMQP or other network protocol clients.

Output modules often have a rescue queue to which events which failed to go out.




Syslog

class wishbone.module.Syslog(name)Writes Wishbone log events to syslog.

Logevents have following format:

(6, 1367682301.430527, ‘Router’, ‘Received SIGINT. Shutting down.’)

The first value corresponds to the syslog severity level.

Parameters

•name(str) : The name of the module.

Queues:

•inbox: incoming events

Null

class wishbone.module.Null(name)Accepts events and purges these without any further processing.

Useful to discard a stream of events.

Parameters:

•name(str): The name of the module

Queues:

•inbox: incoming events

STDOUT

class wishbone.module.STDOUT(name, complete=False, counter=False, prefix=’‘, pid=False)A builtin Wishbone module prints events to STDOUT.

Prints incoming events to STDOUT. When <complete> is True, the complete event including headers is printedto STDOUT.

Parameters:


•complete (bool): When True, print the complete event including headers. Default: False

•counter (bool): Puts an incremental number for each event in front of each event. Default: False

•prefix (str): Puts the prefix in front of each printed event. Default: “”

•pid (bool): Includes the pid of the process producing the output. Default: False

Queues:

•inbox: Incoming events.

1.5. Modules 15


1.5.3 Flow modules

Flow modules accept and forward events from and to other modules. They fulfill a key roll in building a message flow.Since you can only have a 1:1 relationship between queues, you cannot split or merge event streams. That’s whereflow modules come handy. Flow modules are not expected to alter events when transiting the module.

RoundRobin

class wishbone.module.RoundRobin(name, randomize=False)A builtin Wishbone module which round robins incoming events over all connected queues.

Create a “1 to n” relationship between queues. Events arriving in inbox are then submitted in a roundrobin (orrandomized) fashion to the connected queues. The outbox queue is non existent.

Parameters:

name(str): The name of the module.

randomize(bool): Randomizes the queue selection instead of going round robin over all queues.

Queues:

inbox: Incoming events

Fanout

class wishbone.module.Fanout(name, clone=False)A builtin Wishbone module which duplicates incoming events to all connected queues.

Create a “1 to n” relationship with queues. Events arriving in inbox are then copied to each queue connected tothis module. Keep in mind that the outbox queue is never used.

When clone is True, each incoming event is duplicated for each outgoing queue. This might be usefull if yourequire to change the content of the events later down the pipeline. Otherwise references are used which meanschanging the event will change it everywhere in the current Wishbone framework.

Parameters:


clone(bool): When True actually makes a copy instead of passing a reference.

Queues:


Funnel

class wishbone.module.Funnel(name)A builtin Wishbone module which merges incoming events from different queues into 1 queue.

Create a “n to 1” relationship with queues. Events arriving in different queues are all merged into the outbox.

Parameters:



name(str): The name of the module

Queues:

outbox: Outgoing events.

TippingBucket

class wishbone.module.TippingBucket(name, age=0, size=0, events=0, predefined_header=None)A builtin Wishbone module which buffers data.

This module buffers data and dumps it to the output queue on 3 conditions:

•Last data entered in buffer exceeds <age> seconds.

•The length of the data in buffer exceeds <size> size.

•Number of incoming events exceeds <events>.

When the buffer is empty, the header of the first incoming message will be used as the header for the messagegoing out containing the content of the buffer. If you want to override that header with a predefined one thenuse the <predefined_header> option.

Keep in mind to set at least one of the parameters otherwise buffering will be indefinite until your box runs outof memory

Parameters:

•age (int): The time in seconds to buffer before flushing. 0 to disable. (default 0)

•size (int): The total size in bytes to buffer before flushing. 0 to disable. (default 0)

•events (int): The total number of events to buffer before flushing. 0 to disable. (default 0)

•predefined_header (dict): Assign this header to the buffered event when submitting to outbox.

Queues:


•outbox: Outgoing events.

LockBuffer

class wishbone.module.LockBuffer(name, size=100)A builtin Wishbone module with a fixed size inbox queue.

This module shovels events from inbox to outbox. The inbox however is fixed in size which locks the down-stream modules when it reached its limit.

Parameters:


•size (int): The maximum size of inbox. Default: 100

Queues:


•outbox: Outgoing events.

1.5. Modules 17


1.5.4 Function modules

Function modules accept and forward events from and to other modules. They can have a wide range of functionalityand change events when transiting the module.

Header

class wishbone.module.Header(name, key=None, header={})** A builtin Wishbone module which adds the defined dictionary to the header of each passing event.**

Parameters:

•name(str): The name of the module

•key (str): The header key to store the information. Default: <name>

•header(dict): The data to store. Default: {}

Queues:


•outbox: Outgoing modified events.

1.5.5 Logging modules

The logging modules are a specialized set of modules meant to process the logs produced by the modules and collectedby the router.

humanlogformatter

class wishbone.module.HumanLogFormatter(name, colorize=True)A builtin Wishbone module which formats Wishbone log events.

Logs are formated from the internal wishbone format into a more pleasing human readable format suited forSTDOUT or a logfile.

Internal Wishbone format:

(6, 1367682301.430527, ‘Router’, ‘Received SIGINT. Shutting down.’)

Sample output format:

2013-08-04T19:54:43 pid-3342 informational dictgenerator: Initiated 2013-08-04T19:54:43 pid-3342 informa-tional metrics_null: Started

Parameters:

name(str) : The name of the module.



Loglevelfilter

class wishbone.module.LogLevelFilter(name, max_level=6)A builtin Wishbone module which filters Wishbone log events.

name(str) : The name of the module.

max_level(bool) [The maximum log level to show.] Default: 6

1.5.6 Metrics modules

The metrics modules are a specialized set of modules meant to process the metrics produced by the modules andcollected by the router.

graphite

class wishbone.module.Graphite(name, prefix=’‘, script=True, pid=False, source=True)A builtin Wishbone module which formats the internal metric format into Graphite format.

Incoming metrics have following format:

(time, type, source, name, value, unit, (tag1, tag2)) (1381002603.726132, ‘wishbone’, ‘hostname’,‘queue.outbox.in_rate’, 0, ‘’, ())

Parameters:

•name(str): The name of the module.

•prefix(str): Some prefix to put in front of the metric name.

•script(bool): Include the script name. Default: True

•pid(bool): Include pid value in script name. Default: False

•source(bool): Include the source name in the naming schema. Default: True

1.6 Bootstraping on CLI

Wishbone comes with a CLI tool to easily bootstrap a server using a YAML formatted config file. Following filecreates exactly the same environment as the above example:

---modules:

input:module: wishbone.builtin.input.testevent

mixing:module: wishbone.builtin.flow.roundrobin


prefix: "I am number one: "

1.6. Bootstraping on CLI 19



prefix: "I am number two: "

routingtable:- input.outbox -> mixing.inbox- mixing.one -> output1.inbox- mixing.two -> output2.inbox

...

Bootstrapping the environment is just a matter of invoking the wishbone executable with the –config parameter point-ing to the bootstrap file.

1.6.1 Available commands

start

The start command detaches the Wishbone server from console and runs it in the background. This implies that logsare written to syslog unless specifically defined otherwise. Metrics are written to Null unless specifically definedotherwise.

The pidfile contains the pids of the control process and all child processes. When stopping a Wishbone instance makesure you point to the pid file used to start the Wishbone instance.

[smetj@indigo ~]$ wishbone start -husage: wishbone start [-h] [--config CONFIG] [--instances INSTANCES]

[--pid PID]

Starts a Wishbone instance and detaches to the background. Logs are written tosyslog.

optional arguments:-h, --help show this help message and exit--config CONFIG The Wishbone bootstrap file to load.--instances INSTANCES

The number of parallel Wishbone instances tobootstrap.

--pid PID The pidfile to use.

debug

The debug command does pretty much the same as start just that it keeps the Wishbone instance in the foregroundwithout detaching it. Logs are written to STDOUT. The running instance can be stopped gracefully with CTRL+C

[smetj@indigo ~]$ wishbone debug -husage: wishbone debug [-h] [--config CONFIG] [--instances INSTANCES]

Starts a Wishbone instance in foreground and writes logs to STDOUT.

optional arguments:-h, --help show this help message and exit--config CONFIG The Wishbone bootstrap file to load.



--instances INSTANCESThe number of parallel Wishbone instances tobootstrap.

stop

Stops the Wishbone instance gracefully by sending SIGINT to all processes.

smetj@indigo ~]$ wishbone stop -husage: wishbone stop [-h] [--pid PID]

Tries to gracefully stop the Wishbone instance.

optional arguments:-h, --help show this help message and exit--pid PID The pidfile to use.

kill

** Use with caution, sends SIGKILL to the pids in the pidfile. **

[smetj@indigo ~]$ wishbone kill -husage: wishbone kill [-h] [--pid PID]

Kills the Wishbone processes immediately.

optional arguments:-h, --help show this help message and exit--pid PID The pidfile to use.

list

Lists all installed Wishbone modules, given that they have the correct entry-points.

[smetj@indigo ~]$ wishbone listAvailable Wishbone modules:+---------------------------+-------------------+-----------+------------------------------------------------------------------------------------------+| Group | Module | Version | Description |+---------------------------+-------------------+-----------+------------------------------------------------------------------------------------------+| wishbone.builtin.logging | loglevelfilter | 0.4.0beta | A builtin Wishbone module which filters Wishbone log events. || | humanlogformatter | 0.4.0beta | A builtin Wishbone module which formats Wishbone log events. || | | | || wishbone.builtin.metrics | graphite | 0.4.0beta | A builtin Wishbone module which formats the internal metric format into Graphite format. || | | | || wishbone.builtin.flow | roundrobin | 0.4.0beta | A builtin Wishbone module which round robins incoming events || | | | over all connected queues. || | fanout | 0.4.0beta | A builtin Wishbone module which duplicates incoming events to all || | | | connected queues. || | tippingbucket | 0.4.0beta | A builtin Wishbone module which buffers data. |

1.6. Bootstraping on CLI 21


| | funnel | 0.4.0beta | A builtin Wishbone module which merges incoming events from different || | | | queues into 1 queue. || | lockbuffer | 0.4.0beta | A builtin Wishbone module with a fixed size inbox queue. || | | | || wishbone.builtin.function | header | 0.4.0beta | A builtin Wishbone module which adds the defined dictionary || | | | to the header of each passing event. || | | | || wishbone.builtin.input | testevent | 0.4.0beta | A WishBone input module which generates a test event at the chosen interval. || | | | || wishbone.builtin.output | syslog | 0.4.0beta | Writes Wishbone log events to syslog. || | null | 0.4.0beta | Accepts events and purges these without any further processing. || | stdout | 0.4.0beta | A builtin Wishbone module prints events to STDOUT. || | | | || wishbone.input | dictgenerator | 0.1 | A WishBone input module which generates dictionaries build out of words randomly || | | | chosen from a provided wordlist. || | amqp | 0.1 | A Wishbone AMQP input module. || | gearman | 0.1 | A Wishbone input module which consumes jobs from a Gearmand server. || | generator | 0.1 | A WishBone IO module which generates random data for testing purposes. || | namedpipe | 0.1 | A Wishbone IO module which accepts external input from a named pipe. || | tcp | 0.1 | A Wishbone input module which listens on a TCP socket. || | udp | 0.1 | A Wishbone module which handles UDP input. || | uds | 0.1 | A Wishbone input module which listens on a unix domain socket. || | mongodb | 0.1 | A Wishbone output module to write data in MongoDB. || | | | || wishbone.output | amqp | 0.1 | A Wishbone AMQP output module. || | tcp | 0.1 | A Wishbone IO module which writes data to a TCP socket. || | uds | 0.1 | A Wishbone IO module which writes data to a Unix domain socket. || | | | || wishbone.function | skeleton | 0.1 | A bare minimum Wishbone function module. || | msgpack | 0.1 | A Wishbone which de/serializes data into or from msgpack format. || | snappy | 0.1 | A Wishbone module which compresses or decompresses data using Snappy. || | json | 0.1 | A Wishbone module which converts and validates JSON. || | waitseconds | 0.1 | An output module which takes x seconds to finish the <consume> function. || | | | |+---------------------------+-------------------+-----------+------------------------------------------------------------------------------------------+

show

Displays the docstring of the requested module.

[smetj@indigo ~]$ wishbone show wishbone.builtin.flow.fanout

**A builtin Wishbone module which duplicates incoming events to allconnected queues.**

Create a "1 to n" relationship with queues. Events arriving in inboxare then copied to each queue connected to this module. Keep in mindthat the outbox queue is never used.

When clone is True, each incoming event is duplicated for each outgoingqueue. This might be usefull if you require to change the content of theevents later down the pipeline. Otherwise references are used which meanschanging the event will change it everywhere in the current Wishboneframework.



Parameters:


clone(bool): When True actually makes a copy instead of passinga reference.

Queues:


1.7 Patterns and best practices

This section discusses some common usage patterns and best practices. Although most if not all are strictly spokennot required, they might be helpful in building efficient Wishbone solutions.

1.7.1 Event headers

Write data to headers

In its bare minimum, an event has following layout:


Whenever a module writes data into the header section of the event, it should store this under the <self.name> key,which is unique anyway within a router instance.

The <self.name> value of a module is determined when registering the module usingwishbone.router.Default.register().

So when registering a module like this:

router = Default(interval=1)router.register(STDOUT, "on_screen")

Then any information this module instance wants to write into the header should look like:

{ "header":{"on_screen":{"one":1, "two":2}}, "data": object }

Retrieving data from headers

When a module requires specific information from the header which has been stored by another it should be possibleto initiate the module using a parameter named “key”. You should not hard code the name of the header key into themodule because if someone registers a module with another name, your code will not work anymore.

Consider following example module:

class Output(Actor):def __init__(self, name, key=None):

Actor.__init__(self, name)self.name=nameif key == None:

self.key=self.nameelse:

1.7. Patterns and best practices 23


self.key=key

def consume(self, event):print event["header"][self.key]["one"]

1.7.2 Writing output modules

Starting state

An output module should always start with the inbox queue in a wishbone.tools.WishboneQueue.putLock()state. The moment the module detects it is capable of sending incoming events to the outside world, it should unlockthe inbox queue using wishbone.tools.WishboneQueue.putUnlock().

Handle failed and successful events

Output modules are responsible to deliver messages to the outside world. Preferably we want this to be done asreliably. When submitting events to the outside world (aka outside a Wishbone process.) fails or succeeds we mightrequire a specific strategy to deal with that specific situation.

A possible strategy is have 2 additional queues:

• successful

• failed

As you might guess, events which have been submitted successfully to the outside world are then submitted to thesuccessful queue while the events which failed to go out to the failed queue.

It is up the user to connect these queues on their turn to another destination in order come to the desired strategy.

Some practical examples:

• After submitting an event successfully over TCP to the outside world is is submitted to the successful queue.This queue is on its turn connected to the AMQP acknowledge queue to ascertain it is acknowledged fromAMQP.

• After submitting an event over TCP failed, the event is written to the failed queue from where it is forwarded toanother module which writes the event to disk.

Whenever this pattern is not used, the expected behavior should be:

• Successfully submitted events are discarded

• Unsuccessfully submitted events should be send back to the inbox queue usingwishbone.tools.WishboneQueue.rescue().

Retrying and monitors

When possible an output module should have a “monitor” thread running endlessly in a separate greenthread trying tocreate a valid connection object to the outside service.

This monitor process should be blocked until wishbone.Actor.consume() fails to submit an event via theconnection object.

During the time the monitor process is retrying to create a valid connection object, it should block the inbox queueusing wishbone.tools.WishboneQueue.putLock() since it makes no sense to allow events to come intothe module since they can’t be delivered to the outside world anyway.


CHAPTER 2

Indices and tables

• genindex

• search

25


26 Chapter 2. Indices and tables

Index

AActor (class in wishbone), 7alert() (wishbone.tools.QLogging method), 10

Bblock() (wishbone.Actor method), 7block() (wishbone.router.Default method), 12

Cclear() (wishbone.tools.WishboneQueue method), 9connect() (wishbone.router.Default method), 12consume() (wishbone.Actor method), 7createQueue() (wishbone.Actor method), 7crit() (wishbone.tools.QLogging method), 10critical() (wishbone.tools.QLogging method), 10

Ddebug() (wishbone.tools.QLogging method), 10Default (class in wishbone.router), 11deleteQueue() (wishbone.Actor method), 7disableConsuming() (wishbone.Actor method), 8doRescue() (wishbone.router.Default method), 12dump() (wishbone.tools.QueuePool method), 8dump() (wishbone.tools.WishboneQueue method), 9

Eemerg() (wishbone.tools.QLogging method), 10emergency() (wishbone.tools.QLogging method), 10empty() (wishbone.tools.WishboneQueue method), 9enableConsuming() (wishbone.Actor method), 8err() (wishbone.tools.QLogging method), 10error() (wishbone.tools.QLogging method), 10

FFanout (class in wishbone.module), 16Funnel (class in wishbone.module), 16

Gget() (wishbone.tools.WishboneQueue method), 9getLock() (wishbone.tools.WishboneQueue method), 9

getQueueInstances() (wishbone.tools.QueuePoolmethod), 8

getUnlock() (wishbone.tools.WishboneQueue method), 9Graphite (class in wishbone.module), 19

HHeader (class in wishbone.module), 18HumanLogFormatter (class in wishbone.module), 18

Iinfo() (wishbone.tools.QLogging method), 10informational() (wishbone.tools.QLogging method), 10isLocked() (wishbone.tools.WishboneQueue method), 9

LlistQueues() (wishbone.tools.QueuePool method), 8lock() (wishbone.tools.WishboneQueue method), 9LockBuffer (class in wishbone.module), 17LogLevelFilter (class in wishbone.module), 19loop() (wishbone.Actor method), 8loop() (wishbone.router.Default method), 12

MMeasure (class in wishbone.tools), 13messagesLeft() (wishbone.tools.QueuePool method), 8moduleMonitor() (wishbone.router.Default method), 12

Nnotice() (wishbone.tools.QLogging method), 10Null (class in wishbone.module), 15

PputEvent() (wishbone.Actor method), 8putLock() (wishbone.tools.WishboneQueue method), 9putUnlock() (wishbone.tools.WishboneQueue method), 9

QQLogging (class in wishbone.tools), 9QueuePool (class in wishbone.tools), 8

27


Rregister() (wishbone.router.Default method), 12registerConsumer() (wishbone.Actor method), 8registerLogModule() (wishbone.router.Default method),

12registerMetricModule() (wishbone.router.Default

method), 13rescue() (wishbone.tools.WishboneQueue method), 9RoundRobin (class in wishbone.module), 16runTime() (wishbone.tools.Measure class method), 13

Sshutdown() (wishbone.Actor method), 8shutdown() (wishbone.tools.QueuePool method), 8size() (wishbone.tools.WishboneQueue method), 9start() (wishbone.Actor method), 8start() (wishbone.router.Default method), 13stats() (wishbone.tools.WishboneQueue method), 9STDOUT (class in wishbone.module), 15stop() (wishbone.Actor method), 8stop() (wishbone.router.Default method), 13Syslog (class in wishbone.module), 15

TTestEvent (class in wishbone.module), 14throttleMonitor() (wishbone.router.Default method), 13TippingBucket (class in wishbone.module), 17

Uunlock() (wishbone.tools.WishboneQueue method), 9

WwaitUntilData() (wishbone.tools.WishboneQueue

method), 9waitUntilFreePlace() (wishbone.tools.WishboneQueue

method), 9waitUntilGetAllowed() (wishbone.tools.WishboneQueue

method), 9waitUntilPutAllowed() (wishbone.tools.WishboneQueue

method), 9warn() (wishbone.tools.QLogging method), 10warning() (wishbone.tools.QLogging method), 10WishboneQueue (class in wishbone.tools), 8

28 Index

wishbone documentation - media.readthedocs.org · wishbone.module.stdoutmodule instances which...

Documents