Dive into SObjectizer-5.5

SObjectizer Team, Jan 2016

Seventh Part: Message Limits(at v.5.5.15)

This is the next part of the series of presentations with deep introduction into features of SObjectizer-5.5.

This part is dedicated to message limits feature.

This feature is intended to use in protection of agents from overloading.

SObjectizer Team, Jan 2016

Introduction

SObjectizer Team, Jan 2016

The main way of interaction between agents in SObjectizer is asynchronous messages.

It makes very easy to use “fire-and-forget” scheme:

● an agent A finishes its own part of work, sends the result as a message to agent B and starts new processing;

● agent B receives a message, processes it and sends a new message to agent C;

● agent C receives a message, processes it and sends…

This is very simple scheme but...SObjectizer Team, Jan 2016

...what if some agent in this chain processes messages much slower than the other agents which send new messages to it?

This problem is known as overloading of agent.

SObjectizer Team, Jan 2016

Overloading leads to growth of event queues.

Event queues eat more and more memory.

Growing memory usage slows down the application.

Lower speed leads to faster growth of event queues…

As a result of overloading the application is degraded and should be killed in most cases.

SObjectizer Team, Jan 2016

There is no simple solution for overloading problem because a good overload control scheme for a particular application-specific agent will be very domain-specific and, probably, can’t be reused in another project.

So, there is no ready-to-use just out-of-box tool or mechanism in SObjectizer which could be seen as ‘one size fits all’ solution for overloading problem.

SObjectizer Team, Jan 2016

But there is a mechanism which can protect agents from overloading in simple situations.

This mechanism is known as message limits.

SObjectizer Team, Jan 2016

Message limits is an optional feature which can be turned on for individual agent.

If message limits is turned on for an agent SObjectizer will count instances of messages in agent’s event queue.

When number of messages of particular type exceeds limit for that type SObjectizer performs an overlimit reaction which is specified by user.

SObjectizer Team, Jan 2016

Types Of Overlimit Reactions

SObjectizer Team, Jan 2016

There are several overlimit reactions which can be performed by SObjectizer:

● dropping new messages;● killing the whole application by std::abort();● redirecting new messages to another mbox;● transforming new messages to messages of different

types (with possibility to redirect them to different mboxes).

SObjectizer Team, Jan 2016

Message dropping is the simplest form of overlimit reactions.

It means that a new message of type T which exceeds limit for messages of type T will be ignored and will not be pushed to event queue of the subscriber.

This type of reaction can be useful if there is an intensive flow of messages and losing of several messages is not a problem. For example: ignored message will be repeated by sender after some timeout.

SObjectizer Team, Jan 2016

Another simple overlimit reaction is killing the whole application by std::abort().

This overlimit reaction can be used in scenarios where the application must guarantee some throughput but can’t do that.

For example: an agent should process messages quite quickly. But there is a bug in the agent’s implementation and sometimes it fell into infinite loop. The only solution there is killing the app and restart it.

SObjectizer Team, Jan 2016

The next overlimit reaction is redirection of a new message to another mbox.

This type of reaction could be used in different schemes:

● a very simple load balancing with agents A1, A2, ..., An. Agent A1 redirects messages to A2. A2 redirects to A3 and so on;

● different kind of processing by different subscribers. There could be agent N which does a normal processing. On overload it redirects messages to agent P which just sends back a reply “system is busy, try later”.

SObjectizer Team, Jan 2016

The last and most complex reaction is transformation with optional redirection.

This type of reaction could be used in implementing complex domain-specific schemes of overload control.

For example, an overloaded agent could transform new messages into signals about overloaded state and sends it to an overload controller. The controller can perform some actions like changing processing params, disconnecting some clients, switching off some logging and so on...

SObjectizer Team, Jan 2016

Maximum Overlimit Reaction Deep

SObjectizer Team, Jan 2016

There is a possibility of redirection loops for overlimit reactions of ‘redirect’ and ‘transform’ types.

For example: agent A1 redirects message to A2, A2 redirects to A3, …, An redirects back to A1.

Without a control of recursion of overlimit reactions infinite loops inside message delivery procedure could arise.

SObjectizer Team, Jan 2016

Because of that SObjectizer limits the number of overlimit reactions which are performed during the delivery of a message instance.

This is called ‘overlimit_deep’ and the current maximum value of overlimit_deep is 16.

This value is hardcoded in the current SObjectizer implementation and can’t be changed in run-time.

SObjectizer Team, Jan 2016

When maximum value of overlimit_deep is reached then a delivery of message is aborted by raising so_5::exception_t with the appropriate error code.

Please note that if it is a delivery via multi-consumer mbox then some subscribers will not receive a message. Even if they do not use message limits at all.

It is because the whole delivery procedure is canceled.

SObjectizer Team, Jan 2016

How To Specify Message Limits?

SObjectizer Team, Jan 2016

This part of presentation contains some examples of message limits definition.

But before going deep into code samples some important rules first...

SObjectizer Team, Jan 2016

If an agent defines message limit for some message type T it must also define message limits for all other message types it processes.

If an agent defines message limit for type T and tries to make a subscription for message of type M which hasn’t defined limit there will be an error.

SObjectizer Team, Jan 2016

Message limits are defined at the moment of agent’s construction and can’t be changed or removed later.

Definition of message limits are passed to the constructor of so_5::agent_t or to coop_t::define_agent() method via so_5::context_t object.

SObjectizer Team, Jan 2016

Definition of message limits with ‘drop message’ overlimit reaction could look like:// For ordinary agents:class request_processor : public so_5::agent_t {public : request_processor( context_t ctx ) : so_5::agent_t( ctx // No more than 20 messages with dropping of extra messages. + limit_then_drop< request >( 20 ) // No more than 1 message with dropping of extra messages. + limit_then_drop< get_status >( 1 ) ) ...};

// For ad-hoc agents:coop.define_agent( coop.make_agent_context() + so_5::agent_t::limit_then_drop< request >( 20 ) + so_5::agent_t::limit_then_drop< get_status >( 1 ) )...

SObjectizer Team, Jan 2016

Definition of message limits with ‘abort the app’ overlimit reaction could look like:// For ordinary agents:class hardware_interface : public so_5::agent_t {public : hardware_interface( context_t ctx ) : so_5::agent_t( ctx // Usually there must be no more than 10 waiting commands. // But if something goes wrong and hardware part doesn't respond // it is better to terminate the whole application. + limit_then_abort< outgoing_apdu_command >( 500 ) ) ...};

// For ad-hoc agents:coop.define_agent( coop.make_agent_context() + so_5::agent_t::limit_then_abort< outgoing_apdu_command >( 500 ) );

SObjectizer Team, Jan 2016

Definition of message limits with ‘redirect’ overlimit reaction could look like:class normal_request_processor : public so_5::agent_t {public : normal_request_processor( context_t ctx, // Message box of agent for handling requests in overload mode. // In this mode requests are not processed but negative response with the appropriate // result code is sent back (and this fact is stored in the application log). so_5::mbox_t overload_mode_processor ) : so_5::agent_t( ctx // We can hold no more that 10 requests in queue. // All extra messages must be redirected to overload-mode processor // for generation of negative replies. + limit_then_redirect< request >( 10, [overload_mode_processor] { return overload_mode_processor; } ) ...};

SObjectizer Team, Jan 2016

Definition of message limits with ‘transform’ overlimit reaction could look like (for the case of a message):class request_processor : public so_5::agent_t {public : normal_request_processor( context_t ctx ) : so_5::agent_t( ctx // We can hold no more that 10 requests in queue. // For all extra messages negative replies must be generated. + limit_then_transform( 10, [](const request & evt) { return make_transformed< negative_reply >( // Mbox for sending reply is got from original request. evt.reply_to(), // All other arguments are passed to negative_reply constructor. error_code::processor_is_busy ); } ) ) {} ...};

SObjectizer Team, Jan 2016

Definition of message limits with ‘transform’ overlimit reaction could look like (for the case of a signal):class long_operation_performer : public so_5::agent_t{public : long_operation_performer( context_t ctx, so_5::mbox_t mbox_for_notifications ) : so_5::agent_t( ctx // If we cannot process previous get_status signal // then we are busy and can tell about this. + limit_then_transform< get_status >( 1, [this, mbox_for_notifications] { // The result of the transformation is another signal and because of that // there is only one argument for make_transform (the target mbox). return make_transformed< status_busy >( notification_mbox ); } ) ) {} ...};

SObjectizer Team, Jan 2016

Some Final Words

SObjectizer Team, Jan 2016

This presentation is just a short introduction in the topic of message limits.

For more detailed information please see the corresponding section of Project’s Wiki.

But it is necessary to remember that message limits is not a fullfledged solution for overload problem...

SObjectizer Team, Jan 2016

Message limits could be seen as a basic building blocks for more complex application- and domain-specific overload control schemes.

Some examples of such schemes which are based on collector-performer agents pairs are represented in several examples in the SObjectizer’s distribution (they are also described in Project’s Wiki):● work_generation;● collector_performer_pair and collector_many_performers;● simple_message_deadline;● prio_work_stealing.

SObjectizer Team, Jan 2016

Additional Information:

Project’s home: http://sourceforge.net/projects/sobjectizer

Documentation: http://sourceforge.net/p/sobjectizer/wiki/

Forum: http://sourceforge.net/p/sobjectizer/discussion/

Google-group: https://groups.google.com/forum/#!forum/sobjectizer

GitHub mirror: https://github.com/masterspline/SObjectizer