ISIS2 PROCESS GROUPSKen Birman

1

Cornell University

2

Isis2 Process Groups The most important concept in the Isis2

system

A group Has a name (like “myGroup”) Has members (while active) Interacts with those members by delivering

events to them through upcalls to pre-registered methods

The “state” (the data associated with the group) lives within the application programs you build, not in Isis2

3

When a program starts, it isn’t a member of any groups.

Calling IsisSystem.Start() activates Isis2. Then the program would normally join or create groups

Key Concept: Join or Create

4

Group g = new Group(“myGroup”);

Creates an Isis2 object that will be your interface to the group.

The new Group() does not join or create the group. The new Group object isn’t active yet. That

gives you time to attach handlers to it before you join or create it

As soon as the join or create occurs, events might start to be received and reported to the application

5

Handlers These are the important types of event handlers

ViewHander: Used to sense membership changes Message Handlers: Used to process incoming point

to point (unicast) messages and incoming multicasts Checkpoint creation: Used to make a checkpoint of

group state, which can be saved into a log file (for persistency) or sent to a joining member (to initialize it)

After registering handlers, you call g.Join() or g.Create() to bind this instance to others that may be active

6

The View object reports the status of the group. It contains• v.members[]: An array of member Addresses• v.joiners[], v.leavers[]: Members that just joined or left• v.viewid: An integer counter, incremented for each new view• v.gname: name of the associated group• v.offset: when a member joins, it can provide a value (the

“offset”) for use in sending it initial state. The idea is to allow you to “top off” the state if some part of the state was pre-transferred before the join. E.g. suppose the group state is an append-only log. You could pre-transfer 10Mb and have the offset be 10Mb. At join time only the tail of the log needs to be transferred. The offset could also represent the number of updates that have been applied to a database.

Key Concept: View

p

q

r

s

t

Time: 0 10 20 30 40 50 60 70

7

Example: A view handler

This view handler will be called each time membership changes, including when the group is created or initially joined, and when the program leaves the group or “terminates” it

The example updates the title on the console window in Windows

g.ViewHandlers += (ViewHandler)delegate(View v){ Console.Title = "[" + g.gname + "]: " + Isis.Isis.my_address + ": ViewId=" + v.viewid + ", MyRank=" + v.GetMyRank();};

8

Within Isis2, each program has an associated Address. The Address includes the IPv4 address of the host on which it is running, port numbers for contacting it, and a process id.Isis2 groups also have addresses.You can print an Address in short or long-format. Call a.ToStringVerboseFormat() for long format.

Key Concept: Address

9

myAddress, my rank After IsisSystem.Start()

IsisSystem.GetMyAddress() returns your own address.

Used to look yourself up in a View to learn your “rank” within that view. Use v.GetMyRank()

• If a process has rank r in view v, this just means that v.members[r].Equals(myAddress)

• Note: Compare Addresses with .Equals(), not ==

10

Address of some other process? You can obtain this from

The view of a group you both belong to The process could write it down for you in a

file If you know the machine on which the

process will run, you can use new Address(hostIPv4, pid);

In RunAsWorker you can use 0 for the pid. We’ll learn about this later.

11

Uses of the Address? You can send a message directly to a

process

You can query a process

You can tell Isis2 about a failure, if the process belongs to a group to which you also belong.

12

The central property that makes Isis2 awesome!

Key Concept: Consistency

13

Consistency One aspect of consistency is that each

member of a group sees the same membership views, in the identical order This is useful for coordinating actions Notice that since each has its own rank, it is

easy to subdivide a data set using the rank v.members.Length: the total number of

members. Call it N. v.GetMyRank(), v.GetRankOf(a): gets rank

values as integers in the range [0 .. (N-1)]

14

Simple thought problem A group has view v. The group

maintains an array of data with R rows. If we want each member to do roughly equal work, how would you do that?Name Score Assign to member

rank…Sally 27 0Sam 14 1Alice 17 2Doug 33 0Mary 19 1Mahesh 43 2Helen 41 0Harry 29 1

If the group had N members (3 in this

example) the caller will need to merge the 3

subreplies to obtain the answer to the query.

Thus work occurs both in the group and in the

caller. The caller aggregates the answers

Name Score Assign to member rank…

Sally 27 0Sam 14 1Alice 17 2Doug 33 0Mary 19 1Mahesh 43 2Helen 41 0Harry 29 1

1 20

15

Why does this work? Key insight: when they receive the

query, all members have the same current view for the group A virtual synchrony guarantee … and all members receive the Query.

So if we divide the work this way, each member knows what its share of the work is!

What about failures?

p

q

r

s

t

Time: 0 10 20 30 40 50 60 70

16

Overcome failureswith redundancy

1. How do we generate redundant replies?2. Can the query caller “tell” if it is

missing part of the answer? Not without help: When Query is issued, we

don’t know what the membership will be at the time of delivery

… so have the reply include “This is reply k of R”. The caller can build an array of R replies, and if it got one copy of each, the set of replies is complete.

p

q

r

s

t

Time: 0 10 20 30 40 50 60 70

17

Redundant replies: Option A Have each member handle two sets of

data g.Reply(# members, myRank, primary,

secondary)Name

Score

Primary

Secondary

Sally 27 0 1Sam 14 1 2Alice 17 2 0Doug 33 0 1Mary 19 1 2Mahesh

43 2 0

Helen 41 0 1Harry 29 1 2

If the caller is told how many members were in the group, and which parts of the answer this reply includes, it can merge the results and discard duplicates.

It will either have a full set of results, or if any are missing, we

know we had (unlikely) two “correlated” failures.

One Reply but it contains two

answers

18

Redundant replies: Option B In a large enough group, we can have 3 members as

primary reply senders, 3 more as backups The primary and backup members send Replies The non-participants call g.NullReply(); Caller will get 6 responses, each marked “1 of 3” etc,

with each duplicated. Keeps just one

Either option is perfectly reasonable. With option B we can also spread the work around if a big group will receive lots of concurrent queries Later when we learn about sharded data, the DHT, and

subset queries, we’ll discover an option C…

19

Message handler

The actions taken when a message arrives are determined by the code in the message handler

This handler gets a query that passes in a string, and then replies by telling the caller how many members were in the group, its own rank, and sending back its share of the Query answer.

myGroup.Handlers[2] += (Action<string>)delegate(string name){ . . . Compute my share of the reply . . . g.Reply(GetView().members.Length, GetMyRank(), myShare);};

20

Message handler Notice that each handler

Has an associated request code. A small integer in the range [0..127]. We use request codes to simplify the application logic.

Has a “type signature”: the arguments to the upcall and their types. For example, string, int, etc. You can specify application-defined types too, but if so

must register them with Isis2 before using them An incoming message must match a handler with

the proper request code, and must exactly match the given types. The type match is highly restrictive.

21

In C#, an Action is a special type that denotes a method. It can specify the types of arguments, as in this example: Action<string>

A delegate is a kind of anonymous method. It should match the Action type signature. A delegate doesn’t need a name (C# makes one up). Example: delegate(string name)

Key Concept: Action. Delegate.

22

Isis2 needs to be able to put object into messages and extract them from messagesYou can supply types of your own but must register them first.

Key Concept: Tell Isis2 about a type

23

Example: A user-defined type Each type is given a

small integer “type code” [0..127]

This example is automatically marshalled by Isis2: all public data fields will be transmitted

AutoMarshalled types must have a public constructor that takes no arguments.

[AutoMarshalled] public class myData { public int d;

public myData() { }

public myData(int v) { d = v; }

public override string ToString() { return bprint(d); } }

Msg.RegisterType(typeof(myData), 0);

24

More about user-defined types You can also define classes in which only

some fields will be marshalled Provide a method ToBArray() that Isis2 can call,

and that will convert the object into a byte-array representation

Provide a constructor myData(byte[] ba) to convert a byte-array back into the object

You can use C# serialization. The Isis2 Msg class has methods that will convert to and from byte arrays this quite efficiently. They handle endian ordering.

25

When a member joins a group, we use “state transfer” to initialize it• Your code will decide what this state

contains• You register a “checkpoint maker” and

“checkpoint loader” methods.

Key Concept: state transfer

26

Example: Checkpoint methods

This code sends two messages, one with an integer in it, one with a double. The loader methods accept the incoming checkpoint data.

The example is very inefficient. A single call to SendChkpt could have many data objects in it (including complex objects such as arrays, lists, etc)

g.MakeChkpt += (Isis.ChkptMaker)delegate(View nv) { g.SendChkpt(istuff); g.SendChkpt(dstuff); g.EndOfChkpt(); }; g.LoadChkpt += (loadichkpt)delegate(int what) { istuff = what; }; g.LoadChkpt += (loaddchkpt)delegate(double what) { dstuff = what; };

27

Fancier checkpointing ideas Consider pre-transferring as much state as you can

prior to the join. Then use the offset argument to g.Join(); it will show up as the v.offset value in the checkpoint maker, which can send just the delta!

It is ok to call SendChkpt() more than once, but we do that mostly if the data will be large. Normally, one call is enough.

Isis2 lets you specify a “checkpoint choser” method. If you do, you can control which existing member sends state for a particular joining member.

28

Out of Band Transfer A very good way to pre-transfer data is this:

Call IsisSystem.Start(); Join a helper group that has no state in it. Using the Isis.Client() method, do an RPC request to

some member of the “data-heavy” group you plan to join. Ask it to do an “out of band” transfer to the joining member. (We’ll discuss the OOB layer soon)

Once this data has arrived, dispose of the Client handle and do a g.Join(offset) for the data heavy group, specifying an offset that tells how much data you already have.

Only transfer this remaining delta during the join itself.

29

Only legal after g.Join() has succeeded.

g.P2PSend(dest, rcode, args): sends a point-to-point message (FIFO in this case) to a single specified member. The request code and args must match some handler within the group.

Key Concept: Unicast

30

Only legal after g.Join() has succeeded.

g.P2PQuery(dest, rcode, args, EOL, p2presults): sends a point-to-point query (FIFO in this case) to a single specified member. It calls g.Reply(results), and the result value(s) are returned in the “p2presults” list.

Key Concept: Unicast Query

31

A special object used by Isis2 as a separatorSeparates the list of arguments being sent from the results: end of list marker.

Key Concept: EOLMarker

32

Only legal after g.Join() has succeeded.

g.Send(rcode, args): sends a multicast (FIFO in this case) to all members. The request code and args must match some handler within the group.

Key Concept: Multicast

33

Only legal after g.Join() has succeeded.

Nr = g.Query(N, rcode, args, EOL, qresults): sends a multicast (FIFO in this case) to all members. N of them (you can specify ALL, or any number 0) call g.Reply(results), and the values collected are returned in the lists of qresults. Nr gives the actual number of replies that came back.

Key Concept: Query

34

Every query must have an associated timeout, but you can let Isis2 define a default valueIf you specify it, you provide the amount of time (in milliseconds) to wait for a reply, and the action to take if the timer expires: new Isis.Timeout(2000, Isis.Timeout.TO_NULLREPLY)Possible actions are:• TO_ABORT: On timeout, throw an “abort reply” exception.• TO_FAILURE: On timeout, declare the slow responder to

have failed.• TO_NULLREPLY: Ignore slow responders

Key Concept: Timeout

35

Isis2 considers a process to have failed if:• The process doesn’t respond to messages for

ISIS_TIMEOUT seconds (default: 45)• You tell Isis2 that the process has failed by

calling Isis.ProcessFailed() (this is legal only if the failed process is a member of some group to which your process also belongs).

• The process terminated gracefully, ideally by calling Isis.Shutdown().

Key Concept: Failure

36

Modern systems use threads.Isis2 uses them extensively

Key Concept: Threads

37

Threads in Isis2

C# makes it easy to use threads, and Isis2 definitely does… Normally will have 8 or so system threads (they spend

most of their time sleeping) … plus 2 event delivery threads per group (one for

views and multicast messages, the other for unicast messages). Implication: if the thread that delivers a multicast blocks for some reason, no more messages can be delivered until it wakes up!

… a few more are created from time to time as it runs. The system runs best if you dedicate 1 or 2 cores

to it, but no more than 2.

38

If the main thread exits, Isis2 automatically shuts down.

What should you do if your main thread has no additional work to do?• You call IsisSystem.WaitForever(); • This blocks until something calls

IsisSystem.Shutdown();

Key Concept: WaitForever()

39

For most purposes, processes join the groups that they participate in or useBut you can also be a “client” of a group

Key Concept: Client of a group

40

Client c = new Client(“group name”);

A Client is a process “bound” to some member of the group, picked at random. Allows the process to issue unicast (point-

to-point) requests to the member (that member could relay them within the group if the logic is coded appropriately).

The member must call g.AllowClientRequests(rcode) for each request code that can accept external client-initiated requests this way.

A client should dispose of the handle (c.Dispose()) if it wishes to Join the group later.

41

Isis2 has two of them

Key Concept: System groups

42

ORACLE and ISISMEMBERS The system itself creates two groups.

You can’t access them directly. ORACLE: a small group that manages group

membership on behalf of the system. It runs in the first processes you launch and has ISIS_ORACLESIZE members (typically 3). The load on the ORACLE is normally very light.

ISISMEMBERS: A group containing all processes using this instance of Isis2.

43

Our first complete Isis program Let’s switch to Visual Studio now and

develop a complete Isis2 program that maintains a list of (key,value) tuples (strings and doubles), updates them, includes state transfer, and queries them.

For queries, we’ll have each of the N members do 1/N’th of the work.