Global States

Topics

Usefulness of global state Difficulties in determining snapshot Determining global state Examples

Readings

Van Steen and Tanenbaum: 5.3 Coulouris: 10.5“Distributed Snapshots: Determining Global

States of Distributed Systems”, K. Mani Chandy and Leslie Lamport, ACM Transactions on Computer Systems, vol 3, no 1, February 1985.

Motivating Example

Assume that we have processes interacting in a client-server mode.

Client makes request to server Waits for response While waiting for response, client simply

blocks; does not answer to requests from other nodes.

A

B

C

This is a deadlock

Motivating Example

Assume that you have a centralized server.

It queries each node Each node responds with a list of requests

that are pending (requests for which a response has not been sent).

Centralized server can then build a “waits-for” graph. Cycle in graph implies deadlock

This is an example where being able to determine the global state is useful.

Other Motivating Examples

Distributed Debugging Distributed Garbage Collection

An object is considered to be garbage if there are no longer any references to it anywhere in the distributed system.

p2p1

message

garbage object

objectreference

Global State Occasionally it is useful to know the global state

of a distributed system. The global state of a distributed system consists

of the local state of each process, together with the messages that are currently in transit, that is, that have been sent but not delivered.

Exactly what a local state of a process is, depends on our interest. In a distributed database system, the local state may

consist of only those records that form part of the database and excludes temporary records used in computation.

Global State in a Single Process Application

Assume that we have the following code:x = y = z =0;x = z+1;y = x;z = y

The state can be represented as the values of x,y,z. The sequence of states for the execution of the above

program is the following:000, 100, 110, 111

Deterministic Computation At any point in computation there is at most one event

that can happen next. There is only one sequence of states for the same

input.

Global State for a Multiple Process Application

Say in process P1 you have a code segment as follows:

1.1 x = 5;1.2 y = 10*x;1.3 send(y,P2);

1.4 z = 8;

Say in process P2 you have a code segment as follows:

2.1 a=8;2.2 b=10*a;2.3 rcv(c,P1);

2.4 b = b+c;

Global State for a Multiple Process Application

Non-Deterministic Computation At any point in computation there can be

more than one event that can happen next There is more than one possible

sequence of states for the same input. Assume that a state is represented as

(x,y,z,a,b,c). Observation: In this example, regardless

of the sequence, eventually all sequences reach these global states (5,50,0,8,80,0), (5,50,0,8,80,50)

Global State for Multiple Process Application

0,0,0,0,0,0

5,0,0,0,0,0

5,0,0,8,0,0

5,0,0,8,80,0

0,0,0,8,0,0

5,0,0,8,0,0

5,50,0,8,0,0

5,50,0,8,80,50

5,50,8,8,80,50

5,50,8,8,80,130

5,50,0,8,80,130

5,50,8,8,80,130

5,50,0,8,80,0

A Non-Deterministic Computation

Each of the paths represent a possible sequence of events that could have occurred during execution. This assumes that the same inputs were

applied. Different paths occur for different reasons that

include machines that have different loads at different runs, the network load at different runs, etc;

The different paths correspond to a different sequence of global states that the distributed program goes through.

Viewing Global States

A process in a distributed system has only a limited view of the system, which consists of its local state and the messages it has sent or received.

Given this, is it possible for processes in the system to determine a global system state?

Instantaneous recording not possible No global clock: Distributed recording of local

states cannot be synchronized based on time. Random network delays; What messages are

in transit?

Viewing Global States

The main difficulty is that global state computation requires computation and communication, and by the time such effort is completed, the global state has changed.

A snapshot of the system is a single configuration of the system. It is not necessarily a global state that has actually occurred, but one that could have occurred during the computation.

A Naive Snapshot Algorithm

Processes record their state at any arbitrary point

A designated process collects these states

Simple, but not correct.

Example of Naive Snapshot Algorithm in Use

P records its state

P Q

X = 5

Example of Naive Snapshot Algorithm in Use

P Q

m

Let m be a message to update a variable Y in Q by incrementing it by the value of X.

Example of Naive Snapshot Algorithm in Use

Q records its state after receiving m but before applying the operation.

P Q

m

Example of Naive Snapshot Algorithm in Use

Process P records its state before it actually sends m

Process Q records its state after it has received m.

This is bad. The result is a global state that has

recorded the receive event but no send event.

Distributed Snapshot Represents a state in which the

distributed system might have been in. A distributed snapshot should reflect a

consistent state. If we have recorded that a process P has

received a message from a process Q, then we should have also recorded that process Q had actually send that message.

Otherwise, a snapshot will contain the recording of messages that have been received but never sent.

The reverse condition (Q has sent a message that P has not received) is allowed.

Distributed Snapshot

The notion of a global state can be graphically represented by what is called a cut (example on the next slide).

A cut represents the last event that has been recorded for each process.

Consistent State

a) A consistent cutb) An inconsistent cut

Assumptions for Algorithm

The algorithm to determine global states records process states locally and the states are collected by a designated server.

No failures in channels and processes – exactly once delivery.

Unidirectional channel and FIFO ordered message delivery.

Always a path between two processes. Global snapshot initiation at any process at

any time. No process activity halt during snapshot.

Features of Algorithm

Does not promise us to give us exactly what is there i.e., an actual state

However does give us a consistent state

Algorithm P starts by recording its own local state P subsequently sends a marker along each of its

outgoing channels When Q receives a marker through channel C, its

action depends on whether it had already recorded its local state: Not yet recorded: It records its local state and sends the

marker along each of its outgoing channels Already recorded: The marker on C indicates that the

channel’s state should be recorded: all messages received before this marker and since the last time Q records its own state.

Q is finished when it has received a marker along each of its incoming channels.

Algorithm

a) Organization of a process and channels for a distributed snapshot

Algorithm

b) Process Q receives a marker for the first time and records its local state

c) Q records all incoming messagesd) Q receives a marker for its incoming channel and finishes

recording the state of the incoming channel

Algorithm

How do we know when we are done? If a process Q receives the marker

requesting a snapshot for the first time, it considers the process that sent the marker as its predecessor

When Q completes its part of the snapshot, it sends its predecessor a DONE message.

By recursion, when the initiator of the distributed snapshot has received a DONE message from all its successors, it knows that the snapshot has been completely taken.

Example

Assume two process p1 and p2 connected by two unidirectional channels, c1 and c2.

The two processes trade in widgets. Process p1 sends orders for widgets over c2 to p2,

enclosing payment at the rate of $10 per widget. Process p2 sends widgets along channel c1 to p1. The processes have the initial states shown in the

figure on the next slide. It is assumed that p2 has already received an order for 5 widgets, which it will shortly dispatch to p1.

Example

p1 p2c2

c1

account widgets

$1000 (none)

account widgets

$50 2000

This is the initial state when the algorithm was initiated;Before initiation, the process p2 had received an order for 5 widgets;Thus account has $50

Example

Upon request for global snapshot, process p1 records its state in the actual global state, S0, when p1’s state is <$1000,0>.

p1 p2(empty)<$1000, 0> <$50, 2000>

(empty)

c2

c1

1. Actual Global State S0

Example

Process p1 then emits a marker message over its outgoing channel c2 before it sends the next application-level message: (Order 10, $100) over channel c2.

The system enters actual global state S1.

2. Actual Global state S1

p1 p2(Order 10, $100), M<$900, 0> <$50, 2000>

(empty)

c2

c1

Example

Before p2 receives the marker, it emits an application message (5 widgets) over c1 in response to p1’s previous order, yielding a new actual global state S2.

3. Actual Global State S2 p1 p2(Order 10, $100), M<$900, 0> <$50, 1995>

(five widgets)

c2

c1

Example

Now process p1 receives p2’s message (5 widgets) and p2 receives the marker. p2 records its state as <$50,1995> and that of channel c2 as the empty sequence.

p2 sends a marker message over c1. When process p1 receives p2’s marker

message, it records the state of c1 which is the single message (5 widgets) that it received after it first recorded its state.

Example

The final recorded state is p1: <$1000,0>; p2:<$50,1995>;c1:<(five widgets)>; c2:<>

4. Actual Global state S3

p1 p2(Order 10, $100)<$900, 5> <$50, 1995>

(empty)

c2

c1

Example

The sequence of actual global states was the following: S0: p1: <$1000,0>; p2:<$50,2000>;c1:<>;

c2:<>

S1: p1: <$900,0>; p2:<$50,2000>; c1:<>; c2:<(order 10, $100)>;

S2: p1: <$900,0>; p2:<$50,1995>; c1:<five widgets>; c2:<(order 10, $100)>;

S3: p1: <$900,5>; p2:<$50,1995>; c1:<empty>;c2:<(order 10, $100)>;

Example

A possible sequence of global states is the following: S0: p1: <$1000,0>; p2:<$50,2000>;c1:<>;

c2:<>

S1’: p1: <$1000,0>; p2:<$50,1995>;c1:<(five widgets)>; c2:<>

Recorded State: p1: <$1000,0>; p2:<$50,1995>;c1:<(five

widgets)>; c2:<>

Example

S0

S1

S2

S3

S1’

………

Example

The recorded state is not the same as one of the actual states.

The recorded state is one that was possible in one of the possible sequences of global states.

Why is this useful? What can we do with this? To illustrate we will look at another

example.

Example

What if p1 initiates the snapshot algorithm after it issues a request for the 10 widgets and before it receives the 5 widgets?

This would imply that p2 gets the order for the 10 widgets before the marker. This causes p2 to update its account and number of widgets to $150 and 1985 respectively. p2 will record this state when it receives the marker. p2 sends the 10 widgets.

The state of c1 is <(5 widgets),(10 widgets)> and c2 is marked as empty.

Example 2Say in process P1 you

have a code segment as follows:

1.1 a = 5;1.2 recv(b, P2);

1.3 c = a+ b;1.4 send(c,P2);

Say in process P2 you have a code segment as follows:

2.1 x = 5;2.2 recv(y,P1);

2.3 z = x+y;2.4 send(z, P1);

State includes values of a,b,c,x,y,z and for each process records what that process is waiting (if there is anything).

Example 2

p1 p2c2

c1

a=5b=0c=0

This is the state when the algorithm was initiated;1.1 has been executed but 2.1 has not; marker is sent out after 1.1has been executed but before trying to execute 1.2Channels are empty

x=0y=0z=0

S0

Example 2

p1 p2c2

c1

a=5b=0c=0

2.1 has been executed before it receives the marker from p1

Upon receiving the marker it records that c2 is empty , thatx=5,y=0,z=0 and that it is waiting for a message from p1

It sends out a marker on c1

x=5y=0z=0

S1

p2 is waiting for message from p1

Example 2

p1 p2c2

c1

a=5b=0c=0

p1 receives a marker on c1

It records the state of its incoming channel as empty.The final state recorded is this:a=5,b=0,c=0, x=5,y=0,z=0; p2 is waiting for a message from p1 and channels are empty.

x=5y=0z=0

S2

p2 is waiting for message from p1

Example 2

So far the deadlock has not been detected. Let’s say that after P1 starts waiting for a message

from P2 that it initiates the global snapshot algorithm again.

P1 records its state as a=5,b=0,c=0 and that it is waiting for a message from P2

P1 sends the marker to P2 . P2 records its state as x=5,y=0,z=0, incoming

channel is empty and it is waiting for a message from P1

P2 sends a marker on c1; When P1 receives the marker it records the state of channel c1 as empty

Example 2

At this point the global state recorded includes the following: a=5,b=0,c=0 x=5,y=0,z=0 Both channels are empty P1 is waiting for a message from P2 and P2 is

waiting for a message from P1

This recorded state has enough information to determine that there is a deadlock.

If the algorithm is initiated again the global state still shows a deadlock.

Example 3

p0 p1

p2

M

B

C

A

M

B

A

C

M

Send A

Send B

Record State

Send M

Recv B

Recv A

Recv C

Recv M, Record State, Channel (2,1)empty

Recv M, Record State, Channel (0,1)empty, Send M

Recv M, Record State, Channel (0,1)A

Send C

Stable Properties

The global state computed using the algorithm described can be used for detecting stable properties, e.g.,Distributed deadlock detectionDistributed garbage collection

A stable property is a property that once it holds true in a state Si then it holds true in state Sj where j > i

Stable Properties

Let’s look at deadlock detection. Assume each process has a list waits-for which records the list of processes that the process is waiting for a message from.

The global state includes values of the waits-for list from each process.

Let’s assume two processes p1 and p2. Let p1.waits-for = {p2} and p2.waits-for = {p1}.

This represents a deadlock. If there is a deadlock in state Si then there is a deadlock in Sj (where j > i).

Stable PropertiesS0

S1

S2

S3

S1’

S2’

If deadlock has happened in S3 then it happens in subsequent states.

Summary

No shared memory in distributed systems

Very difficult to determine exactly what a system’s current state is.

Determining the global state of a distributed system can be done by synchronizing all processes so that each collects its own local state along with the messages that are currently in transit.