distributed operating systems: theoretical...

Department of Computer Science, The University of Texas at Dallas CS 6378: Advanced Operating Systems – Chapter 5: Theoretical Foundations Slide 1/44

Distributed Operating Systems:Theoretical Foundations

Neeraj Mittal

Department of Computer Science

The University of Texas at Dallas

[email protected]

Inherent Limitations

❖ Two Important Characteristics

❖ Absence of Global Clock

❖ Absence of Shared Memory

Ordering of Events

Abstract Clocks

Ordering of Messages

State of a Distributed System

Monitoring a Distributed

System


Two Important Characteristics

■ Absence of Global Clock◆ there is no common notion of time

■ Absence of Shared Memory◆ no process has up-to-date knowledge about the system





Ordering of Events

Abstract Clocks




System


Absence of Global Clock

■ Different processes may have different notions of time

I invented AIDS cure!

Who did it first?

Patent Officer


Earth Mars

Problem: How do we order events on different processes?





Ordering of Events

Abstract Clocks




System




I invented AIDS cure!I invented AIDS cure!

Earth Mars


Who did it first?

Patent Officer


Earth Mars






Ordering of Events

Abstract Clocks




System





Who did it first?

Patent Officer


Earth Mars






Ordering of Events

Abstract Clocks




System


Absence of Shared Memory

■ A process does not know current state of other processes

Bob

Harry

What is Harry doingat present?

0

Problem: How do we obtain a coherent view of the system?





Ordering of Events

Abstract Clocks




System




Bob

Harry

0

Bob

Harry


0






Ordering of Events

Abstract Clocks




System




Bob

Harry


0



Ordering of Events

❖ When is it possible to order

two events?❖ Happened-Before Relation

❖ Concurrent Events

Abstract Clocks




System


When is it possible to order two events?

Three cases:

1. Events executed on the same process:

■ if e and f are events on the same process and e

occurred before f , then e happened-before f

2. Communication events of the same message:■ if e is the send event of a message and f is the receive

event of the same message, then e happened-before f

3. Events related by transitivity:■ if event e happened-before event g and event g

happened-before event f , then e happened-before f


Ordering of Events




Abstract Clocks




System



Three cases:

1. Events executed on the same process:■ if e and f are events on the same process and e







Ordering of Events




Abstract Clocks




System



Three cases:



2. Communication events of the same message:

■ if e is the send event of a message and f is the receiveevent of the same message, then e happened-before f




Ordering of Events




Abstract Clocks




System



Three cases:








Ordering of Events




Abstract Clocks




System



Three cases:





3. Events related by transitivity:

■ if event e happened-before event g and event g



Ordering of Events




Abstract Clocks




System



Three cases:








Ordering of Events




Abstract Clocks




System


Happened-Before Relation

■ Happened-before relation is denoted by →

■ Illustration:

g h i

a b c

d e fP2

P3

P1

◆ Events on the same process:examples: a → b, a → c, d → f

◆ Events of the same message:examples: b → e, f → i

◆ Transitivity:examples: a → e, a → i, e → i


Ordering of Events




Abstract Clocks




System


Concurrent Events

■ Events not related by happened-before relation

■ Concurrency relation is denoted by ‖

■ Illustration:

g h i

a b c

d e fP2

P3

P1

◆ Examples: a ‖ d, d ‖ h, c ‖ e

■ Concurrency relation is not transitive:example: a ‖ d and d ‖ c but a ∦ c


Ordering of Events




Abstract Clocks




System


Concurrent Events

■ Events not related by happened-before relation■ Concurrency relation is denoted by ‖

■ Illustration:

g h i

a b c

d e fP2

P3

P1

◆ Examples: a ‖ d, d ‖ h, c ‖ e

■ Concurrency relation is not transitive:example: a ‖ d and d ‖ c but a ∦ c


Ordering of Events

Abstract Clocks

❖ Different Kinds of Clocks

❖ Logical Clock

❖ Implementing Logical Clock

❖ Implementing Logical Clock:

An Illustration❖ Limitation of Logical Clock

❖ Vector Clock

❖ Comparing Two Vectors

❖ Implementing Vector Clock

❖ Implementing Vector Clock:

An Illustration❖ Properties of Vector Clock




System


Different Kinds of Clocks

■ Logical Clocks◆ used to totally order all events

■ Vector Clocks◆ used to track happened-before relation

■ Matrix Clocks◆ used to track what other processes know about other

processes

■ Direct Dependency Clocks◆ used to track direct causal dependencies


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Logical Clock

■ Implements the notion of virtual time

■ Can be used to totally order all events

■ Assigns timestamp to each event in a way that isconsistent with the happened-before relation:

e → f ⇒ C(e) < C(f)

C(e): timestamp for event e

C(f): timestamp for event f


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Implementing Logical Clock

■ Each process has a local scalar clock, initialized to zero◆ Ci denotes the local clock of process Pi

■ Action depends on the type of the event:

■ Protocol for process Pi:

◆ On executing an interval event:Ci := Ci + 1

◆ On sending a message m:Ci := Ci + 1piggyback Ci on m

◆ On receiving a message m:let tm be the timestamp piggybacked on m

Ci := max{Ci, tm} + 1


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Implementing Logical Clock: An Illustration

1

1

P2

P3

P1

a

a: internal event C(a) = 1

1 2 3 4

1 3 4

1 52

2

4

P2

P3

P1

b e

a d

c


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



1

1

1

P2

P3

P1

c

b

a

b and c: internal events C(b) = 1 and C(c) = 1

1 2 3 4

1 3 4

1 52

2

4

P2

P3

P1

b e

a d

c


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



2

1

1 2

1

P2

P3

P1

a d

c

b

d: send event C(d) = 2

1 2 3 4

1 3 4

1 52

2

4

P2

P3

P1

b e

a d

c


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



2

1 3

1 2

1

P2

P3

P1

b e

a d

c

e: receive event C(e) = 3

1 2 3 4

1 3 4

1 52

2

4

P2

P3

P1

b e

a d

c


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



1 2 3 4

1 3 4

1 52

2

4

P2

P3

P1

b e

a d

c


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Limitation of Logical Clock

■ Logical clock cannot be used to determine whether twoevents are concurrent

e ‖ f does not imply C(e) = C(f)


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Vector Clock

■ Captures the happened-before relation

■ Assigns timestamp to each event such that:

e → f ⇐⇒ C(e) < C(f)

C(e): timestamp for event e

C(f): timestamp for event f


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Comparing Two Vectors

■ Vectors are compared component-wise:

◆ Equality:

V = V ′ iff 〈∀i : V [i] = V ′[i]〉

◆ Less Than:

V < V ′ iff 〈∀i : V [i] ≤ V ′[i]〉 ∧ 〈∃i : V [i] < V ′[i]〉

1

2

0

<

2

3

1

and

2

1

1

<

2

3

4

but

1

2

1

6<

2

1

3


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Implementing Vector Clock

■ Each process has a local vector clock◆ Ci denotes the local clock of process Pi

■ Action depends on the type of the event:


◆ On executing an interval event:Ci[i] := Ci[i] + 1

◆ On sending a message m:Ci[i] := Ci[i] + 1piggyback Ci on m

◆ On receiving a message m:let tm be the timestamp piggybacked on m

∀k Ci[k] := max{Ci[k], tm[k]}Ci[i] := Ci[i] + 1


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Implementing Vector Clock: An Illustration

P2

P3

P1

a[

1

0

0

]

[

0

0

1

]

a: internal event C(a) =

1

0

0

b e

a d

c

P2

P3

P1 [

1

0

0

] [

3

0

0

] [

4

0

0

][

2

0

0

]

[

2

0

0

]

[

2

2

0

][

0

1

0

] [

2

3

0

]

[

2

3

0

]

[

0

0

1

] [

0

0

2

] [

2

3

3

]


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



c

P2

P3

P1

b

a[

1

0

0

]

[

0

1

0

]

[

0

0

1

]

b and c: internal events C(b) =

1

0

0

and C(c) =

0

0

1

b e

a d

c

P2

P3

P1 [

1

0

0

] [

3

0

0

] [

4

0

0

][

2

0

0

]

[

2

0

0

]

[

2

2

0

][

0

1

0

] [

2

3

0

]

[

2

3

0

]

[

0

0

1

] [

0

0

2

] [

2

3

3

]


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



a d

c

P2

P3

P1

b

[

1

0

0

] [

2

0

0

]

[

2

0

0

]

[

0

1

0

]

[

0

0

1

]

d: send event C(d) =

2

0

0

b e

a d

c

P2

P3

P1 [

1

0

0

] [

3

0

0

] [

4

0

0

][

2

0

0

]

[

2

0

0

]

[

2

2

0

][

0

1

0

] [

2

3

0

]

[

2

3

0

]

[

0

0

1

] [

0

0

2

] [

2

3

3

]


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



b e

a d

c

P2

P3

P1 [

1

0

0

] [

2

0

0

]

[

2

0

0

]

[

2

2

0

][

0

1

0

]

[

0

0

1

]

e: receive event C(e) =

2

2

0

b e

a d

c

P2

P3

P1 [

1

0

0

] [

3

0

0

] [

4

0

0

][

2

0

0

]

[

2

0

0

]

[

2

2

0

][

0

1

0

] [

2

3

0

]

[

2

3

0

]

[

0

0

1

] [

0

0

2

] [

2

3

3

]


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System



b e

a d

c

P2

P3

P1 [

1

0

0

] [

3

0

0

] [

4

0

0

][

2

0

0

]

[

2

0

0

]

[

2

2

0

][

0

1

0

] [

2

3

0

]

[

2

3

0

]

[

0

0

1

] [

0

0

2

] [

2

3

3

]


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System


Properties of Vector Clock

■ How many comparisons are needed to determine whetheran event e happened-before another event f?

◆ As many as N integers may need to be compared in theworst case, where N is the number of processes

◆ Suppose e and f occurred on processes Pi and Pj

e → f

if and only if

(i = j) ∧ (C(e)[i] < C(f)[i])∨

(i 6= j) ∧ (C(e)[i] ≤ C(f)[i])


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System






e → f

if and only if

(i = j) ∧ (C(e)[i] < C(f)[i])

∨

(i 6= j) ∧ (C(e)[i] ≤ C(f)[i])


Ordering of Events

Abstract Clocks


❖ Logical Clock




❖ Vector Clock








System






e → f

if and only if

(i = j) ∧ (C(e)[i] < C(f)[i])∨

(i 6= j) ∧ (C(e)[i] ≤ C(f)[i])


Ordering of Events

Abstract Clocks


❖ Ordering of Messages

❖ Useful Notations

❖ Causal Delivery of Messages

❖ A Causally Ordered Delivery

Protocol❖ The BSS Protocol

❖ The BSS Protocol: An

Illustration❖ Another Causally Ordered

Delivery Protocol❖ When to Deliver a Message?

❖ The SES Protocol

❖ The SES Protocol (Continued)

❖ The SES Protocol: An

Illustration



System



■ For many applications, messages should be delivered incertain order to be interpreted meaningfully

■ Example:

m1: Have you seen the movie “Shrek”?

m2: Yes I have and I liked it

Bob

Alice

Tom

◆ m2 cannot be interpreted until m1 has been received◆ Tom receives m2 before m1: an undesriable behavior


Ordering of Events

Abstract Clocks













Illustration



System


Useful Notations

■ For a message m:◆ src(m): source process of m

◆ dst(m): destination process of m

◆ snd(m): send event of m

◆ rcv(m): send event of m

e

m

dst(m)

f

rcv(m)

snd(m)src(m)

Pi

Pj


Ordering of Events

Abstract Clocks













Illustration



System


Causal Delivery of Messages

■ A message w causally precedes a message m ifsnd(w) → snd(m)

■ An execution of a distributed system is said to be causallyordered if the following holds for every message m:

every message that causally precedes m and isdestined for the same process as m is deliveredbefore m

Mathematically, for every message w:

(snd(w) → snd(m)) ∧ (dst(w) = dst(m))

⇒

rcv(w) → rcv(m)


Ordering of Events

Abstract Clocks













Illustration



System


A Causally Ordered Delivery Protocol

■ Proposed by Birman, Schiper and Stephenson (BSS)

■ Assumption:

◆ communication is broadcast based: a process sends amessage to every other process

■ Each process maintains a vector with one entry for eachprocess:

◆ let Vi denote the vector for process Pi

◆ the jth entry of Vi refers to the number of messages thathave been broadcast by process Pj that Pi knows of


Ordering of Events

Abstract Clocks













Illustration



System


The BSS Protocol


◆ On broadcasting a message m:piggyback Vi on m

Vi[i] := Vi[i] + 1

◆ On arrival of a message m from process Pj :let Vm be the vector piggybacked on m

deliver m once Vi ≥ Vm

◆ On delivery of a message m sent by process Pj :

Vi[j] := Vi[j] + 1


Ordering of Events

Abstract Clocks













Illustration



System


The BSS Protocol: An Illustration

P2

P3

P1[

0

0

0

]

a e

[

0

0

0

]

[

0

0

0

]

[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System



P2

P3

P1[

0

0

0

] [

1

0

0

]

[

0

0

0

]

a

[

0

0

0

]

e

[

0

0

0

]

b

[

0

0

0

]

[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System



P2

P3

P1[

0

0

0

] [

1

0

0

]

[

0

0

0

]

a

[

0

0

0

]

e

[

0

0

0

][

1

0

0

]

b

[

0

0

0

]

[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System



P2

P3

P1[

0

0

0

] [

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

1

0

0

]

[

0

0

0

]

a

d

[

0

0

0

]

e

cb

[

0

0

0

]

[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System



[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

1

0

0

]

[

0

0

0

]

a e

d

[

0

0

0

]

cb

[

0

0

0

]

[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System



[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

d f

cb

[

0

0

0

]

[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System



[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

d f

cb

[

0

0

0

] [

1

0

0

]

[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System



[

0

0

0

][

1

1

0

]

P2

P3

P1[

1

0

0

]

[

0

0

0

][

1

1

0

][

1

0

0

]

[

1

0

0

]

[

0

0

0

]

[

0

0

0

]

a e

gf

[

1

0

0

]

cb

[

0

0

0

] [

1

0

0

] [

1

1

0

]


Ordering of Events

Abstract Clocks













Illustration



System


Another Causally Ordered Delivery Protocol

■ Proposed by Schiper, Eggli and Sandoz (SES)

■ Assumption:

◆ communication is point-to-point◆ processes are using vector clock algorithm

■ To determine whether a message has arrived out of causalorder:

◆ let Vm denote the vector timestamp of message m

◆ let Vi denote the vector clock of process Pi

Does there exist a message w that causallyprecedes m and destined for Pi such that Vi 6> Vw?


Ordering of Events

Abstract Clocks













Illustration



System


When to Deliver a Message?

■ A message m destined for process Pi can be delivered if:

◆ for every message w that causally precedes m anddestined for Pi:

Vi > Vw

or, equivalently

◆ let past(m, j) denote the set of all messages thatcausally precede m and are destined for process Pj :

Vi > maxw∈past(m,i)

{Vw}


Ordering of Events

Abstract Clocks













Illustration



System


The SES Protocol

■ Each process maintains a list of tuples with at most onetuple for every other process

◆ let DLi denote the list for process Pi

■ The list is piggybacked on every message a process sends

◆ let DLm denote the list for message m

■ if past(m, j) = ∅, then DLm does not contain a tuplefor process Pj

■ Otherwise, the tuple for process Pj is given by:

(j, maxw∈past(m,j)

{Vw})


Ordering of Events

Abstract Clocks













Illustration



System


The SES Protocol (Continued)


◆ On sending a message m to process Pj :piggyback DLi on m

remove entry for Pj from DLi, if anyadd (j, Vm) to DLi

◆ On arrival of a message m from process Pj :if DLm does not contain any tuple for Pi then

deliver m

elselet (j, V ) be the tuple for Pi in DLm

deliver m once Vi > V

endif

◆ On delivery of a message m:merge DLi and DLm


Ordering of Events

Abstract Clocks













Illustration



System


The SES Protocol: An Illustration

P1

P2

P3

{}[

0

0

0

]

a

{}

[

0

0

0

]

[

0

0

0

]

{}[

1

0

1

]

P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

g f

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

2

2

0

]{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

{}[

2

2

2

]


Ordering of Events

Abstract Clocks













Illustration



System



P1

P2

P3

[

1

0

0

]

{}[

0

0

0

]

a

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

{}

[

0

0

0

]

[

0

0

0

]

{}[

1

0

1

]

P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

g f

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

2

2

0

]{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

{}[

2

2

2

]


Ordering of Events

Abstract Clocks













Illustration



System



P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

c

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

[

0

0

0

]

{}[

1

0

1

]

P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

g f

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

2

2

0

]{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

{}[

2

2

2

]


Ordering of Events

Abstract Clocks













Illustration



System



P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

c

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

0

0

0

]

{}[

1

0

1

]

P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

g f

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

2

2

0

]{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

{}[

2

2

2

]


Ordering of Events

Abstract Clocks













Illustration



System



P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

e

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}

[

2

1

0

] {(3,

[

1

0

0

]

)}

{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

2

2

0

]

[

0

0

0

]

{}[

1

0

1

]

P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

g f

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

2

2

0

]{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

{}[

2

2

2

]


Ordering of Events

Abstract Clocks













Illustration



System



P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

ge

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}

[

2

1

0

] {(3,

[

1

0

0

]

)}

{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

g f

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

2

2

0

]{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

{}[

2

2

2

]


Ordering of Events

Abstract Clocks













Illustration



System



P1

P2

P3

[

2

0

0

][

1

0

0

]

{}[

0

0

0

]

a b

[

2

0

0

]

{}

[

1

0

0

]

{(3,

[

1

0

0

]

)}

dc

{(3,

[

1

0

0

]

)}

g f

{(3,

[

1

0

0

]

), (2,

[

2

0

0

]

)}

{}

[

0

0

0

]

{(3,

[

1

0

0

]

)}{(3,

[

1

0

0

]

)}

[

2

1

0

]

[

2

2

0

]{(3,

[

2

2

0

]

))}

[

2

2

0

]

[

0

0

0

]

{} {}[

1

0

1

]

{}[

2

2

2

]


Ordering of Events

Abstract Clocks



❖ State of a Distributed System

❖ Events and Local States

❖ When is a Global State

Meaningful?❖ Revisiting Happened-Before

Relation❖ A Consistent Global State

❖ Recording a Consistent

Global Snapshot❖ The CL Protocol

❖ The CL Protocol (Continued)

❖ The CL Protocol: An

Illustration❖ The CL Protocol: An

Illustration (Continued)


System



■ State of a distributed system is a collection of states of allits processes and channels:

◆ a process state is given by the values of all variables onthe process

◆ a channel state is given by the set of messages in transitin the channel

■ can be determined by examining states of the twoprocesses it connects

■ State of a process is called local state or local snapshot

◆ the textbook refers to local state as cut event

◆ cut event is not same as event

■ State of a distributed system is called global state or globalsnapshot


Ordering of Events

Abstract Clocks















System


Events and Local States

■ Processes change their states by executing events

■ Example:

P2

y = 3y = 2y = 1

P1

x = 0 x = 2 x = 3

a b

dc

◆ Process P1 changes its state from x = 0 to x = 2 onexecuting event a

◆ Process P2 changes its state from y = 2 to y = 3 onexecuting event d


Ordering of Events

Abstract Clocks















System


When is a Global State Meaningful?

■ Example:

P2

y = 3y = 2y = 1

P1

x = 0 x = 2 x = 3

a b

dc

◆ Is it possible for x to be 0 and y to be 3 at the same time?

◆ Does {s, x} form a meaningful global state?


Ordering of Events

Abstract Clocks















System


When is a Global State Meaningful?

■ Example:

P2

y = 3y = 2y = 1

P1

a bs t u

dc xwv

x = 0 x = 2 x = 3

◆ Is it possible for x to be 0 and y to be 3 at the same time?

◆ Does {s, x} form a meaningful global state?


Ordering of Events

Abstract Clocks















System


Revisiting Happened-Before Relation

■ Typically, happened-before relation is defined on events

■ The relation can be extended to local states as follows:

◆ let s be a local state of process Pi

◆ let t be a local state of process Pj

◆ s → t if there exist events e and f such that:1. Pi executed e after s,

2. Pj executed f before t, and

3. e → f


Ordering of Events

Abstract Clocks















System







◆ s → t if there exist events e and f such that:

1. Pi executed e after s,


3. e → f


Ordering of Events

Abstract Clocks















System







◆ s → t if there exist events e and f such that:1. Pi executed e after s,


3. e → f


Ordering of Events

Abstract Clocks















System


A Consistent Global State

■ For a global state G, let G[i] refer to the local state ofprocess Pi in G

■ A global state G is meaningful or consistent if

∀i, j : i 6= j : G[i] ‖ G[j]


Ordering of Events

Abstract Clocks















System





∀i, j : i 6= j : (G[i] 9 G[j]) ∧ (G[j] 9 G[i])

∀i, j : i 6= j : G[i] ‖ G[j]


Ordering of Events

Abstract Clocks















System





∀i, j : i 6= j : G[i] ‖ G[j]


Ordering of Events

Abstract Clocks















System


Recording a Consistent Global Snapshot

■ Proposed by Chandy and Lamport (CL)

■ Assumptions and Features:

◆ channels satisfy first-in-first-out (FIFO) property◆ channels are not required to be bidirectional◆ communication topology may not be fully connected

■ Messages exchanged by the underlying computation(whose snapshot is being recorded) are called applicationmessages

■ Messages exchanged by the snapshot algorithm are calledcontrol messages


Ordering of Events

Abstract Clocks















System





◆ channels satisfy first-in-first-out (FIFO) property

◆ channels are not required to be bidirectional◆ communication topology may not be fully connected




Ordering of Events

Abstract Clocks















System





◆ channels satisfy first-in-first-out (FIFO) property◆ channels are not required to be bidirectional

◆ communication topology may not be fully connected




Ordering of Events

Abstract Clocks















System





◆ channels satisfy first-in-first-out (FIFO) property◆ channels are not required to be bidirectional◆ communication topology may not be fully connected




Ordering of Events

Abstract Clocks















System


The CL Protocol

■ Initially: all processes are colored blue

■ Eventually: all processes become red

■ On changing color from blue to red:record local snapshotsend a marker message along all outgoing channels

■ On receiving marker message along incoming channel C:if color is blue then

change color from blue to redendifrecord state of channel C as application messages

received along C since turning red

■ Any process can initiate the snapshot protocol byspontaneously changing its color from blue to red◆ there can be multiple initiators of the snapshot protocol


Ordering of Events

Abstract Clocks















System


The CL Protocol







■ Any process can initiate the snapshot protocol byspontaneously changing its color from blue to red

◆ there can be multiple initiators of the snapshot protocol


Ordering of Events

Abstract Clocks















System


The CL Protocol







■ Any process can initiate the snapshot protocol byspontaneously changing its color from blue to red◆ there can be multiple initiators of the snapshot protocol


Ordering of Events

Abstract Clocks















System


The CL Protocol (Continued)

■ Global Snapshot: local snapshots of processes just afterthey turn red

■ In-Transit Messages: blue application messages receivedby processes after they have turned red

■ Why is the global snapshot consistent?◆ Assume an application message has the same color as

its sender◆ Can a blue process receive a red application message?


Ordering of Events

Abstract Clocks















System





■ Why is the global snapshot consistent?

◆ Assume an application message has the same color asits sender

◆ Can a blue process receive a red application message?


Ordering of Events

Abstract Clocks















System






its sender

◆ Can a blue process receive a red application message?


Ordering of Events

Abstract Clocks















System






its sender◆ Can a blue process receive a red application message?


Ordering of Events

Abstract Clocks















System


The CL Protocol: An Illustration

■ Three processes: P1, P2 and P3

■ Five channels: C1,2, C1,3, C2,1, C2,3 and C3,1

P1

P2P3

C2,3

C1,2

C2,1

C1,3

C3,1


Ordering of Events

Abstract Clocks















System


The CL Protocol: An Illustration (Continued)

m3

m1

m4

m5

m2

P1

P2

P3

1. All processes are blue

m3

m1

m4

m5

m2

P1

P2

P3

1. P1 receives the marker message along C3,1

2. P1 records the state of C3,1 as {m2, m3}


Ordering of Events

Abstract Clocks















System



m3

m1

m4

m5

m2

P1

P2

P3

1. P1 turns red2. P1 sends a marker message along C1,2 and C1,3

m3

m1

m4

m5

m2

P1

P2

P3




Ordering of Events

Abstract Clocks















System



m3

m1

m4

m5

m2

P1

P2

P3

1. P2 receives the marker message along C1,2 and turns red2. P2 sends a marker message along C2,1 and C2,3

3. P2 records the state of C1,2 as ∅

m3

m1

m4

m5

m2

P1

P2

P3




Ordering of Events

Abstract Clocks















System



m3

m1

m4

m5

m2

P1

P2

P3

1. P3 receives the marker message along C1,3 and turns red2. P3 sends a marker message along C3,1

3. P3 records the state of C1,3 as ∅

m3

m1

m4

m5

m2

P1

P2

P3




Ordering of Events

Abstract Clocks















System



m3

m1

m4

m5

m2

P1

P2

P3



m3

m1

m4

m5

m2

P1

P2

P3




Ordering of Events

Abstract Clocks















System



m3

m1

m4

m5

m2

P1

P2

P3


2. P3 records the state of C2,3 as {m1}

m3

m1

m4

m5

m2

P1

P2

P3




Ordering of Events

Abstract Clocks















System



m3

m1

m4

m5

m2

P1

P2

P3




Ordering of Events

Abstract Clocks




System

❖ A Subclass of Distributed

Computation❖ Distributed Computation: An

Illustration❖ Termination Detection

❖ Huang’s Algorithm


(Continued)❖ Huang’s Algorithm: An

Illustration


A Subclass of Distributed Computation

■ Many distributed computations obey the followingparadigm:

◆ A process is either in active state or passive state◆ A process can send an application message only when

it is active◆ An active process can become passive at any time◆ A passive process, on receiving an application

message, becomes active

■ Intuitively:

◆ if a process is active, then it is doing some work◆ if process is passive, then it is idle◆ an active process uses an application message to send

a part of its work to another process


Ordering of Events

Abstract Clocks




System







Illustration




◆ A process is either in active state or passive state

◆ A process can send an application message only whenit is active

◆ An active process can become passive at any time◆ A passive process, on receiving an application


■ Intuitively:




Ordering of Events

Abstract Clocks




System







Illustration





it is active

◆ An active process can become passive at any time◆ A passive process, on receiving an application


■ Intuitively:




Ordering of Events

Abstract Clocks




System







Illustration





it is active◆ An active process can become passive at any time

◆ A passive process, on receiving an applicationmessage, becomes active

■ Intuitively:




Ordering of Events

Abstract Clocks




System







Illustration







■ Intuitively:




Ordering of Events

Abstract Clocks




System







Illustration







■ Intuitively:

◆ if a process is active, then it is doing some work

◆ if process is passive, then it is idle◆ an active process uses an application message to send



Ordering of Events

Abstract Clocks




System







Illustration







■ Intuitively:

◆ if a process is active, then it is doing some work◆ if process is passive, then it is idle

◆ an active process uses an application message to senda part of its work to another process


Ordering of Events

Abstract Clocks




System







Illustration







■ Intuitively:




Ordering of Events

Abstract Clocks




System







Illustration


Distributed Computation: An Illustration

��

P2

P3

P1

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

P2

P3

P1

m1

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

��

P2

P3

P1

m1

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

��

P2

P3

P1

m1

m2

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

P2

P3

P1

m1

m2

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

P2

P3

P1

m1

m2

m3

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

P2

P3

P1

m1

m2

m3

m4

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

��

P2

P3

P1

m1

m2

m3

m4

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

��

��

P2

P3

P1

m1

m2

m3

m4


Ordering of Events

Abstract Clocks




System







Illustration


Termination Detection

■ To detect if the computation has finished doing all the work

◆ all processes have become passive, and◆ all channels have become empty

■ Different types of computations:

◆ diffusing: only one process is active in the beginning◆ non-diffusing: any subset of processes can be active in

the beginning■ no process knows which processes are active and

which processes are passive


Ordering of Events

Abstract Clocks




System







Illustration




◆ all processes have become passive, and

◆ all channels have become empty






Ordering of Events

Abstract Clocks




System







Illustration










Ordering of Events

Abstract Clocks




System







Illustration






◆ diffusing: only one process is active in the beginning

◆ non-diffusing: any subset of processes can be active inthe beginning■ no process knows which processes are active and



Ordering of Events

Abstract Clocks




System







Illustration







the beginning

■ no process knows which processes are active andwhich processes are passive


Ordering of Events

Abstract Clocks




System







Illustration










Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm

■ Assumption: computation is diffusing

◆ the initially active process is called the coordinator■ coordinator is responsible for detecting termination

■ Initially:

◆ coordinator has a weight of 1◆ all other processes have a weight of 0

■ Invariants:

◆ total amount of weight in the system is 1◆ a non-coordinator process has a non-zero weight if and

only if it is active◆ a channel has a non-zero weight if and only if it is

non-empty■ weight of a channel is the sum of the weight of all its

messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm


◆ the initially active process is called the coordinator

■ coordinator is responsible for detecting termination

■ Initially:


■ Invariants:




messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm



■ Initially:


■ Invariants:




messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm



■ Initially:

◆ coordinator has a weight of 1

◆ all other processes have a weight of 0

■ Invariants:




messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm



■ Initially:


■ Invariants:




messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm



■ Initially:


■ Invariants:

◆ total amount of weight in the system is 1

◆ a non-coordinator process has a non-zero weight if andonly if it is active

◆ a channel has a non-zero weight if and only if it isnon-empty■ weight of a channel is the sum of the weight of all its

messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm



■ Initially:


■ Invariants:


only if it is active

◆ a channel has a non-zero weight if and only if it isnon-empty■ weight of a channel is the sum of the weight of all its

messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm



■ Initially:


■ Invariants:



non-empty

■ weight of a channel is the sum of the weight of all itsmessages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm



■ Initially:


■ Invariants:




messages


Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm (Continued)

■ Actions:

◆ On sending an application message:■ send half of its weight along with the message

◆ On receiving an application message:■ add the weight of the message to the current weight

◆ On becoming passive:■ send the current weight to the coodrinator

■ Coordinator announces termination once:

◆ it has become passive and◆ it has collected all the weight


Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:

◆ On sending an application message:

■ send half of its weight along with the message






Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:







Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:


◆ On receiving an application message:

■ add the weight of the message to the current weight





Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:







Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:



◆ On becoming passive:

■ send the current weight to the coodrinator




Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:







Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:





◆ it has become passive and

◆ it has collected all the weight


Ordering of Events

Abstract Clocks




System







Illustration



■ Actions:







Ordering of Events

Abstract Clocks




System







Illustration


Huang’s Algorithm: An Illustration

��

P2

P3

P1

1

0

0

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

P2

P3

P1

m1(1

2)

1 1

2

0

0

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

��

P2

P3

P1

m1(1

2)

1 1

2

0 1

2

0

P2

P3

P1

m1(1

2)

1 1

2

0 1

2

0

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

P2

P1

m1(1

2)

m2(1

4)

1 1

2

0 1

2

1

4

0P3

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

0 1

2

1

4

0 1

4

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

0 1

2

m3(1

8)

1

4

0 1

4

1

8

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

0 1

2

m4(1

16)

m3(1

8)

1

4

0 1

4

1

8

1

16

3

8

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

0 1

2

m4(1

16)

m3(1

8)

1

4

0 1

4

1

8

1

16

3

8

9

16

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

��

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

16

0

0

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0


Ordering of Events

Abstract Clocks




System







Illustration



��

��

��

��

��

P2

P3

P1

m1(1

2)

m2(1

4)

1 1

2

1

16

0 1

2

m4(1

16)

m3(1

8)

3

8

1

4

0 1

4

1

8

1

16

3

8

9

16

15

161

0

0

distributed operating systems: theoretical...

Documents