Replication (1)

Topics

Why Replication? System Model Consistency Models – How do we reason about

the consistency of the “global state”? Data-centric consistency Client-centric consistency

We will examine consistency protocols which describe an implementation of a specific consistency model.

Other Implementation Issues Examples

Readings

Van Steen and Tanenbaum: 6.1, 6.2 and 6.3, 6.4

Coulouris: 11,14

Why Replicate? Replication refers to the maintenance of

copies at multiple site Reliability

If one replica is unavailable or crashes, use another

Avoid single points of failure Performance

Placing copies of data close to the processes using them can improve performance through reduction of access time.

If there is only one copy, then the server could become overloaded.

Common Replication Examples

DNA naming service Web browsers often locally store a copy

of a previously fetched web page. This is referred to as caching a web

page. Replication of a database Replication of game state

System Model A collection of replica managers (RMs) provide

a service to clients One replica manager per replica

The front-end for a client allows a client to see a service that gives it access to logical objects, which are in fact replicated at the RMs

Clients request operations: some are read-only operations and some are update operations

•

System Model

FE

Requests andreplies

C

ReplicaC

Service

ClientsFront ends

managers

RM

RMFE

RM

•

Clients’ request are handled by front ends. A front end makes replication transparent.

Phases in Executing Request Issue request

the FE either• sends the request to a single RM that passes it on to the others • or multicasts the request to all of the RMs

Coordination The RMs decide whether to apply the request; and decide

on its ordering relative to other requests (according to FIFO or total ordering)

Execution The RMs execute the request

Agreement RMs agree on the effect of the request, e.g., perform 'lazily'

or immediately Response

One or more RMs reply to FE.

Group Communication Replication systems need to be able to

add/remove RMs A group membership service provides:

Interface for adding/removing members• Create, destroy process groups, add/remove members• A process can generally belong to several groups

Implements a failure detector• This monitors members for failures

(crashes/communication), and excludes them when unreachable

Notifies members of changes in membership Expands group addresses

• Multicasts addressed to group identifiers• Group expanded to a list of delivery addresses• Coordinates delivery when membership is changing

Services provided for process groups

Join

Groupaddress

expansion

Multicastcommunication

Group

send

FailGroup membership

management

Leave

Process group

•

A process outside the group sends to the group without knowing the membership

Membership service provides leave and join operations

Failure detector notes failures and evicts failed processes from the group

The group address is expanded

Members are informed when processes join/leave

Group Communication Replication systems need to be able to

add/remove RMs A group membership service provides:

Interface for adding/removing members• Create, destroy process groups, add/remove members• A process can generally belong to several groups

Implements a failure detector • This monitors members for failures

(crashes/communication), and excludes them when unreachable

Notifies members of changes in membership Expands group addresses

• Multicasts addressed to group identifiers• Group expanded to a list of delivery addresses• Coordinates delivery when membership is changing

Developing Replication Systems

Consistency Faults Changes in Group Membership

A Replication Problem

Multiple copies may lead to consistency problems.

Whenever a copy is modified, that copy becomes different from the rest.

Modifications have to be carried out on all copies to ensure consistency.

The type of application has an impact on the consistency requirements needed and thus on the implementation.

Consistency Model

A consistency model describes the rules to be used in updating replicated data

The rules define the order of operations. Rules used depend on the application Consistency defined within the context

of read and write operations on shared data is data-centric Strict Sequential Causal FIFO

Strict Consistency

Strict consistency is defined as follows: Read is expected to return the value resulting

from the most recent write operation Assumes absolute global time All writes are instantaneously visible to all

Suppose that process pi updates the value of x to 5 from 4 at time t1 and multicasts this value to all replicas Process pj reads the value of x at t2 (t2 > t1). Process pj should read x as 5 regardless of

the size of the (t2-t1) interval.

Strict Consistency

What if t2-t1 = 1 nsec and the optical fiber between the host machines with the two processes is 3 meters. The update message would have to travel at

10 times the speed of light Not allowed by Einsten’s special theory of

relativity. Can’t have strict consistency

Sequential Consistency

Sequential Consistency: The result of any execution is the same as if the

(read and write) operations by all processes on the data were executed in some sequential order

Operations of each individual process appears in this sequence in the order specified by is programs.

We have seen this in the banking example One implementation used Lamport’s clocks.

Causal Consistency

Causal Consistency: That if one update, U1, causes another update, U2, to occur then U1 should be executed before U2 at each copy.

Application: Bulletin board Possible Implementation: Using vector clocks

FIFO Consistency Writes done by a single process are seen

by all other processes in the order in which they were issued

… but writes from different processes may be seen in a different order by different processes.

i.e., there are no guarantees about the order in which different processes see writes, except that two or more writes from a single source must arrive in order.

FIFO Consistency

Caches in web browsers All updates are updated by page owner. No conflict between two writes Note: If a web page is updated twice in a very

short period of time then it is possible that the browser doesn’t see the first update.

Implementation: Each process adds the following to an update

message: (process id, sequence number) Each other process applies the update

messages in the order received from a single process.

Implementation Options: Sequential Consistency

We saw how to use Lamport’s logical clocks for sequential consistency.

Another option is to have a centralized processor that is a sequencer.

Implementation Options: Sequential Consistency

We saw how to use Lamport’s logical clocks for sequential consistency.

Another option is to have a centralized processor that is a sequencer.

Each update request it sent to the sequencer which Assigns the request a unique sequence

number Update request is forwarded to each replica Operations are carried out in the order of

their sequence number

Implementation Options: Sequential Consistency

The use of a sequencer also does not solve the scalability problem. It may become a performance bottleneck. What if it goes down?

A combination of Lamport timestamps and sequencers may be necessary.

The approach is summarized as follows: Each process has a unique identifier, pi, and keeps a

sent message counter ci. The process identifier and message counter uniquely identify a message.

Active processes (or a sequencer) keep an extra counter: ti. This is called the ticket number. A ticket is a triplet (pi, ti, (pj, cj)).

All other processes are passive

Implementation Options: Sequential Consistency

Approach Summary (cont) Passive processes (non-sequencer) send their

messages to their sequencer. Lamport’s totally ordered multicast algorithm is used

among the sequencers to determine the order of update operations.

When an operation is allowed, each sequencer sends the ticket to its associated passive processes. It is assumed that the passive process receives these tickets in the order sent.

Implementation Options: Sequential Consistency

Approach Summary (cont) If a sequencer terminates abnormally, then

one of the passive processes associated with it can become the new sequencer.

An election algorithm may be used to choose the new sequencer.

Implementation Options: Sequential Consistency

Let’s say that we have 6 processes: p1,p2,p3,p4,p5,p6

Assume that p1,p2 are sequencers; p3,p4 are associated with p1 and p5,p6 are associated with p2

Let’s say that p3 sends a message which is identified by (p3 , 1).

p1 generates a ticket as follows: (p1, 1, (p3 , 1)) The ticket number is generated using the

Lamport clock algorithm.

Ticket number

Implementation Options: Sequential Consistency

Let’s say that p5 sends a message which is identified by (p5 , 1).

p2 generates a ticket as follows: (p2, 1, (p5 , 1))

Which update gets done first? Basically, p1,p2 will apply Lamport’s algorithm for totally ordered multicast.

When an update operation is allowed to proceed, the sequencers send messages to their associated processes.

Data-Centric Consistency Models

The consistency models just discussed are called data-centric consistency models.

Assumptions: Concurrently processes may be

simultaneously updating Updates need to be propagated quickly.