cdc

UBSS UBSS UNIXUNIX Based System SoftwareBased System Software

CDCCDCConsistent Datastore CopiesConsistent Datastore Copies

MCC/CPC/CIS

01/04/2004

UBSS/CDC 2MCC/CPC/CIS

Version 6

CDC ModelCDC Model

Network Shared Memory with Update Notifications

Simple abstraction already understoods by programmers

Natural transistion from sequential to distributed applications

Shared Memory Persistance

01/04/2004


Version 6

CDC BenefitsCDC Benefits

Applications loosely coupled

Replicated Data Services for Fault tolerant applications

01/04/2004


Version 6

ATC OPERATOR RequirementsATC OPERATOR Requirements

ATC controllers require : Rapid response time

and

Fixed response time

to commands and commands acknowledgment

01/04/2004


Version 6

ATC Centers Physical Lay OutATC Centers Physical Lay Out

FDPFDP

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

RequestandAck

01/04/2004


Version 6

Push or Pull modelPush or Pull model

Pull model clients request information from the server when

needed Push model

servers push the information to clients before needed

01/04/2004


Version 6

Push modelPush model

FDPFDP

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

OperatorPosition

01/04/2004


Version 6

Push model and data replication

• Push model implies : data replication

• Data replication implies data consistency between copies :

Strong consistencyOr Weak consistency

01/04/2004


Version 6

Strong consitency

Strong consistency implies :

Atomicity : all receive the message or no one

Causality : all messages are received by all in the same order

01/04/2004


Version 6

AtomicityAtomicityA B C

Data Update

Update Available

Ack

Group Ack

01/04/2004


Version 6

Causality (case study)Causality (case study)A B C

Data Update

Ack

Ordering Message (Atomic)

Ack

Data Updatenon causal event = same time

01/04/2004


Version 6

CDC lightweight protocolCDC lightweight protocol

CDC provids weak consistency :

Replicated information may temporarily differ while data copies are updated = data are always available

Serial consistency on the same record is guaranteed = an old data will never replace a newer data, but data update may be lost

Periodic consistency check = One global acknowledgment at periodic time (master/slave configuration)

01/04/2004


Version 6

Temporary inconsistencyTemporary inconsistencyData Update

= Version N

N+1 N N

N+1 N+1 N

N+1 N+1 N+1

N N N

= Version N +1

01/04/2004


Version 6

Periodic Consistency checkingPeriodic Consistency checking

T

A B C D E

Master

Update

Update

Update

CAM

T

T

T

T

A B C D E

CAM

CAM

CAM

CAM

(CAM : Central Announcement Message)RRM : Retransmission Request Message

01/04/2004


Version 6

CDC algorithmsCDC algorithms

Full replication provides : Multiple Reader Single Writer : MRSW (read

replication consistency). Multiple Readers Multiple Writers : MRMW

(write shared consistency).

01/04/2004


Version 6

CDC Write Shared Consistency SemanticCDC Write Shared Consistency Semantic

Write Shared : Write Shared Data are those that are frequently and concurrently writen by multiple threads without intervening synchronization.

Concurreny writes are safe, because the programmers knows that each thread updates separate portions of the Data.

Because of the way the data are laid out in memory, false sharing may be present.

01/04/2004


Version 6

Write Sharing ModelWrite Sharing Model

Array sharing, Unshared record : notification false shared : filtering.

Shared record, unshared fields : false shared : filtering.

Shared fields : real shared : Application synchronisation

(Transactionnal).

01/04/2004


Version 6

Write Shared : Array False sharingWrite Shared : Array False sharing

CDC shared record array

thread 1

thread 2

thread 3

Unshared Record

01/04/2004


Version 6

Write Shared : Record False sharing => ColisionWrite Shared : Record False sharing => Colision

thread 1 thread 2 thread 3

Shared Record

Unshared fields

01/04/2004


Version 6

CDC shared memory access semantic CDC shared memory access semantic

READ(data, record #)WRITE(data, record #)SUBSCRIBENOTIFY (record #)

01/04/2004


Version 6

Consistent Datastore Copies (CDC) : SCOPE 1/2Consistent Datastore Copies (CDC) : SCOPE 1/2

Distributed algorithm for managing replicated information with stringent response time (access) requirements on a large number of nodes

Replicated information may temporarily differ while data copies are updated. The copies converge when data modification ceases.

The algorithm is fail-safe against node failures and insertions. A predefined node is the master and provides periodic consistency checking and

recovery. In case of failure, a new master is automatically allocated by means of priority. Application process can subscribe to datastore. It will then be informed about any

changes of the datastore. User has no knowledge about other producers or consumers. Serial consistency on the same record is guaranteed.

01/04/2004


Version 6

Consistent Datastore Copies (CDC) : SCOPE 2/2Consistent Datastore Copies (CDC) : SCOPE 2/2

A datastore copy is a table of records Sorting algorithm with application defined key A datastore copy has fixed known attribute (size of record, number of records,

name and identification of the datastore). All datastore copy with the same name will contain the same informations on all

nodes.The application may claim access to the datastore copy. The lock is node wide. (It is possible to group the operations). Access time must be limited. Application must take care of deadlocks. Application defined automatic conversion function call backfor read and write Four different ways to access store records :

Physical access using pointers Sequential access Direct access Indexed sequential access (Each user process can provide a compare function of

key to built a table of indexes)

01/04/2004


Version 6

CDC : Distributed Shared Memory ModelCDC : Distributed Shared Memory Model

Structure : Layout of the shared data memory :

Array of C or Ada record data typeGranularity :

Size of the unit of sharing C or Ada strutures size, which can vary to match

the application requirements

01/04/2004


Version 6

CDC MECHANISMSCDC MECHANISMS

Data Store

Data Store

LAN

Appli

Appli

Master

Slave

SlaveSlave

Data Store

Appli Appli

Data Store

Appli

Data Store

Appli

Slave

data bus

01/04/2004


Version 6

UBSS CDC : System DesignUBSS CDC : System Design

RFP

Radar1 Radar2 Radar3 Radar4

RDP FDP

System Tracks Identification Flight Plan

MMIMMIMMIMMI

Radar

01/04/2004


Version 6

CDC : CDC : Context diagramContext diagram

CDC

Application

LibraryInterface

MessageInterface

EventNotification

CDC store collision store

CDC protocol

LAN

LibraryInterface

01/04/2004


Version 6

CDC : distribution and processing 1/3CDC : distribution and processing 1/3

For each record, CDC maintains two field :- the Record Update Counter (RUC); identifies the version of the record. It is

incremented by one for each modification.- the Record Update Originator (RUO); identifies the node where the last update has

been initiated. A user gains exclusive access to the local datastore copy and update one or several

records. RUC and RUO are also updated by CDC. When exclusive access is released, the updated records are packed into one or more

Update Initiation Messages (UIMs) and distributed to other nodes on the LAN. UDP datagram broadcast facility is used.

Each receiver node (master and slave) applies the same algorithm to update their data store.

For each record received in UIM message, if the RUC(UIM) > RUC (datastore), it replaces the datastore record

If RUC(UIM) = RUC (datastore) and RUO(UIM) /= RUO (datastore) then a collision occured

RRM : Retransmission Request MessageUIM : Update Initiation Messages

01/04/2004


Version 6


RUC RUO User record

data store copy

32 Ko max.

12000 records max.

(RUC : Record Update Counter)

(RUO: Record Update Originator)

{CAM (CAM : Central Announcement Message)RRM : Retransmission Request MessageUIM : Update Initiation Messages

01/04/2004


Version 6


slave slave

slave master slave

UIM

Appli-cation

Appli-cation

Appli-cation

Appli-cation

Appli-cation


01/04/2004


Version 6

CDC : periodic consistency checkingCDC : periodic consistency checking

slave slave

slave slave

CAM

Appli-cation

Appli-cation

Appli-cation

Appli-cation

Appli-cation

masterAppli-cation


01/04/2004


Version 6

CDC : periodic consistency checking and recoveryCDC : periodic consistency checking and recovery

At regular intervals (period is a CDC start-up parameter), the master initiates a consistency check by broadcasting a special message.

The Central Announcement Message (CAM) contains the RUC and RUO fields of each record of the Master datastore copy.

Each slave node performs the same algorithm. - CDC requests retransmission from the master when :

RUC(CAM record) > RUC(datastore copy) (master has a new version)RUC(CAM record) = RUC (datastore copy) and RUOs are different. (The

collision has not been resolved by application).- CDC retransmits Update in broadcast mode when :

RUC(CAM record) < RUC(datastore copy) (master has an old version) CDC retransmission requests are transmitted using a Retransmission Request Message

(RRM) with UDP point to point facility. The master distributes its version of the record in question by mean of the Update

Retransmission Message (URM) with UDP broadcast facility.

RRM : Retransmission Request Message

01/04/2004


Version 6

CDC : retransmission requestCDC : retransmission request

slave slave

slave slave

RRM

Appli-cation

Appli-cation

Appli-cation

Appli-cation

Appli-cation

masterAppli-cation


01/04/2004


Version 6

CDC : Update RetransmissionCDC : Update Retransmission

slave slave

slave slave

URM

Appli-cation

Appli-cation

Appli-cation

Appli-cation

Appli-cation

masterAppli-cation

RRM : Retransmission Request MessageUIM : Update Initiation MessagesURM : Update Retransmission Message

01/04/2004


Version 6

CDC : update collision detection 1/2CDC : update collision detection 1/2

The detection of the collision by CDC triggers the following actions first, the original contents of the local record is put in the collision store the received update of the record is placed in the datastore copy an update event is generated for all subscribed processes, except the originator of

the saved collision the originator of the saved collision which made the update, is notified that a

collision occurred.

It is the responsibility of the originator to recover from the collision


01/04/2004


Version 6

CDC : update collision detection 2/2CDC : update collision detection 2/2

UIM

node A node B

local update record RRUC= n, RUO=A

local update record RRUC= n, RUO=B

remote update record RRUC(UIM) = n, RUO(UIM) = B

remote update record RRUC(UIM) = n, RUO(UIM) = A

- node A and node B update the same record R in a close time- CDC of node A(B) broadcasts an UIM message with record R update- node A (B) receives an UIM message with a remote update of record R- received record R has the same RUC and a different RUO as the local record R.- a collision is detected

RRM : Retransmission Request MessageUIM : Update Initiation MessagesURM : Update Retransmission Message

01/04/2004


Version 6

CDC : behaviour in exceptional caseCDC : behaviour in exceptional case

Failure of a slave has no influence for other slaves or the master. Consistency of datastore copies is guaranteed if the master is available. The master must have a high availability (located in a dual node). When a master switches or is about to terminate it sends a Master Died Message

(MDM). Absence of master is detected by reception of MDM or after no periodic reception of

CAM. The event "master not alive" is then reported to the application. The event "master alive" occurs when a slave detects a master (CAM reception).

01/04/2004


Version 6

CDC : control node accessCDC : control node access

A user application is able to disconnect/reconnect the datastore copy from/to the LAN using a library function

"Connected" node access state is the normal CDC behavior. Local and remote (received) updates are normally processed.

When the node access state is "isolated", received CDC messages are ignored. The local update are only performed. UIM messages are not sent

After a transition from "isolated" to "connected", the master records will overwrite local records (regardless of the local RUC).

The "no_lan_support" node access state is used when the CDC does not support LAN. This state can not be changed

01/04/2004


Version 6

CDC : datastore designCDC : datastore design

Data normalization should be a solution for a good design Three rules can be used:

1 : Remove repeating fields to a separate record2 : Similarly, remove fields which can refer to another key3 : Similarly, remove fields which can refer to other non-key fields elsewhere

Try to avoid records which are bigger than the medium packet-size. Don't forget that CDC does not support integrity neither between stores nor

between records of the same CDC store.

01/04/2004


Version 6

CDC library functions 1/3CDC library functions 1/3

Datastore copy user registration:cdc_open Open a specified datastore copy.cdc_close Close a datastore copy.cdc_subscribe Open a specified datastore copy and subscribe for CDC events.

Get and free exclusive datastore copy access:cdc_store_acc Access to a datastore copy.

Create / Remove indexed sequential access environment:cdc_is_build Create indexed sequential access environment.cdc_is_remove Remove indexed sequential access environment.

Controlling master/slave state:cdc_master_slave Change status of datastore copy.

01/04/2004


Version 6


Accessing datastore copy records:

cdc_record_acc() Get physical access to a datastore copy record.cdc_coll_acc() Get physical access to a saved collision record.cdc_direct_read() Obtain a copy of a datastore copy record by record number.cdc_direct_coll_read() Obtain a copy of a saved collision record by record number.cdc_direct_update() Update a datastore copy record by record number.cdc_direct_delete() Delete a datastore copy record by record number.cdc_seq_read() Obtain datastore copy records sequentially by record number.cdc_is_read() Obtain a copy of a datastore copy record by key.cdc_is_update() Update a datastore copy record by record number.cdc_is_create() Create a ’new’ record using a deleted one.cdc_is_key_delete() Delete all datastore records with a specified key.

01/04/2004


Version 6


Handling events:cdc_event Get next event from the event chain.cdc_upd_recnum Get number of next updated record.cdc_coll_recnum Get number of next save collision record.cdc_event_ctl Control sequence of event reporting.

Retrieving datastore copy information:cdc_nr_records Get number of records in datastore copy.cdc_size_record Get the size of the datastore copy records.cdc_statistics Get/reset statistics.cdc_store_acc_status Get the store access status.cdc_req_cons_rep Request consistency reply message.cdc_is_key_exists Check for existence of datastore records with a specified key.

Controlling node access state:cdc_control_node_access Change status of node access.

01/04/2004


Version 6

CDC event managementCDC event management

ApplicationProcess

RUC RUO User record

CDCProcess

ApplicationProcess

FIFO

FIFO

CDC event

Shared memory

CDC event notiification

CDC events chain

01/04/2004


Version 6

CDC Reader Side Typical useCDC Reader Side Typical usedeclaration of FIFOCDC with their associated typeinitialization of QUEUE_DECKinitialization of FIFOCDCdeclaration of maskinitialization of CDC utilisationOPEN FIFOCDC for system wide communicationinitialization of mask with priority of FIFOCDC=1OPEN CDCrecover eventsdo forever

WAIT_FROM(mask)case receiving FIFO id of

FIFOCDC :do forver

case CDC event of NOEVENT : exitUPDATE : treament 2COLLISION : treatment 3.....

end caseend do

time out : treatmentend case

end doRRM : Retransmission Request Message

01/04/2004


Version 6

CDC debugger : CDD 1/2CDC debugger : CDD 1/2

CDD is a tool to inspect and /or change contents or behavior of one or more CDC stores CDD is a kind of application program and must be specified in the (ncf.n) Node

Configuration File. It can be actived by an interactive session. CDD main commands :

close / open close / open a CDC store.define format define one or all records from a CDC store.delete delete one or all records from a CDC store.force close / unlock force one or all processes to close or unlock a CDC store.format assign a format to a CDC store.get events display events that are currently received for a CDC store.isolate / reconnect disconnect / reconnect the CDC store from lan.lock / unlock lock / unlock a CDC store.log events enable or disable the logging of events.master / slave become master / slave of a CDC store.read / write read from or write to a CDC store.

01/04/2004


Version 6

CDC debugger : CDD 2/2CDC debugger : CDD 2/2

CDD main commands (continued): recover events force CDC to send events for all modified records of a CDC store.reset statistics reset the statistical information, kept by CDC for a CDC store.show users / status show the users or status and locks of one or all CDC stores.skip events skip all events that are currently received for a CDC store.use use a CDC store to be the current active CDC store .wait consistency wait for (a record of) the CDC store to become consistent.wait events wait for and display the receipt of events for a CDC store.

CDD offers others general commandslog input to a logfilelog output and error information to a logfileprovides options for starting with a commandfile...

cdc

Technology

shared data

version n n

data consistency

data copies

order01042 version

newer data

old data

data replicationpush