31 Aug 2004 1

Asynchronous data communication mechanisms

ACMs

Fei Xia, Ian Clark, Alex Yakovlev

EECE, Univ of Newcastle upon Tyne

31 Aug 2004 2

Data communication

• Point to point: connecting two processes

writer readerdata

Data is a stream of items of a set type.Writer and reader are cyclic processes.Writer provides one item of data per cycle. Reader uses one item per cycle.

31 Aug 2004 3

Data communication

• Traditional approach

writer reader w/r

31 Aug 2004 4

Data communication

• Traditional approach is synchronized– Either reader or writer must wait for the

other side during the transfer of every data item

– Not ideal for many concurrent systems (esp. embedded, real-time, and low power systems)

31 Aug 2004 5

The delay ratio problem – systems are getting bigger

Delay (ps)

VLSI Generation (m)

Transistor/Gate Interconnect

Transistors get smaller/faster

Interconnects get longer/slower

Delays are unknown until we have the layout – The TIME CLOSURE problem – delays change if the layout changes.

31 Aug 2004 6

• Asynchronous processing.

• Improved EMC - dependent on data being processed.

• Lower power - energy only used when work is done.

Example – A to D conversion.

Moreover …

31 Aug 2004 7

Asynchrony in concurrent systems

• Inevitable– High degrees of integration mean that

distribution of global clocks becomes impractical, so even on the same chip there will be asynchrony

– Distributed systems naturally have local clocks. Synchronyzing such clocks can be problematic.

– Self-timed systems (esp. useful for low power) have no regularly pulsing clocks.

31 Aug 2004 8

Asynchrony in concurrent systems

• Real-time elements– Most obvious characteristic is timing

predictability – Such processes thus should not be

delayed by outside influences, e.g. data communications with another process

– Data communications should therefore follow “not obliged to wait for data” protocols

31 Aug 2004 9

Asynchrony in concurrent systems

• Low power elements– In such things as battery powered remote

sensors, timing for data items can depend on two factors

– Some sensors only produce data when they detect changes in the input signal

– Other sensors may produce data when their communication partners request data from them

– Each requires a different approach to data communication synchrony

31 Aug 2004 10

Asynchronous data communication

• A buffer increases scope for asynchrony

writer reader

r

w

31 Aug 2004 11

writer timing domain reader timing domain

ACMs

• An ACM is implemented with shared memory

writer readerACM

temporal divide

31 Aug 2004 12

ACMs

• An ACM can specify the following protocols – Writer may be required to wait for reader– Writer may not be required to wait for reader– Reader may be required to wait for writer– Reader may not be required to wait for writer

• Qualitative asynchrony specifications naturally divides all ACMs into four types

31 Aug 2004 13

Simpson’s Classification of ACMsDestructive read (read can be held up)

Non-destructive read (read cannot be held up)

Destructive write (write cannot be held up)

Signal

(event data)

Pool

(reference data)

Non-destructive write (write can be held up)

Channel

(message data)

Constant

(configuration data)

Other ACM classifications: e.g. L. Lamport, 1986 (safe, regular and atomic registers)

There is symmetry between Pool and Channel but no symmetry between Signal and Constant

31 Aug 2004 14

ACMs

• Consequences of waiting/no waiting– Overwriting – new item superseding a

previous one when no item in an ACM has been read

– Rereading – reading previously read item when no newer one is available

31 Aug 2004 15

Petri net capture of Simpson’s protocols

Signalnon-destr write empty

full

destr write

non-destr write

empty

full

destr read

non-destr write

empty

full

full

destr write non-destr read

destr read

ConstantChannel

Pool

non-destr read

31 Aug 2004 16

Our interpretation

OW-BB

writeread

unread

over-writeread

unread

writeread

unreadread

RR-BBBB

OW-RR-BB

writeread

re-read

read

unread

over-write

write re-read

read

read

Constant is a special case

31 Aug 2004 17

Our interpretation

OW-BB

writeread

unread

over-writeread

unread

writeread

unreadread

RR-BBBB

OW-RR-BB

writeread

re-read

read

unread

over-write

write re-read

read

read

Busy Writer

Lazy Writer

Busy ReaderLazy Reader

31 Aug 2004 18

Our classification of ACMs

RR-BBBBLazy writer = write only if previous read data(write can be held up)

OW-RR-BBOW-BBBusy writer = may over-write unread data(write cannot be held up)

Busy reader = may re-read data already read(read cannot be held up)

Lazy reader = read only previously unread data(read can be held up)

31 Aug 2004 19

S IG N A L FA M ILY P O O L

C H A N N E L FA M ILY B B R R -B B

O W -B B O W -R R -B B

C O N S TA N T

E ven t D a ta R efe ren ce D a ta

C on figu ra tion D a taM essage D a ta

In te ractio n P ro to co ls an d A C M s

D W

D R R B N R R N D R R N B R R

N D RD WW BR BN R RN O W

N D W

W N B

W B

O W

N O W

n 1

n 1n n

n n

R ange o f n

(S ens ib le )

L egend :

n 0 n =

B B

D R

N D W

R N B

W N B

R R

O W

nB ounded B uffe r

D estruct ive R ead ing O b liga to ry

N on D estructive W riting O b lig ato ry

R eader N on B lock ing

W rite r N on B lock ing

R e-R ead ing P e rm itted

O ver-W riting P erm itted

M ax im um N um ber o f I tem s

N on D estructive R ead ing P e rm it ted

D estruct ive W riting P erm itted

W rite r B lock ing P oss ib le

R eader B lock ing P oss ib le

N o R e-R ead ing O b liga to ry

N o O ver-W riting O b liga to ry

31 Aug 2004 20

OW-BBOver-writingNo re-reading

OW-RR-BBOver-writingRe-reading

BBNo over-writingNo re-reading

RR-BBNo over-writingRe-reading

n=0 n=1 n>1

n=0 n=1 n>1

n=0 n=1 n>1

n=0 n=1 n>1

0

Flash Data

0 1 1 n n

Signal Over writing buffer

h/t

0 1 n

h/t

Pool

0

0

1

1

n

n

Rendezvous Channel Bounded Buffer

0 1 n

Constant

Real-time writerLow-power reader

Real-time writerReal-time reader

Low-power writerLow-power reader

Low-power writerReal-time reader

31 Aug 2004 21

Heterogeneously Timed Nets (Hets)(based on MASCOT standard symbols)

31 Aug 2004 22

Hets

A1 A4

2

Time/event/data-drivenData processing elements(active)

31 Aug 2004 23

Hets

C1

C3

Data communication elements(passive) - ACMs

31 Aug 2004 24

The Timing Modes Spectrum

Ana

logu

e

Asy

nchr

onou

s (s

elf-

timed

)

Sin

gle

cloc

k sy

nchr

onou

s

GA

LS

Het

erog

eneo

us

Non-sampled Sampled data

Continuous time Discrete time?

Par

alle

l

Mul

tiple

clo

ck

dom

ains

HE

TS

GA

LA

31 Aug 2004 25

Hets - characteristics

To stop unnecessary processing

Either one input must be non-RR or one output must be non-OW

31 Aug 2004 26

ACMs

• Writer may be required to wait

writer

wunread

read

31 Aug 2004 27

ACMs

• Writer may not be required to wait

writer w

unread

read

ow

31 Aug 2004 28

ACMs

• Reader may be required to wait

reader

r

unread

read

31 Aug 2004 29

ACMs

• Reader may not be required to wait

readerr

rrread

unread

31 Aug 2004 30

ACMs

• No wait on either side

readerr

rrread

writer w

unreadow

31 Aug 2004 31

ACMs

• No wait on either side– “Atomic register” – L. Lamport– “Pool” – Mascot (e.g. H. Simpson)– Connecting together two independently

timed processes (e.g. real-time processes)– Reference data– Clock example– Spiders example

31 Aug 2004 32

ACMs

• May be required to wait on both sides– Traditional buffer in computers– “Channel” – Mascot (e.g. H. Simpson)– Full data continuity (no OW/RR)– Message data– Increasing the size of the buffer increases

quantitative asynchrony, at the expense of latency (true for all ACMs)

– Anti-shock CD player example

31 Aug 2004 33

ACMs

• May be required to wait on one side – Connecting self-motivated timing with

reactive timing– Signal or command data– One side real-time the other low power, etc.– Smartdust example

31 Aug 2004 34

“Slots” and asynchrony• A “slot” is a portion of shared memory of

correct size for containing one data item. – One slot is enough data storage with

conventional mutexing on the data storage (say semaphores)

– One slot is not enough if mutexing on data storage is not acceptable

• Intuitively– At least two slots are needed to avoid the

reader and writer clashing on the same slot

31 Aug 2004 35

ACM implementation• Traditionally ACMs have been

implemented with multi-slot shared memory and control variables:

Reader

Writer

Shared memory

Control variables

ACM

data data

31 Aug 2004 36

ACM implementation• When accessing data, a process is fully

synchronized with a slot in the shared memory

• Any asynchrony between the two processes is concentrated on the control variables, which must be– as small as possible– with atomic setting and reading assumptions

31 Aug 2004 37

ACM implementation• It’s best to use unidirectional control

variables– set and reset from one side and referenced

from the other side or both sides– easily implemented with an asynchronous

latch:

process 1 process 2latchDclk

Q

31 Aug 2004 38

ACM implementation• Even pure synchronization can be

implemented with unidirectional control variables:

pr.2pr.1

pr.2

pr.1x=0

x=1

y=1

y=0

31 Aug 2004 39

ACMs with buffering

ACMs are by definition buffering devices

Buffering of arbitrary size assumed

Lots of work on small buffer size ACMs

We want to look at significant buffer sizes

Is the difference quantitative or qualitative?

31 Aug 2004 40

ACMs with bufferingACMs with buffering

ACMReade

rWriter

Shared memory

data data

Control variables

enough memory to enough memory to contain an arbitrary contain an arbitrary number of itemsnumber of items

31 Aug 2004 41

ACMs with buffering

CC00CC00

CC22CC22

CC11CC11CCnn-1-1CCnn-1-1

nn cells in a ring … … cells in a ring … …nn cells in a ring … … cells in a ring … …

31 Aug 2004 42

RR-BB

Global viewRe-reading

Writing

Reading

Writer waiting

CC00CC00

CC22CC22

CC11CC11CCn-n-11CCn-n-11

RR

WW

31 Aug 2004 43

More buffering:

Increases quantitative asynchrony

No change to qualitative asynchrony

Worse latency

Better data continuity

Less data repetition and replacement

31 Aug 2004 44

ACM implementation• RR-BB: from an interleaving specification

wr0

rd1

rd1

wr0

rd2

rd2 wr0

rd0

rd2

wr1

wr1 rd0

rd2

wr1

rd0

wr2

wr2

rd0

wr2

wr2

rd1

rd1

s0 s1

s0 s1

(wr wait)

(wr wait)

(wr wait)

rereading

cyclic

waiting

31 Aug 2004 45

ACM implementation• RR-BB: adding “silent actions”

12

rd2

11

11

wr0

rd1

wr0 wr2

wr2

rd1

20

rd1

rd1

20

01

01

rd2

rd2

wr0

22

12

rd0

rd0

rd0

wr2

01

22

wr0

wr1

01

12

wr1

rd2

20

20

wr1

wr1

00

00

wr2 12

rd0 s0 s1

s0 s1

prepare to reread

λ: writer silent actions;μ: reader silent actions

31 Aug 2004 46

ACM implementation• RR-BB: synthesis

12

rd2

11

11

wr0

rd1

wr0 wr2

wr2

rd1

20

rd1

rd1

20

01

01

rd2

rd2

wr0

22

12

rd0

rd0

rd0

wr2

01

22

wr0

wr1

01

12

wr1 rd2

20

20

wr1

wr1

00

00

wr2 12

rd0

s0 s1

s0 s1

var w: 0..n-1; r: 0..n-1; initialized sensibly (say r=w-1) and initialize data items in the cells.

writer

wr: write cell w;

ww: wait until r≠w;

w0: w:=(w+1 mod n);

reader

r0: if (r+1 mod n)≠w then r:=(r+1 mod n);

rd: read cell r;

31 Aug 2004 47

ACM implementation• RR-BB: synthesis (modular design)

ReaderWriter

cell memory

w and r

ACM cell i

data data

cell memory

w and r

ACM cell i+1

ReaderWriter

data data

advance advance

31 Aug 2004 48

ACM implementation• RR-BB: synthesis (modular design)

var w: 0..1; r: 0..1; initialized sensibly (one cell has w=1 and one has r=1, all others being 0) and initialize data in the cells.

Writer Reader

wr: write; r0: if wnext=0 then

w0: w:=0; wnext:=1; begin r:=0; rnext:=1;

ww: wait until rnext=0; advance to next end

wa: advance to next; rd: read;

31 Aug 2004 49

ACM analysis/verification• Do ACM algorithms behave the same as

their specifications?

• If not entirely, how good is the approximation?

• The current SOTA is to set up a number of crucial data properties, derive mathematical models of the algorithms highlighting these properties, then verify the models

31 Aug 2004 50

ACM analysis/verification• Data properties of ACMs:

– Data coherence:• ACM specification/description models represent

data items with individual tokens. This implies that individual data items should not be modified or otherwise changed during transit.

– Data loss and repetition:• Not all items from the writer must reach the reader

and the reader may obtain the same item of data more than once (OW and RR).

31 Aug 2004 51

ACM analysis/verification• Data properties of ACMs:

– Data freshness:• In OW ACMs with very

small (0 and 1) buffer size, it is important that the newest item of data available is obtained by the reader. For instance, with an OW ACM with buffer size 1, the place “unread” represents the newest written item at the beginning of a read.

writer

w

unread

read

ow

31 Aug 2004 52

ACM analysis/verification• Data properties of ACMs:

– Data freshness:• Data freshness is important because in OW

ACMs it is not possible to maintain data coherence with a single memory space for each cell.

• At lease two memory spaces (slots) are needed if the writer and the reader need to avoid each other in the shared memory.

• With multiple slots comes the possibility that the reader might obtain a non-fresh item.

31 Aug 2004 53

ACM analysis/verification• Data properties of ACMs:

– Data freshness failure example:

ReaderWriter

slot 2

slot 1

OW-RR-BBdata in

data out

– No data coherence problem and fully asynchronous but no data passed

31 Aug 2004 54

ACM analysis/verification• Data properties of ACMs:

– Data sequencing:• Reader should obtain data items in the same

sequence as the writer delivered them. • Writer writes: 0, 1, 2, 3, 4, 5, …• Reader may read: 1, 3, 4, …• But not: 1, 4, 3, …• This is especially important for buffered ACMs

with FIFO buffer arrangements, but may not be an important issue with OW ACMs with buffer size 0 or 1

31 Aug 2004 55

ACM analysis/verification• ACM modelling:

– Modelling for analysis and verification must highlight the properties being analysed.

– The resulting models must be conducive to a systematic solution.

– We choose to model ACM algorithms with Petri nets and solve these models with reachability techniques.

– This requires that the important properties are highlighted by states and transitions.

31 Aug 2004 56

ACM analysis/verification• ACM modelling: data coherence ―

simultaneous access of the same slot

Writing slot x

Reading slot x

31 Aug 2004 57

ACM analysis/verification• ACM modelling: data freshness

write post

pre read

Write is set to fresh and validother slot is set to not fresh

Clear validexcept fresh

Read should be validread is set to not fresh

31 Aug 2004 58

Exercise• Synthesize an OW-BB with a single cell

– Data coherence required– Data freshness required– Full asynchrony required between writer and

reader, i.e. the only waiting happens when reader waits for a new data item

– How many data slots are needed?– Try two slots first, 40 minutes to find an

algorithm (consult slides 44-46, if you are getting nowhere in 10 minutes, read slide 59)

31 Aug 2004 59

Exercise• Synthesize an OW-BB with a single cell

– Hint slide (30 minutes to find an algorithm)– The following slide may confuse you

wr0

rd1

rd1 wr0 wr1

rd0

wr1

rd0

s0

(rd wait)

wr0 s0

wr1

(rd wait)

31 Aug 2004 60

ACM analysis/verification

• ACM modelling: example algorithm

Single-cell OW-BB with two data slots:var w: 0..1; r: 0..1;

Writer Reader

wg: g:=1; r0: r:=(not)r;

wr: write slot w; rd: wait until w=(not)r and

wg0: g:=0; g=0 then

w0: w:=(not)r; read r;

31 Aug 2004 61

ACM analysis/verification• ACM modelling: a binary control variable

r=1

r=0

w=0

w=1

finishstart

w:=(not)r

31 Aug 2004 62

• ACM modelling: the overall model scheme

W0 subnet

write subnet

read subnet

R0 subnet

w=0/1

r=0/1

Slot_0/1read/unread

31 Aug 2004 63

wr start wr writing 0 end wr w0

rd start rd reading 0 end rd r0

w=0slot 0read

slot 0unread

w=1

r=0

• ACM modelling: one of the slots

31 Aug 2004 64

OR arcs

Example statement :- “a := 0”

a=0

a=1

a=0

a=1

a=2 a=2

a=3a=3

31 Aug 2004 65

wr start wr writing 0 end wr w0

rd start rd reading 0 end rd r0

w=0slot 0read

slot 0unread

w=1

r=0

Re-read 0

• ACM modelling: one of the slots

31 Aug 2004 66

write post

Write is set to fresh and validother slot is set to not fresh

write end fresh and validsub-model

This should appear as an atomic action to the other process

• ACM modelling: assembling the model

31 Aug 2004 67

write end fresh and validsub-model

enable

part of the reader model

• ACM modelling: assembling the model

31 Aug 2004 68

ACM analysis/verification

• Analysis of the example algorithm:

Single-cell OW-BB with two data slots:var w: 0..1; r: 0..1;

Writer Reader

wg: g:=1; r0: r:=(not)r;

wr: write slot w; rd: wait until w=(not)r and

wg0: g:=0; g=0 then

w0: w:=(not)r; read r;

31 Aug 2004 69

Models constructed

Reachability search

Asynchrony requirements

Data coherence

Data freshness The data read is always valid

It is not always marked fresh

• Analysis of the example algorithm:

31 Aug 2004 70

This is due to interleaving and the initial conditions

Write 0 Write 1 Write 0

Read 1

This can be prevented by introducing the guard signal shown in Figure 6

• Analysis of the example algorithm:

31 Aug 2004 71

If a reader cycle immediately follows a writer cyclewhat data does it get?

Write X post

Does the reader read X?

• Analysis of the example algorithm: data latency

31 Aug 2004 72

Data latency

Write X post

w=0r=0

write slot w;w := not r;

r := not r;if* w == r then wait;read slot r;

Write slot 0

w:=not r = 1

pre

r:=not r = 1

w==rtherefore

made to wait

31 Aug 2004 73

Data latency

w=0r=0

write slot w;w := not r;

r := not r;if* w == r then wait;read slot r;

Write X post

Write slot 0

w:=not r = 1

pre

r:=not r = 1

w==rtherefore

made to wait

Write Y post

Write slot 1

w:=not r = 0

Read

This implies 0 capacity

31 Aug 2004 74

Metastability is....

Not being able to decide…

Q

Q

Reset

Set

31 Aug 2004 75

History & Philosophy

• Abu Hamid Ibn Muhammad Ibn Muhammad al-Tusi al-Shafi’i al-Ghazali ~1100– “Suppose two similar dates in front of a man who has a strong

desire for them, but who is unable to take them both. Surely he will take one of them through a quality in him, the nature of which is to differentiate between two similar things”

– He felt that this demonstrated free will

• Jehan Buridan, Rector of Paris University ~1340– Buridan’s Ass (A dog with two bowls?)– “Should two courses be judged equal, then the will cannot break

the deadlock, all it can do is to suspend judgment until the circumstances change, and the right course of action is clear”

– He’s not so sure

31 Aug 2004 76

Does noise affect ?

• Probability of escape from metastability does not change with gaussian noise (Couranz and Wann 1975)

Trajectories

-0.7

-0.5

-0.3

-0.1

0.1

0.3

0.5

0.7

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

Time

Vol

ts

Trajectories

-0.7

-0.5

-0.3

-0.1

0.1

0.3

0.5

0.7

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

Time

Vo

lts

31 Aug 2004 77

Metastable transients

3

1

3

1

3

1

13

31 Aug 2004 78

ACMs

• Metastability in ACMs– In a well-designed ACM, metastability is

restricted to control variables– This is one of the reasons why binary or

ternary control variables are so much better than multi-bit ones

– In a well-designed ACM, metastability in a control variable does not matter so long as it has settled before use

31 Aug 2004 79

ACM applications

• Concurrent real-time systems– Example: Missile systems

• BAe Dynamics, now MBDA• In commercial products• ACMs of various types providing the data

connections between such nodes as processors, sensors, controllers, etc.

• Emphasized real-time requirements (temporal decoupling) and data requirements satisfied by ACMs working under real-time kernels

31 Aug 2004 80

ACM applications

• Concurrent real-time systems– Example: Intelligent distributed CCTV

systems• EU project ADVISOR• Tested on subway/underground monitoring

system• ACMs of various types providing the data

connections between such nodes as cameras and computers

31 Aug 2004 81

ACM applications

• Concurrent real-time systems– Example: Distributed control system

• Motor control system with fast inner (current) and slow outer (position) control loops

• Matlab models derived for various types of ACMs which are used to simulate system with ACMs connecting the components

• OW-RR ACMs especially relevant in control systems as they mimic a direct analogue connection (a wire)

31 Aug 2004 82

ACM applications

• OW-RR ACMs ― digital wires– Reference data, temporal decoupling– Equivalent to:

D/A A/Ddigital data

analogue data

digital data

31 Aug 2004 83

ACM applications

• On-chip systems– Networks on chip

• Based on traditional circuit switch/packet switch network solutions

• Data loss/repetition are treated as abnormalities needing to be “corrected”

• Problems in data transmission normally considered under the QoS umbrella

• If no faults or all faults are corrected the point to point behaviour is equivalent to a BB ACM

31 Aug 2004 84

ACM applications

• On-chip systems– ACMs on chip

• Instead of regarding the BB behaviour as the ideal and any differences as QoS problems, the starting point of ACMs is data requirements

• SOTA directly considers the individual point to point connections at the application level, lacks a general architecture onto which solutions can be mapped

• We propose a shared memory fabric to be used in conjunction with traditional networks

31 Aug 2004 85

ACM applications

• ACMs on chip– Shared memory fabric

• ACMs work better when the physical and logical distance between accessing processes is close

• In on-ship solutions, this indicates a basic fabric of shared memory components between neighbouring nodes (IPs)

• Different types of ACMs can then be mapped onto this shared memory fabric by system designers

• SMF can coexist with a network architecture

31 Aug 2004 86

ACM applications

• ACMs on chip– Shared memory fabric together with

network architecture• Each servicing different data requirements• Remote ACMs can be mapped onto the network

architecture or a combination of local SMF and remote network connections

• ACM data-requirement oriented specifications can be used to supplement QoS-based considerations

31 Aug 2004 87

ACM applications

• ACMs on chip as a complement to NoC

SMF

31 Aug 2004 88

ACMs

• Further reading – http://www.async.org.uk/comfort– http://www.async.org.uk/coherent

31 Aug 2004 89

async.org.uk

• Hardware security (smartcards)

• Asynchronous circuit design– Synthesis, verification, design automation,

visualisation, etc.

• Collaborations with industry– MBDA, Atmel, etc.

• Academic collaborations– Kingston, Manchester, Barcelona, Turino,

etc.

31 Aug 2004 90

Acknowledgements: Acknowledgements:

Delong ShangDelong ShangAlbert KoelmansAlbert KoelmansAlex BystrovAlex BystrovDavid KinnimentDavid KinnimentHugo SimpsonHugo SimpsonEric CampbellEric Campbell