Treadmarks: Distributed Shared Treadmarks: Distributed Shared Memory on Standard Memory on Standard

Workstations and Operating Workstations and Operating SystemsSystems

P. Keleher, A. Cox, S. Dwarkadas, and W. Zwaenepoel The Winter Usenix Conference

1994

2008-22952Jun Lee

DSM (distributed shared DSM (distributed shared memory)memory) A software system for parallel

computation• Shares distributed memories• Easier programming

−Provide a single global address space

DSM (distributed shared DSM (distributed shared memory)memory) No widely available DSM implementations• In-house research platforms• Kernel modifications• Poor performance

−Imitating consistency protocols of hardware−False sharing

TreadmarksTreadmarks

Objectives• Commercially available workstations and

OS−Standard Unix system on DECstation

• Efficient user-level DSM implementation−Reduce communication overhead

Design• LRC (lazy release consistency)• Multiple writer protocol• Lazy diff creation

Consistency protocol (SC)Consistency protocol (SC)

Sequential Consistency• Every write visible “immediately”• Single writer

P0 P1

R(a):0

W(a):1

R(a):1

Consistency protocol (SC)Consistency protocol (SC)

Sequential Consistency• Every write visible “immediately”• Single writer

P0 P1

R(a):0

W(a):1

R(a):?

R(a):1

Big problem with page size granularity

Page X

Consistency protocol (SC)Consistency protocol (SC)

Sequential Consistency• Every write visible “immediately”• Single writer

W(x0):a W(x1):b a

W(x2):c

W(x3):d

P0 P1

a

Page X

bb c c d

False sharing

Consistency protocol (RC)Consistency protocol (RC)

Release Consistency• Relaxed memory consistency model

−delay making its changes visible to other processors until certain synchronization accesses occurs

• Synchronization points−Acquire(), Release() (similar to locks,

barriers)

• Two types−ERC (eager), LRC (lazy)

Consistency protocol (RC)Consistency protocol (RC)

Release Consistency• Acquire() and release() are sequentially

consistent−Release() is performed after all previous

operations have completed−Operations are performed after previous

acquire() have been performed

• Acquire() and release() pair between conflicting accesses−SC and RC produce the same results.

Consistency protocol (RC)Consistency protocol (RC)

ERC• Write information is delivered at the

release pointP0 P1

Acquire(L)

R(a):0

W(a):1

Release(L)

Acquire(L)

R(a):?

Release(L)

R(a):1

Write Notice

Consistency protocol (RC)Consistency protocol (RC)

ERC• Write information is delivered at the

release pointP0 P1

Acquire(L)

R(a):0

W(a):1

Release(L)

Acquire(L)

R(a):1

Release(L)

Acquire(K)

Release(K)

Consistency protocol (RC)Consistency protocol (RC)

LRC• The delivery is postponed until the

acquire• Fewer messages than ERCP0 P1

Acquire(L)

R(a):0W(a):1

Release(L)Acquire(L)

R(a):?

Release(L)

R(a):1

Consistency protocol (RC)Consistency protocol (RC)

ERC vs. LRC

P0 P1

Acquire(L) R(a):0

W(a):1

Release(L)

P2

R(a):0

P3

R(a):0

Acquire(L)

Release(L)

R(a):1

ERC

Consistency protocol (RC)Consistency protocol (RC)

ERC vs. LRC

P0 P1

Acquire(L) R(a):0

W(a):1

Release(L)

P2

R(a):0

P3

R(a):0

Acquire(L)

Release(L)

R(a):1

ERC

P0 P1

Acquire(L) R(a):0

W(a):1

Release(L)

P2

R(a):0

P3

R(a):0

Acquire(L)

Release(L)

R(a):1

LRC

Page X

Multiple writer protocolMultiple writer protocol

Page X

W(x0):a W(x1):b

W(x2):c

W(x3):?

P0 P1

a bc

Acquire(L)

Release(L)

Acquire(L)

a c

W(x0):a W(x1):b

W(x2):c

W(x3):d

P0 P1a

Page X

bc

Page X

abcd

False sharing

Page X

Multiple writer protocolMultiple writer protocol

Page X

W(x0):a W(x1):b

W(x2):c

W(x3):?

P0 P1

a bc d

W(x0):a W(x1):b

W(x2):c

W(x3):d

P0 P1a

Page X

bc

Page X

abcd

False sharing

Acquire(L)

Release(L)

Acquire(L)

Release(L)

W(x3):d

a c

Page X

Twin and DiffTwin and Diff

Page X

W(x0):a W(x1):b

W(x2):c

W(x3):?

P0 P1

a bc d

Acquire(L)

Release(L)

Acquire(L)

Release(L)

W(x3):d

Twin X Twin XDiff

a c diff

a c

Page X

Twin and DiffTwin and Diff

Page X

W(x0):a W(x1):b

W(x2):c

W(x3):?

P0 P1

a bc d

Acquire(L)

Release(L)

Acquire(L)

Release(L)

W(x3):d

Twin X Twin XDiff

a c diff

a c

Twin X

ba cDiff

b diff

a c diffinterval

ImplementationImplementation

Etc.Etc.

Lock & barrier• Statically assigned manager

Garbage collection• reclaim the space used by write notice

records, interval records, and diffs• Triggered when the free space drops

below a threshold

EvaluationEvaluation

Experimental Environment• 8 DECstation-5000/240• connected to a 100-Mbps ATM LAN and a 10-

Mbps Ethernet

Applications• Water – molecular dynamics simulation • Jacobi – Successive Over-Relaxation• TSP – branch & bound algorithm to solve the

traveling salesman problem• Quicksort – using bubblesort to sort subarray of

less than 1K element• ILINK – genetic linkage analysis

EvaluationEvaluation

Speedup

Execution statistics

EvaluationEvaluation

Execution time breakdown

EvaluationEvaluation

Unix overhead breakdown TreadMarks overhead breakdown

EvaluationEvaluation

Execution time breakdown for Water

EvaluationEvaluationERC vs. LRC

Speedup Message rate

EvaluationEvaluationERC vs. LRC

Data rate Diff creation rate

ImplementationImplementation

. . .

. . .

. . .

pages

time stamp0 0 0

. . .

. . .

. . .

pages

time stamp0 0 0

Acq(L)

P0 side P1 side

P P

ImplementationImplementation

. . .

. . .

. . .

pages

time stamp1 0 0

. . .

. . .

. . .

pages

time stamp0 0 0

Acq(L)

P0 side P1 side

W(a)

Rel(L)a W(b) b

twin twin

P0W.N

P1W.N

P P

ImplementationImplementation

. . .

. . .

. . .

pages

time stamp2 0 0

. . .

. . .

. . .

pages

time stamp0 0 0

Acq(L)

P0 side P1 side

W(a)

Rel(L)a

Acq(L)

W(b) b

twin twin

P0W.N

P1W.N

ImplementationImplementation

. . .

. . .

. . .

pages

time stamp2 0 0

. . .

. . .

. . .

pages

time stamp0 0 0

Acq(L)

P0 side P1 side

W(a)

Rel(L)a

Acq(L)

W(b) b

twin twin

P0W.N

P0

1 0 0

P1W.N

ImplementationImplementation

. . .

. . .

. . .

pages

time stamp2 0 0

. . .

. . .

. . .

pages

time stamp1 1 0

Acq(L)

P0 side P1 side

W(a)

Rel(L)a

Acq(L)

W(c)

W(b) b

twin twin

P0W.N

P0W.N

P1

bdiff

adiff

P

a

W.N

P0 P1

1 0 0

ImplementationImplementation

. . .

. . .

. . .

pages

time stamp2 0 0

Acq(L)

P0 side P1 side

W(a)

Rel(L)a

Acq(L)

W(c)

Rel(L)

W(b)P0W.N

adiff

P

. . .

. . .

. . .

pages

time stamp1 1 0

b P0W.N

P1

bdiff

adiff

aW.N

P0 P1

1 0 0

c P1W.N

P

twinba

ImplementationImplementation

. . .

. . .

. . .

pages

time stamp2 0 0

Acq(L)

P0 side P1 side

W(a)

Rel(L)a

Acq(L)

W(c)

Rel(L)

W(b)P0W.N

adiff

P

. . .

. . .

. . .

pages

time stamp1 2 0

b P0W.N

P1

bdiff

adiff

aW.N

P0 P1

1 0 0

c P1W.N

twinba

ImplementationImplementationP1 side P2 side

P0 P1

1 0 0

. . .

. . .

. . .

pages

time stamp0 0 0

P

Acq(L). . .

. . .

. . .

pages

time stamp1 2 0

b P0W.N

P1

bdiff

adiff

aW.N

c P1W.N

twinba

0 0 01 1 0

ImplementationImplementationP1 side P2 side

P0 P1

1 0 0

. . .

. . .

. . .

pages

time stamp1 1 1

P

Acq(L). . .

. . .

. . .

pages

time stamp1 2 0

b P0W.N

P1

bdiff

adiff

aW.N

c P1W.N

twinba

0 0 01 1 0

P0W.N

P1W.N

P1W.N

Thank you !Thank you !Any questions?