hardware support for acid transactions in...

Arpit Joshi, Vijay Nagarajan, Marcelo Cintra, Stratis Viglas

Hardware Support for ACID Transactions in Persistent Memory

NVMW 2019

Persistent Memory SystemsL1

LLC

Persistent Memory

L1

!2


LLC

Persistent Memory

L1• Persistent Memory

- Non-volatility over the memory bus- Load/Store interface to persistent data

!2


LLC

Persistent Memory



!2

System Crashes


LLC

Persistent Memory



• Crash Consistency- Is the persistent state consistent?- Programming Model: ACID Transactions

!2

System Crashes


LLC

Persistent Memory




!2

System Crashes

“Ensuring failure atomicity for all this computation without failure-atomic transactions is practically infeasible, if not impossible.”

Marathe et al. [HotStorage’17]


LLC

Persistent Memory




!2

System Crashes

“Ensuring failure atomicity for all this computation without failure-atomic transactions is practically infeasible, if not impossible.”

Marathe et al. [HotStorage’17]

How fast can we support ACID?

ACID TransactionsL1

LLC

Persistent Memory

L1

!3

ACID TransactionsL1

LLC

Persistent Memory

L1

!3

Atomic Visibility

ACID TransactionsL1

LLC

Persistent Memory

L1

!3

Atomic Visibility

Atomic Durability

ACID TransactionsL1

LLC

Persistent Memory

L1

!3

Atomic Visibility

Atomic Durability

Locks HTMSTM

ACID TransactionsL1

LLC

Persistent Memory

L1

!3

Atomic Visibility

Atomic Durability

Locks HTMSTM

Check-pointing

H/W Logging

S/WLogging

Atomic Visibility: HTM

!4


• Commercial HTMs [Intel, IBM]

!4

L1 Cache

Cache Line

A = 15

R

B = 20

W

11


• Commercial HTMs [Intel, IBM]- Version Management: read/write sets in

L1 cache

!4

L1 Cache

Cache Line

A = 15

R

B = 20

W

11



L1 cache- Conflict Detection: piggy back on the

coherence protocol

!4

L1 Cache

Cache Line

A = 15

R

B = 20

W

11




coherence protocol- Commit: make updates non-speculative

!4

L1 Cache

Cache Line

A = 15

R

B = 20

W




coherence protocol- Commit: make updates non-speculative- Abort: invalidate write set

!4

L1 Cache

Cache Line R

B = 20

W




coherence protocol- Commit: make updates non-speculative- Abort: invalidate write set

!4

L1 Cache

Cache Line R

B = 20

W

Write-sets in commercial HTMs limited by the size of the L1 cache.

Atomic Durability: Logging

!5


• Logging for durability [Doshi’16, Joshi’17, Shin’17, Ogleari’18]

!5

Persistent Memory

In-place Values

A = 10B = 20C = 30


• Logging for durability [Doshi’16, Joshi’17, Shin’17, Ogleari’18]- Write a log entry for every update

!5

Persistent Memory

In-place Values

A = 10B = 20C = 30

Transaction Log

A = 15B = 25


• Logging for durability [Doshi’16, Joshi’17, Shin’17, Ogleari’18]- Write a log entry for every update- Commit: Update the values in-place

!5

Persistent Memory

In-place Values

A = 15B = 25C = 30

Transaction Log


• Logging for durability [Doshi’16, Joshi’17, Shin’17, Ogleari’18]- Write a log entry for every update- Commit: Update the values in-place- Abort: Undo any in-place updates

!5

Persistent Memory

In-place Values

A = 15B = 25C = 30

Transaction Log

A = 10B = 20


• Logging for durability [Doshi’16, Joshi’17, Shin’17, Ogleari’18]- Write a log entry for every update- Commit: Update the values in-place- Abort: Undo any in-place updates

!5

Persistent Memory

In-place Values

A = 15B = 25C = 30

Transaction Log

A = 10B = 20

In-place updates in the critical path of commit High memory write bandwidth requirement

ACID = HTM + Logging

Goals:- Support fast commits - Minimise memory bandwidth consumption- Extend the supported transaction size- Maintain the simplicity of commercial HTMs

!6

DHTM: Durable Hardware Transactional Memory

L1

LLC

Persistent Memory

L1

!7

Log Writes

Commercial HTM + Hardware Redo Log


L1

LLC

Persistent Memory

L1

!7

Log Writes

Commercial HTM + Hardware Redo Log- H/W Redo Log + Log Buffer

Reduced memory bandwidth Fast commits


L1

LLC

Persistent Memory

L1

!7

Log Writes

Commercial HTM + Hardware Redo Log- H/W Redo Log + Log Buffer

Reduced memory bandwidth Fast commits

- H/W Log + Sticky State Extended transaction size to the LLC Simplicity of commercial HTM


L1

LLC

Persistent Memory

L1

!7

Log Writes

!8

L1

LLC

Persistent Memory

L1

Log Writes

DHTM: Log Buffer

!8

L1

LLC

Persistent Memory

L1• Redo Log Bandwidth Problem

Log Writes

DHTM: Log Buffer

!8

L1

LLC

Persistent Memory


- write a log entry for every storeLog Writes

DHTM: Log Buffer

!8

L1

LLC

Persistent Memory


- write a log entry for every store- multiple stores create multiple log entriesLog Writes

DHTM: Log Buffer

!8

L1

LLC

Persistent Memory


- write a log entry for every store- multiple stores create multiple log entries

• Solution: Log Buffer

Log Writes

DHTM: Log Buffer

!8

L1

LLC

Persistent Memory



• Solution: Log Buffer - track cache lines being modified

Log Writes

DHTM: Log Buffer

!8

L1

LLC

Persistent Memory



• Solution: Log Buffer - track cache lines being modified- multiple writes coalesced in a log entry

Log Writes

DHTM: Log Buffer

!8

L1

LLC

Persistent Memory



• Solution: Log Buffer - track cache lines being modified- multiple writes coalesced in a log entry - log entry written to persistent memory on eviction

from log buffer

Log Writes

DHTM: Log Buffer

DHTM: Transaction States

!9


!9

Active

Begin Transaction


!9

Active Commit

Begin Transaction

End Transaction&

Log RecordsPersisted


!9

Active Commit CommitComplete

Begin Transaction

End Transaction&


In-place DataPersisted


!9

Active Commit CommitComplete

Abort

Begin Transaction

End Transaction&


In-place DataPersisted

Conflict

DHTM: Commit ExampleL1 Cache

Cache Line R W

Persistent Memory

In-place Values

A = 15B = 25A = 10B = 20C = 30

Transaction Log

A = 10B = 20

State

Log Buffer Begin_Transaction

Write (A=15)

Read (B)

Write (B=25)

End_Transaction

!10

Active


Cache Line R W

Persistent Memory

In-place Values

A = 15B = 25A = 10B = 20C = 30

Transaction Log

A = 10B = 20

State


Write (A=15)

Read (B)

Write (B=25)

End_Transaction

!10

ActiveActive


Cache Line R W

Persistent Memory

In-place Values

A = 15B = 25A = 10B = 20C = 30

Transaction Log

A = 10B = 20

State


Write (A=15)

Read (B)

Write (B=25)

End_Transaction

!10

A = 15 1A

ActiveActive


Cache Line R W

Persistent Memory

In-place Values

A = 15B = 25A = 10B = 20C = 30

Transaction Log

A = 10B = 20

State


Write (A=15)

Read (B)

Write (B=25)

End_Transaction

!10

A = 15 1A

A = 15 1AB = 20 1

ActiveActive


Cache Line R W

Persistent Memory

In-place Values

A = 15B = 25A = 10B = 20C = 30

Transaction Log

A = 10B = 20

State


Write (A=15)

Read (B)

Write (B=25)

End_Transaction

!10

A = 15 1A

A = 15 1AB = 20B = 25 11 1

A = 15

B

ActiveActiveCommit


Cache Line R W

Persistent Memory

In-place Values

A = 15B = 25A = 10B = 20C = 30

Transaction Log

A = 10B = 20

State


Write (A=15)

Read (B)

Write (B=25)

End_Transaction

!10

A = 15 1A

A = 15 1AB = 20B = 25 11 1

A = 15

B

B = 25

B = 25

A = 15

Commit

1

ActiveActiveCommit


Cache Line R W

Persistent Memory

In-place Values

A = 15B = 25A = 10B = 20C = 30

Transaction Log

A = 10B = 20

State


Write (A=15)

Read (B)

Write (B=25)

End_Transaction

!10

A = 15 1A

A = 15 1AB = 20B = 25 11 1

A = 15

B

B = 25

B = 25

A = 15

CommitA = 15B = 25

Complete

CommitComplete

CommitB = 25

1

DHTM: Supporting Overflow

!11


• Problems with Overflow:

!11


• Problems with Overflow:- Version Management:

- global operation on write-set on a commit/abort- overhead infeasible in larger caches (beyond L1)

!11


• Problems with Overflow:- Version Management:

- global operation on write-set on a commit/abort- overhead infeasible in larger caches (beyond L1)

- Conflict Detection: - additional metadata to detect conflicts- increased complexity due to NACK based protocols

!11


!12


!12

• Solution


!12

LLC

Persistent Memory

• Solution- Version Management:

- Overflow List


!12

LLC

Persistent Memory

Overflow List

C

AB


- Overflow List


!12

LLC

Persistent Memory

Overflow List

C

AB


- Overflow List- Conflict Detection:

- maintain sticky state on overflow (similar to LogTM)

- avoid NACK by restricting overflow to LLC

Evaluation

• System Configuration- We evaluate an 8-core machine with a 2-level cache hierarchy- HTM’s implement (first) writer wins conflict resolution policy

!13

Atomic Visibility Atomic Durability

ATOM Locks Hardware Undo Log

LogTM+ATOM HTM (LogTM) Hardware Undo Log

DHTM HTM Hardware Redo Log (Log Buffer)

Evaluation

!14

Evaluation

!14

1

1.25

1.5

1.75

2

queue hash sdg sps btree rbtree gmean

ATOM LogTM+ATOM DHTM

Evaluation

!14

1

1.25

1.5

1.75

2



26%

Evaluation

!14

1

1.25

1.5

1.75

2



17%

Conclusion

• Persistent memory systems require crash consistency• ACID Transactions: widely understood crash consistency mechanism

• DHTM: ACID transactions in hardware- Atomic Visibility: commercial HTM- Atomic Durability: bandwidth optimized hardware redo log- Leverage hardware logging to extend transaction size unto LLC

!15

Arpit Joshi, Vijay Nagarajan, Marcelo Cintra, Stratis Viglas

Hardware Support for ACID Transactions in Persistent Memory

NVMW 2019

hardware support for acid transactions in...

Documents