tomasz kapela, @tomaszkapela linuxcon europe, berlin 2016...the fancy typed pointer...

Tomasz Kapela, @tomaszkapela

LinuxCon Europe, Berlin 2016

Hype

3

The NVM programming model

The solution

5

NVMLOpen source set of libraries available at www.github.com/pmem/nvml

Library Description

libpmem The basic library, flush to persistence

libvmem Volatile Memory Allocator

libvmmalloc Transparent use of libvmem

libpmemblk Persistent memory carved into blocks

libpmemlog Log file (append-mostly)

libpmemobj Transactional Object Store

6

libpmemobj

Application

libpmem

Load/Store

memory-mapped pmem

libpmemobj

transactionsatomic

operations

allocatorlists

locks

API API API

Pointers by libpmemobj

8

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

9

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

10

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

11

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

12

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

Type declarations necessary

13

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

14

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

Type declarations necessary

A lot of macros to make this feasible

15

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

16

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

Type declarations necessary

A lot of macros to make this feasible

A lot of macros to handle the TOID

17

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

18

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

Type declarations necessary

A lot of macros to make this feasible

A lot of macros to handle the TOID

– Why?

19

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

20

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

Type declarations necessary

A lot of macros to make this feasible

A lot of macros to handle the TOID

– Why?

Is it usable?

21

The fancy typed pointer – TOID

Type safety introduced to a fancy based pointer.

Type declarations necessary

A lot of macros to make this feasible

A lot of macros to handle the TOID

– Why?

Is it usable?

Addressing the pain points

23

The fancy typed pointer – persistent_ptr

Encapsulate type info into a smart pointer.

No explicit type declaration

24

The fancy typed pointer – persistent_ptr

Encapsulate type info into a smart pointer.

No explicit type declaration

– typeid(T).hash_code()

25

The fancy typed pointer – persistent_ptr

Encapsulate type info into a smart pointer.

No explicit type declaration

– typeid(T).hash_code()

No more macros!

26

The fancy typed pointer – persistent_ptr

Encapsulate type info into a smart pointer.

27

The fancy typed pointer – persistent_ptr

Encapsulate type info into a smart pointer.

No explicit type declaration

– typeid(T).hash_code()

No more macros!

Mimics a well known construct

– std::shared_ptr

Now is it usable?

28

The fancy typed pointer – persistent_ptr

Encapsulate type info into a smart pointer.

29

The fancy typed pointer – persistent_ptr

Encapsulate type info into a smart pointer.

30

The fancy typed pointer – persistent_ptr

Known issues

No polymorphic types allowed in persistent memory

– Vtables rebuild issue

typeid(T).hash_code()

– Returns an unspecified value, except that within a single execution of the program, it shall return the same value for any two type_info objects which compare equal.

Transactions by libpmemobj

32

Transactions

Consistency critical roll-back transactions.

ACID like properties

33

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

34

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

35

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

36

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

Macros to make this usable

37

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

Macros to make this usable

38

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

Macros to make this usable

39

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

Macros to make this usable

A bit error prone

– Even with the macros

40

Transactions

Consistency critical roll-back transactions.

41

Transactions

Consistency critical roll-back transactions.

ACID like properties

„C” exceptions

Macros to make this usable

A bit error prone

– Even with the macros

Snapshotting

42

Transactions

Consistency critical roll-back transactions.

Addressing the pain points

44

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

45

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

46

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

– The problem with std::uncaught_exception

47

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

– The problem with std::uncaught_exception

– CppCon2015/Declarative Control Flow - Andrei Alexandrescu

48

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

– The problem with std::uncaught_exception

– Still can’t throw an exception though

49

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

– The problem with std::uncaught_exception

– Still can’t throw an exception though

The lambda version

50

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

– The problem with std::uncaught_exception

– Still can’t throw an exception though

The lambda version

51

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

– The problem with std::uncaught_exception

– Still can’t throw an exception though

The lambda version

52

Transactions

Making life easier with RAII and lambdas.

Manual and automatic transaction

– The problem with std::uncaught_exception

– Still can’t throw an exception though

The lambda version

– Self contained

– Somewhat ugly looking

Using the containers

54

Persistent memory containers

A proof of concept under way.

Targeting libc++ and libstdc++

55

Persistent memory containers

A proof of concept under way.

Targeting libc++ and libstdc++

56

Persistent memory containers

A proof of concept under way.

Targeting libc++ and libstdc++

– Libc++ is mostly working

– Libstdc++ needs a bit more work

57

Persistent memory containers

A proof of concept under way.

Targeting libc++ and libstdc++

– Libc++ is mostly working

– Libstdc++ needs a bit more work

58

Persistent memory containers

A proof of concept under way.

Targeting libc++ and libstdc++

– Libc++ is mostly working

– Libstdc++ needs a bit more work

Main pros

– It’s for free! (well, almost)

– Familiar interface

– Well tested implementation of containers

59

Persistent memory containers

What are the known issues

Internally held data is not „aware” of persistence

T* used instead of allocator_traits::pointer

Layout versioning

Current implementation works only on x86_64

61

Handy links

The allocator branch:

https://github.com/tomaszkapela/nvml/tree/cpp-allocator

Libpmemobj++ documentation:

http://pmem.io/nvml/cpp_obj/

Pmem Google Groups page:

https://groups.google.com/forum/#!forum/pmem