andy rudoff (intel data center group) september 5th, 2019 · 2019-09-24 · file system application...
TRANSCRIPT
![Page 1: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/1.jpg)
Andy Rudoff (Intel Data Center Group)
September 5th, 2019
![Page 2: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/2.jpg)
SPDK, PMDK & Vtune™ Summit
Agenda
Goals
History
Current State of PMDK
Future
2
![Page 3: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/3.jpg)
Goals of PMDK
3
![Page 4: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/4.jpg)
SPDK, PMDK & Vtune™ Summit 4
Back When We Heard:“Persistent memory is coming…”Byte-addressable, use it like memory
But it is persistent
Actually had been shipping from some vendors
Later named NVDIMM-N
Small capacity 16-32 GB
All access was through a driver interface when I first started looking at them
![Page 5: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/5.jpg)
SPDK, PMDK & Vtune™ Summit 5
Persistent memory First Steps…Step 1: how should it be exposed to applications
How to name it, re-attach to it
How to enforce permissions
How to back it up, manage it
And some less technical goals, but just as important
– Represent the interests of the ISVs
– Avoid vendor lock-in to a product-specific API
– As an Intel employee, acknowledge that Intel-specific doesn’t work here
Headed to SNIA…
Persistent Memory
UserSpace
KernelSpace
Standard
File API
NVDIMM Driver
Application
File System
ApplicationApplication
Standard
Raw Device
Access
Storage File Memory
Load/Store
Management Library
Management UI
Standard
File API
Mgmt.
PM-AwareFile System
MMU
Mappings
![Page 6: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/6.jpg)
SPDK, PMDK & Vtune™ Summit 6
Ancient historyJune 2012
Formed the NVM Programming TWG
Immediate participation from key OSVs, ISVs, IHVs
January 2013
Held the first PM Summit (actually called “NVM Summit”)
July 2013
Created first GitHub thought experiments (“linux-examples”)
January 2014
TWG published rev 1.0 of the NVM Programming Model
![Page 7: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/7.jpg)
SPDK, PMDK & Vtune™ Summit 7
SNIA Model Success… and then what?!Open a pmem file on a pmem-aware file system
Map it into your address space
Okay, you’ve got a pointer to 3TB of memory, have fun!
The model is necessary, but not sufficient for an easy to program resource
Gathering requirements yielded fairly obvious top priorities:
Need a way to track pmem allocations (like malloc/free, but pmem-aware)
Need a way to make transactional updates
Need a library of pmem-aware containers: lists, queues, etc.
Need to make pmem programming not so error-prone
![Page 8: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/8.jpg)
SPDK, PMDK & Vtune™ Summit 8
The first few tries
// volatilechar *ptr = malloc(size);
// persistentchar *ptr = pm_malloc(size);
// crash before using ptr => pmem leak!
NAME libpmemalloc -- Persistent Memory malloc-like library
SYNOPSIS #include <pmemalloc.h> cc ... -lpmemalloc
void *pmemalloc_init(const char *path, size_t size); void *pmemalloc_static_area(void *pmp); void *pmemalloc_reserve(void *pmp, size_t size); void pmemalloc_persist(void *pmp, void **parentp_,
void *ptr_); void pmemalloc_onactive(void *pmp, void *ptr_,
void **parentp_, void *nptr_); void pmemalloc_onfree(void *pmp, void *ptr_,
void **parentp_, void *nptr_); void pmemalloc_activate(void *pmp, void *ptr_); void pmemalloc_free(void *pmp, void *ptr_); void pmemalloc_check(const char *path);
PMEM(pmp, ptr_)
![Page 9: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/9.jpg)
SPDK, PMDK & Vtune™ Summit 9
GOALSMake persistent programming easier
Especially allocation, transactions, atomic operations
Validate thoroughly to save developers implementation time
Performance tune it, improving over time
Later we realized we needed additional goals…
Help simplify RAS (bad block tracking, recovery)
Create new libraries for new use cases as they come up
Track new hardware features (example: MOVDIR64B)
![Page 10: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/10.jpg)
SPDK, PMDK & Vtune™ Summit 10
The result… PMDKPMDK Provides a Menu of Libraries
Developers pull in just what they need
– Transaction APIs
– Persistent memory allocators
Instead of re-inventing the wheel
– PMDK libraries are fully validated
– PMDK libraries are performance tuned
PMDK Provides Tools for Developers
PMDK is Open Source and Product-Neutralpmem
UserSpace
KernelSpace
Application
Load/StoreStandard
File API
PM-AwareFile System
MMU
Mappings
PMDKLibraries
![Page 11: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/11.jpg)
SPDK, PMDK & Vtune™ Summit 11
Current State of PMDKCore libraries, roughly ten of them, in PMDK repo on GitHub
Over 8000 commits over a period of about five years
Dozens of users that we know about
– Some open source
– Some closed source
– Some code stealers (which we encourage)
Most intense activity has been on libpmemobj, the most flexible library
Team took over maintenance of libmemkind
For volatile use cases
Lots of interesting additions since the initial set of libraries…
![Page 12: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/12.jpg)
12
100.00%
167.49%199.07%
377.12%351.09%
511.33% 510.59%549.66%
1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7
LIBPMEMOBJ RELATIVE PERFORMANCE ACROSS VERSIONS
(B-TREE BENCHMARK)
![Page 13: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/13.jpg)
SPDK, PMDK & Vtune™ Summit 13
PMDK EvolutionNew libraries based on use cases and customer feedback (see talks on many of these!)
Java support
C++ support
– Some of the most interesting & challenging work in this space
– Lots more in this summit about C++
Libpmemkv
libvmemcache
Tools support (VTune, pmemcheck, pmreorder, etc.)
![Page 14: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/14.jpg)
SPDK, PMDK & Vtune™ Summit 14
PMDK FutureWe’re not done!
As more use cases emerge, decide if current libraries cover them
– Invent new libraries when it makes sense
Get community more engaged
– So far, only a few pull requests from outside PMDK team
– Use SPDK as a model!
Continue to tune, enhance, refine what we have
– Example: more C++ containers, better C++ performance
– Example: support more languages
![Page 15: Andy Rudoff (Intel Data Center Group) September 5th, 2019 · 2019-09-24 · File System Application Raw Device Access Storage File Memory Load/Store Management Library Management](https://reader033.vdocuments.mx/reader033/viewer/2022042217/5ec0f28861ca0e05f87c4caa/html5/thumbnails/15.jpg)