programming emerging storage interfaces€¦ · abstract handle to your controller / namespace...
TRANSCRIPT
![Page 1: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/1.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 1
Programming Emerging Storage Interfaces
Simon A. F. LundSamsung / SSDR
![Page 2: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/2.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 2
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
![Page 3: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/3.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 3
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
Command (sq-entry)
Response (cq-entry)
![Page 4: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/4.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 4
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
Command (sq-entry)
Response (cq-entry)
![Page 5: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/5.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 5
Programming Emerging Storage Interfaces
Block Storage Zoned Block Storage Object Storage Computational Storage
Command (sq-entry)
Response (cq-entry)
![Page 6: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/6.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 6
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage
![Page 7: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/7.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 7
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage Setup virtual NVMe devices
![Page 8: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/8.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 8
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage Setup virtual NVMe devices User Space tools and libraries
![Page 9: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/9.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 9
Programming Emerging Storage Interfaces
Host responsibilities for Zoned Block Storage Setup virtual NVMe devices User Space tools and libraries Example library and tool usage
![Page 10: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/10.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 10
Host ResponsibilitiesFrom Open-Channel to ZNS
![Page 11: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/11.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 11
Host Responsibilities
Device media represented as Physical blocks (OCSSD 1.2) Chunks (OCSSD 2.0 + Denali) Zones (Zoned Namespaces)
![Page 12: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/12.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 12
Host Responsibilities
Device media represented as Physical blocks (OCSSD 1.2) Chunks (OCSSD 2.0 + Denali) Zones (Zoned Namespaces)
Nomenclature: zone
![Page 13: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/13.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 13
Host Responsibilities: Zones
Zone Layout
![Page 14: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/14.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 14
Host Responsibilities: Zones
Zone Layout Zone Attributes and Condition
![Page 15: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/15.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 15
Host Responsibilities: Zones
Zone Layout Zone Attributes and Condition Zone Constraints
1. Write contiguously (within a zone)2. Reset before write (again)
![Page 16: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/16.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 16
Host Responsibilities: Zone Layout
Block Storage How many LBAs Size of an LBA
![Page 17: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/17.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 17
Host Responsibilities: Zone Layout
Block Storage How many LBAs Size of an LBA
Zoned Block Storage How many Zones Attributes and Condition
![Page 18: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/18.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 18
Host Responsibilities: Zone Attributes
![Page 19: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/19.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 19
Host Responsibilities: Zone Attributes
![Page 20: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/20.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 20
Host Responsibilities: Zone Write
![Page 21: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/21.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 21
Host Responsibilities: Zone Write
![Page 22: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/22.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 22
Host Responsibilities: Zone Write
![Page 23: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/23.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 23
Host Responsibilities: Zone Write
![Page 24: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/24.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 24
Host Responsibilities: Zone Write
![Page 25: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/25.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 25
Host Responsibilities: Zone Write
![Page 26: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/26.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 26
Host Responsibilities: Zone Write QD > 1
![Page 27: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/27.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 27
Host Responsibilities: Zone Write QD > 1
![Page 28: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/28.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 28
Host Responsibilities: Zone Write QD > 1
![Page 29: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/29.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 29
Host Responsibilities: Zone Write QD > 1
![Page 30: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/30.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 30
Host Responsibilities: Zone Write QD > 1
![Page 31: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/31.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 31
Host Responsibilities: Zone Write QD > 1
![Page 32: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/32.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 32
Host Responsibilities: Zone Write QD > 1
Synchronize:Wait for a write to WP finish before writing to WP + k
Submission Order != Execution / Completion Order
![Page 33: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/33.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 33
Host Responsibilities: Zone Append
![Page 34: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/34.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 34
Host Responsibilities: Zone Append
![Page 35: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/35.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 35
Host Responsibilities: Zone Append
![Page 36: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/36.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 36
Host Responsibilities: Zone Append
![Page 37: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/37.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 37
Host Responsibilities: Zone Append
![Page 38: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/38.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 38
Host Responsibilities: Zone Append
![Page 39: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/39.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 39
Host Responsibilities: Zone Append
![Page 40: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/40.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 40
Host Responsibilities: Append QD > 1
![Page 41: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/41.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 41
Host Responsibilities: Append QD > 1Where is my data?
![Page 42: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/42.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 42
Host Responsibilities: Append QD > 1Where is my data?Read the completion entry for location
Response (cq-entry)
![Page 43: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/43.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 43
Host Responsibilities: Zone Reset
![Page 44: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/44.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 44
Host Responsibilities: Zone Reset
![Page 45: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/45.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 45
Host Responsibilities
How to manage this as storage developer? You need to get retrieve information on Zone Layout Zone Attributes and Condition
Read, Write, Append, and Reset commands
![Page 46: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/46.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 46
User Space tools and libraries
![Page 47: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/47.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 47
Open-Source Ecosystem: OS Support
UNIX-like OS: everything is a file
![Page 48: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/48.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 48
Open-Source Ecosystem: OS Support
UNIX-like OS: everything is a file System Calls ioctl(), read(), write(), pread(), pwrite(), etc. aio_read(), aio_write() io_uring_prep_readv / io_uring_submit
![Page 49: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/49.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 49
Open-Source Ecosystem: OS Support
UNIX-like OS: everything is a file System Calls ioctl(), read(), write(), pread(), pwrite(), etc. aio_read(), aio_write() io_uring_prep_readv / io_uring_submit
Wrapped in libraries libc, libaio, liburing
![Page 50: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/50.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 50
Open-Source Ecosystem: OS Support
Pros: General block storage infrastructure Efficient async R/W with io_uring / liburing
Cons: Syncronous ioctl() interface Limited control over command construction
![Page 51: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/51.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 51
Open-Source Ecosystem: OS Bypass
API: Everything is a function call to opaque*
![Page 52: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/52.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 52
Open-Source Ecosystem: OS Bypass
API: Everything is a function call to opaque* Driver in User Space Intel SPDK libnvme (SPDK without DPDK)
![Page 53: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/53.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 53
Open-Source Ecosystem: OS Bypass
API: Everything is a function call to opaque* Driver in User Space Intel SPDK libnvme (SPDK without DPDK)
Driver in Kernel access from User Space NVMe-Direct
![Page 54: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/54.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 54
Open-Source Ecosystem: OS Bypass
Pros: Full control over command construction Efficient async interface for ANY command
Cons: Controller detachment from kernel Non-trivial controller sharing
Demo detach
![Page 55: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/55.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 55
Open-Source Ecosystem: tools
nvme-cli Built on Linux ioctl() Limited port for FreeBSD ioctl() Port built on SPDK
nvmecontrol FreeBSD base system, built on ioctl()
![Page 56: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/56.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 56
Virtual NVMe DevicesSetup, usage, and modification with
![Page 57: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/57.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 57
QEMU: The Quick Emulator
A generic machine emulator and virtualizer ia32, x86_64, mips, sparc, arm, risc-v KVM-client e.g. using Intel VT-x
Includes a huge collection of emulated devices
![Page 58: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/58.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 58
QEMU: The Quick Emulator
A generic machine emulator and virtualizer ia32, x86_64, mips, sparc, arm, risc-v KVM-client e.g. using Intel VT-x
Includes a huge collection of emulated devices Active community ~130 subsys maintainers ~1500 individual contributors)
![Page 59: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/59.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 59
QEMU: Contributions
Upstream contributions (Klaus A. B. Jensen) Full NVMe 1.3 support Ongoing NVMe 1.4 support Full ZNS support Upcoming TPs
![Page 60: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/60.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 60
QEMU: Contributions
Upstream contributions (Klaus A. B. Jensen) Full NVMe 1.3 support Ongoing NVMe 1.4 support Full ZNS support Upcoming TPs
Extending the QEMU NVMe drive model
![Page 61: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/61.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 61
QEMU: NVMe Device Model
QEMU/hw/block/nvme.c
![Page 62: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/62.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 62
QEMU: NVMe Device Model Usage
![Page 63: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/63.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 63
QEMU: NVMe Usage Hands On!
QEMU configure and build:
QENV setup environment config machine config device config run and access
mkdir buildcd build../configure--target-list=x86_64-softmmu
make -j $(nproc)
QEMU build demo link
QENV setup demo link
![Page 64: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/64.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 65
User Space tools and libraries
![Page 65: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/65.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 66
Open-Source Ecosystem: an overview
![Page 66: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/66.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 67
Open-Source Ecosystem: an overview
![Page 67: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/67.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 68
Open-Source Ecosystem: an overview
![Page 68: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/68.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 69
Open-Source Ecosystem: a contribution
![Page 69: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/69.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 70
xNVMeCross-platform libraries and tools for NVMe devices
![Page 70: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/70.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 71
xNVMe: API: xnvme_dev
Abstract handle to your controller / namespace
Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES, ) Reduce overhead of kernel retrieving handles for each IO
SPDK Backend Initialize and attach to controller spdk_env_opts_init() / spdk_env_init() / spdk_nvme_probe()
![Page 71: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/71.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 72
xNVMe: API: xnvme_buf
Allocate and free memory for use by the xnvme_cmd interface
Linux Backend Pagesize aligned for ioctl() posix_memalign() / free() io_uring_register(IORING_REGISTER_BUFFERS, ...)
SPDK Backend Allocate physical memory / DMA transferable spdk_dma_{malloc,realloc,free}()
![Page 72: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/72.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 73
xNVMe: API: xnvme_async
Context for asynchronous / non-blocking xnvme_cmd interface
Linux Backend Threadpool allocation for pseudo-async behavior via ioctl() SQ / CQ setup for io_uring
SPDK Backend NVMe QP setup spdk_nvme_ctrlr_{alloc,free}_io_qpair()
![Page 73: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/73.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 74
xNVMe: API: xnvme_cmd
Synchronous and asynchronous command interface
Linux Backend io_uring_{submit, peek , wait} Jobs to ioctl() threadpool
SPDK Backend spdk_nvme_ctrlr_cmd_{admin_raw, io_raw_with_md}() spdk_nvme_qpair_process_completions()
![Page 74: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/74.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 75
Open-Source Ecosystem: a contribution
![Page 75: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/75.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 76
Open-Source Ecosystem: a contribution
![Page 76: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/76.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 77
Open-Source Ecosystem: a contribution
![Page 77: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/77.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 78
Open-Source Ecosystem: a contribution
![Page 78: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/78.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 79
Usage of `zoned`Demo link
![Page 79: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/79.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 80
Open-Source Ecosystem: a contribution
![Page 80: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/80.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 81
Open-Source Ecosystem: a contribution
![Page 81: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/81.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 82
Usage of `ZROFS`Demo link
![Page 82: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/82.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 83
Open-Source Ecosystem: a contribution
![Page 83: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/83.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 84
Open-Source Ecosystem: a contribution
![Page 84: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/84.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 85
Usage of `xnvme`Demo link
![Page 85: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/85.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 86
Open-Source Ecosystem: a contribution
![Page 86: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/86.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 87
xNVMe
Building Open-Source ecosystem
![Page 87: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/87.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 88
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces
![Page 88: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/88.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 89
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces Tools
![Page 89: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/89.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 90
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces Tools Libraries
![Page 90: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/90.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 91
xNVMe
Building Open-Source ecosystem Cross-platform for existing and emerging
storage interfaces Tools Libraries VALUE
![Page 91: Programming Emerging Storage Interfaces€¦ · Abstract handle to your controller / namespace Linux Backend Open FDs for NVMe controller and namespace io_uring_register(.., IORING_REGISTER_FILES,](https://reader030.vdocuments.mx/reader030/viewer/2022013022/5f6185f176d74e3f1e39a8b1/html5/thumbnails/91.jpg)
2019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved. 922019 Storage Developer Conference. © Samsung / SSDR. All Rights Reserved.
Thanks
WWW https://xnvme.ioMAIL [email protected]
www.linkedin.com/in/simonlund