multitenant storage-scale11x
DESCRIPTION
Adding multitenant storage capability to the illumos kernelTRANSCRIPT
![Page 1: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/1.jpg)
Adding Multitenant Storage
Capability to the illumos Kernel
Richard Elling and Garrett D’Amore
![Page 2: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/2.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
• Why multitenancy?
• Survey of available technologies
• OS virtualization
• Multitenant storage services in illumos
Agenda
![Page 3: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/3.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Why Multitenancy?
![Page 4: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/4.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Kernel
Userland Process
Device Driver
Userland Process
Simple model: multitasking OS
![Page 5: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/5.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Strike up the band!
![Page 6: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/6.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Guest
Native Hypervisor Model
Kernel
Userland Process
Hypervisor Device Driver
Examples: VMware ESX, XenServer, KVM
Device Driver
![Page 7: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/7.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Java
Java Virtual Machines are Virtual Machines, too
JVM
Java Application
OS Device Driver
APIs
![Page 8: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/8.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Hosted Hypervisor Model
UserlandProcess
Examples: VMware Workstation, VirtualBox
Guest
Kernel
Userland Process
Hypervisor Device Driver
Device Driver
Kernel Device Driver
![Page 9: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/9.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
HVMGuestOS
HostOS
QEMU
Device Driver
Userland Process
Device Driver
KVM
I/O Proxy
![Page 10: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/10.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
DomU
Front-end driver
Device Driver
Userland Process
XenServer
Dom0
Back-end driver
Device Driver
Userland I/O Proxy
![Page 11: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/11.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Kernel
Userland Process
Device Driver
Userland Process
Return to simple model: multitasking OS
![Page 12: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/12.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Zone
KernelDevice Driver
Userland Process
Zone
Userland Process
![Page 13: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/13.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
BSD Kernel
Jail
BSD Jails
Storage Device Drivers
IP AddressDirectorystructure
File system
Userland Process
NIC Driver
![Page 14: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/14.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Kernel
Zone
illumos OS
Storage Device Drivers NIC Driver
Classification Engine
TCP/IP
Virtual NICFile system
Userland Process
![Page 15: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/15.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
The first thing we do, let’s kill all the lawyers
King Henry VIAct IV, Scene II
![Page 16: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/16.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
The first thing we do, let’s kill all the lawyers
King Henry VIAct IV, Scene II
The first thing we do, let’s kill all the layers
UNIX Performance Gurus
![Page 17: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/17.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Zone
KernelDevice Driver
Userland Process
Zone
Userland Process
![Page 18: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/18.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Kernel
Storage Device Drivers NIC Driver
VirtualSwitch
Zone A
Zone B
TCP/IP TCP/IP
File systemBlock device Virtual NIC Virtual NIC
Transactional Object Store and Data Management Unit
NFS
File system
SMB
File system
NFSiSCSI
![Page 19: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/19.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Global state Per-zone state
NFS - rpc now zoned
SMB - in-kernel, multithreaded, zoned
iSCSI - per-zone block devices
➾
![Page 20: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/20.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Big shared caches Inefficient small caches
~ 100 MB/tenant > 1000 MB/tenant
Kernel
Zone
Adaptive Replacement Cache
NFS
Zone
ZoneiSCSI
NFS
SMBTraditionalHypervisor
VMNFS
kernel
VM
kernel
Samba
FS Cache
FS Cache
![Page 21: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/21.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Kernel
Deep access to devices
Storage Device Drivers NIC Driver
VirtualSwitch
Zone A
Zone B
TCP/IP TCP/IP
Awesome New
DeviceBlock device
Virtual NIC Virtual NIC
Transactional Object Store and Data Management Unit
New interface?
ANP
Block device
iSCSIiSCSI
![Page 22: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/22.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Zone
IP Multipath Group
Virtual Networking
TCP
NFS iSCSI
Zone
IP Multipath Group
TCP
NFS SMB
Physical NIC
Virtual Switch
Physical NIC
Virtual Switch
Virtual NICVirtual NIC Virtual NICVirtual NIC
![Page 23: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/23.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Kernel statistics kstats✦
Zone A
Resource capsNetwork usageI/O usage
Zone B
Resource capsNetwork usageI/O usage
All kstats
![Page 24: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/24.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
All in due time?
![Page 25: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/25.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
Tree AB
Domain B
Domain A
Domain C
Tenant A
Tenant B
Tenant C
SMB
SMB
NFS SMB✦
A
B
C
![Page 26: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/26.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
• Efficient, multitenant storage in illumos
• Sit out of the traditional VM dance
• OS virtualization rulez!
Summary
![Page 27: Multitenant storage-scale11x](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5597069b1a28abbb0e8b465f/html5/thumbnails/27.jpg)
Presented at the SCALE 11x ConferenceFebruary 24 2012
www.illumos.org
www.dey-sys.com
www.RichardElling.com