an experiment on bare-metal bigdata provisioning · bigdata configuration bigdata installation os...
Post on 11-Jul-2020
46 Views
Preview:
TRANSCRIPT
An Experiment on Bare-Metal BigData Provisioning
Ata Turk, Ravi S. Gudimetla, Emine Ugur Kaynar, Jason Hennessey, Sahil Tikale, Peter Desnoyers, Orran Krieger
1
BigData Analytics on the Cloud
• BigData deployments are moving to the cloud • On-demand usage (Cost), Elasticity,
Agility, Simplicity, … • Virtualized IaaS solutions: Amazon
EMR, Azure HDInsight, …
• Virtualization drawbacks • Overhead, unpredictability, security
concerns, device functionality, … • Bare-metal cloud solutions: IBM,
Rackspace, and Internap, …
2
Bare-Metal BigData Cloud Solutions
• Bare-Metal cloud provisioning • Automated provisioning: Ironic,
MaaS, … • Image copy to local disk => long
waits => loss of agility & elasticity
• OS streaming*, Lazy copy & de-virtualization**
• What about network booting? • incur an ongoing unacceptable
overhead during runtime
** Y. Omote, T. Shinagawa, and K. Kato, “Improving Agility and Elasticity in Bare-metal Clouds,” in ASPLOS’15, pp. 145–159, 2015.
3
* David Clerc, “OS Streaming Deployment”, in IPCCC’10, pp. 169–179, 2010.
• Large parts of the HPC community has been doing it for the last 20 years.
• Virtualized IaaS is doing it all the time.
• Why not bare-metal cloud?
4
Network-Mounted BigData System
• Clients access kernel and init ramdisk via PXE
• Mount OS & BigData apps from a remote iSCSI volume
• Use local disk for ephemeral storage (HDFS, /swap, /tmp,…)
5
Bare-Metal BigData Provisioning Prototype
6
Bare-Metal BigData Provisioning Prototype
Hardware Isolation Layer: A service to allocate bare-metal nodes out of a shared pool and isolate network
7
Bare-Metal BigData Provisioning Prototype
CEPH: Central image store hosting user images with BigData applications
8
Bare-Metal BigData Provisioning Prototype
Provisioning VM: Gateway between isolated servers and image store
9
Bare-Metal BigData Provisioning Prototype
10
Bare-Metal BigData Provisioning Prototype
11
Bare-Metal BigData Provisioning Prototype
12
Bare-Metal BigData Provisioning Prototype
13
Provisioning Time
Local Disk iSCSI0
200
400
600
800
1000
1200
1400
Elap
sed
Tim
e (S
ecs)
Bigdata ConfigurationBigdata InstallationOS RebootFirmware InitializationPost Setup Software InstallationPackage InstallationOS Boot(inc. Kernel+Initrd Download)DHCP requestFirmware InitializationHaas Power CycleCeph CloningHaas Initilization
14
Firmware initialization
OS Boot (inc. kernel+initrd download)
OS Installation
OS Reboot
BigData App Installation
Post Config
DHCP Request
Local diskinstallation iSCSI boot
Provisioning Time
Local Disk iSCSI0
200
400
600
800
1000
1200
1400
Elap
sed
Tim
e (S
ecs)
Bigdata ConfigurationBigdata InstallationOS RebootFirmware InitializationPost Setup Software InstallationPackage InstallationOS Boot(inc. Kernel+Initrd Download)DHCP requestFirmware InitializationHaas Power CycleCeph CloningHaas Initilization
15
Firmware initialization
OS Boot (inc. kernel+initrd download)
OS Installation
OS Reboot
BigData App Installation
Post Config
DHCP Request
Local diskinstallation iSCSI boot
Provisioning Time Scaling
2 Node 4 Node 8 Node0
50
100
150
200
250
300
Elap
sed
Tim
e (S
ecs)
Bigdata Post ScriptBootingCeph CloningHaas Initilization
16
Read Traffic over Boot Drive
Initi
alPr
ovis
ioni
ngD
ata
Gen
erat
ion
1
Sort
1
Dat
aG
ener
atio
n 2
Sort
2
Dat
aG
ener
atio
n 3
Sort
3
Dat
aG
ener
atio
n 4
Sort
4
Dat
aG
ener
atio
n 5
Sort
50
100
200
300
Cum
ulat
ive
iSC
SI re
ads
per n
ode
(MB)
iSCSI Reads: Runs with 256GB DataiSCSI Reads: Runs with 128GB Data
17
Read Traffic over Boot Drive
Initi
alPr
ovis
ioni
ngD
ata
Gen
erat
ion
1
Sort
1
Dat
aG
ener
atio
n 2
Sort
2
Dat
aG
ener
atio
n 3
Sort
3
Dat
aG
ener
atio
n 4
Sort
4
Dat
aG
ener
atio
n 5
Sort
50
100
200
300
Cum
ulat
ive
iSC
SI re
ads
per n
ode
(MB)
iSCSI Reads: Runs with 256GB DataiSCSI Reads: Runs with 128GB Data
~170MB / 8GB Boot Image => 2%18
Read Traffic over Boot Drive
Initi
alPr
ovis
ioni
ngD
ata
Gen
erat
ion
1
Sort
1
Dat
aG
ener
atio
n 2
Sort
2
Dat
aG
ener
atio
n 3
Sort
3
Dat
aG
ener
atio
n 4
Sort
4
Dat
aG
ener
atio
n 5
Sort
50
100
200
300
Cum
ulat
ive
iSC
SI re
ads
per n
ode
(MB)
iSCSI Reads: Runs with 256GB DataiSCSI Reads: Runs with 128GB Data
3KB/sreadafter initial boot
19
Write Traffic over Boot Drive
Initi
alPr
ovis
ioni
ngD
ata
Gen
erat
ion
1
Sort
1
Dat
aG
ener
atio
n 2
Sort
2
Dat
aG
ener
atio
n 3
Sort
3
Dat
aG
ener
atio
n 4
Sort
4
Dat
aG
ener
atio
n 5
Sort
50
100
200
300
400
500
600
700
Cum
ulat
ive
iSC
SI w
rites
per
nod
e (M
B)
iSCSI Writes - Runs with 256GB DataiSCSI Writes - Runs with 128GB Data
20
Write Traffic over Boot Drive
Initi
alPr
ovis
ioni
ngD
ata
Gen
erat
ion
1
Sort
1
Dat
aG
ener
atio
n 2
Sort
2
Dat
aG
ener
atio
n 3
Sort
3
Dat
aG
ener
atio
n 4
Sort
4
Dat
aG
ener
atio
n 5
Sort
50
100
200
300
400
500
600
700
Cum
ulat
ive
iSC
SI w
rites
per
nod
e (M
B)
iSCSI Writes - Runs with 256GB DataiSCSI Writes - Runs with 128GB Data 14KB/s
write
21
Runtime Performance of Network-Mounted Boot Drive
171 31
9
616
1187
2314
171 31
8
617
1176
2281
64 115 30
0
542
1073
69 120 23
8
555
1361
60 75 76 118 19
9
52 63 86 125 20
1
Data Size (GB)0
400
800
1200
1600
2000
2400
2800
Elap
sed
Tim
e (s
ecs)
WordCount - Local DiskWordCount - iSCSI MountedSort - Local DiskSort - iSCSI MountedGrep - Local DiskGrep - iSCSI Mounted
8GB 16GB 32GB 64GB 128GB
22
Runtime Performance of Network-Mounted Boot Drive
171 31
9
616
1187
2314
171 31
8
617
1176
2281
64 115 30
0
542
1073
69 120 23
8
555
1361
60 75 76 118 19
9
52 63 86 125 20
1
Data Size (GB)0
400
800
1200
1600
2000
2400
2800
Elap
sed
Tim
e (s
ecs)
WordCount - Local DiskWordCount - iSCSI MountedSort - Local DiskSort - iSCSI MountedGrep - Local DiskGrep - iSCSI Mounted
8GB 16GB 32GB 64GB 128GB
23
Runtime Performance of Network-Mounted Boot Drive
171 31
9
616
1187
2314
171 31
8
617
1176
2281
64 115 30
0
542
1073
69 120 23
8
555
1361
60 75 76 118 19
9
52 63 86 125 20
1
Data Size (GB)0
400
800
1200
1600
2000
2400
2800
Elap
sed
Tim
e (s
ecs)
WordCount - Local DiskWordCount - iSCSI MountedSort - Local DiskSort - iSCSI MountedGrep - Local DiskGrep - iSCSI Mounted
8GB 16GB 32GB 64GB 128GB
24
Take-aways• Network booting the OS for bare-metal BigData
• uses only a fraction of boot disk during start-up • improves provisioning time with no runtime degradation
• provisioning time < 5 mins, boot disk reads: ~3KB/s, writes: ~14KB/s
• Enormous effort on bare-metal provisioning on local disks may be unnecessary, especially for BigData deployments
• We are building a new Bare Metal Imaging Service using remote network boot mechanisms • enable capabilities available on virtualized platforms (e.g.
snapshotting, cloning, …) to bare metal cloud solutions
25
Questions
26
Provisioning Time
Local Disk iSCSI0
200
400
600
800
1000
1200
1400
Elap
sed
Tim
e (S
ecs)
Bigdata ConfigurationBigdata InstallationOS RebootFirmware InitializationPost Setup Software InstallationPackage InstallationOS Boot(inc. Kernel+Initrd Download)DHCP requestFirmware InitializationHaas Power CycleCeph CloningHaas Initilization
27
Local diskinstallation iSCSI boot Emulab* Ironic*
* A. Chandrasekar and G. Gibson, “A comparative study of baremetal provisioning frameworks,” Parallel Data Laboratory, Carnegie Mellon University, Tech. Rep. CMU-PDL-14-109, 2014.
Provisioning Time
Local Disk iSCSI0
200
400
600
800
1000
1200
1400
Elap
sed
Tim
e (S
ecs)
Bigdata ConfigurationBigdata InstallationOS RebootFirmware InitializationPost Setup Software InstallationPackage InstallationOS Boot(inc. Kernel+Initrd Download)DHCP requestFirmware InitializationHaas Power CycleCeph CloningHaas Initilization
28
Local diskinstallation iSCSI boot Emulab* Ironic*
* A. Chandrasekar and G. Gibson, “A comparative study of baremetal provisioning frameworks,” Parallel Data Laboratory, Carnegie Mellon University, Tech. Rep. CMU-PDL-14-109, 2014.
copy (rather than install) an image to local disk
top related