new towards near-data processing in deep and cold storage … · 2020. 5. 8. · towards near-data...
TRANSCRIPT
![Page 1: New Towards Near-Data Processing in Deep and Cold Storage … · 2020. 5. 8. · Towards Near-Data Processing in Deep and Cold Storage Hierarchies DLR.de • Chart 1 > XLDB’19 >](https://reader036.vdocuments.mx/reader036/viewer/2022081407/6050595127fb933d9c70bb75/html5/thumbnails/1.jpg)
Towards Near-Data Processing in Deep and Cold Storage Hierarchies
> XLDB’19 > Lightning Talk > Marcus Paradies > 03.04.2019 DLR.de • Chart 1
Marcus Paradies, German Aerospace Center (DLR)
04/03/2019, XLDB
![Page 2: New Towards Near-Data Processing in Deep and Cold Storage … · 2020. 5. 8. · Towards Near-Data Processing in Deep and Cold Storage Hierarchies DLR.de • Chart 1 > XLDB’19 >](https://reader036.vdocuments.mx/reader036/viewer/2022081407/6050595127fb933d9c70bb75/html5/thumbnails/2.jpg)
The Storage Hierarchy and Its (Current) Coverage in DB Research
> XLDB’19 > Lightning Talk > Marcus Paradies > 03.04.2019 DLR.de • Chart 2
Tape Disk Memory LLC Flash
Nearline/ Offline
Online
DNA
Offline/ Nearline
![Page 3: New Towards Near-Data Processing in Deep and Cold Storage … · 2020. 5. 8. · Towards Near-Data Processing in Deep and Cold Storage Hierarchies DLR.de • Chart 1 > XLDB’19 >](https://reader036.vdocuments.mx/reader036/viewer/2022081407/6050595127fb933d9c70bb75/html5/thumbnails/3.jpg)
> Lecture > Author • Document > Date DLR.de • Chart 3
![Page 4: New Towards Near-Data Processing in Deep and Cold Storage … · 2020. 5. 8. · Towards Near-Data Processing in Deep and Cold Storage Hierarchies DLR.de • Chart 1 > XLDB’19 >](https://reader036.vdocuments.mx/reader036/viewer/2022081407/6050595127fb933d9c70bb75/html5/thumbnails/4.jpg)
> Lecture > Author • Document > Date DLR.de • Chart 4
Active Data Archives for Scientific Data
![Page 5: New Towards Near-Data Processing in Deep and Cold Storage … · 2020. 5. 8. · Towards Near-Data Processing in Deep and Cold Storage Hierarchies DLR.de • Chart 1 > XLDB’19 >](https://reader036.vdocuments.mx/reader036/viewer/2022081407/6050595127fb933d9c70bb75/html5/thumbnails/5.jpg)
Scientific Application Domains
> XLDB’19 > Lightning Talk > Marcus Paradies > 03.04.2019 DLR.de • Chart 5
Earth Observation Radio Astronomy Weather Forecasting
Archive: 14 PB
Disk Cache: 175 TB
Archive: 50 PB
Disk Cache: 750 TB
Archive: 100 PB
Disk Cache: 1.34 PB
![Page 6: New Towards Near-Data Processing in Deep and Cold Storage … · 2020. 5. 8. · Towards Near-Data Processing in Deep and Cold Storage Hierarchies DLR.de • Chart 1 > XLDB’19 >](https://reader036.vdocuments.mx/reader036/viewer/2022081407/6050595127fb933d9c70bb75/html5/thumbnails/6.jpg)
Data Movement as Major Performance Bottleneck
> XLDB’19 > Lightning Talk > Marcus Paradies > 03.04.2019 DLR.de • Chart 6
Active data archives (and their catalogs) are like Amazon, but just for data.
No SLAs on access latency, usually between minutes and hours Tail latency can be multiple days Historic data analysis can easily request 100s of TB
… Compute
Facilities
Disk Cache N Disk Cache 1 Data Archive
![Page 7: New Towards Near-Data Processing in Deep and Cold Storage … · 2020. 5. 8. · Towards Near-Data Processing in Deep and Cold Storage Hierarchies DLR.de • Chart 1 > XLDB’19 >](https://reader036.vdocuments.mx/reader036/viewer/2022081407/6050595127fb933d9c70bb75/html5/thumbnails/7.jpg)
CryoDrill---Near-Data Processing for Cold Storage
> XLDB’19 > Lightning Talk > Marcus Paradies > 03.04.2019 DLR.de • Chart 7
Focus on nearline storage (archival disks, tape)
Consider all NDP opportunities (in-
network, in-storage)
Push data reduction ops down the storage hierarchy