oasis: self-tuning storage for applications · - slos specified in the largest data centers are...
TRANSCRIPT
![Page 1: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/1.jpg)
IBM Research
14th NASA Goddard- 23rd IEEE Mass Storage Systems Technologies, College Park, MD, May 17, 2006
OASIS: Self-tuning Storage for Applications
Kostas Magoutis, Prasenjit Sarkar, Gauri Shah
IBM Research
![Page 2: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/2.jpg)
2
IBM Research
Outline
Motivation and background
OASIS Architecture
- Environment
- Protocol
- Algorithms
Evaluation
Conclusions
![Page 3: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/3.jpg)
3
IBM Research
Congestion Control Congestion Control
Motivation
Storage System
Database System
Workload #1
(transactional)
SLO ?SLO ?
Data gathering agents
Workload #2
(aggregation)
Statistical reports
Congestion
![Page 4: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/4.jpg)
4
IBM Research
Background Allocating storage resources to match application needs is hard
- Best-effort allocations are often insufficient
Systems offering Service Level Objectives (SLOs) have drawbacks- Administrators cannot give detailed description of storage requirements- Instead, they rely on rough estimates based on intuition or past experience- Descriptions of storage resource requirements can only be statistical- SLOs specified in the largest data centers are often punitive rather than descriptive
Complexity of applications and storage infrastructure growing- Need simple interface between applications and storage infrastructure
![Page 5: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/5.jpg)
5
IBM Research
Overview of OASIS Manages the allocation of storage resources to applications
- Monitors application requirements and conveys to a storage manager- Applies proportional fairness mechanisms in resource allocation- Improvement over best-effort systems
Self-tuning: The allocations are set automatically and dynamically- Does not require operator involvement- OASIS does not conflict (and can co-exist) with SLO-based systems
Introduces simple interface between applications and storage
![Page 6: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/6.jpg)
6
IBM Research
Proportional Fairness Derives from economic framework
- Participants are associated with utility functions- Suitable choice for utility functions is logarithmic (“diminishing returns”)- Optimal allocation maximizes sum of all utilities- Applied to network bandwidth sharing by Kelly [1998]
Resembles fairness achieved by TCP protocol- Proportional revocation of resources
OASIS applies proportional fairness mechanisms to storage systems
SLO-based storage systems offer performance isolation/differentiation
![Page 7: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/7.jpg)
7
IBM Research
OASIS Architecture Environment
- Multiple host servers running multiple applications- Storage access through storage-area network (SAN)- Applications, storage are complex domains in different areas of expertise
OASIS Protocol- Insulates the application domain from the storage domain- Conveys application requirements to Storage Manager
Resource Management Algorithms- Map application requirements to storage resources- Govern allocation of resources subject to proportional fairness
![Page 8: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/8.jpg)
8
IBM Research
Environment
SANdata path
Storage Manager
Monitor
Control
protocol end-point
protocol end-point
![Page 9: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/9.jpg)
9
IBM Research
OASIS Protocol
Commodities Throughput Latency
Resources Disk bandwidth Cache
Mapping
![Page 10: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/10.jpg)
10
IBM Research
Application Protocol End-point Functionally distinct from the application
- Does not require application modifications or human involvement- Operates using standard interfaces (e.g., UNIX iostat, database logs, etc.)
Determines application requirements of commodities- For data path sourced at Q[Pi]j over ∆t = (t, t + δt), app end-point measures
• Di,j (∆t) : Average I/O arrival rate
• Xi,j (∆t) : Average I/O completion rate
• Ri,j (∆t) : Average I/O latency
• Si,j (∆t) : Average I/O size
- Bandwidth or latency request:
• ci,j (∆t) = Di,j (∆t) * Si,j (∆t).
• li,j (∆t) = Ri,j (∆t) – Ri,j (t) , if Ri,j (∆t) > R
Communicates requirements to Storage Manager protocol end-point
![Page 11: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/11.jpg)
11
IBM Research
Storage Manager Protocol Endpoint Maps requested commodities to resources
- E.g., maps throughput, latency to disk bandwidth, cache, etc.
Identifies data paths involved in commodity requests- Data paths connect host I/O queues and storage volumes through SAN
Manages (allocates, revokes) resources- Determines quantity of resources required
• Resource type #1: Measurable, fixed quantity at time t; e.g., cache• Resource type #2: Estimable, varies over time; e.g., disk bandwidth
- Determines whether system resource-constrained- Current implementation revokes resources proportionately
![Page 12: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/12.jpg)
12
IBM Research
Storage Manager Protocol Endpoint: Bandwidth Current state:
- Bandwidth allocated to application i is Ai
- Bandwidth used by application i is Ui
Upon receiving bandwidth request B from application p- if Ap = Up then
• Allocate bandwidth B to p, i.e., Ap = Ap + B
- else (Ap > Up )
• A = Σi=1..n Ai
• U = Σi=1..n Ui
• Revoke Ai – (Ai / A)∗ U from all applications except p
![Page 13: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/13.jpg)
13
IBM Research
Storage Manager Protocol Endpoint: Latency Current state:
- Cache allocated to application i is Ai
Upon receiving request to reduce latency for application p- Convert request to hit ratio requirement- Determine additional cache C required to meet new hit ratio
If cache space C not available
- A = Σi=1..n Ai
- Revoke [ Ai / (A - Ap) ] * C from all applications except p
Allocate cache space C to p
![Page 14: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/14.jpg)
14
IBM Research
Experimental Setup
Linux 2.6.8
2.4GHz Intel CPUs (4), 4GB
worker
set disk bandwidth allocation
APendpoin
t
SMendpoin
t
request
revoke
diskcache set cache allocation
direct async I/O
![Page 15: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/15.jpg)
15
IBM Research
Evaluation: Bandwidth Two competing workers
- worker #1: sequential access- worker #2: random access
Workers identically configured- raw (unbuffered) 4KB reads to logical disk device- requests produced at constant rate 60 IOPS- logical disk saturates at about 100 IOPS for random 4KB I/Os
Worker #1 overwhelms #2 in a best-effort system
![Page 16: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/16.jpg)
16
IBM Research
Evaluation: Bandwidth, Best Effort
Worker #1 (Sequential)
Worker #2 (Random)
Avg. Q size: 13.4
Avg. Q size: 2.1
![Page 17: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/17.jpg)
17
IBM Research
Evaluation: Bandwidth, OASIS
Bandwidth demand
Bandwidth usage
Equal avg. Q size
![Page 18: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/18.jpg)
18
IBM Research
Evaluation: Bandwidth, OASIS Allocation vs. Usage
Bandwidth demand
Bandwidth usage
Bandwidth allocation
![Page 19: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/19.jpg)
19
IBM Research
Evaluation: Latency (Summary)
For simplicity, the OASIS prototype maps latency to cache via hit ratio
Cache allocation is triggered by explicit requests for latency reduction
OASIS achieves faster convergence to a fair allocation of cache
However, adaptation is slower than in the case of bandwidth due to statistical nature of latency shift estimates
![Page 20: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/20.jpg)
20
IBM Research
Evaluation: Database Workload
Storage System
Database System
Workload #1
(transactional)
Data gathering agents
Workload #2
(aggregation)
Statistical reports
Congestion
(PostgreSql)
SMendpoin
t
set disk bandwidth allocation
APendpoin
t
APendpoin
t
revokerequest request
revoke
OASIS
![Page 21: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/21.jpg)
21
IBM Research
Database Workload Transaction workload (#1):
- Mostly random access- Reads: 6%- Average I/O size: 0.8 KB- Average inter-arrival time: 0.121 ms
Aggregation workload (#2):- Mostly sequential access- Reads: 99.9%- Average I/O size: 3.9 KB- Average inter-arrival time: 0.049 ms
![Page 22: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/22.jpg)
22
IBM Research
Database Workload (contd.)
0135840102902078731.82358
015400080302293319.51424
017290036002517815.8822
02047201630267807.2491
Aggr Queue
Len
Aggr Perf
(TPS)
Trans Queue
Len
Trans Perf
(TPS)
Aggr Queue
Len
Aggr Perf
(TPS)
Trans Queue
Len
Trans Perf
(TPS)
Trans Workload Scaling
OASISBest EffortMode
Improvement of up to 5x for transaction workload
Aggregation workload impacted by about 30-40%
![Page 23: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/23.jpg)
23
IBM Research
Future work
Fairness issues: Take efficiency of I/O streams into account?
Complex mappings of commodities to resources
Coupling between inter-related commodities
Re-organization of data layout as a response to overload
![Page 24: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/24.jpg)
24
IBM Research
Conclusions Self-tuning protocol to fairly share storage resources
- Does not require operator involvement- Improves over best-effort systems
OASIS automatically- Detects congestion- Ensures underperforming workloads receive fair share of resources
Simple interface between applications and storage
![Page 25: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/25.jpg)
25
IBM Research
Backup
![Page 26: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/26.jpg)
26
IBM Research
Related work “Black-box” systems
- OASIS maps commodities to storage resources and requires control of the storage subsystem
SLO-based systems- OASIS can co-exist with SLO-based systems- OASIS is fundamentally about the protection of overwhelmed I/O flows
![Page 27: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/27.jpg)
27
IBM Research
Fairness Weighted Fair Queuing (WFQ)
- Ri/Rj = p/q
- Performance isolation & differentiation
Economic frameworks- Based on utility functions
• Proportional Fairness [Kelly98]
TCP- Additive increase/multiplicative decrease
![Page 28: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/28.jpg)
28
IBM Research
Evaluation: Bandwidth, Worker #1 (Sequential)
![Page 29: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/29.jpg)
29
IBM Research
Evaluation: Bandwidth, Worker #2 (Random)
![Page 30: OASIS: Self-tuning Storage for Applications · - SLOs specified in the largest data centers are often punitive rather than descriptive Complexity of applications and storage infrastructure](https://reader031.vdocuments.mx/reader031/viewer/2022011911/5f87a99d7ebd3d50e37d2d57/html5/thumbnails/30.jpg)
30
IBM Research
Evaluation: Bandwidth, Worker #1 OASIS Allocation vs. Usage