non-blocking writes to files - usenix · non-blocking writes to files operating systems block...
TRANSCRIPT
Non-Blocking Writes to Files
Daniel Campello, Hector Lopez, Luis Useche1, Ricardo Koller2, and Raju Rangaswami
1Google, Inc. 2IBM TJ Watson
Non-blocking Writes to Files
Applications have different alternatives to persist data
Synchrony vs Asynchrony
2
Process
Page Cache
Backend Storage
Process
Kernel Thread
Page Cache
Backend Storage
Synchronous Writes Asynchronous Writes
Mem
ory
Mem
ory
Non-blocking Writes to Files
Memory access granularity is smaller than disk’s ⇒ Partial writes to pages not present in the cache
require page fetch.
Introduction
Process OS
Backing Store
Page Cache
0 1 0
1 1 1
0 0 0
1. Write (x) 2. Miss
3. Issue Fetch4. Complete
5. Return
Why wait for data that the application doesn’t need?
3
Non-blocking Writes to Files
Motivation: Breakdown of write operations
• On an average, 63.12% of the writes involved partial page overwrites.
• Depending of page cache size, these overwrites could result in varying degrees of page fetches.
0%10%20%30%40%50%60%70%80%90%
100%
% o
f W
rite
Op
era
tio
ns
Types of Page Writes (%)
No overwrite; writeis to a new page
Overwrite; multipleof 4 KB aligned
Overwrite; >= 4 KBunaligned
Overwrite; < 4 KB
4
Non-blocking Writes to Files
Motivation: LRU Cache Simulation
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
0.001 0.1 10 1000 100000
Par
tial
Ove
r-W
rite
s to
Ou
t-o
f-co
re P
age
(%
)
Page cache size (MB)
ug-filesrv
gsf-filesrv
moodle
backup
usr1
usr2
5
Non-blocking Writes to Files
Non-blocking Writes: Asynchronous Fetch
Process OS
Backing Store
0 1 0
1 1 1
0 0 0
1. Write (x) 2. Miss
3. Issue Fetch6. Complete
5. Return
Page CachePatchPatchPatch
4. Buffer
7. Merge
Benefits:1. Application execution time reduction2. Increased backing store bandwidth usage
6
Non-blocking Writes to Files
Non-blocking Writes: Asynchronous Fetch
Process OS
Backing Store
0 1 0
1 1 1
0 0 0
1. Write (x) 2. Miss
3. Issue Fetch6. Complete
5. Return
Page CachePatchPatchPatch
4. Buffer
7. Merge
Why issue a page fetch for data that the application doesn’t need?
7
Non-blocking Writes to Files
Non-blocking Writes: Lazy Fetch
Benefits and Drawbacks:1. Allocating page memory and fetching only if necessary2. Resource utilization is unpredictable and can be bursty
Process OS
Backing Store
0 1 0
1 1 1
0 0 0
1. Write (x)
2. Miss
7. Issue page fetch
10. Return
PatchPatchPatch
3. Buffer
9. Merge
4. Return
5. Read (x)
8. Complete6. Miss
Page Cache
8
Non-blocking Writes to Files
Managing Page Fetches
Page Fetch Policies
Asynchronous
Lazy
Page Fetch Mechanisms
Foreground
Background
9
Non-blocking Writes to Files
Metadata misses generate extra blocking fetches
Foreground vs Background Mechanisms
10
Process
Page Cache
Backend Storage
Process
NBW Thread
Page Cache
Backend Storage
Foreground Background
Mem
ory
Mem
ory
Non-blocking Writes to Files
Implementation
We found this problem in other OSes (FreeBSD, Xen)
We implemented non-blocking writes for files in the Linux kernel by modifying the generic virtual file system (VFS) layer
Workloads
Filebench micro-benchmark
SPECsfs2008 benchmark
Mobibench system call trace-replay
Evaluation Specifics
11
Non-blocking Writes to Files
0100020003000400050006000700080009000
128 256 512 1024 2048 4096
Op
era
tio
ns
/ se
con
d
Size of operation (bytes)
Random Writes
BW
NBW-Async-FG
NBW-Async-BG
NBW-Lazy
0
2000
4000
6000
8000
128 256 512 1024 2048 4096
Op
era
tio
ns
/ se
con
d
Size of operation (bytes)
Random Read-Write
Filebench evaluation: Writes and Read-Writes
• NBW-Async-FG: 50-60% performance improvement for Random Writes
12
0
200
128 256 512 1024 2048 4096
Non-blocking Writes to Files
0100020003000400050006000700080009000
128 256 512 1024 2048 4096
Op
era
tio
ns
/ se
con
d
Size of operation (bytes)
Random Writes
BW
NBW-Async-FG
NBW-Async-BG
NBW-Lazy
0
2000
4000
6000
8000
128 256 512 1024 2048 4096
Op
era
tio
ns
/ se
con
d
Size of operation (bytes)
Random Read-Write
Filebench evaluation: Writes and Read-Writes
• NBW-Async-FG: 50-60% performance improvement for Random Writes
• NBW-Async-BG: 6.7x-30x (Random Writes) and 3.4x-20x (Random RW)
• NBW-Lazy: Up to 45x performance improvement
13
Non-blocking Writes to Files
Filebench evaluation: Reads
Non-blocking writes do not affect the performance of read-only workloads
0
20
40
60
80
100
120
140
160
128 256 512 1024 2048 4096
Op
era
tio
ns
/ se
con
d
Size of operation (bytes)
Random Reads
BW
NBW-Async-FG
NBW-Async-BG
NBW-Lazy
14
Non-blocking Writes to Files
SPECsfs2008 evaluation
0
20
40
60
80
100
Write (10%) Read (18%) Others (72%) Overall (100%)
No
rmal
ize
d L
ate
ncy
SPECsfs2008
BW
NBW-Async-FG
NBW-Async-BG
NBW-Lazy
• Around 70% write latency decrease
• NBW-Lazy read latency slightly affected by delayed fetching
• Overall latency is in direct relation to write and read latency
15
Non-blocking Writes to Files
Mobibench System-call Trace Replay
Average operation latency decreased by 20-40%
0
20
40
60
80
100
120
Facebook Twitter
No
rmal
ize
d L
aten
cy
Mobibench HDD
BW
NBW-Async-FG
NBW-Async-BG
NBW-Lazy
0
20
40
60
80
100
120
Facebook Twitter
No
rmal
ize
d L
aten
cy
Mobibench SSD
16
Non-blocking Writes to Files
Huge body of work on optimizing writes performance
O_NONBLOCK flag for OPEN system call
• If no data can be read or written, returns EAGAIN (or EWOULDBLOCK)
• Requires application modification
Asynchronous I/O Library (POSIX AIO)
• Multiple threads to perform I/O operations (expensive and scales poorly)
• Requires application modification
Related Work
17
Non-blocking writes do not require application
modification
Non-blocking Writes to Files
Operating systems block processes on asynchronous writes in many cases!
With non-blocking writes, we demonstrate that such blocking is unnecessary and detrimental to performance
General solution for any kind of modern OS
Does not require application modification
Evaluation demonstrated how non-blocking writes effectively reduces write latency across a wide range of workloads
Conclusions
18
Thank You!
The traces used in this paper are available athttp://sylab-srv.cs.fiu.edu/dokuwiki/
doku.php?id=projects:nbw:start
Non-blocking Writes to Files
Page Fetch Asynchrony
Blocking write
Time
Waiting I/O: Thinking:
Write P Read P
Non-blocking write
Write P Read P
20
Non-blocking Writes to Files
Page Fetch Parallelism
Blocking write
Time
Blocking I/O: Background I/O:
Write P Write Q Read P
Non-blocking write
Write P
Write Q
Read P
21
Non-blocking Writes to Files
Production system trace descriptions
Motivation: Production workloads
Workload Description
ug-filserv Undergrad NFS/CIFS fileserver
gsf-filesrv Grad/Staff/Faculty NFS/CIFS fileserver
moodle Web & DB server for department CMS
backup Nightly backups of department servers
usr1 Researcher 1 desktop
usr2 Researcher 2 desktop
facebook Mobibench facebook trace
twitter Mobibench twitter trace
22
Non-blocking Writes to Files
Ordering of page updates
Handling of disk errors
Journaling File Systems
OS-initiated page accesses
Page persistence and syncs
Handling read-write dependencies
Multi-core and kernel preemption
Highlights On Correctness
23