vDrive: An Efficient and Consistent Virtual I/O System

Mohit Saxena†, Pin Zhou*and David A. Pease††IBM Research Almaden, *Datos IO

AbstractThe most popular methods for managing storage and pro-viding crash consistency are I/O virtualization and jour-naled filesystems respectively. This popularity is due totheir widespread use in production environments. How-ever, both of these methods have evolved separately indifferent contexts in the past.

This paper presents a first look on providing crash con-sistency for virtual I/O caches through journaled filesys-tems. We find that nested filesystem journaling in guestand host operating systems has a significant performancecost. This cost is attributed to the use of traditional diskinterfaces for cache flushes and lack of coordination be-tween the two journaling levels. We present vDrive, aconsistent virtual I/O system architecture, with a new vir-tual disk interface and semantic journaling mechanismdesigned to provide high performance.

We have implemented vDrive interface extensions inthe KVM/QEMU virtual I/O system and semantic jour-naling in the Linux ext4 filesystem. We show through ex-periments that vDrive outperforms nested journaling byup to 142% and correctly recovers to a consistent stateafter a crash.

1 Introduction

In recent years, virtualization has improved hardware uti-lization allowing service providers to offer a wide rangeof application and infrastructure services [3, 16, 24]. I/Ovirtualization enables efficient and flexible allocation ofstorage resources across different virtualized workloads.

To achieve high bare-metal performance, I/O intensiveworkloads require direct access to physical disks or log-ical volumes. For this purpose, modern hypervisors im-plement a technique called PCI Passthrough or Direct-Path I/O [40]. Passthrough I/O achieves high perfor-mance by bypassing the host software on the I/O path.However, it gives up a lot of virtualization flexibility in-

cluding complicated VM live migration and no storagespace overcommit.

As a result, the most popular I/O virtualization tech-nique today is paravirtual I/O1 [29, 38]. A virtual I/Osystem consists of a modified driver in the guest oper-ating system, and a virtual disk exported to the guest asa block device but stored as a file on the host filesys-tem. This enables flexible allocation of storage spaceand additional management features embedded in VMimages [28]. Previous work has pointed to the perfor-mance implications of virtual I/O resulting from nestedfile systems [21], and addressed the overheads for fre-quent switches between guest and host operating sys-tems [14, 15].

In this paper, we present a first look on the perfor-mance implications of crash consistency for virtual I/O.The most popular and widely adopted technique to pro-vide crash consistency in virtualized environments is theuse of journaled filesystems in guest and host operat-ing systems. However, the two journal levels in nestedfilesystem journaling interact in ways, which have notbeen studied earlier, and result in significant performancecost. We find that this performance cost is attributedto two major reasons embedded in the original designof filesystem journaling. First, journaling has been de-signed to provide crash resilience on bare-metal hard-ware for data resident in a physical disk cache. Sec-ond, journaling uses the traditional storage interfaces forflushing data to disk storage. In contrast, virtual I/O in-troduces a more complex hierarchy of cache levels, butat the same time provides greater flexibility to rethink thesoftware interface to virtualized storage.

The first source of overhead for nested filesystem jour-naling arises because the two journaling protocols pro-vide crash resilience for the same application data writeindependently across different cache levels in guest andhost without any coordination. The guest filesystem

1we refer paravirtual I/O as virtual I/O in the rest of the paper

1

journal provides crash resilience against the virtual diskcache, while the host filesystem journal protects againstthe physical disk cache. As a result, each write from theguest application waits longer because of multiple stallsand additional writes introduced during journal commitsat each level. This significantly degrades the combinedsystem performance for applications running atop nestedjournal filesystems. We investigate the design of a nestedjournaling protocol, which is aware of the virtual I/O se-mantics to minimize these overheads.

The second source of overhead arises due to the use ofthe traditional storage interface of expensive disk cache-flushes for nested filesystem journaling. The disk cache-flush operations used in modern drives impose both or-dering and durability guarantees for writes [36, 27]. Re-cent work on optimistic crash consistency [5] decouplesthe costs of ordering and durability for disk cache-flusheson a single level of filesystem journal. We investigate thefirst application of optimistic crash consistency to vir-tual I/O for reducing the commit cost of nested filesystemjournaling. We find that it requires revisiting the softwarecache-flush interface to virtual disk and major changes tothe virtual I/O system. However, these changes simplifythe integration of optimistic crash consistency with theguest filesystem than what is required for a single jour-nal level [5]. We exploit this opportunity to reduce theoverheads of nested journaling.

We present vDrive: an efficient and consistent virtualI/O system. vDrive solves the above problems by usinga new technique to provide low overhead crash consis-tency: semantic journaling for virtual I/O. vDrive ex-tends the virtual disk interface by introducing two newprimitives: vorder and vflush, which are exported to thevirtual I/O drivers in the guest operating system. Thevorder operation only guarantees ordering of preceedingwrites, while vflush enforces both order and durability.The guest drivers use these primitives selectively basedon the semantics of the data being persisted from the vir-tual cache hierarchy to the physical disk. With both prim-itives, vDrive always enforces the order of committedwrites so as to enable a consistent state recovered aftera crash.

The main contributions of this paper are as follows:

• We present the first experimental analysis of theconsistency and performance tradeoffs for virtualI/O. We conduct our experiments across differentstorage technologies. We identify a key prob-lem with virtualized disk storage: overhead ofnested filesystem journaling for different virtual I/Ocaching modes.• We design, implement and evaluate vDrive, an ef-

ficient and consistent virtual I/O system, which ex-ploits a new virtual disk interface to implement se-mantic journaling for virtual I/O traffic. vDrive im-

Applica'on  

Guest  Filesystem  

Virtual  I/O  Driver  

Storage  HW  Emula'on  

Image  Format  

Host  Filesystem  

Host  I/O  Driver  

Disk  

vir$o-­‐blk  req  

Posix  API    

SCSI/SATA/PCI-­‐e  

Guest  Page  Cache  

Guest  Disk  Cache  

Host  Page  Cache  

Disk  Write  Cache  

Guest  OS  

Virtual  Disk  

Host  OS  

Figure 1: Virtual I/O Storage Stack: The figure showsthe different software and caching layers in the virtualI/O storage stack.

proves virtual I/O performance by up to 142% overnested journaling and correctly recovers to a consis-tent state after crash.

The remainder of the paper is structured as follows.Section 2 and 3 motivate vDrive by describing a qual-itative and quantitative analysis of virtual I/O perfor-mance tradeoffs respectively. Section 4 and 5 present thedetailed description of vDrive design and implementa-tion. Finally, we evaluate vDrive design techniques inSection 6, and finish with related work and conclusions.

2 Motivation

In this section, we describe how the standard virtual I/Omodel achieves crash consistency through nested filesys-tem journaling and different caching modes within guestand host operating systems. We demonstrate how thesetechniques and the use of a traditional disk interface (i.e.cache-flush commands) result in negative performanceimpact for virtual I/O.

Virtual I/O Storage Stack. Figure 1 shows the vir-tual I/O stack comprised of different software layers andcache levels in the guest and host operating systems.An application I/O request in the guest can be servedfrom the guest OS page cache, otherwise it is forwardedthrough the frontend guest virtual I/O device driver to thebackend virtual disk running in the host user-space. Thevirtual disk is typically a file on the host fileystem whoselayout can vary based on the feature or performance re-quirements [28].

There are two sets of interfaces for a virtual disk:virtio-blk interface [29] with the guest driver and theposix filesystem interface with the host OS. Apart from

2

Virtual  I/O  Cache  Mode  

Nested  Journaling  Mod

e  

WB   Unsafe  WT  None  Direct  Sync  

ord-­‐data  

sync-­‐sync  

data-­‐data  

data-­‐ord  

ord-­‐ord  

ord-­‐wb  

wb-­‐ord  

wb-­‐wb  

ord-­‐ord  (w/o  flush)  

no-­‐journal  

(Low  Performance,    Consistent,  Fresh)  

(Low  Performance,    Inconsistent,  Stale)  

(High  Performance,    Inconsistent,    

Stale)  

(High  Performance,    Consistent,  Bounded)  

vDrive  

Figure 2: Virtual I/O Tradeoffs: The figure shows theimpact of different nested journaling and virtual cachingmodes on the performance, consistency and freshness ofdata for virtual I/O. We show the most relevant combi-nations of filesystem journaling modes in guest and hostOS: data, ordered, writeback and ordered without cache-flush. We show the popular cache modes for guest diskcache, host page cache and disk write cache: direct,none, write-through, write-back and unsafe.

the read and write requests; the guest driver can sendcache-flush commands to the virtual disk. The virtualdisk further translates them into the host filesystem fsyncsystem calls. Finally, the host filesystem sends cache-flush commands to flush data from the physical diskwrite cache.

A guest I/O request can get cached within the host atthree levels: virtual disk cache, host page cache or thephysical disk cache. Each guest virtual machine can beconfigured from the host to use one of the five differentcombinations for host cache modes: write-back (all threecaches are enabled), write-through (guest disk cache isdisabled), none (host page cache is disabled), direct (bothguest disk cache and host page cache are disabled) andunsafe (all caches are enabled and any cache-flush com-mands from guest are ignored).

The guest and host filesystems can use journaling toprovide write ordering and durability across the virtualand physical disk write caches. There are three majormodes for journaling: data (both metadata and data arecommitted into the journal before being written into themain filesystem), ordered (data is written to the mainfilesystem before metadata is committed into the jour-nal), and write-back (no ordering is preserved, data maybe written into the main filesystem after metadata hasbeen committed into the journal). The filesystem jour-nals send disk cache-flush commands to ensure orderingand durability of writes.

Virtual I/O Tradeoffs. The different virtual cachingand nested filesystem journaling modes provide differ-ent tradeoffs between performance, consistency and datafreshness. Figure 2 shows a classification of such trade-offs among the four most relevant combinations of cacheand journaling modes.

The first lowermost region in blue corresponds to thecombination of ordered or data journaling modes in bothguest and host, and any of the four cache modes whichpass guest cache-flush commands for journal commitsthrough all cache levels. After recovery, this combina-tion provides a consistent and the most recent or freshstate, which existed before a crash. However, these guar-antees comes at the performance cost to provide write or-dering and immediate durability through expensive diskcache-flushes. The use of synchronous I/O mode in guestand host (shown as the origin of the two axes in Figure 2)provides immediate durability for all writes with a hugeperformance loss.

The second region in red corresponds to the combi-nation of write-back or ordered journaling modes in bothguest and host. The use of write-back journaling mode ineither guest or host could result in unordered writes to thephysical disk, and an inconsistent and stale virtual diskimage after a crash. The use of ordered journaling mode,especially in the guest, results in frequent cache-flushrequests. As a result, these journaling mode combina-tions provide lower performance with the three cachingmodes: direct, none and write-through; all of which by-pass some cache levels at host for unordered writes fromjournal commits. Apart from the consistency cost, thesethree cache modes provide different levels of cache ex-clusiveness and differ in performance due to differentcache hit rates (we elaborate this further in Section 3).

The third region in dark green corresponds to no-journaling or combination of ordered journaling modesin guest and host without cache flushes enabled for jour-nal commits. This is equivalent to the unsafe cachingmode, which ignores all cache-flush requests from guestregardless of journaling mode. These combinations sac-rifice both consistency and freshness of data in the virtualdisk for a guest to achieve high virtual I/O performance.

Why vDrive? The fourth circular region in light greencorresponds to the space around the combination of or-dered journaling modes in both guest and host, and write-back virtual caching mode. This is the default combi-nation in KVM/QEMU, which provides consistent andfresh states but at a high performance cost of cache-flushes in nested ordered journaling mode. We focuswithin this region because all three spaces described ear-lier converge here. As a result, moving in different di-rections within this region result in different tradeoffs forperformance, consistency and freshness. Therefore, weaim to find an approach to improve the performance of

3

0

5000

10000

15000

20000

25000

30000

varmailrandom-write

seq-write

IO O

pera

tions

/s

WBNone

WTDirect

Unsafe

Figure 3: Caching modes for virtual I/O: The figureshows the impact of different caching modes for virtualdisk storage on varmail, random write and seq writeworkloads.

nested ordered journaling mode with the ability to re-cover to a consistent state. To achieve this, we use op-timistic crash consistency for virtual I/O by decouplingthe guarantees of write ordering and durability for nestedjournal commits. The vDrive design provides write or-dering but eventual durability to achieve higher runtimeperformance, and recover to a consistent virtual disk im-age, which existed within a bounded time interval beforea crash.

3 Quantifying Virtual I/O Tradeoffs

We now quantitatively show the performance impact ofdifferent virtual caching and nested filesystem journalingmodes. We use the Linux ext4 journaled filesystem inKVM guest and host for our experiments.

Virtual I/O Caches. Figure 3 shows the performanceimpact of the different virtual caching modes for the de-fault ext4 ordered journaling mode with cache-flushesenabled (at filesystem mount-time) in both guest andhost OS. We use the filebench suite [1] of write-intensiveworkloads (random and sequential writes to isolate theimpact of cache hit rates, and varmail: fsync intensive toisolate the cost of crash consistency).

The varmail benchmark simulates a mail server andincludes many small synchronous updates, thereby ex-ercising both the journaling and caching mechanismsthrough frequent cache-flush requests. As a result, allcache modes perform similar and upto 3.5x slower thanthe unsafe mode. None of these four traditional cachingmodes can provide better performance for this workloadwith their existing consistency semantics.

For random write workload, the write-back cachemode is better than other caching modes and compara-ble to unsafe mode. The is because write-back cacheis inclusive of all cache levels and fewer cache-flushcommands from the workload result in larger effective

0 500

1000 1500 2000 2500 3000 3500 4000 4500

7.2K Disk10K Disk

Flash SSD

IO O

pera

tions

/s

Nested Host None

Figure 4: Filesystem journaling for virtual I/O: Thefigure shows the impact of filesystem journaling on dif-ferent storage devices (7.2K RPM SATA, 10K RPM SASDisks and PCI-e Flash SSD). Cache flushes on commitare performed in both guest and host OS (Nested), hostOS only (Host), and disabled in both OS (None).

cache size. The none mode performs about 20% betterthan write-through for both random and sequential writemodes. Disabling the host page cache in none modeturns all write operations into direct I/O operations todisk write cache, which perform better than synchronousI/O in write-through mode. Direct mode performs worstbecause it converts all writes into direct synchronous op-erations each of which result in a disk cache-flush.

Overall, we find that all the virtual caching modesincluding the default write-back mode always performslower than unsafe caching mode. The peformance dif-ference is especially significant when the application andthe filesystem journal within the guest OS issue frequentcache-flush commands to achieve a consistent virtualdisk image.

Nested Filesystem Journaling. We now isolate theimpact of nested filesystem journaling in guest and hostOS on virtual I/O performance. Figure 4 shows the per-formance of three different ext4 ordered journal modeconfigurations: Nested (both guest and host filesystemsuse ordered journal mode with cache-flushes enabled),Host (unlike Nested mode, cache-flushes are disabled forguest filesystems) and None (cache-flushes are disabledfor both guest and host filesystems). We conduct experi-ments on three different storage devices: a nearline 7.2KRPM SATA disk, an enterprise 10K RPM SAS disk, anda high-end PCI-e Flash SSD. For brevity, we only showthe results for the varmail workload and write-back cachemode.

Nested filesystem journaling has significant perfor-mance impact on disk storage. For a nearline disk, dis-abling cache-flushes for guest filesystem journal com-mits results in nearly a factor of five performance im-provement. The enterprise disk has relatively lower seekand data transfer times from disk write cache to disk be-

4

cause of a faster spindle. Nonetheless, the performanceloss is still more than 3x for nested filesystem journaling.Disabling host filesystem journal cache-flushes result inonly about 8% performance improvement over no flushesin guest (Host configuration in Figure 4). Overall, nestedfilesystem journaling with cache-flushes enabled on bothguest and host has significant negative performance im-pact on virtual I/O performance for disk storage.

Figure 4 also shows the performance impact of nestedjournaling on a high-end PCI-e Fusion-io SSD [11]. Wealso found similar results on an Intel SATA SSD [17].As expected, flash SSDs have faster random access laten-cies than disks. Therefore, cache-flushes resulting fromnested filesystem journal commits in guest and host oper-ating systems have less than 10% performance impact onboth SSDs. One approach to boost nested journaling per-formance on disk storage is the use of an external journalplaced on a flash SSD. However, an external journal ona different device cannot be synchronously updated withfilesystem updates, and the result is an inconsistent orread-only filesystem [8].

These findings indicate that providing low overheadcrash consistency with the use of traditional cache-flushinterface for virtual caches and nested filesystem journalsis challenging on virtualized disk storage. To overcomethese challenges, we design vDrive with the followingthree goals:

• Consistent interface to provide a consistent virtualdisk image across a system crash or power failure.• Cache management to improve performance for the

virtual cache hierarchy.• Filesystem journaling to reduce the cost of cache-

flushes for nested filesystems.

4 System Design

vDrive is a new virtual I/O system designed to improveperformance for virtual I/O and provide a consistent vir-tual disk image after a crash or power failure. This sec-tion describes the design of the virtual disk interfaceand consistency guarantees of vDrive, nested journalingprotocol, and cache management. Figure 5 shows theflow of write and cache-flush requests from the guest tovDrive and from vDrive to the host filesystem.

4.1 vDrive InterfacesThe vDrive design provides a consistent disk interfacethat reflects the need of a virtual disk to: (i) provide in-order guest journal commits to achieve a consistent vir-tual disk image across a crash, and (ii) provide immediateor eventual durability to trade between performance andfreshness of a virtual disk.

The vDrive interfaces, as shown in Table 1, are a smallextension to the standard virtio interface [29] between

Name Functionvflush flush I/O operations from the virtual

disk write cache.vorder order I/O operations queued ahead

of vorder.aio-notify notify that all I/O operations queued

before vorder have been synchro-nized to physical disk.

preadv/pwritev gather/scatter a vector of blocksfrom virtual disk file at a given off-set to host memory buffers.

Table 1: The vDrive Interfaces

guest OS virtual driver and host user-space QEMU vir-tual disk driver, and the posix interface between the vir-tual disk cache manager and the host filesystem.

4.1.1 Guest InterfacesThe standard virtio interface between the guest OS andvirtual disk is based on a virtio ring buffer implemen-tation. The guest driver enqueues read/write/flush re-quests into the ring buffer and kicks-off the buffer to thehost. The guest disk cache manager implemented withinthe QEMU emulation block driver dequeues the requestsfrom the ring buffer and then translates them in a separateI/O thread to the underlying system calls to the virtualdisk file on the host filesystem.

vDrive provides two new synchronization primitivesto decouple ordering and durability guarantees of guestflushes to the virtual disk.

Write Flush. The vflush primitive is similar to the tra-ditional flush interface used for ATA/SAS disks, alsotermed as “cache-flush” or “synchronize cache”, whichflushes all buffered writes from the disk write cache. Thisprimitive provides both immediate durability and correctwrite ordering. It only returns when the buffered writeshave been acknowledged to be flushed from all three hostcache levels including the guest disk write cache, hostpage cache and the physical disk write cache. As a result,the cost of the vflush primitive is larger for a virtual diskthan the traditional cache-flush for physical disk writecaches.

Write Order. The vorder primitive provides orderingfor all writes buffered within the three host cache lev-els. When the operation returns, this request has beenonly submitted to the I/O queue in the host OS. Immedi-ate durability of the preceeding writes is not guaranteedupon its return. However, all preceeding writes completein the order as they were submitted by the guest driverto the host emulation framework. As a result, new writesissued after vorder will always be durable after the writespreceeding vorder. The cost of using this primitive is al-

5

ways less than that of vflush as it does not acknowledgefor durability.

By default, all cache-flush requests from the guestfilesystem are converted to vorder requests from theguest virtual I/O driver. The application and guest oper-ating system also has an option of using the vflush prim-itive to force flush through all cache levels in the host.We describe the selection mechanisms and policies forthe use of these primitives in more detail in our discus-sion on semantic journaling in Section 4.2.

4.1.2 Host InterfacesThe vDrive cache manager is implemented within theQEMU user-space virtual disk driver and controls whenblocks are flushed from the three host caches: guest diskcache, host page cache and physical disk cache. It ex-tends the existing interface used between the traditionalQEMU user-space virtual disk driver and the host filesys-tem.

Flush Notification. The vDrive block driver can is-sue new writes after vorder, which reach disk after allpreceeding writes issued before vorder. However, thevorder primitive does not wait to acknowledge the dura-bility of preceeding writes. The vDrive cache managerinstead uses the aio-notify interface to asynchronouslyacknowledge the completion of all preceeding writes.The aio-notify is a new interface implemented within thevDrive cache manager as a signal handler. Specifically,this primitive is used to receive a notification when allwrites buffered within the host caches prior to a vorderoperation have been flushed to the disk. The vDrivecache manager also updates additional information suchas the number of pending write operations and resetsa timer within the aio-notify call. As we will discusslater in Section 4.2, this information is useful to boundthe freshness of the virtual disk image recovered after acrash.

Scatter/Gather I/O. The vDrive cache manager usesthe existing scatter-gather vectored interface imple-mented within the QEMU block driver to issue read-/write system calls to the host filesystem. In addition, itupdates the number of write operations pending to be ac-knowledged through aio-notify or vflush on each pwritevoperation.

4.2 Semantic JournalingThe new vDrive interfaces enable the design of semanticjournaling to reduce the cost of crash consistency for I/Ovirtualization. To understand semantic journaling better,we first examine the interaction between the guest andhost filesystems using Linux ext3/ext4 ordered journal-ing mode protocols. Much of our discussion is also ap-plicable to other journaling filesystems such as IBM JFS,SGI XFS and Windows NTFS.

Journaling Protocol. Figure 5 shows the flow of ap-plication writes to a data block (D) through the guestfilesystem, virtual I/O driver, guest cache manager withinthe host user-space QEMU block driver and the hostfilesystem. When a guest application updates the filesys-tem state, either the filesystem metadata (M), user data(D), or ofen both need to be updated in an ordered man-ner. The atomic update of the filesystem metadata in-cluding the inode and allocation bitmap to the journalis referred to as a transaction. The filesystem must firstwrite data blocks (D) and log the metadata updates (JM )to the journal (filesystem write W1), then finally writea commit block (JC) to the journal to mark transactioncommit (filesystem write W2). Finally, the metadata (M)can be written in place to reflect the change (filesystemwrite W3). Therefore, the journaling protocol is: D andJM before JC before M, or more simply: D | JM →JC → M. The data (D) and the journal metadata entries(JM ) can represent multiple disk blocks within a transac-tion, whereas the commit record (JC) is always a singlesector. In summary, for each application write to data(D), there are three logical filesystem write operations:W1, W2 and W3 (see Figure 5). A logical write can becomposed of multiple physical disk writes to discontigu-ous blocks. There is no ordering required within a logicalwrite itself for ordered journal mode [27].

The guest filesystem also issues cache-flush com-mands (shown as F1 and F2 in Figure 5), wherever or-der (→) is required between the different writes in thisprotocol. This is because the protocol has been origi-nally designed to use a physical disk cache-flush inter-face, which provides both write ordering and immediatedurability. Recent work on optimistic crash consistencyproposes to decouple the cache-flush ordering and dura-bility semantics with modifications to a single level offilesystem journal and the physical disk interface [5]. Aswe will show next, implementing optimistic crash con-sistency for vDrive through semantic journaling mainlyrequires changes to the virtual disk interface and the vir-tual I/O system, and minimal changes to the guest filesys-tem.

The traditional KVM/QEMU virtual I/O system (guestdriver and host QEMU block driver) further translateseach cache-flush command into a fsync system call tothe virtual disk file on the host filesystem. As shown inFigure 5, the host filesystem uses another level of jour-naling protocol (d | jm → jc → m) separately for eachof the three logical writes (d being W1, W2 and W3 re-spectively). The nesting of the two journaling protocolshas two negative performance impacts. First, it results inwrite amplification: a guest application write can be con-verted into upto nine writes to the physical disk. Second,it results in write stalls: the first guest cache-flush (F1)only required for guest write ordering stalls the virtual

6

Applica'on  

Guest  FS  journal  

Virtual  I/O  Driver  

virtual  disk  OR  vDrive  (Host  User-­‐Space)  

Host  FS  journal  

Host  I/O  Driver  

Disk  

F1  

D  |  JM  à  JC  à  M  

write(D)  

F1  :  flush1  F2  :  flush2  

F31  

d  |  jm  à  jc  à  m  

W1   W2   W3  

F2  

W1  W2  

W3  

F32  

F1  

D  |  JM  à  JC  à  M  

write(D)  

F1  :  vorder1  F2  :  vorder2  

F31  

d  |  jm  à  jc  à  m  

W1   W2   W3  

F2  

W2  

W3  

F32  

W1  

ack   aio-­‐no'fy  

Nested  Journaling  on  Virtual  Disk  

Seman'c  Journaling  on  vDrive  

stall  

stall  

Figure 5: Nested and Semantic Journaling: This figure shows the interaction between guest and host filesystemjournaling for nested journaling on traditional virtual disk and semantic journaling on vDrive. The vDrive designuses vorder and aio-notify interfaces.

I/O system until the first group of host filesystem updates(d | jm → jc → m corresponding to W1) have been ac-knowledged to be flushed to disk. The second group ofupdates corresponding to W2 is only issued after receiv-ing the acknowledgement for F1. Similarly, F2 results inanother stall between the group of writes correspondingto W2 and W3.

In contrast, we observe that the cache manager andvirtual disk are emulated entirely within the host soft-ware. As a result, the vDrive design has the flexibilityto address the cost of nested journaling by using its newsynchronization interfaces for cache-flush requests. Weclassify the cache-flush requests from the guest filesys-tem based on the semantics of the data being persisted.For simplicity, we pass the semantic information used forclassifying the cache-flush requests by annotating themwithin the guest filesystem and virtual memory subsys-tems. With the use of more sophisticated approachesproposed in the past [35, 37], this classification can bealso done by rather discovering the semantic informationwithin the virtual I/O system.

The vDrive classifies the cache-flush command basedon its semantic requirements for ordering and durabilityto provide a consistent virtual disk image after a crash.There are broadly four major classifications based onwhen the cache-flush is issued: journal transaction com-mit, virtual memory page write-backs, checkpoints forjournal truncation, and flushing I/O queues after a diskwrite failure.

Journal Commits. vDrive converts all guest filesys-tem cache-flush commands after the journal commit

record (JC) into vorder request to the virtual disk. Thisensures correct write ordering both within and across dif-ferent guest filesystem transactions without incurring thecost of immediate durability for each journal commit.The vDrive also keeps track of the time elapsed since thelast vorder completion has been notified through the aio-notify handler. If this time interval exceeds the freshnessthreshold for vDrive and there are pending write opera-tions, it issues a vflush to the host filesystem. This en-sures that the virtual disk image recovered after a crashis always consistent and has all updates older than thefreshness threshold before the crash. If the workloadis read-intensive, there are fewer pending write opera-tions and the cost of a vflush operation is not incurred byvDrive.

VM Page Write-backs. In addition to the journal com-mits, the guest virtual memory subsystem also writes-back pages when the dirty to clean page ratio exceedsa configured threshold. As these write-backs happen inthe background and do not require immediate durability,vDrive only requires correct ordering for them with otherwrites. As a result, vDrive uses the vorder primitive forsuch VM page write-backs.

Journal Truncation. When the guest journal gets full,a cleanup is required for the journal tail to re-use spacein memory and disk pre-allocated for the journal. Thejournal metadata checkpoint (M) and all transactions cor-responding to the re-used journal space are flushed tothe virtual disk before the cleanup starts. vDrive issuesa vflush request for all such cache-flush requests to en-

7

force immediate durability and avoid any transactions orcheckpoints lost due to cleanup.

Write Failure. In addition to the journal truncation,vDrive also classifies a cache-flush request issued when anew write fails because of a stopped guest I/O schedulerqueue. The guest I/O queue is stopped when the devicedriver signals it cannot service further requests becauseof full virtual/hardware disk queue. The vDrive issues avflush request to flush all enqueued requests with imme-diate durability guarantee and only then allows the queu-ing of the new write request.

4.3 Crash Behavior and RecoveryWith the use of default write-back virtual cache and or-dered nested filesystem journaling modes, the filesys-tem recovers by replaying the journal upto the last fullycommitted transaction. However, a partially committedtransaction could happen if a crash happens after the datablock (D) or the JM , but before the commit record hasbeen written out to the journal. If the data write is to anew allocated block, it would result in garbage data afterrecovery. Otherwise, if it was an overwrite to an alreadyallocated block, it would result in partially updated dataafter recovery. There is no data corruption possible in or-dered journaling mode because metadata of the filesys-tem always points to valid data that existed before thecrash. As a result, both the filesystems in guest and hostwill always recover to a consistent state that existed be-fore the crash.

The vDrive design provides similar crash recoveryguarantees: the filesystem always recovers to a consis-tent state that existed before the crash. The host filesys-tem still uses the ordered filesystem, and provides similarguarantees as before. With the use of vorder for journalcommits and vflush operations for journal truncation, thevDrive journaling mechanism has the following orderinginvariants for the guest filesystem writes:

1. Data block (D) and journal metadata entries (JM )within a transaction always reach disk before thejournal commit record (JC).

2. Two different transactions are always committed in-order.

3. A transaction is never released for journal re-usebefore all previous checkpointed blocks (M) areflushed to disk.

Upon recovery, a partially committed guest transactioncould still result in garbage data or partially updated dataas before. However, these vDrive invariants guaranteethat the metadata in the filesystem never points to invaliddata, which did not exist. However, within this consis-tency semantics, we relax the guarantee about freshnessof data to trade for higher performance. Journal replaybrings the guest filesystem to a consistent state, which

existed before the crash, but not necessarily the mostfresh state. The vDrive architecture bounds the consis-tent state to be no older than the freshness threshold ofthe virtual disk.

5 Implementation

The implementation of vDrive entails three components:the semantic classifier, the vDrive virtual disk interfaceand the cache manager. The first two are implementedinside the Linux 3.2 kernel as modifications to the jbd2journaling module for ext4 filesystem and virtio blk vir-tual I/O block driver respectively. The cache manageris implemented within the QEMU block driver for rawposix I/O.

We base the semantic classifier on the jbd2 journalinglayer and the virtual memory subsystem in Linux, whichenables to classify the different cache-flush requests sentto the block layer. This makes our modifications modu-lar and does not require changes to the ext4 filesystemstructures. At the block layer, we sought to leave asmuch code as possible unmodified. Thus, we augmentcache-flush requests from the higher layers with an ad-ditional field, effectively adding our new interface com-mands as sub-types of the existing cache-flush command.The write to the commit record in the journal transactioncommit adds the vorder sub-type. Similarly, the writesfrom the virtual memory writeback thread also adds thevorder sub-type to the associated flush command. How-ever, the writes during the journal cleanup for check-pointing add the vflush sub-type to ensure that journalspace is not re-used for a transaction before it is com-mitted and its metadata is checkpointed. The I/O queuerestart code-path also uses the vflush sub-type to flush allthe preceeding requests in the queue on a write failure.The block layers pass the sub-type field down to the nextlayer. We modified the I/O scheduler to consider bothsub-types as a traditional cache-flush command whileservicing them. This enables us to achieve the same or-dering guarantees for vorder as for a traditional cache-flush request in the guest operating system.

We modify the guest virtio blk driver, which imple-ments the interface to the virtual block device in the hostemulated using the QEMU block driver. It inserts the I/Orequests in the virtio ring buffer and triggers the QEMUblock driver to service those requests using a work queueof threads. We modified the virtio blk driver to inserta different vorder and vflush request into the virtio ringbuffer based on the sub-type of the command receivedfrom the block request queue in guest. The QEMUemulation framework pops off the request from the vir-tio ring and hands it to the QEMU block driver.

We base our cache manager implementation on theblock driver for raw posix I/O in QEMU. The raw

8

posix driver provides best performance than other driverssupporting different image formats and implementingadditional features [28]. We modify the code pathfor handling cache-flush requests in the block driver.The QEMU block driver issues a fdatasync system callfor each cache-flush command received from the guestdriver. We modify it to issue an aio fsync system callwith D SYNC flag for a corresponding vorder commandreceived from guest. On vDrive initialization, we setupan asynchronous I/O control block, signal event handlerfor aio fsync implementing the aio-notify notification,and a freshness threshold timer. We reset the freshnesstimer on each aio-notify notification or vflush comple-tion. Similarly, we update the number of write opera-tions issued so far on each pwritev call to the host filesys-tem. The cache manager implementation has the flexibil-ity to use alternative notification interfaces with differentsemantics such as posix inotify, asynchronous durabil-ity notifications [5], and callbacks implemented on PCI-e based storage systems [41, 33]. The cache manageralso forces a vflush operation if there are pending writeoperations and the freshness timer exceeds the fresh-ness threshold of vDrive. We configure the freshnessthreshold to match the average latency of a single cache-flush request: 50 ms. We find through experiments thatthis provides good runtime performance and still tightlybounds the virtual disk freshness within the cache-flushrequest latency.

6 Evaluation

We evaluate vDrive’s design components against tradi-tional virtual disk storage on two axes: performance andreliability.

6.1 MethodsExperiments were performed on a machine equippedwith two sockets and 8 cores-per-socket Intel XeonE5530 CPU running at 2.4 GHz with 24 GB of memory,146 GB 10K RPM IBM 42D0421 SAS drive, and run-ning Linux 3.2 and KVM/QEMU 1.5.3. We use an IntelS3700 SATA SSD [17] and PCI-e Fusion-io SSD [11] forexperiments reported in Section 3. The guest virtual ma-chine is configured with 4 GB memory, 4 core processorand a virtual disk of 60 GB.

We compare the vDrive virtual I/O system usingsemantic journaling and new virtual disk interfacesagainst the baseline system, which uses the unmodifiedKVM/QEMU virtual disk. All systems use Linux ext4ordered mode filesystems in both guest and host oper-ating systems. The baseline system is configured inthree different modes by using different barrier optionsto configure cache-flushes when mounting the filesys-tems: nested journal with disk cache-flushes enabled in

both guest and host, only the guest journal enabled withcache-flushes, no journal enabled with cache-flushes.These three modes present different performance andconsistency tradeoffs for the baseline system. We useraw posix I/O image and write-back guest disk cachemode as they both provide best performance across dif-ferent virtual disk image configurations.

We first measure performance of vDrive under a num-ber of micro- and macro-benchmarks (Section 6.2) run-ning in the guest. We use random and sequential writeworkloads with 200 K writes with an fsync every 1Kwrites over a 10 GB file. These micro-benchmarks areused to exercise the journaling code paths and are dif-ferent from the micro-benchmarks used in Section 3to isolate the performance difference between differentcache modes. We run the filebench varmail (1:1:1 read-s/writes/fsyncs), fileserver (1:2 reads/writes), and web-server (10:1 reads/writes) workloads [1] and MySQLbenchmark (200K OLTP transactions on a table of 1Mrows) from sysbench [2]. These macro-benchmarks ex-ercise all the read, write and fsync code paths of the vir-tual storage stack.

We also perform two case studies (Section 6.3) todemonstrate how vDrive provides crash consistency andrecovery using atomic writes used within a text editor(gedit) and log management within a database (SQLite)running within the guest. We write a tool to captureblock-level I/O traces at the host virtual disk layer andanalyze the crash behavior by applying a subset of writesfrom the trace to the original disk image to reconstructthe image after crash.

6.2 System ComparisonFigure 6 shows the performance of the vDrive systemand three different journaling mode combinations (seeSection 6.1) normalized to nested journaling in the base-line system. The first four workloads: varmail, MySQL,rand-write, and seq-write are write/fsync intensive, thusexercise the journaling disk cache-flush interface. Thefileserver benchmark is write-intensive and is boundby the overhead of frequent metadata updates for ap-pend/delete system calls to the guest filesystem. Sim-ilarly, the webserver benchmark is read-intensive andis bound by the random read performance of the vir-tual disk. As a result, these two workloads get most oftheir requests hit in host caches and have minimal per-formance impact of 8%-11% because of journaling orcache-flushes.

For the four write/fsync-intensive workloads, dis-abling cache-flushes for host journal (guest system) im-proves performance by 15%-56%, but disabling bothguest and host (no system) cache-flushes has a muchlarger impact. The guest journal impacts performancemore because it flushes out guest disk cache and host

9

2.42  

1.96  

1.54   1.47  

1.11   1.08  

varmail   MySQL   rand-­‐write   seq-­‐write   fileserver   webserver  

Nested   Guest   No   vDrive  

1  

2  

3  

8.30  

28.14  10.75  

1312    op/s  

153    tx/s  

394    op/s  

13378  op/s  

2603  op/s  

3178  op/s  

Performance  normalize

d  to  Nested  Journaling  

Figure 6: Application Performance: The performanceof guest-only journaling, no journaling, and vDrive’s se-mantic journaling normalized to nested journaling. Theabsolute performance of nested journaling and improve-ment of vDrive’s semantic journaling are shown as labelsin the figure.

page cache on a journal commit. This corrobates withthe vDrive design, which exploits the guest journal se-mantics to reduce its overhead.

The vDrive system with semantic journaling outper-forms nested journaling by 47%-142%. This improve-ment is mainly due to the use of the vorder interface,which allows for lower write stall times and more parallelwrites to the host filesystem while still maintaining thecorrect order for writes to disk. In contrast, the no cache-flush system does not provide any ordering or durabilityfor writes and result in substantial performance improve-ment over nested journaling, especially for the write-only random and sequential workloads (10x-28x im-provements over nested journaling). This is because allwrites in these workloads complete out-of-order and getacknowledged as soon as they hit any of the in-memorycache levels within the host. The vDrive system is 22%-95% slower than the no cache-flush system, which is at-tributed to the cost it pays for correct write ordering usingthe vorder and bounded freshness using the vflush inter-face for guest journal commits.

6.3 Case StudiesWe now evaluate how vDrive’s crash behavior comparesagainst nested journaling. We use two applications: anatomic write to a file within a popular text editor (gedit)and log management within the SQLite database.

Table 2 shows the crash behavior for nested journal-ing on the baseline system and semantic journaling onvDrive respectively. We capture block-level traces at thehost virtual disk layer when executing the application inguest. We then simulate crashpoints by applying subsets

Gedit SQLiteSystem Nested vDrive Nested vDriveTotal Crashpoints 10 10 30 30Old State 3 5 12 21New State 7 5 18 9

Table 2: Crash Behavior. The number of crashpoints sim-ulated for two applications, and the resulting applicationstates after remounting the filesystem in the guest on thevirtual disk image obtained after crash. Old and newstates correspond to before and after application execu-tion.

of writes from the trace to the original virtual disk imageto reconstruct the crashpoint virtual disk image. We thenrestart the guest virtual machine, remount the filesystemon the reconstructed virtual disk image, and run recoveryto test for crash behavior.

Atomic Write. The atomic write to a file within geditcreates a new version of the file under a temporary name,issues a fsync to flush the atomic write to disk, and re-names the temporary file to the original file name. Asa result, an atomic write ensures that either the old con-tents before the write (old state) or new contents after thewrite (new state) are available in the file in entirety, bothof which are consistent states.

As shown in Table 2, nested journaling uses the cache-flush interface and points to either the old or new contentsof the file after recovery. Semantic journaling preservesthe same crash behavior as nested journaling by alwaysrecovering to a consistent state. However, it recovers tothe old state more frequently than nested journaling. Thisis because journal commits use the vorder interface invDrive, which delays durability for writes. Neither ofthe two modes result in an inconsistent state where thefilesystem was not able to recover or the recovered filedid not have old or new contents or have a mix of con-tents.

Log Management. The SQLite database uses a twofsync sequence for committing its transactions: first af-ter committing its writes to a log file, and second aftermaking in-place updates to the database. We simulatea SQLite transaction transferring data across a set of ta-bles. After the filesystem recovery, the SQLite databaseperforms its own recovery to reach to a consistent state,which reflects either pre-transaction (old state) or post-transaction (new state) changes.

As earlier, recovery on vDrive results in the old pre-transaction state for about 70% of the crashpoints. ThevDrive design trades off the freshness of the virtual diskimage for application runtime performance. For thenested journal, we find that the database recovers to thepost-transaction state more often. This is because there

10

is a higher chance that either the filesystem or the SQLitewill recover to the new state. If the first fsync is notcommitted, the filesystem recovers to an old state andthe database recovery performs a roll-back from the logfile. If the first fsync is committed, the filesystem recov-ers to an old state but the database recovers to the newstate by replaying the log changes. If both the fsyncs arecommitted, both the filesystem and database recover tothe new state. In all cases, both vDrive and baseline sys-tems always recover to either the old or new state, bothof which are consistent.

We also tested the no cache-flush journaling modefrom Section 6.2, which provides performance higherthan semantic journaling, but causes delayed as well asre-ordered writes. In this mode, the crashpoints resultedin different inconsistent images with unmountable orread-only filesystems, orphaned filesystem inodes, anddata corruption with mixed contents from the old andnew states.

7 Related Work

Our work builds on past work on virtual I/O, new storageinterfaces, and consistent storage systems.

Virtual I/O Performance. Recent work has soughtto improve the performance of I/O virtualization.ELVIS [15] and ELI [14] improve virtual I/O perfor-mance for network and read/write I/O in KVM/QEMU.They propose the use of exit-less interrupts and pollingto reduce the number of switches between guest andhost. Further, they scale virtual I/O performance by us-ing x86 hardware support for posted interrupts. Pastwork [31, 25] has also investigated guest schedulingmechanisms to improve virtual network and I/O perfor-mance in the context of Xen [4]. Similarly, the per-formance overheads for nested filesystems have beenanalyzed with advice implied for new filesystem de-signs [21]. In contrast, vDrive is the first system im-plementation to account for the performance and con-sistency tradeoffs for journaling in nested filesystems.vDrive provides high performance for crash consistencywith the use of journaled filesystems, and does not re-quire any costly additional data copies as in other consis-tency techniques such as local snapshots and distributedreplication [28, 39].

New Storage Interfaces. Recent proposals have in-vestigated storage interface extensions for specific usecases such as flash caching [32, 10, 23], new filesys-tems [41, 19, 5], flash databases [26], and key-valuestores [12]. Most of these works have benefitted from theinternal mechanisms within a flash SSD such as logging,garbage collection and sparse address spaces. How-ever, such a tight integration mostly requires offloading

software functionality to hardware with changes to thestandard SATA/SCSI protocols [33, 41, 5, 30] or cus-tom communication channels [12, 22] on the PCI-e bus.In contrast, this work specifically targets disk storageand virtual environments where the benefits of provid-ing high performance data consistency are significantlyhigher than flash storage. In addition, Xen and virtiosupport paravirtual I/O barriers for SCSI block devicesto enforce write ordering only limited for write-throughdisk caching [30, 29, 4]. vDrive provides significantlybetter control than these operations by using a combina-tion of vflush, vorder and flush notifications that work forall caching modes in guest and host systems.

Consistent Storage. A number of approaches to build-ing higher performing and crash-consistent filesystemshave been investigated in the past [13, 9, 6]. How-ever, most of these proposals either increase filesystemcomplexity significantly [13, 6] or provide very general-ized frameworks to order filesystem updates [9]. Recentworks on filesystem [5], databases [7] and storage proto-cols [30] propose the use of optimistic crash consistencyand closely relate to vDrive’s design principles. In thiswork, we explore the first use of optimistic crash con-sistency to address the performance overheads of nestedjournaling. We find that it requires major changes to thevirtual I/O system and rethinking the software interfacebetween the guest and host operating systems. Recently,new optimizations have also been proposed to minimizethe overheads for nested journaling arising from SQLitelog and ext4 filesystem journal in the Android I/O stackon flash storage [18, 20, 34]. However, those optimiza-tions focus primarily on cheap and slow flash storagepopularly used in smartphones or tablets, and are not di-rectly applicable to virtual disk I/O stack.

8 Conclusions

Virtual I/O has been widely adopted in production en-vironments for application and infrastructure services inthe recent past. We present a first look on the crash con-sistency and performance tradeoffs for virtual I/O. Ourfindings reveal deep insights into the performance costsand interaction of nested filesystem journals. We pro-pose a new virtual disk interface and semantic journalingtechnique to address the performance overheads associ-ated with nested journaling. As new non-volatile mem-ory technologies become available, such as phase-changeand storage-class memory, it will be important to revisitthe interface to further address the system call overheadsfor virtual I/O.

11

Acknowledgements

We thank our shepherd Nitin Agrawal and the anony-mous reviewers for their helpful comments and usefulfeedback.

References

[1] Filebench. http://sourceforge.net/projects/filebench.

[2] Sysbench benchmark. https://launchpad.net/sysbench.

[3] AMAZON. Elastic Compute Cloud2. http://aws.amazon.com/ec2.

[4] BARHAM, P., DRAGOVIC, B., FRASER, K.,HAND, S., HARRIS, T., HO, A., NEUGEBAUER,R., PRATT, I., AND WARFIELD, A. Xen and theart of virtualization. In SOSP (2003).

[5] CHIDAMBARAM, V., PILLAI, T. S., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H.Optimistic crash consistency. In SOSP (2013).

[6] CHIDAMBARAM, V., SHARMA, T., ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H.Consistency without ordering. In FAST (2012).

[7] CIPAR, J., GANGER, G., KEETON, K., III, C.B. M., SOULES, C. A., AND VEITCH, A. Lazy-base: Trading freshness for performance in a scal-able database. In EuroSys (2012).

[8] EXT4 FILESYSTEM. external journal caveats.http://www.raid6.com.au/posts/fs_ext4_external_journal_caveats.

[9] FROST, C., MAMMARELLA, M., KOHLER, E.,DE LOS REYES, A., HOVSEPIAN, S., MAT-SUOKA, A., AND ZHANG, L. Generalized filesys-tem dependencies. In SOSP (2007).

[10] FUSION-IO INC. directCache. http://www.fusionio.com/data-sheets/directcache.

[11] FUSION-IO INC. ioDrive SSD Datasheet.http://www.fusionio.com/products/iodrive.

[12] FUSION-IO INC. ioMemory Application SDK.http://www.fusionio.com/products/iomemorysdk.

[13] GANGER, G. R., AND PATT, Y. N. Metadata up-date performance in filesystems. In OSDI (1994).

[14] GORDON, A., AMIT, N., HAR’EL, N., AND BEN-YEHUDA, M. Eli: bare-metal performance for i/ovirtualization. In ASPLOS (2012).

[15] HAR’EL, N., GORDON, A., LANDAU, A., BEN-YEHUDA, M., TRAEGER, A., AND LADELSKY,R. Efficient and scalable paravirtual i/o system. InUsenix ATC (2013).

[16] IBM. Cloud Computing. www.ibm.com/cloud-computing.

[17] INTEL INC. S3700 Series 200GB SSD.http://www.intel.com/content/www/us/en/solid-state-drives/solid-state-drives-dc-s3700-series.html.

[18] JEONG, S., LEE, K., LEE, S., SON, S., ANDWON, Y. I/o stack optimizations for smartphones.In Usenix ATC (2013).

[19] JOSEPHSON, W. K., BONGO, L. A., FLYNN, D.,AND LI, K. DFS: a file system for virtualized flashstorage. In FAST (2010).

[20] KIM, W.-H., NAM, B., PARK, D., AND WON, Y.Resolving journaling of journal anomaly in androidi/o: Multi-version b-tree with lazy split. In FAST(2014).

[21] LE, D., HUANG, H., AND WANG, H. Understand-ing performance implications of nested file systemsin a virtualized environment. In FAST (2012).

[22] MARVELL CORP. Dragonfly platform fam-ily. http://www.marvell.com/storage/dragonfly/, 2012.

[23] MESNIER, M., AKERS, J. B., CHEN, F., ANDLUO, T. Differentiated storage services. In SOSP(2011).

[24] MICROSOFT. Windows Azure. http://www.windowsazure.com.

[25] ONGARO, D., COX, A. L., AND RIXNER, S.Scheduling i/o in virtual machine monitors. In VEE(2008).

[26] OUYANG, X., NELLANS, D., WIPFEL, R.,FLYNN, D., AND D.K.PANDA. Beyond block i/o:Rethinking traditional storage primitives. In HPCA(feb. 2011), pp. 301 –311.

[27] PRABHAKARAN, V., ARPACI-DUSSEAU, A. C.,AND ARPACI-DUSSEAU, R. H. Analysis and evo-lution of journaling filesystems. In Usenix ATC(2005).

[28] QEMU. Virtual Disk Image Formats. http://en.wikibooks.org/wiki/QEMU/Images.

[29] RUSSELL, R. virtio: Towards a de-facto standardfor virtual i/o devices. ACM SIGOPS OperatingSystem Review 42, 5 (July 2008).

12

[30] RUSSELL, R. Virtio pci card specification,May 2012. http://ozlabs.org/˜rusty/virtio-spec/virtio-0.9.5.pdf.

[31] SANTOS, J. R., TURNER, Y., JANAKIRAMAN, G.,AND PRATT, I. Bridging the gap between softwareand hardware techniques for i/o virtualization. InUsenix ATC (2008).

[32] SAXENA, M., SWIFT, M. M., AND ZHANG, Y.FlashTier: A lightweight, consistent and durablestorage cache. In EuroSys (2012).

[33] SAXENA, M., ZHANG, Y., SWIFT, M. M.,ARPACI-DUSSEAU, A. C., AND ARPACI-DUSSEAU, R. H. Getting real: Lessons intransitioning research simulations into hardwaresystems. In FAST (2013).

[34] SHEN, K., PARK, S., ZHU, M., SHEN, K., PARK,S., AND ZHU, M. Journaling of journal is (almost)free. In FAST (2014).

[35] SIVANTHU, M., PRABHAKARAN, V., POPOVICI,F. I., DENEHY, T. E., ARPACI-DUSSEAU, A. C.,AND ARPACI-DUSSEAU, R. H. Semantically-smart disk systems. In FAST (2003).

[36] STEIGERWALD, M. Imposing order. In Linux Mag-azine (2007).

[37] TARASOV, V., JAIN, D., HILDEBRAND, D.,TEWARI, R., KUENNING, G., AND ZADOK, E.Improving i/o performance using virtual disk intro-spection. In HotStorage (2013).

[38] VMWARE. Performance evaluation of vmxnet3 vir-tual network device. In Technical Report (2009).

[39] VMWARE INC. VMWare Virtual MachineFile System: Technical Overview and BestPractices. http://www.vmware.com/pdf/vmfs-best-practices-wp.pdf.

[40] YASSOUR, B.-A., BEN-YEHUDA, M., ANDWASSERMAN, O. Direct device assignment for un-trusted fully-virtualized virtual machines. In IBMResearch Report H-0263 (2008).

[41] ZHANG, Y., ARULRAJ, L. P., ARPACI-DUSSEAU,A., AND ARPACI-DUSSEAU, R. De-indirectionfor flash-based ssds with nameless writes. In FAST(2012).

13