qufiles: the right file at the right time
DESCRIPTION
quFiles: The right file at the right time. Kaushik Veeraraghavan Jason Flinn Ed Nightingale * Brian Noble University of Michigan *Microsoft Research (Redmond). Users need different data for different contexts. Screen size & battery lifetime. Network bandwidth & latency. Platform. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/1.jpg)
quFiles: The right file at the right time
Kaushik VeeraraghavanJason Flinn
Ed Nightingale*
Brian Noble
University of Michigan*Microsoft Research (Redmond)
![Page 2: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/2.jpg)
Users need different data for different contexts
Kaushik Veeraraghavan 2
Users want to see the right file at the right time
Screen size &battery lifetime Platform
Network bandwidth & latency
![Page 3: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/3.jpg)
• Problem: each application builds both, an adaptation system and a data management system
• Our contribution: common abstraction for context-aware data management
Free developers to build interesting adaptation schemes!
Decouple adaptation from management
Kaushik Veeraraghavan 3
Energy-aware adaptation: SOSP ‘99
Distillation: ASPLOS ‘96
Proxy
![Page 4: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/4.jpg)
Layer context-awareness in existing FS
• The way data is presented to users can be different from how it is stored– Change the interface used to access data
Kaushik Veeraraghavan 4
• Create new context-aware systems by just writing policies- We built two new applications in a couple weeks!
• Existing applications that use the file system become context-aware without any modification
![Page 5: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/5.jpg)
• quFiles multiplex different views of a single logical object
• Context-aware mechanism selects the best representation
quFiles: a unifying abstraction
5Kaushik Veeraraghavan
![Page 6: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/6.jpg)
Talk outline
• What are quFiles?• Design & Implementation• Case studies• Evaluation• Related work• Conclusion
Kaushik Veeraraghavan 6
![Page 7: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/7.jpg)
Life of a quFile
• Utility creates alternate representations of video
quFile utilityfoo.mp4
foo_low.mp4 foo.tivo
7Kaushik Veeraraghavan
/videos/foo.qufile
• Utility creates a quFile and moves representations into it
~~~~~~
~~~~~~
~~~~~~
• Links in the policies
![Page 8: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/8.jpg)
Name policy: choosing the right name
• Name policy: 0 or more file names
Kaushik Veeraraghavan 8
foo.mp4
foo.tivo
foo.mp4
Name policy:If (device == TiVo) { return “foo.tivo”;} else { return “foo.mp4”;}
• Policy may dynamically instantiate a new name
![Page 9: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/9.jpg)
Content policy: choosing the right content
Kaushik Veeraraghavan 9
• Content policy: specific content for file name
• Policy may dynamically create a new file and content
foo.mp4
foo.tivo
foo.mp4
![Page 10: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/10.jpg)
quFile edit and cache policies
• Edit policy: allow, disallow or version
• Cache policy: which representation to cache
Kaushik Veeraraghavan 10
foo.mp4
foo.tivo
foo.mp4
ALLOW
DISALLOW
![Page 11: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/11.jpg)
% % ls video.qufile.raw%
quFiles support multiple views
• Raw view: shows all contents i.e. representations, policies,…
11Kaushik Veeraraghavan
~~~~~~
~~~~~~
~~~~~~
% ls video.qufile.highres%
• Custom view: policy may return any representation it wishes
• No application modification is required to see other views
![Page 12: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/12.jpg)
Talk outline
• What are quFiles?• Design & Implementation• Case studies• Evaluation• Related work• Conclusion
Kaushik Veeraraghavan 12
![Page 13: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/13.jpg)
• Cache policy: use spare storage to cache WAV• Name & content policy: return WAV if cached, else
mp3• 4-11% battery lifetime gain; lines of policy code:
94
Power management
13Kaushik Veeraraghavan
![Page 14: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/14.jpg)
Copy-on-write versioning
• Custom versions view– Name policy: returns names of all past versions (1, 2 or more)
– Content policy: dynamically generates past version
• Lines of policy code: 55
Kaushik Veeraraghavan 14
$ ls paper
V
$ ls paper.qufile.yesterday
V3
V1 V2
V3
$ ls paper.qufile.versions
V1
V2
V3
• Edit policy: save information to an undo log
![Page 15: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/15.jpg)
Resource-aware directory listing
• Default view: list files viewable given network quality
• Custom “all’’ view: “currently_unplayable” suffix• Lines of policy code: 98
15Kaushik Veeraraghavan
GPR
S
![Page 16: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/16.jpg)
Application-aware adaptation: Odyssey
• Name: bar.jpeg to all clients
• Content: best image served in 1 second
• Edit: disallows content writes, allows metadata writes
• Lines of policy code: 8216Kaushik Veeraraghavan
bar.jpeg
bar.jpeg
bar.jpeg
![Page 17: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/17.jpg)
Talk outline
• What are quFiles?• Design & Implementation• Case studies• Evaluation• Related work• Conclusion
Kaushik Veeraraghavan 17
![Page 18: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/18.jpg)
quFiles are easy to implement and use
Component Name
Content
Edit
Cache
Total
Power Management 32 18 8 36 94
Copy-on-write versioning 29 18 8 N/A 55
Security 20 33 8 N/A 61
Resource-aware directory listing
64 26 8 N/A 98
Odyssey 23 27 32 N/A 82
Platform spec. video display
31 30 8 43 112Kaushik Veeraraghavan 18
• quFiles are easy to incorporate in a file system• quFiles add 1,600 lines to BlueFS’s 28,000.
• Almost all policies (see table) require less than 100 lines.
- Each case study in a week or two. Some 1-2 days.
![Page 19: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/19.jpg)
Micro-benchmark: Directory listing overhead
• Worst-case quFile overhead as there’s no activity to amortize cost– Only 3% overhead for warm; 0.5 ms overhead per file for cold
19Kaushik Veeraraghavan
Warm client & server
0
50
100
150
200
No replication quFile-OdysseyReplication
Tim
e (
mill
iseco
nds)
0
5
10
15
No replication quFile-OdysseyReplication
Tim
e (
mill
iseco
nds)
Warm Cold
• quFiles are 2X-3X better than Replication
![Page 20: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/20.jpg)
Kernel grep
• grep Linux 2.6.24 source: grep –Rn “foo” linux (9 occurrences)
• 1% overhead for warm; 6% overhead for scenario
Warm Cold
20Kaushik Veeraraghavan
0
20
40
60
Series1
Tim
e (
seco
nds)
0
1
2
Series1
Tim
e (
seco
nds)
• Search all versions: grep –Rn “foo” linux.qufile.versions (18 occurrences)
• 2X overhead in warm; 31% in cold case
![Page 21: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/21.jpg)
Related work
• Semantic File System– Only expands name space but not content
• Adaptation systems: Ninja, Odyssey, Puppeteer, …– No application or OS modification, no proxy. Adaptation policies.
• Partial-replication: Cymbiosis, PRACTI, Perspective– Filter-based caching policies can be augmented with context
• Dynamic resolution of file content: OS X bundles, AFS @sys– General abstraction w/o baking resolution policies in FS
• Materialized views in databases– Context-aware generation of views; operate on data without schema
Kaushik Veeraraghavan 21
![Page 22: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/22.jpg)
• quFiles provide first-class support for context in file systems– Multiplex different views onto single logical object– Context-aware policies select the best view
• Context-aware systems can be easily built by simply providing quFile policies
• Thank you!
Conclusion
22Kaushik Veeraraghavan
![Page 23: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/23.jpg)
Kaushik Veeraraghavan 23
![Page 24: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/24.jpg)
Building blocks of quFiles
• Policies are file system extensions– User-level software fault isolation is fine
• File system change notifications– To trigger quFile utilities (automation)
• File system should support directories
• Context library– Simple to build: ours is ~250 LOC
24Kaushik Veeraraghavan
![Page 25: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/25.jpg)
Why put quFiles in the file system and not middleware, library, …
• Any application that uses the file system now becomes context-aware– Transparency ensures backward compatibility
• quFiles are a simple abstraction in the FS– Hooking into POSIX API is simple– readdir, lookup, commit_write, unlink, rename
25Kaushik Veeraraghavan
![Page 26: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/26.jpg)
Case study applicability
• Local & Distributed file systems– Resource management
• E.g.: if battery is low, display low-res video
– Copy-on-write versioning– Context-aware redaction
• Distributed file systems– Resource-aware directory listing– Application-aware adaptation: Odyssey
Kaushik Veeraraghavan 26
![Page 27: quFiles: The right file at the right time](https://reader033.vdocuments.mx/reader033/viewer/2022052510/56813820550346895d9fcf8a/html5/thumbnails/27.jpg)
Andrew-style make
• Make Linux 2.6.24 kernel– quFile: version all source files (.c, .h or .S) – 19,844 of 23,062
files
• Negligible overhead for warm, 1% overhead for cold scenario
Kaushik Veeraraghavan 27
Warm Cold0
100
200
300
400
500
Series1
Tim
e (
seco
nds)
0
100
200
300
400
500
Series1
Tim
e (
seco
nds)