nebulas: using distributed voluntary resources to build cloudsnebulas: using distributed voluntary...
TRANSCRIPT
![Page 1: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/1.jpg)
University of Minnesota
Nebulas: Using Distributed Voluntary Resources to Build Clouds
Abhishek Chandra and Jon Weissman Department of Computer Science
University of Minnesota
![Page 2: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/2.jpg)
University of Minnesota
Clouds
Cloud: Hides details of actual service deployment from users
Users
![Page 3: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/3.jpg)
University of Minnesota
Current Cloud Model
Cloud: Hides details of actual service deployment from users
Users
![Page 4: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/4.jpg)
University of Minnesota
Current Cloud Model
Largely centralized (or small degree of distribution)
Pay-as-you-go model Strong guarantees
Question: Are there services that do not need/fit this cloud model?
![Page 5: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/5.jpg)
University of Minnesota
Class 1: “Experimental” Services
Experimental deployment for: Debugging, viability, requirement estimation
![Page 6: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/6.jpg)
University of Minnesota
Class 1: “Experimental” Services
SOSP 07 USENIX 09
OSDI 08
SOSP 2009
Experimental deployment for: Debugging, viability, requirement estimation
![Page 7: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/7.jpg)
University of Minnesota
Class 2: Dispersed-Data-Intensive Services Data is geographically distributed
Costly, inefficient to move to central location
![Page 8: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/8.jpg)
University of Minnesota
Class 2: Dispersed-Data-Intensive Services
blog1 blog2
blog3
Data is geographically distributed Costly, inefficient to move to central location
![Page 9: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/9.jpg)
University of Minnesota
Class 3: Shared “Public” Services
Tour of
Paris
Personal application offered as free service User-demand driven, scale-up/scale-down needed
![Page 10: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/10.jpg)
University of Minnesota
Class 3: Shared “Public” Services
Tour of
Paris
Personal application offered as free service User-demand driven, scale-up/scale-down needed
![Page 11: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/11.jpg)
University of Minnesota
Common Service Characteristics
Elastic resource consumption Scale up/down based on demand
Geographical data/user distribution Execution dependent on location of data/user
Low/no cost Do not want to pay for resources
Weak performance/robustness requirements Some failures may be ok
![Page 12: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/12.jpg)
University of Minnesota
Users
Cloud Cloud: Hides details of actual service deployment
from users
![Page 13: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/13.jpg)
University of Minnesota
Nebula Decentralized, less-managed cloud
Dispersed storage/compute resources No/low user cost
Users
![Page 14: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/14.jpg)
University of Minnesota
Building Nebulas
Idea: Use distributed voluntary resources Resources donated by end-users ala @home, P2P systems
![Page 15: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/15.jpg)
University of Minnesota
Why Voluntary Resources?
Scalability: Large number of resources available SETI@Home: Over 2.2 million computers contributing
~510 TFlops of compute power Kazaa: Over 3.5 million users
Low cost: Minimal deployment, management costs [Kondo09]: 2 orders of magnitude difference in EC2
vs. SETI@home resources/$ Dispersion: Geographically distributed
Users can be located worldwide
![Page 16: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/16.jpg)
University of Minnesota
How is Nebula different from @home?
Cloud-oriented services impose new requirements
Requirement Nebula @home
Collective performance
High None
Locality/Context-awareness
High Low
Statefulness High/medium Low
![Page 17: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/17.jpg)
University of Minnesota 17
Challenges
Heterogeneity Different nodes have different CPU speeds, network
bandwidth, loads
Resource dispersion Data sources and compute resources may be widely
distributed
Unreliability Node/link failures, high churn
![Page 18: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/18.jpg)
University of Minnesota
Handling Heterogeneity
Heterogeneity-aware resource selection and allocation Allows better collective performance
Trivedi et al. [IJHPCA06]: Fit tasks to node capability
Heterogeneity-aware allocation reduces execution time
![Page 19: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/19.jpg)
University of Minnesota
Handling Data Dependence
Find compute nodes and data sources with high accessibility to each other
Kim et al. [UM-TR08]: Use passive accessibility estimation
Data accessibility-based selection improves download time
![Page 20: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/20.jpg)
University of Minnesota
Handling Failures
Replication, state-maintenance Sonnek et al. [TPDS07]: Reliability-aware dynamic replication
20
Dynamic replication improves performance, reliability
![Page 21: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/21.jpg)
University of Minnesota
Other Issues/Challenges
Incentivizing Nebulas Market economy, bartering, auctions How to prevent cheating/freeloading?
Deployment tools/APIs/client support Virtualization, Middleware?
Privacy/security issues How to secure systems and applications? We think: Nebulas not suitable for privacy-
sensitive services
![Page 22: Nebulas: Using Distributed Voluntary Resources to Build CloudsNebulas: Using Distributed Voluntary Resources to Build Clouds Abhishek Chandra and Jon Weissman Department of Computer](https://reader033.vdocuments.mx/reader033/viewer/2022053009/5f0c9d037e708231d436464c/html5/thumbnails/22.jpg)
University of Minnesota
Summary
Current Cloud models: Well-provisioned, well-managed, centralized
Some service classes: Need loose performance, low/no cost, distributed
data-intensive
Nebula: Distributed, less-managed clouds Use voluntary resources