expanding the horizons of cloud computing beyond the data ...€¦ · expanding the horizons of...
TRANSCRIPT
![Page 1: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/1.jpg)
Expanding the Horizons of Cloud Computing Beyond the Data Center
Jon Weissman and Abhishek Chandra
(+ great UMn students)
Department of CS&E
University of Minnesota
![Page 2: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/2.jpg)
Key Trends
• Client technology
– devices: smart phones, ipods, tablets, sensors
• Big data
– located at the network edge
• Privacy/trust
– local clouds
• Multiple DCs/clouds
– global services
![Page 3: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/3.jpg)
Minnesota Cloud Research
• Eye towards cloud evolution
• Projects
– Nebula
– DMapReduce
– Mobile cloud
– Proxy cloud
– Active cloud storage
– Virtualization
![Page 4: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/4.jpg)
Project Clusters
• Power at the edge
– big data, locality
• Cloud-2-Cloud
– multiple clouds/DCs
• Mobile user
– user-centric cloud
Nebula
Mobile cloud
DMapReduce
![Page 5: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/5.jpg)
Big Data Trend
• Big data is distributed
– earth science: weather data, seismic data
– life science: GenBank, NCI BLAST, PubMed
– health science: GoogleEarth + CDC pandemic data
– web 2.0: user multimedia blogs
– “everyone is a sensor”
• Cost in moving data to the cloud
![Page 6: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/6.jpg)
Big Data Trend: Nebulas
• Process data “close by”
– fully and/or on-route to the central cloud
– cost: save time and money
– privacy (think: patient records)
• Close by
– network distance
– trusted peers
![Page 7: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/7.jpg)
Example: Dispersed-Data-Intensive Services
Data is geographically distributed Costly, inefficient to move to central location
![Page 8: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/8.jpg)
Example Instance: Blog Analysis
blog1 blog2
blog3
![Page 9: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/9.jpg)
Nebula: A New Cloud Model
• Make the cloud more “distributed”
– exploit the rich collection of edge computers
– volunteers (P2P, @home), commercial (CDNs)
– enormous computing potential, network diversity
– lower latency: “on demand”, native client sandboxing
Nebula Central
![Page 10: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/10.jpg)
Example: Blog Analysis
blog1 blog2
blog3
![Page 11: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/11.jpg)
Blog Results
0
20000
40000
60000
80000
100000
120000
140000
40 80 120 160 240 320
Tim
e t
ake
n (
sec)
Amazon emulator Nebula testbed
# Blogs
![Page 12: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/12.jpg)
Current Status
• Prototype running on PlanetLab
• Distributed Data-Store Service
• Network Dashboard Service
• Node software stack
– native client
![Page 13: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/13.jpg)
Nebula Going Forward
• Organize Nebulas
– around trusted peers
– social groups
– communities of interest
– local resources
• Nebula + commercial cloud
“use the edge opportunistically”
![Page 14: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/14.jpg)
Big Data Trend: DMapReduce
14
Input
Data
Data Push
Output
Data
Map Reduce
![Page 15: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/15.jpg)
Big Data Trend: Distribution • Big data is distributed
– earth science: weather data, seismic data
– life science: GenBank, NCI BLAST, PubMed
– health science: GoogleEarth + CDC pandemic data
– web 2.0: user multimedia blogs
![Page 16: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/16.jpg)
Wide-Area MapReduce
DFS push
• Data in different data-centers
• Run MapReduce across them
• Data-flow spanning wide-area networks
![Page 17: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/17.jpg)
17
Option 1: Local MapReduce
Data
Source
(US)
Data
Source
(EU)
Data Center (US) Data Center (EU)
MapReduce
Job
Final Result
Data Push
(Fast)
Data Push
(Slow)
![Page 18: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/18.jpg)
18
Option 2: Global MapReduce
Data
Source
(US)
Data
Source
(EU)
Data Center (US) Data Center (EU)
MapReduce Job Final
Result
Data Push
(Fast)
Data Push
(Slow)
Data Push
(Fast)
Data Push
(Slow)
![Page 19: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/19.jpg)
19
Option 3: Distributed MapReduce
Data
Source
(US)
Data
Source
(EU)
Data Center (US) Data Center (EU)
MapReduce
Job
Final Result
Data Push
(Fast)
MapReduce
Job
Combine Results
Data Push
(Fast)
![Page 20: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/20.jpg)
PlanetLab: WordCount (text data)
• Distributed MR works best in presence of data aggregation
20
0
200
400
600
800
1000
1200
1400
1600
1800
Push US Push EU Map Reduce ResultCombine
Total
Tim
e in s
eco
nds
Local MR
Global MR
Distributed MR
![Page 21: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/21.jpg)
PlanetLab: WordCount (random data)
• Local MR works best in presence of data ballooning
21
0
100
200
300
400
500
600
700
800
900
Push US Push EU Map Reduce ResultCombine
Total
Tim
e in s
eco
nds
Local MR
Global MR
Distributed MR
![Page 22: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/22.jpg)
EC-2 Results
22
0100200300400500600700800900
1000
Push US Push EU Map Reduce ResultCombine
Total
Tim
e in s
eco
nds Local MR
Distributed MR
0
100
200
300
400
500
600
700
800
900
Push US Push EU Map Reduce ResultCombine
Total
Tim
e in S
eco
nds Local MR
Distributed MR
0
50
100
150
200
250
300
Push US Push EU Map Reduce ResultCombine
Total
Tim
e in S
eco
nds
Local MR
Distributed MR
WordCount (Text) WordCount (Random)
Sort
![Page 23: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/23.jpg)
Intelligent Data Placement
• HDFS push
– local node, same rack, random rack
Data placement
Scheduling
Resource Topology Application
Characteristics
/DCi/rackA/nodeX Data expansion factors input->intermediate, a
Intermediate->output, b
select LMR, DMR, GMR
Generalize beyond HDFS/MapReduce!
![Page 24: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/24.jpg)
Mobility Trend: Mobile Cloud
• Mobile users/applications: phones, tablets
– resource limited: power, CPU, memory
– applications are becoming sophisticated
• Improve mobile user experience
– performance, reliability, fidelity
– tap into the cloud dynamically based on current resource state, preferences, interests, privacy
![Page 25: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/25.jpg)
Cloud Mobile Opportunity
• Dynamic outsourcing
– move computation, data to the cloud dynamically
• User context
– exploit user behavior to pre-fetch, pre-compute, cache
• Multi-user sharing
– discover implicit cloud sharing based on interests, social ties
![Page 26: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/26.jpg)
Outsourcing
• Partitioning across (Mobile <->Server <-> Cloud)
– local data capture + cloud processing
– images/video, speech, digital design, aug. reality
Server Server Server Server Server
Proxy Code repository
….
….
Application Profiler
Outsourcing Client
Outsourcing Controller
cloud end (EC-2) mobile end (Android)
![Page 27: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/27.jpg)
Experimental Results -Image Sharpening
• Response time
– both WIFI & 3G
– up to 27× speedup
– 219K, WIFI
• Power consumption
– save up to 9× times
– 219K, WIFI
27
Avg. Time
Avg. Power
![Page 28: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/28.jpg)
Cloud Speculation
• Dynamic user profile
– contains activities in time and space
– “read nytimes.com bet. 9-9:30am on the train; likes technology articles”
– associate with context
• Patterns are relationships between activities
– repetitive, sequential, concurrent, time-bounded
– “user always does X and then does Y”
• Exploiting patterns: pre-fetching, pre-computing, caching in the cloud
![Page 29: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/29.jpg)
Smart cloud
Back-end speculation and optimization
![Page 30: Expanding the Horizons of Cloud Computing Beyond the Data ...€¦ · Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman and Abhishek Chandra (+ great UMn](https://reader035.vdocuments.mx/reader035/viewer/2022081405/5f0c9d047e708231d4364653/html5/thumbnails/30.jpg)
Summary
• Cloud Evolution
– mobile users, big data, privacy/trust, global
• Our vision of the Cloud
– locality of users, data, other clouds/data centers, user-centric behavior