ccgrid 2014 panel: architect cloud and hpc for big data era rajkumar buyya cloud computing and...
TRANSCRIPT
CCGrid 2014 Panel: Architect Cloud and HPC for Big Data Era
Rajkumar BuyyaCloud Computing and Distributed Systems (CLOUDS) LabDept. of Computer Science and Software EngineeringThe University of Melbourne, Australia
www.cloudbus.orgwww.buyya.comwww.manjrasoft.com
Major Sponsors/Supporters
Manjrasoft
Dr Rajkumar BuyyaChief Executive Officer
Manjrasoft Pty LtdOffice No. 7.22, Doug McDonell BuildingMelbourne University - Parkville Campus
Melbourne, VIC 3010, Australia P: +61-3-8344 1344 | M : +61-431799078
E: [email protected] | W: http://www.manjrasoft.com
ManjrasoftInnovative Solutions for Cloud Computing
2
Manjrasoft
Panel Questions
1. Is Big Data the same as data intensive computing? Do they bring in the same technical challenges from the computing point of view?
2. Can we “ignore” the Big Data problem? If not, what are the challenges which the Cloud and HPC community must address?
3. In your opinion, what are the solutions or possible solutions of these challenges?
4. In your opinion, what will be the programming model/environment or models/environments of next generation of Cloud and HPC?
5. Will Cloud and HPC go different way to provide services for different users and applications?
6. Will HPC and data-center become more and more the same or will become more and more apart?
3
Manjrasoft
About Me – Prof. Raj Buyya
Future Fellow of the Australian Research Council (ARC) @ University of Melbourne.
Director of CLOUDS Lab @ Melbourne Led development of some popular software
– CloudSim, Aneka, InterCloud Brokers, GridSim, Workflow Engine– used in 40+ countries.
Founder CEO of Manjrasoft Pty Ltd Editor-In-Chief of IEEE Transactions
on Cloud Computing (TCC)
4
Manjrasoft
Cloudbus@CLOUDS Lab: Melbourne Cloud Computing Initiative
Market-Oriented Clouds SLA-based Resource Management Global Cloud Exchange
Aneka – .NET-based Cloud Application Platform PaaS for Enterprise and Public Clouds
InterCloud - Scaling Across Clouds (Meta Brokering) Federation of clouds for application scaling and reliability
3rd Party Cloud Services (e.g., MetaCDN) Content Delivery Networks using different “vendors” Storage Clouds
Workflow Engine for Cloud Computing Scheduling applications with multiple interlinked tasks and dependencies
Green Clouds / Data Centers Energy Efficient and QoS Oriented Resource Allocation
CloudSim: Toolkit for Simulation of Clouds Evaluation of resource management policies & algorithms
IoT (Internet of Things) for Smart Cities Big Data Environment for Disaster Management Apps
5
Manjrasoft
1. Is Big Data the same as data intensive computing? Do they bring in the same technical challenges from the computing point of view?
Data-intensive computing Associated with “massive”
scientific data management and processing.
Solving “Big science” problems.
Conducing “Blue sky” research.
Sources: Instruments LHC/Telescopes
Big Data Associated with
supporting enterprises and governments in gaining greater insights into people's behaviors and sentiments.
“targeted” services or policies: RoI/satisfaction
“Ethical challenges” Tracking people.
Sources: CCTV
6
Manjrasoft
2.Can we “ignore” the Big Data problem? If not, what are the challenges which the Cloud and HPC community must address?
No. If we “ignore” it, we will be “ignored”! We need to be relevant and be in business
High-performance messaging Offer low-latency and high bandwidth substrate for BigData
Large-scale data management Reliability, performance, and policy driven data replication and access
management. Large-scale computations
E.g., multi-site computing QoS-based Resource Management (T, $, accuracy, …)
Heavy Vs. light-weight workload/operations Harness Big Data investment/paradigm for HPC/Clouds
Unification of HPC/BigData scientific and business paradigm
7
Manjrasoft
3. What are the solutions or possible solutions of these challenges?
Cloud/Big Data Platforms
DatacentersClusters
Desktop PCsPublic Clouds
(High Performance) Communication Infrastructure
HPC Application Platform
Application Development
Unified Programming Models
QoS (Time, Budget,
Accuracy, Secrecy)
8
Manjrasoft
4. What will be the programming models/environments of next generation
of Cloud and HPC?
Aneka Container
Core Services
Aneka Container
Core Services
Aneka Container
Core Services
Aneka Container
Core Services
Aneka Container
Core Services
Aneka Container
Core Services…….
Extensive Framework for implementation of Multiple Programming Models
9
Manjrasoft
5. Will Cloud and HPC go different
way to provide services for different users and applications?
HPC Tend to be dominated by scientific apps Massive scale computing
Cloud Consumer apps Enterprise… BigData
Good to have some “unified” programming and execution environment to benefit from “Clouds” economy of scale.
10
Manjrasoft
Aneka: Cloud Application Platform (CAP) for Resource-Intensive/Elastic Apps
Multiple Infrastructures
Multi-core Cluster Grid Cloud
Thread Task ... MapReduce
2100 2100 2100 2100
2100 2100 2100 2100
Aneka
Multiple Applications
1. SDK
2. Runtime
World-first platform supporting multiple Cloud programming models (Task, Thread, MapReduce)
SDK (Software Development Kit) containing APIs for multiple programming models and tools
Runtime Environment for managing application execution on Clouds
Suitable for Development of Enterprise
Cloud Applications Cloud enabling legacy
applications Portability for Customer
Apps: Enterprise ↔ Public Clouds .NET/Win ↔ Mono/Linux
11
Manjrasoft
Aneka: The Cloud Application Platform (CAP) for Resource-Intensive Apps(Available as Manjrasoft Product)
Private Cloud
LAN network
AmazonMicrosoft Google
IBM
Data Center
Hardware Profile Services
Container
Persiste
nce
TaskModel
ThreadModel
Map Reduce Model
OtherModels
.NET @ Windows Mono @ Linux
Secu
rity
Programming Models
Software Development Kit
ManagementStudio
Application
Foundation Services
MembershipServices
ReservationServices
LicenseServices
APIsDesign Explorer
Management Kit
AdministrationPortal
SLA-NegotiationWeb Services
ManagementWeb Services
StorageServices
AccountingServices
Fabric Services
Dynamic Resource Provisioning Services
Infrastructure
Physical Machines/Virtual Machines
Private Cloud
LAN network
Private Cloud
LAN network
AmazonMicrosoft Google
IBM
Data Center
AmazonMicrosoft Google
IBM
Data Center
Hardware Profile Services
Container
Persiste
nce
TaskModel
ThreadModel
Map Reduce Model
OtherModels
.NET @ Windows Mono @ Linux
Secu
rity
Programming Models
Software Development Kit
ManagementStudio
Application
Foundation Services
MembershipServices
ReservationServices
LicenseServices
APIsDesign Explorer
Management Kit
AdministrationPortal
SLA-NegotiationWeb Services
ManagementWeb Services
StorageServices
AccountingServices
Fabric Services
Dynamic Resource Provisioning Services
Infrastructure
Physical Machines/Virtual Machines
Patent (USA)
World-first platform supporting multiple Cloud programming models (Task, Thread, MapReduce)
SDK (Software Development Kit) containing APIs for multiple programming models and tools
Runtime Environment for managing application execution on Clouds
Suitable for Development of Enterprise
Cloud Applications Cloud enabling legacy
applications Portability for Customer
Apps: Enterprise ↔ Public Clouds .NET/Win ↔ Mono/Linux
13
Manjrasoft
6. Will HPC and data-center become more and more the same or more
and more apart?
Depends…
Mid and low-end HPC systems will look more and more like Cloud Data Centers
Extreme HPC (like those in Top 10 in Top500) Will be more and more apart from Data Centers Continues to be funded by Govt… Focuses on “few” “scientific” applications.
14
Manjrasoft
Thanks for your attention!
Are there any Questions? Comments/Suggestions
We welcome you to:Study/Research with Us | Do Business with us!
http:/www.cloudbus.org | [email protected] | [email protected]
Manjrasoft