ccgrid 2014 panel: architect cloud and hpc for big data era rajkumar buyya cloud computing and...

14
CCGrid 2014 Panel: Architect Cloud and HPC for Big Data Era Rajkumar Buyya Cloud Computing and Distributed Systems (CLOUDS) Lab Dept. of Computer Science and Software Engineering The University of Melbourne, Australia www.cloudbus.org www.buyya.com www.manjrasoft.com Major Sponsors/Supporters M anjra soft DrRajkum arBuyya Chief Executive Officer M anjrasoftPty Ltd Office No. 7.22, Doug M cDonellBuilding M elbourne University -ParkvilleCam pus M elbourne,VIC 3010, Australia P: +61-3-8344 1344 | M :+61-431799078 E: raj@ m anjrasoft.com |W :http://www.manjrasoft.com M anjra soft Innovative Solutions for Cloud Computing

Upload: noreen-potter

Post on 22-Dec-2015

214 views

Category:

Documents


1 download

TRANSCRIPT

CCGrid 2014 Panel: Architect Cloud and HPC for Big Data Era

Rajkumar BuyyaCloud Computing and Distributed Systems (CLOUDS) LabDept. of Computer Science and Software EngineeringThe University of Melbourne, Australia

www.cloudbus.orgwww.buyya.comwww.manjrasoft.com

Major Sponsors/Supporters

Manjrasoft

Dr Rajkumar BuyyaChief Executive Officer

Manjrasoft Pty LtdOffice No. 7.22, Doug McDonell BuildingMelbourne University - Parkville Campus

Melbourne, VIC 3010, Australia P: +61-3-8344 1344 | M : +61-431799078

E: [email protected] | W: http://www.manjrasoft.com

ManjrasoftInnovative Solutions for Cloud Computing

2

Manjrasoft

Panel Questions

1. Is Big Data the same as data intensive computing? Do they bring in the same technical challenges from the computing point of view?

 2.  Can we “ignore” the Big Data problem? If not, what are the challenges which the Cloud and HPC community must address?

3. In your opinion, what are the solutions or possible solutions of these challenges?

4. In your opinion, what will be the programming model/environment or models/environments of next generation of Cloud and HPC?

5. Will Cloud and HPC go different way to provide services for different users and applications?

6. Will HPC and data-center become more and more the same or will become more and more apart?

3

Manjrasoft

About Me – Prof. Raj Buyya

Future Fellow of the Australian Research Council (ARC) @ University of Melbourne.

Director of CLOUDS Lab @ Melbourne Led development of some popular software

– CloudSim, Aneka, InterCloud Brokers, GridSim, Workflow Engine– used in 40+ countries.

Founder CEO of Manjrasoft Pty Ltd Editor-In-Chief of IEEE Transactions

on Cloud Computing (TCC)

4

Manjrasoft

Cloudbus@CLOUDS Lab: Melbourne Cloud Computing Initiative

Market-Oriented Clouds SLA-based Resource Management Global Cloud Exchange

Aneka – .NET-based Cloud Application Platform PaaS for Enterprise and Public Clouds

InterCloud - Scaling Across Clouds (Meta Brokering) Federation of clouds for application scaling and reliability

3rd Party Cloud Services (e.g., MetaCDN) Content Delivery Networks using different “vendors” Storage Clouds

Workflow Engine for Cloud Computing Scheduling applications with multiple interlinked tasks and dependencies

Green Clouds / Data Centers Energy Efficient and QoS Oriented Resource Allocation

CloudSim: Toolkit for Simulation of Clouds Evaluation of resource management policies & algorithms

IoT (Internet of Things) for Smart Cities Big Data Environment for Disaster Management Apps

5

Manjrasoft

1. Is Big Data the same as data intensive computing? Do they bring in the same technical challenges from the computing point of view?

Data-intensive computing Associated with “massive”

scientific data management and processing.

Solving “Big science” problems.

Conducing “Blue sky” research.

Sources: Instruments LHC/Telescopes

Big Data Associated with

supporting enterprises and governments in gaining greater insights into people's behaviors and sentiments.

“targeted” services or policies: RoI/satisfaction

“Ethical challenges” Tracking people.

Sources: CCTV

6

Manjrasoft

2.Can we “ignore” the Big Data problem? If not, what are the challenges which the Cloud and HPC community must address?

No. If we “ignore” it, we will be “ignored”! We need to be relevant and be in business

High-performance messaging Offer low-latency and high bandwidth substrate for BigData

Large-scale data management Reliability, performance, and policy driven data replication and access

management. Large-scale computations

E.g., multi-site computing QoS-based Resource Management (T, $, accuracy, …)

Heavy Vs. light-weight workload/operations Harness Big Data investment/paradigm for HPC/Clouds

Unification of HPC/BigData scientific and business paradigm

7

Manjrasoft

3. What are the solutions or possible solutions of these challenges?

Cloud/Big Data Platforms

DatacentersClusters

Desktop PCsPublic Clouds

(High Performance) Communication Infrastructure

HPC Application Platform

Application Development

Unified Programming Models

QoS (Time, Budget,

Accuracy, Secrecy)

8

Manjrasoft

4. What will be the programming models/environments of next generation

of Cloud and HPC?

Aneka Container

Core Services

Aneka Container

Core Services

Aneka Container

Core Services

Aneka Container

Core Services

Aneka Container

Core Services

Aneka Container

Core Services…….

Extensive Framework for implementation of Multiple Programming Models

9

Manjrasoft

5. Will Cloud and HPC go different

way to provide services for different users and applications?

HPC Tend to be dominated by scientific apps Massive scale computing

Cloud Consumer apps Enterprise… BigData

Good to have some “unified” programming and execution environment to benefit from “Clouds” economy of scale.

10

Manjrasoft

Aneka: Cloud Application Platform (CAP) for Resource-Intensive/Elastic Apps

Multiple Infrastructures

Multi-core Cluster Grid Cloud

Thread Task ... MapReduce

2100 2100 2100 2100

2100 2100 2100 2100

Aneka

Multiple Applications

1. SDK

2. Runtime

World-first platform supporting multiple Cloud programming models (Task, Thread, MapReduce)

SDK (Software Development Kit) containing APIs for multiple programming models and tools

Runtime Environment for managing application execution on Clouds

Suitable for Development of Enterprise

Cloud Applications Cloud enabling legacy

applications Portability for Customer

Apps: Enterprise ↔ Public Clouds .NET/Win ↔ Mono/Linux

11

Manjrasoft

Aneka: The Cloud Application Platform (CAP) for Resource-Intensive Apps(Available as Manjrasoft Product)

Private Cloud

LAN network

AmazonMicrosoft Google

IBM

Data Center

Hardware Profile Services

Container

Persiste

nce

TaskModel

ThreadModel

Map Reduce Model

OtherModels

.NET @ Windows Mono @ Linux

Secu

rity

Programming Models

Software Development Kit

ManagementStudio

Application

Foundation Services

MembershipServices

ReservationServices

LicenseServices

APIsDesign Explorer

Management Kit

AdministrationPortal

SLA-NegotiationWeb Services

ManagementWeb Services

StorageServices

AccountingServices

Fabric Services

Dynamic Resource Provisioning Services

Infrastructure

Physical Machines/Virtual Machines

Private Cloud

LAN network

Private Cloud

LAN network

AmazonMicrosoft Google

IBM

Data Center

AmazonMicrosoft Google

IBM

Data Center

Hardware Profile Services

Container

Persiste

nce

TaskModel

ThreadModel

Map Reduce Model

OtherModels

.NET @ Windows Mono @ Linux

Secu

rity

Programming Models

Software Development Kit

ManagementStudio

Application

Foundation Services

MembershipServices

ReservationServices

LicenseServices

APIsDesign Explorer

Management Kit

AdministrationPortal

SLA-NegotiationWeb Services

ManagementWeb Services

StorageServices

AccountingServices

Fabric Services

Dynamic Resource Provisioning Services

Infrastructure

Physical Machines/Virtual Machines

Patent (USA)

World-first platform supporting multiple Cloud programming models (Task, Thread, MapReduce)

SDK (Software Development Kit) containing APIs for multiple programming models and tools

Runtime Environment for managing application execution on Clouds

Suitable for Development of Enterprise

Cloud Applications Cloud enabling legacy

applications Portability for Customer

Apps: Enterprise ↔ Public Clouds .NET/Win ↔ Mono/Linux

12

Manjrasoft

Cloud-centric IoT Framework

13

Manjrasoft

6. Will HPC and data-center become more and more the same or more

and more apart?

Depends…

Mid and low-end HPC systems will look more and more like Cloud Data Centers

Extreme HPC (like those in Top 10 in Top500) Will be more and more apart from Data Centers Continues to be funded by Govt… Focuses on “few” “scientific” applications.

14

Manjrasoft

Thanks for your attention!

Are there any Questions? Comments/Suggestions

We welcome you to:Study/Research with Us | Do Business with us!

http:/www.cloudbus.org | [email protected] | [email protected]

Manjrasoft