31.08.2008 grid08 - grid virtualization by s.t.selvi, mit, anna university 1 organized by pbs...

31.08.2008GRID08 - Grid Virtualization by S.T.Selvi, MIT, Anna

University 1

Organized by

PBSGridWorks, AltairPresentation

by

S.THAMRAI SELVI

Professor and Head

Department of Information Technology

MIT Campus, Anna University

Chromepet, Chennai

on 31.08.2008

Virtualization and Grid @

GRID 08, Bangalore


University 2

Agenda

Introduction Virtualization Trust management in Grid Semantic discovery of Resources in Grid Conclusion


University 3

The laws influencing Information Technology

The laws that are generally accepted as governing the spread of technology:

1. Moore's Law

2. Gilder's Law

3. Metcalfe's Law

4. Disk Law


University 4

IT Architecture Trends

Resource Sharing

Fle

xib

ility

Range of solutions

Infrastructure Consolidation

VirtualComputing

Service OrientedArchitecture

Web Services


University 5

Service Oriented Architecture modelService Oriented Architecture model


University 6

XML Web Services FoundationSimple, Open, Broad Industry SupportPublish, Find, Use Services:Service Descriptions :

UDDIWSDL

Service Interactions: SOAP

Universal Data Format: XML

Ubiquitous Communications: Internet(HTTP)


University 7

Web 2.0: Evolution Towards a Read/Write Platform

Web 1.0(1993-2003)

Pretty much HTML pages viewed through a browser

Web 2.0(2003- beyond)

Web pages, plus a lot of other “content” shared over the web, with more interactivity; more like

an application than a “page”

“Read” Mode “Write” & Contribute

“Page” Primary Unit of content

“Post / record”

“static” State “dynamic”

Web browser Viewed through…

Browsers, RSS Readers, anything

“Client Server” Architecture “Web Services”

Web Coders Content Created by…

Everyone

“geeks” Domain of… “mass amatuerization”


University 8

Core concepts used in Web 2.0 AJAX- Asynchronous JavaScript + XML (User

interface/Portal) JSON - JavaScript Object Notation (SOAP) REST Representational State Transfer

(WSDL). REST proposes to use Internet as programming platform with only HTTP

Widgets, Gadgets, Start page for Portal


University 9Source: http://blogs.zdnet.com/Hinchcliffe/


University 10


University 11


University 12


University 13


University 14

Today’s IT architecture is complex and unmanageable…

access tier

web tier

application tier

database tier

edge routers

routingswitches

authentication, DNS,intrusion detect, VPN

web cache1st level firewall

2nd level firewall

load balancingswitches

web servers

web page storage(NAS)

databaseSQL servers

storage areanetwork(SAN)

applicationservers

files(NAS)

switches

switches

internetinternetinternetinternet


University 15

Putting together network, storage and computing speed

…rates of growth of digital power

1,000,000,000,000

100,000,000,000

1970

Moore/Transistors

Gilder/Bandwidth

Metcalf/NetworkNodes

10,000,000,000

1,000,000,000

100,000,000

10,000,000

1,000,000

100,000

10,000

1,000

100

101

1975 1980 1985 1990 1995 2000 2005 2010

2,300 6,000 29,000 275,000 1.2 mil 5.5 mil 42 mil 252 mil 1.344 bil

50 50 56 1,544 45,000 145,000 10 mil 2.43 bil 200.49 bil

4 111 200 10,000 300,000 1 mil 140 mil 3.5 bil 300 bil

Moore’s Law. Transistors on a single

chip doubles approximately every 18–24 months.

Gilder’s Law. Aggregate bandwidth

triples approximately every year.

Metcalfe’s Law. The value of a network

may grow exponentially with the number of participants(community Law).

Disk Law Storage doubles every 12

months

Source: Cambridge Energy Resource Associates 10616-17


University 16

The Need for New Technology

“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”


University 17

Technologies influencing a new era in computing…

Microprocessor Technology Optical Networking Technology Storage Technology Wireless Technology Sensor Technology Global Internet Infrastructure WWW and Web Services (SOA) Open Source Movements


University 18

Grid

A kind of open standard distributed

infrastructure that enables flexible,

secure, coordinated resource sharing

among dynamic collections of trusted

resources belonging to diverse

organizations across the globe

ensuring user’s Quality of Service

requirements.


University 19

Grid enables …

Virtual Organization Dynamic Resource sharing


University 20

Building Blocks Network Computing Nodes Storage

PARAM PADMA


University 21

Context

Middleware

Map to

resources

jobs / legacy code /binary executables

Resources


University 22

Overall GRID Architecture

Application

Collective

Resource

Connectivity

Fabric

Application

Transport

InternetLink

GRIDInternet

Source: The Anatomy of the GRID, Foster, Kesselman and Teucke


University 23

Protocols, Services,and APIs Occur at Each Level

Languages/Frameworks

Fabric Layer

Applications

Local Access APIs and Protocols

Collective Service APIs and SDKs

Collective ServicesCollective Service Protocols

Resource APIs and SDKs

Resource ServicesResource Service Protocols

Connectivity APIs

Connectivity Protocols


University 24

Three Generations of Grid

Source: Charlie Catlett

StandardizationStandardization is key for third-generation grids! is key for third-generation grids!StandardizationStandardization is key for third-generation grids! is key for third-generation grids!

• Local “metacomputers“– Distributed file systems– Site-wide single sign-on

• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept

• Local “metacomputers“– Distributed file systems– Site-wide single sign-on

• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept

• Utilize software services and communications protocols developed by grid projects:

– Condor, Globus, UNICORE, Legion, etc.

• Need significant customization to deliver complete solution• Interoperability is still very difficult!

• Utilize software services and communications protocols developed by grid projects:

– Condor, Globus, UNICORE, Legion, etc.

• Need significant customization to deliver complete solution• Interoperability is still very difficult!

• Common interface specifications support interoperability of discrete, independently developed services

• Competition and interoperability among applications, toolkits, and implementations of key services

• Common interface specifications support interoperability of discrete, independently developed services

• Competition and interoperability among applications, toolkits, and implementations of key services

We are here!

We are here!


University 25

Grids – Where to ?

Grid 1.0 – concerned with the virtualization, aggregation and sharing or compute resources

Grid 2.0 – focused on the virtualization, aggregation and sharing of all compute, storage, network and data resources

The promise is that in Grid 2.0 the resources will be easier to define, test, install, transport and adjust on demand.


University 26

Grid 2.0 EmergingGrid 2.0 Emerging

Grid 1.0Grid 1.0Compute Intensive Compute Intensive Cycle AggregationCycle Aggregation

SOASOASoftware Services Software Services with SLA & QoS with SLA & QoS

MetricsMetrics

Virtualization Virtualization Consolidation of Consolidation of

ResourcesResources

Grid 2.0*Grid 2.0* Virtualized Compute, Virtualized Compute,

Storage, Network, DataStorage, Network, Data

Service OrientedService Oriented

Policy Driven AutomationPolicy Driven Automation

Distributed across Distributed across firewallsfirewalls

Parallel, stateless, Parallel, stateless, stateful and transactional stateful and transactional appsapps

*The 451 Group: 'grid 2.0' is focused on the virtualization, aggregation and sharing of all *The 451 Group: 'grid 2.0' is focused on the virtualization, aggregation and sharing of all compute, storage, network and data resources. It is both Service-oriented and automated.compute, storage, network and data resources. It is both Service-oriented and automated.


University 27

Grid 3.0?

(Grid) Middleware

User Applications

Resources

Semantic Grid We

b 2

.0

Not just stitching Web and Grid together (Semantic

Web and Web 2.0 ARE the Web) The Grid is part of the

Web. The Web is a platform for the Grid.

A rich and relevant experience for users and developers.

Smashing and mashing.


University 28

Platform – Infrastructure Rapid Application development Agile, Loosely coupled Designed for Reuse

People - Social Keep it Simple Users count

Grid 3.0? The Participatory Grid.


University 29

NextGeneration

Grids

• En

d-us

er e

mpo

wer

men

t

• Li

fe-s

uppo

rt to

busi

ness

proc

esse

s

• Societal behaviour (millions of self-organising nodes)• Computational semantics, ontologies, meta-descriptions• Pervasive virtual organisations

• Continuously changing requirem

ents

• Grid services developm

ent

environments

Virtualization

End-

User

Vision

Software Vision

Architectural Vision

Sim

plifi

cation

Abstraction

Next Generation Grid(s) – 3-fold Vision

Expert Group Reports:


University 30

Virtualization


University 31

Virtualization

Virtualization is a framework or methodology of dividing the resources of a computer into multiple execution environments, by applying one or more concepts or technologies such as hardware and software partitioning, time-sharing, partial or complete machine simulation, emulation, quality of service (Source: Wiki)


University 32

Benefits from Virtualization Load balancing

Create VM for new work loads

Increased Resource utilization Create VM in the under utilized resources

Reusable One VM created can be reused somewhere

else

Isolation Destruction of one VM doesn’t affect the other

VM


University 33

Continued… High Availability

Reduction in downtime

Improved ROI More production with less Investment,

operational and maintenance

Reduced TCO Eliminate the need of new Infrasturctures such

as Server


University 34

Continued… Live migration

Support suspend resume and migration of created VMs

Persistent and Non Persistent Support for keep/drop the changes made in

VM images

Undoable and Append Allows undo and append options in VM

imagaes


University 35

“If you want people to use your system then make it as easy as possible to use.”Werner Vogels, Amazon CTO

Simple to use is not the same as simpleYes, Grids are doing complicated hard stuff. But why make the developer and user suffer so? Also, Trust is another major issue

Motivation for our Research


University 36

CARE Resource Broker (CRB ) A Grid Meta Scheduler


University 37

User PoolRequest Handler

Job Pool

RESOURCE MANAGER

Information System

Physical Machine Manager

Virtual Machine Manager

Transporter ExecutorAggregator

CONTROLLER

Job Tracker

Host IdentifierHost Pool

Scheduler

DB

Grid Resources

Virtual Cluster Service

CLI or Job Submission Portal

CARE Resource Broker

GridFTP FTP Pre-WS GRAM WS GRAMMDS4GT Components

Drivers

CR

B w

ith

Vir

tuali

zati

on

su

pp

ort


University 38

CRBRequest Handler: Gets the Job request, create Job

instance and store it in the Job Pool.

Controller: Initiates the Aggregator to collect the

information about all available resources and updates the Host object and Host Pool object.


University 39

Continued …

Scheduler:

Gets the unordered resource list and orders the resources based on their trust value and return back to Controller.

Virtual Cluster Service:

The controller gives the resource list to the VCS in order to create virtual cluster.


University 40

Continued…

Transporter

Initializes the RFT and transfer the input and executables into selected host

Executor:

Initializes the GRAM to submit the job into local scheduler.

Aggregator

Initializes MDS to get the resource information in the Grid


University 41

Trust Management System


University 42

We define Trust…

The degree of belief in the resource

provider’s competence to complete user’s

task dependably, securely and reliably in a

specific context at a given time

usersAgent / Resource Broker

Resources


University 43

Trust Integration

Trust MetricIdentification

Trust ValueCalculation

Trust MetricEvaluation

Trust ValueUpdation

TMSTMS

• Identify suitable parameters with which the respective trust can be defined

• Apply suitable methodology to determine the value of those metrics

• Determine the overall trust value using the values for various trust metrics obtained

• Integrate the trust mechanism in the resource broker / Scheduler to find out the most trustworthy resource provider for successful job execution/task completion

• Modify or update the value of trust periodically for each resource provider

Trust Management Life Cycle


University 44

User’sTrust

Broker’sTrust

InformationTrust

EquipmentProvision

Trust

ServiceProvision

Trust

VariousTrusts

Describes relying party’s trust in a service provider. The trustor trusts the trustee to provide a service that does not involve access to the trustor’s resource

•It describes trust in principals for the purpose of accessing resources owned by the relying party. A trustor trusts a trustee to use resources that he owns or controls. •It measures whether a resource provided by the resource provider is trustworthy.• The QoS is the main concern in this case

• It is the belief that information provided by the Information provider is reliable and accurate.

• It is a measure of belief that a resource broker has discovered a trustworthy resource

Types of trust

• Measures whether a resource provider is willing to offer his services to the user.• The previous behaviour / payment record may be considered for this trust


University 45

EquipmentProvision

Trust

Resource Registration Metrics

Resource Performance Metrics

User Feedback Metrics

Dependency Metrics

These metrics reflect reputation of the resource in the user community

These metrics reflect the reputation of the organization . It is used to identify initial trust value of the resource provider

These metrics reflect the throughput of the resources and their QoS

Availability, Bandwidth, Latency, Estimated Execution

Time, Actual Execution time, Success, Failure

Recommendation, Reputation

Government / Private, Registration Number


University 46

TrustUpdation

TrustCalculation

TrustMetrics Evaluation

User Feedback

Resource Registration

ResourcePerformance

Underlying Middleware, Resources

User

Resource

Provider

Block Diagram of Trust Management System

Trust Metrics IdentificationDetermines values of metrics using different methodologies

Determines Overall trust value by using suitable mathematical model

Periodically updates the trust value and stores in the database

Database


University 47

Conventional Gridway Flow Trust Enabled Gridway Flow

Job Submit

<job template>

Gathers AvailableResource

MatchesAgainst JobReq

R1R2 R3

Invokes Scheduling Operation

Performs Matchmaking

Selects and submits

Job Submit

<job template>

Gathers AvailableResource

MatchesAgainst JobReq

R1R2 R3

Invokes Scheduling Operation

Performs Matchmaking

Selects and submits

TMSInvokes TMS

TrustDB

Selects Most TrustedResource


University 48

PATENT Pending Filed Application for patent (Patent

application No.593/CHE/2007 A) Title of the Invention: Trust

Resource Broker International Classification:G06F

17/30 Patent has been published in

Journal No. 16/2007 dated 20/04/2007.


University 49

Integrated with Gridway Meta Scheduler

http://www.gridway.org/doku.php?id=related:related

Developed by other projects Trust Management

Description - This project provides GridWay with a mechanism to evalute the trust of the computational resources. The Trust module consists of modified host attribute parsers and match-making algorithms. Additionally the Trust module needs to modify the information providers of Grid resources.

Contact - This component has been developed by the team of Prof Dr.S.Thamarai Selvi ([email protected]) at Madras Institute of Technology


University 50

Semantic Component


University 51

Semantic Description and Discovery of

Grid Resources and its integration using

Gridbus Broker Semantic Component enables semantic

description and discovery of grid resources.

Protégé-OWL APIs are used to create

knowledge base of grid resources using

a resource ontology template.

MDS is used to aggregate resource

information

We use Algernon Inference Engine to

discover suitable resource

This module has been successfully integrated

With Garuda, a National Grid Computing

Initiative


University 52

Deployment in Garuda Grid

Semantic component has been deployed in Garuda Grid


University 53

Sample Screenshots


University 54


University 55


University 56


University 57


University 58


University 59


University 60


University 61


University 62


University 63

Contact

E-Mail: [email protected]

Web site: www.annauniv.edu/care


University 64

31.08.2008 grid08 - grid virtualization by s.t.selvi, mit, anna university 1 organized by pbs...

Documents