grid architecture developments aslam parvez memon & shakil akhtar shaheed zulifkar ali bhutto...

Grid Architecture Developments

Aslam Parvez Memon & Shakil Akhtar

Shaheed Zulifkar Ali Bhutto Institute of Science and Technology (SZABIST), Karachi Pakistan.

http://www.szabist.edu.pk

December 21, 2006

Outlines

• Defining Grid Computing

• What is Grid and Grid Computing

• Grid Technology

• Why Grid

• Study of Grid Computing

• Grid Technology Problem Space

• Virtual Organization

• Virtual Organization Problem Space

• Grid Architecture

• Some Solutions

• Globus Toolkit

• Working With Gird

• Key Concepts of GT4

• Major Grid Projects

• Research Bodies

• Global Community

• Research Areas

• The Research Processes

Ian Foster’s 3 point checklist

• A Grid is a system that is able to• coordinate “resources that are not subject to

centralized control”• Use “standard, open, general-purpose

protocols and interfaces”• “to deliver nontrivial qualities of service.”

Defining Grid Computing

• There are several competing definitions for “The Grid” and Grid computing

• These definitions tend to focus on:• Implementation of Distributed computing• A common set of interfaces, tools and APIs• Some stress the inter-institutional aspect of

grids and Virtual Organizations• “The Virtualization of Resources” abstraction

of resources

What is Grid and Grid Computing?

• Grid computing must provide basic functions1. resource discovery and information collection &

publishing2. data management on and between resources3. process management on and between resources4. common security mechanism underlying the

above5. process and session recording/accounting

Grid Technology

• Emerging enabling technology.

• Natural evolution of distributed systems and the Internet.

• Middleware supporting network of systems to facilitate sharing, standardization and openness.

• Infrastructure and application model dealing with sharing of compute cycles, data, storage and other resources.

• Publicized by prominent industries as on-demand computing, utility computing, etc.

• Move towards delivering “computing” to masses similar to other utilities (electricity and voice communication).

• Currently used for high performance computing however the trend is towards Service Oriented Applications (SOA).

Why Grid?

• What can the grid do that existing technology cannot do?

• Grid infrastructure and application architecture form a global computing framework facilitating sharing of resources and schedulability of jobs by matching their needs with available pool of compute and storage resources.

• Compute cycles can be tapped on demand from sources other then yours.

• Wasted cycles from idle sources can be utilized for use in needed application.

• Grid is molding computing into an utility similar to utilities we are used to: electricity and telephone.

Study of Grid Computing

• Components: Core, system defined and user defined

• Infrastructure

• Application model

• Standards

• Application Programming Interfaces

• Technology Support (enabling technologies)

• Job submission and associated functions

• Service creation and deployment and related functions

Grid Technology Problem Space

• Grid technologies and infrastructures support the sharing and coordinated use of diverse resources in dynamic, distributed “virtual organizations”.

• Grid technologies are distinct from technology trends such as Internet, enterprise, distributed and peer-to-peer computing. But these technologies can benefit from growing into the “problem space” addressed by grid technologies.

The Grid Problem

• Flexible, secure, coordinated sharing of computation among dynamic collections of individuals, institutions, and resources

• Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of…• central location• central control• omniscience• existing trust relationships

Virtual Organization

Grid’s virtual organizations (VOs) concept provides seamless access to federated heterogeneous resources—computers, mobile devices, network bandwidth, storage, databases, scientific instruments, servers etc. by creating illusion of supercomputing infrastructure. A grid user can have on demand access to such resources, distributed across various organizations in different geographical locations, yet in a controlled and secure resource sharing environment.

Elements of the Problem

• Resource sharing• Computers, storage, sensors, networks, …• Sharing always conditional: issues of trust,

policy, negotiation, payment, …

• Coordinated problem solving• Beyond client-server: distributed data

analysis, computation, collaboration, …

• Dynamic, multi-institutional virtual orgs• Community overlays on classic org

structures• Large or small, static or dynamic

The Programming Problem

• Applications require resources (compute power, storage, data, instruments, displays) at many sites for many users.

• Some requirements:• Abstractions and models to increase

speed/robustness/etc. of development• Tools to ease application development

and diagnose common problems, ease deployment

• Code/tool sharing to allow reuse of code components developed by others

Grid must suspport computational workflows

• Locate “suitable” computers

• Authenticate with appropriate sites

• Allocate resources on those computers

• Initiate computation on those computers

• Configure those computations

• Select “appropriate” communication methods

• Compute with “suitable” algorithms

• Access data files, return output

• Respond “appropriately” to resource changes

Grid Requirements

• identity & authentication

• authorization & policy

• resource/service discovery

• resource allocation

• (co-)reservation, workflow

• remote data access

• rapid data transfer

• monitoring

• intrusion detection

• resource management

• accounting

• fault management

• system evolution

• and more…

Grid Computing - Functions

• Grid computing must provide typically these basic functions (Foster/Kesselman)• resource discovery and information

collection & publishing• data management on and between

resources• process management on and between

resources• common security mechanism underlying the

above

• In addition, it should include:• process and session recording/accounting

Grid Architecture

• Architecture identifies the fundamental system components, specifies purpose and function of these components, and indicates how these components interact with each other.

• Grid architecture is a protocol architecture, with protocols defining the basic mechanisms by which VO users and resources negotiate , establish, manage and exploit sharing relationships.

• Grid architecture is also a services standards-based open architecture that facilitates extensibility, interoperability, portability and code sharing.

• API and Toolkits are also being developed.

Layered Grid Architecture

• Fabric Layer - provides the local services of a resource:

• computational, storage, network

• Connective Layer - core communication and authentication protocols

• Enables exchange of data between fabric layer resources

• Security and authentication important here

Layered Grid Architecture (cont.)

• Resource Layer – enables resource sharing• Builds on connectivity layer to control

and access resources (Ex: data servers)

• Collective Layer - coordinates interactions across multiple resources

• Ties multiple resources and services together

• (Ex: metacatalogues)

• Application Layer - user applications use collective, resource, and connective layers to perform grid operations in a virtual organization

Some Solutions

• Middleware Toolkits: not all speak (or spoke) Globus:• Condor• Globus Toolkit• Legion/Avaki• Condor (now

Sun Grid Engine)

• Unicore

• Higher Level Toolkits (build on Globus)• JavaCoG• GridPortal

Toolkit, Grid Portal Development Toolkit (GPDK)

• Condor-G• SGE

The Globus Toolkit

• Open-source reference software base for developing Grid infrastructure and applications

• Implements GGF standards

• Service-oriented• Services can be decoupled from any

fixed resource• A service consumes resources, but

how is not most important • A better base abstraction for managing

dependability, end-to-end quality of service

Slide Courtesy of Ian Foster presentation at Comdex04

Globus Protocols - Connectivity Layer

• Grid Security Infrastructure (GSI):

• Authentication/authorization, message protection across institutions

• Single sign-on, delegation, identity mapping

• Public key technology

• Certificate authorities, certificate & key management

Ian Foster, et. al., “Anatomy of the Grid”

Globus Protocols - Collective Layer

• Metadirectory services

• Resource brokers• Condor

• Co-reservation/co-allocation services

• Workflow management services

Ian Foster, et. al., “Anatomy of the Grid”

Globus Protocols – Resource Layer

• Grid Resource Allocation Management (GRAM)

• Remote allocation, control of compute resources• Furnishes information on state of the resources

to the Metacomputing Directory Service (MDS)

• GridFTP• High-performance data access and transport

• Grid Resource Information Service (GRIS)• Access to structure and state info (MDS)

• All built on connectivity layer

Ian Foster, et. al., “Anatomy of the Grid”

Grid Security Infrastructure (GSI)

• Public key cryptography• Encryption relies on two keys, related

mathematically so that if either key encrypts a message, the other must be used to decrypt it

• One key is public, the other is kept private• A user proves own identity by encrypting a

message; if the public key can decrypt, the user is indeed holding the private key

• No password is ever exchanged

Ian Foster, et. al., “Anatomy of the Grid”

Working With Grids

1. A user enrolls himself or his machine with grid system

2. The user establishes his identity with CA (this process may require alternate ways other than internet)

3. The CA takes steps to make sure that the user is in fact who, he claims to be.

4. The CA makes special certificate available to the software, which needs to check the identity of user and his requests to the grid system.

5. Steps 1-4 may be repeated for the donor machine(s). A user must keep his security credentials secure.

6. User installs the software provided by the grid system to use the grid and/ donate the machine. The software may be auto configured or manual by the user, this configuration is required for:• Grid nodes management• Machine identification information• Implement constraints on resources’ access such as time, type etc.• Providing user’s IDs on other machines that exist on grid.

1. A user is required to login to a grid system using user ID that is enrolled in the grid.

2. The user can use gird system IDs or operating system IDs, which ever is enrolled with grid, but grid system ID is recommended for two reasons:• It eliminates the need for matching IDs form machine to

machine• A user can access entire grid as a one large virtual computer

using common ID across the grid.

3. Globus, as mentioned above uses proxy login model, which keeps a user logged in for a specified amount of time, even if a user logs off and logs back on the operating system, and even if the machine is rebooted.

4. Once the user is logged in he can query the grid or submit job using localization interfaces.

Key Concepts for GT4

• OGSA, WSRF, and GT4• These are basic architecture components for GT4• Open Grid Services Architecture (OGSA)

• Web Services: • OGSA, WSRF, and GT4 are based on standard Web Services

technologies such as SOAP and WSDL. • Ned to be familiar with the Web Services architecture and

languages.

• The Web Services Resource Framework: • WSRF is the core of GT4.

• Based on WS-Resources and Web Services, and grid computing

• Java & XML: • to use GT4, you need to be able to program in Java, and to

understand basic XML.

OGSA Key Requirements

• Interoperability and Support for Dynamic and Heterogeneous Environments

• Resource Sharing Across Organizations

• Job Execution

• Data Services

• Security

• Optimization

• Quality of Service (QoS) Assurance

• Administrative Cost Reduction

• Scalability

• Availability

• Ease of Use and Extensibility

OGSA Defines Basic Capabilities

• Infrastructure Services

• Execution Management Services

• Data Services

• Resource Management Services

• Security Services

• Information Services

• Security Considerations

GT Architecture

• GT4 comprises both a set of service implementations (“server” code) and associated “client” libraries.

• GT4 provides both Web services (WS) components and non-WS components

• All GT4 WS components use WS-Interoperability-compliant transport and security mechanisms• can interoperate with each other and with other

WS components.

• All GT4 components support X.509 certificates

• both WS and non-WS• client can use the same credentials to authenticate

with any GT4 WS or non-WS component.

GT4 Services

• Nine GT4 services implement Web services (WS) interfaces:• Job management (GRAM)• Reliable File Transfer• Delegation• Monitoring and Discovery System (MDS)

• MDS-Index, MDS-Trigger, and MDSArchive• Community Authorization (CAS) • OGSA-DAI data access and integration• Grid TeleControl Protocol (GTCP) Grid

• remote instrumentation control

OGSA, WSRF, GT4 Relationship Diagram

WS Software stack used by GT4 WSRF

• HTTP Server• Apache HTTP Server

• Application Server• Apache Tomcat

• SOAP Engine• Apache AXIS• Supports wsdl2java tool -

build Java proxies and skeletons from WSDL docs.

• Web Service• User App

GT4 Roadmap

Major Grid Projects

• Earth System Grid, www.earthsystemgrid.org

• Virtual Observatory, http://skyview.gsfc.nasa.gov/

• European Data Grid, http://cern.ch/eu-datagrid

• GriPhyN Project, http://www.griphyn.org/

• PPDG, http://www.ppdg.net/

• HEPGRID, http://www.buyya.com/hepgrid/

• Virtual Laboratory Grid, http://www.jhu.edu/virtlab/virtlab.html

• NEESGRID, http://www.neesgrid.org/

• GEOSIDE, http://www.geodise.org/

• Fusion Grid, http://www.fusiongrid.org/

• IPG Grid, http://www.nas.nasa.gov/About/IPG/ipg.html

• ActiveSeets http://www.csse.monash.edu.au/~davida/nimrod/activesheets.htm

• China National Grid (CNGrid), http://www.cngrid.org

• China Science Grid (CSGrid) http://www.ssc.net.cn/en/showinfo.asp? categoryid=84

• China Semantic Grid, http://kg.ict.ac.cn/

• Shanghai City Information Grid http://www.gridtoday.com/05/0131/104536.html

Research Bodies and Consortiums

• Global Grid Forum (GGF), http://www.ggf.org

• OASIS, http://www.oasis.org

• DMTF, http://www.dmtf.org

• CIM, http://www.dmtf.org/standards/cim

• WBEM, http://www.dmtf.org/standards/wbem

• W3C, http://www.w3.org

• Globus Alliance, http://www.globus.org

• GridBus Project, http://www.gridbus.org/

• Condor Project Home Page, http://www.cs.wisc.edu/condor/

• Legion Project, http://legion.virginia.edu/

• Unicore, http://www.unicore.org

GlobalCommunity

Slide Courtesy of Ian Foster

Applications

Proof concepts

• Academic prototype applications

Long terms applications

• Grid based Agro-MIS

• Grid based LRMIS

• Grid Based LLDP

And more……..

Research Areas

• Protocols and Standards Development

• Grid Security

• Service Oriented Architecture

• Resource Management

• Scheduling

• Grid Operating Environments

• Grid Software Development Environments

• Quality of Services

• Grid Localization

• Grid Simulations

• Toolkits and Portals developmentb

The Process

• Literature Review

• Research Question/ Hypothesis

• Analysis (Generic Specifications)

• Design (Technical Specification)

• Implementation

• Testing

• Documentation

• Presentation

Thank You!