GRID COMPUTING

Net-535 Fall 2013

Grid Computing Definitions

The term Grid computing originated in the early 1990s as a metaphor for making computer power as easy to access as an electric power grid. The definitive definition of a Grid is provided by Ian

Foster in his article "What is the Grid? Computing resources are not administered centrally. Open standards are used. Non-trivial quality of service is achieved.

IBM : "A Grid is a type of parallel and distributed system that enables the sharing, selection, and aggregation of resources distributed across multiple administrative domains based on the resources availability, capacity, performance, cost and users' quality-of-service requirements"

What is Grid Computing

“Grid computing, most simply stated, is distributed computing taken to the next evolutionary level. The goal is to create the illusion of a simple yet large and powerful self managing virtual computer out of a large collection of connected heterogeneous systems sharing various combinations of resources” IBM Redbook .

4

Electrical Power Grid Analogy

Electrical power grid users (or electrical appliances)

get access to electricity through wall sockets with no care or consideration for where or how the electricity is actually generated.

“The power grid” links together power plants of many different kinds

The Grid Computing users (or client applications) gain

access to computing resources (processors, storage, data, applications, and so on) as needed with little or no knowledge of where those resources are located or what the underlying technologies, hardware, operating system, and so on are

"the Grid" links together computing resources (PCs, workstations, servers, storage elements) and provides the mechanism needed to access them.

5

Properties of the Grid

• Distributed• Dynamic• Heterogeneous• Virtual environment• Collaborative environment• Transparent access to all the

available resources

6

Who needs Grid Computing?

Not just computer scientists… scientists “hit the wall” when faced with

situations: The amount of data they need is huge and the data is

stored in different institutions. The amount of similar calculations the scientist has to do

is huge. Other areas:

Government Business Education Industrial design etc

What grid computing can do(1) Exploiting underutilized resources:

The easiest use of grid computing is to run an existing application on a different machine

The machine on which the application is normally run might be unusually busy due to an unusual

peak in activity. The job in question could be run on an idle machine elsewhere on the

grid. There are at least two prerequisites for this scenario

First, the application must be executable remotely and without undue overhead Second, the remote machine must meet any special hardware, software, or

resource requirements imposed by the application The processing resources are not the only ones that may be

underutilized also grid data Often, machines may have enormous unused disk drive capacity. Grid

computing, more specifically, a “data grid”, can be used to aggregate this unused storage into a much larger virtual data store

Another function of the grid is to better balance resource utilization. An organization may have occasional unexpected peaks of activity that demand more resources. If the applications are grid enabled, they can be moved to underutilized machines during such peaks

What grid computing can do Parallel CPU capacity is one of the most

attractive features of a grid

Subjobs on different machines Barriers often exist to perfect scalability.

What grid computing can do Applications

Grid-enabled applications no practical tools for transforming arbitrary

applications to exploit the parallel capabilities of a grid.

What grid computing can do

Virtual resources and virtual organizations for collaboration Another important grid computing

contribution is to enable and simplify collaboration among a wider audience.

Grid computing takes these capabilities to an even wider audience, while offering important standards that enable very heterogeneous systems to work together to form the image of a large virtual computing system offering a variety of virtual resources, as illustrated

What grid computing can do Access to additional resources

special equipment, software, licenses, and other services

Some machines on the grid may have special devices Resource balancing

An unexpected peak can be routed to relatively idle machines in the grid.

If the grid is already fully utilized, the lowest priority work being performed on the grid can be temporarily suspended or even cancelled and performed again later to make room for the higher priority work.

What grid computing can do

Reliability Now: redundancy in

hardware Future: Software Utilize “autonomic

computing”• Management– More disperse IT

infrastructure– Priority among projects

Grid concepts and componentsTypes of resources

Computation Storage

Primary/secondary storage Mountable networked filed system

AFS, NFS, DFS, GPFS Capacity increase (multiple machine ) Uniform name space Data Stripping

Grid concepts and components(2)Types of resources (cont)

Communications Redundant communication paths

Software and licenses License management software

Special equipment, capacities, architectures, and policies different architectures, operating systems, devices,

capacities, and equipment. Jobs and applications

Application is a collection of jobs Specific dependencies

Grid concepts and components(3)Types of resources (cont)

Scheduling, reservation, and scavenging scheduler

automatically finds the most appropriate machine on which to run any given job

scavenging report its idle status to the grid management node.

This management node would assign to this idle machine the next job that is satisfied by the machine’s resources.

Reserved Reserve of resources in advance to improve the quality

of service

Grid concepts and components(4)

Intragrid to Intergrid cluster

same hardware/software

Intragrid heterogeneous

machines/software multiple

department/same organization

Intergrid heterogeneous

machines/software multiple

department/multiple organization

Grid components

Management components First, there is a component that keeps track

of the resources available to the grid and which users are members of the grid.

Second, there are measurement components that determine both the capacities of the nodes on the grid and their current utilization rate at any given time.

Third, advanced grid management software can automatically manage many aspects of the grid

Grid components

Donor software Each machine contributing resources typically needs

to enroll as a member of the grid and install some software that manages the grid’s use of its resources. Usually, some sort of identification and authentication procedure must be performed before a machine can join the grid

The donor machine will usually have some sort of monitor that determines or measures how busy the machine is and the rate or amount of resources utilized. This information is “bubbled up” to the management software of the grid and used to schedule use of those resources accordingly.

Grid components

Submission software Software to submit the job Schedulers

Most grid systems include some sort of job scheduling software. This software locates a machine on which to run a grid job that has been submitted by a user. In the simplest

cases, it may just blindly assign jobs in a round-robin fashion to the next machine matching the resource requirements. However, there are advantages to using a more advanced

scheduler. Some schedulers implement a job priority system. This is sometimes done by using

several job queues, each with a different priority. As grid machines become available to execute

jobs, the jobs are taken from the highest priority queues first. Policies of various kinds are also implemented using schedulers. Policies can include various kinds of constrains on jobs, users, and resources. For example, there may be a policy that restricts grid jobs from executing at certain times of the day.

Grid components

CommunicationsA grid system may include software to help jobs

communicate with each other. For example, an application may split itself into a large number of subjobs. Each of these subjobs is a separate job in the grid. However, the application may implement an algorithm that requires that the subjobs communicate some information among them. The subjobs need to be able to locate other specific subjobs,establish a communications connection with them, and send the appropriate data. The open standard Message Passin Interface (MPI) and any of several variations is often included as part of the grid system for just this kind of communication.

Grid components

Observation, management, and measurementWe mentioned above the schedulers react to

current loads on the grid. Usually, the donor software will include some tools that measure the current load and activity on a given machine using either operating system facilities or by direct measurement. This software is sometimes referred to as a “load sensor.” Some grid systems provide the means for implementing custom load sensors for other than CPU or storage resources.

22

Grid User Roles ---A User’s Perspective

Enrolling and installing grid software Logging onto the grid Queries and submitting jobs Data configuration Monitoring progress and recovery Reserving resources

23

Grid User Roles---An Administrator’s Perspective

Planning Installation Managing enrollment of donors and

users Certificate authority Resource management Data sharing

Using a grid: An application developer’s perspective(1)

Applications that are not enabled for using multiple processors but can be executed on different machines.

Applications that are already designed to use the multiple processors of a grid setting.

Applications that need to be modified or rewritten to better exploit a grid Tools for debugging and measuring the

behavior of grid applications

Using a grid: An application developer’s perspective(2)

Globus developer’s toolkit

Manage grid operations Measurement Repair Debug grid applications

Open Grid Services Architecture (OGSA)

26

Grid Architecture

Application

Collective

Resource

Connectivity

Fabric

Application

Transport

Internet

Link

GRID

Internet

Grid Computing

27

Grid Architecture

Fabric layer: Provides the resources to which shared access is mediated by Grid protocols.

Connectivity layer: Defines the core communication and authentication protocols required for grid-specific network functions.

Resource layer: Defines protocols, APIs, and SDKs for secure negotiations, initiation, monitoring control, accounting and payment of sharing operations on individual resources.

Collective Layer: Contains protocols and services that capture interactions among a collection of resources.

Application Layer: These are user applications that operate within VO (Virtual Organization ) environment.

28

Standards for Grid Environments

Global Grid Forum (GGF)http://www.ggf.org Organization for the Advancement of

Structured Information Standards (OASIS)

http://www.oasis-open.org/ World Wide Web Consortium (W3C)

http://www.w3.org/ Distributed Management Task Force (DMTF)

http://www.dmtf.org/ Web Services Interoperability Organization

(WS-I)http://www.ws-i.org/

Globus Toolkit v5

The Globus Alliance is made up of organizations and individuals that develop and make available various technologies applicable to grid computing.

The Globus toolkit v5 includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications.

For more information visit http://www.globus.org

Challenges

Trust (security is built on trusted parties or trusted third-party CA) Problem: how to trust VO members and its agents (autonomous apps) Solution: proxy credentials provided by a CA in public key infrastructure

Sharing of applications and data Problem: incompatible machines and OS, need to limit access Solution: virtualization, Grid resource allocation policies

Communication of Grid policies and metadata: Grid interoperability Problem: incompatible protocols Solution: XML-based protocols and open standards

Reliability and robustness (a non-functional requirement) Problem: Grid-based systems can be brittle (network connections) Solution: two-phase commit, transaction-based protocols

Quality of service (QoS) (a non-functional requirement) Problem: need end-to-end resource management, transactions Solution: budgeting of cycles, bandwidth, and storage capacity

Finally

grid computing assumes and/or requires technologies that include: Support for executing programs on a

variety of platforms A secure infrastructure Data movement/replication/federation Resource discovery Resource management

References

Introduction to Grid Computing IBM Redbook

The Grid 2: Blueprint for a New Computing Infrastructure, 2nd Edition from Ian Foster, Carl Kesselman.

The Anatomy of the Grid: Enabling Scalable Virtual Organizations by I. Foster, C. Kesselman and S. Tuecke , International J. Supercomputer Applications, 15(3), 2001 http://www.globus.org/alliance/publications/papers/anatomy.pdf