military technical academy bucharest, 2004 grid models grid functions reasoning & benefits adina...

Military Technical Academy Bucharest, 2004

GRID MODELSGRID MODELS

GRID FUNCTIONSGRID FUNCTIONS

REASONINGREASONING & BENEFITS & BENEFITS

ADINA RIPOSANADINA RIPOSANApplied Information TechnologyApplied Information Technology

Department of Computer EngineeringDepartment of Computer Engineering

Military Technical Academy BucharMilitary Technical Academy Bucharest, 2004est, 2004

GRID REASONINGGRID REASONING

Exploiting underutilizedExploiting underutilized resourcesresources

Resource balancingResource balancing effecteffect

Massive Massive Parallel CPU capacityParallel CPU capacity

((Computational GridComputational Grids)s)

Grid-enabled ApplicationsGrid-enabled Applications

Scheduling, reservation,Scheduling, reservation, and scavengingand scavenging

Disk Disk DDrive capacityrive capacity ((Data GridData Grids)s)

Data Communication capacityData Communication capacity

Grid Grid AAccountingccounting

ReliabilityReliability

ManagementManagement

VVirtual irtual OOrganizationsrganizations (VOs); (VOs); Virtual resourcesVirtual resources


Some grids are designed to take advantage of Some grids are designed to take advantage of

extra processing resourcesextra processing resources, ,

whereas some grid architectures are designed whereas some grid architectures are designed to to support collaborationsupport collaboration between various between various organizations.organizations.

=> The => The type of gridtype of grid selected is based primarily on selected is based primarily on the business problem that is being solved.the business problem that is being solved.

=> The selection of a specific => The selection of a specific grid typegrid type will have a will have a direct impact on the grid solution design.direct impact on the grid solution design.


GRID MODELSGRID MODELS

1. Computational grid1. Computational grid A computational grid aggregates the processing A computational grid aggregates the processing

power from a distributed collection of systems.power from a distributed collection of systems.

2. Data grid2. Data grid While computational grids are more suited for While computational grids are more suited for

aggregating resources, data grids focus on aggregating resources, data grids focus on providing secure access to distributed, providing secure access to distributed, heterogeneous pools of data.heterogeneous pools of data.

3. Access grid3. Access grid


GRID REASONINGGRID REASONING

In creating the Grid, there are different In creating the Grid, there are different possible approaches:possible approaches:

1.1. To scavenge CPU cyclesTo scavenge CPU cycles from existing desktops from existing desktops throughout the institutions that join the grid.throughout the institutions that join the grid.

2.2. Alternatively, Alternatively, to have dedicated servers and to have dedicated servers and machinesmachines for use in the computational grid. for use in the computational grid.

3.3. To To BOTHBOTH scavenge existing desktops and scavenge existing desktops and establish dedicated resources for the establish dedicated resources for the computational grid.computational grid.


In case of SCAVENGE existing desktops, In case of SCAVENGE existing desktops,

a a protective “SANDBOX”protective “SANDBOX” should be should be

implemented on the Grid member-machines, so implemented on the Grid member-machines, so

that:that:

=> It cannot cause any disruption to the => It cannot cause any disruption to the donating donating machinemachine if it encounters a problem during if it encounters a problem during execution. execution.

=> => Rights to accessRights to access files and other resources on files and other resources on

the grid machine from inside the Gridthe grid machine from inside the Grid may be may be restrictedrestricted..


Exploiting underutilized Exploiting underutilized resourcesresources


Grid computing provides a framework for Grid computing provides a framework for exploiting exploiting underutilized resourcesunderutilized resources

and thus has the possibility of substantially and thus has the possibility of substantially increasing the efficiency of resource usage.increasing the efficiency of resource usage.

This applies toThis applies to::

CPU, storage, software, services, licensesCPU, storage, software, services, licenses

and many other kinds of resources that may be and many other kinds of resources that may be available on a grid.available on a grid.

• The easiest use of grid computing is to run an The easiest use of grid computing is to run an existing application on a different machineexisting application on a different machine

The job in question could be The job in question could be run on an idle run on an idle machinemachine elsewhere on the grid. elsewhere on the grid.


Special equipment, capacities, architecturesSpecial equipment, capacities, architectures

• Platforms on the Platforms on the GGrid will often have different rid will often have different architectures, operating systems, devices, architectures, operating systems, devices, capacities, and equipmentcapacities, and equipment::

=> => Represent Represent different kinds of resourcedifferent kinds of resource that the that the GGrid can rid can use as criteria and attributesuse as criteria and attributes for assigning jobs to for assigning jobs to machinesmachines. .

• The administrator of a The administrator of a GGrid may create rid may create a new a new artificial resource typeartificial resource type

• that is used by schedulers to assign work according to that is used by schedulers to assign work according to policy rules or other constraints.policy rules or other constraints.

=> => The administrators would need to impose a The administrators would need to impose a classification classification on each kind of jobon each kind of job through some certification procedure through some certification procedure to use this kind of approach.to use this kind of approach.


Some machines on the grid may have special Some machines on the grid may have special devicesdevices

• Some machines on the grid may be Some machines on the grid may be connected to scanning electron microscopes connected to scanning electron microscopes that can be operated remotely that can be operated remotely => => In this case, scheduling and reservation are In this case, scheduling and reservation are

important.important. A specimen could be sent in advance to the facility A specimen could be sent in advance to the facility

hosting the microscope. hosting the microscope. Then the user can remotely operate the machine, Then the user can remotely operate the machine,

changing perspective views until the desired image changing perspective views until the desired image is captured.is captured.

• The Grid can enable more elaborate access,The Grid can enable more elaborate access, potentially to remote medical diagnostic and potentially to remote medical diagnostic and

robotic surgery tools with two-way interaction from robotic surgery tools with two-way interaction from a distancea distance..


Software and licensesSoftware and licenses

• Some machines may have Some machines may have expensive licensed expensive licensed softwaresoftware installed that the user requires, installed that the user requires,

• His jobs can be sent to such machines on which His jobs can be sent to such machines on which this software happens to be installed, thus more this software happens to be installed, thus more fully exploiting the software licenses.fully exploiting the software licenses.

The software may be too expensive to install on The software may be too expensive to install on every grid machine. every grid machine.

When the licensing fees are significant, this When the licensing fees are significant, this approach can save significant expenses for an approach can save significant expenses for an organization.organization.


• Some Some Software licensing arrangementsSoftware licensing arrangements permit permit the software to be installed on all of the the software to be installed on all of the machines of a machines of a GGrid rid

but may limit the number of installations that can but may limit the number of installations that can be simultaneously used at any given instant. be simultaneously used at any given instant.

• License management softwareLicense management software keeps track of how many concurrent copies of the keeps track of how many concurrent copies of the

software are being usedsoftware are being used,, and and prevents more than that number from executing at prevents more than that number from executing at

any given time. any given time.

The The grid job schedulersgrid job schedulers can be configured to can be configured to take software licenses into account, take software licenses into account, optionally balancing them against other optionally balancing them against other priorities or policies.priorities or policies.


Resource balancing Resource balancing

effecteffect


• Another function of the grid is to Another function of the grid is to better better balance resource utilizationbalance resource utilization

• In fact, some In fact, some GGrid implementations rid implementations can can migrate partially completed jobsmigrate partially completed jobs. .

For example, a batch job that spends a significant For example, a batch job that spends a significant amount of time processing a set of input data to amount of time processing a set of input data to produce an output set is perhaps the most ideal produce an output set is perhaps the most ideal and simple use for a grid.and simple use for a grid.

• In general, a In general, a GGrid can provide a consistent rid can provide a consistent way way to balance the loads on a wider to balance the loads on a wider federation of resourcesfederation of resources..


For applications that are gridFor applications that are grid--enabled, the enabled, the GGrid can rid can offer a resource balancing effect byoffer a resource balancing effect by SCHEDULING grid jobs on machines with low SCHEDULING grid jobs on machines with low utilizationutilization..

Jobs are migrated to less busy parts of the Jobs are migrated to less busy parts of the GGrid rid to balance resource loads and to balance resource loads and absorb unexpected peaks of activity in a part of an absorb unexpected peaks of activity in a part of an

organization. organization.

Without a Without a GGrid infrastructure, such balancing decisions rid infrastructure, such balancing decisions are difficult to prioritize and execute.are difficult to prioritize and execute.

• An An ADVANCED SCHEDULERADVANCED SCHEDULER could schedule them could schedule them to minimize communications trafficto minimize communications traffic,, or or minimize the distance of the communicationsminimize the distance of the communications

=> => This can potentially reduce communication This can potentially reduce communication and other forms of contention in the grid.and other forms of contention in the grid.


Handling occasional peak loadsHandling occasional peak loads of activity of activity in parts of an larger organization in parts of an larger organization

This can happen in two ways:This can happen in two ways:

• An unexpected peak can be An unexpected peak can be routedrouted to relatively to relatively idle machines in the idle machines in the GGrid.rid.

• If the If the GGrid is already fully utilized, the lowest rid is already fully utilized, the lowest prioritypriority work being performed on the work being performed on the GGrid can be rid can be temporarily suspended or even cancelled and temporarily suspended or even cancelled and performed again later to make room for the performed again later to make room for the higher priority work.higher priority work.


Massive Massive Parallel CPU capacity Parallel CPU capacity



Massive Massive Parallel CPU capacityParallel CPU capacity


• The potential for The potential for massive parallel CPU massive parallel CPU capacitycapacity will be one of the most attractive will be one of the most attractive features of a grid.features of a grid.

=> => The most common resource is The most common resource is computing cycles computing cycles provided by the processorsprovided by the processors of the machines on of the machines on the grid.the grid.

• The The processors can varyprocessors can vary in speed, in speed, architecture, software platform, and other architecture, software platform, and other associated factors, such as memory, associated factors, such as memory, storage, and connectivity.storage, and connectivity.


A COMPUTATIONAL GRID aggregates the A COMPUTATIONAL GRID aggregates the processing power from a distributed processing power from a distributed collection of systems.collection of systems.

• One benefit would be to modify specific One benefit would be to modify specific vertical vertical applicationsapplications for for parallel computingparallel computing opportunities opportunities

• Another benefit: the processes may require Another benefit: the processes may require more more computer capacitycomputer capacity than is available. than is available.

• Reduced Total Cost of Ownership (Reduced Total Cost of Ownership (TCOTCO), and ), and shorter deployment life cycles.shorter deployment life cycles.

• The next generation of computational grid: shift The next generation of computational grid: shift focus towards solving focus towards solving real-time computational real-time computational problemsproblems..


There are There are 33 primary ways to exploit the primary ways to exploit the computation resources of a Grid: computation resources of a Grid:

1.1. The first and simplest is to use it The first and simplest is to use it to run an to run an existing application on an available machineexisting application on an available machine on the on the GGrid rather than locally.rid rather than locally.

2.2. The second is The second is to use an application to use an application designed to split its workdesigned to split its work in such a way that in such a way that the the separate parts can execute in parallel separate parts can execute in parallel on different processorson different processors. .

3.3. The third is The third is to run an application that needs to run an application that needs to be executed many timesto be executed many times on many on many different machinesdifferent machines in the in the GGrid.rid.


Regarding the second utilizationRegarding the second utilization type, the type, the common attribute among such uses is thatcommon attribute among such uses is that::

=>=>the the AApplications have been written to use pplications have been written to use algorithmsalgorithms that can be partitioned into that can be partitioned into independently running parts.independently running parts.

(see Jobs and Applications)(see Jobs and Applications)

• A A CPU intensive CPU intensive GGrid rid AApplicationpplication can be can be thought of as many smaller thought of as many smaller “subjobs”“subjobs”, each , each executing on a different machine in the executing on a different machine in the GGrid.rid.

• To the extent that these subjobs do not To the extent that these subjobs do not need to communicate with each other, the need to communicate with each other, the more “more “scalablescalable” the application becomes.” the application becomes.


““Scalability” is a measure of how efficiently the Scalability” is a measure of how efficiently the multiple processors on a multiple processors on a GGrid are used rid are used

• If twice as many processors makes an If twice as many processors makes an application complete in one half the time, then application complete in one half the time, then it is said to be it is said to be perfectly scalableperfectly scalable. .

• A A perfectly scalable applicationperfectly scalable application will, for will, for example, finish 10 times faster if it uses 10 example, finish 10 times faster if it uses 10 times the number of processors.times the number of processors.

• However, there may be However, there may be limits to scalabilitylimits to scalability when when applications can only be split into a limited applications can only be split into a limited number of separately running parts or if those number of separately running parts or if those parts experience some other contention for parts experience some other contention for resources of some kind.resources of some kind.


Barriers to perfect scalability:Barriers to perfect scalability:

• The first barrier depends on The first barrier depends on the algorithms used for the algorithms used for splitting the applicationsplitting the application among many CPUs among many CPUs

=> => If the algorithm can only be split into a If the algorithm can only be split into a limited number of limited number of independently running partsindependently running parts, then that forms a , then that forms a scalability barrier. scalability barrier.

• The second barrier appears The second barrier appears if the parts are not if the parts are not completely independentcompletely independent

=> T=> This can cause his can cause contentioncontention, which can limit scalability. , which can limit scalability.

For example, if all of the subjobs need to read and write For example, if all of the subjobs need to read and write from one common file or database, the access limits of from one common file or database, the access limits of that file or database will become the limiting factor in the that file or database will become the limiting factor in the application’s scalability. application’s scalability.

• Other sources of inter-job contention in a parallel grid Other sources of inter-job contention in a parallel grid application include: application include: message communications message communications latencieslatencies among the jobs, among the jobs, network communication network communication capacitiescapacities, , synchronization protocolssynchronization protocols, , input-output input-output bandwidth to devicesbandwidth to devices and and storage devicesstorage devices, and , and latencies interfering with real-time requirementslatencies interfering with real-time requirements


Grid-enabled ApplicationsGrid-enabled Applications


NNot all ot all AApplicationspplications can be transformed to run can be transformed to run in parallel on a grid and achieve scalability.in parallel on a grid and achieve scalability.

Grid Grid AApplications can be categorized in one of the pplications can be categorized in one of the following 3 categories:following 3 categories:

• Applications that Applications that are not enabledare not enabled for using for using multiple processors but multiple processors but can be executedcan be executed on on different machines. different machines.

• Applications that Applications that are already designedare already designed to use to use the multiple processors of a the multiple processors of a GGrid setting.rid setting.

• Applications that Applications that need to be modifiedneed to be modified or or rewrittenrewritten to better exploit a to better exploit a GGrid.rid.


There are many factors to consider in grid-There are many factors to consider in grid-enabling an enabling an AApplicationpplication

• New computation intensive applicationsNew computation intensive applications written written today are being today are being designed for parallel executiondesigned for parallel execution => => and these will be and these will be easily grideasily grid--enabledenabled, if they do not , if they do not

already follow emerging grid protocols and already follow emerging grid protocols and standards.standards.

• There are some There are some practical toolspractical tools that skilled that skilled application designers can use to write a application designers can use to write a parallel parallel grid applicationgrid application..

• There are There are NONO practical tools practical tools for transforming for transforming arbitrary applicationsarbitrary applications to exploit the parallel to exploit the parallel capabilities of a grid. capabilities of a grid. => => Automatic transformation of applications is a Automatic transformation of applications is a

science in its infancy.science in its infancy.


JOBS AND APPLICATIONSJOBS AND APPLICATIONS

Although various kinds of resources on the Although various kinds of resources on the GGrid may rid may be shared and used, they are usually accessed be shared and used, they are usually accessed via an executing via an executing “Application”“Application” or or “Job”“Job”..

““Application”Application” = the highest level of a piece of = the highest level of a piece of work on the grid work on the grid

SSometimes the term ometimes the term “job”“job” is used is used equivalently equivalently

An An ApplicationApplication is is one or more jobsone or more jobs that are that are scheduled to run on machines in the scheduled to run on machines in the GGrid rid

=> => the results are collected and assembled to the results are collected and assembled to produce the answer.produce the answer.


• ApplicationsApplications may be broken down into any may be broken down into any number of individual number of individual JobsJobs..

Those, in turn, can be further broken down Those, in turn, can be further broken down into into “subjobs”“subjobs” (transactions, work units, (transactions, work units, submissions etc.)submissions etc.)

• JobsJobs are programs that are executed at an are programs that are executed at an

appropriate point on the appropriate point on the GGrid. rid.

They may They may computecompute something, something, executeexecute one or one or more system commands, more system commands, move or collectmove or collect data, or data, or operateoperate machinery. machinery.

A A GGrid rid AApplicationpplication that is organized as a that is organized as a collection of collection of JJobsobs is usually designed to have is usually designed to have these these jobs execute in paralleljobs execute in parallel on different on different machines in the machines in the GGrid.rid.


The The jjobs may have obs may have specific DEPENDENCIESspecific DEPENDENCIES that that may prevent them from executing in may prevent them from executing in parallel in all cases.parallel in all cases.

They may require some specific input data They may require some specific input data that must be that must be copied to the machinecopied to the machine on which on which the job is to run. the job is to run.

Some jobs may require Some jobs may require the output produced the output produced by certain other jobsby certain other jobs and cannot be executed and cannot be executed until those prerequisite jobs have completed until those prerequisite jobs have completed executing. executing.

Jobs may spawn additional subjobs, Jobs may spawn additional subjobs, depending on the data they process. depending on the data they process.

This work flow can create a This work flow can create a hierarchy of jobs hierarchy of jobs and subjobsand subjobs..


Finally, the results of all of the Finally, the results of all of the JJobsobs must bemust be

collectedcollected

and and

appropriately assembledappropriately assembled

to produce the ultimate answer for the to produce the ultimate answer for the AApplicationpplication..


Scheduling, reservation, Scheduling, reservation,

and scavengingand scavenging


Scheduling, reservation, Scheduling, reservation,

and scavengingand scavenging

The The Grid systemGrid system is responsible for is responsible for sending a jobsending a job to a given machine to be to a given machine to be executed. executed.

Advanced Advanced GGrid systemsrid systems => use various => use various combinations of combinations of

schedulingscheduling, ,

reservationreservation,, and and

scavengingscavenging

to more completely utilize the to more completely utilize the GGrid.rid.


Job “SCHEDULER”Job “SCHEDULER” - automatically finds the - automatically finds the most appropriate machine on which to run most appropriate machine on which to run any given job that is waiting to be executed.any given job that is waiting to be executed.

Schedulers react to current availability of Schedulers react to current availability of resources on the resources on the GGrid.rid.

““Scheduling”Scheduling” ≠ ≠ ““RReservation”eservation”

““RESERVATIONRESERVATION““ of resources in advanceof resources in advance

=> => to improve the quality of serviceto improve the quality of service (QoS) (QoS)

If If “Scheduler”“Scheduler” = = ““RResource broker”esource broker”

=> => it implies that some it implies that some bartering capabilitybartering capability is factored into scheduling.is factored into scheduling.


1.1. ““Scavenging” Scavenging” GGrid system:rid system:

• Any machine that becomes idle would typically Any machine that becomes idle would typically report its report its idle statusidle status to the to the GGrid rid MManagement anagement node. node.

• This This MManagement node would assign to this anagement node would assign to this idle idle machinemachine the next jobthe next job that is satisfied by the that is satisfied by the machine’s resources. machine’s resources.

Scavenging is usually implemented in a way that isScavenging is usually implemented in a way that is unobtrusive to the normal machine userunobtrusive to the normal machine user. .

• If the machine becomes busy with local non-grid If the machine becomes busy with local non-grid work, work, the grid jobthe grid job is usually is usually suspended/delayedsuspended/delayed

=> => This situation creates somewhatThis situation creates somewhat unpredictable unpredictable completion timescompletion times for grid jobsfor grid jobs, although it is not , although it is not disruptive to those machines donating resources to disruptive to those machines donating resources to the the GGrid.rid.


2.2. Machines “dedicated” to the Machines “dedicated” to the GGrid:rid:

• To create To create more predictable behaviormore predictable behavior

• The The GGrid machines rid machines are not preemptedare not preempted by by outside workoutside work

=> This enables => This enables SSCHEDULERS to compute the CHEDULERS to compute the

approximate completion timeapproximate completion time for a set of jobs, for a set of jobs,

when their running characteristics are known.when their running characteristics are known.


““RESERVATION” in advanceRESERVATION” in advance for a designated for a designated set of jobs.set of jobs.

• Grid resources can be Grid resources can be reserved in advancereserved in advance, , as a further step as a further step

=> To meet => To meet deadlinesdeadlines and guarantee and guarantee QoSQoS (quality of service). (quality of service).

• When POLICIES permit, resources When POLICIES permit, resources reserved reserved in advancein advance could could also be scavengedalso be scavenged

=> T=> To run o run lower priority jobslower priority jobs when they are when they are not busy during a reservation period, not busy during a reservation period, yyielielding ding to jobsto jobs for which they are reserved for which they are reserved


Scheduling & reservation for Scheduling & reservation for

single / multiple resources:single / multiple resources:

• Scheduling and reservation is fairly Scheduling and reservation is fairly straightforward straightforward when only one resource when only one resource typetype, usually CPU, is involved. , usually CPU, is involved.

• Additional Additional GGrid optimizationsrid optimizations can be can be achieved by considering achieved by considering more resourcesmore resources in in the scheduling and reservation process. the scheduling and reservation process.

It would be desirable to assign It would be desirable to assign executing jobsexecuting jobs to to machines nearest to the datamachines nearest to the data that these jobs that these jobs requirerequire

reduce network trafficreduce network traffic and and reduce scalability limitsreduce scalability limits (possibly) (possibly)


Optimal schedulingOptimal scheduling, considering , considering multiple multiple

resourcesresources, is a , is a difficult mathematics problemdifficult mathematics problem. .

• Such Such SSchedulerschedulers may use may use HEURISTICSHEURISTICS = rules = rules

designed to improve the probability of finding the designed to improve the probability of finding the

best combination of job schedules and best combination of job schedules and

reservations to optimize throughput or any other reservations to optimize throughput or any other

metric.metric.


Disk drive capacity Disk drive capacity

((Data GridData Grids)s)

available available unused storageunused storage


Disk drive capacityDisk drive capacity

The processing resources are not the only ones that The processing resources are not the only ones that may be underutilized.may be underutilized.

Often, machines may have enormous unused Often, machines may have enormous unused disk disk drive capacitydrive capacity. .

=> => SHARING starts with DATA in the form of SHARING starts with DATA in the form of files or databasesfiles or databases

• Files or databasesFiles or databases can seamlessly span many systems can seamlessly span many systems and thus have and thus have larger capacitieslarger capacities than on any single than on any single system. system.

• Such spanning can Such spanning can improve data transferimprove data transfer rates rates through the use of striping techniques.through the use of striping techniques.


““DATA GRID” = A DATA GRID” = A GGrid providing an integrated rid providing an integrated view of view of data storagedata storage

• Each machineEach machine on the on the GGrid usually provides rid usually provides some some quantity of storagequantity of storage for for GGrid use, even if rid use, even if temporary. temporary.

• ““Data grid” can be used to aggregate this Data grid” can be used to aggregate this unused storageunused storage into a much larger into a much larger virtual virtual data storedata store, ,

=> => possibly configured to achieve possibly configured to achieve improved improved performance and reliabilityperformance and reliability over that of any over that of any single machine.single machine.


If a batch job needs to read a large amount of data, If a batch job needs to read a large amount of data, this this data could be automatically replicateddata could be automatically replicated at at various strategic points in the various strategic points in the GGrid. rid.

• Thus, if the job must be executed on a remote Thus, if the job must be executed on a remote machine in the machine in the GGridrid

=> => the data is already there and does not need to be the data is already there and does not need to be moved to that remote point. moved to that remote point.

=> t=> this offers clear performance benefitshis offers clear performance benefits

Data can be hosted on or near the machines most Data can be hosted on or near the machines most likely to need the data, in conjunction with likely to need the data, in conjunction with advanced scheduling techniquesadvanced scheduling techniques..

Also, such copies of data can be used as Also, such copies of data can be used as backupsbackups when the primary copies are damaged or when the primary copies are damaged or unavailable.unavailable.


Storage capacityStorage capacity

=> => The second most common The second most common resourceresource used used in a in a GGridrid

Storage can be:Storage can be:

• Memory attached to the processorMemory attached to the processor

• ““Secondary storage”, using hard disk drives or other Secondary storage”, using hard disk drives or other permanent storage media.permanent storage media.

Memory attached to the processor: Memory attached to the processor:

Usually has very fast access but is volatile. Usually has very fast access but is volatile.

It would best be used to cache data to serve as It would best be used to cache data to serve as

temporary storage for running applications.temporary storage for running applications.


““Secondary storage”, using hard disk drives or Secondary storage”, using hard disk drives or

other permanent storage media. other permanent storage media.

Can be used to increase capacity, performance, Can be used to increase capacity, performance,

sharing, and reliability of data. sharing, and reliability of data.

Many grid systems use mountable networked file Many grid systems use mountable networked file

systems, such as Andrew File System (AFS), systems, such as Andrew File System (AFS),

Network File System (NFS), Distributed File System Network File System (NFS), Distributed File System

(DFS), or General Parallel File System (GPFS). (DFS), or General Parallel File System (GPFS).

=> => These offer varying degrees of performance, These offer varying degrees of performance,

security features, and reliability features.security features, and reliability features.


CapacityCapacity can be increased by using the storage on can be increased by using the storage on multiple machines with a unifying file system. multiple machines with a unifying file system.

• Any individual file or data base can span several Any individual file or data base can span several storage devices and machines,storage devices and machines,

=> => eliminating eliminating maximum size restrictionsmaximum size restrictions often imposed often imposed by file systems shipped with operating systems.by file systems shipped with operating systems.

• A unifying file system can also provide a A unifying file system can also provide a single uniform single uniform name spacename space for for GGrid storage. rid storage.

=> This makes it easier for users to => This makes it easier for users to referencereference datadata residing in the residing in the GGrid, without regard for itsrid, without regard for its exact exact location.location.

• In a similar way, special In a similar way, special database softwaredatabase software can can “federate”“federate” an assortment of individual databases and an assortment of individual databases and files files

=> => to form a larger, more comprehensive data base, to form a larger, more comprehensive data base, accessible using database query functions.accessible using database query functions.


More advanced file systemsMore advanced file systems on a on a GGrid can rid can automatically duplicateautomatically duplicate sets of data, sets of data,

• to provide REDUNDANCY for to provide REDUNDANCY for increased reliabilityincreased reliability and and increased performanceincreased performance. .

An intelligent An intelligent GGrid rid SSchedulercheduler can help select can help select the appropriate storage devices to hold data, the appropriate storage devices to hold data, based on based on usage patternsusage patterns. .

• Jobs can then be Jobs can then be scheduled closer to the datascheduled closer to the data, , preferably on the machines directly connected to the preferably on the machines directly connected to the storage devices holding the required data.storage devices holding the required data.


A grid file system can also implement A grid file system can also implement

JOURNALINGJOURNALING

=>=>Data can be Data can be recovered more reliablyrecovered more reliably after certain after certain kinds of failures.kinds of failures.

Some file systems implement Some file systems implement

Advanced Synchronization mechanismsAdvanced Synchronization mechanisms

• to to reduce contentionreduce contention when data is shared and when data is shared and updated by many usersupdated by many users..


DATA STRIPINGDATA STRIPING can also be implemented by can also be implemented by grid file systemsgrid file systems

• When there are When there are sequential or predictable access sequential or predictable access patterns to datapatterns to data, this technique can create the , this technique can create the virtual effectvirtual effect of having storage devices that can of having storage devices that can transfer data at a faster rate than any individual transfer data at a faster rate than any individual disk drive.disk drive.

• This can be important for This can be important for multimedia data multimedia data streamsstreams or when or when collecting large quantities of collecting large quantities of data at extremely high ratesdata at extremely high rates from CAT scans or from CAT scans or particle physics experiments.particle physics experiments.

DATA STRIPING DATA STRIPING = writing or reading successive records = writing or reading successive records to/from different physical devices, overlapping the access to/from different physical devices, overlapping the access for faster throughput; additional techniques increase for faster throughput; additional techniques increase reliability.reliability.



• CCommunications within the ommunications within the GGridrid • EExternal xternal communicationcommunication



This includes This includes communications within the gridcommunications within the grid and and external to the gridexternal to the grid..

If a user needs If a user needs to increase his total bandwidth to the to increase his total bandwidth to the InternetInternet, the work can be split among , the work can be split among GGrid machines that rid machines that have independent connections to the Internet. have independent connections to the Internet.

• If the machines had shared the connection to the If the machines had shared the connection to the Internet, there would not have been an effective Internet, there would not have been an effective increase in bandwidth.increase in bandwidth.

• Potential use: to implement a data mining search engine Potential use: to implement a data mining search engine => the total searching capability is multiplied.=> the total searching capability is multiplied.



A A GGrid provides excellent infrastructure for rid provides excellent infrastructure for brokering resourcesbrokering resources => =>

=> => This can form the basis for This can form the basis for GGrid rid AAccountingccounting and the ability to more fairly distribute work on and the ability to more fairly distribute work on the the GGrid.rid.

Individual resources can be profiled to determine Individual resources can be profiled to determine their their availabilityavailability and their and their capacitycapacity, and this can , and this can be factored into be factored into SScheduling on the cheduling on the GGridrid. .

Different organizations participating in the Different organizations participating in the GGrid can rid can build up build up GGrid creditsrid credits and use them at times when and use them at times when they need additional resources. they need additional resources.



Redundant Redundant ggrid configurationrid configuration and and RRedundant job submissionedundant job submission

=> => used to achieve high reliabilityused to achieve high reliability

Grid systems will utilize Grid systems will utilize ““AAutonomic computing”utonomic computing”

• This is a type of software that This is a type of software that automatically heals automatically heals problems in the gridproblems in the grid, perhaps even before an , perhaps even before an operator or manager is aware of them.operator or manager is aware of them.

• In principle, most of the reliability attributes achieved In principle, most of the reliability attributes achieved using using hardwarehardware in today’s high availability systems in today’s high availability systems can be achieved using can be achieved using softwaresoftware in a in a GGridrid setting in setting in the future.the future.


Fail-over scenarios / Recovery scenariosFail-over scenarios / Recovery scenarios

Of prime importance is understanding the Of prime importance is understanding the fail-over scenariosfail-over scenarios for the given Grid system for the given Grid system

=> so that the => so that the Grid can continue operatingGrid can continue operating even even if any of the management machines fails in if any of the management machines fails in some way some way

Machines should be configured and Machines should be configured and connected to facilitate connected to facilitate recovery scenariosrecovery scenarios..


ManagementManagement


Management can use a Management can use a GGrid to better view the rid to better view the usage patternsusage patterns in the larger organization, in the larger organization,

== > > permitting permitting better planningbetter planning when when upgrading systemsupgrading systems

increasing capacity, or increasing capacity, or

retiring computing resources no longer neededretiring computing resources no longer needed

Autonomic computingAutonomic computing => => Various tools may be able Various tools may be able

to to identify important trendsidentify important trends throughout the throughout the GGrid,rid,

informing management of those that require informing management of those that require

attention.attention.


The management of priorities The management of priorities

among among different Projectsdifferent Projects::

In the past, each project may have been responsible for its In the past, each project may have been responsible for its own IT resource hardware and the expenses associated own IT resource hardware and the expenses associated with it.with it.

Aggregating utilization dataAggregating utilization data over a larger set of projects: over a larger set of projects: => => AA project may suddenly rise in importance with a specific project may suddenly rise in importance with a specific

deadline.deadline.• If the size of the job is known, if it is a kind of If the size of the job is known, if it is a kind of job that can be job that can be

sufficiently split into subjobssufficiently split into subjobs, and if enough resources are , and if enough resources are available after preempting lower priority work, a available after preempting lower priority work, a GGrid can bring rid can bring a very large amount of processing power to solve the problem.a very large amount of processing power to solve the problem.

• In such situations, a In such situations, a GGrid can, with some rid can, with some planningplanning, succeed in , succeed in meeting a surprise deadline.meeting a surprise deadline.

When When maintenancemaintenance is required, is required, GGrid work can be rerouted rid work can be rerouted to other machines without crippling the projects involved.to other machines without crippling the projects involved.


VVirtual irtual OOrganizationsrganizations (VOs) (VOs)

Virtual resourcesVirtual resources


Virtual resources and Virtual resources and

VVirtual irtual OOrganizationsrganizations (VOs) (VOs)

for collaborationfor collaboration

Another important Another important GRID benefit is to GRID benefit is to enable and enable and simplify collaborationsimplify collaboration among a wider audience, among a wider audience, • offering offering important standardsimportant standards that enable very that enable very

heterogeneous systems to work togetherheterogeneous systems to work together

The users of the GRID can be organized dynamically into The users of the GRID can be organized dynamically into a a number ofnumber of Virtual Organizations (VOs)Virtual Organizations (VOs), ,

each with different each with different POLICY REQUIREMENTSPOLICY REQUIREMENTS

=> These => These Virtual OrganizationsVirtual Organizations can share their can share their resources collectively as a larger resources collectively as a larger GridGrid..


• Administrators can change Administrators can change any number of any number of policiespolicies that affect how the different organizations that affect how the different organizations might share or compete for resources.might share or compete for resources.

• Administrators can adjust POLICIES to Administrators can adjust POLICIES to better better allocate resourcesallocate resources

=> The Grid can help in enforcing => The Grid can help in enforcing SECURITY RULESSECURITY RULES among them and implement among them and implement POLICIESPOLICIES, which can , which can resolve resolve

prioritiespriorities for bothfor both

resourcesresources andand usersusers


Virtual Organization (VO) Virtual Organization (VO)

Consists of Consists of resources, services, and peopleresources, services, and people collaborating across institutional, geographical, collaborating across institutional, geographical, and political boundaries. and political boundaries.

military technical academy bucharest, 2004 grid models grid functions reasoning & benefits adina...

Documents