1st grid@asia workshop, beijing, china, june 21 – 23, 2005 1 marian bubak institute of computer...

1st Grid@Asia Workshop, Beijing, China, June 21 – 23, 2005 1

Marian BubakMarian BubakInstitute of Computer Science and ACC Institute of Computer Science and ACC CYFRONET CYFRONET

AGHAGH Cracow Cracow, Polan, [email protected]@agh.edu.pl

andand

CrossGrid, K-WfGrid, and (future) GridSpace CrossGrid, K-WfGrid, and (future) GridSpace CollaborationsCollaborations

From Interactive Applications From Interactive Applications and Knowledge Based Workflowsand Knowledge Based Workflows

to Transparent Semantic Grid to Transparent Semantic Grid (from X# and ~# to # )(from X# and ~# to # )


OverviewOverview

Trends in applications and computing Trends in applications and computing systemssystems

CrossGrid - Environment for application CrossGrid - Environment for application steeringsteering

Workflow applicationsWorkflow applications

K-WfGrid - knowledge-based environmentK-WfGrid - knowledge-based environment

New proposal – GridSpace: transparent New proposal – GridSpace: transparent semantic grid semantic grid


Trends in ApplicationsTrends in Applications

Large scale numerical simulationsLarge scale numerical simulations Computationally demanding data analysisComputationally demanding data analysis Distributed computing and storageDistributed computing and storage Remote access to experimental Remote access to experimental

equipmentsequipments A need for integration heterogeneous A need for integration heterogeneous

environments into one applicationenvironments into one application Collaborative problem solvingCollaborative problem solving Virtual organisationsVirtual organisations


Evolution in Distributed ComputingEvolution in Distributed Computing

Distributed systems operate in heterogenous Distributed systems operate in heterogenous environmentsenvironments

• Large scale resource sharingLarge scale resource sharing• InteroperabilityInteroperability• Communication via protocol stacksCommunication via protocol stacks

Service oriented architecturesService oriented architectures

Open standard integrationOpen standard integration

Virtualisation of resourcesVirtualisation of resources

Complexity of computing systems close to the Complexity of computing systems close to the limits of human capabilitylimits of human capability


CrossGridCrossGrid

•21 21 partnerspartners•2002-2005, EC IST F22002-2005, EC IST F2•Coordinated by CYFRONETCoordinated by CYFRONET•Research areasResearch areas

• CrossGrid ApplicationsCrossGrid Applications• Grid Tool Environment Grid Tool Environment • New Grid ServicesNew Grid Services• International Testbed International Testbed • ArchitectureArchitecture

www.eu-crossgrid.orgwww.eu-crossgrid.org


Main Main CrossGrid CrossGrid ObjectivesObjectives

New category of GridNew category of Grid--enabled applicationsenabled applications• computcompute-e- and data and data--intensiveintensive• distributeddistributed• nearnear--realreal--time response (a person in a loop)time response (a person in a loop)• layered layered

New programming toolsNew programming tools

Grid more user friendly, secure and efficientGrid more user friendly, secure and efficient

Interoperability with other GridsInteroperability with other Grids

Implementation of standardsImplementation of standards


Portal Login and Grid Proxy Creation

Bypass Placementand LB Mesh

Generation

Simulation Job Submission

Job MonitoringVirtual Node Navigation and

Grid Data Transfer

Patient in an MRI Scanner

MR Image Storage MR Image Segmentation

globus-lumc (Leiden)

mn (Virtual Operating Theatre at the UvA)

ce (Linz)ce2 (NIKHEF)

Blood Flow Visualization

Blood Flow Simulation

Blood Flow Rendering in VR

Virtual Medical SupportVirtual Medical Support


Flood SimulationFlood Simulation

Data sources

Hydrological simulation

Hydraulic simulation

Portal

Meteorological simulation


HHigh Energy Physics Aigh Energy Physics Applicationspplications

Interactive ANN trainingInteractive ANN training• MPI MPI

Parallel Sleuth algorithmParallel Sleuth algorithm

ATLAS DAQ events remoteATLAS DAQ events remote

processingprocessing• FeasFeasiibilitybility study study of using the of using the GGrid to process difficult events rid to process difficult events

in one in one of of the LHC experiments the LHC experiments ((DAQDAQ))

Distributed Distributed ddata ata aaccess prototypeccess prototype• Distributed filtering of ntuples (data files) distributed Distributed filtering of ntuples (data files) distributed onon

CrossGrid Storage ElementsCrossGrid Storage Elements• The output can be used in the ANN application.The output can be used in the ANN application.


MeteoMeteo / Pollution / Pollution AApppplicationlication

Weather Weather fforecastorecastinging

Air Air ppollution ollution fforecastorecastinging

Wave Wave modelingmodeling

Data miningData mining

Weather forecastWeather forecastinging is is thethe common link and provides common link and provides input data input data forfor all other all other activitiesactivities


CrossGrid Tools and ServicesCrossGrid Tools and Services

T e s t b e d T e s t b e d

Applications

Services

Tools

• 17 sites• 9 countries

• over 200 CPUs• 4 TB of storage

RoamingAccess Server

MigratingDesktop

OCM-G

PerformancePrediction

Data Access Globus

Toolkit

MPI Verification

MPI Library

Portal

Post-processing

InfrastructureMonitoring

Plugin

Plugin

Plugin

SOAP

SOAP

SOAP

SOAP

SOAP

Protocol

API

API

API

API

Links

API

API API

SOAP

SOAP

API

API(JMX)

(OMIS)

VisualizationKernel

Links

PerformanceAnalysis

API

ApplicationMonitoring

Benchmarks

NetworkMonitoring

MedicalSupport

ParticlePhysics

Meteo/Pollution

FloodSimulatio

n

DataGrid

Scheduler


Migrating Desktop ToolsMigrating Desktop Tools

Desktop toolsDesktop tools- Job Wizard- Job Wizard- Job Monitor- Job Monitor- Application Container and Application Plugin- Application Container and Application Plugin- GridFTP Commander- GridFTP Commander- User Profile Manager- User Profile Manager- Private Storage Management- Private Storage Management

- VNC/SSH console- VNC/SSH console


Migrating Desktop FunctionalityMigrating Desktop Functionality

Main Features:Main Features: Single sign-on / authorisationSingle sign-on / authorisation Platform independentPlatform independent Batch jobsBatch jobs MPI jobsMPI jobs Running interactive applications Running interactive applications

using java plugins or VNCusing java plugins or VNC Monitoring grid applicationsMonitoring grid applications Flexible Application frameworkFlexible Application framework User profile managementUser profile management Easy application add onEasy application add on Local and grid file management Local and grid file management


Roaming Access ServerRoaming Access Server

Well-defined set of web-services Well-defined set of web-services An interface for accessing HPC An interface for accessing HPC

systems and services (based on systems and services (based on various technologies) in a various technologies) in a common and standardised waycommon and standardised way

Interconnection between variousInterconnection between various grid middleware and applicationsgrid middleware and applications

Additional features:Additional features:• Virtual Directory supportVirtual Directory support• Plug-in for various grid Plug-in for various grid

middlewaremiddleware

JobSubmissionServices

Roaming Access Server

Interactive Session Services

File Management Services

Profile Management Services

Application Management Services


InteractivityInteractivity

1.1. Interactive workflow - uInteractive workflow - user ser interactivityinteractivity onon the Grid the GridA A user can continuously interact with user can continuously interact with a a Grid client without Grid client without waiting for twaiting for termination ermination of the jobsof the jobs submittedsubmitted

2.2. Online output control - uOnline output control - user ser oone-ne-wway ay iinteractivity nteractivity A A user can see the output of the application running user can see the output of the application running inin the Grid the Grid testbed on testbed on an an MD client synchronously with the applicationMD client synchronously with the application

3.3. Runtime steering - uRuntime steering - user ser ttwo-wo-wway ay iinteractivitynteractivityA useA use can steer the running application, either providing some can steer the running application, either providing some input data online as requested by the application input data online as requested by the application (also (also asynchronouslyasynchronously)), or suspending the process, changing some , or suspending the process, changing some input data and resuming itinput data and resuming it


Interactive Job Submission Interactive Job Submission The user submits an The user submits an

application job through the application job through the CG portal or Migrating CG portal or Migrating Desktop and the Roaming Desktop and the Roaming Access Server which Access Server which supports individual user supports individual user environments,environments,

The job is handled by the The job is handled by the Scheduler, which selects Scheduler, which selects the appropriate computing the appropriate computing resources,resources,

DataGrid software DataGrid software components are used for components are used for low-level Grid operations low-level Grid operations (submission for (submission for processing and delivery of processing and delivery of results),results),

The system bases on The system bases on Globus Toolkit v2,Globus Toolkit v2,

Other CrossGrid tools and Other CrossGrid tools and services can be used in services can be used in conjunction with running conjunction with running jobs, as requirements jobs, as requirements dictatedictate


Interactivity Interactivity via via MDMD

Job Submission Services CrossBroker

Logging&

Bookkeeping

Roaming Access Server

Computing Element

LRMS

Gatekeeper

JDLMigrating Desktop

RAS shadow port,

RAS shadow host

Process Launched

Interactive data

Control data

Submission flow

In/Out/Err job data

010011000

WorkerNode

Job Shadow

stdin – stdout -stderr

Console Agent

JobJob

Condor ByPass SystemCondor ByPass System

Job Shadow

Java Visualisation

plug-in


Scheduler (CrossBroker)Scheduler (CrossBroker)

Automatic job management for parallel Automatic job management for parallel applications:applications:

Search and selection of available resources, job Search and selection of available resources, job conditioning, job launching, job monitoring, job retry conditioning, job launching, job monitoring, job retry (in case of failures) and results retrieval.(in case of failures) and results retrieval.

• MPICH-P4 (intra-cluster)MPICH-P4 (intra-cluster)• MPICH-G2 (inter-cluster)MPICH-G2 (inter-cluster)• Computational WorkflowsComputational Workflows

Best effort approach to deal with failures/problems Best effort approach to deal with failures/problems


Performance Analysis ToolPerformance Analysis Tool

Monitors

Managers

ServiceManager

High-LevelAnalysisComponentService

OCM-GInterface(OMIS)

Main

MeasurementInterface

G-PMAnalysis ToolPerformance

User Interface& Visualization

P1

Component

Monitoring System

...

OCM-G

P2Processes

P3

...

...Application Pn

ModulesApplication Performance

MeasurementComponent

Local

...


Definition of MeasurementsDefinition of Measurements

Aggregation in time Aggregation in space

Objects Functions Partner objectsMetrics

Sites

Nodes

Processes


Definition of VisualizersDefinition of Visualizers

Visualization type:• Bar graph• Curve diagram• Histogram• Pie chart• Matrix diagram

Parameters:• Scales• Update interval


0102030405060708090

100

2 4 6 8

P1

P2

P3

Application MonitoringApplication Monitoring

RUNNING APPLICATIONRUNNING APPLICATIONRUNNING APPLICATIONRUNNING APPLICATION

OCM-G – MONITORING OCM-G – MONITORING OF APPLICATIONOF APPLICATION

OCM-G – MONITORING OCM-G – MONITORING OF APPLICATIONOF APPLICATION

TOOL - VISUALIZATIONTOOL - VISUALIZATIONTOOL - VISUALIZATIONTOOL - VISUALIZATION

Standard Interface (OMIS)

System-specific interface


OCM-G Monitoring SystemOCM-G Monitoring System

G-PMG-PM

Local Monitor

Local Monitor

Local Monitor

Local Monitor

ApplicationProcess 1


User Interface

Computing Element

Worker Node

Worker Node

Site ASite A

Site CSite C

Service Manager

Service Manager

Local Monitor

Local Monitor



ComputingElement

Worker Node

Service Manager

Service Manager

Main Service

Manager

Main Service

Manager

Site BSite B





G-PMG-PM

Local Monitor

Local Monitor

Local Monitor

Local Monitor



User Interface

Computing Element

Worker Node

Worker Node

Site ASite A

Site CSite C

Service Manager

Service Manager

Local Monitor

Local Monitor



ComputingElement

Worker Node

Service Manager

Service Manager

Main Service

Manager

Main Service

Manager

Site BSite B






OCM-G – FeaturesOCM-G – Features On-line On-line operationoperation Support for Support for multi-site grid applicationsmulti-site grid applications Low perturbationLow perturbation

• Techniques for data rate Techniques for data rate reduction reduction • LLightweight and fast socket-based communicationightweight and fast socket-based communication

FlexibleFlexible, , sservices-drivenervices-driven designdesign• No fixed metrics but a set of flexible services to construct metrics with No fixed metrics but a set of flexible services to construct metrics with

desired semanticsdesired semantics• Enables custom metrics in G-PMEnables custom metrics in G-PM

ExtendibleExtendible• AAdditional servicesdditional services can easily be added can easily be added• LLoaded dynamically at run-timeoaded dynamically at run-time

SSecureecure• GSI-based security GSI-based security • Minimal security requirements Minimal security requirements

Autonomous and standardizedAutonomous and standardized • SStandard interface tandard interface • MMinimizeinimizedd effort of porting OMIS-based tools across platforms effort of porting OMIS-based tools across platforms • Enabled interoperability of multiple tools monitoring a single application.Enabled interoperability of multiple tools monitoring a single application.


GridBench: The SuiteGridBench: The Suite

Layered Layered approachapproach

Worker-nodeWorker-node

SiteSite

and VO leveland VO level

Micro-benchmarksMicro-benchmarks

Micro-kernel Micro-kernel BenchmarksBenchmarks

Application-kernel Application-kernel BenchmarksBenchmarks


UDAL - UDAL - Unified Data Access LayerUnified Data Access Layer

DatabaseStorage

NAS

Replica ManagerEDG Reptor

Applications Portal

HSM

Componentsare kinds of

plug-ins

Components are grouped by types but they differ from each other by specialization. The rule based system takes the decision which particulartype component is the best in a context.

Applications

Applications

Unified Data Access Layer based on Component-Expert Architecture

Component TypeTID1

Component TypeTID1 Component Type

TID1

Component TypeTID1


TID1


TID1

Component TypeTID1

Component TypeTID4

Unified Data Access Cost Estimation Unified Data Access via GridFTP Protocol


BrokerClient

BrokerService

Site A

Site B

RegistryService

SiteC

RegistryService

SiteC

UserBenchmark

Services

BenchmarkAnaliserServices

InfrastructureMonitoringServices

ApplicationMonitoringServices

MigrationService

Broker SupportServices

MigrationSupportServices

HLA SpeakingService

RTIExecService

HLA ManagementServices

N-th Gridsitesupporting HLA

ApplicationFederate

Code

Gridsitesupporting HLA

HLA SpeakingService

RTIExecService

HLA Management

Services

RTIExec

HLABus

Broker SupportServices

MigrationSupportServices

PerformanceDecisionService

ApplicationMonitoring

MainServiceManager

Grid HLAGrid HLA Management System Management System HLA management servicesHLA management services

• HLA-speaking Service for HLA-speaking Service for managing federatemanaging federatess

• RTIExec Service for RTIExec Service for managing RTIExec managing RTIExec (coordination process in (coordination process in RTI)RTI)

• Broker Broker for setting up a for setting up a federationfederation and and makmakinging migration decisionsmigration decisions

Broker decision servicesBroker decision services• Registry Registry for for storingstoring

location of location of HLA-speaking HLA-speaking serviceservicess

• Infrastructure Infrastructure Monitoring/BenchmarksMonitoring/Benchmarks for checking environment for checking environment of HLA serviceof HLA service

Migration support servicesMigration support services• Application Monitoring Application Monitoring

for monitoring for monitoring performanceperformance

• Migration SMigration Serviceervice


CrossGrid TestbedCrossGrid Testbed

• Testbed sites in 9 Testbed sites in 9 countriescountries

• 17 testbed sites17 testbed sites

• Three types of Three types of testbeds: testbeds: production, production, development, testdevelopment, test

• Communication: Communication: national research national research networks and networks and GEANTGEANT


CrossGrid: Innovation, Interactivity,CrossGrid: Innovation, Interactivity, InteroperabilityInteroperability

FeaturesFeatures Brings interactive applications to the GridBrings interactive applications to the Grid Enables easy access toEnables easy access to the Gridthe Grid viavia WWeb eb SServices ervices Extends and enhances Extends and enhances DataGrid, GridLab, and EuroGridDataGrid, GridLab, and EuroGrid DDeveloped according to Geveloped according to GGGF and software engineering standardsF and software engineering standards

Potential CustomersPotential Customers EEnd-users: nd-users: hospitalshospitals, , environmental authoritiesenvironmental authorities, physicists, physicists CCompanies developing computeompanies developing compute--intensive softwareintensive software SService and infrastructure providerservice and infrastructure providers

StatusStatus SStable version available table version available sincesince March 2004 as open source March 2004 as open source LLicensing: CrossGrid license based on EDG, GPLicensing: CrossGrid license based on EDG, GPL CrossGrid Tutorial available for potential usersCrossGrid Tutorial available for potential users


K-WfGridK-WfGrid

www.kwfgrid.netwww.kwfgrid.net

Fraunhofer FIRST, Berlin, GermanyFraunhofer FIRST, Berlin, Germany Institute of Computer Science, Institute of Computer Science,

University of Innsbruck,University of Innsbruck, Innsbruck, AustriaInnsbruck, Austria Institute of Informatics of the Slovak Institute of Informatics of the Slovak

Academy of Sciences, Bratislava, Academy of Sciences, Bratislava, SlovakiaSlovakia

ACC CYFRONET AGH, Kraków, PolandACC CYFRONET AGH, Kraków, Poland LogicDIS S.A., Athens, GreeceLogicDIS S.A., Athens, Greece Softeco Sismat SpA, Genova, ItalySofteco Sismat SpA, Genova, Italy

Bratislava

Athens

Genova

KrakówInnsbruck

Berlin


Flood Flood SimulationsSimulations - Workflow - Workflow

User Portal WorkflowService

Workflow Knowledge

StorageService

MeteorologyService

HydrologyService

HydraulicsService

MeteorologyVisualization

HydrologyVisualization

HydraulicsVisualization


Workflow Applications and KnowledgeWorkflow Applications and Knowledge

• Integrating services into Integrating services into

coherent application scenarioscoherent application scenarios

• Enabling automatic Enabling automatic construction and reuse of construction and reuse of workflows with knowledge workflows with knowledge gathered during operationgathered during operation

• Involving monitoring and Involving monitoring and knowledge acquisition knowledge acquisition services in order to provide services in order to provide added value for end usersadded value for end users

TechnologiesTechnologies: service-oriented Grid architecture,: service-oriented Grid architecture, software agents, software agents, ontologies, dynamicontologies, dynamic instrumentationinstrumentation

Execute workflow

Capture knowledge

Reuse knowledge

K-WfGrid

Monitor environment

Analyze information

Construct workflow


ArchitectureArchitecture of K-WfGrid of K-WfGrid

• Capturing and Capturing and reusing reusing knowledge knowledge about Grid about Grid environmentsenvironments

• Ontology-based Ontology-based optimization of optimization of workflowsworkflows

• FFramework for ramework for collaborative collaborative knowledge reuseknowledge reuse

Grid Middleware

Web Portal

User Assistant Agent

Automatic Application Builder

Workflow Composition Tools

Grid ServiceInvocation and Control

Grid Performance Analysis Service

Grid OrganizationMemory

Grid Application Building

Users

Knowledge Builder Agent

Grid Workflow Execution Service

<templates><workflows>

<users><components>

<resources>

Grid Resources Grid Performance Monitoring Service

Knowledge


User

Workflow Orchestration and Execution

Web Portal

Grid Resources

Scheduler

Grid Performance Monitoringand Instrumentation Service

Automatic Application Builder

Workflow Composition Tool

KnowledgeAssimilation Agent

Grid OrganizationalMemory

Ontological store of

knowledge

Low Level Grid Middleware (WS-RF)

Knowledge

Grid Workflow Execution Service

PerformanceAnalysis


Grid Workflow User Interface

User interactionthrough the Portal

Guidancesfor the user

Workflow composition and execution visualizationUser’s decisions in crucial points of execution

Information onavailable

resources andtheir description

Execution of chosenGrid services

Informationabout workflow

execution

Analysed andextractedknowledge

Information aboutresources andenvironment

Information aboutperformance of

particular resources

Flow of ActionsFlow of Actions


Stages of Workflow ConstructionStages of Workflow Construction

Initial, abstract grid job

Abstract Workflow withService classes

Partially concretized Workflow prior to

execution

Fully concretized Workflow after

Successful execution


Workflow EnvironmentWorkflow Environment


Circulation of KnowledgeCirculation of Knowledge

Grid wf Exec System

Initial conditionsprovided by user

Abstract workflowmade by WCT

Concrete workflowmade by AAB

Running workflowScheduler

Service class functionality

Service instance properties

Service instance performance

Hints and guidelines

USER

Services and resourcesmonitoring data

On-line monitoring infrastructure

Events occurred duringwf composition and execution

Event publishing subsystem

Defined metrics

predefined forthe system

supplied by grid service providers


Knowledge Assimilation Agent

Performance analysis


Monitoring and Performance AnalysisMonitoring and Performance Analysis

• Monitoring and instrumentation service (MIS)Monitoring and instrumentation service (MIS)• Performance analysis service (PAS)Performance analysis service (PAS)• Data representations and service interfaces Data representations and service interfaces


Monitoring and Performance AnalysisMonitoring and Performance Analysis Monitoring and Instrumentation ServiceMonitoring and Instrumentation Service

• Instrument code regions, and activitiesInstrument code regions, and activities• Monitor infrastructure, code regions, activity execution statusMonitor infrastructure, code regions, activity execution status

Performance Analysis ServicePerformance Analysis Service• Define performance metrics for workflows Define performance metrics for workflows • Analyze monitoring data and relate the data to the workflowAnalyze monitoring data and relate the data to the workflow• Define performance properties and search performance Define performance properties and search performance

bottlenecks of workflowsbottlenecks of workflows

Data Presentations and Service InterfacesData Presentations and Service Interfaces• XML schemas for describing CPU usage,TCP bandwidth, generic XML schemas for describing CPU usage,TCP bandwidth, generic

events, profiling data, workflow activity execution status, etc. events, profiling data, workflow activity execution status, etc. • WIRL, PDQS, common service operations and specific service WIRL, PDQS, common service operations and specific service

operationsoperations

WP InterdependenciesWP Interdependencies• Scheduler and Grid Workflow Execution Service (GWES) - WP2Scheduler and Grid Workflow Execution Service (GWES) - WP2• Grid Organizational Memory - WP4Grid Organizational Memory - WP4• Knowledge Assimilation Agent - WP5 Knowledge Assimilation Agent - WP5


Ontologies in GOMOntologies in GOM

Some general concepts, Some general concepts, that could be shared by that could be shared by many domain ontologiesmany domain ontologies

Domain specific concepts, Domain specific concepts, water flowwater flow

Computational resource, Computational resource, visualisation resourcevisualisation resource

Rainfall measurement Rainfall measurement resourceresource

File metadataFile metadata

File replicasFile replicas

File formatsFile formats

Meaning of data Meaning of data

Data formatData format

Other flood samples time Other flood samples time perdiodperdiod

Average monitoring Average monitoring parametersparameters

Service class (e.g. FFT Service class (e.g. FFT solver)-Service inputssolver)-Service inputs;; Service outputsService outputs; ; Service Service preconditionspreconditions, effects, effects

Workflow patternsWorkflow patterns

Workflow historyWorkflow history

Domain specific Domain specific requirements – no parallelrequirements – no parallel

Data

Serv

ice

Ap

plicati

on

Work

flow

Ontology Scope

On

tolo

gy T

yp

e

Generic Specific

Resou

rces


Levels of KnowledgeLevels of Knowledge Knowledge is separated into three levels Knowledge is separated into three levels

• Generic – definitions of concepts, taxonomyGeneric – definitions of concepts, taxonomy• Domain specific – definition of domain specific topicsDomain specific – definition of domain specific topics• Data – individuals from a concrete applicationData – individuals from a concrete application

Knowledge is gathered in different registries Knowledge is gathered in different registries


Knowledge Knowledge BBase ase AArchitecturerchitecture

GOM Client

Grid Organisational Memory Service

Service Matchmaker ... Service Annotation ...

Knowledge Usage Knowledge Provision

Organisational Memory Interface

State Handler Event Handler

Security Model

Event ModelCurrent State Model

Persistence

Notifier

Ontology


Technologies and StandardsTechnologies and Standards

JavaJava

Web Service, SOAPWeb Service, SOAP

Tomcat, Maven, JIBXTomcat, Maven, JIBX

WSRF, Globus Toolkit 4WSRF, Globus Toolkit 4

XML, OWL, RDF, RDQLXML, OWL, RDF, RDQL

OWL-DL (Description logic)OWL-DL (Description logic)

JENA (Java Semantic Web Toolkit)JENA (Java Semantic Web Toolkit)


Users of K-WfGridUsers of K-WfGridUser communityUser community

Environment:Environment: Flood decision crisis team support system Flood decision crisis team support system

Business:Business: Enterprise resource planning Enterprise resource planning

Public sector:Public sector: Coordinated traffic management Coordinated traffic management

Developer communityDeveloper community

Grid software developers:Grid software developers: Workflow and knowledge management Workflow and knowledge management toolstools

Application developers:Application developers: Complex distributed application Complex distributed application constructionconstruction

Interested Institutions:

Municipality of Genova Slovak Water Research Institute, Bratislava Slovak Hydrometeorological Institute, Bratislava Slovak Watermanagement Enterprise, Banska Stiavnica


Project Proposal for F2 Call 5Project Proposal for F2 Call 5

GridSpaceGridSpace

Transparent Semantic Transparent Semantic GridGrid


Programming grid Programming grid applicationapplication

• Various Grid middleware platforms for uniform access to Various Grid middleware platforms for uniform access to resources – difficult, complex to programresources – difficult, complex to program

• RRecent initiatives ecent initiatives onon Grid programming do not address the Grid programming do not address the dynamic nature of the Griddynamic nature of the Grid

Important features of the GridImportant features of the Grid• Grid is dynamicGrid is dynamic• Resource users do not instantiate resources on their ownResource users do not instantiate resources on their own• RResources esources are in differentare in different administrative domains administrative domains

Therefore, a Grid application has to beTherefore, a Grid application has to be• More loosely coupled (combined of autonomous elements)More loosely coupled (combined of autonomous elements)• Flexible to overcome (and benefit from) Grid dynamic natureFlexible to overcome (and benefit from) Grid dynamic nature• Adaptable to cross boundaries of various administration Adaptable to cross boundaries of various administration

policiespolicies

MotivationMotivation


FeaturesFeatures• Abstract, semantically rich layer between a user and the Abstract, semantically rich layer between a user and the

middlewaremiddleware• Set of tools for Grid application developer to make the Grid Set of tools for Grid application developer to make the Grid

programming easierprogramming easier• Strong support for developing flexible and adaptable Strong support for developing flexible and adaptable

applicationsapplications

ComponentsComponents• Grid programming languageGrid programming language• Interpreter with dynamic Interpreter with dynamic ad-hocad-hoc binding capabilities binding capabilities• Runtime environment based on tuple space ideaRuntime environment based on tuple space idea• Evolving language library to share and reuse applicationsEvolving language library to share and reuse applications

Technology:Technology: components, services, objects components, services, objects

Concept of the Concept of the GridSpaceGridSpace


Separates the developer from Separates the developer from ever-changing Grid resource ever-changing Grid resource layerlayer

Seamlessly introduces Seamlessly introduces dynamism into newly created dynamism into newly created applicationsapplications

Provides unified access to Provides unified access to resources by means of resources by means of semantically described semantically described abstractionsabstractions

Supports evolving and well Supports evolving and well organized library of organized library of applications used up-to-dateapplications used up-to-date

Allows easy reuse of already Allows easy reuse of already built built applicationsapplications

GridSpace abstraction of data

abstraction of computations abstraction of components

ReplicaMngmnt Schedulers Resource

Brokers ...

Unified Component Access Layer

Abs

trac

tion

ComponentDeveloper

ApplicationDeveloper

GridUserU

sers

Mid

dlew

are

GRAM Condor LSF ...GridFTP

GridSpaceGridSpace - - Additional Abstract LayerAdditional Abstract Layer


Operation of GridSpaceOperation of GridSpace

GridSpace

Interpreter

Inactive Data Tuple

Inactive Component Tuple

A

B

Provides- com(x)- print(x)

Provides- y fun(x)- fun2(x)

input A, Bloop 10x call com(A)endloopC = call fun(B)output C

Puts Script to execute

Provides Component

May reuse Script as a new Component

Active ExecutionTuple

Provides Data for computation

Provides Data for community

Dynamic binding links


From Resource Abundance to Programmable GridFrom Resource Abundance to Programmable Grid

Similar semantic description of all the resourcesSimilar semantic description of all the resources• Using common notions: dependency, requirement, Using common notions: dependency, requirement,

capabilitycapability• Based on growing Semantic Web/Grid achievementsBased on growing Semantic Web/Grid achievements• With natural ability to be extended by multiple usersWith natural ability to be extended by multiple users

AdvantagesAdvantages• Helps build new applicationsHelps build new applications• Common language for various tools and platformsCommon language for various tools and platforms• Does not enforce unification of underlying technologyDoes not enforce unification of underlying technology

Global Grid Environment

Computingpower

Data

Software

Sensors,devices

Networktransfer

DatastoragespaceCommon

SemanticDescription

Layer

The environment provides everything for The environment provides everything for anan applicationapplication: v: vast space, multiplication ast space, multiplication of resources, multitude of access of resources, multitude of access standards and protocolsstandards and protocols

IIssuessuess

• Plenty of resources to build Plenty of resources to build sophisticated applicationsophisticated applications froms from

• Each new application requires huge Each new application requires huge effort to overcome integration effort to overcome integration problemsproblems

Data/events- Sources

- Retainers- Tranformers- Consumers

Global Grid EnvironmentComputingpower

Data

Software

Sensors,devices

Networktransfer

Datastoragespace


Transparent Transparent Global GridGlobal Grid for Everyday Use for Everyday Use

The applications The applications should beshould be buil builtt/rebuil/rebuiltt almost at runtime almost at runtime Abstraction and transparencyAbstraction and transparency

• Hierarchy of resources enables customisable abstraction Hierarchy of resources enables customisable abstraction levellevel

• No more No more jjob submission black boxesob submission black boxes – control over running – control over running application will be enabled application will be enabled

• AA level of transparency helps users understand level of transparency helps users understand the the GridGrid The path The path to our main objective: plugto our main objective: pluggingging problems into the Grid problems into the Grid

to have them solveto have them solved!d!

Global Grid Environment


ContactContact

www.cyfronet.krakow.pl/en/www.cyfronet.krakow.pl/en/

[email protected]@agh.edu.pl

1st grid@asia workshop, beijing, china, june 21 – 23, 2005 1 marian bubak institute of computer...

Documents

efficient grid

asia workshop

grid data transfer patient

interactive applications

storage distributed

input data

crossgrid tools

ann application