the globus project infrastructure for computational grids

The Globus Project

Infrastructure for Computational Grids

The Globus Project Teamhttp://www.globus.org/

2SIAC 2000, Wright State University, August 21, 2000

Session Goals• Provide an introduction to…

computational grids the capabilities of the Globus Toolkit pragmatic issues with grids & Globus

• Enable attendees to… start building grids using the Globus Toolkit start building & using grid applications


Overview• Introduction to computational grids• Introduction to the Globus Toolkit

Portability Security Information services Resource management Data management Communication

• Case studies• Other Globus services, and future directions


What is a computational grid?• A pool of computational resources that can

be “plugged into” via standard interfaces.• Processors• Data storage devices• Instruments

• As the power grid is to electrical power, and the telephone grid is to voice communication, so will the computational grid be for computation.


Computational Collaboration• 1975-1995: Collaboration

We work together, but use our own unique systems. It’s hard to share data, computing power, instrumentation.

• 1995-2005: Virtual Organizations We build systems that combine our resources with those of our

collaborators. We learn how to manage the heterogeneity of systems,

management, and users.• 2005-?: Computational Grid

Computation is a commodity, that can be bought and sold by anyone.

Computational services use standard interfaces. Organizations and individuals typically don’t need to build their

own computing infrastructures.


Why do we need the Grid?• The Grid will enable applications that include

people, computers, databases, instruments, etc. Online instruments Collaborative engineering Parameter studies Browsing of remote datasets Use of remote software Data-intensive computing Very large-scale simulation


tomographic reconstruction

real-timecollection

wide-areadissemination

desktop & VR clients with shared controls

Advanced Photon Source

Online Instruments

archival storage

DOE X-ray source grand challenge: ANL, USC/ISI, NIST, U.Chicago


Collaborative Engineering

CAVERNsoft: UIC, Electronic Visualization Laboratory

• Manipulate shared virtual space, with Simulation components Multiple flows: Control, Text,

Video, Audio, Database, Simulation, Tracking, Haptics, Rendering

• Issues: (un)reliable uni/multicast Security Reservation & QoS


Control Text Video Audio Database

Tele-immersion

“5 Gflop/sec, flowspecs, design db” Multiple access modalities

Multiple flows Simulation Tracking Haptics Rendering

Leigh et al: UIC, Electronic Visualization Laboratory


Distributed Supercomputing

• Issues: Resource discovery, scheduling Configuration Multiple comm methods Message passing (MPI) Scalability Fault tolerance

NCSAOrigin

CaltechExemplar

ArgonneSP

MauiSP

SF-Express Distributed Interactive Simulation: Caltech, USC/ISI


High-Throughput Computing

Nimrod-G: Monash University

CostDeadline

AvailableMachines

• Schedule many independent tasks Parameter studies Data analysis

• Issues: Resource discovery Data Access Scheduling Reservatation Security Accounting Code management


• Examples: Problem solving env. for

computational chemistry Application web portals

• Issues: Remote job submission,

monitoring, and control Resource discovery Distributed data archive Security Accounting

Problem Solving Environments

ECCE’: Pacific Northwest National Laboratory


The Grid“Dependable, consistent,

pervasive access to[high-end] resources”

• Dependable: Can provide performance and functionality guarantees

• Consistent: Uniform interfaces to a wide variety of resources

• Pervasive: Ability to “plug in” from anywhere


Early Steps Toward the Grid• Metacomputing: late 80s

Focus on distributed computation• Gigabit testbeds: early 90s

Research, primarily on networking• I-WAY: 1995

Demonstration of application feasibility• NFS PACIs (National Technology Grid): 1998• NASA Information Power Grid: 1999• DOE ASCI DISCOM DRM: 1999• European Grid: 2000


Technical Challenges• Complex application structures that combine

aspects of parallel, multimedia, distributed, collaborative computing

• Resource characteristics that vary dynamically in both time and space.

• Requirements for guaranteed high “end to end” performance, despite heterogeneity and lack of global control.

• Desire to retain local policies for security, fees, usage restrictions, management, and technical standards.


Domain 2

Domain 1

•Authenticate once•Specify simulation (code, resources, etc.)•Locate resources•Negotiate authorization, acceptable use, etc.•Acquire resources•Initiate computation•Steer computation•Access remote datasets•Collaborate on results•Account for usage

Issues


Architectural Approaches• Distributed systems: DCE, CORBA, Jini, etc.

Rich functionality eases app development Complexity hinders deployment

• especially in absence of global control Performance difficulties

• Internet/Web Protocols and Tools Simple protocols facilitate deployment Missing functionality hinders app development Performance difficulties


Standards & Commodity Tech• Where appropriate, exploit standards and

commodity technology in core infrastructure LDAP, SSL/TLS, X.509, GSS-API, HTTP, FTP,

XML, SOAP, etc. Provides leverage

• Interface with other common standards CORBA, Java/Jini, DCOM, Web, etc While our core infrastructure may not be built

on one of these distributed architectures, we can and must cleanly interface with them


The Globus Project• Basic research in grid-related technologies

Resource & data management, security, QoS, policy, communication, adaptation, etc.

• Development of the Globus Toolkit Core services for grid-enabled tools & applications

• Construction of production grids & testbeds Environments in which grid software can be deployed

and experiments can be run.• Experimentation with real grid applications

Verifying that the grid works and is useful.


Grid Services ArchitectureApplications

Grid ServicesSecurity

Data Management Fault Detection

Information Services

Resource Management

Grid FabricData Transport

Schedulers

Control Interfaces

Operating Systems

Application ToolkitsData Intensive Remote VisualizationCollab Design

High Throughput Remote Control

Portability

Instrumentation QoS Services

High Energy Physics Data Analysis Climate StudiesCollab Engineering

Online Instrumentation

Message Passing


Globus Project Participants• Globus Project is a large community effort

Globus Toolkit core development• Argonne, USC/ISI, NCSA, SDSC

Globus Toolkit contributors• NASA, DOE ASCI DRM (SNL, LBNL, LLNL), Raytheon, and

numerous others Collaborators

• University, lab, industrial, and international partners spanning many scientific and engineering disciplines

• Active in Grid Forum http://www.gridforum.org


Globus Approach• A toolkit and collection of services addressing

key technical problems Modular “bag of services” model Not a vertically integrated solution General infrastructure tools (aka middleware)

that can be applied to many application domains• Inter-domain issues, rather than clustering

Integration of intra-domain solutions• Distinguish between local and global services


Globus Hourglass

• Focus on architecture issues Propose set of core services

as basic infrastructure Use to construct high-level,

domain-specific solutions• Design principles

Keep participation cost low Enable local control Support for adaptation “IP hourglass” model

Diverse global services

Core Globusservices

Local OS

A p p l i c a t i o n s


Technical Focus & Approach• Enable incremental development of grid-enabled

tools and applications Model neutral: Support many programming models,

languages, tools, and applications Evolve in response to user requirements

• Deploy toolkit on international-scale production grids and testbeds Large-scale application development & testing

• Information-rich environment Basis for configuration and adaptation


Layered ArchitectureApplications

Grid ServicesGRAM

GSI HBM

Nexus

globus_io

Grid FabricLSF

Condor MPI

NQEPBS

TCP

NTLinux

UDP

Application ToolkitsDUROC globusrunMPICH-G Nimrod/GCondor-G HPC++

GlobusView Web Portals

GASS

Solaris DiffServ

GSI-FTPMDS


Globus Toolkit Grid Services• Security (GSI)• Information services (MDS)• Resource management (GRAM)• Data management (GASS, GSI-FTP, replicas)• Communication (globus_io, Nexus) • Fault detection (HBM)• Portability (globus_dc, globus_thread)


Other Globus Project Grid Services• Coming Soon

Data transfer (GSI-FTP) Replica Management

http://www.globus.org/datagrid

• Experimental Prototypes Advanced Reservations & QoS (GARA) Distributed Events & Logging


Sample of High-Level Services• Resource brokers and co-allocators

DUROC, HTB, Nimrod/G, Condor-G, ASCI DRM• Communication & I/O libraries

MPICH-G, PAWS, RIO (MPI-IO), PPFS, MOL• Parallel languages

HPC++, CC++• Collaborative environments

CAVERNsoft, ManyWorlds• Others

MetaNEOS, NetSolve, LSA, AutoPilot, WebFlow


Condor-G: Condor for the Grid• Condor is a high-throughput scheduler• Condor-G uses Globus Toolkit libraries for:

Security (GSI) Managing remote jobs on Grid (GRAM) File staging & remote I/O (GSI-FTP)

• Grid job management interface & scheduling Robust replacement for Globus Toolkit programs

• Globus Toolkit focus is on libraries and services, not end user vertical solutions

Supports single or high-throughput apps on Grid• Personal job manager which can exploit Grid resources


Production Grids & Testbeds• Production deployments underway at:

NSF PACIs (National Technology Grid) NASA Information Power Grid DOE ASCI European Grid

• Research testbeds EMERGE: Advance reservation & QoS GUSTO: Globus Ubiquitous Supercomputing Testbed

Organization Particle Physics Data Grid Earth Systems Grid


Production Grids & Testbeds

NASA’s Information Power Grid The Alliance National Technology Grid

GUSTO Testbed


Application Experiments• Computed microtomography (ANL, ISI)

Real-time, collaborative analysis of data from X-Ray source (and electron microscope)

• Hydrology (ISI, UMD, UT; also NCSA, Wisc.) Interactive modeling and data analysis

• Collaborative engineering (“tele-immersion”) CAVERNsoft @ EVL

• OVERFLOW (NASA) Large CFD simulations for aerospace vehicles


Application Experiments• Distributed interactive simulation (CIT, ISI)

Record-setting SF-Express simulation• Cactus

Astrophysics simulation, viz, and steering Including trans-Atlantic experiments

• Particle Physics Data Grid High Energy Physics distributed data analysis

• Earth Systems Grid Climate modeling data management


Where Are We? (August 2000)• Research is focused on data management,

resource management, and web portals.• Globus Toolkit v1.1.3 has been released.

Runs on most versions of Unix, Win32 clients.• Production deployment is underway.

NSF PACIs, NASA IPG, DOE ASCI DRM• Many research applications and tools are

using these testbeds.• We’re always looking for interesting

applications.


For More Information on Globushttp://www.globus.org/

• Papers on most components• Tutorials

User, Developer, Administrator• Manuals

Quick Start Guide, System Administration Guide• Mailing lists

[email protected], [email protected]• Software & API documentation• Application descriptions• Attend Supercomputing 2000 (Nov. 2000)


The Grid:Blueprint for a New Computing Infrastructure

I. Foster, C. Kesselman (Eds), Morgan Kaufmann, 1999• Available July 1998; ISBN 1-55860-475-8 • 22 chapters by expert

authors including Andrew Chien, Jack Dongarra, Tom DeFanti, Andrew Grimshaw, Roch Guerin, Ken Kennedy, Paul Messina, Cliff Neuman, Jon Postel, Larry Smarr, Rick Stevens, and many others

http://www.mkp.com/grids

“A source book for the historyof the future” -- Vint Cerf


Session Approach• Five sections, each illustrating a basic

Globus service• Laboratory material is available to allow

practice with the use of each technique See http://www.globus.org/tutorial/


Desktop Supercomputing• Seamlessly, from the desktop

Sign-on once Locate available computers Start computation on an appropriate

system Monitor progress Get output files Manipulate locally

• E.g. ECCE’, Cactus, Hotpage, Chemical Eng. Workbench, WebFlow, LSA


WebFlow Grid Interface • Dataflow computing

interface to grid computing Fox, Haupt: Syracuse

• Globus services for Authentication Process creation and

management• Applications include

nanomaterials


Application Challenges• Security

How do we authenticate ourselves at the remote site?

• Resource specification How do we locate and request a resource?

• Staging of code and data How do we stage a user’s executables and data to

the remote resource?• Computation

How do we start & manage computation?


Grid Services• Single sign-on for all resources

No need for user to keep track of accounts and passwords at multiple sites

No plaintext passwords• Uniform interface to various local scheduling

mechanisms PBS, Condor, LSF, NQE, LoadLeveler, fork, etc. No need to learn and remember obscure

command sequences at different sites• Support for file staging, remote I/O, etc.


Grid Authentication Model• Authentication is done on a “user” basis

Single authentication step allows access to all grid resources

• No communication of plaintext passwords• Most sites will use conventional account

mechanisms You must have an account on a resource to use that

resource• Sites may use “generic” Grid accounts

Not common, but Globus can deal with it


Grid Security Infrastructure (GSI)• Based on public key technology

Standard X.509 certificate, same as certificates used for the Web

• Each user has: a Grid user id (called a Subject Name) a private key (like a password) a certificate signed by a Certificate Authority

(CA)• A “gridmap” file at each site specifies

grid-id to local-id mapping


Certificate Based Authentication• User has a certificate, signed by a trusted “certificate

authority” (CA) Certificate contains users name and public key Globus project operates a CA

• User’s private key is used to encode a challenge string

• Public key is used to decode the challenge If you can decode it, you know the user

• Treat your private key carefully!! Private key is stored in encrypted form


User Proxies• Minimize exposure of user’s private key• A temporary credential for use by our

computations We call this a user proxy certificate Allows process to act on behalf of user User-signed user proxy certificate stored in

local file• Proxy’s private key is not encrypted

Rely on file system security, proxy certificate file must be readable only by the owner


Delegation• Remote creation of a user proxy• Allows remote process to act on behalf of

the user• Avoids sending passwords or private keys

across the network


Single sign-onvia “grid-id”

User

User Proxy

GlobusGlobusCredentialCredential

Site 1

Kerberos

GRAM Process

Process

ProcessGSI

TicketTicket

Site 2

Public Key

GRAM

GSI

CertificateCertificate

Process

Process

Process

Authenticatedinterprocess

communication

CREDENTIAL

GSSAPI:multiplelow-level

mechanisms

Mutualuser-resourceauthentication

Mappingto local ids

Assignment of credentials to“user proxies”


Globus Authentication Setup• Before you can run Globus applications:

Install Globus Obtain a Grid certificate and key Set up your environment so Globus knows where to

find certificates and keys Contact sites to set up local accounts and globusmap

entries Create proxy certificate for each application run

• Documentation Globus Quick Start Guide (on website)


Your New CertificateCertificate: Data: Version: 3 (0x2) Serial Number: 28 (0x1c) Signature Algorithm: md5WithRSAEncryption Issuer: C=US, O=Globus, CN=Globus Certification Authority Validity Not Before: Apr 22 19:21:50 1998 GMT Not After : Apr 22 19:21:50 1999 GMT Subject: C=US, O=Globus, O=NACI, OU=SDSC, CN=Richard Frost Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (1024 bit) Modulus (1024 bit): 00:bf:4c:9b:ae:51:e5:ad:ac:54:4f:12:52:3a:69: <snip> b4:e1:54:e7:87:57:b7:d0:61 Exponent: 65537 (0x10001)Signature Algorithm: md5WithRSAEncryption 59:86:6e:df:dd:94:5d:26:f5:23:c1:89:83:8e:3c:97:fc:d8: <snip> 8d:cd:7c:7e:49:68:15:7e:5f:24:23:54:ca:a2:27:f1:35:17:

NTP is highlyrecommended


“Logging on” to the Grid• To run programs, authenticate to Globus:

% grid-proxy-initEnter PEM pass phrase: ******

• Creates a temporary, short-lived credential for use by our computationsPrivate key is not exposed past grid-proxy-init

• Options for grid-proxy-init:-hours <lifetime of credential>-bits <length of key>-help


Proxy Information• To get proxy information run grid-proxy-info

% grid-proxy-info -subject/C=US/O=Globus/O=ANL/OU=MCS/CN=Ian Foster

• Options for printing proxy information-subject -issuer-type -timeleft-strength -help

• Options for scripting proxy queries-exists -hours <lifetime of credential>-exists -bits <length of key> Returns 0 status for true, 1 for false:


Sample Gridmap File

# Distinguished name Local# username"/C=US/O=Globus/O=NPACI/OU=SDSC/CN=Rich Gallup” rpg"/C=US/O=Globus/O=NPACI/OU=SDSC/CN=Richard Frost” frost"/C=US/O=Globus/O=USC/OU=ISI/CN=Carl Kesselman” u14543"/C=US/O=Globus/O=ANL/OU=MCS/CN=Ian Foster” itf

• Gridmap file maintained by Globus administrator

• Entry maps Grid-id into local user name(s)


Remote Startup Mechanism

key

cert

gatekeeperclient

1. Exchange certificates, authenticate, delegate

2. Check gridmap file3. Lookup service4. Run service program

(e.g. jobmanager)

jobmanager

key

cert

1.

2.

map

4.

services3.


Simple job submission• globus-job-run provides a simple RSH

compatible interface% grid-proxy-init Enter PEM pass phrase: *****% globus-job-run host program [args]

• Job submission will be covered in more detail in Part 5


Exercise: Sign-On & Remote Process Creation

• Use grid-proxy-init to create a proxy certificate:

% grid-proxy-initEnter PEM pass phrase:......................................+++++.....+++++

• Use grid-proxy-info to query proxy:% grid-proxy-info -subject

• Use globus-job-run to start remote programs:

% globus-job-run jupiter.isi.edu /usr/bin/ls -l /tmp


Globus Components Being Used• GSI: Grid Security Infrastructure

Authenticate to remote system• GRAM: Globus Resource Allocation Manager

Create process on remote resource, deal with local resource managers

• GASS: Global Access to Secondary Storage Redirect standard output

(More on GRAM and GASS later!)


Summary• Grid Security Infrastructure (GSI) provides

single sign-on capability• globus-job-run can be used to create a

remote process Difference between schedulers managed by

Globus Strong authentication provided

• Remote process creation can be added to applications by using Globus services


Grid Information Services• System information is critical to operation

of the grid and construction of applications How does an application determine what

resources are available? What is the “state” of the computational

grid? How can we optimize an application based

on configuration of the underlying system?• We need a general information

infrastructure to answer these questions


Using Information forResource Brokering

“10 GFlops, EOS data,20 Mb/sec -- for 20 mins”

MetacomputingDirectoryService

GRAMGRAMGRAM

ResourceBroker

Info service:location + selection

Globus ResourceAllocation Managers

GRAM

ForkLSFEASYLLCondoretc.

“What computers?”“What speed?”“When available?”

“50 processors + storage from 10:20 to 10:40 pm”

“20 Mb/sec”


Examples of Useful Information• Characteristics of a compute resource

IP address, software available, system administrator, networks connected to, OS version, load

• Characteristics of a network Bandwidth and latency, protocols, logical

topology• Characteristics of the Globus infrastructure

Hosts, resource managers


Grid Information Service• Provide access to static and dynamic

information regarding system components• A basis for configuration and adaptation in

heterogeneous, dynamic environments• Requirements and characteristics

Uniform, flexible access to information Scalable, efficient access to dynamic data Access to multiple information sources Decentralized maintenance


The Globus Toolkit’sMetacomputing Directory Service (MDS)

• Information is maintained in a distributed directory. Information is accessed via a standard protocol Information is distributed across many servers Each server can be optimized for particular functions

• Information comes from many sources. Information providers and tools Applications (i.e., users) Resources that generate info on demand

• Information is available everywhere.


MDS Features• White Pages

Look up the IP number, amount of memory, etc., associated with a particular machine

• Yellow Pages Search for computers of a particular class or with a

particular property • Information is dynamic!

In a distributed system, things change without warning. Information often has an expiration date or other

measures of uncertainty.


MDS Approach• Based on LDAP

Lightweight Directory Access Protocol v3 (LDAPv3)

Standard data model Standard query protocol

• Globus specific schema Host-centric representation

• Globus specific tools GRIS, GIIS Data discovery, publication,… SNMP

GRIS

NIS

NWS

LDAP

LDAP API

Middleware

…

Application

GIIS…


LDAP Details

• Lightweight Directory Access Protocol Stripped down version of X.500 DAP protocol Supports distributed storage/access (referrals) Supports replication Becoming de facto standard

• Defines: Network protocol for accessing directory contents Information model defining form of information Namespace defining how information is

referenced and organized


LDAP Directory Structure• Directory contents

Called Object Classes and Entries What information is stored in directory Group related information into entries

• Directory organization Called Directory Information Tree (DIT) Objects are organized into tree structure Position of object in tree uniquely names

entry within the server


Sample Object Classes• Compute Resources

Operating System Memory Hierarchy Health and Status

• Network Interfaces IP address Interface types

• Performance Data Schedule Jobs CPU Loads Network Traffic

• Resource Managers Contact strings Scheduled jobs Free nodes

• Software Configuration Version Control Contact information

• Organizations• People


LDAP Directory Information Tree• Directory entries are organized into a tree.

Called Directory Information Tree (DIT) Subtrees can be distributed or replicated.

• Position in tree uniquely names entry within a server.

• Each object in a server is uniquely determined by its distinguished name (DN). List of unique attribute names and values

along path from root of DIT to object, e.g.:<hn=sp2.sdsc.edu, dc=sdsc, dc=edu, o=Grid>


Global Namespace• A DN uniquely names an entry within an LDAP

server.• The host and port uniquely names an LDAP

server.• An LDAP URL is the combination of these.

ldap://<host>:<port>/<DN>• LDAP URL can be used as the global name of

an entry.


MDS Components• Uses standard LDAP servers

OpenLDAP, Netscape, Oracle, etc• Tools for populating & maintaining MDS

Integrated with Globus Toolkit server release, not of concern to most Globus users

Discover/update static and dynamic info• APIs for accessing & updating MDS contents

C, Java, PERL (LDAP API, JNDI)• Various tools for manipulating MDS contents

Command line tools, Shell scripts & GUIs


Three Classes Of MDS Servers• Grid Resource Information Service (GRIS)

Supplies information about a specific resource Mostly “white pages” lookups

• Grid Index Information Service (GIIS) Supplies collection of information which was gathered

from multiple GRIS servers Supports efficient queries against information which is

spread across multiple GRIS servers Mostly “yellow pages” lookups

• Referral Service Links together GRIS and/or GIIS servers


Grid Resource Information Service• GRIS is an LDAP server which runs on each resource.

GRIS runs on a well known port (2135).• GRIS provides resource-specific information.

Load, process information, storage information, etc. GRIS gathers this information on demand with a TTL

• GRIS supports white & yellow pages queries. Configuration settings Capabilities (queues, extra equipment, etc.)

• GRIS can be configured to register with a GIIS on startup, deregister on shutdown.


Grid Index Information Service• GIIS describes a class of servers.

Gathers information from multiple GRIS servers. Each GIIS is optimized for particular queries.

• Ex1: Which Alliance machines are >16 process SGIs?• Ex2: Which Alliance storage servers have >100Mbps bandwidth to

host X? GIIS is similar to web search engines.

• GIIS can be used to inventory an organization’s resources (“organizational information service”). The Globus Toolkit includes a GIIS. Caches GRIS info with long update frequency.

• Useful for queries across an organization that rely on relatively static information. (Ex1 above.)


Referral Service• Links together multiple GRIS and/or GIIS servers

into a single LDAP namespace• Referral servers contain no actual content• Current limitations

Requires LDAPv3 referrals Some LDAP clients to not have good support for

referrals But this is improving…

• Can be implemented as a GIIS or as a commercial LDAP server.


Pulling Together a DIT• In order for a GRIS’s content to appear in a GIIS, the

GIIS must first find it. GRISes can register with GIIS during startup. GIIS can be configured with GRISes’ hostnames. GIIS can walk a referral tree to find GRISes.

• MDS can support any of these approaches. The “right” approach depends on the organization’s

requirements. Can use a combinations of these approaches. This is the subject of ongoing research.


Finding a GIIS• How does a user or application find a GIIS server?

Options include: Runs on well-known host/port Use DNS server records

• Returns host/port of server for a particular domain Part of referral tree

• Recursive: How do you find the referral service? Indexed by another GIIS

• Recursive: How do you find the referral service?• This is also the subject of ongoing research.


Server Registration• A GRIS or GIIS server can be configured to

(de-)register itself during startup/shutdown Basic capability that is useful in many scenarios

• Support for multiple registration protocols Can register with a GIIS using custom protocol Can dynamically add itself to a referral tree

using LDAPv3 referral protocol• Allows for federations of information servers

E.g. Argonne GRIS can register with both Alliance and DOE GIIS servers


MDS Tools• Java LDAP browser

http://www.mcs.anl.gov/~gawor/ldap• Web-based browsers and displays

http://www.globus.org/mds• CGI-based MDS browser• MDS Object Class Browser

• Various APIs and search tools • Translators from “Globus Object Definition

Language” Commented LDIF - LDAP schema definition Converts to LDIF, HTML


MDS Access/Update Commands• LDAP defines a set of standard commands

ldapsearch, ldapmodify, ldapdelete, etc.• We also define MDS-specific commands

grid-info-search, grid-info-create, grid-info-update, grid-info-remove, grid-info-host-search

Routines to ensure data consistency and to insert metadata

• APIs are defined for C, Java, etc. C: OpenLDAP client API

• ldap_search_s(), ldap_modify_s(), … Java: JNDI


Searching an LDAP Databasegrid-info-search [options] filter [attributes]

• Default grid-info-search options-h mds.globus.org MDS server-p 389 MDS port-b “o=Grid” search start point-T 30 LDAP query timeout-s sub scope = subtree

alternatives: base : lookup this entry one : lookup immediate children


Searching a GRIS Servergrid-info-host-search [options] filter [attributes]

• Exactly like grid-info-search, except defaults:-h localhost GRIS server-p 2135 GRIS port

• Example:grid-info-host-search –h pitcairn “dn=*” dn


Filtering• Filters allow selection of object based on

relational operators (=, ~=,<=, >=) grid-info-search “cputype=*”

• Compound filters can be construct with Boolean operations: (&, |, !) grid-info-search “(&(cputype=*)(cpuload1<=1.0))” grid-info-search “(&(hn~=sdsc.edu)(latency<=10))”

• Hints: white space is significant use -L for LDIF format

required


Example: Filtering

% grid-info-host-search -L “(objectclass=GlobusSoftware)”

dn: sw=Globus, hn=pitcairn.mcs.anl.gov, dc=mcs, dc=anl, dc=gov, o=Gridobjectclass: GlobusSoftwarereleasedate: 2000/04/11 19:48:29releasemajor: 1releaseminor: 1releasepatch: 3releasebeta: 11lastupdate: Sun Apr 30 19:28:19 GMT 2000objectname: sw=Globus, hn=pitcairn.mcs.anl.gov, dc=mcs, dc=anl, dc=gov, o=Grid


Example: Attribute Selection% grid-info-host- search -L “(objectclass=*)” dn hn

Returns the distinguished name (dn) and hostname (hn) of all objects

Objects without hn fields are still listed DNs are always listed

dn: sw=Globus, hn=pitcairn.mcs.anl.gov, dc=mcs, dc=anl, dc=gov, o=Grid

dn: hn=pitcairn.mcs.anl.gov, dc=mcs, dc=anl, dc=gov, o=Gridhn: pitcairn.mcs.anl.gov

dn: service=jobmanager, hn=pitcairn.mcs.anl.gov, dc=mcs, dc=anl, dc=gov, o=Gridhn: pitcairn.mcs.anl.gov

dn: queue=default, service=jobmanager, hn=pitcairn.mcs.anl.gov, dc=mcs, dc=anl, dc=gov, o=Grid


Example: Discovering CPU Load• Retrieve CPU load fields of compute resources% grid-info-search -L “(objectclass=GlobusComputeResource)” \ dn cpuload1 cpuload5 cpuload15 dn: hn=lemon.mcs.anl.gov, ou=MCS, o=Argonne National Laboratory, o=Globus, c=UScpuload1: 0.48cpuload5: 0.20cpuload15: 0.03 dn: hn=tuva.mcs.anl.gov, ou=MCS, o=Argonne National Laboratory, o=Globus, c=UScpuload1: 3.11cpuload5: 2.64cpuload15: 2.57


MDS and Security• Authentication is required to perform certain

operations (e.g., write operations)• Each site has a Directory Manager

cn=Directory Manager, o=Organization, c=US• Users registered with the MDS have a DN

cn=Jane Doe, o=Organization, c=US% grid-info-search -D “cn=Jane Doe, o=Organization, c=US” \

-w <passwd>• GSI based authentication coming soon

Working prototype of GSI with OpenLDAP v2


Summary• MDS provides the information needed to

perform dynamic resource discovery and configuration. This is a critical component of resource

brokers!• MDS is based on existing directory service

standards (LDAPv3).• Although MDS is based on the LDAPv3

protocol, we are not tied to commercial LDAP servers.


SF-ExpressDistributed Interactive Simulation

P. Messina et al., Caltech

• Developed at Caltech• 100K vehicles (2002 goal) using 13

computers, 1386 nodes, 9 sites• Globus mechanisms for

Resource allocation Distributed startup I/O and configuration Fault detection

NCSAOrigin Caltech

Exemplar

CEWESSP

MauiSP


Grid Services• Resource Specification: the power to express the

required resources and constraints• Resource Co-allocation: using resources together

simultaneously• Executable staging, remote data access and I/O

streaming (more on these later)• These services should be integrated with higher-

level tools MPICH-G, globus-job-*, Condor, PBS, GRD, etc.


Globus Components in Actionglobus-job-run

jobmanager

fork

P1 P2

gatekeeper

jobmanager

LSF

P1 P2

gatekeeper

jobmanager

LoadLeveler

P1 P2

gatekeeperGRAM


Resource Management• Resource Specification Language (RSL) is

used to communicate requirements • The Globus Resource Allocation Manager

(GRAM) API allows programs to be started on remote resources, despite local heterogeneity

• A layered architecture allows application-specific resource brokers and co-allocators to be defined in terms of GRAM services


GRAM GRAM GRAMLSF EASY-LL NQE

Application

RSL

RSL

Information Service

Localresourcemanagers

Broker

Co-allocator

Building a Resource Broker

RSL

LDAP

RSLRSLStatus

Globuscomponents


Resource Specification Language• Common notation for exchange of

information between components Syntax similar to MDS/LDAP filters

• RSL provides two types of information: Resource requirements: Machine type,

number of nodes, memory, etc. Job configuration: Directory, executable,

args, environment• API provided for manipulating RSL


RSL Syntax• Elementary form: parenthesis clauses

(attribute op value [ value … ] )• Operators Supported:

<, <=, =, >=, > , != • Some supported attributes:

executable, arguments, environment, stdin, stdout, stderr, resourceManagerContact,resourceManagerName

• Unknown attributes are passed through May be handled by subsequent tools


Constraints: “&”• For example:

& (count>=5) (count<=10) (max_time=240) (memory>=64) (executable=myprog)

• “Create 5-10 instances of myprog, each on a machine with at least 64 MB memory that is available to me for 4 hours”


Disjunction: “|”• For example:

& (executable=myprog) ( | (&(count=5)(memory>=64)) (&(count=10)(memory>=32)))

• Create 5 instances of myprog on a machine that has at least 64MB of memory, or 10 instances on a machine with at least 32MB of memory


GRAM Components

Globus SecurityInfrastructure

Job Manager

GRAM client API calls to request resource allocation

and process creation.

MDS client API callsto locate resources

Query current statusof resource

Create

RSL Library

Parse

Request Allocate &create processes

Process

Process

Process

Monitor &control

Site boundary

Client MDS: Grid Index Info Server

Gatekeeper

MDS: Grid Resource Info Server

Local Resource Manager

MDS client API callsto get resource info

GRAM client API statechange callbacks


Multirequest: “+”• A multirequest allows us to specify multiple

resource needs, for example+ (& (count=5)(memory>=64) (executable=p1)) (&(network=atm) (executable=p2)) Execute 5 instances of p1 on a machine with at least

64M of memory Execute p2 on a machine with an ATM connection

• Multirequests are central to co-allocation


Co-allocation• Simultaneous allocation of a resource set

Handled via optimistic co-allocation based on free nodes or queue prediction

In the future, advance reservations will also be supported

• globusrun and globus-job-* will co-allocate specific multi-requests Uses a Globus component called the Dynamically

Updated Request OnlineCo-allocator (DUROC)


A Co-allocation Multirequest+( & (resourceManagerContact= “flash.isi.edu:754:/C=US/…/CN=flash.isi.edu-fork”) (count=1) (label="subjob A") (executable= my_app1) ) ( & (resourceManagerContact= “sp139.sdsc.edu:8711:/C=US/…/CN=sp097.sdsc.edu-lsf") (count=2) (label="subjob B") (executable=my_app2) )

Different executables

Differentcounts

Different resourcemanagers


Job Submission Interfaces• Globus Toolkit includes several command

line programs for job submission globus-job-run: Interactive jobs globus-job-submit: Batch/offline jobs globusrun: Flexible scripting infrastructure

• Others are building better interfaces General purpose

• Condor-G, PBS, GRD, Hotpage, etc Application specific

• ECCE’, Cactus, Web portals


globus-job-run• For running of interactive jobs• Additional functionality beyond rsh

Ex: Run 2 process job w/ executable stagingglobus-job-run -: host –np 2 –s myprog arg1 arg2

Ex: Run 5 processes across 2 hostsglobus-job-run \

-: host1 –np 2 –s myprog.linux arg1 \-: host2 –np 3 –s myprog.aix arg2

For list of arguments run:globus-job-run -help


globus-job-submit• For running of batch/offline jobs

globus-job-submit Submit job• Same interface as globus-job-run• Returns immediately

globus-job-status Check job status globus-job-cancel Cancel job globus-job-get-output Get job stdout/err globus-job-clean Cleanup after job


globusrun• Flexible job submission for scripting

Uses an RSL string to specify job request Contains an embedded globus-gass-server

• Defines GASS URL prefix in RSL substitution variable:(stdout=$(GLOBUSRUN_GASS_URL)/stdout)

Supports both interactive and offline jobs• Complex to use

Must write RSL by hand Must understand its esoteric features Generally you should use globus-job-* commands

instead


Brokering via Lowering• Resource location by refining a RSL

expression (RSL lowering):(MFLOPS=1000)

(& (arch=sp2)(count=200)) (+ (& (arch=sp2) (count=120)

(resourceManagerContact=anlsp2)) (& (arch=sp2) (count=80) (resourceManagerContact=uhsp2)))


SF-ExpressDistributed Interactive Simulation

P. Messina et al., Caltech

• Developed at Caltech• 100K vehicles (2002 goal) using 13

computers, 1386 nodes, 9 sites• Globus mechanisms for

Resource allocation Distributed startup I/O and configuration Fault detection

NCSAOrigin Caltech

Exemplar

CEWESSP

MauiSP


Grid Services• Resource Specification and Resource

Co-allocation (GRAM)• Executable Staging• Remote Data Access• These services should be integrated with higher-

level tools globus-job-*, Condor, etc.


What is GASS?

1. RSL extensions URLs used to name executables, stdout, stderr

2. A file access API Replace open/close with globus_gass_open/close;

read/write calls can then proceed directly3. Remote cache management utility


GASS Architecture

Cache

GASS Server

HTTP Server

FTP Server

% globus-gass-cacheRemote cache management

GRAM

GASS file access API

&(executable=https://…)RSL extensionsmain( ) {

fd = globus_gass_open(…) … read(fd,…) … globus_gass_close(fd)}

1

23


GASS File Naming• URL encoding of resource names

https://quad.mcs.anl.gov:9991/~bester/myjob

protocol server address file name• Other examples

https://pitcairn.mcs.anl.gov/tmp/input_dataset.1https://pitcairn.mcs.anl.gov:2222/./output_datahttp://www.globus.org/~bester/input_dataset.2

• Currently supports http & https• Next release will also support ftp & gsiftp.


GASS RSL Extensions• executable, stdin, stdout, stderr can be

local files or URLs• executable and stdin loaded into local

cache before job begins (on front-end node)

• stdout, stderr handled via GASS append mode

• Cache cleaned after job completes


GASS/RSL Example

&(executable=https://quad:1234/~/myexe) (stdin=https://quad:1234/~/myin) (stdout=/home/bester/output) (stderr=https://quad:1234/dev/stdout)


Example GASS Applications• On-demand, transparent loading of data

sets• Caching of data sets• Automatic staging of code and data to

remote supercomputers• (Near) real-time logging of application

output to remote server


GASS File Access API• Minimum changes to application• globus_gass_open(), globus_gass_close()

Same as open(), close() but use URLs instead of filenames

Caches URL in case of multiple opens Return descriptors to files in local cache or

sockets to remote server• globus_gass_fopen(), globus_gass_fclose()


GASS File Access API (cont)• Support for different access patterns

Read-only (from local cache) Write-only (to local cache) Read-write (to/from local cache) Write-only, append (to remote server)


Remove cachereference

Upload changes

Modified noyes

globus_gass_open()/close()

Download Fileinto cache

open cached file,add cachereference

URL in cache? noyes

globus_gass_open()globus_gass_close()


GASS File API Semantics

• Copy-on-open to cache if not truncate or write-only append and not already in cache

• Copy on close from cache if not read only and not other copies open

• Multiple globus_gass_open() calls share local copy of file

• Append to remote file if write only append: e.g., for stdout and stderr

• Reference counting keeps track of open files


Remote Cache Management Utilities

• Remote management of caches, for Prestaging/poststaging of files Cache cleanup and management

• Support operations on local & remote caches

• Functionality encapsulated in a program: globus-gass-cache


globus-gass-server• Simple file server

Run by user wherever necessary Secure https protocol, using GSI APIs for embedding server into other programs

• Exampleglobus-gass-server –r –w -t

-r: Allow files to be read from this server -w: Allow files to be written to this server -t: Tilde expand (~/… $(HOME)/…) -help: For list of all options


1. Build RSL string2. Startup GASS server3. Submit to request4. Return output

jobmanager

gatekeeper

program

GRAM & GASS: Putting It Together

stdout

GASS server

2

3

globus-job-run

RSLstring

1CommandLine Args

3

34

444


Globus Components In ActionLocal Machine

mpirun

globusrun

GRAMClient GSI

GRAMClient GSI

Remote Machine

AppNexus

AIX

PBS

MPI

grid-proxy-initX509UserCert

UserProxyCert

Machines

GRAM GatekeeperGSI

GRAM Job ManagerGASS Client

Remote Machine

AppNexus

Solaris

Unix Fork

MPI

GRAM GatekeeperGSI

GRAM Job ManagerGASS Client

RSL string

RSL multi-request

RSL single requestDUROC

GASS Server

RSL parser


Summary• We learned how to dynamically select and

configure computations MDS and DUROC/GRAM enable construction

of brokers• GASS enables access to remote files and

executables• Demonstrates the interaction of many

Globus components


IPC with globus_io• TCP, UDP, IP multicast, and File I/O

Familiar socket and file abstractions• Asynchronous & synchronous interfaces

Robust implementation & applications• Easy to use security, socket options, etc

Turn on security and other options with just a few function calls

• Designed for Win32 support Hides difference between socket & file handles Compatible with completion ports


Motivation• Numerous modules were using various

combinations of TCP, UDP, IP multicast, and file I/O Write very robust code once, and exploit it

throughout• In the process, solve other issues:

Win32 portability Ease of use of security, socket options, QoS


Approach• Provide familiar socket and file abstractions• Provide both synchronous and asynchronous

versions of everything Can easily write code that will not block for

anything• Handle security, socket options, and QoS

through attributes


Win32• Unlike Unix, in Win32 “file handles” and

“socket handles” are treated differently Select only works on socket handles Different Win32 calls for file and socket I/O globus_io allows us to mask this difference

• Win 32 “completion ports” give the best I/O performance globus_io’s asynchronous callback interface

matches well with completion ports globus_callback also designed for this


TCP Security Attributes• TCP authentication and delegation

characteristics globus_io_attr_set_secure_authentication_mode()

• GLOBUS_IO_SECURE_AUTHENTICATION_MODE_NONE• GLOBUS_IO_SECURE_AUTHENTICATION_MODE_GSSAPI

globus_io_attr_set_secure_delegation_mode()• GLOBUS_IO_SECURE_DELEGATION_MODE_NONE• GLOBUS_IO_SECURE_DELEGATION_MODE_LIMITED_PROXY• GLOBUS_IO_SECURE_DELEGATION_MODE_FULL_PROXY


TCP Security Attributes• TCP authorization and channel characteristics

globus_io_attr_set_secure_authorization_mode()• GLOBUS_IO_SECURE_AUTHORIZATION_MODE_SELF• GLOBUS_IO_SECURE_AUTHORIZATION_MODE_IDENTITY• GLOBUS_IO_SECURE_AUTHORIZATION_MODE_CALLBACK

globus_io_attr_set_secure_channel_mode()• GLOBUS_IO_SECURE_CHANNEL_MODE_CLEAR• GLOBUS_IO_SECURE_CHANNEL_MODE_GSI_WRAP• GLOBUS_IO_SECURE_CHANNEL_MODE_SSL_WRAP


TCP Socket Attributes• TCP socket options

globus_io_attr_set_socket_reuseaddr() globus_io_attr_set_socket_keepalive() globus_io_attr_set_socket_linger() globus_io_attr_set_socket_oobinline() globus_io_attr_set_socket_sndbuf() globus_io_attr_set_socket_rcvbuf() globus_io_attr_set_tcp_nodelay()


Other Attributes• File attributes

globus_io_attr_set_file_type()• GLOBUS_IO_FILE_TYPE_TEXT• GLOBUS_IO_FILE_TYPE_BINARY

• Restricting anonymous ports to a particular port range globus_io_attr_set_tcp_restrict_port()

• IP multicast globus_io_attr_set_udp_multicast_loop() globus_io_attr_set_udp_multicast_ttl()


Core Functions• Common functions used for all forms of I/O

globus_io_[register]_select() globus_io_[register]_cancel() globus_io_[register]_close() globus_io_[register]_read() globus_io_[register]_write() globus_io_[register]_writev() globus_io_try_{read,write,writev}() globus_io_get_handle_type() globus_io_handle_{set,get}_user_pointer()


TCP Connection Setup• Typical functions for creating TCP

connections globus_io_tcp_create_listener() globus_io_tcp_[register]_listen() globus_io_tcp_[register]_accept() globus_io_tcp_[register]_connect()

• Setting and getting attributes globus_io_tcp_set_attr() globus_io_tcp_get_attr()


File Setup• Typical functions for establishing file I/O

globus_io_file_open() globus_io_file_seek()


Portability Features• A collection of fundamental software modules.

Module activation/deactivation Portable thread library (POSIX subset) Thread-safe and portable libc wrappers Timed and periodic callbacks Data object and error object management Modules to manipulate lists, fifos, URLs, …

• The rest of Globus relies on these features.• Developers can (should) use these features for

maximum portability and convenience.


Activation and Deactivation• globus_module_*()

Functions for activation (initialization) and deactivation (shutdown)

Support for multiple independent activations and deactivations of a module through reference counting

Support for dependencies amongst modules Support for thread-safe initialization Modules can use atexit()...


Threads• globus_thread_*(), globus_mutex_*(), globus_cond_*()

Simple POSIX threads (pthreads) subset Same arguments and semantics as pthreads Simply change “pthread” to “globus” or “globus_thread”

in the function name Provides portability to pre-standard pthread libraries, and

non-pthread based systems Simple pass-through on systems with standard pthreads Co-exists with programs using pthreads directly


Threaded vs Non-threaded• Globus supports pthreads, cthreads, Solaris threads,

and sproc.• Globus also supports non-threaded applications.

Most thread-related functions are stubbed out. If you write code carefully which uses the mutex, cond,

and thread specific storage functions, if should work both threaded and non-threaded.

Most Globus Toolkit APIs use an event driven approach, which works well with or without threads.


globus_libc• Wrappers around standard libc functions

Thread safe, even if underlying libc is not• Caveat: Application must also use globus_libc to ensure

thread safety. Same interface for threaded and non-threaded

• POSIX reentrant functions (*_r()) always work.• Example: globus_libc_gethostbyname_r()

Fixes or enhances some functions• Example: globus_libc_gethostname() tries to figure out a

fully qualified hostname despite system configuration.


Callbacks• Driver for timed and periodic callbacks

globus_callback_register_oneshot() globus_callback_register_periodic() globus_callback_unregister() globus_callback_poll()

• All modules rely on this for periodic polling• Uses threads where possible• Uses thread blocking callbacks to guarantee

progress even when callback blocks


Convenience Modules• globus_common also contains a small set of

convenience modules globus_fifo: First-In-First-Out queue globus_hashtable: Hashtable globus_list: List functions globus_symboltable: Symbol table management globus_url: URL parsing globus_strptime: Y2K-compliant strptime() globus_object globus_error

the globus project infrastructure for computational grids

Documents

wright state university

computational services

power grid

grid applications

computational gridcomputation

telephone grid

globus projectinfrastructure

globus toolkitstart