the open grid computing environments collaboratory

The Open Grid Computing The Open Grid Computing Environments CollaboratoryEnvironments Collaboratory

Marlon PierceMarlon PierceCommunity Grids LabCommunity Grids Lab

Indiana UniversityIndiana Universitywww.collab-ogce.orgwww.collab-ogce.org

NSF NMI Project for Reusable NSF NMI Project for Reusable Portal Components: Who We ArePortal Components: Who We Are

University of Chicago/ANLUniversity of Chicago/ANL• Gregor von LaszewskiGregor von Laszewski

Indiana UniversityIndiana University• Marlon Pierce, Dennis Gannon, Geoffrey Fox, Marlon Pierce, Dennis Gannon, Geoffrey Fox,

and Beth Plaleand Beth Plale University of MichiganUniversity of Michigan

• Charles Severance, Joseph HardinCharles Severance, Joseph Hardin NCSA/UIUCNCSA/UIUC

• Jay Alameda, Joe FutrelleJay Alameda, Joe Futrelle Texas Advanced Computing CenterTexas Advanced Computing Center

• Jay BoisseauJay Boisseau San Diego State UniversitySan Diego State University

• Mary ThomasMary Thomas

Presentation OutlinePresentation Outline

OGCE OverviewOGCE Overview Portlet Standards for OGCEPortlet Standards for OGCE

• New JSR 168 compatible Grid portletsNew JSR 168 compatible Grid portlets• Demonstrate compatibility with uPortal Demonstrate compatibility with uPortal

and GridSphere containersand GridSphere containers Grid Portlet Programming Grid Portlet Programming Sample Application: TeraGrid User Sample Application: TeraGrid User

PortalPortal Sample Application: LEAD PortalSample Application: LEAD Portal

What a Grid Portal Is/Is NotWhat a Grid Portal Is/Is Not It isIt is

• A tool for aggregating and managing web contentA tool for aggregating and managing web content• A user customizable view of these Web content pieces.A user customizable view of these Web content pieces.

You see what you want/can see.You see what you want/can see. But you must log in.But you must log in.

• Implemented on top of standard servicesImplemented on top of standard services Like login, authorization, customization.Like login, authorization, customization. May include collaboration, etc, that depend on login.May include collaboration, etc, that depend on login.

• A way to accomplish Grid tasks through browsers:A way to accomplish Grid tasks through browsers: Launch, monitor jobsLaunch, monitor jobs Move filesMove files Run science applications based on these services.Run science applications based on these services.

• Compatible with emerging standards and best practices Compatible with emerging standards and best practices (such as portlets, JSR 168 and WSRP).(such as portlets, JSR 168 and WSRP).

It is not (just)It is not (just)• A web pageA web page• A collection of linksA collection of links• An appletAn applet

Por

tal U

ser

Inte

rfac

e Grid ResourceBroker Service

Grid and Web

Protocols

Informationand

Data Services

DatabaseService Database

HPCor

Compute Cluster

Grid InformationServices, SRB

PortalClientStub

PortalClientStub

PortalClientStub

JDBC,Local, orRemote

Connection

Three-Tiered ArchitectureThree-Tiered Architecture

Three-tiered architecture is accepted standard for Three-tiered architecture is accepted standard for accessing Grid and other servicesaccessing Grid and other services

Building Portals from Reusable Building Portals from Reusable ComponentsComponents

The component architecture of choice for the The component architecture of choice for the Portal community is the one based on Portal community is the one based on portletsportlets• (Java) components that generate content, make local (Java) components that generate content, make local

and remote connections to services.and remote connections to services.• Portlet containersPortlet containers manage portlet lifecycles manage portlet lifecycles• Standardized now by Standardized now by JSR 168JSR 168

A portlet is a piece of Java code similar to a A portlet is a piece of Java code similar to a servlet that does two things:servlet that does two things:• Creates a fragment of HTML display that becomes part Creates a fragment of HTML display that becomes part

of a web pageof a web page• Handles any link clicks and HTML <Form> actions.Handles any link clicks and HTML <Form> actions.

May involve accessing local and remote servicesMay involve accessing local and remote services There are now many, many portlet components.There are now many, many portlet components.

• So don’t start from scratch. So don’t start from scratch. JSR 168JSR 168 is the Java standard for portlets. is the Java standard for portlets.

OGCE’s Release 1OGCE’s Release 1 The OGCE Portal release is based on The OGCE Portal release is based on

CHEF/Jetspeed 1.4CHEF/Jetspeed 1.4 Available for download and installation from Available for download and installation from

http://www.collab-ogce.orghttp://www.collab-ogce.org.. It comes with many pre-configured capabilities if It comes with many pre-configured capabilities if

you want a grid portal “out of the box”.you want a grid portal “out of the box”.• Except for the mysql jar.Except for the mysql jar.• You must still set up Grid services (MyProxy servers, You must still set up Grid services (MyProxy servers,

Globus, etc).Globus, etc).• Globus version compatibility through the Java CoG.Globus version compatibility through the Java CoG.

Apache Ant-based installation procedure:Apache Ant-based installation procedure:• Edit one properties file, run ant, and away you go.Edit one properties file, run ant, and away you go.

http://www.collab-ogce.org/

PortalPortal CapabilitiesCapabilities DescriptionDescription

Grid Proxy Certificate ManagerGrid Proxy Certificate Manager Get MyProxy certs after logging inGet MyProxy certs after logging in

ScheduleSchedule Interactive individual and group calendars Interactive individual and group calendars

DiscussionDiscussion Persistent topic-based discussion for groupsPersistent topic-based discussion for groups

ChatChat Live chat services and interfacesLive chat services and interfaces

Document managersDocument managers WEBDav based document system for group WEBDav based document system for group file sharingfile sharing

MDS/LDAP BrowsersMDS/LDAP Browsers Basic Globus MDS browsing and navigatingBasic Globus MDS browsing and navigating

GridContext PortletsGridContext Portlets Access context services for managing Access context services for managing metadatametadata

GRAM Job SubmissionGRAM Job Submission Run simple executables on remote hostsRun simple executables on remote hosts

GridFTPGridFTP Upload, download, crossload remote files.Upload, download, crossload remote files.

GPIR PortletsGPIR Portlets View, interact with HPC status, job, etc View, interact with HPC status, job, etc information.information.

AnabasAnabas Access to Anabas shared display appletsAccess to Anabas shared display applets

Newsgroups and citation portletsNewsgroups and citation portlets Post topics to newsgroup, manage group Post topics to newsgroup, manage group references and citations with access controlsreferences and citations with access controls

User PortletsUser Portlets

PortalPortal CapabilitiesCapabilities DescriptionDescription

CoG Workflow demonstration CoG Workflow demonstration portletportlet

Set up and run task graphs using the Java Set up and run task graphs using the Java CoGCoG

Job Scheduling and SequencingJob Scheduling and Sequencing Schedule sequences of jobs on several hosts Schedule sequences of jobs on several hosts using Community Scheduling Framework.using Community Scheduling Framework.

Condor PortletsCondor Portlets Interact with Condor through browser.Interact with Condor through browser.

Application ManagementApplication Management Combine GridFTP and GRAM into application Combine GridFTP and GRAM into application wizard forms.wizard forms.

Newsgroup ServicesNewsgroup Services Download and install server side of the OGCE Download and install server side of the OGCE newsgroup system.newsgroup system.

OGRE Job Management ServicesOGRE Job Management Services Manage complicated grid tasks through an Manage complicated grid tasks through an extensible, Apache Ant like task list.extensible, Apache Ant like task list.

XDirectory ServicesXDirectory Services The backend for the Grid Context portletThe backend for the Grid Context portlet

More User Portlets and ServicesMore User Portlets and Services

What’s New for the OGCE2?What’s New for the OGCE2? JSR 168JSR 168 Compatible Grid portlet suite Compatible Grid portlet suite

• Basic capabilities: MyProxy, GridFTP, GRAM, GPIR.Basic capabilities: MyProxy, GridFTP, GRAM, GPIR.• Working in uPortal, GridSphere containersWorking in uPortal, GridSphere containers

Container independentContainer independent services for sharing data between services for sharing data between portlet applications.portlet applications.• GSSCredential objects, global session data.GSSCredential objects, global session data.• Limitation in JSR 168Limitation in JSR 168

Support for Support for VelocityVelocity development development • Velocity is the Apache web application development tool of Velocity is the Apache web application development tool of

choice for Jetspeed1.choice for Jetspeed1.• Provide backward compatibility with OGCE1Provide backward compatibility with OGCE1

Maven-basedMaven-based build and deploy system build and deploy system• Choose either uPortal or GridSphereChoose either uPortal or GridSphere• One-command install, but you must still install a Grid toolkit.One-command install, but you must still install a Grid toolkit.

NMI GRID Center: www.grids-center.orgNMI GRID Center: www.grids-center.org ModularModular portlet extensions portlet extensions

• Collaboration tools, etc.Collaboration tools, etc.

Portlet DemoPortlet Demo

Grid Portlet Grid Portlet ProgrammingProgramming

Marlon Pierce, Gregor von Marlon Pierce, Gregor von Laszewski, Eric RobertsLaszewski, Eric Roberts

Grid Programming Interfaces for Grid Programming Interfaces for PortletsPortlets

Portlet form actions may result in remote Portlet form actions may result in remote calls to Grid or Web Servicescalls to Grid or Web Services• A Portlet is just java codeA Portlet is just java code

We use two programming APIs you can We use two programming APIs you can choose fromchoose from• CoG4CoG4 API provides abstraction of Grid tasks to API provides abstraction of Grid tasks to

hide Grid toolkit version differences.hide Grid toolkit version differences.• GridPortGridPort provides integrated information, CSF- provides integrated information, CSF-

based job submission and sequencing, and file based job submission and sequencing, and file manipulation services.manipulation services.

Office of Science

U.S. Department of Energy

Java CoG Kit and PortalsJava CoG Kit and Portals

• The Java CoG Kit is a bridge between Grids and Portals as it provides an abstraction layer that is supportive for portals developers.

• The Goals of the Java CoG Kit include reuse of a variety of commodity tools, protocols, approaches, methodologies, while integrating Grid software to enable – Easier development of advanced Grid services– Easier and more rapid application development– Easier deployment of Grid services – Code reuse and use of component repositories– Use of Web services as part of the Grids– Widespread use of the Grid– Use of commodity technology is not limited to the client.

• As a result we make Portal development and Grid computing easier

Office of Science


Java CoG Kit AbstractionsJava CoG Kit Abstractions

• We provide a number of abstractions that build the foundation why we make Grid programming easier:– File transfer, job submission, authentication

• We provide a workflow abstraction that makes the specification of Grid workflows possible helping those with complex job management scenarios.

• We provide a software methodology that makes adoption to new standards easier– We defined so called providers that allow reusing various Grid

and commodity services including GT2, GT3, GT4 (under development), SSH, Unicore and Condor will be

provided by the community. This feature is unique amongst the Grid toolkits.

Office of Science


GT2GT2

GT3OGSA/WS-RF

GT3OGSA/WS-RF

SSHSSH CondorCondor UnicoreUnicore

CoG Abstraction Layer (job submission, file transfer, authentication)

CoG Abstraction Layer (job submission, file transfer, authentication)

CoGCoG CoGCoG CoGCoG CoGCoG CoGCoG

ApplicationsApplications

Data and Task Management Layer (workflow)

Data and Task Management Layer (workflow)

Gridfaces Layer (portals, Swing, SWF)

Gridfaces Layer (portals, Swing, SWF)

OthersAvaki

OthersAvaki

CoGCoG

OthersSETI

OthersSETI

CoGCoG

NanomaterialsNano

materials

Bio-Informatics

Bio-Informatics

DisasterManagement

DisasterManagement

PortalsPortals

GridID

EG

ridIDE

DevelopmentSupport

Concept of the Java CoG Kit Architecture Concept of the Java CoG Kit Architecture

Office of Science


Java CoG Kit informationJava CoG Kit information

• More information on the Java CoG Kit can be found at http://www.cogkit.org or visit the posters at SC2004

P09: Karajan: A Grid Orchestration Framework: Mihael Hategan, Gregor von Laszewski, Kaizar Amin

P11: The Next Generation of the Java CoG Kit (version 4) Gregor von Laszewski, Kaizar Amin, Matt Bone, Mike Hategan, Pankaj Sahasrabudhe, Mike Sosonkin, Robert Winch,Nithya Vijayakumar, David Angulo

What is GridPort?• High-level middleware that aids grid portal

developers by easing use of low level grid tools• GridPort’s Role in Grid Computing

– Aggregate services from grid software packages• Globus Toolkit• Community Scheduling Framework (CSF)• Storage Resource Broker (SRB)

– Simple, consistent API– Custom Web services

• GPIR• Job Sequencer• Advanced File Transfer

• GridPort portlets are compatible with, part of OGCE releases.

Web ServicesPortlets

J2EE App ServerContains: Spring,Hibernate

WebClient

HTTP / SSL

PostgreSQL

J2EE WebServer

GPIR

JobSequencer

FileTransfer

JobSubmission

DataAccess

GS

I Security

GridP

ort Service A

PI

JDB

C

GridFTP

SRB

OGSI

GRAMWS-GRAM

CSF

Grid ServicesJob

Sequencer

Adv. FileTransfer

GPIR

Adv. FileTransfer

Grid Data

HT

TP

/SO

AP

JSP/Servlet

GPIRAdminClient

GridPort’s 3-tier Architecture

GridP

ort Client A

PI

GPIRPostgreSQL

Information Providers

Resources ClientsDB

Web Scraping

JavaClient

Ingester WS

Query WS

Portals

OtherMiddleware

SOAP-XML

HTTP

PerlClient

Portletsedu.tacc.

gridport.gpir

JDBC

GPIR Web Services

MDS

OGSA(Future)

Other

GridPort Job Sequencer

• Web service for simple job workflow composition

• Portlet interface available for specifying sequences

• Specify job submissions and file transfers

• More information available at

http://www.gridport.net

CSF Integration

• Community Scheduling Framework (CSF)• Meta-scheduler for submitting jobs to a grid

– No need to specify a resource– CSF will schedule jobs to appropriate resource for

you• OGSI grid service• CSF is an open source project developed by

Platform computing– Now available for download from Sourceforge.net

http://sourceforge.net/projects/gcsf/

More GridPort Information

GridPort Demo in TACC Booth 120

2:00-2:20 PM

Thursday, Nov 11

TeraGrid User PortalTeraGrid User Portal

Eric RobertsEric Roberts

Texas Advanced Computing Texas Advanced Computing CenterCenter

MotivationMotivation Make joining the TeraGrid easier for usersMake joining the TeraGrid easier for users Single place for users to find user Single place for users to find user

information and get user supportinformation and get user support Certain information can be displayed Certain information can be displayed

better in a web page than in a command better in a web page than in a command shellshell

Allow novice users to start using grid Allow novice users to start using grid resources securely through a Web resources securely through a Web interfaceinterface

Increase productivity of TeraGrid Increase productivity of TeraGrid researchers – do more science!researchers – do more science!

TeraGrid User PortalTeraGrid User PortalService AggregationService Aggregation

TeraGrid User Portal

InteractiveJob Submission

File Transfer

DocumentationUser Guides

NotificationUser News

User SupportConsulting

CollaborativeCalendar

Chat

InformationResource

Grid

Client Browser

HTTP/SSL

HTTP/SSL/SOAP

Current TeraGrid User Portal Current TeraGrid User Portal CapabilitiesCapabilities

User ServicesUser Services• User Information (User Guides, Support,etc.)User Information (User Guides, Support,etc.)

Information ServicesInformation Services• SystemSystem• GridGrid• NetworkNetwork

InteractiveInteractive• GSI AuthenticationGSI Authentication• Remote Command ExecutionRemote Command Execution• Job SubmissionJob Submission• File ManagementFile Management

Future DirectionsFuture Directions Central gateway for TeraGrid servicesCentral gateway for TeraGrid services TeraGrid allocations and account TeraGrid allocations and account

creation/management through portalcreation/management through portal• Streamline the processStreamline the process

Application portalsApplication portals• Science gateways that expose scientific Science gateways that expose scientific

applications through interfacesapplications through interfaces We will be contacting users for We will be contacting users for

“friendly-user” testing (2005)“friendly-user” testing (2005)

LEAD Application PortalLEAD Application Portal

Marcus ChristieMarcus Christie

Indiana UniversityIndiana University

Linked Environments for Atmospheric Discovery


The LEAD GoalThe LEAD GoalProvide the IT necessary to allow

People (scientists, students, operational practitioners)

andTechnologies (models, sensors, data

mining)

TO INTERACT WITH WEATHER


Analysis/Assimilation

Quality ControlRetrieval of Unobserved

QuantitiesCreation of Gridded Fields

Prediction/Detection

PCs to Teraflop Systems

Product Generation, Display,

Dissemination

End Users

NWSPrivate Companies

Students

Traditional MethodologyTraditional Methodology

STATIC OBSERVATIONS

Radar DataMobile Mesonets

Surface ObservationsUpper-Air BalloonsCommercial Aircraft

Geostationary and Polar Orbiting Satellite

Wind ProfilersGPS Satellites

The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!

The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!


The Consequence: Model Grids The Consequence: Model Grids Fixed in Time – No AdaptivityFixed in Time – No Adaptivity


Analysis/Assimilation

Quality ControlRetrieval of Unobserved

QuantitiesCreation of Gridded Fields

Prediction/Detection

PCs to Teraflop Systems

Product Generation, Display,

Dissemination

End Users

NWSPrivate Companies

Students

STATIC OBSERVATIONS

Radar DataMobile Mesonets

Surface ObservationsUpper-Air BalloonsCommercial Aircraft

Geostationary and Polar Orbiting Satellite

Wind ProfilersGPS Satellites

The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static

Models Responding to Observations


In LEAD, Everything is a ServiceIn LEAD, Everything is a Service• Finite number of services – they’re the “low-level” elements

but consist of lots of hidden pieces…services within services.

Service A(ADAS)

Service B(WRF)

Service C(NEXRAD Stream)

Service D(MyLEAD)

Service E(VO Catalog)

Service F(IDV)

Service G(Monitoring)

Service H(Scheduling)

Service I(ESML)

Service J(Repository)

Service K(Ontology)

Service L(Decoder)

Many others…


Start by Building Simple Prototypes to Start by Building Simple Prototypes to Establish the Services/Other Capabilities…Establish the Services/Other Capabilities…


Service F(IDV)

Service L(Decoder)

Prototype X




Service F(IDV)

Service L(Decoder)

Prototype Y

Service D(MyLEAD)





Service F(IDV)

Service L(Decoder)

Prototype Z

Service A(ADAS)

Service I(ESML)

Service J(Repository)

Service D(MyLEAD)



new load saved

Dynamic Workflow – Changes in ResponseTo Observed Events, Model Output, User Input,


The DemoThe Demo

Using the portal to launch

A Distributed Grid Workflow


The Big PictureThe Big Picture

PortalServer

PortalServer

The user’s browser

ThreddsServer

At Unidata

ThreddsServer

At Unidata

DataDecoderIndiana

DataDecoderIndiana

1. User buildsexperiment

2. portrallaunches

workflow

3. pulls datafrom server

and decodes

OpenDapServer

In Alabama

OpenDapServer

In Alabama

Threddscatalog

GeneratorAt NCSA

Threddscatalog

GeneratorAt NCSA

4. decoderpushesdata toserver

5. tells cat.Generatorabout data

6. pushes metadatato opendap

the open grid computing environments collaboratory

Documents