OGCEOGCEConsortium

Reusable Grid Portal Components

Mary Thomas

OGCEOGCEConsortium

Outline

• The Portal Challenge

• The Solution: – The Open Grid Computing Environments

Consortium (OGCE)

• The OGCE Portal Architecture

• Application Portal Examples

• NMI OGCE Plans

OGCEOGCEConsortium

The Portal Challenge

OGCEOGCEConsortium

Por

tal U

ser

Inte

rfac

e Grid ResourceBroker Service

Grid and Web

Protocols

Informationand

Data Services

DatabaseService Database

HPCor

Compute Cluster

Grid InformationServices, SRB

PortalClientStub

PortalClientStub

PortalClientStub

JDBC,Local, orRemote

Connection

Three-Tiered Architecture

Three-tiered architecture is accepted standard for accessing Grid and other services

OGCEOGCEConsortium

Portal are Gateways to Grid

• Early Grid Portals were great success:– Applications oriented,– Provided scientists with some useful tools– Encapsulated some complexities.

• But they were:– All “stovepipe” solutions.

• Very little reusable code and infrastructure

• Nothing could be shared between portals

– Many ignored emerging architecture of the Grid• A framework based on distributed ubiquitous services

OGCEOGCEConsortium

Impact of Distributed Grid and Web Services

Resource layer1000s of PCs ->massive supercomputers

Grid Portals

Launch, configureAnd control Application Interface

Workflow service

App InstanceApp InstanceApp Instance

SecuritySecurity

Data ManagementService

Data ManagementService

AccountingService

AccountingService

LoggingLogging

Event/MesgService

Event/MesgService

PolicyPolicy

Administration& Monitoring

Administration& Monitoring

Grid OrchestrationGrid Orchestration

Registries andName binding

Registries andName binding

Reservations And SchedulingReservations

And Scheduling

Open Grid Service Architecture Layer

Open Grid Service Infrastructure (web service component model)

Online instruments

(slide by Dennis Gannon)

OGCEOGCEConsortium

Advantage of Portlet Architecture

• Portlet/component approach very compatible with Web/Grid services model

• Each Grid service can be associated with a unique portlet– Very easy to add new services– Many different Groups can contribute portlets which can be

plugged into a portal.• Currently Indiana, Argonne, Michigan, NCSA,Texas + GridSphere

portal group following this model. Most likely others as well.– Jetspeed is Apache standard. Basis for IBM WebSphere portal.

Also Oracle, Sun and others. • Now a Java Standard (waiting for JSR 168)

• User can select and configure portlets he/she wishes to use– Selection becomes part of persistent context

OGCEOGCEConsortium

The Next Generation Grid Portal

• Building on Standard Technologies– Portlet Design (JSR-168) IBM, Oracle, Sun, BEA, Apache– Grid standards: Java CoG, Web/Grid Services– Web server: JetSpeed (open source)

• User configurable, Service Oriented– Philosophy: The Portal is a gateway to distributed Grid and Web

Services• With common API, portlets can be exchanged, interoperate

Portal Server

MyProxyServer

MetadataDirectoryService(s)

Directory& indexServices

ApplicationFactoryServices

Messagingand group

collaboration

Event andlogging

Services

OGCEOGCEConsortium

User Requirements Have Not Changed

• Users & Clients are:– Portal clients/users– Portal developers

• Security– Single sign-on, authentication to all grid resources

• Certificate based / uses MyProxy– VO management

• File Management– Grid FTP– File staging– Metadata management

• Remote Job Management– Job submission, scheduling, and tracking. Log and output mgmt.– Workflow management tools

OGCEOGCEConsortium

User Requirements

• Information Services– Grid resource and status information directories– Data and metadata catalogs – Personal and VO metadata and service index/directories– Database Interfaces

• Application Interfaces– Configure and launch workflow from domain specific interface

• Access to Collaboration Tools– VO collaboration. Synchronous and asynchronous– Access Grid venues

• Developers need common API

OGCEOGCEConsortium

The Solution:The Open Grid Computing

Environment (OGCE) Consortium

OGCEOGCEConsortium

Open Grid Computing Environments (OGCE) Consortium

• Motivation:– Emerging technologies impacting portal developers:

• Grid/Web services; ubiquitous database use, XMl

– Lessons learned from GCE-RG and other projects (GridPort/HotPage, Alliance Portal, DOE Components, NEESGrid, Gateway, GRaDS, others)

– Multiple projects repeating similar tasks, work, development;

• OGCE established in Fall 2003 to foster collaborations and sharable components with portal developers worldwide.

OGCEOGCEConsortium

OGCE

• Funded by:– NSF NMI: software releases will be part of NMI

and synchronized with GRIDS releases– Also: NCSA Alliance, NPACI, DOE, NASA,

others

• Release Schedule:– First Release: 11-15-03– Next Release: April, 2004

• Website: http://www.ogce.org

OGCEOGCEConsortium

OGCE Tasks/Goals

• Development of interoperable Grid portal architecture – Define standards for common API’s between portlet

service and remote services• Establishment of a Grid Portal Collaboratory

– Repository of portlet and portal service components• An online forum for developers of Grid Portals• The building of reusable portal components:

– Can be integrated in a common portal container system• Contribute to standards organizations:

– NSF NMI program software releases– GGF if appropriate or applicable

OGCEOGCEConsortium

The OGCE Portal Architecture

OGCEOGCEConsortium

OGCE Initial Architecture

Po

rta

l Local

Portlets

Teamlets

Proxy Portlets

JetspeedInternalServices

JavaCOGAPI

Java CoGKit

Grid Services

GridProtocols

GRAM,MDS-LDADMyProxy

ServiceAPI

CHEFServices

Rem

ote

Inte

rfac

es CoG

StubsHTTP

Grid Services

Other ServicesSOAP

Initial architecture aggregates multiple services into a single portal using portlet containers

OGCEOGCEConsortium

OGCE Initial Arch: Features & Capabilities

• OGCE approach encompasses multiple portal development systems:– Preserving development investments

• Central portal system (Jetspeed) acts as a central portal container:– Portal responsible for managing the lifecycle of pluggable portal

components, or portlets.

• Grid/computational portlets must provide services on the behalf of the user:– Not just a collection of links/static pages

• User view (from the left) is of a single portal composed of portlets.

OGCEOGCEConsortium

Supports Aggregation of Diverse Tasks

• Portlets provide a uniform management interface to the portal (on the left) connect to heterogeneous set of APIs on the right. – First horizontal thread

• Portlets interact with web interfaces written in other languages such as Perl, Python, PHP, ColdFusion, etc. – Architecture for bridged support is depicted in the second

horizontal thread • Examples:

– Locally running Myproxy Grid portlet uses Java CoG kit to interact with remote Grid services

– Job Scheduler portlet uses GridPort Toolkit to submit job via Community Scheduling Frameworks (CSF) service

– Teamlets (sharable portlets) share collaboration and Grid services, such as calendars, message boards, file services.

OGCEOGCEConsortium

OGCE - Revised Architecture

• Revised portal architecture provides a unified interface for portal services.• Supports aggregation of services in backend

Po

rtal

Portletsand

Teamlets

JetspeedInternal Services

ServiceAPI

GridProtocolsGrid

ServiceStubs

RemoteContentServices

Remote Content ServersHTTP

GridServices

Java

Co

G K

it

LocalPortal

Services

OGCEOGCEConsortium

Features

• All services will employ a single (generic) service API that allows them to be deployed and accessed in a uniform fashion.

• Developer then chooses (during portal configuration) the implementations of particular services.

• Services may be local, may access remote sites via Grid protocols, or may access remote content, such as web interfaces written in Perl.

• OGCE Twist:– Entire portal will auto-install and include all required

software components

OGCEOGCEConsortium

Key Technologies????

• Core Portal Framework: – CHEF/JetSpeed (U. Mich)

• Core Grid Technologies:– JavaCoG (provides Globus interface)– GridPort Toolkit (aggregates JavaCoG, and

other grid services)

OGCEOGCEConsortium

Portal/Application Gallery

OGCEOGCEConsortium

TeraGrid Portal • Access Point for TeraGrid

Services• Version 0: Collecting Initial

Services– Public Information about

Resources– Private Information for the

developers.

• Version 1: User centered portal– HotPage/GridPort style access to

user accounts, credentials, job submission & management.

• Version 2: Portals for Science Collaborations (Q3 2004)– Shared spaces, whiteboards, AG

access, group authorization, shared application services

OGCEOGCEConsortium

TeraGrid Integrated Architecture

Diagram demonstrates how existing software projects (such as GridPort) can be adapted to support NMI Portals software system

Po

rtal

Portletsand

Teamlets

JetspeedInternal Services

GridServiceStubs

RemoteContentServices

Remote Content ServersHTTP

GridServices

JavaCoG Kit

LocalPortal

Services

ServiceAPI

Gri

dP

ort

To

olk

it

WebServices

OGCEOGCEConsortium

Create job

description

Create job

description

Store andrecordresults

Store andrecordresults

Generateand storederived

products

Generateand storederived

products

monitoringmonitoring

myMEAD personal

informationspace

myMEAD personal

informationspace

Broker ServiceBroker Service

Teragriddata service

Teragriddata service

TeragridTeragridGlobus Toolkit 3.0Globus Toolkit 3.0

GridFTPGridFTP MetaCatalogService(MCS)MetaCatalogService(MCS)

NEESgrid Meta-Catalog service

NEESgrid Meta-Catalog service

Storage Resource Broker (SRB)

Storage Resource Broker (SRB)

Gather data files, resources

Gather data files, resources

Execute applicationExecute

application

Need ‘x’put to ‘l’by time y:00

Need tostore ‘y’ and ‘z’

Provides contextto resolve ‘x’, ‘y’, ’z’

Retrieve customized run description templatefrom data managementservices Monitoring results captured in event

services persistent channel. myMEADrecords address of service and channel

Delayed invocationof GridFTP

Teragrid Data Service-single point of contact to server-side data servers

Alliance scientific portal

OGCEOGCEConsortium

DOE Fusion Grid Portal

• Portal Development on the DOE SciDAC Grid– Initially, Fusion Grid

• Multi-site collaborative project: – TACC, Indiana, SDSC,

General Atomics

• Developing interoperable portal and Web services

• Portal architecture based on NMI Portal software

OGCEOGCEConsortium

LEAD Portal

OGCEOGCEConsortium

CHEF - NEESGrid

• NEES Tools– Data Viewers (Live/Stored)– Telepresense– Video as Data– Electronic Notebook– Data/Metadata Repository

• Grid Technologies– Grid sign on using myproxy– Grid computational portal– GridFTP– ..Many more

OGCEOGCEConsortium

OGCE Plans

OGCEOGCEConsortium

OGCE Portal System

System Integration

OGSAGlobus

COG

CHEFToolkit

Gridport

AlliancePortal

Workflow

Testing

NMIDist

Slide courtesy of Chuck Severance

OGCEOGCEConsortium

NMI Features List (April 04 Release)

• Content Management:– Resources: Users and groups can upload and share documents

and URLs. – News: Several news feeds are available, including XMLHack and

BBC News. – Grid Context: Store arbitrary web objects (movies, web pages,

audio files) in a customizable, annotated directory tree. • Collaborative:

– Chat: Engage other members of your group in online discussion. – Discussion: Participate in discussions with other members of your

group – Calendar Program:schedule appointments and attach associated

documents and URLs.  Calendars can be used to schedule both user and group activities. 

– Anabas Impromptu: system for real time shared display, audio, and chats.

OGCEOGCEConsortium

NMI Features List (April 04 Release)

• GT 3.0/Interactive:– Proxy Manager and Job Submit: Obtain and manage one or more Grid credentials

that can be used to access Grid resources through your browser. – GridFTP: Use Grid credential to browse remote directories and to upload and

download files between your desktop and a remote host. – Application Manager– Application Factory Manager

• Remote Job Management – GridAnt for workflow– Job submission portal– Community Scheduling Framework (CSF): submit jobs to scheduler

• Information Services:– LDAP Browser:– Navigate the LDAP server of your choice. – GridPort Information Repository Portlets (GPIR): Portlets are available for browsing

Machine Summaries,  System Status, and Total Grid Capabilities.• Application Portlets:

– Specialized portlets for projects

OGCEOGCEConsortium

Key Feature – Download a Portal & Install

• Prerequisite Software– Java 2 Standard Edition version

1.4.x or higher. – Apache Ant version 1.5.x or higher. 

• Preparing the Installation– Optionallly, create a "portal"

account on your web server. – Download the portal bundle – Put the download bundle into the

home directory of "portal" account. – Untar or unzip the portal bundle.  – Globus: installation process will

create the directories .globus and .globus/certificates in the

• Software automatically installed includes:

– Globus/Java CoG– CHEF

• Install the Portal– Move into the portal/chef-1.0.7

directory.– Edit file "build.properites" – Type "ant deploy" to build in and

install the portal. – Start the Tomcat web server.  – Point your browser at http://<your

host name>:10081/ogce and create an account.

• OS/Machines Tested:– MAC OSX– PC/Windows XP– Linux – Cygwin

OGCEOGCEConsortium

Current Status

• SC03 Demonstration portal and software download available– Beta1

• Updated Release available now– Bug fixes– Now supporting GT3.x with J2SDK 1.4.2

• Working on NMI April 2004 release– Including new features to support science portal applications

• Open GCE Runtime Engine (OGRE) workflow services and portlets• Sample template portlets for simple science applications

– Based on aggregation of existing projects– OGSA/OGSI/GT3.x compatible

OGCEOGCEConsortium

Future Directions

• Moving towards new portlet standard– JSR 168– Should allow compatibility with

• GridSphere• Academic education and business portals Oncourse, OneStart, CHEF

(of course), and uPortal • Major commercial products offerings from Sun, IBM, Oracle, etc.

• Incorporate new features:– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)– Access Grid, Polycom and other AV Portlets

• Collaborations and Contributions– DOE SciDAC projects (such as CMCS)

• Expand online repository– Develop process for third party contributions

OGCEOGCEConsortium

Future Directions

• Moving towards new portlet standard– JSR 168– Should allow compatibility with

• GridSphere• Academic education and business portals Oncourse, OneStart, CHEF

(of course), and uPortal • Major commercial products offerings from Sun, IBM, Oracle, etc.

• Incorporate new features:– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)– Access Grid, Polycom and other AV Portlets

• Collaborations and Contributions– DOE SciDAC projects (such as CMCS)

• Expand online repository– Develop process for third party contributions

OGCEOGCEConsortium

Participants

Indiana University

University of Michigan

University of Texas at Austin

Argonne National Labs

JavaCoG

National Center for Supercomputing Applications

the globus alliance

OGCEOGCEConsortium

References

• OGCE Website– http://www.ogce.org

• NSF Middleware Initiative– http://www.nsf-middleware.org