Reusable Grid Portal

Components

Mary Thomas

OGCE
Consortium
 Outline
• The Portal Challenge
• The Solution:
– The Open Grid Computing Environments
Consortium (OGCE)
• The OGCE Portal Architecture
• Application Portal Examples
• NMI OGCE Plans

OGCE
Consortium
The Portal Challenge

OGCE
Consortium
 Three-Tiered Architecture
Grid and JDBC,
Web Local, or
Protocols Remote
Connection

Portal
Database
Client Database
Service
Stub

Portal HPC
Grid Resource
Client or
Broker Service
Stub Compute Cluster
ec af r et nI r es Ul at r o P

Portal Information
Grid Information
Client and
Services, SRB
Stub Data Services

Three-tiered architecture is accepted standard for

accessing Grid and other services

OGCE
Consortium
 Portal are Gateways to Grid
• Early Grid Portals were great success:
– Applications oriented,
– Provided scientists with some useful tools
– Encapsulated some complexities.
• But they were:
– All “stovepipe” solutions.
• Very little reusable code and infrastructure
• Nothing could be shared between portals
– Many ignored emerging architecture of the Grid
• A framework based on distributed ubiquitous services

OGCE
Consortium
Impact of Distributed Grid and Web
Services
Launch, configure
And control Application Interface

Workflow service

Grid Portals
App Instance
App Instance
App Instance

Open Grid Service Architecture Layer Data Management

Data Management
Service
Registries and Service
Registries and Security
Name binding Security
Name binding Policy
Policy Accounting
Administration Logging Accounting
Administration Logging Service
Reservations & Monitoring Service
Reservations & Monitoring
And Scheduling Event/Mesg Grid Orchestration
And Scheduling Event/Mesg Grid Orchestration
Service
Service

Open Grid Service Infrastructure (web service component model)

Resource layer
1000s of PCs ->massive supercomputers

Online instruments
OGCE
(slide by Consortium
Dennis Gannon)
Advantage of Portlet Architecture
• Portlet/component approach very compatible with
Web/Grid services model
• Each Grid service can be associated with a unique
portlet
– Very easy to add new services
– Many different Groups can contribute portlets which can be
plugged into a portal.
• Currently Indiana, Argonne, Michigan, NCSA,Texas + GridSphere
portal group following this model. Most likely others as well.
– Jetspeed is Apache standard. Basis for IBM WebSphere portal.
Also Oracle, Sun and others.
• Now a Java Standard (waiting for JSR 168)
• User can select and configure portlets he/she wishes
to use
– Selection becomes part of persistent context

OGCE
Consortium
The Next Generation Grid Portal
• Building on Standard Technologies
– Portlet Design (JSR-168) IBM, Oracle, Sun, BEA, Apache
– Grid standards: Java CoG, Web/Grid Services
– Web server: JetSpeed (open source)
• User configurable, Service Oriented
– Philosophy: The Portal is a gateway to distributed Grid and Web
Services
• With common API, portlets can be exchanged, interoperate

Event and
logging
Services Application
Factory
Services
Messaging
Portal Server and group
collaboration
Directory
& index
MyProxy Metadata Services
Server Directory
Service(s) OGCE
Consortium
 User Requirements
Have Not Changed
• Users & Clients are:
– Portal clients/users
– Portal developers
• Security
– Single sign-on, authentication to all grid resources
• Certificate based / uses MyProxy
– VO management
• File Management
– Grid FTP
– File staging
– Metadata management
• Remote Job Management
– Job submission, scheduling, and tracking. Log and output mgmt.
– Workflow management tools

OGCE
Consortium
 User Requirements
• Information Services
– Grid resource and status information directories
– Data and metadata catalogs
– Personal and VO metadata and service index/directories
– Database Interfaces
• Application Interfaces
– Configure and launch workflow from domain specific interface
• Access to Collaboration Tools
– VO collaboration. Synchronous and asynchronous
– Access Grid venues
• Developers need common API

OGCE
Consortium
 The Solution:
The Open Grid Computing
Environment (OGCE) Consortium

OGCE
Consortium
 Open Grid Computing Environments
(OGCE) Consortium
• Motivation:
– Emerging technologies impacting portal developers:
• Grid/Web services; ubiquitous database use, XMl
– Lessons learned from GCE-RG and other projects
(GridPort/HotPage, Alliance Portal, DOE Components,
NEESGrid, Gateway, GRaDS, others)
– Multiple projects repeating similar tasks, work,
development;
• OGCE established in Fall 2003 to foster
collaborations and sharable components with
portal developers worldwide.
OGCE
Consortium
 OGCE
• Funded by:
– NSF NMI: software releases will be part of NMI
and synchronized with GRIDS releases
– Also: NCSA Alliance, NPACI, DOE, NASA,
others
• Release Schedule:
– First Release: 11-15-03
– Next Release: April, 2004
• Website: http://www.ogce.org

OGCE
Consortium
 OGCE Tasks/Goals
• Development of interoperable Grid portal
architecture
– Define standards for common API’s between portlet
service and remote services
• Establishment of a Grid Portal Collaboratory
– Repository of portlet and portal service components
• An online forum for developers of Grid Portals
• The building of reusable portal components:
– Can be integrated in a common portal container system
• Contribute to standards organizations:
– NSF NMI program  software releases
– GGF if appropriate or applicable

OGCE
Consortium
The OGCE Portal Architecture

OGCE
Consortium
 OGCE Initial Architecture
Grid
Java Java Protocols
Local
COG CoG Grid Services
Portlets API Kit GRAM,
MDS-LDAD
MyProxy

Grid Services

Interfaces
CoG

Remote
Proxy
Portlets HTTP Stubs Other Services
Portal

SOAP

Service CHEF
Teamlets API Services

Jetspeed
Internal
Services
Initial architecture aggregates multiple services into a single
portal using portlet containers

OGCE
Consortium
 OGCE Initial Arch:
Features & Capabilities
• OGCE approach encompasses multiple portal
development systems:
– Preserving development investments
• Central portal system (Jetspeed) acts as a central
portal container:
– Portal responsible for managing the lifecycle of pluggable portal
components, or portlets.
• Grid/computational portlets must provide services on
the behalf of the user:
– Not just a collection of links/static pages
• User view (from the left) is of a single portal composed
of portlets.

OGCE
Consortium
 Supports Aggregation of
Diverse Tasks
• Portlets provide a uniform management interface to
the portal (on the left) connect to heterogeneous set of
APIs on the right.
– First horizontal thread
• Portlets interact with web interfaces written in other
languages such as Perl, Python, PHP, ColdFusion, etc.
– Architecture for bridged support is depicted in the second
horizontal thread
• Examples:
– Locally running Myproxy Grid portlet uses Java CoG kit to interact
with remote Grid services
– Job Scheduler portlet uses GridPort Toolkit to submit job via
Community Scheduling Frameworks (CSF) service
– Teamlets (sharable portlets) share collaboration and Grid services,
such as calendars, message boards, file services.

OGCE
Consortium
 OGCE - Revised Architecture
Grid
Grid Protocols

CoG Kit
Java
Grid
Service
Services
Stubs
Portlets Local
Service
and Portal
API
Teamlets Services
Remote Remote
l at r o P

Content Content
Services HTTP Servers

Jetspeed
Internal Services

• Revised portal architecture provides a unified interface for portal services.

• Supports aggregation of services in backend

OGCE
Consortium
 Features
• All services will employ a single (generic)
service API that allows them to be deployed
and accessed in a uniform fashion.
• Developer then chooses (during portal
configuration) the implementations of
particular services.
• Services may be local, may access remote
sites via Grid protocols, or may access remote
content, such as web interfaces written in Perl.
• OGCE Twist:
– Entire portal will auto-install and include all required
software components

OGCE
Consortium
 Key Technologies????

• Core Portal Framework:

– CHEF/JetSpeed (U. Mich)
• Core Grid Technologies:
– JavaCoG (provides Globus interface)
– GridPort Toolkit (aggregates JavaCoG, and
other grid services)

OGCE
Consortium
Portal/Application Gallery

OGCE
Consortium
 TeraGrid Portal
• Access Point for TeraGrid
Services
• Version 0: Collecting Initial
Services
– Public Information about
Resources
– Private Information for the
developers.
• Version 1: User centered portal
– HotPage/GridPort style access to
user accounts, credentials, job
submission & management.
• Version 2: Portals for Science
Collaborations (Q3 2004)
– Shared spaces, whiteboards, AG
access, group authorization,
shared application services

OGCE
Consortium
 TeraGrid Integrated Architecture

Grid Grid Java

CoG Kit
Service Services

GridPort Toolkit
Stubs
Web
Portlets Local Services
Service
and Portal
API
Teamlets Services
Remote Remote
l at r o P

Content Content
Services HTTP Servers

Jetspeed
Internal Services

Diagram demonstrates how existing software projects (such as GridPort) can

be adapted to support NMI Portals software system

OGCE
Consortium
 Alliance scientific portal
Create Gather
Gather Generate
Generate
Create Store
Store and
data
data files,
files, Execute
Execute and
and store
store
job
job record
resources application
application record derived
derived
description
description resources
results
results products
products

Retrieve customized
run description template
from data management Broker Service
services Monitoring results captured in event
services persistent channel. myMEAD
Need ‘x’ Need to records address of service and channel
put to ‘l’ store
by time y:00 ‘y’ and ‘z’

monitoring
monitoring

Teragrid
data service
Teragrid Data Service-
Provides context
to resolve ‘x’, ‘y’, ’z’ single point of contact
Delayed invocation
myMEAD
myMEAD of GridFTP to server-side data
personal
personal servers
information
information
space
space MetaCatalog NEESgrid
NEESgrid Meta- Storage
Storage Resource
GridFTP MetaCatalog
Service(MCS)
Service(MCS) Catalog
Meta-
Catalog service
service Broker
Resource
Broker (SRB)
(SRB)

Globus Toolkit 3.0

Teragrid

OGCE
Consortium
 DOE Fusion Grid Portal

• Portal Development on
the DOE SciDAC Grid
– Initially, Fusion Grid
• Multi-site collaborative
project:
– TACC, Indiana, SDSC,
General Atomics
• Developing interoperable
portal and Web services
• Portal architecture based
on NMI Portal software

OGCE
Consortium
LEAD Portal

OGCE
Consortium
 CHEF - NEESGrid

• NEES Tools
– Data Viewers (Live/Stored)
– Telepresense
– Video as Data
– Electronic Notebook
– Data/Metadata Repository
• Grid Technologies
– Grid sign on using myproxy
– Grid computational portal
– GridFTP
– ..Many more

OGCE
Consortium
OGCE Plans

OGCE
Consortium
 OGCE Portal System
OGSA
Globus
COG
NMI
Dist
CHEF System
Toolkit Integration

Gridport

Alliance
Portal Testing

Workflow

Slide courtesy of Chuck Severance

OGCE
Consortium
 NMI Features List (April 04 Release)

• Content Management:
– Resources: Users and groups can upload and share documents and
URLs.
– News: Several news feeds are available, including XMLHack and BBC
News.
– Grid Context: Store arbitrary web objects (movies, web pages, audio
files) in a customizable, annotated directory tree.
• Collaborative:
– Chat: Engage other members of your group in online discussion.
– Discussion: Participate in discussions with other members of your
group
– Calendar Program:schedule appointments and attach associated
documents and URLs. Calendars can be used to schedule both user
and group activities.
– Anabas Impromptu: system for real time shared display, audio, and
chats.

OGCE
Consortium
 NMI Features List (April 04 Release)
• GT 3.0/Interactive:
– Proxy Manager and Job Submit: Obtain and manage one or more Grid credentials
that can be used to access Grid resources through your browser.
– GridFTP: Use Grid credential to browse remote directories and to upload and
download files between your desktop and a remote host.
– Application Manager
– Application Factory Manager
• Remote Job Management
– GridAnt for workflow
– Job submission portal
– Community Scheduling Framework (CSF): submit jobs to scheduler
• Information Services:
– LDAP Browser:
– Navigate the LDAP server of your choice.
– GridPort Information Repository Portlets (GPIR): Portlets are available for browsing
Machine Summaries, System Status, and Total Grid Capabilities.
• Application Portlets:
– Specialized portlets for projects

OGCE
Consortium
 Key Feature –
Download a Portal & Install
• Prerequisite Software
– Java 2 Standard Edition version
1.4.x or higher.
– Apache Ant version 1.5.x or higher.
• Preparing the Installation
– Optionallly, create a "portal"
account on your web server.
– Download the portal bundle
– Put the download bundle into the
home directory of "portal" account.
– Untar or unzip the portal bundle.
– Globus: installation process will
create the directories .globus and
.globus/certificates in the
• Software automatically installed
includes:
– Globus/Java CoG
– CHEF
• Install the Portal
– Move into the portal/chef-1.0.7
directory.
– Edit file "build.properites"
– Type "ant deploy" to build in and
install the portal.
– Start the Tomcat web server.
– Point your browser at http://<your
host name>:10081/ogce and create
an account.
• OS/Machines Tested:
– MAC OSX
– PC/Windows XP
– Linux
– Cygwin

OGCE
Consortium
 Current Status
• SC03 Demonstration portal and software download
available
– Beta1
• Updated Release available now
– Bug fixes
– Now supporting GT3.x with J2SDK 1.4.2
• Working on NMI April 2004 release
– Including new features to support science portal applications
• Open GCE Runtime Engine (OGRE) workflow services and portlets
• Sample template portlets for simple science applications
– Based on aggregation of existing projects
– OGSA/OGSI/GT3.x compatible

OGCE
Consortium
 Future Directions
• Moving towards new portlet standard
– JSR 168
– Should allow compatibility with
• GridSphere
• Academic education and business portals Oncourse, OneStart, CHEF
(of course), and uPortal
• Major commercial products offerings from Sun, IBM, Oracle, etc.
• Incorporate new features:
– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)
– Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions
– DOE SciDAC projects (such as CMCS)
• Expand online repository
– Develop process for third party contributions

OGCE
Consortium
 Future Directions
• Moving towards new portlet standard
– JSR 168
– Should allow compatibility with
• GridSphere
• Academic education and business portals Oncourse, OneStart, CHEF
(of course), and uPortal
• Major commercial products offerings from Sun, IBM, Oracle, etc.
• Incorporate new features:
– Grid Services (CSF, OGSA-DAI portlets, Job Sequencer)
– Access Grid, Polycom and other AV Portlets
• Collaborations and Contributions
– DOE SciDAC projects (such as CMCS)
• Expand online repository
– Develop process for third party contributions

OGCE
Consortium
 Participants

the globus
alliance
University of Texas
at Austin Indiana University

JavaCoG

University of Michigan National Center for Argonne National Labs

Supercomputing Applications

OGCE
Consortium
 References

• OGCE Website
– http://www.ogce.org
• NSF Middleware Initiative
– http://www.nsf-middleware.org

OGCE
Consortium