TeraGrid Science Gatewaysin Portland

Nancy Wilkins-DiehrTeraGrid Area Director for

Science Gatewayswilkinsn@sdsc.edu

SC09, November 14-20, 2009

TeraGrid Today

SC09, November 14-20, 2009

Follow-on program to TeraGrid now being competedeXtreme Digital (XD)

The XD Solicitation (NSF 08-571) consists of 5 services:

Current planning grants address the CMS/AUSS/TEOS integrative services.

Service Acronym Funding (all 5 yrs)

Expected Start Date

Coordination and Management CMS $12M/yr April 2011

Advanced User Support AUSS $8M/yr April 2011

Training, Education and Outreach

TEOS $3M/yr April 2011

Technology Audit and Insertion TAIS $3M/yr April 2010

High-Performance Remote Visualization and Data Analysis (2 awards)

RVDAS 2 * $3M/yr Late 2009Mid 2010

SC09, November 14-20, 2009Source: Richard Moore, SDSC

XD as Integrator

•XD will be in a position similar to TeraGrid Grid Infrastructure Group (GIG) and EGI – Deliver common services: e.g. allocations, central

documentation/portal/helpdesk, standard user software environments, networking, advanced user support, training/education/outreach, etc.

– Provide integration function(s) across Resource Providers (RPs), including governance

– Define/implement a high-level integrating system architecture– Provide continuity of services for TeraGrid users – Work with TAIS awardee in an integrated fashion

•XD will not– Deploy its own HPC systems– Prescribe how Resource Providers operate resources

•But will work with RPs to define a common user environment

– Define the science/engineering objectives for users

SC09, November 14-20, 2009Source: Richard Moore, SDSC

TeraGrid/XD Resource Evolution

IU Big RedNCSA Cobalt and Mercury (IA-64)

ORNL NSTGPSC Big Ben

Purdue Brutus and Condor PoolSDSC IA-64

UC/ANL IA-64

IU Big RedNCSA Cobalt and Mercury (IA-64)

ORNL NSTGPSC Big Ben

Purdue Brutus and Condor PoolSDSC IA-64

UC/ANL IA-64

Ending Serviceon or before

3/31/10*

IU QuarryLONI Queen Bee

NCAR Frost NCSA Abe

NCSA LincolnPurdue Steele

PSC PopleSDSC Dash

TACC Lonestar

Track 2CTrack 2D: Data-Intensive Computing

Track 2D: High-Throughput ComputingTrack 2D: Experimental Architecture

Track 2D: Grid TestbedOther resources as designated by NSF

Track 2CTrack 2D: Data-Intensive Computing

Track 2D: High-Throughput ComputingTrack 2D: Experimental Architecture

Track 2D: Grid TestbedOther resources as designated by NSF

Start dates TBD

XD RV-DA Systems TACC Longhorn (~12/09) & NICS Nautilus (~6/10)

XD RV-DA Systems TACC Longhorn (~12/09) & NICS Nautilus (~6/10)

* Check the TeraGrid Resource Catalog at www.teragrid.org for

start/end dates.

To be continued through 3/31/11*

TACC Ranger (Track 2A)NICS Kraken (Track 2B) Continuing

XDTeraGrid

SC09, November 14-20, 2009Source: Richard Moore, SDSC

TeraGrid resources today include:•Tightly Coupled Distributed Memory Systems, 2 systems in the top 10 at top500.org

– Kraken (NICS): Cray XT5, 99,072 cores, 1.03 Pflop– Ranger (TACC): Sun Constellation, 62,976 cores, 579 Tflop, 123 TB RAM

•Shared Memory Systems– Cobalt (NCSA): Altix, 8 Tflop, 3 TB shared memory– Pople (PSC): Altix, 5 Tflop, 1.5 TB shared memory

•Clusters with Infiniband– Abe (NCSA): 90 Tflops– Lonestar (TACC): 61 Tflops– QueenBee (LONI): 51 Tflops

•Condor Pool (Loosely Coupled)– Purdue- up to 22,000 cpus

•Gateway hosting– Quarry (IU): virtual machine support

•Visualization Resources– TeraDRE (Purdue): 48 node nVIDIA GPUs– Spur (TACC): 32 nVIDIA GPUs

•Storage Resources– GPFS-WAN (SDSC)– Lustre-WAN (IU)– Various archival resources

SC09, November 14-20, 2009 Source: Dan Katz, U Chicago

How did the Gateway program develop?A natural result of the impact of the internet on worldwide

communication and information retrieval

SC09, November 14-20, 2009

Only 17 years since the release of Mosaic!

Access to supercomputers hasn’t changed much in 20 years

But the world around them sure has!

SC09, November 14-20, 2009

vt100 in the 1980s and alogin window on Ranger today

SC09, November 14-20, 2009

Why are gateways worth the effort?

•Increasing range of expertise needed to tackle the most challenging scientific problems–How many details do you

want each individual scientist to need to know?•PBS, RSL, Condor•Coupling multi-scale codes•Assembling data from multiple sources

•Collaboration frameworks

SC09, November 14-20, 2009

#! /bin/sh#PBS -q dque#PBS -l nodes=1:ppn=2 #PBS -l walltime=00:02:00#PBS -o pbs.out#PBS -e pbs.err#PBS -Vcd /users/wilkinsn/tutorial/exercise_3../bin/mcell nmj_recon.main.mdl

+( &(resourceManagerContact="tg-login1.sdsc.teragrid.org/jobmanager-pbs") (executable="/users/birnbaum/tutorial/bin/mcell") (arguments=nmj_recon.main.mdl) (count=128) (hostCount=10) (maxtime=2) (directory="/users/birnbaum/tutorial/exercise_3") (stdout="/users/birnbaum/tutorial/exercise_3/globus.out") (stderr="/users/birnbaum/tutorial/exercise_3/globus.err"))

=======# Full path to executableexecutable=/users/wilkinsn/tutorial/bin/mcell

# Working directory, where Condor-G will write # its output and error files on the local machine.initialdir=/users/wilkinsn/tutorial/exercise_3

# To set the working directory of the remote job, we# specify it in this globus RSL, which will be appended# to the RSL that Condor-G generatesglobusrsl=(directory='/users/wilkinsn/tutorial/exercise_3')

# Arguments to pass to executable.arguments=nmj_recon.main.mdl

# Condor-G can stage the executabletransfer_executable=false

# Specify the globus resource to execute the jobglobusscheduler=tg-login1.sdsc.teragrid.org/jobmanager-pbs

# Condor has multiple universes, but Condor-G always uses globusuniverse=globus

# Files to receive sdout and stderr.output=condor.outerror=condor.err

# Specify the number of copies of the job to submit to the condor queue.queue 1

Gateways democratize access to high end resources

•Almost anyone can investigate scientific questions using high end resources–Not just those in the research groups of those who request

allocations–Gateways allow anyone with a web browser to explore

•Opportunities can be uncovered via google–My 11-year-old son discovered nanoHUB.org himself while his class was studying Bucky Balls

•Foster new ideas, cross-disciplinary approaches–Encourage students to experiment

•But used in production too–Significant number of papers resulting from gateways including

GridChem, nanoHUB–Scientists can focus on challenging science problems rather

than challenging infrastructure problemsSC09, November 14-20, 2009

Today, there are approximately 35 gateways using the TeraGrid

SC09, November 14-20, 2009

Not just ease of useWhat can scientists do that they

couldn’t do previously?•Linked Environments for Atmospheric Discovery (LEAD) - access to radar data•National Virtual Observatory (NVO) – access to sky surveys•Ocean Observing Initiative (OOI) – access to sensor data•PolarGrid – access to polar ice sheet data•SIDGrid – expensive datasets, analysis tools•GridChem –coupling multiscale codes

•How would this have been done before gateways?SC09, November 14-20, 2009

What makes a gateway a TeraGrid gateway?

•TeraGrid gateways use TeraGrid resources•Are they all developed by TeraGrid?

–No, we don’t make gateways the gateways you use, we make the gateways you use better

–The strength of the program lies in the development of end user interfaces by those in the community

•TeraGrid does provide staff to assist with gateway use of the resources–Anyone can request support via the same peer review process

used to request CPU hours or a data allocation

SC09, November 14-20, 2009

3 steps to connect a gateway to TeraGrid

•Request an allocation–Only a 1 paragraph abstract

required for up to 200k CPU hours

•Register your gateway–Visibility on public TeraGrid page

•Request a community account–Run jobs for others via your

portal

•Staff support is available!•www.teragrid.org/gateways

SC09, November 14-20, 2009

Tremendous Opportunities Using the Largest Shared Resources -

Challenges too!•What’s different when the resource doesn’t belong just to me?–Resource discovery–Accounting–Security–Proposal-based requests for resources (peer-reviewed access)

•Code scaling and performance numbers•Justification of resources•Gateway citations

•Tremendous benefits at the high end, but even more work for the developers•Potential impact on science is huge

–Small number of developers can impact thousands of scientists

–But need a way to train and fund those developersSC09, November 14-20, 2009

Gateways in the marketplaceKids control telescopes and share images

•“In seconds my computer screen was transformed into a live telescopic view”–“Slooh's users include

newbies and professional astronomers in 70 countries”

•Observatories in the Canary Islands and Chile, Australia coming soon •Over 1.2M images taken since 2003•Increases public support for investment in these facilities

SC09, November 14-20, 2009

Linked Environments for Atmospheric DiscoveryLEAD

•Providing tools that are needed to make accurate predictions of tornados and hurricanes•Data exploration and Grid workflow

SC09, November 14-20, 2009

PolarGrid brings CI to polar ice sheet measurement

SC09, November 14-20, 2009

•Cyberinfrastructure Center for Polar Science (CICPS)–Experts in polar science,

remote sensing and cyberinfrastructure

– Indiana, ECSU, CReSIS

•Satellite observations show disintegration of ice shelves in West Antarctica and speed-up of several glaciers in southern Greenland–Most existing ice sheet

models, including those used by IPCC cannot explain the rapid changes

•Components of PolarGrid– Expedition grid consisting of ruggedized laptops in a field grid linked to

a low power multi-core base camp cluster– Prototype and two production expedition grids feed into a 17 Teraflops

"lower 48" system at Indiana University and Elizabeth City State (ECSU) split between research, education and training.

– Gives ECSU (a minority serving institution) a top-ranked 5 Teraflop high performance computing system

•Access to expensive data•TeraGrid resources for analysis

– Large level 0 and level 1 data sets require once and done processing and storage

– Filters applied to level 1 data by users in real time via the web

•Student involvement

SC09, November 14-20, 2009Source: Geoffrey Fox

Social Informatics Data GridCollaborative access to large, complex datasets

•SIDGrid is unique among social science data archive projects–Streaming data which

change over time•Voice, video, images (e.g. fMRI), text, numerical (e.g. heart rate, eye movement)

– Investigate multiple datasets, collected at different time scales, simultaneously•Large data requirements•Sophisticated analysis tools

SC09, November 14-20, 2009

http://www.ci.uchicago.edu/research/files/sidgrid.mov

Viewing multimodal data like a symphony conductor

•“Music-score” display and synchronized playback of video and audio files– Pitch tracks– Text– Head nods, pause, gesture

references

•Central archive of multi-modal data, annotations, and analyses– Distributed annotation efforts by

multiple researchers working on a common data set•History of updates

•Computational tools– Distributed acoustic analysis using

Praat– Statistical analysis using R– Matrix computations using Matlab

and Octave

SC09, November 14-20, 2009

Source: Studying Discourse and Dialog with SIDGrid, Levow, 2008

Future Technical Areas•Web technologies change fast

– Must be able to adapt quickly

•Gateways and gadgets– Gateway components

incorporated into any social networking page

– 75% of 18 to 24 year-olds have social networking websites

•iPhone apps?•Web 3.0

– Beyond social networking and sharing content

– Standards and querying interfaces to programmatically share data across sites

•Resource Description Framework (RDF), SPARQL

SC09, November 14-20, 2009

Gateways can further investments in other projects

•Increase access–To instruments, expensive data collections

•Increase capabilities–To analyze data

•Improve workforce development–Can prepare students to function in today’s cross-disciplinary

world

•Increase outreach•Increase public awareness

–Public sees value in investments in large facilities–Pew 2006 study indicates that half of all internet users have been

to a site specializing in science–Those who seek out science information on the internet are more

likely to believe that scientific pursuits have a positive impact on society

SC09, November 14-20, 2009

Where are we going with the program next?

•New gateways receiving support this year–Connecting climate gateways

•Community Climate System Model (CCSM) and Earth System Grid (ESG)

–Arroyo

•Prebuilt VMs with gateway software•Better visibility of apps available through gateways in addition to those available at the command line on TeraGrid•More example-based documentation

–Less talk, more action

•Improved use of remote vis resources by gateways•Continued targeted support of an ever-changing portfolio of projects–Peer-reviewed requests for assistance

•Helpdesk support expanded

SC09, November 14-20, 2009

Tremendous Potential for Gateways

•In only 17 years, the Web has fundamentally changed human communication•Science Gateways can leverage this amazingly powerful tool to:–Transform the way scientists collaborate–Streamline conduct of science– Influence the public’s perception of science

•Reliability, trust, continuity are fundamental to truly change the conduct of science through the use of gateways–High end resources can have a profound impact

•The future is very exciting!

SC09, November 14-20, 2009

SC09, November 14-20, 2009

Thank you for your attention!Questions?

Nancy Wilkins-Diehr, wilkinsn@sdsc.eduwww.teragrid.org