Cactus Tools for the Grid

Gabrielle Allen

Max Planck Institute for Gravitational Physics,

(Albert Einstein Institute)

CactusCactus

THE GRID: Dependable, consistent, pervasive access

to high-end resources

CACTUS is a freely available, modular,

portable and manageable environment

for collaboratively developing parallel, high-

performance multi-dimensional simulations

www.CactusCode.org

What is CactusWhat is Cactus Flesh (ANSI C) provides code infrastructure (parameter, variable, scheduling

databases, error handling, APIs, make, parameter parsing, ) Thorns (F77/F90/C/C++/Java/Perl/Python) are plug-in and swappable modules

or collections of subroutines providing both the computational instructructure and the physical application. Well-defined interface through 3 config files

Just about anything can be implemented as a thorn: Driver layer (MPI, PVM, SHMEM, …), Black Hole evolvers, elliptic solvers, reduction operators, interpolators, web servers, grid tools, IO, …

User driven: easy parallelism, no new paradigms, flexible Collaborative: thorns borrow concepts from OOP, thorns can be shared, lots

of collaborative tools Computational Toolkit: existing thorns for (Parallel) IO, elliptic, MPI unigrid

driver, Integrate other common packages and tools: HDF5, Globus, PETSc, PAPI,

Panda, FlexIO, GrACE, Autopilot, LCAVision, OpenDX, Amira, ... Trivially Grid enabled!

Current Version Cactus 4.0Current Version Cactus 4.0 Cactus 4.0 beta 1 released September

1999 Community code: Distributed under GNU

GPL Currently: Cactus 4.0 beta 8 Supported Architectures:

SGI Origin SGI 32/64 Cray T3E Dec Alpha Intel Linux IA32/IA64 Windows NT HP Exemplar IBM SP2 Sun Solaris Hitachi SR8000-F NEC SX-5 Mac Linux ...

0

20

40

60

80

100

120

0

20

40

60

80

10

0

12

0

Processors

Sc

alin

g

Origin

NT SC

Cactus Scaling on T3E-600

192

760

5980

47900

100

1000

10000

100000

1 10 100 1000

Number of Processors

Cactus on T3E 600 Total Mflops/sec

Cactus Computational Toolkit: Parallel utilities (thorns) for computational scientistCactus Computational Toolkit: Parallel utilities (thorns) for computational scientist

CactusBase Boundary, IOUtil, IOBasic, CartGrid3D,

IOASCII, Time

CactusBench BenchADM

CactusConnect HTTPD, HTTPDExtra

CactusExample WaveToy1DF77, WaveToy2DF77

CactusElliptic EllBase, EllPETSc, EllSOR, EllTest

CactusPUGH Interp, PUGH, PUGHSlab,

PUGHReduce

CactusPUGHIO IOFlexIO, IOHDF5, IsoSurfacer

CactusIO IOJpeg

CactusTest TestArrays, TestCoordinates,

TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce

CactusWave IDScalarWave, IDScalarWaveC,

IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90

CactusExternal FlexIO, jpeg6b

BetaThorns (In Development) IOStreamedHDF5, Renderer,

IOHDF5Util,…, many more

Cactus CommunityCactus Community DLR

Geophysics(Bosl)

Numerical Relativity CommunityCornell

Crack prop.

NASA NS GC

Livermore

SDSS(Szalay)

Intel

Microsoft

Clemson

“Egrid”NCSA, ANL, SDSC

AEI Cactus Group(Allen)

NSF KDI(Suen)

EU Network(Seidel)

Astrophysics(Zeus)

US Grid ForumDFN Gigabit

(Seidel)

“GRADS”(Kennedy, Foster,

Dongarra, et al)

ChemEng(Bishop)

San Diego, GMD, Cornell

Berkeley

Grid ComputingGrid Computing

AEI Numerical Relativity Group has access to high-end resources in over ten centers in Europe/USA

They want: Bigger simulations, more simulations and faster throughput Intuitive IO at local workstation No new systems/techniques to master!!

How to make best use of these resources? Provide easier access … noone can remember ten usernames, passwords,

batch systems, file systems, … great start!!! Combine resources for larger productions runs (more resolution badly

needed!) Dynamic scenarios … automatically use what is available Remote/collaborative visualization, steering, monitoring

Many other motivations for Grid computing ...

Grand PictureGrand PictureRemote steering and monitoring

from airport

Origin: NCSA

Remote Viz in St Louis

T3E: Garching

Simulations launched from Cactus PortalGrid enabled

Cactus runs on distributed machines

Remote Viz and steering from Berlin

Viz of data from previous simulations in

SF café

DataGrid/DPSSDownsampling

Globus

http

HDF5

IsoSurfaces

Remote VisualizationRemote Visualization

IsoSurfaces and Geodesics

Contour plots(download)

Grid FunctionsStreaming

HDF5

Amira

Amira

LCA Vision

OpenDXOpenDX

Remote SteeringRemote Steering

Remote Viz data

Remote Viz data

XML HTTP

HDF5

Amira

Any Viz Client

Remote Offline VisualizationRemote Offline VisualizationViz Client (Amira)

HDF5 VFD

DataGrid (Globus)

DPSS FTP HTTP

VisualizationClient

DPSS Server

FTP Server

Web Server Remote

Data Server

Downsampling, hyperslabs

Viz in Berlin

4TB at NCSA

Only what is needed

Remote Monitoring: Thorn HTTPDRemote Monitoring: Thorn HTTPD Thorn which allows

simulation any to act as its own web server

Connect to simulation from any browser anywhere … collaborate

Monitor run: parameters, basic visualization, ...

Change steerable parameters

See running example at www.CactusCode.org

Wireless remote viz, monitoring and steering

Remote Viz: OpenDXRemote Viz: OpenDX

Open source, (free), multiplatform, large active development community, easy to program

Reads HDF5 (Cactus) data from file or remotely streamed from Cactus

Simple GUI, select different hyperslabs from 3D data

Also support for streamed ASCII data from Cactus

Remote Viz: IsoViewRemote Viz: IsoView

Lightweight, free, isosurface viewer

Distributed from Cactus website

Displays isosurfaces streamed from Cactus (thorn IsoSurfacer calculates isosurfaces inline with the parallel simulation)

Connect to running simulation on any remote machine

Steer isosurface value from client

Cactus PortalCactus Portal KDI ASC Project Technology: Globus, GSI, Java Beans,

DHTML, Java CoG, MyProxy, GPDK, TomCat, Stronghold

Locates resources, builds/finds executables, central management of parameter files/job output, submit jobs to local batch queues, tracks active jobs

Submission of distributed runs Adding more collaborative features

(eg group accounting, …) Accesses the ASC Grid Testbed

(SDSC, NCSA, Argonne, ZIB, LRZ, AEI)

New Grid ApplicationsNew Grid Applications Dynamic Staging: move to faster/cheaper/bigger machine

“Cactus Worm”

Multiple Universe create clone to investigate steered parameter (“Cactus Virus”)

Automatic Convergence Testing from intitial data or initiated during simulation

Look Ahead spawn off and run coarser resolution to predict likely future

Spawn Independent/Asynchronous Tasks send to cheaper machine, main simulation carries on

Thorn Profiling best machine/queue choose resolution parameters based on queue ….

New Grid Applications (2)New Grid Applications (2) Dynamic Load Balancing

inhomogeneous loads multiple grids

Portal resource choosing simulation launching management

Intelligent Parameter Surveys farm out to different machines

Make use of Running with management tools such as Condor, Entropia, etc. Scripting thorns (management, launching new jobs, etc) Dynamic use of eg MDS for finding available resources

Go!

Dynamic Grid ComputingDynamic Grid Computing

Clone job with steered parameter

Queue time over, find new machine

Add more resources

Found a horizon,try out excision

Look forhorizon

Calculate/OutputGrav. Waves

Calculate/OutputInvariants

Find bestresources

Free CPUs!!

NCSA

SDSC RZG

SDSC

LRZ Archive data

Users ViewUsers View

Cactus Grid DevelopmentCactus Grid Development

Projects: GrADs, KDI Astrophysics Collaboratory, TIKSL/GriKSL, EGrid/GGF, Globus team, AEI

Grid Application Development Toolkit: Application developer should be able to build simulations with tools that easily enable dynamic

grid capabilities e.g. Query information server, Network Monitoring, Simulation instrumenation/Contract specification, Spawning and Migrating, Authentification Server, Data handling, Data transfer, Visualization, Steering, …

Remote visualization, steering, data management User Portal Dynamics Applications

More Information ...More Information ... Cactus:

Web Site: www.CactusCode.org (Documentation/Tutorials etc) Cactus Worm: www.CactusCode.org/Development/Egrid.html

Global Grid Forum (Egrid) www.egrid.org www.gridforum.org

ASC Portal www.ascportal.org

TIKSL Gigabit Computing www.zib.de/Visual/projects/TIKSL/

Black Holes and Neutron Star: Pictures and Movies jean-luc.aei.mpg.de

Any questions: cactus@cactuscode.org