PETSc

Portable, Extensible Toolkit for Scientific computing

Overview Parallel & Sequential Object-oriented Available for virtually all UNIX

platforms, as well as Windows 95/NT

Flexible: many options for solver algorithms and parameters

Motivation Developing parallel, non-trivial PDE

solvers that deliver high performance is still difficult and requires months (or even years) of concentrated effort.

PETSc is a toolkit that can ease these difficulties and reduce the development time

Introduction A freely available and supported research code

Available via http://www.mcs.anl.gov/petsc Free for everyone, including industrial users Hyperlinked documentation and manual pages for all

routines. Many tutorial-style examples Support via email: petsc-maint@mcs.anl.gov Usable from Fortran 77/90, C, and C++

Portable to any parallel system supporting MPI, including

Tightly coupled systems Cray T3E, SGI Origin, IBM SP, HP 9000, Sun Enterprise

Loosely coupled systems, e.g., networks of workstations Compaq, HP, IBM, SGI, Sun, PCs running Linux or Windows

History PETSc history

Begun in September 1991

Keep on development

Now: over 8,500 downloads since 1995 (versions 2.0 and 2.1)

PETSc Concepts How to specify the mathematics of the

problem Data objects

- vectors, matrices How to solve the problem

Solvers

- linear, nonlinear, and time stepping (ODE) solvers Parallel computing complications

Parallel data layout

- structured and unstructured meshes

Structure of PETSc

Computation and Communication KernelsMPI, MPI-IO, BLAS, LAPACK

Profiling Interface

PETSc PDE Application Codes

Object-OrientedMatrices, Vectors, Indices

GridManagement

Linear SolversPreconditioners + Krylov Methods

Nonlinear Solvers,Unconstrained Minimization

ODE Integrators Visualization

Interface

PETSc Numerical Components

CompressedSparse Row

(AIJ)

Blocked CompressedSparse Row

(BAIJ)

BlockDiagonal(BDIAG)

Dense Other

Line Search Trust Region

Newton-based MethodsOther

Nonlinear Solvers

AdditiveSchwartz

BlockJacobi Jacobi ILU ICC

LU(Sequential only) Others

Preconditioners

EulerBackward

EulerPseudo Time

Stepping Other

Time Steppers(For ODE)

GMRES CG CGS Bi-CG-STAB TFQMR Richardson Chebychev Other

Krylov Subspace Methods

MatricesMatrix-free

What is not in PETSc? Discretizations

Unstructured mesh generation and refinement tools

Load balancing tools

Sophisticated visualization capabilities

BUT!PETSc has interface to external software that provides some

of this functionality

Linear solvers (AMG BlockSolve95 LUSOL SPAI SuperLU)

Optimization software (TAO, Veltisto)

Mesh and discretization tools (overture, SAMRAI)

ODE solvers (PVODE)

Others (Matlab, ParMETIS)

Flow of Control for PDE Solution

PETSc codeUser code

ApplicationInitialization

FunctionEvaluation

JacobianEvaluation

Post-Processing

PC KSPPETSc

Main Routine

Linear Solvers (SLES)

Nonlinear Solvers (SNES)

Timestepping Solvers (TS)

A simple PETSc program#include “petsc.h”

int main( int argc, char *argv[] ){ int rank;

PetscInitialize(&argc,&argv,PETSC_NULL,PETSC_NULL); MPI_Comm_rank(PETSC_COMM_WORLD,&rank );

PetscSynchronizedPrintf(PETSC_COMM_WORLD, “Hello World from %d\n”,rank);

PetscFinalize(); return 0;}

Sparse Matrix Computation in PETSc

Variety Data Structures

Variety Preconditioners

Variety Iterative Solvers

Variety interface to External Softwares

Data Structure for sparse matrix

Compressed Sparse Row

Blocked compressed sparse Row

Block Diagonal

Example(BCRS) Block Compressed Sparse Row NR: Number of blocks per side of the

matrix NNZB: Number of Non-Zero Blocks NB: Number of elements per side of the

Block N: Number of elements per side of the

matrix

Preconditioners

Incomplete LU factorization Jacobi, Gauss-Seidel,SOR Schwarz; within blocks use LU, ILU,

SOR, etc.

Solvers Direct method (LU) Krylov method (CG, GMRES, BiCGstab,

CGS, QMR, …) Non-Krylov iterative method (Jacobi, G-

S, SOR) Pre-conditioned Krylov method

Interface to external solver

Both Iterative and Director solvers AMG

BlockSolve95

LUSOL

SPAI

SuperLU

Functions Define the linear system (Ax=b)

MatCreate(), MatSetValue(), VecCreate() Create the Solver

SLESCreate(), SLESSetOperators() Solve System of Equations

SLESSolve() Clean up

SLESDestroy()

Example Solve Ax = b www.cs.ucsb.edu/~jh/example.cc

ComparisonI: Iterative methods provided for different packages

ComparisonII: Preconditioners provided by different packages

ComparisonIII: Performance

Test Matrix:

ComparisonIII: Performance

Feed these matrices to Aztec and PETSc

Generate a zero vector as initial guess and a vector of 1 as RHS

Use GMRES without preconditoners, max iteration = 500

Run on 512-processor Cray T3E900

ComparisonIII: Performance

ComparisonIII: Performance

ComparisonIII: Performance

Aztect suffers from setup varization

PETSc did not optimize partition algorithm( PE256 -> PE512, performance decrease). Communication takes more time than computation

Conclusion PETSc - Well designed and widely used

- First MPI-based program to public - Good set of iterative methods and

preconditioners - Good Support and excellent technical

documents - Still under developement

Thanks

Comments ?