scidac tops petsc work
DESCRIPTION
SciDAC TOPS PETSc Work. SciDAC TOPS Developers Satish Balay Chris Buschelman Matt Knepley Barry Smith Hong Zhang Other PETSc Team Members Lois Curfman McInnes, Bill Gropp, Dinesh Kaushik. http://www.mcs.anl.gov/petsc [email protected]. PETSc. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/1.jpg)
1
SciDAC TOPS PETSc Work
http://www.mcs.anl.gov/[email protected]
SciDAC TOPS DevelopersSatish Balay
Chris BuschelmanMatt KnepleyBarry SmithHong Zhang
Other PETSc Team MembersLois Curfman McInnes, Bill Gropp, Dinesh Kaushik
![Page 2: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/2.jpg)
2
PETSc
• Portable, Extensible, Software for the scalable solution of nonlinear PDEs
• Focusing on algebraic solvers– Preconditioners, multigrid, Krylov methods,
Newton, FAS, time integrators
![Page 3: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/3.jpg)
3
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG,SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
![Page 4: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/4.jpg)
4
Derivatives
• Automatic tools ADIC/ADIFOR are great but require manual processing (painful when developing code)
• Would like to have a variety of derivatives “for free” (or nearly)– Matrix element based Newton which requires explicit
sparse Jacobian– Matrix-free Newton– SOR preconditioners and FAS require one equation at
at time
![Page 5: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/5.jpg)
5
To Fully Automate
• Code comments to indicate what needs to be processed
• Predefined calling sequences
• Libraries that can use generated code
• Make system automatically generates required derivative code
![Page 6: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/6.jpg)
6
../* Process adiC: FFL */int FFL (DALocalInfo *info,Field **x, Field **f,void *ptr) { AppCtx *user = (AppCtx*)ptr; ..
DMMGSetSNESLocal(dmmg,FFL,0,ad_FFL,admf_FFL);
typedef struct { PassiveReal lidvelocity,prandtl,grashof; PetscTruth draw_contours;} AppCtx;
typedef struct { double u,v,omega,temp;} Field;
![Page 7: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/7.jpg)
7
> make BOPT=g_c++ ex19Running adiC on function FFLExtracting structure FieldExtracting structure AppCtxExtracting function FFL
adiC -a -d gradient -i /sandbox/bsmith/petsc/bmake/adic.init ex19.c.tmpADIC 1.1 beta 4 08/07/00 18:06:31This software was created in the course of academic and/or researchendeavors by Argonne National Laboratory. See the disclaimer in the
g++ -fPIC -o ex19.o -c -Dad_GRAD_MAX=36 -I/sandbox/bsmith/petsc/include/adic -g -I/sandbox/bsmith/petsc ..
![Page 8: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/8.jpg)
8
Matrix-free Multiply
• Differencing (well understood)
• AD
int mf_FFL(DALocalInfo *info,mf_Field **x, mf_Field **f,void *ptr) {
typedef struct { double value;double grad[1];} mfDERIV_TYPE;typedef struct { mfDERIV_TYPE u, v, omega, temp;}
![Page 9: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/9.jpg)
9
Matrix-free Multigrid Preconditioners
• Krylov smoothing with – no preconditioner– (point block) Jacobi preconditioner– (point block) SOR preconditioning
![Page 10: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/10.jpg)
10
Matrix-free Jacobi/SOR
• If function is provided an “(block) equation at a time” matrix-free multiply can be done an equation at a time– can be computed by providing a seed of – For SOR
• Similar computations with differencing
iiJ ie1
:( )i i ii i iu u J b J u
![Page 11: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/11.jpg)
11
FAS – Full Approximation Scheme (Nonlinear Multigrid)
• Smoothing uses Newton to solve a single (block) equation at a time– Need the same “(block) equation at a time” and
its derivative as for matrix-free Newton!
![Page 12: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/12.jpg)
12
We are developing a common software infrastructure and user interface for many Newton methods and FASwhich can use AD or differencing for derivatives.
![Page 13: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/13.jpg)
13
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG, SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
![Page 14: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/14.jpg)
14
Philosophy on Solvers
• Use any solver without changing the application code
• Add to application code to take advantage of special solver features (optimizations)– but additions cannot break other solvers
PCILUSetDropTolerance(pc,1.e-5);PCSPAISetEpsilon(pc,1.e-2);
![Page 15: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/15.jpg)
15
HYPRE-PETSc Matrix Element based preconditioners
• No vector copies
• Currently a matrix copy (plan to remove)
-pc_type hypre -pc_hypre_type <pilut,parasails,boomerAMG,euclid>-pc_hypre_pilut_tol 1.e-4-pc_hypre_parasails_thresh .1-pc_hypre_parasails_filter .2
![Page 16: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/16.jpg)
16
RAMG, SAMG – PETScSequential Preconditioners
• AMG – written by John Ruge, Klaus Stueben, and Rolf Hempel. A classic scalar PDE AMG code.
• SAMG – written by Klaus Stueben. A complete rewrite that supports systems of PDEs (interface coming soon).– PETSc interfaces provided by Domenico
Lahaye
![Page 17: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/17.jpg)
17
SuperLU_Dist• MPI parallel LU factorization and solve written by Sherry Li and Jim Demmel
• Henry Tufo – Paul Fischer XYT fast direct parallel solver for “small problems”, i.e. the coarse grid problem in multigrid
-mat_aij_superlu_dist-mat_aij_superlu_dist_colperm <MMD_AT_PLUS_A,NATURAL,MMD_ATA,COLAMD>
![Page 18: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/18.jpg)
18
ESI Equation Solver Interface
• A C++ “standard” of abstract classes for setting up and solving linear systems
• At least three implementations– Trilinos (Sandia, NM)– ISIS (Sandia, CA)– PETSc
![Page 19: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/19.jpg)
19
PETSc ESI Goal
• Allow PETSc to use matrices and solvers written by others in ESI
• Allow anyone to use PETSc matrices and solvers from ESI– Achieved by using two sets of wrappers
• PETSc objects “look like” ESI objects
• ESI objects “look like” PETSc objects
![Page 20: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/20.jpg)
20
-is_type esi –is_esi_type Petra_ESI_IndexSpace-vec_type esi -vec_esi_type Petra_ESI_Vector -mat_type esi –mat_esi_type Petra_ESI_CSR_Matrix
PETSc using Trilinos objects
ESI using PETSc objects
esi::Vector<double,int> *v = new esi::petsc::Vector<double,int>(petscvector)esi::Operator<double,int> *o = new esi::petsc::Matrix<double,int>(petscmatrix)
![Page 21: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/21.jpg)
21
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG, SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
![Page 22: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/22.jpg)
22
SIDL/Babel (LLNL)
• Scientific Interface Definition Language
• Babel – generates language stubs for– C, C++, Fortran 77, Python, Java, …
• Provides – Object model– Exception (error) model
• Immature, but very exciting development
![Page 23: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/23.jpg)
23
PETSc 3.0
• SIDL/Babel for inter-language, object model and exception model
• Multiple precisions – Single for preconditioners?
• Vector and matrix sizes mutable– Adaptive mesh refinement
• Not a rewrite, just a rewrap!
![Page 24: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/24.jpg)
24
Activities• Derivative calculations – ADIC/ADIFOR
– Matrix-free preconditioners– FAS (future)
• Accessing other software– HYPRE preconditioner usage– RAMG, SAMG– SuperLU– ESI
• SIDL/Babel/CCA – PETSc 3.0 plans• SIDL/Babel MPI
![Page 25: SciDAC TOPS PETSc Work](https://reader036.vdocuments.mx/reader036/viewer/2022081603/56814491550346895db13091/html5/thumbnails/25.jpg)
25
Conclusions
• Many new tools and solvers
• Accessible via – PETSc 2.0 interface– ESI– (coming) SIDL/Babel/CCA
• We are happy to support users, especially those interested in nontrivial problems