data structures and methods for unstructured distributed ...roystgnr/parmesh_talk.pdf · 1...

PECOSPredictive Engineering and Computational Sciences

Data Structures and Methods for UnstructuredDistributed Meshes

Roy H. Stogner

The University of Texas at Austin

May 23, 2012

Roy H. Stogner Distributed Meshes May 23, 2012 1 / 19

1 Introduction2 Data Structures

Unstructured MeshingDistributed Meshing

3 AlgorithmsParallel UtilitiesParallel Mesh Manipulation

4 ApplicationsPerformanceChallenges

5 Conclusions


Introduction

Context and Motivations

PetscMetis

Mpich

Laspack

LibMesh

STL

RBM

TumorAngiogenesis

CompressibleNS

DD

Elem

NodeElem FaceEdge Cell InfQuad InfCell

Prism Hex Pyramid Tet

Hex8 Hex20 Hex27

DofObject

Node

libMesh design• C++ library classes

I Hybrid parallel FEM calculationsI Unstructured adapted meshesI Abstract base classes like Elem,

FEBaseI Concrete leaf classes like Hex27,

FEHierarchic

ParallelMesh design goals• Eliminate serialized mesh bottleneck

• O (NE/NP ) memory use

• Interchangeable with Mesh (nowSerialMesh)

• Incremental design and testing


Data Structures Unstructured Meshing

Unstructured Mesh Data: Elements, Nodes

• Heap memory allocation formixed geometric, finite elementtypes

• Mesh provides iterators, randomaccess

• Each element provides pointersto neighbors, nodes

• Adaptive refinement addsparents, children

• Boundary adds null neighbors


Data Structures Unstructured Meshing

Unstructured Mesh Data: Degrees of Freedom

Indexing Requirements• Multiple systems of equations on each Mesh

• Multiple variables in each system

• (Varying) multiple components to each variable

• Extra components on some hanging node types

• Second-order geometric nodes give topologicalstorage for higher-order finite elements


Data Structures Distributed Meshing

Mesh Class Hierarchy

MeshBase

+boundary_info: AutoPtr<BoundaryInfo>

+elem(unsigned int)

+node(unsigned int)

+elements_begin/end()

+nodes_begin/end()

+active_local_elements_begin/end()

+read()

+write()

ParallelMesh

#_elements: mapvector<Elem*>

+allgather()

+delete_remote_elements()

UnstructuredMesh

+find_neighbors()

+read/write()

+add/delete_elem()

SerialMesh

#_elements: vector<Elem*>

+elements_begin()

CartesianMesh

Object-Oriented Design• Predicated iterators for

traversing subsets of elements,nodes

• Abstract iterator interface hidesmesh type from mostapplications

• UnstructuredMesh ”branch” formost library code

• ParallelMesh leaf implementsdistributed storage,synchronization



Distributed Mesh Data Structures

SerialMesh• Contiguous vector of pointers to Elem, Node

• elements[i]->id() == i

• O (1) sequential traversal

• O (1) random access

ParallelMesh• map (binary tree) or unsorted map (hash table)

• elements[i]->id() == i

• O (1) (sequential or unsorted) traversal

• O (logNE/NP ) or O (1) random access



Distributed Mesh Data Structures - RemoteElem

• Boundary neighbors still point toNULL

• Neighbors owned by remoteranks are “ghost” elements withfull data

• Neighbors which exist only onremote ranks point toremote elem

• O (1) memory, tests

• Code “walking” past ghostelements triggers runtime errors


Algorithms Parallel Utilities

Utility Functions - Parallel::

MPI interfacestd::vector<Real> send, recv;

...

if (dest processor id == libMesh::processor id() &&

source processor id == libMesh::processor id())

recv = send;

#ifdef HAVE MPI

else

unsigned int sendsize = send.size(), recvsize;

MPI Status status;

MPI Sendrecv(&sendsize, 1, datatype<unsigned int>(),

dest processor id, 0,

&recvsize, 1, datatype<unsigned int>(),

source processor id, 0,

libMesh::COMM WORLD,

&status);

recv.resize(recvsize);

MPI Sendrecv(sendsize ? &send[0] : NULL, sendsize, MPI DOUBLE,

dest processor id, 0,

recvsize ? &recv[0] : NULL, recvsize, MPI DOUBLE,

source processor id, 0,

libMesh::COMM WORLD,

&status);

#endif // HAVE MPI




Parallel:: interfacestd::vector<Real> send, recv;

...

Parallel::send receive(dest processor id, send,

source processor id, recv);

Advantages• Recompilable without MPI• MPI implementation optimizable “under the hood”

I MPI probe for vector lengthsI Asynchronous I/O

• Data type independent

• Container type independent




Parallel:: algorithmstemplate <typename Iterator,

typename SyncFunctor> void

sync dofobject data by id

(const Iterator& range begin,

const Iterator& range end,

SyncFunctor& sync);




Advantages• Generic communication for

Elements, Nodes

• SyncFunctor::act on dataupdates local mesh subset

• Communications algorithm isindependent of functor




Round-Robin• Processor P sends to processorP +K while receiving fromP −K

• Old data discarded after eachstep - O (NG/NP ) in transit

• O (NG) worst-case executiontime




Future optimizations• Precalculate, skip

non-neighboring processors

• Queue multiple asynchronoussends at a time


Algorithms Parallel Mesh Manipulation

Mesh Distribution

Mesh Creation• File 1-to-N input, serial generation

I Remote elements must be removedI One pass: flag nodes on local elementsI Second pass: delete disconnected

elements, replace with remote elem

• File N-to-N input

I Ghost elements, nodes must be gatheredfrom neighboring processors

I Local boundary node id lists sent toadjacent processors

I Ghost nodes, elements received in return

• Parallel generation

I Ids i : i mod (NP + 1) = p are ownedby processor p

I Very algorithm-dependent



Mesh Distribution




• File N-to-N inputI Ghost elements, nodes must be gathered

from neighboring processorsI Local boundary node id lists sent to

adjacent processorsI Ghost nodes, elements received in return

• Parallel generation

I Ids i : i mod (NP + 1) = p are ownedby processor p

I Very algorithm-dependent



Mesh Distribution




• File N-to-N inputI Ghost elements, nodes must be gathered

from neighboring processorsI Local boundary node id lists sent to

adjacent processorsI Ghost nodes, elements received in return

• Parallel generationI Ids i : i mod (NP + 1) = p are owned

by processor pI Very algorithm-dependent



Distributed Degree of Freedom Numbering

One-pass indexing

• Index processor P starting from∑P−1

1 NEp

• Pass∑P

1 NEp to processor P + 1

• O(NE)

Two-pass indexing• Count processor P indices starting from 0

• Gather NEp on all processors

• Re-index processor P from∑P−1

1 NEp

• Trade ghost indices with remote processors

• O(NE/NP ) execution time



Distributed Mesh RefinementError Estimation

• Local residual, jump error estimators

• Refinement-based estimators

• Adjoint-based estimators

• Recovery estimators

Refinement Flagging

• Flagging by error tolerance: η2K < η2

tol/NE

I Embarrassingly parallel

• Flagging by error fraction: ηK < rmaxK ηK

I One global Parallel::max to find maximum error

• Flagging by element fraction or target NE

I Parallel::Sort to find percentile levels?I Binary search in parallel?I TBD




• Local residual, jump error estimators: embarrassingly parallel

• Refinement-based estimators: use solver parallelism

• Adjoint-based estimators: use solver parallelism

• Recovery estimators: require partitioning-aware patch generation

Refinement Flagging


tol/NE

I Embarrassingly parallel

• Flagging by error fraction: ηK < rmaxK ηK

I One global Parallel::max to find maximum error

• Flagging by element fraction or target NE





• Local residual, jump error estimators: embarrassingly parallel

• Refinement-based estimators: use solver parallelism

• Adjoint-based estimators: use solver parallelism

• Recovery estimators: require partitioning-aware patch generation

Refinement Flagging


tol/NE

I Embarrassingly parallel• Flagging by error fraction: ηK < rmaxK ηK

I One global Parallel::max to find maximum error• Flagging by element fraction or target NE




Distributed Mesh Refinement

Elem, Node creation

• Ids i : i mod (NP + 1) = p areowned by processor p

Synchronization• Refinement Flags

I Data requested by idI Iteratively to enforce smoothing

• New ghost child elements, nodes

I Id requested by data

• Hanging node constraint equations

I Iteratively through subconstraints,subconstraints-of-subconstraints...




Elem, Node creation• Ids i : i mod (NP + 1) = p are

owned by processor p



• New ghost child elements, nodes

I Id requested by data

• Hanging node constraint equations

I Iteratively through subconstraints,subconstraints-of-subconstraints...




Elem, Node creation• Ids i : i mod (NP + 1) = p are

owned by processor p



• New ghost child elements, nodesI Id requested by data

• Hanging node constraint equationsI Iteratively through subconstraints,

subconstraints-of-subconstraints...


Applications Performance

“Typical” PDE exampleTransient Cahn-Hilliard, Bogner-Fox-Schmidt quads or hexes

101

102

103

104

103

104

105

106

107

RA

M (

MB

)

Mesh Size (quad elements)

2D SerialMesh vs. ParallelMesh Per-Node Memory Usage

1 Node Serial 2D

1 Node Parallel 2D

2 Node Serial 2D

2 Node Parallel 2D

4 Node Serial 2D

4 Node Parallel 2D

8 Node Serial 2D

8 Node Parallel 2D

16 Node Serial 2D

16 Node Parallel 2D

32 Node Serial 2D

32 Node Parallel 2D

101

102

103

104

103

104

105

106

RA

M (

MB

)

Mesh Size (hex elements)

3D SerialMesh vs. ParallelMesh Per-Node Memory Usage

1 Node Serial 3D

1 Node Parallel 3D

2 Node Serial 3D

2 Node Parallel 3D

4 Node Serial 3D

4 Node Parallel 3D

8 Node Serial 3D

8 Node Parallel 3D

16 Node Serial 3D

16 Node Parallel 3D

32 Node Serial 3D

32 Node Parallel 3D

Results• Parallel codes using SerialMesh are unchanged for ParallelMesh

• Overhead, distributed sparse matrix costs are unchanged

• Serialized mesh, indexing once dominated RAM use


Applications Challenges

Distributed Correlation Length PostprocessingCahn-Hilliard FEM has only semilocal dependencies

(∂c

∂t, φ)Ω =−

(Mc∇f ′0(c),∇φ

)Ω− ε2c

(∆c,∇ ·MT

c ∇φ)

Ω

+((Mc∇

(f ′0(c)− ε2c∆c

))· ~n, φ

)∂Ω

+ ε2c(∆c,MT

c ∇φ · ~n)∂Ω

Correlation lengths involve global integrals

< f(~x) > ≡∫

Ω f(~x) dΩ∫Ω 1 dΩ

r(~y) ≡< c(~x)c(~x+ ~y) > − < c(~x) >2

• Communication of remote data unavoidable

• Communication of remote geometry avoidedvia coarse mesh integration


Applications Challenges

Hypersonic Flow Calculations

Flow, transport, shock capturing, reacting chemistry, thermalnon-equilibrium, turbulence: all passing regression tests onParallelMesh at 10−9 tolerancesSerialMesh-dependent exceptions:Shock fitting, transition integrals


Conclusions

Ongoing Work• DofObject serialization, redistribution

• Adaptive coarsening

• Benchmarking, optimization

• Extend Parallel:: APIs from vector,set→ map

• APIs for integro-differential terms

Any Questions?


data structures and methods for unstructured distributed ...roystgnr/parmesh_talk.pdf · 1...

Documents