the charm++ parfum framework parallel framework for unstructured meshing presented by: isaac dooley...

The Charm++ ParFUM FrameworkPARallel Framework for Unstructured Meshing

presented by:

Isaac DooleyParallel Programming Lab

University Illinois Urbana-Champaign

http://charm.cs.uiuc.edu

ParFUM Overview

Charm Workshop 2005 3

Why use the ParFUM Framework?

• Publicly Available (included with Charm++)

• Makes parallelizing a serial code faster and easier

• Handles mesh partitioning

• Handles communication

• Handles load balancing (via Charm++)

• Bindings for Fortran, C++, C

• Provides Advanced features

• Parallel Adaptivity and Incremental Mesh Modification

• Visualization Support with ParaView or NetFEM

• Collision Detection Library

• Iterative Matrix Solver Library (IFEM)


Features Overview• Originally Designed for Finite Element/Volume Problems • Parallel Partitioning • Field registration and updating for shared nodes, ghosts -- “User can own Data”•Arbitrary element types, and arbitrary user data associated to elements or nodes• Boundary condition representation for faces, edges and nodes• Efficient ID translation mechanism for communication•Mesh Modification Support

3-D Fractography in ParFUM

Rocket Burn Simulation, CSAR


ParFUM Framework Users• CSAR

• Rocflu: Fluids solver• RocRem: Remeshing / Parallel Solution Transfer

• CPSD• SpaceTime meshing

• Frac3D• Fracture Mechanics

• Dendritic Growth• Metal Solidification process


Outline for the rest of this talk

• ParFUM concepts & philosophy• Program Structure• Synchronization of Ghost & Shared entities• Topological Relationships (Adjacencies)• Parallel Mesh Adaptivity• Obtaining & Compiling ParFUM• Libraries within ParFUM

• Collision Detection• Visualization Tools

• One Sample Program

ParFUM Concepts


ParFUM Basics

• Nodes• Points in the problem domain’s space.

• Elements• Discretized areas or volumes representing portions of a

problem domain• Arbitrary Dimension and Type• Each element is defined by a set of nodes

• Data Attributes, can be associated with nodes or Elements

• Data can be owned by either Framework or Application


Serial Mesh

N5N4N2E3

N4N2N1E2

N4N3N1E1

Surrounding Nodes

Element


Partitioning

• Partition the ParFUM Mesh into multiple chunks

• Distribute elements, replicate shared nodes and/or add ghosts• Keep track of communication

• Partition so that communication is minimized


Partitioned Mesh

N3N2N1E2

N4N3N1E1

Surrounding Nodes

Element

N3N2N1E1

Surrounding Nodes

Element Shared Nodes

N3N4

N1N2

BA


ParFUM Parallel Model: Shared Nodes

• “Shared Node” model• Element computations based on values of

surrounding nodes• Node values are sum of surrounding elements

• Example: Mechanical Simulation• Element stresses are computed from element’s

nodal displacements


ParFUM Parallel Model: Ghosts

• “Ghost” model• Element computations based only on values of

surrounding nodes and elements

• Example: Fluid Dynamics• Element pressures and velocities come from

neighboring element pressures and velocities, and node locations


Virtualization• Charm++ Runtime System

• Applications built using migratable objects

• Each ParFUM mesh chunk mapped to a migratable object.

• Virtualization = multiple migratable objects per processor

• Load Balancing

• High(90-100%) Processor Utilization and Scaling.

System Implementation

User View


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1 2 4 8 16 32 64 128 256 512 1024 2048

Migratable Objects Per Processor

Time (Seconds) per Iteration

ParFUM Application On Eight Physical Processors

Benefit of Virtualization to Structural Dynamics Applications

ParFUM Program Structure


• Consists of at least two user-written subroutines• init• driver

• init is called on chunk 0

• driver is called on every chunk

Structure of an ParFUM Application


init()

subroutine init read the serial mesh and configuration data inform the framework about the mesh end subroutine


driver()

subroutine driver get local mesh chunk time loop ParFUM computations communication more ParFUM computations end time loop end subroutine


Structure of an ParFUM Application

init()

Update Update Update

driver driver driver


• Framework Owns Data• User gives framework mesh in init using

FEM_Mesh_Data()

• User Accesses mesh by calling FEM_Mesh_Data() in driver to copy data into the user’s arrays

• User Owns Data• User registers mesh with framework in init using

FEM_Register_entity() and FEM_Register_array()

• User uses their own data arrays in driver

• User must also supply a resize function, to be called when framework partitions mesh and resizes mesh in adaptive refinement calls.

Data Management In ParFUM


void Mesh_data(

int mesh,

int entity,

int attr,

void *data,

int first,

int length,

int datatype,

int width

);

Get/Set, and multi-mesh supportGet/Set, and multi-mesh support

NODE, ELEM, SPARSE (+GHOST)NODE, ELEM, SPARSE (+GHOST)

DATA, CONN, SYM,GLOBALNO,…DATA, CONN, SYM,GLOBALNO,…

User data: width x length arrayUser data: width x length array

Apply to fir st…first+length-1Apply to fir st…first+length-1

User data formattingUser data formatting

Setting/Getting Mesh Data


Mesh_data(

FEM_Mesh_default_read()FEM_Mesh_default_read(),FEM_NODE,FEM_DATA+23, coord, 0,nNodes,

FEM_DOUBLE,3);

Mesh Access: Example 1

Read from default meshRead from default mesh

Access nodal dataAccess nodal data

Some user defined attribute #23Some user defined attribute #23

User data: 3 x nNodes arrayUser data: 3 x nNodes array

Get data for nodes 0 to nNodesGet data for nodes 0 to nNodes

3 doubles for each node3 doubles for each node


Mesh_data(

FEM_Mesh_default_write()FEM_Mesh_default_write(),FEM_ELEM,FEM_CONN, conn, 0,nElem,

FEM_INDEX_0, 3);


Write to default meshWrite to default mesh

Modify element dataModify element data

Element ConnectivityElement Connectivity

Connectivity Array: 3 x Connectivity Array: 3 x nElemnElem

For elements 0 to nElemFor elements 0 to nElem

3 integers for each element3 integers for each element

(0-based C-style indexing)(0-based C-style indexing)


Mesh_data(

FEM_Mesh_default_read()FEM_Mesh_default_read(),

FEM_ELEM+FEM_GHOST,FEM_CONN, conn, 0,nElem, FEM_INDEX_0,3

);


Read from default meshRead from default mesh

Access ghost element dataAccess ghost element data

ParFUM Ghost Elements


Ghost Elements: Overview

• Most ParFUM programs communicate via shared nodes

• Some computations require read-only copies of remote elements—“ghosts”• Stencil-type finite volume computation• Many kinds of mesh modification


Ghosts: 2D Example

1 2 3 4

1 2Ghostof3

Ghost of2

3 4

Serial Mesh

Left Chunk Right Chunk


Defining Ghost Layers:

• Add ghost elements layer-by-layer from init

• A chunk will include ghosts of all the elements it is connected to by “tuples”—sets of nodes

• For 2D, a tuple might be a 2-node edge

• For 3D, a tuple might be a 4-node face

• You specify a ghost layer with Add_ghost_layer(tupleSize,ghostNodes)

• ghostNodes indicates whether to add ghost nodes as well as ghost elements.


Ghosts: Node adjacency

0

1 2

/* Node-adjacency: triangles have 3 nodes */

FEM_Add_ghost_layer(1,0); /* 1 node per tuple */

const static int tri2node[]={0,1,2};

FEM_Add_ghost_elem(0,3,tri2node);


Ghosts: Edge adjacency

/* Edge-adjacency: triangles have 3 edges */

FEM_Add_ghost_layer(2,0); /* 2 nodes per tuple */

const static int tri2edge[]={0,1, 1,2, 2,0};

FEM_Add_ghost_elem(0,3,tri2edge);

0

1 2

ParFUM Ghosts & Shared Node Synchronization


Node Fields

• Framework handles combining data for shared nodes and keeps them in sync

• Framework does not understand meaning of node fields, only their location and types

• Framework needs to be informed of locations and types of fields

• Create_field once, Update_field every timestep


Create a Field

integer function FEM_Create_simple_field(datatype, len)

integer, intent(in) :: datatype, len


Update Field: Shared Nodes

subroutine FEM_Update_Field(fid,nodes) integer, intent(in) :: fid varies, intent(inout) :: nodes

Can be used to sum forces contributed to a node in FE Method


Update Field: Ghosts

subroutine Update_ghost_field(fid,elType,elts) integer, intent(in) :: fid,elType varies, intent(inout) :: elts

Can be used to update ghost copies of local nodes(with current displacements) in FE Method

Topological Relationships:Adjacencies


Adjacencies

• User registers element to node connectivity• Framework can derive other relationships:

• Element to Element

• Node to Node

• Node to Element

• User specifies a “tuple” to specify when two arbitrary elements are adjacent. The framework uses this to construct the other adjacencies if needed.


Adjacencies

const int triangleFaces[6] = {0,1,1,2,2,0};FEM_Add_elem2face_tuples(mesh,0,2,3,triangleFaces);

FEM_Mesh_create_elem_elem_adjacency(mesh);FEM_Mesh_create_node_elem_adjacency(mesh);FEM_Mesh_create_node_node_adjacency(mesh);

• User must explicitly have framework generate the adjacency tables


Adjacencies

Get a list of elements adjacent to element e:

e2e_getAll(int e, int *neighbors);

Get a list of elements adjacent to node n:

n2e_getAll(int n, int **adjelements, int *sz);

• The adjacency tables are created as attributes which can be accessed via Mesh_Data() or by simple accessor functions.


Adjacencies

• Caveats or Limitations:• Currently some of the adjacency functions only work

correctly for meshes containing one type of element.

• Two nodes are adjacent if and only if the two nodes are listed in the connectivity for some element. Thus in a rectangular grid, two nodes on opposite corners of a rectangle will be listed as adjacent nodes.

How Do I get a copy of ParFUM and

Run a ParFUM Application?


ParFUM or FEM?• FEM is the old name for ParFUM• FEM and ParFUM are currently the same library in

Charm++• The FEM framework as described in the manual

does not contain some new features:• Adjacency Data Structures, Adaptivity, Mesh

Modification

• The codebase currently uses the old naming conventions of FEM, functions start with “FEM_”

• The current version of FEM/ParFUM can no longer run apart from Charm++


Where to Get It ?ParFUM is included in Charm++ CVS distribution

CSH:setenv CVSROOT ":pserver:[email protected]:/cvsroot"Or BASH:export CVSROOT=":pserver:[email protected]:/cvsroot"

You should now be able to do a> cvs login(no password needed, just type [Enter] at prompt)

and then> cvs co -P charm

ParFUM-FEM is in charm/src/libs/ck-libs/fem


How to Build It ?

> cd charm

and do

> ./build LIBS net-linux -O

This will make a net-linux directory, with bin, include, lib etc subdirectories.


How to Compile & Link ?

• Use “charmc”: available under bin• a multi-lingual compiler driver, understands f90

• Knows where modules and libraries are

• Portable across machines and compilers

• Linking• use “-language femf” : for F90

• Use “–language fem” : for C/C++

• See example Makefiles• charm/examples/fem/…


How to Run ?

• Charmrun• A portable parallel job execution script• Specify number of processors: +pN• Specify number of chunks: +vpN• Special “nodelist” file for net-* versions

./charmrun ./pgm +vp100 +p16


Example

./charmrun pgm +p4 +vp70

Nodelist File: $(HOME)/.nodelist

group main ++shell ssh host tur0001.cs.uiuc.edu host tur0002.cs.uiuc.edu host tur0003.cs.uiuc.edu host tur0004.cs.uiuc.edu

ParFUM Parallel Mesh Adaptivity


Mesh Adaptivity in ParFUM

• 2D refinement

• 2D coarsening

• 3D refinement

• 3D coarsening -- coming soon

• Support for user defined mesh operations.


Mesh Modification Primitives• ParFUM provides incremental asynchronous parallel

mesh modificiation primitives:• Add_node()• Remove_node()• Add_element()• Remove_element()

• Higher Level Operations are built using these primitives• edge_bisect()• edge_flip()• edge_collapse()• …

Mesh Adjacencye2e,e2n,n2e,n2n

Generate, Modify …

Mesh ModificationLock(),Unlock()

Add/Remove Node()Add/Remove Element()

Nodes:Local, Shared, Ghost

Elements:Local,Ghost

SDG Application API (serial or parallel)

ParFUM Structure

Mesh AdaptivityEdge Flip, Edge Bisect,

Edge Contract, …

ParFUM


Mesh Modification Examples

Edge Flip:

Remove elements e1

Remove element e2

Add element (n1,n2,n4)



Mesh Modification Examples

Edge Bisect:

Remove elements e1

Remove element e2

Add node






Mesh Modification in Parallel

Mesh on Processor 1 before edge flip

Mesh on Processor 2 before edge flip

Mesh on Processor 2 after edge flip


Mesh Modification in ParFUM

Primitive Operations must do the following:

•Perform the operation on local and all applicable remote processors•Convert local nodes to shared nodes when they become part of the new boundary•Update ghost layers(nodes and elements) for all applicable processors. The ghost layers can grow or shrink


Parallel Mesh Refinement/Coarsening

Parallel Refinement and Coarsening

algorithms in action:

A wave propagates through a block

across 8 processors


Parallel 3D Refinement• Built from same primitives• Longest edge based refinement• Longest face based refinement• Currently implementing 3D coarsening


Refinement and Coarsening in a 2D ParFUM Application

Shock propagation and reflection down the length of the bar

Adaptive mesh modification to capture the shock propagation


Solution Comparison

Initial Mesh

Adaptive Mesh

Fine Mesh

ParFUM Collision Detection


Charm++ Collision Detection

Detect collisions/intersections/contacts between objects scattered across processors

Built on Charm++ Arrays Overlay regular 3D sparse grid of voxels (boxes) Send objects to all voxels they touch Collect collisions from each voxel

Collision response is left to caller

ParFUM Mesh Visualization


Mesh Visualization in ParFUM

• Currently we support:• NetFEM Online Visualization• NetFEM Offline Visualization• ParaView Offline Visualization


NetFEM Client: pretty picturesWave dispersion off a crack (simplified

frac3d)


ParFUM Unified Visualization Interface

n=NetFEM_Begin(…);

NetFEM_Nodes(…);NetFEM_Vector(…);NetFEM_Vector(…);

NetFEM_Elements(…);NetFEM_Scalar(…);NetFEM_Scalar(…);NetFEM_Scalar(…);

NetFEM_End(n);

Start output for one timestep, specifying Output method(online/offline)

Specify number of nodes and coord.Give an array of nodal vector dataGive a second array of nodal data

Specify number of elements and conn.Give an array of scalar element dataGive a second array of element dataGive a third array of element data

NetFEM_End(n);


NetFEM n=NetFEM_Begin(2,t,NetFEM_POINTAT); NetFEM_Nodes(n,nnodes,(double *)g.coord,"Position (m)");NetFEM_Vector(n,(double *)g.d,"Displacement (m)");NetFEM_Vector(n,(double *)g.v,"Velocity (m/s)");

NetFEM_Elements(n,nelems,3,(int *)g.conn,"Triangles");NetFEM_Scalar(n,g.S11,1,"X Stress (pure)");NetFEM_Scalar(n,g.S22,1,"Y Stress (pure)");NetFEM_Scalar(n,g.S12,1,"Shear Stress (pure)");

NetFEM_End(n);

ParFUM Unified Visualization Interface


NetFEM Online Visualization

• To allow the NetFEM client to connect, you add NetFEM registration calls to your server• Register nodes and element types• Register data items: scalars or spatial vectors

associated with each node or element• You provide the display name and units for each data

item• Link your program with “-module netfem”• Run with “++server”, and connect!


NetFEM: Setup

• n=NetFEM_Begin(FEM_My_partition(),timestep, dim,NetFEM_POINTAT)• Call this each time through your timeloop; or skip• timestep identifies this data update• dim is the spatial dimension—must be 2 or 3• Returns a NetFEM handle n used by everything else• NetFEM_POINTAT for online visualization

• NetFEM_End(n)• Finishes update n


NetFEM: Nodes

• NetFEM_Nodes(n,nnodes,coord,”Position (m)”)• Registers node locations with NetFEM—future vectors and scalars will be

associated with nodes

• n is the handle returned by NetFEM_Begin

• nnodes is the number of nodes

• coord is a dim by nnodes array of doubles

• The string describes the coordinate system and meaning of nodes

• Currently, there can only be one call to nodes


NetFEM: Node Displacement


NetFEM: Elements

• NetFEM_Elements(n,nelem,nodeper, conn,”Triangles”)• Registers elements with NetFEM—future vectors and scalars will be

associated with these elements• n is the handle returned by NetFEM_Begin• nelem is the number of elements• nodeper is the number of nodes per element• conn is a nodeper by nelem array of node indices• The string describes the kind of element

• Repeat to register several kinds of element • Perhaps: Triangles, squares, pentagons, …


NetFEM: Element Stress


NetFEM: Vectors• NetFEM_Vector(n,val,”Displacement (m)”)

• Registers a spatial vector with each node or element• Whichever kind was registered last

• n is the handle returned by NetFEM_Begin• val is a dim by nitems array of doubles

• There’s also a more general NetFEM_Vector_field in the manual

• The string describes the meaning and units of the vectors

• Repeat to register multiple sets of vectors• Perhaps: Displacement, velocity, acceleration, rotation,

…


NetFEM: Element Stress


NetFEM: Element Velocity


NetFEM: Zoom in


NetFEM: Outline Elements


Offline Visualization with NetFEM

• Call NetFEM_Begin() with NetFEM_WRITE,

• ParFUM application will then write out binary output dump files to a new directory called “NetFEM” containing directories for each timestep

• Visualize with command netfem NetFEM/10

• In offline mode, the ``update'' button fetches the next extant timestep directory.


Offline Visualization with ParaView

• The directory NetFEM can be converted to a ParaView/VTK compatible XML format.

• Converter can be built by calling “make” in charm/tmp/libs/ck-libs/netfem/ParaviewConverter/

• Run the resulting program NetFEM_To_Paraview to convert the NetFEM directory to a new directory called ParaViewData.

• ParaView can open the files inside this directory either as individual chunks, or as the entire mesh.

• Scan through the timesteps using the standard timestep slider.


Offline Visualization with ParaView• Limitations:

• Multiple element types with varying numbers of data attributes may not work

• Current converter produces ASCII XML files, not binary, so they use excessive disk space.

• Benefits:• Advanced visualization functionality• More stable than offline NetFEM which crashes if

you try to access non-existent timesteps, or huge meshes

Migration and Load Balancing


Advanced API: Migration

• Chunks may not be computationally equal: Results in load imbalance

• Multiple chunks per processor

• Chunks cannot have writable global data

• Automatic load balancing• Migrate chunks to balance load

• How to migrate allocated data for chunks ?• Embed it in a user-defined type


Chunk Data Example

MODULE my_block_mod TYPE my_block INTEGER :: n1,n2x,n2y REAL*8, POINTER, DIMENSION(:,:) :: arr END TYPE END MODULE


Pack/Unpack (PUP) RoutineSUBROUTINE pup_my_block(p,m) USE my_block_mod USE pupmod INTEGER :: p TYPE(my_block) :: m call fpup_int(p,m%n1) call fpup_int(p,m%n2x) call fpup_int(p,m%n2y) IF (fpup_isUnpacking(p)) THEN ALLOCATE(m%arr(m%n2x,m%n2y)) END IF call fpup_doubles(p,m%arr,m%n2x*m%n2y) IF (fpup_isDeleting(p)) THEN DEALLOCATE(m%arr) END IF END SUBROUTINE


Registering Chunk Data

!- Fortran driver subroutine use my_block_mod interface subroutine pup_my_block(p,m) use my_block_mod INTEGER :: p TYPE(my_block) :: m end subroutine end interface

TYPE(my_block) :: m

CALL FEM_Register(m,pup_my_block)


Migration

• Every chunk driver calls FEM_Migrate()• Framework calls PUP

• for getting the size of packed data• For packing data

• Chunk migrates to new processor• Framework calls PUP for unpacking• Driver returns from FEM_Migrate()

A sample ParFUM Program


Simple2D

• Can be found in charm/examples/fem/simple2D

• Generates simple but interesting visualizations

• A constrained triangle Finite Element Code

• Here we will show the init and driver routines, almost exactly copied from pgm.C, omitting some comments and trivial code fragments.


extern "C" void init(void){ CkPrintf("init started\n"); double startTime=CmiWallTimer(); int nPts=0; //Number of nodes vector2d *pts=0; //Node coordinates int nEle=0; connRec *ele=NULL;

// Omitted for readability:// Read in mesh from users files to ele and pts

// Tell framework we will be writing to the mesh int fem_mesh=FEM_Mesh_default_write();


FEM_Mesh_data(fem_mesh, // Add nodes to the current meshFEM_NODE, // We are registering nodesFEM_DATA+0, // Register the point locations which(double *)pts, // The array of point locations

0, // First node is 0 nPts, // The number of points FEM_DOUBLE, // Coordinates are doubles 2); // Points have dimension 2 (x,y)

FEM_Mesh_data(fem_mesh, // Add elements to the current mesh FEM_ELEM+0, // Element type 0 FEM_CONN, // Register the connectivity table for this

(int *)ele,// The array of point locations 0, // 0 based indexing

nEle, // The number of elements FEM_INDEX_0,// We use zero based node numbering 3); // Elements have three nodes


delete[] ele; delete[] pts; CkPrintf("Finished with init\n");}


extern "C" void driver(void){ int nnodes,nelems,ignored; int i, myId=FEM_My_partition(); myGlobals g;

FEM_Register(&g,(FEM_PupFn)pup_myGlobals); int mesh=FEM_Mesh_default_read();

// Get node data nnodes=FEM_Mesh_get_length(mesh,FEM_NODE);

g.coord=new vector2d[nnodes];

FEM_Mesh_data(mesh, FEM_NODE, FEM_DATA+0, (double*)g.coord, 0, nnodes, FEM_DOUBLE, 2);


// Read element data from framework

nelems=FEM_Mesh_get_length(mesh,FEM_ELEM+0);

g.nnodes=nnodes; g.nelems=nelems; g.conn=new connRec[nelems]; g.S11=new double[nelems]; g.S22=new double[nelems]; g.S12=new double[nelems];

FEM_Mesh_data(mesh, FEM_ELEM+0, FEM_CONN, (int *)g.conn, 0, nelems, FEM_INDEX_0, 3);


//Initialize associated data g.R_net=new vector2d[nnodes]; //Net force g.d=new vector2d[nnodes];//Node displacement g.v=new vector2d[nnodes];//Node velocity g.a=new vector2d[nnodes];//Node acceleration

for (i=0;i<nnodes;i++) g.R_net[i]=g.d[i]=g.v[i]=g.a[i]=vector2d(0.0);

int fid=FEM_Create_simple_field(FEM_DOUBLE,2);


int tSteps=5000;

for (int t=0;t<tSteps;t++) {//Structural mechanics //Compute forces on nodes exerted by elementsCST_NL(g.coord,g.conn,g.R_net,g.d,matConst,nnodes,nelems,g.S11,g.S22,g.S12);

//Communicate net force on shared nodesFEM_Update_field(fid,g.R_net);

//Advance node positionsadvanceNodes(dt,nnodes,g.coord,g.R_net,g.a,g.v,g.d,0);

/* perform migration-based load balancing */ if (t%1024==0) FEM_Migrate();


if (t%1024==0) { //Publish data for Visualization NetFEM n=NetFEM_Begin(FEM_My_partition(),t, 2,NetFEM_POINTAT); NetFEM_Nodes(n,nnodes,(double*)g.coord,"Position (m)"); NetFEM_Vector(n,(double*)g.d,"Displacement (m)"); NetFEM_Vector(n,(double*)g.v,"Velocity (m/s)"); NetFEM_Elements(n,nelems,3,(int*)g.conn,"Triangles"); NetFEM_Scalar(n,g.S11,1,"X Stress (pure)"); NetFEM_Scalar(n,g.S22,1,"Y Stress (pure)"); NetFEM_Scalar(n,g.S12,1,"Shear Stress (pure)"); NetFEM_End(n); }

} // end timeloop}// end driver

Conclusions


Conclusions: ParFUM

• ParFUM is a functional, freely available Unstructured Mesh Framework.

• ParFUM has a wide range of useful features:• Load Balancing, Fault Tolerance, …• Parallel Partitioning• Parallel Incremental Adaptivity, Remeshing, and

Refinement• Visualization Tools• Collision/Contact Detection Library• Matrix Solver

The EndThanks

the charm++ parfum framework parallel framework for unstructured meshing presented by: isaac dooley...

Documents