A.C. Bauer, M.S. Shephard, E. Seol and J. Wan, (acbauer@scorec.rpi.edu)

Scientific Computation Research Center Rensselaer Polytechnic Institute, Troy, NY 12180

Outline: TSTT simulation infrastructure development Structures to support the construction of adaptive loops Mesh modification tools for adaptive mesh control Adaptive loop for DOE accelerator modeling code Adaptive loop for evolving geometry problem solved using commercial

analysis engine

The TSTT Interface for Mesh-Based Simulations

Terascale Simulation Tools & Technologies (TSTT) Center

Goal: To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales Adaptive methods Composite or hybrid solution strategies High-order discretization strategies

Barrier: The lack of easy-to-use interoperable meshing, discretization and adaptive tools requires too much software expertise by application scientists

Organizations: DOE Labs: Argonne, Brookhaven, Lawrence Livermore, Oak Ridge, Pacific

Northwest, Sandia Universities: Rensselaer Polytechnic Institute, SUNY Stony Brook, University of

British Columbia Project Managers:

David Brown: dlb@llnl.gov Lori Freitag Diachin: diachin2@llnl.gov Jim Glimm: glimm@ams.sunysb.edu

More information: www.tstt-scidac.org

TSTT Interoperability Goal

Accomplishing this goal requires: An abstract data model that encompasses a

broad spectrum of mesh types and usage scenarios

A set of common interfaces● Implementation and data structure

neutral● Small enough to encourage adoption● Flexible enough to support a broad

range of functionality

Accomplishing this goal requires: An abstract data model that encompasses a

broad spectrum of mesh types and usage scenarios

A set of common interfaces● Implementation and data structure

neutral● Small enough to encourage adoption● Flexible enough to support a broad

range of functionality

GeometryInformation(Level A)

Full GeometryMeshes(Level B)

MeshComponents(Level C)

To provide interchangeable and interoperable access to different meshmanagement and discretization strategies Ease experimentation with different

technologies Combine technologies together for hybrid

solution techniques

A different way of thinking about simulation code development

TSTT Simulation Infrastructures

7 Interacting structures Mathematical problem definition

1) Space/time domain ***2) Physical and mathematical model attributes

First level of discretization3) Mesh ***

Second level of discretization4) Discretization operators acting on mathematical equations and degrees-of-

freedom (dof) Application of #4 over mesh entities

5) Algebraic system contributors Solving for dof values

6) Algebraic system Description of the physical parameters

7) Fields ***

*** Main TSTT focus to date

Space/Time Domain

Overall domain definition - geometric model and timedomain Want a “true definition” of the domain - not a specific

approximation Define other structures in relation to the domain definition

Problem attributes defined over portions of the domain Mesh represents discretization of domain Fields are discretizations of physical tensors defined over the

domain

At SCOREC, direct geometric model access via topology Shape independent abstraction Need general combination of solids, surfaces and curves Non-manifold topology representations known Effectively supported by commercial solid modelers

Topology built from topology in modeling source Geometric queries passed through modeler API - supports

auto-meshing

Attributes

Problem definition attributes Governing equations, material properties, loads, b.c.’s

Solution strategy attributes Meshing parameters, PDE discretization, basis functions, etc.

Attributes applied to geometric model

Problem definition attributes: Are tensors Have distributions Have relationships

Classification used to retrieve attributes

TSTT Mesh Data Model

Data model consists of entities and entity sets Entity definition

Unique topology Canonical ordering defines adjacency

relationships Entity set definition

Arbitrary grouping of TSTT mesh entities There is a single “Root Set” Relationship among entity sets

Contained-in Hierarchical

Mesh Definition Simple meshes are a set of connected,

non-overlapping entities related through adjacency relationships

Meshes can contain collections of simple meshes

TSTT Mesh Interface Functionality

Core functionality The smallest set of functions required to be considered TSTT compliant –

extensions for implementation of advanced functionality Provides basic access to vertex coordinates and adjacency information

Mesh loading and saving Accessing global information such as the root set, geometric dimension, number of

entities of a given type or topology Primitive array access for entity sets Global entity arrays for entity sets

Traversal functionality Advanced traversal functions provide additional flexibility and performance

Single Entity Traversal Iterator functionality (init, next, reset, end) Query (topology, type, adjacency)

Workset Traversal Block iterators allow access of entities in a user-defined array size Array based query functions for topology, type, adjacency Increases performance by reducing the number of calls through the interface Interface is not as intuitive

TSTT Mesh Modification Operators

Simple mesh modification operations are also supported Modifications

Changing vertex location Add and delete entities

Caveats No validity checks Requires simple classification mechanisms

against the geometric model Intended to support higher-level

functionality Mesh quality improvement Adaptive schemes with validity checking Front tracking procedures Basic mesh generation algorithms

Domain Discretization and Mesh Representation

Geometric model discretized into a mesh Discretized geometric domain defined as “union” of the

mesh entities Maintain link to shape information Support the field’s distributions used in equation

discretization Maintain the dof multiplying the distributions and support

assembly of global system

Mesh topology provides flexible support for these functions Mesh entities, Md, of dimension d Mesh entities are constructed from lower dimensional

mesh entities e.g. A face is constructed from edges which are constructed

from vertices Entities are adjacent if one is used to construct the other Typical complete mesh topological structures:

Mesh/Model Relationship

Relationship is termed “classification” Mesh classification: unique

association of a mesh entity, Midi,

to a geometric model entity, Gjdj,

where di≤dj is denoted by

Midi Gj

dj indicates the left-hand entity

(or set) represents a portion of the right-hand entity in the discretization

Multiple Midi classified on a Gj

dj

Boundary mesh entities are identifiedin terms of their classifications

Classification critical to supporting adaptive simulations and high level problem definitions

Reverse classification is the set of equal order mesh entities classified on a specific model entity

Simulation Fields

The discretization of the tensors used in the PDE in terms of distribution functions times degrees of freedom written over mesh entities

Field definition Tensor qualification - order, symmetries,

spatial dimension, coordinate system, constraints

Distribution functions - equations, mesh entities they act over, dof used

dof - mesh entities they are on, values after evaluation

ljki

Num

iiljkljk DOFtxNFT

N

...,1

...... ),(

* Typical but not unique form

Fields Relationship With Other Simulation Structures

Relationship with the space/time domain Tensor qualifiers specified over the space/time domain

Specifies the spatial domain dimension for the tensor Constraints (e.g. boundary conditions, initial conditions, divergence free/solenoidal field,

etc.) can be specified with attributes

Relationship with the mesh Information used to discretize the tensor specified over the mesh Distributions -- defined over mesh entities that are the same dimension as the

tensor dimension DOFS -- associated with mesh entities that are the dimension of the distribution

mesh entities and lower

DOF locations on associated entities

5 point FD stencil

FV with 1 distribution per element

C0 FEM quadratic distributions

Functionality of Field Library

Load, save, create and delete functions Point-wise interrogation functions using

combinations of operators Tensor operators

Addition Multiplication Transpose

Differential operators Identity Derivatives (x,y,z,t) Gradient

Integral functions Domain defined through TSTT entity sets

and/or model entities Uses point-wise interrogation operators for

integration of general quantities Mapping fields between meshes

Requires mesh information and above operations

Curl Divergence Laplacian

Locally adapted mesh

Meshes sharing a boundary

TSTT Interface Implementations

Implementations are software libraries that can be used through the TSTT interface Implementations are required to contain a core functionality Includes extensions allowing certain software requirements to be met Multiple implementations can be used simultaneously Tests to verify each implementations compliance with TSTT interface

standard

Implementations of the TSTT Mesh Interface currently available at https://svn.scorec.rpi.edu/wsvn/TSTT FMDB - http://www.scorec.rpi.edu/AOMD FronTier - http://frontier.ams.sunysb.edu GRUMMP - http://tetra.mech.ubc.ca/GRUMMP MOAB - http://cubit.sandia.gov/MOAB NWGrid - http://www.emsl.pnl.gov/nwgrid/index_nwgrid.html

SCOREC Involvement With TSTT

Development of interface standards TSTT Mesh Interface TSTT Geometry Interface TSTT Fields Interface

Software development FMDB (Flexible Mesh DataBase) library which is compliant with the TSTT

Mesh Interface FANS (Field Approximation for Numerical Simulations) library which is

being developed in conjunction with development of the TSTT Fields Interface

Model library which is being developed in conjunction with development of the TSTT Geometry Interface

Notable uses of software Development of adaptive loops for mesh-based simulations Multiphysics simulations

Constructing Adaptive Solution Procedures

Components of an an adaptive mesh control loop High level problem definition (domain and attributes) Domain discretization (mesh generation) Equation discretization Solvers to deal with the large systems of equations Error estimators/indicators Correction indicators Spatial and/or equation discretization enrichment methods

Implementation Tightly coupled using a single set of structures

Advantage: Computationally efficient if done well Disadvantage: Complex algorithm and code development

Loosely coupled building on existing components Advantage: Ability to use existing analysis codes and adaptive tools and to

mix and match tools Disadvantage: Overhead of multiple structures and data conversion

Information needed by an adaptive loop for proper creation of input for a code Overall Domain Definition - Geometric Model

Want a “true definition” of the domain - not a specific approximation Physical Attributes

Sets of tensors needed to qualify the loads, material properties, boundary conditions and initial conditions

Domain Discretizations (meshes) First piece of the two level discretization that is input to analysis codes

Simulation Fields Discretizations The discretized versions of tensor variables defined in terms of distributions

and degrees of freedom over the meshes as dictated by the PDE discretization process performed within the analysis code

The goal of the TSTT development is to provide all of the above information in a uniform manner

Structures/Services for Adaptive Loops

Adaptive Loop Driver Code Input and Output

Input Domain definition

Mathematical problem attributes Adaptive loop attributes/a priori

mesh controls Adaptive goal Geometry approximation

controls for mesh Mesh size limits (memory,

computation constraints) a posteriori mesh controls

Field information Error estimates Convergence rates Adaptive strategies

Model modifications through mesh motion Output

Simulation code input Adaptive loop convergence information Mesh

Simulation Code Input and Output

Input in form manageable by simulation code Sufficient mesh representation

Constructed from complete mesh representation from interface

Usually only nodes and elements Boundary and possibly initial condiitons

Element “sides” and nodal boundary conditions from classification/reverse classification information

Tensor values from field and attributes Other problem attributes

From complete problem definition Ideally code input files do not change

Output Computed field information

May include error estimates/indicators Field information from visualization output may be

sufficient Ideally code output files do not change

Adaptive Mesh Control

Need adaptive mesh control when Discretization is inadequate - mesh refinement Discretization computationally inefficient - mesh coarsening Element shape becomes unacceptable – mesh improvement

Two approaches for h-type mesh adaptation Re-meshing

Define new mesh size field Provide mesh generator with domain definition and mesh size field Map any history dependent solution fields from the old to the new mesh

Local mesh modifications Determine local operations needed to perform desired modifications Execute the local modifications Apply incremental history dependent solution field updates

Mesh Modification Operators

Single step operators Face swap Edge swap Edge collapse Region collapse Edge split Vertex motion Curving interior mesh entity

Compound operators modify two or more entities to allow the modification of the problem entity

Examples of two step compound operators Swap entity A first, and then swap entity B Collapse entity A first, and then collapse entity B Swap entity A first, and then collapse entity B Swap entity A first, and then curve new entity B Swap entity A first, and then reposition vertex B Etc.

Moving refinement vertices to boundary required during mesh modification (see IJNME paper, vol58 pp247-276, 2003 )

Coarse initial mesh and the mesh after multiple refinement/coarsening

Operations to move refinement vertices

Accounting for Curved Domains During Refinement

xy z

Iterations ofadaptations

# ofvertices tobe snapped

# of verticessnapped by a

reposition

# of verticessnapped by local

modifications

# of vertices snappedrequiring local re-

triangulations1 342 204 136 22 485 369 110 63 340 286 52 24 74 34 40 -5 26 3 23 -

Field Transfer on Adapted Meshes

Two approaches Global

Computationally expensive Procedures tend to diffuse information - accuracy loss

Local Performed with each local mesh modification Limited number of elements involved - efficient No accuracy loss with some operations, others easier to control

due to local nature

Edge marked for collapse

Zone updated by the operation

Mesh before collapse Mesh after collapse

Adaptive Loop for Accelerator Design

Collaboration with SLAC (I. Malik, K. Ko and Z. Li) Complex CAD geometry Physics modeling by the Omega3P code from SLAC

Solves Maxwell’s equations in frequency domain Determine power loss for specific modes

High level modeling accuracy needed E.g., 0.1% error in frequency predictions

Adaptive mesh control needed to provide desired accuracy Adaptive loop constructed using unmodified Omega3P

Providing the Adaptive Loop Components

Components added to construct adaptive loop High level problem definition Automatic mesh generation directly from CAD a-posteriori error indication and convergence rate to define mesh size field

a-posteriori error indicator constructed from knowledge of the tensor and its discretized field

General mesh modification to construct requested mesh size field accounting for the CAD geometry

Approach to the coupling of the components Interoperability procedures to provide information needed by components

Geometry - integration to multiple CAD systems Mesh - integration to various mesh generation and modification procedures Field - integration with different analysis procedures

Interoperability procedures used in this example from TSTT SCOREC Simmetrix

Integration of Components

Using geometry operatorsmeans alternate solid modelers can be inserted

Using geometry operatorsmeans alternate solid modelers can be inserted

Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted

Using TSTT mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted

Using TSTT field operators allows easy construction of alternative error estimators

Operator Interfaces

Projection-based error estimatorused to construct new mesh sizefield given to mesh modification

Mesh adaptation based on local modification linkeddirectly to CAD

Mesh adaptation based on local modification linkeddirectly to CAD

UnalteredSLAC code

UnalteredSLAC code

Error estimatorsfrom RPI and SLAC

Error estimatorsfrom RPI and SLAC

Adaptive Results for Trispal Model

1.0560

1.0580

1.0600

1.0620

1.0640

1.0660

1.0680

1.0700

1.0720

0 1E+05 2E+05 3E+05 4E+05 5E+05Number of DOF

F (

GH

z)

12000.0

12500.0

13000.0

13500.0

14000.0

0 1E+05 2E+05 3E+05 4E+05 5E+05

Number of DOF

Q

Frequency Convergence Q Convergence

Distribution of Wall-loss

Level 0 Level 2Level 1

Metal Forming Simulation

Large plastic deformation Meshes often become invalid Evolving geometry Evolving contact

Components of automated simulation Commercial analysis engine (DEFORMTM) Monitoring of mesh discretization errors and element shapes Mesh model topology update Construct mesh size field based on discretization errors and

geometry approximation General mesh modification to obtain the desired mesh size field Adjust mesh size and shape to control geometric approximations Local field transfer

Model Topology Update

Geometric components in forming simulations Workpiece Dies Die motions

Model topology needs to be updated Contact conditions change as simulation proceeds Mesh updates require complete model topology Simulation engine tracts only nodal contact

Must update model topology based on this information Model update procedure

Maintain non-manifold model representation Simulation contact information and “mesh geometry” Uses initial classification to build topology and then corrects ambiguities Mesh classified against updated model topology

Mesh modifications controlled Attributes properly associated to mesh

Steering Link Problem

Adapted based on the error indicators on the effective strain

Step 120

Step 160

Step 180

Step 200

Step 220

Step 240

Step 260 Step 279 (Completion)

Summary of TSTT Work

TSTT Mesh 0.6 Interface complete Working document describing the mesh and geometry data models as

well as the overall vision Implementations of TSTT Mesh 0.6 Interface available

https://svn.scorec.rpi.edu/wsvn/TSTT FMDB - http://www.scorec.rpi.edu/AOMD FronTier - http://frontier.ams.sunysb.edu GRUMMP - http://tetra.mech.ubc.ca/GRUMMP MOAB - http://cubit.sandia.gov/MOAB NWGrid - http://www.emsl.pnl.gov/nwgrid/index_nwgrid.html

Preliminary interfaces for geometry, field data, and mesh/geometry data model manager

C, C++, and Fortran language interoperability through SIDL/Babel (CCA) Analyzing performance ramifications of SIDL/Babel language

interoperability tools (joint with the CCA) TSTT Information

http://www.tstt-scidac.org

Adaptive loops built from available components Software for multiple sources combined to create functional adaptive

loops that provide superior results Initial SLAC adaptive loop constructed in less than 3 months

By people with little background with components Included development of an error estimation and correction

indication routines

Approach taken focused on optimizing time to getting adaptive results (not the computing resources needed for the simulation) Did not alter Omega3P or DEFORM codes

Used meshing components and operators to create input files Used field operators to extract results from output file

Closing Remarks on Adaptive Loops