computational engineering at nacad - coppe/ alvaro/  · computational engineering at nacad

Download Computational Engineering at NACAD - COPPE/ alvaro/  · Computational Engineering at NACAD

Post on 30-Nov-2018




0 download

Embed Size (px)


  • Computational Engineering at NACADAlvaro L.G.A. Coutinho

    NACAD-Center for Parallel ComputingCOPPE/Federal University of Rio de Janeiro, Brazil

    October, 2003

  • Alvaro LGA Coutinho 2/48

    Contents:Contents:Introduction: Who we are and what we doField Equations for Grid-based ApplicationsFinite Element DiscretizationComputational IngredientsGrid-based Demonstration ProblemsConcluding Remarks

  • Alvaro LGA Coutinho 3/48

    IntroductionIntroductionWho are we ?

    NACAD Center for Parallel Computing, COPPE/Federal University of Rio de Janeiro, Brazil

    Associated Laboratories

    LAMCE, NTT, LAB2M, CEMONComputer Methods in Engineering Lab, Data Mining Lab, Basin Simulation Lab, Environment Monitoring Lab Civil Engineering Department

    LASPOTPower Systems LaboratoryElectrical Engineering Department

    Parallel Computing LaboratoryComputer Science Department

  • Alvaro LGA Coutinho 4/48

    Introduction (contIntroduction (contd)d)What we do ? High Performance Computing: research and development

    Parallel, vector, and cluster computing Scientific visualization Applications to:

    Petroleum EngineeringPower SystemsAerospace Engineering Environment Data MiningGovernanceFinancial EngineeringMeteorology

    Cray SV1

    InfoServer Itautec

  • Alvaro LGA Coutinho 5/48

    Field Equations for GridField Equations for Grid--based based ApplicationsApplications

    General Form of PDEs for Engineering Systems

  • Alvaro LGA Coutinho 6/48

    Governing Equations in Eulerian Framework









    Navier-Stokes Equations

    Energy Transport Equation

    =+ inThTkTc




    Mass Transfer Equations

    =+ inT



  • Alvaro LGA Coutinho 7/48

    Eulerian Governing Equations

    Multi-phase Darcy-flow in Porous Media:







    zg = p

    ( )




    ij +



    =1, 2, ... , nphases

    From Mendona, 2003

  • Alvaro LGA Coutinho 8/48

    Governing Equations in Lagrangian Framework

    Equation of Motion for Solids and Structures:

  • Alvaro LGA Coutinho 9/48

    Lagrangian Governing Equations


    From Quaranta&Alves, 2002

  • Alvaro LGA Coutinho 10/48

    Arbitrary Eulerian Lagrangian Governing Equations

    Incompressible N_S equations in ALE frame moving with velocity w:

    Velocity w is conveniently adjusted to Eulerian (w=0), far from moving object to Lagrangian (w=u) on the fluid-structure interface.Fluid is considered attached to the body.Need to solve extra-field equation to define mesh movement: our choice is to solve the Laplacian.

    From Felippa, Park and Farhat (CMAME, 2001)

  • Alvaro LGA Coutinho 11/48

    FEM DiscretizationFEM Discretization

    Good mathematical background and ability to handle complex geometries by using unstructured grids

  • FEM FEM DiscretizationDiscretization

    Variational Formulation

  • Alvaro LGA Coutinho 13/48

    FEM Computational IngredientsFEM Computational Ingredients

    Space-Time AdaptationAdaptive step sizeMesh refinement/unrefinement

    Non-linear Solution Methods, Iterative SolversData Structures: Memory complexity O(meshparameters)Partitioned Time-Marching SchemesHigh Performance Computing Issues

  • Alvaro LGA Coutinho 14/48

    Adaptive Step size Control for Time Step Selection


    Valli, Coutinho, Carey, CNME, 2002

  • Alvaro LGA Coutinho 15/48

    Adaptive Mesh Refinement

    Fundamental for high accuracy computationsWe prefer adaptive remeshing with Delaunay triangulation with a coarse background meshZZ viscous stress error indicator do guide adaptationALE we need to move both background and current meshes

    Sampaio&Coutinho, IJNMF, 1999

  • Alvaro LGA Coutinho 16/48

    Nonlinear Solution Method: Inexact Newton Method

    Given utol, rtol, relative unknown and residual tolerances and RHS vector, b do i while convergence Compute residual vector, 11 = iii uJbr Update jacobian matrix, iJ Compute tolerance for iterative driver, i Solve ii ruJ = for tolerance i Update solution, uuu +

    If toluuu

    and toli


    then convergence

    End while

    Backtracking is sometimes useful !Coutinho et al, IJNME, 2001

  • Alvaro LGA Coutinho 17/48

    Iterative Solution Methods

    Symmetric systems: PCGNon-symmetric systems: GMRESMatrix-vector products Element-by-element


    Preconditioning keeping same data structures





    ( )=


    1eepLwpK )(,

  • Alvaro LGA Coutinho 18/48

    Edge-based Solution

    FE mesh Graph representation

    Sparse matrix

  • Alvaro LGA Coutinho 19/48

    Edge-based FE Scheme

    Disassembling of Element Matrix







    321 EdgeEdgeEdgeElement

    Assembling of Edge Matrix






    EdgeIJ Elem Elem



    1 2

  • Alvaro LGA Coutinho 20/48

    Edge Matrices and Matvec







    Element matrices disassemblingm is the number of element edges, which is 6 for tetrahedra or 28 for hexahedra.

    Edge Matrix


    ess KK E is the set of all elements sharinga given edge s

    Edge-by-edge matrix-vector product



    ss pKpK



  • Alvaro LGA Coutinho 21/48

    Computational costs for symmetric sparse matrix-vector products in tetrahedral meshes


    Memory flop i/a

    EBE 429 nnodes 1,386 nnodes 198 nnodes

    Edges 63 nnodes 252 nnodes 126 nnodes

    nel 5.5nnodes, nedges 7nnodes

  • Alvaro LGA Coutinho 22/48

    SuperedgesIdea introduced by Lhner (94) and implemented in CSM and CFD by Martins et al (97,98,02) and Coutinho et al (01) for tetrahedraand hexahedraDesigned to improve i/a ratio and flop balanceOnce data have been gathered from memory to processor (registers), reuse them as much as possibleFormed by edge list reorderingDifferent grouping are possible increasing code complexityNodes reordered in increasing order as they appear in the superedge list (Lhner, 93)2D triangle, 3D tetrahedra

    Superedges in blue

    Guanabara Bay

  • Alvaro LGA Coutinho 23/48

    Partitioned Time Marching Scheme

    Mesh partitioning algorithms for time-marching: I/E, E/E, Iterative/Direct, etc

    Partition can evolve in time

    Implicit Edges in RED

  • Alvaro LGA Coutinho 24/48

    High Performance Computing High Performance Computing IssuesIssues

    FEM is a unstructured grid method characterized by:

    Discontinuous data no i-j-kaddressingGather-scatter operationsRandom memory access patternsData dependenceMinimize indirect addressing is a must

  • Alvaro LGA Coutinho 25/48

    Parallel Solution Strategies

    Shared Memory: Mesh Coloring

    Distributed Memory: Mesh Partitioning

  • Alvaro LGA Coutinho 26/48

    GridGrid--based Demonstration based Demonstration ProblemsProblems

    Fluid Flow in Deformable Porous Media -Well Stability: What you can do in a PCReservoir Engineering: Effects of Memory SpeedHydrodynamic computations in Araruama Lagoon: Example of Cluster ComputingFluid-Structure Interaction in Rio-Niteroi BridgeStress analysis of sedimentary basins

  • Alvaro LGA Coutinho 27/48

    Fluid Flow in Deformable Porous Media Well Stability: What you can do in a PC

    Quasi-static deformation of plastic porous media coupled with 1-phase flowStrain depends on poro-pressurePorosity is function of volumetric changeStaggered coupling

  • Alvaro LGA Coutinho 28/48

    Coupled 1-phase (water) and Solid in 3D: Vertical and Horizontal Wells

    Mesh Data

    36,105 nodes,

    191,163 elements,

    236,090 edges (23% simple, 18% s3 and 59% s6)

    Solid Material Data

    Internal radius 0.11 m; External Radius 20.0 m

    Formation Pressure: 32.4 MPa

    Insitu stresses (V/H): 32.1 MPa; 9.0 MPa

    Youngs modulus: 1.2 GPa, Poisson: 0.2

    Internal angle: 45; Cohesion: 8.5 MPa; Biot: 1.00

  • Alvaro LGA Coutinho 29/48

    Coupled 1-phase (water) and Solid in 3D: Vertical and Horizontal Wells

    Stiffness Updating

    PCG total PCG Average

    NR BCT Time (s)

    Secant 931 93,1 10 0 579,1 Tangent 931 93,1 10 0 551,4

    Stiffness Updating

    PCG total PCG average

    NR BCT Time (s)

    Secant 6.049 40,1 148 0 4.709,2Tangent 1.291 92,2 14 4 981,4

    3 poro-pressure load steps: 34,088, 14,9 e 4,9 MPa;Non-linear Solver: Edge-based Inexact Newton; PIII 1GHz

    Vertical Well

    Horizontal Well

  • Alvaro LGA Coutinho 30/48

    Numerical Results for Horizontal Well

    Plastic fringes around well Total displacements around well

  • Alvaro LGA Coutinho 31/48

    Reservoir Engineering: Effects of Memory Speed

    True heterogeneous reservoir: SPE 10thcomparative project: dimensions: 1200x2200x170 ftUnstructured grid generated from 60x220x85 cells

    5,610,000 tetrahedra1,159,366 points6,843,365 edges

  • Alvaro LGA Coutinho 32/48

    Effects Memory Speed

    From Jack Dongarra, 2002

  • Alvaro LGA Coutinho 33/48

    Preprocessing and Matvec performance on the CRAY SV1





View more >