parallelization of an extended finite element two-phase flow solver · 2011. 7. 5. · chair for...
TRANSCRIPT
CHAIR FOR
COMPUTATIONAL
ANALYSIS OF
TECHNICAL
SYSTEMS
Parallelization of an Extended Finite Element
Two-Phase Flow Solver
H. Sauerland and T.-P. Fries
July 20, 20109. WCCM
Sydney, Australia
History of our XFEM flow solver
Navier-Stokes solver for quadrilateral/hexahedral meshes withsemi-implicit time integration.
←−
increasingNo.
DOF
serial direct solverC, UMFPACK1
⇓
parallel direct solverMPI, MUMPS2
⇓
parallel iterative solverMPI, PETSc3
1http://www.cise.ufl.edu/research/sparse/umfpack/
2http://graal.ens-lyon.fr/MUMPS/
3http://www.mcs.anl.gov/petsc/
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 2
Outline
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 3
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 4
Challenges in two-phase flows
Moving interface:
Topological changes of the phases can occur→ Level-set method.
Discontinuous field variables:
Density differences
→ kink in the velocity/pressure field,Viscosity differencesSurface tension → jump in the pressure field→ XFEM.
up
ρ2, µ2
ρ1, µ1
u
p
ρ2, µ2
ρ1, µ1
γ
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 5
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 6
Introduction to the XFEM
Interface is implicitly described using the level-set method.
Discontinuities inside elements accounted for with the XFEM.
Here: Pressure approximation space is locally enriched:
ph(x) =∑
i∈I
Ni (x)pi
︸ ︷︷ ︸
strd. FE approx.
+∑
i∈I⋆
Ni (x) · [ψ(x, t)− ψ(xi , t)] ai
︸ ︷︷ ︸
enrichment
.
Sign-enrichment:
ψ(x, t) = sign (φ(x, t)) =
−1 : φ < 0,
0 : φ = 0,
1 : φ > 0.
I ⋆
φ(x, t) = 0
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 7
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 8
Iterative solvers with the XFEM
Occurrences of enrichment basis functions with very smallsupport seriously increase the condition number of the systemmatrix → slow/no convergence with iterative solvers.
Spectral condition number of the system matrix K:
κ(K) =
∣∣∣∣
λmax(K)
λmin(K)
∣∣∣∣, λ: Eigenvalues
Case κ(K)
Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444
Possibilities: Blocking degrees of freedom,’special’ (additional) preconditioning.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 9
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 10
Blocking degrees of freedom
Reusken (CVS, 2008) introduced a modified XFEM space,excluding the basis functions with very small support.
Here: Identify basis functions with very small support simplyusing the area ratio of the two phases occupying the element:
Aphase 1
Aphase 2< Cblock ≪ 1, where Aphase 1 < Aphase 2.
Spectral condition number of the system matrix K:
Case κ(K)
Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444Cblock = 0.01 ∼ 22849 Cblock = 0.01
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 11
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 12
Preconditioning: Diagonal scaling
Bechet et al. (IJNME, 2005) developed a preconditioner forthe XFEM based on a local Cholesky decomposition.
Here: Simple diagonal scaling of the linear system:
D−1/2KD−1/2(D1/2x
)= D−1/2b, with D = diag(K)
Applied in addition to the general purpose preconditioners.
Spectral condition number of the system matrix:
Case κ(K)
Standard FEM ∼ 15XFEM (p-enriched) ∼ 97444Cblock = 0.01 ∼ 22849Diagonal scaling ∼ 27
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 13
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 14
Convergence study: Blocking DOFs, diagonal scaling
Steady state test case with surface tension:
ρ1/ρ2 = 1.0,
µ1/µ2 = 1.0,
γ = 1 kg/s2.
Exact solution given by:
u(x) = 0, p2 − p1 = γ ·1
r.
Ω1
Ω2
2r4r
4r
Solver: GMRES, LU preconditioner.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 15
Convergence study: Blocking DOFs, diagonal scaling
No convergence to a steady state without blocking andwithout diagonal scaling.
Steady state test case with surface tension:
10−2
10−1
10−4
10−3
10−2
10−1
100
Element size h
Pre
ssur
e L2
−no
rm
dia. scal.c=0.01c=0.1c=0.5c=1
1/15 1/20 1/40 1/80 1/1600
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Element size h
Rat
io o
f num
ber
of b
lock
ed a
nd e
nric
hed
node
s
c=0.01c=0.1c=0.5c=1
Conclusion: Diagonal scaling is very powerful in this case.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 16
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 17
Study on iterative solvers and preconditioners
PETSc allows to change solver/preconditioner on execution.
Considered Krylov-Subspace methods:
GMRES(m), l-GMRES, f-GMRES,BiCGstab, i-BiCGstab, BiCGstab-l.
GMRES: satisfies an optimality condition; Comp. costs risewith the iteration count.
BiCGstab: Comp. costs independent of the iteration count;does not satisfy an optimality condition.
Considered preconditioners:
ASM, KSP, LU, ICC, ILU, SOR, SSOR.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 18
Study on iterative solvers and preconditioners
Steady state test case with surface tension:
ρ1/ρ2 = 1.0,
µ1/µ2 = 1.0,
γ = 1 kg/s2,
80× 80 elements.
Exact solution given by:
u(x) = 0, p2 − p1 = γ ·1
r.
Ω1
Ω2
2r4r
4r
Convergence criteria:Residual of the solution vector: ε ≤ 10−10.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 19
Study on iterative solvers and preconditioners
GMRES
PC ε× 10−10 # t [s]lu 0.00060 528 7.92
f-GMRES
PC ε× 10−10 # t [s]lu 0.00057 522 7.84
l-GMRES
PC ε× 10−10 # t [s]asm 0.00190 1473 7.13ilu 0.00058 1524 6.93lu 0.00058 453 7.78
i-BiCGstab
PC ε× 10−10 # t [s]asm 3083. 3410 22.03ilu 8455. 3410 19.72
BiCGstab-l
PC ε× 10−10 # t [s]asm 3992. 2936 20.32lu 12. 1000 24.29ssor 7604. 4894 19.61
BiCGstab
PC ε× 10−10 # t [s]asm 1918. 3059 21.25ilu 4542. 3085 19.15ssor 875. 7678 27.29
GMRES methods converge very fast.
BiCGstab methods do not reach the residual bound.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 20
Study on iterative solvers and preconditioners
Tank sloshing test case:
ρ1/ρ2 = 1000.0,
µ1/µ2 = 0.01,
∆t = 0.05 s,
80× 120 elements,400 time-steps.
Ω2
Ω1
1.0
1.5
g
Convergence criteria: ε ≤ 10−4.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 21
Study on iterative solvers and preconditioners
Tank sloshing test case:
Solver PC t [min]
GMRES lu 36f-GMRES lu 34
l-GMRESasm 37ilu 33lu 36
i-BiCGstab asm 25BiCGstab-l lu 19BiCGstab ssor 30
0 2 4 6 8 10 12 14 16 18 200.9
0.95
1
1.05
1.1
1.15
Time
Hei
ght
BiCGstab methods perform better here, as the convergencecriteria is not that strong as in the steady case.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 22
Study on iterative solvers and preconditioners
Rising bubble test case:
ρ1/ρ2 = 1000.0,
µ1/µ2 = 0.01,
γ = 0.001 kg/s2 ,
r = 0.1m,
∆t = 0.025 s,
80× 160 Elements,1000 time-steps.
Ω1
Ω2
4r
8r
g
2r
Convergence criteria: ε ≤ 10−4.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 23
Study on iterative solvers and preconditioners
Rising bubble test case:
Solver PC t [min] Cblock
l-GMRES asm 159 0.0BiCGstab-l lu - 0.0BiCGstab-l lu 118 0.001
GMRESBiCGs
Diagonal scaling and blocking with small Cblock makes sensehere.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 24
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 25
Parallelization in the XFEM context
Basic approach is the same as for standard FEM:
Partitioning(ParMETiS)
Assembly onpartition-level
Assembly in parallelPETSc matrix
Parallel solver
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 26
Parallelization in the XFEM context
XFEM: Enriched elements are computationally more expensive(more integration points, more shape function evaluations andthe additional DOFs)
Tank sloshing for 2 PEs
PE0: 75 Elements (12 enriched)PE1: 75 Elements (0 enriched)→ Imbalance
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 27
Parallelization in the XFEM context
Solution: Assign weights to the elements before the partitionphase (larger ones for enriched).
ParMETiS uses the weights as an additional constraint.
Tank sloshing for 2 PEs (enriched elements weighted with 10):
PE0: 48 Elements (9 enriched)PE1: 102 Elements (3 enriched)→ Balanced?
Reasonable values for the weights have to be determined.
E.g. for the rising bubble test case a static load balancing willnot be sufficient (also cf. adaptive meshes).
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 28
Dynamic load balancing
Dynamic load balancing:
Adapt the partitioning duringexecution to maintain a good loadbalance.
Specify criteria when to perform a load balancing step (ideallytaking into account migration/rebuilding costs).
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 29
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 30
Scale test
Olmsted dam on the Ohio river (Illinois, USA):
http://www.lrl.usace.army.mil/olmsted_specproj/
Computational domain:
195m
1.16m3/s →
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 31
Scale test
140000 DOFs(performed on an Intel Xeon cluster with 1840 cores in total).
Interface motion and pressure:
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 32
Scale test
Parallel performance:
Speedup:
S(n) =T (1)
T (n).
Efficiency:
E (n) =T (1)
n · T (n),
n: Number of processors,
n ts/h S(n) E(n)
1 1.5 - -2 2.8 1.87 0.9354 5.1 3.38 0.8458 8.6 5.69 0.71116 15.3 10.10 0.63132 16.6 10.90 0.340
T (i): Execution time with i processors.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 33
Challenges in two-phase flows
Introduction to the XFEM
Iterative solvers with the XFEMBlocking degrees of freedomPreconditioning: Diagonal scalingConvergence study: Blocking DOFs, diagonal scalingStudy on iterative solvers and preconditioners
Parallelization in the XFEM contextDynamic load balancing
Scale test
Conclusion
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 34
Conclusion
Usage of standard iterative solvers and preconditioners withthe XFEM is difficult.
Diagonal scaling of the system matrix significantly improvesthe condition number.
Additional blocking of DOFs makes sense in extreme cases.
Dynamic load balancing is required for the XFEM in order toachieve optimal parallel performance.
Current implementation already scales quite good.
Outlook: Complete dynamic load balancing.
Parallelize our adaptive mesh refinement.
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 35
Thank you for your attention.
www.xfem.rwth-aachen.de
H. Sauerland and T.-P. Fries Parallelization of an Extended Finite Element Two-Phase Flow Solver 36