parallel wave propagation and topological operators for fragmentation simulation

Parallel Wave Propagation and Topological Operators for Fragmentation Simulation

Glaucio H. Paulino

Professor, Faculty CEE, MechSE, CSE

Donald Biggar Willett Professor of Engineering

6th Annual Workshop on CHARM++ and its Applications

4/15/2008 2http//cee.uiuc.edu/paulino paulino@uiuc.edu

Acknowledgments

CEE COLLABORATOR:

Mr. Kyoungsoo Park

CS COLLABORATORS:

Prof. Laxmikant V. Kale (UIUC)

Dr. Celso L. Mendes (UIUC)

Dr. Terry L. Wilmarth (UIUC)

Mr. Aaron Becker (UIUC)

Mr. Isaac Dooley (UIUC)

Prof. Waldemar Celes (PUC-Rio)

Mr. Rodrigo Espinha (PUC-Rio)

4/15/2008 3http//cee.uiuc.edu/paulino paulino@uiuc.edu

Stress waves

Waves

Rosakis AJ, Samudrala O, Coker D, 1999, Science 284

4/15/2008 4http//cee.uiuc.edu/paulino paulino@uiuc.edu

Outline

Waves Wave Propagation

Rayleigh wave Parallel computing: ParFUM Results: Geological simulation

Dynamic Fracture Cohesive zone modeling Node Perturbation & Edge-Swap Operation Results: Fracture & Compact compression tests

Summary Future Work

4/15/2008 5http//cee.uiuc.edu/paulino paulino@uiuc.edu

Wave Propagation: Rayleigh Wave

Surface Wave

Lord Rayleigh, 1885

Seismology, Geology, Material Science, etc

Homogeneous & Orthotropic materials (2005)

Large-Scale 3D Analysis for Graded media

• Rayleigh L. 1885, On waves propagated along the plane surface of an elastic solid, Proc. R. Soc. Lond. A 17, 4-11

• Vinh PC, Ogden RW. 2005, On the Rayleigh wave speed in orthotropic elastic solids, Meccanica 40, 147-161.

4/15/2008 6http//cee.uiuc.edu/paulino paulino@uiuc.edu

Parallel Computing: ParFUM

Finite Element Analysis

Bill Gropp: Very “Easy to write code that

scales and performs poorly.”

Time Integration

Central difference method

Communications

Update Rint

Shared-node summation operation

21

12n n n nt tu u u u+ = +D + D& &&

1 int1 1 1( )ext

n n n-

+ + += -u M R R&&

1 1( )2n n n n

t+ +

D= + +u u u u& & && &&

Lawler OS, Chakravorty S, Wilmarth TL, Choudhury N, Dooley I, Zheng G, Kale LV, 2006, ParFUM: a parallel framework for unstructured meshes for scalable dynamic physics applications, Engineering with Computers 22, 215-235.

4/15/2008 7http//cee.uiuc.edu/paulino paulino@uiuc.edu

Machine Specification

Dell Cluster [Abe] Peak FLOPS: 89.47 TF Number of Blades (nodes): 1200 Number of CPUs (cores): 9600 Processor: Intel 64 2.33GHz dual socket quad core 8 MB L2 cache (2 MB) Memory: 8GB (1GB) Total: 9600 GB

Dell Xeon Cluster [Tungsten] Peak FLOPS: 16.38 TF Number of nodes: 1280 Number of processors: 2560 Processor: Intel Xeon 3.2 GHz (32-bit) Memory: 1.5 GB Total: 3840 GB

4/15/2008 8http//cee.uiuc.edu/paulino paulino@uiuc.edu

0

2000

4000

6000

8000

10000

32 64 128 256 512 1024

Runtime Performance

Number of processors

Par

alle

l ru

nti

me

(sec

) # of Elements: 0.4 million

91%

90%

81%62% 43%

Dell Cluster [Abe]

4/15/2008 9http//cee.uiuc.edu/paulino paulino@uiuc.edu

Geology Simulation: Rayleigh Wave

Layer #4

Layer #3

Layer #2

Layer #1 (Graded)

4km

2km

2km

2km

10 km

10 km

Layer #4

Homogeneous

10km

10 km

10 km

z

P(t) P(t)

( ) sin(2 ) (0 2)

( ) 0 ( 2)

P t t t

P t t

p= £ £

= >

V. Pereyra, E. Richardson, S. E. Zarantonello, Large scale calculations of 3D elastic wave propagation in a complex geology, Proceedings of the 1992 ACM/IEEE conference on Supercomputing, Minneapolis, Minnesota, 301-309.

4/15/2008 10http//cee.uiuc.edu/paulino paulino@uiuc.edu

Homogeneous Media

Material Properties

Dynamic Responses

CP (km/s)

CS (km/s)

ρ(kg/m3)

E (MPa) v

Media 2 1.2 2 7.02 0.219

Tim

e (sec)

5

4

3

2

1

( ) 1/f r r

1

1.01

CR = 1.1 km/s

4/15/2008 11http//cee.uiuc.edu/paulino paulino@uiuc.edu

Graded & Layered Media

( ) 1/f r r

( ) 1/f r r

CP (km/s)

CS (km/s)

p(kg/m3)

E (MPa) v

Graded Layer #1

23.6

1.22.16

23.6

7.0240.94

0.2190.219

Layer #2 3.5 2.1 2.3 24.7 0.336

Layer #3 4.5 2.1 2.3 27.6 0.361

Layer #4 5.5 2.1 2.3 28.7 0.430

Tim

e (sec)

5

4

3

2

1

1

1.25

4/15/2008 12http//cee.uiuc.edu/paulino paulino@uiuc.edu

Waves Wave Propagation

Rayleigh wave Parallel computing: ParFUM Results: Geology simulation

Dynamic Fracture Cohesive zone modeling Node Perturbation & Edge-Swap Operation Results: Fracture & Compact compression tests

Summary Future Work

4/15/2008 13http//cee.uiuc.edu/paulino paulino@uiuc.edu

Dynamic Fracture: Cohesive Zone

Cohesive Zone Model

Computational Simulation TRULY Extrinsic cohesive surface elements

Several claims of extrinsic simulations in the literature are NOT truly extrinsic (e.g. activated elements are not extrinsic)

Ce-basedTi-basedXi XK et al., 2005,, Physical Review Letters, 94, 125510

4/15/2008 14http//cee.uiuc.edu/paulino paulino@uiuc.edu

Topology-based Data Structure

Complete Topological Data

Reduced Representation

Support for Adaptive Analysis

• W. Celes, G.H. Paulino, R. Espinha, 2005, Efficient handling of implicit entities in reduced mesh representations, Journal of Computing and Information Science in Engineering 5 (4), 348-359.

• W. Celes, G.H. Paulino, R. Espinha, 2005, A compact adjacency-based topological data structure for finite element mesh representation, IJNME 64(11), 1529-1556

• G. H. Paulino, W. Celes, R. Espinha, Z. Zhang, 2008, A general topology-based framework for adaptive insertion of cohesive elements in finite element meshes, EWC 24, 59-78

4/15/2008 15http//cee.uiuc.edu/paulino paulino@uiuc.edu

Entity Enumeration

Model

Entity information

Elapsed time (s)

Topologicalentity

Number of entities

Titan IV model (linear hexahedral

mesh)

Element 1,738,240 0.097

Node 1,845,640 0.046

Facet 5,321,600 0.219

Edge 5,429,000 0.292

Vertex 1,845,640 0.186

W. Celes, G.H. Paulino, R. Espinha, 2005, Efficient handling of implicit entities in reduced mesh representations, Journal of Computing and Information Science in Engineering 5 (4), 348-359.

4/15/2008 16http//cee.uiuc.edu/paulino paulino@uiuc.edu

4K Structured Mesh

Mesh Orientation Dependence 4 direction Maximum error: 45º

8 direction Maximum error: 22.5º

Undesirable crack pattern

4/15/2008 17http//cee.uiuc.edu/paulino paulino@uiuc.edu

Node Perturbation

Edge Swap

Proposed Remediation

0.0 0.1 0.3

4/15/2008 18http//cee.uiuc.edu/paulino paulino@uiuc.edu

Crack Length Convergence

Simulation Outline Find the shortest path

(e.g. ) Node Perturbation (NP) Factor: 0.3 Edge Swap Square 4K structured mesh Element size: 0.1 Simulate 100 randomly perturbed

meshes for each node perturbation factor

1

2.4

1tan (2.4) 67.38

2.6ExactL

4/15/2008 19http//cee.uiuc.edu/paulino paulino@uiuc.edu

Results

NP 0.3

Avg. Error = 5.5%

NP 0.3 & Edge Swap

Avg. Error = 4.5%NP0, Error = 8.2%

4/15/2008 20http//cee.uiuc.edu/paulino paulino@uiuc.edu

Crack Angle Convergence

67.38

NP Factor = 0 NP Factor (0.3) & Edge swap

4/15/2008 21http//cee.uiuc.edu/paulino paulino@uiuc.edu

Effect of Element Size & Edge-SwapA

ng

le (

º)

Element Size

Activate Edge Swap (ES)

Without Edge Swap (ES)

NP Factor = 0.3Given Angle (α)

0 10 20 30 40 50

50.2

53.1

56.3

59.7

63.4

67.4

71.6

76.0

An

gle

(º)

The number of appearance

Element Size = 0.1

45

50

55

60

65

70

75

80

0 0.05 0.1 0.15 0.2

with edge-swap

without edge-swap

4/15/2008 22http//cee.uiuc.edu/paulino paulino@uiuc.edu

Fracture Test (Verification)

0 0.0124

16 mm

4.2 mm

4K structured mesh (80 X 21)

Material PropertiesE = 3.24 GPav = 0.35ρ = 1190 kg/m3

GI = 352 N/mσmax = 129.6 MPa

4/15/2008 23http//cee.uiuc.edu/paulino paulino@uiuc.edu

[mm]

Compact Compression Specimen

Incident bar Transmitter bar

60

70

2035

16

E = 5.76 GPav = 0.42ρ = 1182 kg/mm3

GI = GII = 4800 N/mσmax = 105 MPa

striker

4/15/2008 24http//cee.uiuc.edu/paulino paulino@uiuc.edu

Summary

Large-Scale Parallel Wave Propagation 1024 processors

Rayleigh Wave Speed in 3D Functionally Graded Media

Crack Path Representation thru Topological representation Node perturbation & Edge swap operators

Adaptive Dynamic Fracture Simulation V & V

4/15/2008 25http//cee.uiuc.edu/paulino paulino@uiuc.edu

Future Work

Wave propagation for complex geology systems Provide guidance to estimate Rayleigh wave speed in

smoothly graded heterogeneous media

Incorporate data from geological surveys

Parallel Dynamic Fracture Simulation Parallel adaptive insertion of cohesive surface element

Dynamic adaptive load balancing

There is a lot of exciting work to do !