optimistic parallel discrete event simulations of physical systems using reverse computation

Optimistic Parallel Discrete Event Simulations of Physical Systems

using Reverse Computation

06/01/2005

Yarong TangKalyan S. PerumallaRichard M. Fujimoto

College of Computing, Georgia Tech

Homa Karimabadi

Jonathan DriscollYuri

Omelchenko

SciberNet Inc.

PADS 06/01/05 Georgia Institute of Technology

The Big Picture

Global Multi-Scale Kinetic Simulations of the Earth's Magnetosphere

Multi-physics; multi-scale

The goal: to understand how solar wind interacts with the Earth’s magnetosphere (space weather)

People PDES simulation: lead by Richard Fujimoto and Kalyan

Perumalla Compiler optimization: lead by Pande Santosh Physics modeling: lead by Homa Karimabadi @ Scibernet

Inc.


Outline

MotivationOverview

The PIC model (Particle-in-Cell)

ImplementationPerformanceConclusion & Future work


Why optimistic simulation?

Synchronization in PDES: ensures LPs to process events in time stamp order

Conservative approachAvoids violation of local causality constraintInefficient for applications with little lookahead

(low “predictability”)Plasma simulation highly dynamic, nearly-zero lookahead in certain regions

Optimistic Approach


Why reverse computation?

Optimistic synchronization: Allows causality errors to happen, but uses a

rollback mechanism to recoverState saving: saves state prior to computation; restores saved state

Plasma simulation has a lots of events → lots of memory!Event computation cost is small compared to state saving overhead → significant overhead!

Reverse computation: rollback by performing the inverse of the event computation

Reduces both memory and time needed for state savingCan be automatedExcellent match for large, high-performance models


Outline

MotivationOverview

The PIC model (Particle-in-Cell)

ImplementationPerformanceConclusionFuture work


Particle-in-Cell (PIC) ModelCharging a spacecraft due to a periodical beam injection from its surface Conceptually simple, yet sufficiently complex to capture characteristics of plasma dynamics

What are we interested in the simulation? Charge on spacecraft surface Plasma dynamics


Challenges

Complicated interactions between physical entities: cells (modeled by LPs) and particles

Cell1 sends a particle to cell2

Cell1 updates state and may “wake up” all its particles

Cell1 Cell2

Cell2 updates state and may “wake up” all its particles


Cell 1

Cell 2

Simulation Time

Cell 3

Move

Add Move

AddAdd Move

Add Move

Particle Movement Illustrated


Challenges cont.d

Memory requirementParticle state

Velocity, acceleration, position, direction, type,

exit_time, lastmove_time, etc.Cell state

Center field, boundary fields, coordinates, particle queue

Q: How to minimize the number of auxiliary bits

for efficient reverse code?


Challenges cont.d

Floating point arithmetic throughout simulation

Simple increment/decrement won’t work

Destructive operations are dominantParticle’s exit_time is computed by solving a pair of quadratic equations → not reversible“Wakeup” event → recompute all particle statesDelete operations in particle queue


Our observations

“delete” in queue is the inverse of “insert” and vise versa

Particles never “disappear”; one deletion in a cell corresponds to an insertion in another cell

Partial particle state is carried over by msgsNo need to re-compute these state variables

Reversing physical states can resort to physics laws

No need for “brute-force” when reversing codeApplication semantics is the magic for irreversible code


Implementation exampleShell::arrival( ParticleArrivalEvent *e ) { if( this cell is active ) { update cell state; insert particle in cell; } else if ( e is a beam particle ) { activate cell; }}Shell::departure( ParticleDepartureEvent *e ) { if( particle bounced from right neighbor ){ bounce particle back; // no state change } else { update cell state; }}Shell::inject( ParticleInjectEvent *e ) { update cell state; insert beam particles;}

Shell::undo_arrival( ParticleArrivalEvent *e ) { if ( cell was activated ) { undo_activate cell; } else if ( cell already active ) { delete particle in cell; undo_update cell state; }}Shell::undo_departure( ParticleDepartureEvent *e ) { if ( particle was bounced from right neighbor ) { undo_bounce particle; } else { undo_update cell state; }}Shell::undo_inject( ParticleInjectEvent *e ) { delete beam particles; undo_update cell state;}


Cell 2

Simulation Time

Cell 3

Add@ 20

E

Move@ 40

E

Move@ 5Ion

Add@ 10Ion

Q

a

v

x

S

'

'

'

'

'

Q

a

v

x

S

ForwardAddParticleToPQ()

Q = Q'+q/c.w2

a = Q.q/mv = v'+a.dtx = x'+v.dt+½a.dt2

Reversex' = x-v.dt-½a.dt2

v' = v-a.dtQ' = Q-q/c.w2

a' = Q'.q/mDelParticleFromPQ()

Move@ 30

E

dt

?…

Reverse Computation Illustrated


Experiment

Simulation engine: μsik (by Kalyan Perumalla)

General-purpose parallel/distributed sim engineSupports multiple synchronization approaches

Hardware platform: SMP machines running Red Hat Linux with a customized 2.4.18-10smp kernel8 PIII 550MHz Xeon processors sharing 4GB of memory


Simulation configuration[1] Karimabadi, H, et al. A New Asynchronous Methodology for Modeling of

Physical Systems: Breaking the Curse of Courant Condition. J.

Computational Physics, 2005

Normalized units Simulate 7000 cells with 70 initially active cellsSpacecraft has a radius of 500 & cell has a width of 0.24Solar wind plasma initially loaded with uniformly distributed electrons and protons (30 of each per cell)Injected positron beam has an energy of 10 keV with a period of 4ms


Phase space comparison

Time-stepped

Sequential DES


Validation by phase space comparison

Sequential DES PDES


PDES vs. sequential DES


Events distribution T

otal

Nu

mb

er o

f C

omm

itte

d E

ven

ts


Conclusions

Being the First to apply reverse computation to parallel simulations of physical systemsPropose application-level reverse computationEncouraging results for 1D model, better performance expected for models with a higher dimension


Ongoing and future work

Performance comparison with check-pointing approachesMore complex applications on larger-scale systemsApply reverse computation to other domains of physics applications involving partial differential equations


Any questions?

Thank you!

optimistic parallel discrete event simulations of physical systems using reverse computation

Documents