the pencil code: multi-purpose and multi-user maintained
Post on 14-Jan-2016
50 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Pencil Code: multi-purpose The Pencil Code: multi-purpose and multi-user maintainedand multi-user maintained
Axel Brandenburg
(Nordita, Copenhagen)
Axel Brandenburg: The Pencil-Code 2
OverviewOverview
• Pencil formulation (advantages, headaches)
• Structure of code, cvs maintainence
• High-order schemes, tests
• Peculiarities on big linux clusters
• Online data processing
Axel Brandenburg: The Pencil-Code 3
Pencil CodePencil Code
• Started in Sept. 2001 with Wolfgang Dobler
• High order (6th order in space, 3rd order in time)
• Cache & memory efficient
• MPI, can run PacxMPI (across countries!)
• Maintained/developed by many people (CVS!)
• Automatic validation (over night or any time)
• Max resolution so far 10243
Axel Brandenburg: The Pencil-Code 4
Range of applicationsRange of applications
• Isotropic turbulence– MHD (Nils), passive scalar, cosmic rays
• Stratified layers– Convection, radiative transport (Tobi)
• Shearing box• MRI (Nils), Planetesimals (Anders), Interstellar (Tony)
• Sphere embedded in box– Fully convective stars (Dobler), geodynamo (McMillan)
Axel Brandenburg: The Pencil-Code 5
Pencil formulationPencil formulation
• In CRAY days: worked with full chunks f(nx,ny,nz,nvar)
– Now, on SGI, nearly 100% cache misses
• Instead work with f(nx,nvar), i.e. one nx-pencil• No cache misses, negligible work space, just 2N• Communication before sub-timestep• Then evaluate all derivatives, e.g. call curl(f,iA,B)
– Vector potential A=f(:,:,:,iAx:iAz), B=B(nx,3)
Axel Brandenburg: The Pencil-Code 6
A few headachesA few headaches
• All operations must be combined– Curl(curl), max5(smooth(divu)) must be in one go
• rms and max values for monitoring– call max_name(b2,i_bmax,lsqrt=.true.)– call sum_name(b2,i_brms,lsqrt=.true.)
• Similar routines for toroidal average, etc
• Online analysis (spectra, slices, vectors)
Axel Brandenburg: The Pencil-Code 7
CVS maintainedCVS maintained
• pserver (password protected)– Public (check-out only), private (ci/co, 20 people)
• Set of 10 test problems
• Nightly auto-test (different machines, web)
• Before check-in: run auto-test yourself
• Mpi and nompi dummy module for single processor machine (or use lammpi on laptops)
Axel Brandenburg: The Pencil-Code 8
Switch modulesSwitch modules
• magnetic or nomagnetic (e.g. just hydro)
• hydro or nohydro (e.g. kinematic dynamo)
• density or nodensity (burgulence)
• entropy or noentropy (e.g. isothermal)
• radiation or noradiation (see Tobi’s talk)
• dustvelocity or nodustvelocity (planetesimals)
Axel Brandenburg: The Pencil-Code 9
Features, problemsFeatures, problems
• Namelist (can freely introduce new params)
• Upgrades forgotten on no-modules (auto-test)
• SGI namelist problem (see pencil FAQs)
10
Pencil Code check insPencil Code check ins
Axel Brandenburg: The Pencil-Code 11
High-order schemesHigh-order schemes
• Alternative to spectral or compact schemes– Efficiently parallelized– No transpose necessary
• 6th order central differences in space• Non-conservative scheme
– Allows use of logarithmic density and entropy– Copes well with strong stratification and
temperature contrasts
Axel Brandenburg: The Pencil-Code 12
High-order spatial schemesHigh-order spatial schemes
x
fffffff iiiiii
i 60
945459 321123'
2321123''
180
227270490270272
x
ffffffff iiiiiii
i
Main advantage: low phase errors
Axel Brandenburg: The Pencil-Code 13
Wavenumber characteristicsWavenumber characteristics
Axel Brandenburg: The Pencil-Code 14
Higher order – less viscosityHigher order – less viscosity
Axel Brandenburg: The Pencil-Code 15
Less viscosity – also in Less viscosity – also in shocksshocks
Axel Brandenburg: The Pencil-Code 16
High-order temporal schemesHigh-order temporal schemes
),( 111 iiiii utFtww
Main advantage: low amplitude errors
iiii wuu 1
3)()(
0 , uuuu nn
2/1 ,1 ,3/1
1 ,3/2 ,0
321
321
1 ,2/1
2/1 ,0
21
21
1
0
1
1
3rd order
2nd order
1st order
2N-RK3 scheme (Williamson 1980)
Axel Brandenburg: The Pencil-Code 17
Shock tube testShock tube test
Axel Brandenburg: The Pencil-Code 18
Hydromagnetic turbulence Hydromagnetic turbulence and subgrid scale models?and subgrid scale models?
• Want to shorten diffusive subrange– Waste of resources
• Want to prolong inertial range– Focus of essential physics
• Reasons to be worried about hyperviscosity– Shallower spectra– Wrong amplitudes of resulting large scale fields
42
2
Axel Brandenburg: The Pencil-Code 19
Simulations at 512^3Simulations at 512^3
Withhyperdiffusivity
Normaldiffusivity
Biskamp & Muller (2000)
Axel Brandenburg: The Pencil-Code 20
256 processor run at 1024256 processor run at 102433
Axel Brandenburg: The Pencil-Code 21
MHD equationMHD equation
JBuA
t
fuuBJu
)(1
lnD
D3122
sc
t
utD
lnD
AB
BJ
Induction
Equation:
Magn.Vectorpotential
Momentum andContinuity eqns
Axel Brandenburg: The Pencil-Code 22
Vector potentialVector potential
• B=curlA, advantage: divB=0• J=curlB=curl(curlA) =curl2A• Not a disadvantage: consider Alfven waves
z
uB
t
b
z
bB
t
u
00 and ,
uBt
a
z
aB
t
u02
2
0 and ,
B-formulation
A-formulation 2nd der onceis better than1st der twice!
Axel Brandenburg: The Pencil-Code 23
Wallclock time versus processor #
Axel Brandenburg: The Pencil-Code 24
Sensitivity to layout onSensitivity to layout onLinux clustersLinux clusters
yprox x zproc
4 x 32 1 (speed)
8 x 16 3 times slower
16 x 8 17 times slower
Gigabituplink 100 Mbit
link only
24 procsper hub
25
Why this sensitivity to layout?
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
6 7 8 9 0 1 2 3 4
All processors need to communicatewith processors outside to group of 24
26
Use exactly 4 columns
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
16 17 18 19
20 21 22 23
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
Only 2 x 4 = 8 processors need to communicate outside the group of 24 optimal use of speed ratio between 100 Mb ethernet switch and 1 Gb uplink
Axel Brandenburg: The Pencil-Code 27
Fragmentation over many switches
Axel Brandenburg: The Pencil-Code 28
Animation of uzAnimation of uz
Axel Brandenburg: The Pencil-Code 29
Animation of B vectorsAnimation of B vectors
Axel Brandenburg: The Pencil-Code 30
Animation of B vectorsAnimation of B vectors
Axel Brandenburg: The Pencil-Code 31
Animation of energy spectraAnimation of energy spectra
Axel Brandenburg: The Pencil-Code 32
Saturation behavior explained by Saturation behavior explained by magnetic helicity conservationmagnetic helicity conservation
termssurface2 BJBA dt
d
0BJSteady state,closed box
0 bjBJSmall scale andlarge scalecurrent helicityin balance
2f
21 bB kk
Axel Brandenburg: The Pencil-Code 33
With hyperdiffusivityWith hyperdiffusivity
23231 f
bB kk for ordinaryhyperdiffusion
42k
Axel Brandenburg: The Pencil-Code 34
ConclusionsConclusions• Subgrid scale modeling can be unsafe (some problems)
– shallower spectra, longer time scales, different saturation amplitudes
• High order schemes– Low phase and amplitude errors
– Need less viscosity
• 100 MB link close to bandwidth limit• Comparable to Origin• 2x faster with GB switch• 100 MB switches with GB uplink optimal
top related