High-Performance ComputingHigh-Performance Computingin Examplesin Examples

Dr. Axel Kohlmeyer

Scientific Computing Expert

Information and Telecommunication SectionThe Abdus Salam International Centre

for Theoretical Physics

http://sites.google.com/site/akohlmey/

akohlmey@ictp.it

2

My Background

● Undergraduate training as physical chemist, PhD in Theoretical Chemistry, University Ulm

● Postdoctoral Research Associate,Theoretical Chemistry, Ruhr-University Bochum

● Senior IT-Support Staff, Associate DirectorCenter for Molecular Modeling,University of Pennsylvania, Philadelphia

● Associate Vice-Dean for Scientific Computation,Assistant VP for High Performance Computing,Temple University, Philadelphia

3

TOP 500 Linux Cluster / 2002

4

HPC Cluster 2002 / The Good

5

HPC Cluster in 2002 / The Ugly

6

HPC Cluster in 2012

7

8

Cluster Deployment and Usage

CPU cores online (incl. storage, admin)

CPU cores online (incl. storage, admin)

Running processesRunning processesLoad averageLoad average

Nodes onlineNodes online

9

Batch System Usage / Service Units

April May June July August September October November December January0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

1,000,000

normalmanycorehighmemgpudevelall

Se

rvic

e U

nits

10

Batch System Usage / Utilization

0 %

10 %

20 %

30 %

40 %

50 %

60 %

70 %

80 %

90 %

100 %

develgpuhighmemmanycorenormal

11

DNA/Nanotube Hybrids as Sensors

VB I

VG

SiSiO2

Experiments on ssDNA-CNT

Staii, C. et al, Nano Letters, 5, (2005)

Air flowDNT flow

■ Bare Nanotube

■ Nanotube + ssDNA

Detected Chemicals

Methanol

Dinitrotoluene Propionic Acid

Dimethyl methylphosphonate

Trimethylamine

DNA Sequence Specific Response

Detected Chemicals

Methanol

Dinitrotoluene Propionic Acid

Dimethyl methylphosphonate

Trimethylamine

■ Sequence A

■ Sequence B

Air flowMethanol flow

Self-Assembly of ssDNA-CNT

Right-handed Helix

Left-handed Helix

Kinked Structure

G

A

T

C

0.7 eV

0.64 eV

0.59 eV

0.53 eV

System equilibrates within ~30 ns

Many possible conformations

Adsorption driven by stacking Captures 67% of adsorption energy

Purines exhibit largest stacking energy G > A > T > C

Kinked structure stabilized by intra-DNA stacking interactions

15

Ergodicity Problem with MDP

oten

tial

E

ner

g y

ssDNA Conformation

300 K

System Temperature

System trapped in local minimum

700 K

Pot

enti

al

En

erg y

Full ssDNA configuration space sampled

16

Replica-Exchange Technical Details

Gromacs 4.x classical MD package, with parallel replica support

Individual MD scales out at 32CPUs

64 Replicas with temperatures ranging from 290 K to 715 K

Exchange attempted every 0.6 ps

Temperatures chosen to maintain exchange probability of 20 – 30%

Run on 2048 CPUs of IBM Blue Gene/L at SDSC in 2007

More than 3.8×106 ssDNA conformations sampled (each RE-attempt)

17

Simulation Results

R. R. Johnson, A. Kohlmeyer, A. T. C. Johnson and M. L. KleinNano Letters, 2009, 9 (2), pp 537–541

Self-Assembly fromCoarse-Grain MD

PEG Surfactant Example

= HOCH2-

= -CH2OCH2-

= -CH2CH2CH2-

= -CH2CH3

Building Blocks of PEG surfactant

C12E2

Shinoda et. al. Mol. Sim., 33, 17 (2007)

Parameterized from the ground up!

One CG particle per ~10 atoms.

Fit bulk systems of alkanes, alcohols, glycols and ethers to reproduce surface tension and density using LJ 9-6.

Application: Phase diagram / C12E6

Jonsson et al., ‘Surfactants and Polymers in Aqueous Solution’

X X X

PEG C12E6 Phase Transition

807,360 CG beads61696 PEG molecules

Start 50 wt% PEG(Hexagonal phase)

80 wt% PEG(Lamellar)

Dehydrate

80 ns

PEG C12E6 Formation of Hexagonal Phase

1,237,760 CG beads64,000 PEG molecules

50 wt% PEGStart random configuration

67 ns

Self-Assembly of Cyclic D,L-α-Peptides with CG-MD

● Antimicrobial Agents● Self assemble into tubes-like monomers● Induce membrane leakage● Leu-Trp repeat

Ekta Khurana, R. H. DeVane, A. Kohlmeyer and M. L. KleinNano Lett., 2008, 8 (11), pp 3626–3630

System Setup

● 220 Peptide Rings in Nonane/Water● ~ 29000 Coarse Grain Beads● 5 fs time step, Velocity Verlet● LAMMPS MD code● > 2µs trajectory

Smaller stacks are formedfrom monomers

Nanotubes assembled from stacks

27

Optimization and Parallelization

●

●

●

●

●

●

● 30M CPU hour INCITE grant on Cray XT5● Plain LAMMPS doesn't scale well enough

● Vesicle fusion study:impact of lipid ratio in binary mixture

● LAMMPS MD software● Experimental size:

4M particles for 1 vesicle and solvent

28

Questions?