Meet the Catalysts

Yuri Alexeev, James Osborn, Katherine Riley

2

Diverse ALCF User Community

ALCF is targeted to a few very large science projects

Minimal award in 2014 expected to be at least 50M core-hours

Diverse base of users with diverse needs

Scientific support helps transition, plan and develop for unique HPC resources

Teams have broad expertise– Scientists/engineers– Scalable algorithms– Performance engineering– Scientific code development

2013 INCITE Allocations at ALCF (Mira & Intrepid)736M Intrepid, 2.1B Mira

•Astro•Particle•Nuclear•Plasma

INCITE 60%, ALCC 60%, Discretionary 10%

3

Scientific Support Teams

Two primary scientific support teams– Catalysts – 10 – Performance Engineers - 6

Diversity needed for Engagement, Planning, & Communication– Adapt to new user profiles– Cross pollinate good solutions form different communities– Understand the growth in science and algorithmic requirements

4

Scientific Support is Collaboration

The ALCF is staffed with a team of computational scientists, expert in their domain, scalable algorithms and performance engineering. Provide a "jump-start" in the use of ALCF resources Align the availability of ALCF resources with the needs of the project team Collaborate to maximize the value that ALCF can bring to our project partners Connect the needs of the scientific community with future and current hardware

• Short term, fast solutions• Compiling, Debugging, System Use

• Targeted problem resolution• Resolve a specific hard problem like

restructuring I/O• Long term collaborations

• In depth work developing significant new capabilities

• Training • Postdocs, students, community

• Understand HPC needs for different communities

• Plan for future needs• Help planning new facilities• Advise/Participate in long term code

development paths

Tactical Strategic

Two categories of collaboration and contribution to teams using the ALCF:

5

Biology

ESP PI Mark Gordon2013 INCITE PI George Schatz

• One of the largest QM/FMO DFT calculations of biological systems (18k+ atoms)

• 4096 water molecules with QM/FMO MP2

• JCTC and SC12 papers

Catalyst team: Yuri Alexeev, Graham Fletcher, Wei Jiang, Ray LoyRunning BG/Q codes: CHARMM, GAMESS, LAMMPS, NAMD, Rosetta BG/P codes: BG/Q codes + Amber, GROMACS, ….

ALCF and MCS contributions to this project include:• ALCF staff Yuri Alexeev formulated and

implemented heuristic load balancing algorithm needed to achieve scalability on 40 racks

• Former MCS staff Ashutosh Mahajan and MCS staff Sven Leyffer solved MINLP problem by using MINOTAUR

• ALCF staff Graham Fletcher and collaborator Dmitri Fedorov ported GAMESS to BG/P/Q and optimized its performance

The ALCF and MCS teams combined with the research team achieved an immense improvement of GAMESS speedup

6

Earth science

2013 INCITE PI Michael Wehner2013 INCITE PI Warren Washington

• One of the highest resolution CESM simulations (1/8 degree)

• Two SC13 paper submissions

Catalyst team: Yuri Alexeev, Ray Loy, Katherine Riley, Timothy Williams Running BG/Q codes: CESM, HIRAM, SCECBG/P codes: BG/Q codes + WRF, …

ALCF and MCS contributions to these projects include:• ALCF staff Yuri Alexeev formulated and

implemented heuristic load balancing algorithm for CESM

• CI – University of Chicago staff Sheri Mickleson ported, optimized and run CESM simulations

• MCS staff Sven Leyffer formulated and solved MINLP problem

• MCS post-dock Prasanna Balaprakash developed and applied machine learning algorithm to improve load balance in CICE

The ALCF and MCS teams combined with the research team achieved an immense speedup over the course of a few months

7

Chemistry

2012 ALCC James Chelikowsky

• Importance of many-body van der Waals forces with self-consistent screening and with density functional theory for biological systems

• PNAS 2012 paper

Catalyst team: Yuri Alexeev, Ramesh Balakrishnan, Jeff Hammond, Graham Fletcher, Anatole Lillienfeld, Nichols RomeroRunning BG/Q codes: BigDFT, CASINO, GAMESS, QMCPACK, NWChem, MADNESS, MPQC

ALCF contributions to this project include:• ALCF staff Anatole Lillienfeld proposed

and designed the systems where many-body effects are significant

• ALCF staff Anatole Lillienfeld and Jeff Hammond wrote scripts to perform ensemble simulations to screen lattice parameters for the molecular crystals

The ALCF team combined with the research team demonstrated the scientific relevance of vdW forces for drug-DNA intercalation energies, for B versus A DNA conformational energies, and for molecular crystal polymorphs

8

Material science

ESP PIs: Larry Curtiss & Jeff Greeley• Quantum Monte Carlo (QMC) codes,

CASINO and QMCPACK, highly efficient and scalable on Mira

• First QMC calculations of solid Ar, Xe and Kr (vdW-bonded materials)

• In progress, study of platinum nanoparticles to elucidate key properties relevant to catalysis

Catalyst team: Ramesh Balakrishnan, Anatole Lillienfeld, Nichols RomeroRunning BG/Q codes: ABINIT, BigDFT, CASINO, CP2K, CPMD, GPAW, QBOX, QMCPACK, Quantum Espresso, MADNESS

ALCF contributions to this project include:• ALCF post-doc Anouar Benali designed,

implemented and tuned new spline algorithms in QMCPACK

• ALCF staff Vitali Morozov designed a new efficient algorithm for one of the main spline kernels of QMCPACK

• ALCF team combined with QMCPACK developers designed and implemented a single precision version of the code allowing to double the size of the systems to study.

The ALCF team combined with the research team achieved a speedup of 2.65 of the total application time over the course of a few months

9

Astrophysics

ESP and 2013 INCITE, PI Salman Habib

• Largest cosmology simulation ever currently running on Mira with HACC

• 69% of peak on Mira and SC12 Gordon Bell Finalist

1000 Mpc

100 Mpc

20 Mpc

2 Mpc

Catalyst team: Katherine Riley, Timothy WilliamsRunning BG/Q codes: CHIMERA, FLASH, GenASIS, HACC BG/P codes: BG/Q codes + ENZO, ….

ALCF contributions to this project include:• ALCF post-doc Hal Finkel designed,

implemented and tuned key tree algorithm needed for a balanced load and data locality

• ALCF staff Vitali Morozov has redesigned the algorithm and data structures for short force evaluation kernel and tuned it to perform at 82% of the peak of the BG/Q machine

• ALCF staff Venkatram Vishwanath designed a new parallel I/O file format visualization-specific outputs, and work on the in-situ analysis

The ALCF team combined with the research team achieved an immense speedup over the course of a few months

10

Nuclear & Particle physics

ESP and 2013 INCITE PI’s:• Paul Mackenzie (Lattice QCD):

Most accurate simulations of strong nuclear force (QCD) to date

• Steve Pieper, James Vary (Nuclear Structure): Largest GFMC simulationof Carbon-12

Catalyst team: James OsbornRunning BG/Q codes: MILC, CPS, Chroma, AGFMC

ALCF contributions to these project include:• ALCF Staff James Osborn wrote an

improved version of MILC“fermion force” routine giving7x reduction in flop count and run time

• ALCF post-doc Heechang Na, with James Osborn, optimized MILC on BG/Q by replacing MPI communications with SPI and adding QPX routines

• ALCF Staff Vitali Morozov helped optimize several key routines in GFMC

The ALCF team combined with the research team achieved over a factor of 2 total speedup in the MILC code

11

Plasma physics

2013 INCITE, PI Jean C. Perez

• Studying Alfven wave turbulence as mechanism for coronal heating and solar wind

• Simulation from solar surface out to 11 solar radii—Alfven critical point

Catalyst team: Timothy WilliamsRunning BG/Q codes: GTC, IRMHD, MHD, pF3d,GYRO

ALCF contributions to this project include:• With Sameer Shende, analyzed

communication phase in code• Helped with use of optimized Blue Gene

math libraries

The ALCF team combined with the research team and Shende achieved a significant speedup—saving millions of CPU hours and allowing more complete study with 2012 INCITE award.

12

Engineering

2013 INCITE George Karniadakis

• Performed the first multiscale simulation of blood flow in a brain of a patient with an aneurysm

• SC11 Gordon Bell Finalist Honorable Mention

Catalyst team: Ramesh Balakrishnan, Marta Garcia, Vitali Morozov, Scott ParkerRunning BG/Q codes: Code_Saturne, DNS3D, DPD-LAMMPS+NekTar, Nek5000, OpenFoam, PHASTABG/P codes: BG/Q codes + Charles, Converge, PSDNS

ALCF contributions to this project include:• ALCF staff Vitali Morozov did detailed

performance analysis and tuning of main kernels for NekTar and DPD-LAMMPS, by getting improved scalability of communications and higher flop rate from vector operations

• Collaborator Leopold Grinberg has lead the effort for implementing topology aware communication strategy and MPI mapping

The ALCF team combined with the research team created highly detailed models of blood flow within the complex blood vessel networks in the brain