INCITE Overview October 31, 2012

Julia C. White, INCITE Manager whitejc@DOEleadershipcomputing.org

2

Origin of Leadership Computing Facilities

Department of Energy High-End Computing Revitalization Act of 2004 (Public Law 108-423): The Secretary of Energy, acting through the Office of Science, shall • Establish and operate

Leadership Systems Facilities

• Provide access [to Leadership Systems Facilities] on a competitive, merit-reviewed basis to researchers in U.S. industry, institutions of higher education, national laboratories and other Federal agencies

3

Separation of selection processes for Leadership-class computing and DOE capability computing

2008 ASCAC Committee of Visitors Recommendation 1

“The selection processes for leadership class and DOE capability class computing should be separated. A significant portion … of computational resources should be allocated to high-end DOE capability-class computing using a similar INCITE-type process.”

4

Leadership Computing Facility systems

Argonne LCF Argonne LCF Oak Ridge LCF System IBM Blue Gene/P IBM Blue Gene/Q Cray XK7 Name Intrepid Mira Titan

Compute nodes 40,960 49,152 18,688 Node

architecture PowerPC, 4 cores PowerPC, 16 cores AMD Opteron, 16 cores NVIDIA Kepler GPU

Processing Units 163,840 Cores 786,432 Cores 299,008 x86 Cores + 18,688 GPUs

Memory per node, (gigabytes) 2 16 32 + 6

Peak performance,

(petaflops) 0.557 ~10 ~20

5

Three primary ways for access to LCFs Distribution of allocable hours

60% INCITE 4.7 billion core-hours in

CY2013 30% ASCR Leadership

Computing Challenge

10% Director’s Discretionary

Leadership-class computing

DOE/SC capability computing

6

What is INCITE?

INCITE promotes transformational advances in science and technology through large allocations of computer time, supporting resources, and data storage at the Argonne and Oak Ridge Leadership Computing Facilities (LCFs) for computationally intensive, large-scale research projects.

Innovative and Novel Computational Impact on Theory and Experiment

7

INCITE seeks high-impact research campaigns

Examples of previous successful INCITE applications that advance the state of the art across a broad range of topics and different mission priorities

• Glimpse into dark matter • Supernovae ignition • Protein structure • Creation of biofuels • Replicating enzyme functions • Membrane channels • Protein folding • Chemical catalyst design • Combustion • Parallel algorithms

• Global climate • Regional earthquakes • Carbon sequestration • Turbulent flow • Propulsor systems • Nano-devices • Batteries • Solar cells • Reactor design • Nuclear structure

8

Nuclear Physics: The limits of the nuclear landscape

Application Performance Science Objectives, Impact • Identify how many protons and neutrons can be

bound within a nucleus • Identify the “drip lines” denoting the limits of binding • Advances toward goal of “designer nuclei,” with

uses ranging from cancer treatments to a better understanding of superconductivity

• Benefited from UNEDF SciDAC collaboration

• Custom code (co-authored by OLCF staff) used density functional theory to solve the nuclear structure, which involved a large algebraic non-linear eigenvalue problem.

• Each nucleus required two hours to calculate on 224,256-processor Jaguar system.

1. Identified ~7,000 possible combinations of protons and neutrons bound in nuclei with up to 120 protons (i.e., unbinilium).

2. Several leading models of nuclear interaction demonstrated to be in significant agreement.

3. Published in Nature, 28 June, 2012, “Limits of the Nuclear Landscape,” Erler, et al.

2012 Accomplishments

INCITE PI: James Vary, Iowa State Univ. POC: Witold Nazarewicz, U. Tennessee & ORNL

There are 767 even–even isotopes known experimentally, both stable (black squares) and radioactive (green squares). Mean drip lines and their uncertainties (red) were obtained by averaging the results of different models.

9

Materials design from first principles

Science Results Science Objectives, Impact • Provide the fundamental understanding and predictions

needed to design new materials for catalysis and energy storage

• Building off of significant effort on the performance and scalability of highly-parallelized electronic structure code, GPAW

Reduce the significant chlorinated waste from producing propylene

• Subnanometer Ag clusters may provide pathways to overcome this problem

• Structure of key molecule optimized and study completed on part of the process

Study the transformation of biofuels with efficient catalysts

• Structure of ZrO2 nanobowls in dry and hydroxylated alumina have been optimized

In support of the Institute for Atomic-efficient Chemical Transformation (IACT), an Energy Frontiers Research Center (EFRC)

INCITE PI: Larry Curtiss, Argonne National Lab.

Left: Thirty-three atom silver cluster studied as a new catalyst for propylene epoxidation. Right: Nanobowl in ZrO2 surface (1nm diam.) that is being studied for biomass conversion to fuels.

10

Science Objectives, Impact • Determine how much sea level will

rise based on three climate-change mitigation scenarios

• Examine mechanisms involved in sea-level rise, including ice melt and thermal expansion

• Impact: Quantifying sea-level rise in response to mitigation strategies

Sea level will continue to rise due to thermal expansion into 2300 under the most aggressive mitigation scenario, RCP2.6, but the rise will be slowed enough to implement adaptation measures. Under RCP4.5 and RCP8.5, there would be much less time for adaptation.

• CCSM4 on simulation ensembles for each mitigation scenario

• Simulation of 4,500 years of future climate to characterize range of response

• OLCF, NERSC, NCAR systems

Supercomputing Application Science Results

• With aggressive mitigation, average temperature increase can be stabilized below 2 or 3 degrees C above pre-industrial levels

• Sea level will continue to rise in the future for all mitigation scenarios • Aggressive mitigation measures strongly affect the rate of increase • G.A. Meehl, et al., Nature Climate Change, August 2012

INCITE PI: Warren Washington, National Center for Atmospheric Research

Aggressive Greenhouse Gas Mitigation Can Help Slow Rate

Sea-level rise: Inevitable if unmitigated

11

Science Objectives, Impact • Understanding why and how can

reduce the carbon footprint of cement usage and lower costs.

• Verifying that supplementing cement with certain materials (like fly ash or slag) improves the rheological properties of cement-based materials.

• Establishing a link between non-Newtonian matrix fluid properties and suspension rheology. How do the shape and size of supplemental materials impact the materials behavior?

• Studies requires many thousands of particles with a wide range of sizes and shapes in a non-Newtonian fluid matrix

A snapshot of one simulation showing the suspended particles with their current stress level painted on their surface. In this image, low-stress particles are drawn in outline style to better focus on the high-stress particles.

Science Results

• Validation with experiment that these models can properly predict key properties.

• Simulation using Intrepid of how stress is propagated through the fluids.

Modeling the rheological properties of concrete

INCITE PI: William George, National Institute of Standards and Technology

12

INCITE program organization

OLCF Director James Hack

ALCF Director Michael Papka

Susan Coghlan, ALCF Deputy Division Director Arthur Bland, OLCF Project Director

INCITE Program Julia White,

Manager

Science Panel Reviewers Chair works with INCITE manager to identify and

invite reviewers Computational Readiness

Review Leaders Bronson Messer, OLCF Kalyan Kumaran, ALCF

Access and Monitoring Leaders Ashley Barker, OLCF Richard Coffey, ALCF

Directors of Science Paul Messina, ALCF Jack Wells, OLCF

13

Management roles

Center directors INCITE manager • Responsible for final award decisions • Approve procedural decisions regarding

the INCITE program • Manage center activities associated

with support for INCITE projects

• Provide program leadership, including – Issue call for proposals – Manage proposal review and work with

panel chairs to identify and invite panel reviewers

– Deliver proposal ranking and initial recommendations

– Engage in outreach for INCITE

• Coordinate activities with and between the centers

• Liaison to the DOE office and point of contact for inquiries about INCITE

14

Reviewer roles

Panel peer-reviewers Computational readiness reviewers • Provide expert input to the INCITE

management • Each submittal is assessed by at least

three reviewers • Renewals are evaluated for delivery of

proposed milestones

• Center staff who assess benchmarking information in the proposal

• Each submittal is assessed by both centers

15

2013 INCITE panel peer reviewers

83 science experts participated in the 2013 INCITE panel review.

• > 50% (e.g. more than 40) of the reviewers are: — Society fellows (AAAS, APS, SIAM, IEEE, etc), — Agency awardees (ex. NSF Early Career), — Laboratory fellows, — National Academy members, — National Society presidents

• 41% participated in the 2012 INCITE review

Reviewer Affiliation

16

2013 INCITE Panel questionnaire Response rate 83% of science panel reviewers

Questionnaire* 2013 The INCITE proposals discussed in the panel represent some of the most cutting-edge computational work in the field.

4.6

The proposals were comprehensive and of appropriate length given the award amount requested.

4.1

The science panel was sufficiently diverse to assess the range of research topics being considered.

4.4

Please rate your overall satisfaction with the 2013 INCITE Science Panel review process. (ranging from 1-"very dissatisfied" to 5-"very satisfied)

4.6

*Scores range from 1 to 5 where 1 is "strongly disagree" and 5 is "strongly agree."

17

Awards decision-making

• LCF involvement from the solicitation through final award provides holistic guidance, checks, and balances

The INCITE solicitation and review seeks grand-challenge-scale proposals and carries out a rigorous peer review to identify those of the highest potential impact.

18

Awards decision-making (cont.)

• The INCITE Awards Committee, composed of the LCF management teams, identifies top-ranked proposals by: – peer-review panel rating and reports – additional considerations, such as the desire to promote use of

HPC resources by underrepresented communities

• Individual project allocations are determined in order to: – Ensure that each awarded project has sufficient allocation to enable

all or part of the proposed scientific or technical achievements – Maximize the scientific support provided to each INCITE project – Allocate all of the available INCITE hours at each site

19

INCITE criteria “access on a competitive, merit-reviewed basis”*

1 Merit criterion Research campaign with the potential for significant domain and/or community impact

3 Eligibility criterion • Grant allocations regardless of funding source

(*see DOE High-End Computing Revitalization Act of 2004: Public Law 108-423)

• Non-US-based researchers are welcome to apply

2 Computational leadership criterion Computationally intensive runs that cannot be done

anywhere else: capability, time-to-solution, architectural needs

20

INCITE twofold review Peer Review INCITE Panels

Computational Readiness LCF Centers

New Proposal Assessment

• Scientific and/or technical merit and originality

• Appropriateness of proposal method, milestones given

• Team qualifications • Reasonableness

of requested resources

• Reasonableness and appropriateness of the request for the computational resources

• Technical readiness (optimization, scaling, method for hardware)

• Clarity of proposal and readiness of project

• Appropriateness of plans for new hardware

Renewal Assessment

• Change in scope • Met scientific milestones • On track to meet

future milestones

• Same • Technical progress toward

milestones • Use of resource

21

INCITE annual timeline

Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Solicitation (new and renewals)

Access and monitoring

Awards

Computational Readiness Review

Panel Peer Review

Jan

Access processing

Selection

Solicitation and Review

Selection and Awards

Access and Monitoring

22

INCITE awards history (2004-2013)

Hours allocated 4.9M 6.5M 18.2M 95M 268M 889M 1.6B 1.7B 1.7B 4.7B Projects 3 3 15 45 55 66 69 57 60 61

2004 2005 2006 2007 2008 2009 2010 2011 2013

NERSC: 3 projects and

4.9M hours awarded

ALCF, OLCF, PNNL join

INCITE

INCITE management by ALCF and OLCF;

PNNL and NERSC transition out at

end of 2009*

ALCF and OLCF: Mira, Titan, and

Intrepid

20X growth in number of projects … but 1000X growth in hours

2012

Hours requested vs allocated: ~2.3X per year ~3X per year

23

2013 INCITE statistics

Contact information Julia C. White, INCITE Manager

whitejc@DOEleadershipcomputing.org

• Request for Information helped attract new projects

• Call closed June 27th, 2012

• Total requests ~15 billion core-hours, 3x more than the 5 billion core-hours requested last year

• Number of proposals submitted increased nearly 20%

• Awards of ~5 billion core-hours for CY 2013

• 61 projects awarded of which 20 are renewals

Acceptance rates 33% of nonrenewal submittals and 100% of renewals

24

2013 award statistics, by system

Jaguar Titan Mira Intrepid

2012 INCITE 2013 INCITE 2013 INCITE 2012 INCITE 2013 INCITE

Number projects* 35 32 27 31 27

Average Project 27M 58M 78M 24M 27M

Median Project 23M 49.5M 45M 20M 25M

Titan Mira Intrepid

Total Awards (Hrs in CY2013) 1.84B 2.11B 0.721B

* Totals of 32 projects at the OLCF, 37 projects at the ALCF (many of the ALCF projects received time on both Mira and Intrepid)

25

New PI’s in INCITE

• A “new” PI has never previously led an INCITE submittal • 32% of the nonrenewal projects are led by new PI’s

– 41 new projects awarded, 13 led by new PI’s

INCITE actively engages with new research teams through outreach such as workshops, email distributions, and individual networking.

26

Examples of 2013 INCITE awards

Project Description Award Michael Klein of Temple University to model the human skin barrier. 65 M Priya Vashishta of the University of Southern California to simulate formation of tiny cracks in highly stressed regions of materials used in energy and nuclear technologies.

240 M

Tom Evans and colleagues of the Consortium for Advanced Simulation of Light-Water Reactors to model the interior of a nuclear reactor and gain information to aid design of next-generation reactors and improve the safety and performance of current reactors.

21 M

Dario Alfe of University College London to create an extensive database of energy benchmarks for non-covalent forces, setting new standards and breaking new ground in examining ubiquitous weak bonding paradigms.

95 M

Boris Kozinsky of the Robert Bosch Corp. to discover and optimize new classes of solid, inorganic lithium-ion electrolytes with properties necessary for electrical energy storage.

48 M

Umesh Paliath of GE to gain a physics-based understanding of factors that affect the efficiency of jet engines and noise generation in wind turbines.

105 M

Complete list at http://www.doeleadershipcomputing.org/awards/2013INCITEFactSheets.pdf

27

PI obligations

• Provide quarterly status updates (on supplied template) – Milestone reports – Publications, awards, etc., related to the work

• Provide highlights on accomplishments • Complete annual surveys • Identify project participants and approve account requests • Acknowledge ALCF or OLCF in products of research

By establishing a connection with the PI and project participants, we’re better positioned to address questions or problems, early.

28

Sample project quarterly report topics

• Status of work on milestones, summarize accomplishments

• Productivity details • Center feedback • Code description and

characterization

29

Sample project usage report

• Track allocation burn rate • Monitor job profile, with

breakdowns of job size distribution by bins based on percent system used

30

LCFs’ support for INCITE

• “Two-pronged” support model is shared • Specific organizational implementations differ slightly

but user perspective is virtually identical

Scientific computing

Liaisons

Visualization

Performance

End-to-end workflows

Data analytics and visualization

Catalysts Performance engineering

User assistance and outreach User service and outreach

31

Processes for scientific support

• Queuing policies are designed to enable capability jobs • Reservations are a “matter of course” and a standard

environment for running • Specialists actively participate in projects, providing e.g.,

– Advanced algorithmic and implementation assistance – Assistance in data analytics and workflow management,

visualization, and performance engineering

Every INCITE project is assigned an LCF staff person for scientific and technical support.

32

Request a start up account • Director’s Discretionary Proposals considered year-round • Award up to millions of hours • Allocated by LCF center directors

• Director’s Discretionary (DD) requests can be submitted anytime

• DD may be used for porting, tuning, scaling in preparation for an INCITE submittal

• Submit applications at least 2 months before INCITE Call for Proposals closes

Argonne DD Program: www.alcf.anl.gov/getting-started/apply-for-dd Oak Ridge DD Program: www.olcf.ornl.gov/support/getting-started/olcf-director-discretion-project-application/

33

Contacts

For details about the INCITE program:

http://www.doeleadershipcomputing.org INCITE@DOEleadershipcomputing.org

2014 Call for Proposals will open in April, 2013

The INCITE program seeks proposals for high-impact science and technology research challenges that require the power of the

leadership-class systems