performance portability of the aeras atmosphere model and ......performance portability of the aeras...

Photos placed in horizontal position

with even amount of white space

between photos and header

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin

Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. SAND NO. 2011-XXXXP

Performance Portability of the Aeras Atmosphere Model and FELIX Land-Ice Model to Next Generation Architectures using Kokkos

Jerry Watkins & Irina Tezaur

Sandia National Laboratories2017 Albany Users Meeting, January 18th, 2017

SAND2017-8122 PE

Climate models need higher resolutions and more computational power

High performance computing architectures are becoming increasingly more heterogeneous

Motivation

SAND2017-8122 PE

Earth System Model (ESM)

ESM has six modular components:1. Atmosphere model

2. Ocean model

3. Sea ice model

4. Land ice model

5. Land model

6. Flux coupler

DOE funded collaboration between several national laboratories

Flux Coupler

Sea Ice

OceanAtmosphere

Land Surface Ice Sheet

SAND2017-8122 PE

Earth System Model (ESM)

ESM has six modular components:1. Atmosphere model

2. Ocean model

3. Sea ice model

4. Land ice model

5. Land model

6. Flux coupler

DOE funded collaboration between several national laboratories

This presentation will focus on atmosphere

Flux Coupler

Sea Ice

OceanAtmosphere

Land Surface Ice Sheet

SAND2017-8122 PE

Aeras

Next Generation Global Atmosphere Model Uncertainty Quantification and Performance Portability

Utilizes state-of-the-art C++ libraries Trilinos (discretizations, meshing, coupling, performance portability)

Albany (multiphysics application code base)

Performance portability through Kokkos Parallel performance across a variety of different architectures

SAND2017-8122 PE

Global Atmosphere Model

3D Hydrostatic Equations Conservation equations similar to Euler/NS

Spectral element, explicit time integration

Gauss-Lobato points

diagonal mass matrix

Quadrilateral elements on a spherical shell domain

Finite difference in hybrid vertical coordinate system

Stabilization through hyperviscosity

SAND2017-8122 PE

Parallelism on modern hardware

Memory access time has remained the same

Single core performance has improved but stagnated

More performance from multicore/manycore processors

Compute is cheap, memory transfer is expensive!

Year Memory Access Time Single Core Cycle Time

1980s ~100 ns ~100 ns

Today ~50-100 ns ~1 ns

SAND2017-8122 PE

Performance Portability

Problem: Computer architectures rapidly changing Leads to heterogeneous clusters

Trends remain the same (i.e. increased computational power through manycore architectures)

Kokkos programming model C++ library which provides performance across multiple

computing architectures

Examples: Multicore CPU, GPU, Intel Xeon Phi and more

Abstracts data layouts for optimal performance

Allows researchers to focus on algorithm development for large heterogeneous clusters Write an algorithm once for multiple architectures

SAND2017-8122 PE

Performance Porting using Kokkos

Aeras finite element assembly is organized into evaluator classes

Simple Example: Computing Pressure

SAND2017-8122 PE


Loops are parallelized using inline functions Added to class definition:

Modified member function:

SAND2017-8122 PE


The same code can be used to run on many “devices” (e.g. OpenMP and CUDA)

For CUDA/GPUs, “A” and “B” must be Kokkos DynRankViewsand declared within the class.

Data transfer from host to device handled by CUDA UVM

“Kokkos::parallel_for” is mostly used in the code, “Kokkos::atomic_fetch_add” used for filling the sparse matrix

SAND2017-8122 PE

Problem Specification

Baroclinic Instability Test Case:

100 explicit RK43 iterations, 3rd order elements, 10 vert levels

Mesh Resolution # Elements Fixed dt Hyperviscosity Tau

uniform_30 1° 5,400 30 5.0e15

uniform_60 0.5° 21,600 10 1.09e14

uniform_120 0.25° 86,400 5 1.18e13

SAND2017-8122 PE

Computer Architectures

Shannon used for testing, performance tests 32 nodes w/ varying types/numbers of GPUs per

node (10 nodes w/ one NVIDIA K80 dual-GPU)

Titan used for full length simulations, performance tests 18,688 nodes w/ one NVIDIA K20X GPU per node

SAND2017-8122 PE

OpenMP Strong Scalability

uniform_30 test case is faster on Shannon

Poor scaling

SAND2017-8122 PE

OpenMP Strong Scalability

Efficiency = 100 ×Speedup

#Cores

Poor scaling

SAND2017-8122 PE

MPI+OpenMP Strong Scalability

uniform_30 test case

16 MPI ranks vs. 2 MPI ranks + 8 OpenMP threads per node

Pure MPI scales better on Shannon

SAND2017-8122 PE

MPI+OpenMP Strong Scalability


16 MPI ranks vs. 2 MPI ranks + 8 OpenMP threads per node

Pure MPI scales better on Titan

SAND2017-8122 PE

Workset Size on Shannon


GPU memory limit reached when using less than 8 GPUs

~4x OpenMP Speedup, ~1x GPU Speedup (675 workset size)

SAND2017-8122 PE

Weak Scalability on Titan

~3x OpenMP Speedup

~0.5x GPU Speedup

Workset size too small (168 elements per GPU)

SAND2017-8122 PE

Methods for Improvement

Reduce excess memory usage

Parallelize over other indices (e.g. vertical levels)

Utilize shared memory for interpolation

Replace CUDA UVM with manual memory transfer

Profile using ‘tau’ and ‘nvprof’

SAND2017-8122 PE

Conclusions

A performance portable implementation of the 3D hydrostatic equations was implemented using Kokkos

Heterogeneous high performance computing architectures can now be utilized for atmospheric research in Aeras

Performance studies show that further optimization is needed to fully utilize all resources

https://github.com/gahansen/Albany/wiki/Albany-performance-on-next-generation-platforms

SAND2017-8122 PE

Future Work (Albany/FELIX)

Generate performance profiles (Xeon Phi, P100 GPU)

Identify performance bottle necks on GPU (nvprof/NVTX)

Finish porting and evaluate next generation solvers for FELIX

Improve performance of the finite element assembly

SAND2017-8122 PE

performance portability of the aeras atmosphere model and ......performance portability of the aeras...

Documents