OCCA

We are grateful and wish to acknowledge the grants from Shell, Department of Energy, and Argonne National Laboratory, together with support from the AMD and the Shell Fellowship

David Medina*, Amik St-Cyr† , & Tim Warburton* *Department of Computational and Applied Mathematics, Rice University; †Computation & Modeling, Shell

OCCA: A Unified Approach to Multi-Threading Languages

Finite Difference OCCA Kernel!// External variables are defined as compiler directives!occaKernel void fd2d(occaKernelInfoArg,! occaPointer double *u1,! occaPointer double *u2,! occaPointer double *u3){! occaOuterFor1{! occaOuterFor0{! occaShared double Lu[Gi + 2*r][Gj + 2*r]; // Shared storage! occaPrivate(double, r_u2);! occaPrivate(double, r_u2);!! occaInnerFor1{! occaInnerFor0{! const int Li = occaInnerId0;! const int Lj = occaInnerId1;!! const int i = occaGlobalId0;! const int j = occaGlobalId1;!! const int id = j*w + i; // w = nodes in x direction! // h = nodes in y direction! if( (i < w) && (j < h) ){ // Bounds check!! r_u2 = u2[id]; // Global to register memory! r_u3 = u3[id];!! Lu[Lj][Li] = u2[nY1*w + nX1];!! const int nX1 = (i - r + w) % w;! const int nY1 = (j - r + h) % h;!! const int nX2 = (i + Gi - sr + w) % w;! const int nY2 = (j + Gj - sr + h) % h;!! if(Lj < 2*r)! Lu[Lj + Gy][Li] = u2[nY2*w + nX1];!

SummaryRiDG

For each element:! For each volume node:! Compute divergence at volume node

Volume Kernel

Surface Flux Kernel

dqhdt

= −∇h ⋅Fh − Lh nh ⋅ Fh* − Fh

−( )( ) For each element:! For each surface node:! Compute flux at surface node! For each volume node:! Calculate flux contribution at volume node

∂q∂t

+∇ iF = S

Discontinuous Galerkin seismic forward modeling

GPU Platforms

Est.

GFL

OPS

0

250

500

750

1000

Polynomial Order

2 3 4 5 6

OpenCL (Tahiti)OpenCL (Titan)CUDA (Titan)

CPU Platforms

Est.

GFL

OPS

0

45

90

135

180

Polynomial Order

2 3 4 5 6

OpenCL-AMDOpenCL-IntelOpenMP

CPU Platforms (Vec)

Est.

GFL

OPS

0

45

90

135

180

Polynomial Order

2 3 4 5 6

OpenCL-AMDOpenCL-IntelOpenMP (g++)OpenMP (icpc)

General Conservative-Law Form

OCCA

CUDA-x86

CPU Processors

NVIDIA GPUs

FPGAs

AMD GPUs

Xeon Phi

CPU Processors

NVIDIA GPUs

FPGAs

AMD GPUs

Xeon Phi

PTX Assembly

GPU Ocelot

CU2CL

SWAN

PGI: CUDA-x86

OCCA-Enabled Applications

• Discontinuous Galerkin seismic forward modeling

RiDG

• Fully accelerated aggregation-based algebraic multi grid

ALMOND3

1

5

7

6

9

11

12

2

4

8

10

2

1

3

• Matrix-free preconditioned conjugate gradient - ALMOND-enabled - Spectral overlapping additive Schwarz pre conditioner

High-order Finite Elements for Elliptic Problems

• Current approaches translate CUDA for non-NVIDIA devices

• Translate to OpenCL due to similarity in languages

• GPU-Ocelot translates at the GPU assembly level (PTX, CAL, …)

Existing Approaches to Cross-Platform CUDA

• Unifies heterogenous platforms by abstracting language APIs

• Takes advantage of the similarity in platform optimization techniques

• Macro-based kernel language masks different supported languages

OCCA: Platform and Hardware Pairing

• Simplest wave model

Acoustic Wave Equation

ptt � c2�p = 0

• The programmer has to expose parallelism in memory access and operations.

Parallelism*:

• Naively porting OpenMP to CUDA or OpenCL likely will yield low performance.

• Manufacturers devote resources to enhancing specific APIs.

Performance*:

• Code life cycle measured in decades.

• Architecture & API life cycles measured in Moore doubling periods.

• Example: if you coded for the IBM Cell processor API…

Uncertainty:

• CUDA, OpenCL, OpenMP, OpenACC, Intel TBB, etc are not code compatible.

• Not all APIs are installed on any given system.

Portability:

CPU Platforms

Est.

MN

odes

/s

0

175

350

525

700

(1D) Stencil Size

5 7 9 11 13

OpenCL-AMDOpenCL-IntelOpenMP-icpcOpenMP-g++

GPU Platforms

0

3250

6500

9750

13000

(1D) Stencil Size

5 7 9 11 13

OpenCL (Tahiti)OpenCL (Titan)CUDA (Titan)

Project References1. J. Hesthaven & T. Warburton, Nodal discontinuous Galerkin methods: algorithms, analysis, and applications, vol. 54, Springer, 2008.

2. R. Gandham, K. Esler, & Y. Zhang, A GPU accelerated aggregation algebraic multigrid method, Computers & Math with Applications.

3. D. Medina, R. Gandham, & T. Warburton, gNek: A GPU accelerated spectral-element Navier-stokes solver, (in progress).