dsl toolchains for performance portable geophysical fluid ...€¦ · dsl toolchains for...

Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwissDSL Toolchains for Performance Portable Geophysical Fluid Dynamic ModelsCarlos OsunaMeteoswiss

[email protected]

V. Clement, O. Fuhrer, S. Moosbrugger, X. Lapillonne, P. Spoerri, T. Schulthess, F. Thuering, H. Vogt, T. Wicky

MPI Seminar, 21st Novemebr 2017 2

DSL Toolchains for Performance PortableGeophysical Fluid Dynamic ModelsCOSMO 2km ensembles (21 members) COSMO 1km

MPI Seminar, 21st Novemebr 2017 3

DSL Toolchains for Performance PortableGeophysical Fluid Dynamic Models

Carlos Osuna, MPI Seminar, 21 November 2017 4Multi-coreGPU MIC 4

Top 500 overview Sunway TaihuLightTianhe-2TitanSequoia K ComputerPiz daint

TrinityCorioakforestGyoukouDisruptvie emergence of accelerators in HPC.Large memory bandwidth have speeded upour models. On NVIDIA GPUs:• dynamical cores – 2-3x• physical parametrizations 3-7x• spectral transforms > 10x

Carlos Osuna, MPI Seminar, 21 November 2017 5

How to achieveportabilityfor our GFD models ?


COSMO global 1kmDiscussion paper under reviewhttps://www.geosci-model-dev-discuss.net/gmd-2017-230/


A domain specific language (DSL) for GFDs:• abstracts away all unnecessary details of a programminglanguage (Fortran) implementation• concise syntax: only those keywords needed to express thenumerical problem The GridTools Ecosystem• Set of C++ APIs / Libraries• DSL for PDEs• Large class of problems• Performance Portability• Separation of concerns• Interface to Fortran• Open source license


DSL for Weather Codes on Unstructured GridsDSLs are a successful approach to deal with the software “productivity gap”ESCAPE develops a DSL based on GridTools: library tools for solving PDEs on multiple grids. The DSL syntax requires only the grid-point operation:Tiled loop templates and parallelization are abstractedUse of fast on-chip memory is also abstracted. Grid is abstracted (in combination with Atlas)��) = -4* u + u(i+1) + u(i-1) + u(j+1) + u(j-1)


Writing Operators using DSL


Global Grids• New DSL constructs for stencils on global grids• Expose a unstructured grid API• Leverage grid structure for performanceIcosahedral Cubed sphere Octahedral


Grid Abstraction15

Icosahedral OctahedralDual-grid Cube sphere

constepxr auto offsets = connectivity<edges, vertices, Color>::offsets();

for( auto off: offsets) {eval(flux) += eval( pD(off) );

}

eval(flux) = eval(sum_on_vertices(0.0, pD));• The DSL syntax elements are grid independent. • The same used code using DSL can be compiled for multiple Grids. • GridTools will support efficient native layouts for octahedral/icosahedral. • GridTools will use Atlas in order to support any grid.


Example of GridTools DSL : MPDATA


m_upwind_fluxes = make_computation<gpu>(domain_uwf, grid_,make_multistage(execute<forward>(),make_stage<upwind_flux, octahedral_topology_t, edges>(p_flux(), p_pD(), p_vn()),make_stage<upwind_fluz, octahedral_topology_t, vertices>(p_fluz(), p_pD(), p_wn()),make_stage<fluxzdiv, octahedral_topology_t, vertices>(p_divVD(), p_flux(), p_fluz(), p_dual_volumes(), p_edges_sign()),make_stage<advance_solution, octahedral_topology_t, vertices>(p_pD(), p_divVD(), p_rho())));Full MPDATA implemented using the GridTools DSL:� upwind fluxes� minmax limiter� compute fluxes� Limit fluxes� Flux solution� Rho correction

Composition of multiple MPDATA operators


Grid Abstraction: Performance Portability19

Thanks to abstraction of the mesh wecan implement any partitioning:

O32 meshEqual partitionerimplemented with Atlas

Structured partitionerimplemented with Atlas


Atlas: a library for NWP and climate modelling20Deconinck et al. 2017

Willem Deconinck


Grid Abstraction: Performance Portability21

Abstraction of the mesh can implement any indexing layoutIndexing has a large impact in performance:NVIDIA P100, 128x128x80, Bandwidth GB/s

SN DA 211

SN IA 270

UN IA 130

HN IA 256

B � �∑ ��


GridTools and Atlas E� solely relies on a C++ compiler � No external tools� provides a high-level of abstraction � Performance portabilityBut writing in this new programming modelE � is hard (C++ expertise) � Decreased productivity� is unsafe � No protection from violating the parallel model� often requires expert knowledge to produce efficient code

DSLs and abstractions: state of the art


What is the Future of GFD Models in a complex HPC environment?1. Need to increase model development productivity2. Need to decrease maintainence cost of models3. Need to extend tools to a large set of models, methods, domains.4. Need to maximize efficiency


Need to increase modeldevelopment productivityHigh level DSLs, concise, safe and intuitive syntaxthat remove boiler-platefrom embedded GridTools


Need to decrease maintainence of modelsCheck for parallel errors,Auto-generation of boundaryconditions and halo exchangesCheck for out of boundsSafe generation of loop datadependenciesE.


Need to extend tools to a large set of models, methods, domains.ICON-atmos IFS COSMOLFRicICON-Ocean

GridTools LisztPsyClone HybridFortranFiredrakeCLAWGFD modelsDSLs and programmingmodels

Options:• Unify all tools in a single DSL software stack -> maintainenceissue.• One DSL (programming model) per model, currently leadingto explosion of DSL solutions.• ESCAPE2


Need to extend tools to a large set of models, methods, domains.ICON IFS COSMOLFRicICON-Ocean

GridTools LisztPsyClone HybridFortranFiredrakeCLAWGFD modelsDSLs and programmingmodels

Options:• Unify all tools in a single DSL software stack -> maintainenceissue.• One DSL (programming model) per model, currently leadingto explosion of DSL solutions.• ESCAPE2


In the ESCAPE2 DNA3• Develop programming languages for GFD models thatcan increase productivity, and lower maintainance• Standardization, avoid explosion of tools.• Support multiple domains (FV weather, FV ocean, structured, unstructured, FE) in the standardization• Multiple domain languages or customizations, singletool• Language to be defined together with scientists


ESCAPE2 base technologyDAWN

GTCLANG

Modular compiler toolchain (based on modern compilers –llvm- )


ESCAPE2 base technologyDAWN

Unstructured

Structured

ICON Ocean pluginIFS collocatedFE pluginMultiple variants of a languagea[i+1]

on_cells(+,a)

Carlos Osuna, MPI Seminar, 21 November 2017 32 11/3Fabian Thüring

Example – Horizontal Diffusion

32

Carlos Osuna, MPI Seminar, 21 November 2017 33 33

stencil_function laplacian {storage phi;Do {return 4.0 * phi – phi[i+1] – phi[i-1] – phi[j+1] – phi[j-1];

}};

stencil hd_type2 {storage hd, in;temporary_storage lap;Do {vertical_region(k_start, k_end) { lap = laplacian(in);hd = laplacian(lap);}

}};

Example – Horizontal Diffusion


stencil hd_type2 {storage u, utens;temporary_storage uavg;Do {vertical_region(k_start, k_end) {uavg = (u[i+1]+u[i-1])*0.5;utens = (uavg[k+1]); }

}};

Safety First


stencil hd_type2 {storage u, utens;temporary_storage uavg;Do {vertical_region(k_start, k_end) {uavg = (u[i+1]+u[i-1])*0.5;utens = (uavg[j+1] + uavg[j-1]); }

}};

Safety First


stencil_function laplacian {storage phi;Do {return 4.0 * phi – phi[i+1] – phi[i-1] – phi[j+1] – phi[j-1];

}};stencil hd_type2 {storage hd, in;temporary_storage lap;Do {vertical_region(k_start, k_end) { lap = laplacian(in);hd = laplacian(lap);}

}};#omp parallel for nowait#acc …for(int i=0; i < isize; ++i) {for(int j=0; j < jsize; ++j) {for(int k=0; k < ksize; ++k) {udiff(i,j,k) = hd(i+1,j,k) + hd(i-1,j,k);

} } }

Interoperatbility : Escaping the languageDSL codeStandard C++ code(+OpenMP, +OpenACC, +CUDA)


Standardization:SIR { «filename» : «/code/access_test.cpp»,«stencils» : [«name» : «hori_diff»,«loc» : {«line»: 24,«column»: 8},«ast»: {«block_stmt»: {«statements»: {«expr_stmt»: {«assignment_expr»: {«left»: {«field_access_expr»: {«name»: «b»,«offset»: [], }},«right»: {«binary_op»: {«left»: {«field_access_expr»: {«name»: «a»,«offset»: [1,0,0], }},«right»: {«field_access_expr»: {«name»: «a»,«offset»: [-1,0,0], } } } } } } } } } } }

b = a(i+1) + a(i-1)GTCLANGMultiple DSL (thin) frontends can generatestandard SIR schemeGridToolsT4PyCLAWESCAPE2 / OceanPsyCloneFortranFortran

C++ Python


DAWN: Compiler optimization passes


COSMO dycore evaluation of DSL toolchainType 2 u SmagorinskyType 2 v Type 2 w Type 2

pp Horizontal Diffusion(P100)Advection( metric and time - P100 -)


Conclusions• The future of GFD models, high resolution, multidecade simulationwill bring serious computational challenges:efficiency and production cost, maintenance cost• We need to find right programming models so that adaptation tonew architectures do not hinder scientific development -> high productivity• Single efforts are not a valid model anymore, we needcollaborations• ESCAPE2 is a great opportunity to define the new language withscientists, we have the technology in place.

MPI Seminar, 21st Novemebr 2017

BACKUPS

MPI Seminar, 21st Novemebr 2017

Internal presentation, Valentin Clément

Federal Department of Home Affairs FDHAFederal Office of Meteorology and Climatology MeteoSwiss

MeteoSvizzeraVia ai Monti 146CH-6605 Locarno-MontiT +41 58 460 92 22www.meteosvizzera.ch MétéoSuisse7bis, av. de la PaixCH-1211 Genève 2T +41 58 460 98 88www.meteosuisse.ch MétéoSuisseChemin de l‘AérologieCH-1530 PayerneT +41 58 460 94 44www.meteosuisse.chMeteoSwissOperation Center 1 CH-8058 Zurich-Airport T +41 58 460 91 11 www.meteoswiss.ch

44

dsl toolchains for performance portable geophysical fluid ...€¦ · dsl toolchains for...

Documents