© Crown copyright Met Office

LFRic Coupling Requirements3rd Workshop on Coupling Technologies for Earth System Models

Steve Mullerworth April 22nd 2015

© Crown copyright Met Office

Gung Ho – 2011-2016

• Met Office, NERC, STFC partnership involving • Imperial College, STFC, Universities of Bath, Exeter,

Manchester, Leeds, Reading and Warwick

• Develop a dynamical core that is• Scientifically as good as ENDGame• Scales on future architecture• Used for future dynamics research

• Gung Ho will probably:• Use finite element/volume methods (FEM/FV)• Support a “higher order” schemes• Support quads (cubed- or diamond-sphere) or triangles

© Crown copyright Met Office

LFRic: Replacement of the UM

• Develop a replacement for the UM by around 2019

• Project name LFRic after Lewis Fry Richardson

• Replacement to go operational around 2022

• This is distinctly a Met Office project, but we would clearly like to continue and extend our collaborations

• High-level plan with 3 distinct phases (dates of phases are indicative)

© Crown copyright Met Office

Phases for LFRic

• Phase 1: 2012-2016

• Delivery of Gung Ho dynamical core

• Design of LFRic infrastructure

• Phase 2: 2016-2019

• Delivery of an LFRic atmosphere

• Results must be promising enough to commit to operational implementation stage

• Phase 3: 2019-202?

• Move to operational implementation

• Ramp down Unified Model development to zero

© Crown copyright Met Office

Gung Ho Single Model Architecture

© Crown copyright Met Office

Driver

Algorithm

PSy

Kernel

Set-upDistributed memoryTime-step controlIOCoupling (internal and external)Modular science componentsField operations (local partition)Concurrent running?Auto-generatedBreak field into chunks for kernelsDistributed memory updatesShared memory concurrencySmall chunk of field-dataContiguous memory dataCurrently, vertical column

© Crown copyright Met Office

Current status of implementation

• Dynamo package provides basic driver layer and classes for:

• Algorithms and kernels

• GungHo fields

• Flexible choice of mesh

• Flexible choice of order of FEM

• PSyclone

• Auto-generates PSy-layer

• Being used for development of GungHo science

© Crown copyright Met Office

Finite Element Methods: the basics!

• Fields represented by functions

• Function parameters defined by values held on mesh entities (cells, edges, vertices and faces)

• Higher order implies multiple degrees of freedom (dofs) per mesh entity

• Mixed finite element scheme

© Crown copyright Met Office

Function spaces

• To fully represent the field, need to know

• Value and location of degrees of freedom

• The basis functions

© Crown copyright Met Office

Degrees of Freedom and dof-maps

© Crown copyright Met Office

3D Cell example

Arrows represent dofs of a vector field

© Crown copyright Met Office

Looping up the column

• Dof-map addresses bottom layer of field

• Data in a column of dofs is contiguous in memory

• Therefore, to get data for a cell on level K, add (K-1) to values in the dof-map of cell at base of column

• By looping up column we hope to balance out cost of dof-map lookup

• Vectorisation

• Cache re-use

© Crown copyright Met Office

Coupling to Physics Schemes

• Still to be explored

• PSyKAl design should support physics schemes

• Physics is commonly column-based

• Initially will transform any high-order GungHo field to low order

• Numerical issues may need to be solved

• Scope for extending the design. E.g.

• Bigger stencils for some schemes

• Change direction of loop ordering to level first, if appropriate for performance

© Crown copyright Met Office

High to lowest order transform – 2D example

A field within a cell, using quadratic basis functions (3x3 dofs)

Transformed to 4 cells with piecewise constant basis functions

© Crown copyright Met Office

Coupling to other models and subsystems

• Meshes and order

• Support for changing semi-unstructured meshes

• (between runs, not adaptive meshes)

• Support flexibility of change of order

• How to couple fields which have basis functions?

• IO support – data formats for high-order fields.

© Crown copyright Met Office

Time-stepping

• Concurrent versus sequential models

• Potential need for flexible deployment options

• Flexible time-step length

• Model time-step may relate to CFL limit which will vary

© Crown copyright Met Office

Questions

Key Dynamo developers:LFRic team: Chris Maynard, Mike Hobson, Mike Rezny, Matthew Hambley, Maff Glover, Ricky WongGungHo Computational Science: Rupert Ford (PSyclone), Graham Riley, Paul Slavin, Eike MuellerGungHo science: Tom Melvin, Iva Kavcic