Algorithms Algorithms OverviewOverview

Hernan G. ArangoHernan G. ArangoInstitute of Marine and Coastal SciencesInstitute of Marine and Coastal Sciences

Rutgers UniversityRutgers University

2004 ROMS/TOMS European Workshop2004 ROMS/TOMS European WorkshopCNR-ISMAR, Venice, October 18-20CNR-ISMAR, Venice, October 18-20

ean M od

earch C o m

T e r r a i n - F o l l o w i n g

O

c e a n M o d e l i n g S y s t e m

O p e r a t i o n a l C o m m u n i t y

OutlineOutline

• ROMS/TOMS algorithms statusROMS/TOMS algorithms status

• ROMS/TOMS future releasesROMS/TOMS future releases

• How does one build an adjoint model?How does one build an adjoint model?

• Ensemble predictionEnsemble prediction

• Variational data assimilation:Variational data assimilation:

Strong constraint 4DVARStrong constraint 4DVAR

Weak constraint 4DVARWeak constraint 4DVAR

• Final remarksFinal remarks

ROMS/TOMS 2.1 FeaturesROMS/TOMS 2.1 Features

• Fasham model revisited (Fennel)Fasham model revisited (Fennel)

• Bio-optical model (up to 84 components), EcoSim Bio-optical model (up to 84 components), EcoSim

(Bissett)(Bissett)

• New bottom boundary layer (Blaas); Fixed Styles and New bottom boundary layer (Blaas); Fixed Styles and

Glenn BBLGlenn BBL

• Sediment model revisited: stratigraphy with Nbed Sediment model revisited: stratigraphy with Nbed

layers (Warner)layers (Warner)

• Momentum and tracer balances (Crowley)Momentum and tracer balances (Crowley)

• Time-averaged quadratic terms: <uu>, <uv>, <vv>, Time-averaged quadratic terms: <uu>, <uv>, <vv>,

<uT>, <vT><uT>, <vT>

• Isobaric Lagrangian trajectories (Warner)Isobaric Lagrangian trajectories (Warner)

ROMS/TOMS 2.1 FeaturesROMS/TOMS 2.1 Features

• Sequential and concurrent coupling with atmospheric Sequential and concurrent coupling with atmospheric

models (Moore, Shaffer)models (Moore, Shaffer)

ESMF (ESMF (initializeinitialize, , runrun, , finalizefinalize))

Atmospheric coupler: Modeling coupling toolkit Atmospheric coupler: Modeling coupling toolkit

(MCT, Argonne National Lab) and WRF I/O API(MCT, Argonne National Lab) and WRF I/O API

MPI communicator is split between atmosphere MPI communicator is split between atmosphere

and ocean nodesand ocean nodes

ROMS/TOMS 2.1 Fixed BugsROMS/TOMS 2.1 Fixed Bugs

• Horizontal viscosityHorizontal viscosity

• Parallel periodic boundariesParallel periodic boundaries

• Tiling in serial applicationsTiling in serial applications

• Added river mass transport to Added river mass transport to DU_avg1DU_avg1 and and DV_avg1DV_avg1

arraysarrays

• MPI parallel bug in restart of floats NetCDFMPI parallel bug in restart of floats NetCDF

ROMS/TOMS 2.2 FeaturesROMS/TOMS 2.2 Features

• Ice modelIce model

• Nesting / composed gridsNesting / composed grids

• Parallel IOParallel IO

• Improvements to sediment modelImprovements to sediment model

• Monotonic tracer advectionMonotonic tracer advection

Serial Versus Parallel NetCDF

(Yang, NCSA)

NCSA IBM P690

16

16

16

16 Serial

Parallel

Serial

Parallel

Timestep

Timestep

Outp

ut

Tim

e (

0.1

s)

Outp

ut

Tim

e (

0.1

s)(246 x 240 x 16)

(656 x 640 x 16)

Serial Versus Parallel NetCDF

Serial 128

64

32

16

128

128

Serial

Parallel

(Yang, NCSA)

NCAR IBM SP Cluster(WinterHawk II)

Timestep

Timestep

Outp

ut

Tim

e (

0.1

s)

Outp

ut

Tim

e (

0.1

s)

(656 x 640 x 16)

(656 x 640 x 16)

Sediment Model New FeaturesSediment Model New Features

• Suspended-sediment stratification effects in wave Suspended-sediment stratification effects in wave

boundary layer (neutral currently)boundary layer (neutral currently)

• Mechanics for cohesive versus non-cohesive Mechanics for cohesive versus non-cohesive

bottom sedimentsbottom sediments

• Gravity-driven transport in bottom boundary layerGravity-driven transport in bottom boundary layer

• Aggregation / dissaggregationAggregation / dissaggregation

• Wetting / dryingWetting / drying

• Bioturbation in sediment layersBioturbation in sediment layers

• Bedload transport (with wave effects)Bedload transport (with wave effects)

• Radiation stressesRadiation stresses

ROMS/TOMS Adjoint and Data ROMS/TOMS Adjoint and Data Assimilation TeamAssimilation Team

Hernan G. ArangoHernan G. Arango

Boon ChuaBoon Chua

Bruce D. CornuelleBruce D. Cornuelle

Emanuele Di LorenzoEmanuele Di Lorenzo

Arthur J. MillerArthur J. Miller

Andrew M. MooreAndrew M. Moore

Julio SheinbaumJulio Sheinbaum

Rutgers UniversityRutgers University

Oregon State UniversityOregon State University

Scripps Institute of Scripps Institute of OceanographyOceanography

Georgia Institute of TechnologyGeorgia Institute of Technology

Scripps Institute of Scripps Institute of OceanographyOceanography

University of ColoradoUniversity of Colorado

CICESECICESE

ObjectivesObjectives

• To provide the ocean modeling community with To provide the ocean modeling community with

analysis and prediction tools that are available in analysis and prediction tools that are available in

meteorology and Numerical Weather Prediction meteorology and Numerical Weather Prediction

(NWP), using a community OGCM (ROMS/TOMS).(NWP), using a community OGCM (ROMS/TOMS).

• To build a Generalized Stability Analysis (GSA) To build a Generalized Stability Analysis (GSA)

platform: platform: eigenmodeseigenmodes, , optimal perturbations /optimal perturbations /

singular vectorssingular vectors, , forcing singularforcing singular vectorsvectors, , stochastic stochastic

optimalsoptimals, , pseudospectrapseudospectra..

• To build an ensemble prediction platform.To build an ensemble prediction platform.

• To build 4D variational assimilation platforms.To build 4D variational assimilation platforms.

OverviewOverview•Let’s represent Let’s represent NLNL ROMS as: ROMS as:

•The The TLTL ROMS is derived by considering a small perturbation ROMS is derived by considering a small perturbation ss to to SS. A first-order Taylor expansion yields:. A first-order Taylor expansion yields:

A is real, non-symmetricA is real, non-symmetric Propagator MatrixPropagator Matrix

•The The ADAD ROMS is derived by taking the inner-product with an ROMS is derived by taking the inner-product with an

arbitrary vector , where the inner-product defines an arbitrary vector , where the inner-product defines an

appropriate norm (L2-norm):appropriate norm (L2-norm):

How To Build an AdjointHow To Build an Adjoint

• The ADM can be derived from:The ADM can be derived from: Continuous equationsContinuous equations Discrete equations (Discrete equations (AA is symmetric; exact) is symmetric; exact)

Hand-codedHand-coded Automatic differentiation adjoint compilers Automatic differentiation adjoint compilers

(TAMC)(TAMC)

• The ADM operator relative to L2-norm can be The ADM operator relative to L2-norm can be computed by multiplying each line of the TLM code computed by multiplying each line of the TLM code by the corresponding adjoint variable, and then by the corresponding adjoint variable, and then differentiating with respect the TLM variable.differentiating with respect the TLM variable.

• Use Geiring and Kaminski (1998) transpose TLM Use Geiring and Kaminski (1998) transpose TLM operators and recipes.operators and recipes.

• Non-differentiable algorithms (vertical mixing).Non-differentiable algorithms (vertical mixing).

Nonlinear ModelNonlinear Model

DO k=1,N

DO i=Istr,Iend+1 FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)* & (Hz(i,k)+Hz(i-1,k))* & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc)) END DO

DO i=Istr,Iend t(i,k,nnew,itrc)=t(i,k,nnew,itrc)+ & dt*pm(i)*pn(i)*(FX(i+1)-FX(i)) END DO

END DO

Tangent Linear ModelTangent Linear Model

DO k=1,N

DO i=Istr,Iend+1!! FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)*!! & (Hz(i,k)+Hz(i-1,k))*!! & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc))

tl_FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)* & ((tl_Hz(i,k)+tl_Hz(i-1,k))* & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc))+ & (Hz(i,k)+Hz(i-1,k))* & (tl_t(i,k,nrhs,itrc) -tl_t(i-1,k,nrhs,itrc))) END DO

DO i=Istr,Iend!! t(i,k,nnew,itrc)=t(i,k,nnew,itrc)+!! & dt*pm(i)*pn(i)*(FX(i+1)-FX(i))

tl_t(i,k,nnew,itrc)=tl_t(i,k,nnew,itrc)+ & dt*pm(i)*pn(i)*(tl_FX(i+1)-tl_FX(i)) END DO

END DO

Adjoint ModelAdjoint Model DO k=1,N

DO i=Istr,Iend!! tl_t(i,k,nnew,itrc)=tl_t(i,k,nnew,itrc) +!! & dt*pm(i)*pn(i)*(tl_FX(i+1)-tl_FX(i))

adfac=dt*pm(i)*pn(i)*ad_t(i,k,nnew,itrc) ad_FX(i+1)=ad_FX(i+1)+adfac ad_FX(i )=ad_FX(i ) -adfac END DO

DO i=Istr,Iend+1!! tl_FX(i)=0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))* pmon_u(i)*!! & ((tl_Hz(i,k)+tl_Hz(i-1,k))*!! & (t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc))+!! & (Hz(i,k)+Hz(i-1,k))*!! & (tl_t(i,k,nrhs,itrc) -tl_t(i-1,k,nrhs,itrc)))

adfac =0.25_r8*(diff2(i,itrc)+diff2(i-1,itrc))*pmon_u(i)*ad_FX(i) adfac1=adfac*(t(i,k,nrhs,itrc)-t(i-1,k,nrhs,itrc)) adfac2=adfac*(Hz(i,k)+Hz(i-1,k)) ad_Hz(i ,k)=ad_Hz(i ,k)+adfac1 ad_Hz(i-1,k)=ad_Hz(i-1,k)+adfac1 ad_t(i ,k,nrhs,itrc)=ad_t(i ,k,nrhs,itrc)+adfac2 ad_t(i-1,k,nrhs,itrc)=ad_t(i-1,k,nrhs,itrc) -adfac2 ad_FX(i) =0.0_r8 END DO

END DO

Ensemble PredictionEnsemble Prediction

• Optimal perturbations / singular vectors and Optimal perturbations / singular vectors and stochastic optimal can also be used to generate stochastic optimal can also be used to generate ensemble forecasts.ensemble forecasts.

• Perturbing the system along the most unstable Perturbing the system along the most unstable directions of the state space yields information directions of the state space yields information about the about the firstfirst and and secondsecond moments of the moments of the probability density function (PDF):probability density function (PDF):

ensemble meanensemble mean

ensemble spreadensemble spread

• Adjoint based perturbations excite the full spectrumAdjoint based perturbations excite the full spectrum

Ensemble PredictionEnsemble Prediction

t

s

HighSpread

U npredic table

timet

sLow

Spread

P redic table

time

For an appropriate forecast skill measure, For an appropriate forecast skill measure, ss

Data Assimilation OverviewData Assimilation Overview

•Cost Function:Cost Function:

wherewhere model,model, background,background, observations,observations,

inverse background error covariance,background error covariance,

inverse observations error covarianceinverse observations error covariance

•Model solution depends on initial conditions ( ), Model solution depends on initial conditions ( ), boundary conditions, and model parametersboundary conditions, and model parameters

•Minimize JMinimize J to produce a best fit between model and to produce a best fit between model and observations by adjusting initial conditions, and/or observations by adjusting initial conditions, and/or boundary conditions, and/or model parameters.boundary conditions, and/or model parameters.

MinimizationMinimization

• Perfect model constrained minimization (Lagrange Perfect model constrained minimization (Lagrange function):function):

We require the minimum of at which:We require the minimum of at which:

, , ,, , ,

yieldingyielding

• AATT is the transpose of is the transpose of AA, often called the adjoint , often called the adjoint operator. It can be shown that: operator. It can be shown that:

The adjoint equation solutionThe adjoint equation solutionprovides gradient informationprovides gradient information

4D Variational Data Assimilation Platforms 4D Variational Data Assimilation Platforms (4DVAR)(4DVAR)

• Strong Constraint (S4DVAR) drivers:Strong Constraint (S4DVAR) drivers: Conventional S4DVAR: outer loop, Conventional S4DVAR: outer loop, NLNL, , ADAD Incremental S4DVAR: inner and outer loops, Incremental S4DVAR: inner and outer loops, NLNL, , TLTL, ,

ADAD (Courtier et al., 1994) (Courtier et al., 1994) Efficient Incremental S4DVAR (Weaver et al., 2003)Efficient Incremental S4DVAR (Weaver et al., 2003)

• Weak Constraint (W4DVAR) - IOMWeak Constraint (W4DVAR) - IOM Indirect Representer Method: inner and outer loops, Indirect Representer Method: inner and outer loops,

NLNL, , TLTL, , RPRP, , AD AD (Egbert et al., 1994; Bennett et al, (Egbert et al., 1994; Bennett et al, 1997)1997)

RP:RP:

“Conventional” S4DVAR

NLM: compute model-observations misfit and cost function

ADM: compute cost function gradients

Compute NLM initial conditions using first guess conjugate gradient step size

NLM: compute change in cost functionCompute NLM initial conditions using refined conjugate gradient step size

CALL initialCALL main3d

CALL ad_initialCALL ad_main3d

CALL initialCALL main3d

CALL descentCALL wrt_ini

CALL descentCALL wrt_ini

Oute

r Lo

op

Ipass=1

Ipass=2

Incremental S4DVARCALL initialCALL main3d

Oute

r Lo

op

CALL tl_initialCALL tl_main3d

CALL ad_initialCALL ad_main3d

CALL tl_initialCALL tl_main3d

CALL descentCALL tl_wrt_ini

CALL descentCALL tl_wrt_ini

Inn

er

Loop

Ipass=1

Ipass=2

CALL ini_adjustCALL wrt_ini

NLM: compute basic state trajectory and extract model at observations locations

TLM: compute misfit cost function between model (NLM+TLM) and observationsADM: compute cost function gradients

Compute TLM initial conditions using first guess conjugate gradient step size

TLM: compute change in cost function

Compute TLM initial conditions using refined conjugate gradient step size

Compute NLM new initial conditions(NLM+TLM)

Efficient Incremental S4DVARNLM: compute basic state trajectory and extract model at observations locationsADM: compute initial estimate of the gradientInitialize conjugate direction as the negative of the gradient (adjoint) solution

RPM: compute misfit cost function between model (NLM+TLM) and observations

ADM: compute cost function gradients

Compute TLM initial conditions using conjugate gradient step size

Compute NLM new initial conditions(NLM+TLM)

CALL initialCALL main3d

Oute

r Lo

op

CALL tl_initialCALL tl_main3d

CALL ad_initialCALL ad_main3d

CALL descentCALL tl_wrt_ini

Inn

er

Loop

CALL ini_adjustCALL wrt_ini

CALL ad_initialCALL ad_main3d

CALL ini_descent

W4DVAR, IOM

iom_roms: compute first guess andmisfitbetween observation and model

nl_roms: compute basic state trajectory

Inner loop, backward (ad_roms) and forward (tl_roms) integrations to compute

an ˆ ( )d n n n nFu h R C β

ˆ nβ

nad_roms: backward integration to compute

ˆnuiom_roms: compute

nl_roms < nl_roms.in

ad_roms < ad_roms.in

tl_roms < tl_roms.in

IOM components

iom_roms < iom_roms.in

ad_roms < ad_roms.in

iom_roms < iom_roms.in

Inn

er

Loop

Oute

r Lo

op

Twin ExperimentsTwin Experiments

• Spin-up an idealized, wind-forced double-gyre for Spin-up an idealized, wind-forced double-gyre for 50 years.50 years.

• Basin dimensions: 1000x2000 kmBasin dimensions: 1000x2000 km22

• Grid resolution: dx=dy=18.518 km (54x108x4)Grid resolution: dx=dy=18.518 km (54x108x4)• Run equilibrium solution for another 5 days and Run equilibrium solution for another 5 days and

extract observations (extract observations (true statetrue state) daily for each ) daily for each state variable at every spatial grid point.state variable at every spatial grid point.

• Initialize 4DVAR algorithms from rest and Initialize 4DVAR algorithms from rest and assimilate observations at day 1.assimilate observations at day 1.

• Force only with the adjoint misfit (model minus Force only with the adjoint misfit (model minus observations) terms.observations) terms.

Free-surface and Currents

Final Adjusted Initial ConditionsAdjusted Minus Truth Solution

Free-surface DifferenceRMS = 1.568e-5

Ubar DifferenceRMS = 1.690e-5Vbar DifferenceRMS = 7.995e-6

S4DVAR

S4DVAR 3D Double Gyre

Final Adjusted Initial Conditions

Free-surface and Currents

Model-Observation Misfit Cost Function

Iteration

Free-surface Difference

Adjusted Minus Truth Solution

Vbar DifferencePotential Temperature Difference

IOMFinal Adjusted Initial Conditions

Free-surface and Currents

Adjusted Minus Truth Solution

Free-surface DifferenceRMS = 2.136e-3

Ubar DifferenceRMS = 2.960e-2Vbar DifferenceRMS = 5.085e-2

True Solution

Ongoing 4DVAR ApplicationsOngoing 4DVAR Applications

• Southern California Bight (Cornuelle, Di Southern California Bight (Cornuelle, Di Lorenzo, Miller)Lorenzo, Miller)

• U.S. East coast (Arango, Moore, Wilkin)U.S. East coast (Arango, Moore, Wilkin)

• Intra-Americas Sea (Moore, Sheinbaum)Intra-Americas Sea (Moore, Sheinbaum)

• Gulf of Mexico (Moore, Sheinbaum)Gulf of Mexico (Moore, Sheinbaum)

• East Australia Current (Arango, Wilkin)East Australia Current (Arango, Wilkin)

• Oregon coast (Durski)Oregon coast (Durski)

Observation Types

plus satellite data (SSH, SST) and radar

Timing considerationsTiming considerations

• SCB – 6 CPU minutes per simulation day per SCB – 6 CPU minutes per simulation day per TLMTLM//ADMADM call on a 833MHz Alpha (78x118x30).call on a 833MHz Alpha (78x118x30).

• GoM – 17 CPU minutes per simulation day per GoM – 17 CPU minutes per simulation day per TLMTLM//ADMADM call on a 833 MHz Alpha. call on a 833 MHz Alpha.

• IAS – 15 CPU minutes per simulation day per IAS – 15 CPU minutes per simulation day per TLMTLM//ADMADM call on a 833 MHz Alpha. call on a 833 MHz Alpha.

• NENA – 60 CPU minutes per simulation day per NENA – 60 CPU minutes per simulation day per TLMTLM//ADMADM call on a 833 MHz Alpha (384x128x30). call on a 833 MHz Alpha (384x128x30).

• Data assimilation scaling factors:Data assimilation scaling factors: S4DVARS4DVAR = 2 = 2 IS4DVARIS4DVAR = 3 inner, 0.5 outer = 3 inner, 0.5 outer EIS4DVAR = 2 inner, 0.5 outerEIS4DVAR = 2 inner, 0.5 outer W4DVAR = 2 inner, 2.5 outer.W4DVAR = 2 inner, 2.5 outer.

Final RemarksFinal Remarks

• Maintenance of Maintenance of TLMTLM, , RPMRPM, and , and ADMADM models. models.

• Parallelization of Parallelization of TLMTLM, , RPMRPM, and , and ADMADM models. models.

• Modeling background error covariance.Modeling background error covariance.

• Training and documentation.Training and documentation.

PublicationsPublications

• Moore, A.M., H.G Arango, E. Di Lorenzo, B.D. Cornuelle, Moore, A.M., H.G Arango, E. Di Lorenzo, B.D. Cornuelle, A.J. Miller and D. Neilson, 2004: A comprehensive ocean A.J. Miller and D. Neilson, 2004: A comprehensive ocean prediction and analysis system based on the tangent linear prediction and analysis system based on the tangent linear and adjoint of a regional ocean model, and adjoint of a regional ocean model, Ocean Modelling,Ocean Modelling, 7, 7, 227-258.227-258.

http://marine.rutgers.edu/po/Papers/Moore_2004_om.pdfhttp://marine.rutgers.edu/po/Papers/Moore_2004_om.pdf

• Arango, H.G., Moore, A.M., E. Di Lorenzo, B.D. Cornuelle, Arango, H.G., Moore, A.M., E. Di Lorenzo, B.D. Cornuelle,

A.J. Miller and D. Neilson, 2003:A.J. Miller and D. Neilson, 2003: The ROMS Tangent Linear The ROMS Tangent Linear and Adjoint Models: A comprehensive ocean prediction and and Adjoint Models: A comprehensive ocean prediction and analysis system, analysis system, Rutgers Tech. ReportRutgers Tech. Report..

http://marine.rutgers.edu/po/Papers/roms_adjoint.pdfhttp://marine.rutgers.edu/po/Papers/roms_adjoint.pdf