the g.f.d.l. modular ocean model users guide

The G.F.D.L.Modular Ocean Model

Users Guide

Version 1.0 - September 1991G.F.D.L. Ocean Group Technical Report Number 2

Ronald C. PacanowskiKeith W. DixonAnthony Rosati

Geophysical Fluid Dynamics LaboratoryPrinceton, New JerseyUNITED STATES DEPARTMENT OF COMMERCENational Oceanic and Atmospheric Administration

<Pacanowski, Dixon & Rosati - GFDL Ocean Group Tech. Report No. 2>

i

TABLE OF CONTENTS

TABLE OF CONTENTS .................................................................................... iDISCLAIMER NOTICE ............................................................................................... iiiAVAILABILITY OF THIS DOCUMENT....................................................................... iiiREFERENCING THE GFDL MODULAR OCEAN MODEL AND THIS DOCUMENT ivFONTS USED IN THIS DOCUMENT ......................................................................... ivAVAILABILITY OF THE G.F.D.L. MODULAR OCEAN MODEL CODE..................... iv

PREFACE ..........................................................................................................viACKNOWLEDGEMENTS...............................................................................vii

CHAPTER 1: Beginning to Work with the GFDL Modular Ocean Model(1.1) BRIEF FILE DESCRIPTIONS ..........................................................................1-2(1.2) REQUIREMENTS FOR RUNNING THE M.O.M. CODE..................................1-3(1.3) EXTRACTING THE INDIVIDUAL FORTRAN SOURCE FILES ......................1-4(1.4) GETTING ACQUAINTED WITH THE M.O.M. CODE ......................................1-5(1.5) RECOMMENDATIONS FOR M.O.M. CODE MAINTENANCE ........................1-5

CHAPTER 2: Coding Design of the GFDL Modular Ocean Model(2.1) GENERAL CODING DESIGN..........................................................................2-2(2.2) SOME NEW FEATURES & NOTES FOR USERS OF THE COX CODE ........2-3(2.3) SIMPLIFIED CALLING TREE DIAGRAMS FOR THE GFDL M.O.M. CODE..2-6

CHAPTER 3: Code Options in the GFDL Modular Ocean Model(3.1) OVERVIEW OF CODE OPTIONS....................................................................3-2(3.2) GRID OPTIONS................................................................................................3-2(3.3) EXTERNAL MODE OPTIONS .........................................................................3-3(3.4) LATERAL MIXING SCHEMES.........................................................................3-4(3.5) VERTICAL MIXING SCHEMES .......................................................................3-5(3.6) HYBRID MIXING SCHEME OPTIONS.............................................................3-7(3.7) OPTIMIZATION OPTIONS ...............................................................................3-8(3.8) FILTERING OPTIONS......................................................................................3-9(3.9) I/O OPTIONS..................................................................................................3-10


ii

(3.10) MISCELLANEOUS CODE OPTIONS..........................................................3-10

INDEX...........................................................................................................IN-1


iii

DISCLAIMER NOTICEMention of a commercial company orproduct does not constitute anendorsement by NOAA EnvironmentalResearch Laboratories.Use for publicity or advertising purposes ofinformation from this report concerningproprietary products or the tests of suchproducts is not authorized.The authors and NOAA EnvironmentalResearch Laboratories assume noresponsibility for errors in, or incorrect useof the GFDL Modular Ocean Model. It isfirst and foremost an ocean modelingresearch tool used by researchers atNOAA’s Geophysical Fluid DynamicsLaboratory. The model and this UsersGuide have been made available as aservice to the oceanographic community.No guarantees concerning the code, thedocumentation, nor their continuedsupport are intended. It is left to theindividual user to satisfy (him/her)self thata particular configuration is workingcorrectly.

AVAILABILITY OF THISDOCUMENTCopies of GFDL Ocean Group TechnicalReports can be obtained by writing theauthors at...Geophysical Fluid Dynamics Lab / NOAAPrinceton University, Box 308Princeton, New Jersey 08542Postscript files of this report may also beobtained via anonymous ftp across theinternet from a GFDL server namedgfdl.gov which has the IP address140.208.1.9

After making the anonymous ftpconnection, the files containing thisdocument can be found in the directorynamed.../pub/GFDL_MOM/UsersGuideUpdates to this Users Guide are planned,but no timetable has been set for thoseupdates. Users of the GFDL ModularOcean Model who have provided theauthors with an e-mail address that can bereached via internet connections, will benotified as updates become available.

<GFDL Modular Ocean Model Users Guide - Version 1.0>

iv

REFERENCING THE GFDLMODULAR OCEAN MODEL ANDTHIS DOCUMENTThose who use the GFDL Modular OceanModel in part of their research shouldreference the model in papers and otherpublications as as follows:

REFERENCE:

Pacanowski, R., K. Dixon and A. Rosati,“The GFDL Modular Ocean Model UsersGuide version 1.0”, GFDL Ocean GroupTechnical Report #2, (1991).

FONTS USED IN THISDOCUMENTThis document was prepared using FrameTechnology Corporation’s copywritedFrameMaker workstation publishingsoftware.In the examples in this document, we haveadopted the convention of representinguser keyboard input in FrameMaker’sbold Courier font, and computerresponses in a regular Courierfont.Fortran variable names, UNIX commandsand file names also appear in a regularCourier font

The names of GFDL Modular OceanModel code options enabled through theuse of ifdef directives are shown in a fontFrameMaker calls demibold AvantG arde.

AVAILABILITY OF THE G.F.D.L.MODULAR OCEAN MODEL CODEThe Fortran code and related support filesfor the GFDL Modular Ocean Model canbe obtained free of cost via anonymousftp on the internet. Those wishing toobtain the model in this manner can findthe files under the /pub/GFDL_MOMdirectory of the GFDL server namedgfdl.gov (IP address 140.208.1.9).Example:First, on your own machine, make adirectory and a subdirectory and changeto it:mkdir my_mom (for the MOM files)cd my_mommkdir prep_data (for the DATA files)cd prep_data

Then, do the ftp and transfer the files:ftp -i 140.208.1.9

Name (140.208.1.9:xxx): ftp

Password: anything (your last namewould do nicely)cd pub/GFDL_MOM/PREP_DATA (thisdirectory contains programs for preparingdata for use with the MOM code. It doesnot contain any datasets, and is notneeded to run the model test case.)mget * (transfer the DATA files)lcd ../ (change local directory)cd ../ (change remote directory)mget * (transfer the MOM files)quit (close the ftp connection)


v

On your local machine, change directoriesand list the contents of the directories tocheck that you have received the files:NOTE: the MOM_NEWS file will change sizeas bug reports and other updates areadded to it over time.cd ../ls -goFtotal 1672-r--r--r-- 1 19413 Apr 24 09:46 MOM_NEWSdr-xr-xr-x 2 512 Dec 5 1990 prep_data/-r--r--r-- 1 48290 Dec 14 1990 READ_ME-r-xr--r-- 1 2109 Dec 14 1990 cray.run*-r-xr--r-- 1 1524 Dec 14 1990 mergelib*-r--r--r-- 1 555072 Dec 14 1990 mom_1.0-r--r--r-- 1 361 Feb 2 1991 ocean.in-r--r--r-- 1 221239 Dec 14 1990 printout-r--r--r-- 1 3735 Dec 14 1990 splitlib.c-r-xr--r-- 1 1188 Dec 14 1990 upgrade*cd prep_datals -goFtotal 157-r--r--r-- 1 8670 Dec 14 1990 READ_ME-r-xr--r-- 1 39 Dec 14 1990 export.batch*-r-xr--r-- 1 1282 Dec 14 1990 export.data*-r--r--r-- 1 14906 Dec 14 1990 ic.F-r-xr--r-- 1 32 Dec 14 1990 ic.batch*-r--r--r-- 1 211 Dec 14 1990 ic.i.salt-r--r--r-- 1 211 Dec 14 1990 ic.i.temp-r-xr--r-- 1 3418 Dec 14 1990 ic.run*-r--r--r-- 1 6634 Dec 14 1990 prep_utl.F-r--r--r-- 1 15068 Dec 14 1990 sbc.F-r-xr--r-- 1 33 Dec 14 1990 sbc.batch*-r--r--r-- 1 110 Dec 14 1990 sbc.i-r-xr--r-- 1 2302 Dec 14 1990 sbc.run*-r--r--r-- 1 18431 Dec 14 1990 topo.F-r-xr--r-- 1 34 Dec 14 1990 opo.batch*-r--r--r-- 1 194 Dec 14 1990 topo.i-r-xr--r-- 1 2011 Dec 14 1990 topo.run*-r-xr--r-- 1 39 Dec 14 1990 verify.batch*-r-xr--r-- 1 1400 Dec 14 1990 verify.data*


vi

PREFACE

The GFDL Modular Ocean Model isintended to be a flexible tool for exploringocean and coupled air-sea applicationsover a wide range of space and timescales. This model code has been writtenas a collaborative effort by RonPacanowski, Keith Dixon and AnthonyRosati at the National Oceanographic andAtmospheric Administration’s Geophys-ical Fluid Dynamics Laboratory inPrinceton, New Jersey. It is the successorto the code written by Michael Cox, whichhe documented in the GFDL Ocean GroupTechnical Report #1, (1984).As was the case for the Cox model, theGFDL Modular Ocean Model is distributedfreely to members of the oceanographiccommunity. However, we can not certifythat the code nor its documentation arefree of errors. Also, we can not provide in-depth support for users of the model.We recognize that many users will find thisfirst version of the GFDL Modular OceanModel Users Guide to be quite incomplete.We consider this Users Guide to be adynamic document, and one that we willrevise and expand. We also expect themodel itself to continue to be improvedand expanded. Our hope is that the modeland documentation will grow together. Ourgoal is to have this Users Guide evolve towhere it thoroughly describes the modeland its options, as well as provides userswith guidelines for its use.For now, version 1 of the Users Guide isintended to serve as an introduction to the

GFDL Modular Ocean Model version 1.0.Users unfamiliar with the earlier Coxmodel are referred to Cox’s GFDL OceanTech Report #1 and Kirk Bryan’s 1969Journal of Computational Physics articlefor a discussion of the numerics. As wasthe case for the Cox model and the modelof Bert Semtner (1974) that preceded it,the GFDL Modular Ocean Model is aFortran implementation of equationsdescribed by Bryan (1969).Comments and suggestions concerningthe GFDL Modular Ocean Model arewelcome. Anyone wishing to developrefinements and/or additions will beacknowledged for their efforts insubsequent releases of the code. Pleasecontact one of us if you have somethingyou feel might be worthwhile to add to themodel or if you uncover an error in theexisting code. We suggest that whendeveloping code, the style and modularityevident in the model be followed. Itsmodular design is specifically intended toaid in ease of use, flexibility, and continueddevelopment without sacrificing clarity.Ronald C. Pacanowski [email protected] W. Dixon [email protected] Rosati [email protected] 1991

REFERENCES:

Bryan, K., J. Computat. Phys., 4, 347-376,(1969).

Cox, M. D., “A Primitive Equation, 3-Dimen-sional Model of the Ocean”, GFDL OceanGroup Technical Report #1, (1984).

Semtner, A., UCLA Dept. of MeteorologyTech. Report No. 9, (1974)


vii

ACKNOWLEDGEMENTS

Various aspects of the GFDL ModularOcean Model can be associated with theefforts of many people over a number ofyears. Its lineage can be traced directlyback in time to the codes of Michael Cox,Bert Semtner, and ultimately to theequations described by Kirk Bryan in1969.Numerous others have added usefulrefinements to the GFDL Modular OceanModel’s predecessors, strengthening itsheritage. We have endeavored to notecontributions by citing names andreferences in this document and in themodel code itself. No doubt some havebeen omitted through our own oversight orsimply because the collective memory ofGFDL ocean modeling can grow fuzzyover time. We apologize to those whosenames have been omitted.Of more recent importance has been thesupport we have received that afforded usthe opportunity and freedom needed todevote time to writing the GFDL ModularOcean Model. This continued support hascome from Jerry Mahlman (Director ofGFDL), Kirk Bryan, (GFDL OceanCirculation Project Leader), KikuroMiyakoda (GFDL Experimental PredictionProject Leader), and other GFDLcolleagues.We also wish to thank Christopher Kerr ofCray Research for his help in getting themodel to autotask and multitask correctly.

This page was intentionally left blank


viii


1-1

Once the GFDL Modular Ocean Model(MOM) files have been obtained, onenaturally asks a question along the linesof... “I’ve got all these files, so now whatdo I do with them?”.This chapter is intended to begin toanswer the above question by giving thenew user an overview of the contents ofthe files and suggestions regarding how towork with them.NOTE: The following descriptions assumethat the files have been arranged in UNIXdirectories as described earlier in thisdocument, in the section entitled“AVAILABILITY OF THE G.F.D.L.MODULAR OCEAN MODEL CODE”.

CHAPTER 1: Beginning to Work with theGFDL Modular Ocean Model


1-2

(1.1) BRIEF FILE DESCRIPTIONS(a) mom_x ... the MOM source code in“mergelibed” form (see the entry formergelib (g) below). The “x” suffixrepresents the version number. i.e.:mom_1.0)

(b) MOM_NEWS ... this file contains an upto date list of bugs and problemsassociated with the MOM code, along withrecommended fixes, and other relevantinformation. This file should be consultedbefore a user constructs an ocean modelfor his/her own purposes. The bug fixeswill not appear in the current mom_x codefile. Bug fixes will of course be included inlater released versions of the GFDL MOMcode.

(c) printout ... output from a 3x4 degreeidealized world ocean test case executedon a Cray YMP. Users should comparethis file to results of the test case run ontheir own machines. For a description ofthe sample test case provided with theGFDL M.O.M. code, see section (1.7) atthe end of this chapter

(d) ocean.in ... the input file containing“namelist” input needed to run the testcase.

(e) cray.run ... a sample run deck forrunning the test case on a Cray YMP.While set up for the Cray YMP, this fileshould be consulted for running the testcase on other computers as well

(f) READ_ME ... an ASCII text filecontaining much of the same informationcovered in this document.

(g) mergelib & splitlib.c (two utilities)mergelib is a UNIX script that can be usedto concatenate all code files with suffixesF, f, or h (abbreviated as *.[Ffh] files) intoone big file.USAGE:mergelib [-h] [-d indir]

[-o ofile] [file...]-h ; print this summary-d ; indir ; input directory,default = current directory

-o ; ofile ; output file,default is standard output

file... ; files to merge,default = *.[hfFc]

splitlib does the inverse ofmergelib... breaking themergelibed file back into itscomponent files (*.[Ffh] files).splitlib is written in the Cprogramming language and needsto be compiled.

USAGE:splitlib [-h] [-d outdir]

[file]-h ; print this summary-d outdir ; outdir is thedirectory that will containthe source files thatsplitlib will extract. The

default is the currentworking directory.

file ;if no file name isgiven standard input isassumed

(NOTE: The UNIX tar command canperform essentially the same functions as


1-3

splitlib and mergelib, but we providethese utilities, since there may be tarmachine dependencies.)

(h) upgrade ... a sample script for sourcemanagement on SUN workstations (forupgrading your changes to future versionsof the MOM code)

(i) PREP_DATA ... is a directory whichcontains a collection of files (routines andrundecks) for interfacing Scripps 1 degreetopography (Smith, Menard and Sharman,[1966]), Hellerman and Rosenstein (1983)wind stress, Oort (198?) air temperature,and Levitus (1982) temperature andsalinity climatologies directly to the MOMcode. Check the READ_ME file included inthe PREP_DATA directory for details.Refer to Chapter 5 for more detaileddescriptions.

REFERENCES:

Hellerman, S., and M. Rosenstein, J. Phys.Oceanogr., 13, 1093-1104, (1983).

Levitus, S.,Climatological Atlas of the WorldOcean: NOAA Prof. Paper 13, US Govt.Printing Office, Washington DC, (1982)

Oort, A., Global Atmospheric CirculationStatistics 1958-1973: NOAA Prof. Paper14, US Govt. Printing Office, WashingtonDC, (198?)

Smith, S., W. Menard, and G. Sharman,World-wide ocean depths and continentalelevations averaged for areasapproximating one-degree squares oflatitude and longitude, Ref. 65-80, 14 pp.,Scripps Instit. of Oceanogr., (1966).

(1.2) REQUIREMENTS FORRUNNING THE M.O.M. CODEIn addition to requiring a Fortran compiler,the C-language preprocessor (cpp) isrequired. Any computer that runs the C-language should have cpp, and manyFortran compilers have cpp functionalitybuilt into them.The cpp is used stand alone or bycompilers to allow sections of the GFDLModular Ocean Model Fortran code to beselectively turned on or off for compilationvia ifdef directives. Many compilersincorporate cpp into their processing, evenif not explicitly stated. We use this featureof cpp to control which options and/ormodules are used in any specificconfiguration of the MOM. code. UNIXmanual pages for the cpp command canhelp users become familiar with ifdefs.


1-4

(1.3) EXTRACTING THEINDIVIDUAL FORTRAN SOURCEFILESWhen beginning to work with the GFDLModular Ocean Model, first create a newwork directory and copy the original filesinto the work directory.This is a good place to be reminded toalways save your originals. They areneeded when upgrading your changes tonewer versions of the MOM code. Westrongly recommend that users changethe file permissions in the directorycontaining the original MOM files to “readonly” (chmod 444 *).Now, in the work directory, break mom_xinto its component files by using thesplitlib utility.

cc splitlib.c -o splitlib(compile the C-languageprogram)

splitlib mom_x (to run it)

Now there should be lots of files. They areclassified by their filename suffix:

(a) *.F and *.f files: ....these are the Fortransource files (subroutines, main programs,and functions)

(b) *.h files:....these are the “include” files (groups ofparameter statements, common blocks,etc., that are inserted into a *.F file (orother files) via #include directives)

To merge the individual code files backinto “mergelibed” form, do the following:mergelib > onebigfile

Note that all the *.[Ffh] files are stillleft in the directory and there is a new onenamed onebigfile.NOTE: be careful that there are no othernon-MOM .F, .f or .h files in thedirectory, or they will be incorporated intothe file onebigfile.


1-5

(1.4) GETTING ACQUAINTEDWITH THE M.O.M. CODEWe recommend that after reading thisdocumentation, you begin your explorationof the GFDL Modular Ocean Model byscanning the “.h” files. A convenient wayto do this (under UNIX) is to use: cat *.h > allh

This concatenates all the “.h” files togetherinto the file named allh.One can then print the allh file andbrowse through its contents. All variablescontained in common blocks should bedescribed in the comments (if we missedsome, please let us know). By readingthrough the “.h” “Include” files, one canbecome acquainted with the variablenames used in the model, and with someof the model organization.Next, take a look at the model results fromthe test case stored in file printout.Locate the section in the printout filewhich lists the model options available tothe user. This list is produced by thesubroutine found in file docmnt.F. One ofthe ways in which the GFDL MOM codemay be customized for a particularapplication is by selecting variouscombinations of these options. Theoptions are described in more detail inchapter 3 of this Users Guide.

(1.5) RECOMMENDATIONS FORM.O.M. CODE MAINTENANCEIt is strongly recommended that sourcemanagement be done on a UNIX basedcomputer (workstation or mainframe).While we do source code managementand version control using the UNIX SourceCode Control System (SCCS), the methodof source management is left to theindividual. We’ve included our ownupgrade utility based on SCCS, as anexample of one way to do sourcemanagement on a SUN workstation:

AN EXAMPLE USING THE “upgrade”UTILITY:

Suppose you have been working with aversion of the GFDL Modular OceanModel (let’s call it MOM.old) and haveadded local changes to makeMOM.yours. When the next version ofMOM (MOM.new) is released, you wouldlike to take advantage of its new features.How do you get MOM.yours upgradedfrom MOM.old to MOM.new?If your modifications are mostly sections ofcode grouped together (as opposed tobeing strewn everywhere) you mightsimply “cut & paste” them into MOM.new.More extensive changes can be handledmore or less in an automated way. Anexample using the upgrade script follows:First, create a new working directory andcopy four files into it: the mergelibed formsof MOM.old, MOM.yours and MOM.newalong with the upgrade script itself.Next, to run the script type upgrade andat the prompts input MOM.old,


1-6

MOM.yours and MOM.new. The script willproduce a file named new_source whichis MOM.yours upgraded to MOM.new.The upgrade script will also likely pointout places where possible problems andconflicts occurred by showing the linenumbers in new_source.After investigating and fixing all problemsand conflicts (start from the highest linenumbers (bottom) and work backwards inyour favorite editor), you should thencompare the new_source with MOM.new,as in:diff MOM.new new_source > my.chgs

Inspect the file my.chgs to make sure thatyour modifications went into new_sourceproperly.NOTE: If code is altered in MOM.yoursand the same code is altered in MOM.newbut NOT in MOM.old, both the alteredcode from MOM.yours and MOM.new willappear in new_source.Once one is satisfied that the upgrade hasbeen completed and checked, thecomponent *.[Ffh] files may begenerated by using:splitlib new_source

(1.6) NOTES ON THE M.O.M.TEST CASE:The GFDL Modular Ocean Model codeobtained from GFDL’s anonymous ftpserver is fully functional and ready to run asample test case. We recommend thatusers attempt to run the test case on theirmachine of choice after they becomefamiliar with the code.To run the sample test case, one needs toselect the set of ifdef code options listedin the sample cray.run file, and thencompile the code. When successfullycompiled, one should use the sampleocean.in file provided, and make a shorttest run that will produce output thatshould be compared with the sampleprintout file.The test case uses an idealizedconfiguration that permits the modelresolution to be changed to fit on variouscomputers without having to worry aboutland/sea geometry, model bathymetry,boundary conditions or initial conditions.All of the above automatically adaptthemselves to changes in resolution.However, it should be noted that theintended purpose of the test case is to testthe model code. The test case is notmeant to be a scientifically meaningfulconfiguration of the GFDL M.O.M. code.The test case’s land/sea geometryresembles the gross features of presentday continental outlines. The modelbathymetry, however, is unrealistic (thereis a ridge in the Pacific, but mosteverywhere else there is a flat bottom).For the default sample provided, a 3x4


1-7

degree horizontal resolution wasspecified.Initial conditions for the test case call forpotential temperatures to vary with depthand latitude in a manner roughlyconsistent with observations. Salinity isinitially constant everywhere.Surface boundary conditions are heldconstant in time, in the test case. Theyvary with latitude, but not longitude. Thesurface momentum flux is given by the xand y components of the zonally averagedannual mean wind stresses (Hellermanand Rosenstein, (1983). Surface salinitiesand temperatures are damped backtowards zonally averaged, annual meanvalues calculated from Levitus (1982).When comparing results with the sampleprintout file, we suggest that users firstcheck that the land/sea geometries match.A quick look at the global surface area andvolume statistics will reveal if there is goodagreement. If this comparison revealsdifferences larger than those one mightexpect from roundoff errors, a check of themodel bathymetry is called for in order todetermine the location(s) where thesample Cray version differs from theuser’s run. The scheme use to determineland/sea boundaries can be sensitive tothe precision or the way in which numberare represented, causing variations in themodel’s continental outlines to arise ondifferent machines.

REFERENCES:

Hellerman, S., and M. Rosenstein, J. Phys.Oceanogr., 13, 1093-1104, (1983).

Levitus, S.,Climatological Atlas of the WorldOcean: NOAA Prof. Paper 13, US Govt.Printing Office, Washington DC, (1982)


1-8



2-1

As mentioned in the Preface, one of theprimary goals in constructing the GFDLModular Ocean Model (MOM) was toproduce a model that was easily toconfigure and work with, while at the sametime being flexible enough to be aresearch tool for various scientificproblems. Increased flexibility and clarityhave been achieved through modularity.This chapter describes the ways in whichthe modular design and structure of theGFDL MOM code works toward this goal.Since the GFDL MOM code is notintended to be a static product, wewelcome user input as to how it mayevolve to better achieve our goal of havinga truly flexible and easy to use researchtool.

CHAPTER 2: Coding Design of the GFDLModular Ocean Model


2-2

(2.1) GENERAL CODING DESIGNUsers of previous versions of GFDL’socean model codes will notice that thereare many more subroutines in the MOMcode. The modular design is intended toaid in the logical organization of the codewith “hooks” readily available for addingnew options to the model.Deciding which modules to use is done atthe pre-processing level. Modules do notinteract with each other, but interact withthe main code in only a few places througha short argument list and/or the includefiles. This approach to interfacing tends tolocalize code modifications, therebykeeping the code structure simple, under-standable, and easily supplemented.Again, we strongly suggest that thisapproach, some of which is outlinedbelow, be followed when users make theirown code modifications.

The GFDL Modular Ocean Model code iswritten in standard Fortran-77, except forthe use of the common extension ofnamelist.

Variable organization: variables have beenorganized and grouped in terms ofphysical (or where appropriate logical)significance. This allows for bettermodularity. Look at the *.h files. Note thatall variables are commented. To find outwhere variable yyy is used, use the UNIXcommand:grep -i -n yyy *.[Ffh]

which searches all files with .F, .f, or.h suffixes for the string yyy. The use ofgrep is invaluable in tracing variables andoptions and its use is stronglyencouraged. Note also that the includefiles may be nested (see the bottom of fileparam.h for example).When modifying the code, one shouldconsider whether it is appropriate to addnew variables to existing common blocksand “.h” files, or if a new “.h” file andcommon block are called for to enhancemodularity. When in doubt, try to minimizethe interactions between various parts ofthe code.

No out-of-bounds references are found inthe MOM code. This can simplifydebugging new code by turning on acompiler’s bounds checking option. If, onexecution, an array subscript goes out ofbounds, the compiler’s bounds checkerwill let you know where and when thisimportant design feature has beenviolated.

Statement functions are used to definephysical operators. This allows one towrite code that more closely resemblesthe equations. Another advantage is thatcomplicated code can be made moreunderstandable. At the same time, moreoperations per loop often allows acompiler to generate more efficient code.On computers like the Cray YMP, this canlead to significantly more parallelism. Lookin file tracer.F and clinic.F to seehow we have used them. We have alsoincluded an ifdef option namedkeepterms, that allows the use of arrays


2-3

(instead of statement functions) to retainterms in momentum and tracer equations.This can be useful when using the termsoften in more than one place. The price forthis is increased memory (but lesscomputation).

Naming conventions have also beenadopted, and appear as comments in themodel. Conventions exist for namingnumerical constants, grid variables,reciprocals, etc. (see file pconst.h). Byadhering to these rules, one can moreeasily analyze unfamiliar sections of thecode.

The slab variables have been defined asfour dimensional variables, utilizing theconcept of a memory slab window. Look atfile slabs.h for details. This increasesthe generality and flexibility of the codeand simplifies management of the dataflow from disk through memory. In additionto longitude and depth indicators, thedimensions on the slabs include time leveland j-row location “pointers”. So instead ofshuffling data into the “correct slabpositions”, the MOM code simply updatespointers to existing data

The MOM code has been written with theunderlying assumption that memory sizehas been increasing for the types ofcomputers that the code is typically run on.So, there are places where we trade-offextra memory usage in order to achieveclarity, modularity, and to minimize theprobability of introducing errors whenmodifying the code in the future.

(2.2) SOME NEW FEATURES &NOTES FOR USERS OF THE COXCODEAs noted before, the GFDL ModularOcean Model code is intended to be muchmore modular in design than the Cox(1984) model and other earlier codes.Increased modularity and an increasednumber of code options result in therebeing many more subroutines and includefiles in the MOM code. The simplified flowchart provided in this document (seesection 2.3) should help users familiar withthe Cox code to follow the flow of theMOM code. In this section, we point outsome of the more substantial differencesusers will encounter when migrating fromthe Cox code to the GFDL MOM code.

Customizing via ifdef directives: Refer to thesample printout file and chapter 3 ofthis Users Guide for the complete list ofcurrently allowable ifdef optionsavailable for customization of the model.Note that there is a skipland option fordoing calculations only over ocean points(as opposed to everywhere), options forchoosing various lateral and verticalmixing parameterizations (explicit orimplicitly solved), plus a selection ofPoisson solvers including a very accurate9 point conjugate gradient scheme whichallows for direct reconstruction of the rigidlid surface pressure.Various MOM code options are controlledby ifdef options, rather than Cox’sUPDOC program. It is recommended thatusers put ifdef directives around


2-4

personal code modules and that users tryto keep modules from interacting, as in thebase MOM code. Some UNIX tools forsource management include: diff,sdiff, diff3, and SCCS. UNIXmanual pages for the cpp command canhelp users become familiar with ifdefs.

No boundary conditions in the slabs: Notealso that the slabs are defined in a waysuch that no boundary conditions havebeen added onto the slabs as in Cox’scode. The vertical boundary conditionscan be found in their own common block infile cvbc.h (common of vertical boundaryconditions). By increasing the number ofdimensions on the slabs to include timelevel and j-row location “pointers”, thenumber of slab variable names and doloops that switch time and j-row positionshave been reduced.

New and redefined variables: New variableshave been added and some old onesredefined (see file coord.h forexamples). There are also “source terms”for the momentum and tracer equations(variables sourcu, sourcv and sourct)which allow easy inclusion of sources andsinks (i.e.: biology, imposing baroclinicflows, etc.).

Consistency checks are carried out forifdef options along with various otherconditions by code found in file checks.F.Notice the “warning” and “error”messages. that are generated bychecks.F. Warnings allow the model toprocede, but error messages terminateexecution.

Island calculations are done by line (ratherthan area) integrals and the islanddefinition has been simplified by theautomatic calculation of island perimeters(see file iperim.F) requiring only thatone point within the island be specified.This allows for easy use of islands withincomplex geometries.

A time manager (see file tmngr.F andctmngr.h) controls all time dependentdecisions within the model. Timedependent information such as length ofintegration, time between energydiagnostics, etc., are entered throughnamelist in units of days (only variablenmix, the number of timesteps betweenmixing timesteps, is not in units of days).This information is used by the timemanager to decide how logical variables infile switch.h are to be set each timestep. These logical variables control timedependent data flow.Included in tmngr.F are utilities for aidingin interpolating time dependent boundaryconditions (i.e.: monthly wind stress, etc.)to the time step. However, these are notused in the test case.

Finding the nearest model grid point:Another utility is function indp, found infile setgrid.F. This function is used inmany places and provides users with aneasy way to map real world coordinates(latitudes, longitudes and depths) to thenearest model grid point. This allows allspatial information to be independent ofchanges in model resolution.


2-5

Minimize equivalence statements: In additionto outlawing out-of bounds references inorder to improve the code’s clarity, anattempt has been made to minimizeequivalence statements. Equivalencestatements should be added with caution.

Regional averages and term balances: Theability to compute volume weighted traceraverages, surface fluxes, and termbalances for the momentum and tracerequations has been added to facilitateanalysis and debugging. Thesecomputations can be done over arbitraryvolumes in arbitrary locations. The volumesizes and locations can be specified usinghorizontal and vertical masks (see filesocean.F and cregin.h and optionreadrmsk).

Writing data for off-line analysis: The GFDLModular Ocean Model code provides theuser with a ”snapshot” feature to writeinstantaneous data for offl-ine analysis.(see variable snaps in file switch.h and fileiounit.h for details). Many users willwant to modify or supplement this featurefor their own local purposes.

Bottom drag in the MOM code is controlledby the linear drag coefficient variablecdbot. There is no bottom drag whencdbot = 0.0 (see files scalar.h andblkdta.F). As in the Cox model, lateralboundaries are no slip for momentum andno flux across boundaries for tracers.

Run-time documentation: The MOM codeuses file docmnt.F to arrange data to be

written out as a form of documentationsummarizing the model characteristicsthat uniquely define a given model run. Italso lists the requested ifdef options.The user will need to review this routinewhen configuring a model run or addingnew options to the code. This feature canbe beneficial to those setting up newmodel runs, comparing different modelruns, and for analysis purposes.

REFERENCE:



2-6

The firstcalling tree represents the EQSTAT program found in file denscoef.F, whichgenerates a dncoef.h file.EQSTAT (main program for deriving coefficients of polynomial approximations for the equation of state)

{KNUEKM} or {UNESCO} (routines returning density as a function of T, S and P)POTEM (returns potential temperature for in situ temperature, salinity and pressure){KNUEKM} or {UNESCO} (routines returning density as a function of T, S and P)LSQSL2 (iteratively computes a least squares fit)

The stand alone depths program generates a thick.h file of model layer thicknesses.DEPTHS

The stand alone size program determines memory and disk requirements for variousconfigurations of the GFDL MOM code.SIZE

The following flowchart is for the GFDL MOM code’s main ocean program.

(2.3) SIMPLIFIED CALLING TREEDIAGRAMS FOR THE GFDLM.O.M. CODEThese calling tree diagrams are intendedto help illustrate how the coding designdiscussed previously has beenimplemented in the GFDL Modular OceanModel code.However, for clarity, not all calls arerepresented in the calling tree for the mainocean model program shown on the nextpage. The calls that have been omittedtend to appear often in the code (a sign oftheir modularity).

Specifically, calls which printout informa-tion using the code in file matrix.Fhave been omitted. Those involving the i/o routines odam.F and getvar.F havebeen omitted also. So too havereferences to function indp, a utilityfound in file setgrd.F that is used tofind the nearest grid point to a specifiedlocation.Calls which are conditional upon theifdef options a user has definedappear within curly brackets {}.


2-7

OCEAN (main program)BLKDTA (initialize variables)GRIDS (set up the grids: in file setgrd.F)

MESH (utility for setting up grids: in file setgrd.F)CMESH (utility for setting up grids: in file setgrd.F)

DOCMNT (documents the run)OCN1ST (set up initial conditions on the first timestep)

TOPOG (set up idealized world topography)SETKMP (set the number of levels for the t,s grid)

RDREST (read the restart file if it is not the first timestep: in file restio.F){REG1ST} (set up for regional averages of tracers){IPERIM} (calculate island perimeters){FINDEX} (calculate indices for filtering)CHECKS (test for consistency of configuration)TMNGR (time manager called once per timestep)

TMNSET (function to set time dependent logical switches: in file tmngr.F)STEP (cycle latitude rows between disk & memory slab window)

{DELSQ} (calculate quantities for biharmonic mixing)SETVBC (set vertical boundary conditions)

BCEST (get surface boundary conditions for test case){PPMIX} (Pacanowski/Philander richardson mixing)

STATEC (density calculation for vertical stability){CNVMIX} (set coefficients for constant vertical mixing)

{STATEC} (densities for vertical stability if implicitvmix: in file state.F){TCMIX} (Mellor-Yamada turbulence closure mixing)

STATEC (density calculation for vertical stability: in file state.F){IMPLQ} (for implicit vetical mixing)

{ISOP0} (Isopycnal mixing set up: in file isopyc.F)STATEC (called 3 times for densities for isopycnal slopes: in file state.F)

{CFL} (for cfl monitoring)CLINIC (calculate internal mode velocities)

STATE (density calculations for pressure gradients){NLMIX} (nonlinear Smagorinsky mixing){INVTRI} (for implicit vertical diffusion){FILUV} (filtering u & v at high latitudes)

{FILTR} (if fourfil option was selected){FILFIR} (if firfil option was selected)

TRACER (calculate tracers){ISOP1} (finish isopycnal mixing: in file isopyc.F){INVTRI} (for implicit vertical diffusion)STATEC (density calculation for vertical stability: in file state.F){FILT} (filtering tracers at high latitudes)

{FILTR} (if fourfil option was selected){FILFIR} (if firfil option was selected)

REGION (periodically does basin averages of tracers)DIAG (periodic energy diagnostic and term balance calculations)

VORT (calculates vorticity){FILZ} (filtering vorticity at high latitudes)

{FILTR} (if fourfil option was selected{FILFIR} (if firfil option was selected)

{RELAX}, {HYPER},{CONGR5},{CONGR9} (Poisson solvers for external mode)DIAG2 (periodic energy diagnostic & term balance printouts)

DOCMNT (document the run)RDREST (write the restart file )



2-8


3-1

As discussed earlier, a very powerfulfeature of the GFDL Modular OceanModel (MOM) is the ability to defineifdef options so as to configure themodel in a manner suitable for theproblem a user wishes to investigate. Wealso feel that the use of cpp (the C-language preprocesor) and ifdefdirectives helped us write a model inwhich numerous code options could co-exist and still leave the code readable andreceptive for future development.This chapter contains brief descriptions ofthe various MOM code customizationoptions that a user may turn on or offthrough the use of cpp and ifdefdirectives. It is suggested that the userexamine at the code itself and read thereferences given, in order to develop afuller understanding of the options.These descriptions introduce users of theGFDL MOM code to what it has to offernow. We will also briefly mention sometentative plans for future work. We inviteusers to sharie with us their ideas andcode modules so that together the oceanmodeling community can enhance thiswidely distributed research tool.

CHAPTER 3: Code Options in the GFDLModular Ocean Model


3-2

(3.1) OVERVIEW OF CODEOPTIONSThe options available in the MOM codeare quite extensive and range fromvarious subgrid mixing parameterizationsto computing performance enhancements.The current implementation is such that ifthere are more than two options within acategory of code options, there is nodefault value, so care should be taken todefine the desired option.Within the MOM code, subroutine checks(see the checks.F file) searches theselected options to determine if the userhas asked for conflicting options.For example, if a user in selecting an I/Oscheme turned on both diskless (fullycontained within memory) and fio (Fortrandirect access), an obvious inconsistencyexists and a message to this effect will beprinted and execution of the model willhalt.Many Fortran compilers allow ifdefoptions to be enabled (defined) by usingthe “-D” option on the compile statement(as shown in the next chapter). This formof using a “-D” option may also be usedwith the cpp command (see filecray.run) to preprocess the code beforesending separately it to the compiler.

(3.2) GRID OPTIONScyclic is an option for setting cyclicboundary conditons in a zonal direction. Ifenabled, anything exiting the eastern sideof the MOM model grid comes back inthrough the western side. If not enabled,then solid walls are assumed on theeastern and western boundaries of themodel.

symmetry is an option for setting asymmetric condition across the northernboundary of the model. The condition is :Tracers at row jmt on the “t” grid = tracersat row jmt-1 on the “t” grid. zonal velocityat row jmt-1 on the “u” grid = zonal velocityat row jmt-2 on the “u” grid. meridionalvelocity at row jmt-1 on the “u” grid is zero.meridional velocity at row jmt on the “u”grid is the negative of the meridionalvelocity at row jmt-2 on the “u” grid. If notenabled, solid walls exist at the northernand southern boundaries.


3-3

(3.3) EXTERNAL MODE OPTIONSBasically, there are four Poisson solversprovided in the GFDL MOM code. All ofthem require the rigidlid option to also beenabled.

oldrelax duplicates the meothod used inMike Cox’s (1984) ocean model basecode. If you use it, you should twiddle withvariable sor (the overrelaxation constant)to minimize the number of scans for yourgeometry. Variable crit (which along withsor can be specified through thenamelist) controls the accuracy. See filerelax.F for more details.

congrad5pt is an option which specifiesthe use of a conjugate gradient technique.Variable sor is not needed here, but theconvergence criterion crit, which whensatisfied halts the iterative scheme, isused. This scheme is generally faster (interms of cpu time) than oldrelax, but maybe a bit less stable in the presence ofsteep topographic gradients. Withcongrad5pt defined, the surfacepressure is calculated only as a basis forcomparison with the congrad9pt option.See file congr5.F for more details.

congrad9pt is another conjugategradient solver that uses a 9 pointlaplacian instead of the 5 point laplaciansused by the other schemes. Its advantageis that it is the most accurate and thusallows the surface pressure to becalculated directly. Reducing crit when

using congrad9pt will give better results(as shown by the closed line surfacepressure integrals in the test case) thancongrad5pt (this is because there issignificant truncation due to the 5 pointnumerics). In fact, the accuracy of thecongrad9pt scheme’s streamfunctionsolution is limited only by the variablecrit and computer precision.The 9 pointscheme’s down sides are that it may havestability problems for topographies withsharp gradients and it will take more cputime. How much you ask? It dependsgreatly on your configuration. Try it andsee. See file congr9.F for more details.hypergrid is a variation of oldrelaxsolved using a checkerboard technique:first on the black squares, then on the redones: Numerically it is very similar tooldrelax (sor and crit commentsapply), however it does run somewhatfaster than oldrelax on some types ofcomputer architectures.

REFERENCES:



3-4

(3.4) LATERAL MIXING SCHEMESLateral mixing applies to both momentumand tracers. One (and only one) schememust be enabled.

consthmix enables a linear second ordermixing scheme with constant eddycoefficients: variable am is the eddycoefficient for momentum while variableah is for heat and the other tracers. Oneusually attempts to choose these values tobe just large enough to suppress smallscale numerical noise. A typical value forvariable am in a one degree resolutionconfiguration of the MOM code might be1.0 x 107 cm2 sec-1. Usually variable ah ischosen to be smaller so as not to diffuseaway frontal features.

biharmonic selects a linear fourth ordermixing with constant coefficients as inconsthmix: am for momentum and ah forheat, salinity, and other tracers if present.The biharmonic scheme is more scaleselective than the consthmix option (ie:more severly damps the small scalefeatures). For a 1/3 degree resolutionversion of the MOM code, variable amshould be roughly 1019 to 1020 cm4 sec-1

(the minus sign is in the equations andthus does not appear in the eddycoefficients).

nlhmix is the non-linear horizontalsubgrid-scale mixingoption, based uponSmagorinsky’s non-linear viscocityscheme as implemented and described in

Rosati and Miyakoda (1988). In thisformulation, the horizontal eddy diffusioncoefficient is proportional to the horizontalgrid length and to the local deformationfield. The eddy coefficients computed bythe scheme are sensitive to the spatialscales of motion. Accordingly, mixingcoefficients are relatively small in the openocean, where the scales of motion arecomparatively large, and are relativelylarge in regions where the scales ofmotion are comparatively small.

REFERENCES:

Rosati, A., and K. Miyakoda, J. Phys.Oceanogr., 18, ??-??, (1988)..


3-5

(3.5) VERTICAL MIXINGSCHEMESVertical mixing applies to both momentumand tracers. One (and only one) schememust be enabled. The default is for explicitsolution. Choosing the implicitvmixoption will cause the vertical mixingequations to be solved implicitly. NOTE: inthe GFDL MOM code, explicitfor thesolution unstable density profiles lead totracers being mixed via convectiveadjustment but NOT momenum. But bothmomentum and the tracers are verticallymixed when the implicitvmix option hasbeen defined by the user.

implicitvmix solves the vertical diffusionterm implicitly for all vertical mixingschemes. This allows for large mixingcoefficients without the need to reduce thetimestep. With this option enabled,convective adjustment is not done in filetracer.F and the unstable densityprofile is mixed using large mixingcoefficient limits (see variables vvclimand vdclim). For the constvmix andppvmix options, these limits are set in theblkdta.F file and for option tcvmix thecoefficients are computed. Selecting theimplicitvmix option also requires thatvaraible aidif (the implicit factor) be set(see files cvmix.h, and blkdta.F andthe namelist inputs).0.5 < aidif < 1 ; always stable0 < aidif < 0.5 ; stable if ...

gdtdz( ) 2×

12 4 aidif×<( )

<

where g is the maximum vertical mixingcoefficient.

constvmix utilizes constant eddycoefficients (variables fkpm and fkph formomentum and heat, respectively) forlinear second order mixing of momentumand tracers in the vertical. In the explicitcase (when implicitvmix has not beendefined), convective adjustment handlesregions of gravitational instability for thetracers. If implicitvmix is defined, thenthe convective section is bypassed andthe large mixing coefficient values (limitsset in variables vdclim and vvclim)are used to set the vertical mixingcoefficients to handle the instability forboth the tracers and momentum.

ppvmix is a vertical mixing option basedon the Pacanowski and Philander (1981)Richardson mixing scheme. Thisparameterization was designed primarilyfor equatorial models with verticalresolution of about 10 meters betweenlevels in the upper ocean (we were mostinterested in the structure of theundercurrent). Previous versions of thiscode assumed the “t,s” grid wascoincident with the “u,v” grid and gavegood results. In the present case, we haverelaxed this assumption. We tried thisnewer, more consistent version a fewways and got lots of numerical noise, atfirst (particularly on the equator off Brazil).The present configuration minimizesnoise. If noise develops off steep shelfbreaks it can sometimes be suppressedby turning on a bottom drag (see variablecdbot). The background mixing variablebvdc should be kept much less than 1.0 to


3-6

avoid diffusing the thermocline away onlong time scales.In regions of gravitational instability,ppvmix uses the vertical mixing limitvariables vdclim and vvclim to set thevertical eddy coefficients. In the explicitcase they must satisfy the “CFL” criterionand may be too small to remove theinstability. However, the convectiveadjustment is operative and tries toremove the instability. Whether it does ornot depends on the ncon variable (setablethrough namelist), which controlls thenumber of passes through the convectiveadjustment section of code in filetracer.F. If implicitvmix is used,vdclim and vvclim are set large andthe convective adjustment section isbypassed. The ppvmix scheme alsoassumes the use of equal timesteps onthe density and momentum equations. If alonger time step is desired, try using theimplicitvmix option.

tcvmix invokes the Mellor-Yamada level2.5 turbulence closure scheme. Here themixing coefficients are a function of theturbulence length scale (l), the turbulentkinetic energy (q2), the analytically derivedstability factors (Sm & Sh), and theboundary conditions. There are twooptions within tcvmix to compute thelength scale:

leq solves an additional equation for q2

from which the length scale may bederived at each level.

lalg, obtains the length scale from analgebraic relationship. The lalg optionmay be sufficient for the boundary layerbut not in cases where there would be

multiple turbulent regimes. The horizontaldiffusion coefficient of turbulent kineticenergy is set using the variable aq (seefiles ctcmix.h, blkdta.F and the namelistinput). For this scheme to be effective theimplicitvmix option should be enabledand also the vertical resolution should besufficient.

REFERENCES:

Pacanowski, R. and S.G. Philander, J. Phys.Oceaonogr., 11, ??-??, (1981)

Mellor, G. and Yamata ?????


3-7

(3.6) HYBRID MIXING SCHEMEOPTIONSWe are using the term “hybrid” here todescribe schemes which apply to eithermomentum or tracers but not both. For allcases in which implicitvmix is notenabled, convective adjustment is thedefault. Recall that when solved explicitlyconvective adjustment mixes tracersvertically in regions of gravitationalinstability, but not momentum. Theeffectiveness of convective adjustment iscontrolled by variable ncon which is thenumber of passes through the convectivesection (use grep -i -n ncon *.[Ffh] to seeits usage). If impllicitvmix is enabled, theconvective adjustment section isbypassed. in file tracer.F. Theassumption is that the vertical mixingschemes will handle it.isopycmix is a scheme in which a mixingtensor is computed from the localisopycnal slope and the diffusion of tracersis then conducted along that direction. Ittherefore influences both the lateral andvertical mixing of tracers in the model. Itsuse is intended to partially mitigate aperceived shortcoming of z-coordinateprimitive equation ocean models inparameterizing oceanic mixing due tomesoscale motions. Since such mixing isbelieved to take place along isopycnalsurfaces, the isopycmix option seeks toorient the mixing of tracers alongisopycnal surfaces rather than purelyhorizontal surfaces, as described by Redi(1982).As described by Cox (1987), in addition tospecifying the mixing coefficient for along

isopycnal diffusion (variable ahisop), anon-zero background horizontal mixingcoefficient (here we use variable ah) isoften needed to supress gridpoint noise.Additionally, a constraint is placed on thesteepest isopycnal slope (see varableslmxr) that the scheme will considerwhen computing the components of themixing tensor. A typical value of slmxr is100.0, which translates into a slope of1:100. Steeper slopes would then beconsidered to be 1:100 for the purpose ofcomputing the mixing tensor.A vertical mixing scheme must bespecified for use with isopycmix. The zzcomponent of the isopycnal mixing isadded to the vertical diffusion coefficientproduced by the vertical mixing scheme.Since isopycmix only affects tracers, onemust also specify a lateral mixing schemefrom the above list to be used formomentum. This additional lateral mixingscheme has no effect on tracer diffusion.

REFERENCES:

Cox, M., Ocean Modelling, issue 74, 1-5,(1987)

Redi, M., J. Phys. Oceanogr., 12, 1154-1158,(1982).


3-8

(3.7) OPTIMIZATION OPTIONSskipland is an option that allowscalculations in the GFDL Modular OceanModel to be done over blocks of oceanwhile skipping land points. The trade offhere is the time saved by not doing thecalculation over land versus the time usedin starting and stopping a lot of vectors.Whether or not cp time will be saved byusing this option is largely a function ofland mass geometry and grid resolution. Aglobal ocean would be a good candidatefor this option, but a idealized basin wouldnot.

timing is an option that cp and wall clocktimes for running on a Cray YMP.Additionally, the time per grid point pertime step is calculated.

multitasking is an option that wasdeveloped with the Cray YMP in mind. Itallows the slabs to be divided into a seriesof tasks. Each task contains approximatelythe same number of slabs and thereshould ideally be one task per processor.This is all handled by simply specifyingvariable ntasks to be the number ofprocessors (we’ve defaulted it to 8 in fileblkdta.F).The idea behind multitasking is that sinceall tasks are independent, all tasks can beworked on simultaneously. This reducesthe total wall clock time for the job butNOT the total cpu time. Note: thentasks variable may be set > 1 evenwhen unitasking (running on oneprocessor), but this is not recommended,

since there is extra work being done at theinterfaces between tasks. Also, ondiagnostic, tracer averaging, and datasaving time steps, the MOM code willrevert to one task of jmt-2 slabs, soeffectively multitasking is shut down forthese time steps.The multitasking option currently doesnot work with the diskless option (seebelow) because only two time levels areused for slabs on simulated disk(ntlev=2) to save memory and this isincompatible with ntasks > 1. Thiscondition may, in principle, be removed bysetting parameter ntlev = 3 whenenabling the diskless option. We haven’texplored it yet.To date, we have confirmed that allcombinations of unitasked, multitasked,and autotasked runs give the sameanswers down to the last hex digit on theCray YMP. To really utilize this feature onthe Cray YMP, the SSD (solid state disk)must be used for the slabs (unless largememory and the diskless option is used).We have not yet added the best i/oscheme for the SSD, so the wall clocktimes are higher than they could be.


3-9

(3.8) FILTERING OPTIONSFiltering is used to combat the effect of theconvergence of meridians (shrinkinglongitudinal grid spacing) near the poles.The problem is that the grid spacingseverly limits the time step (especiallywhen using large density time steps) dueto the “CFL” criterion. Filtering relaxes thisconstraint by truncating high wavenumbers. Users should be aware that thefiltering options can be time consuming,and alter the model solution, and thereforeshould be used only as a last resortConfigurations of the GFDL ModularOcean Model code that are primecandidates for filtering are coarseresolution global models used to examinelong term climate. The timestepconstraints imposed by including the ArcticOcean, and the timescales of the problembeing studied, often lead to the need forfiltering at high latitudes.

fourfil is a fourier filter which acts onlongitudinal strips of ocean points. Thenumber of wave numbers to be allowed ateach latitude is decided upon andwavenumbers above are truncated(without using any reasonable window).This option is provided as a backwardcompatability feature for the Cox (1984)code. Note: Both in the MOM code and inthe Cox code, the amount of filteringdepends on the number of ocean pointswithin the longitudinal strip (determined bylmodel bathymetry). This actually leads topossibly different truncations at the samelatitude!

firfil is a simple finite impluse responsefilter based on the familiar 1/4, 1/2, 1/4weights (the response function is acosine). It comes in two flavors: One towith symmetric boundary conditions (ie:for tracers) and one for asymmetricboundary conditons (ie: momentum).The firfil option also works on longitudinalstrips of ocean points, and may be appliedan arbitrary number of times for anarbitrary number of latitudes. While thefirfil option is far less cp time intensivealternative to fourfil, we really have notdone any long running comparisons tocomapre the two schemes


3-10

(3.9) I/O OPTIONSdiskless simulates disk usage using anarray in memory. Since when using thisoption no “wall clock” time lost during disktransfers (because they are memory tomemory transfers), this method gives thebest efficiency (cp time / total time). Thecost for achieving this cpu time efficiencyis that the model may not be able to fit intoavailable memory.As a memory saving feature, disklesskeeps only two time levels of prognosticvariables (ntlev=2) instead of the usualthree. See the size.F file to examineresource requirements. Users may wish toread the discussion of the multitaskingoption for related information.

crayio is an option tested on GFDL’s CrayYMP. Currently, it uses the “getwa/putwa” routines. If the disk units are setup on “non SSD” type disk, then theefficiency (cp time / total time) will be bad.This shows the mismatch in speedbetween computationand disk use. SSD(solid state disk), however, is really thepreferred way to go. It can be looked at asan extended memory and the efficiency ismuch better.

fio is the option for using standard Fortrandirect access i/o. The comments fromcrayio apply here also.

(3.10) MISCELLANEOUS CODEOPTIONS:keepterms is an option allows the use ofarrays instead of statement functions forcomponent terms in the MOM code’stracer and momentum equations. Thismay be desirable when using componentterms over and over for some purpose.The trade off is the extra memory neededfor the arrays versus the extra timeneeded to recalculate the statementfunctions. See the size.F file to look atresource requirements.

nohilats stands for no high latitudes. If thelatitudinal domain of the model is limited toequatorial redions, for instance, then themetric nonlinear terms in the momentumequations may be dropped. Defining thenohilats option will save some cp time andallows for backward compatibility with theCox (1984) code.

restorst stands for restore surface tracersto some prescribed values by a newtoniandamping term. In the sample test case ofthe GFDL Modular Ocean Model code,temperatures and salinities are restoredback toward zonally averaged annualmean conditions on a time scale of 50days (ee file setvbc.F). Without muchwork, this option could easily be extendedto damp certain regions while leavingothers alone by simple making thedamping time scale a function of latitude,for instance.


3-11

testcfl tests whether any velocity exceedsthe local “CFL” criterion by a factor ofcflcrt(see file blkdta.F for wherevariable cflcrt is set). This option willprint out the places where the CFLcriterion is most nearly met on diagnostictime steps. If the cflcrt factor isexceeded on any time step, the MOMcode will stop its execution and print outthe surrounding variables.NOTE: the testcfl option is not meant tobe left on all the time, since is uses muchcp time. It is best used as an analysis aid.For example, it can be used infrequentlyon a model that is being run for a longtime, or should a version of the model“blow up”, it can be useful in pointing towhere the problem arises.



3-12


4-1

Any general circulation model contains anintimidating number of “things” to keeptrack of. For a new user, deciding whichthings to study in detail first so that otherportions of the model can be understoodcan be important question. This chapterseeks to describe for the user several ofthe basic building blocks that one needs toknow in order to develop workingknowldege of the GFDL MOM code. Weexpect that more seasoned users will referto these pages when seeking to refreshtheir memories.The discussion of building blocks beginswith the model grid itself. However, whiledescribing the grid, we will also introducesome specifics concerning conventionsused in the naming of variables.

CHAPTER 4: Some Basics of the GFDLModular Ocean Model


4-2

(4.1)THE THREE DIMENSIONALGRIDThe structure of the model grid itself is areasonable topic to tackle first whenbuilding up knowledge of the GFDLModular Ocean Model code.Understanding the layout of the grid andsome of the MOM code’s namingconventions that relate to grid locations iscrucial if one it to be able to mentallytranslate many parts of the code intovisual images.Figure 4.1.1 displays the horizontal gridstructure used in the MOM code andFigure 4.1.2 displays the vertical grid..Variables are arranged in the manner ofthe “B-grid” convention described byArakawa and Lamb (1977). Users mayrefer to the coord.h and grdvar.h filesfor lists and descriptions of many of thevariables relating to the grid.Parameters used to set the number ofmodel grid points in the west-to-east,south-to-north, and top-to-bottomdirections are imt, jmt, and km,respectively (see the param.h file). Thus,variables aligned along a specific latitudecan be referred to as being in the same “j-row”, with values of i increasing as onemoves eastward. The first j-row (j=1) liesat the southern limit of the model domain,and values of j increase as one movesnorthward. In the vertical, values of kincrease as one moves downward togreater depths.Since the GFDL MOM code employs theB-grid, tracer quantities (potentialtemperature, salinity, and other tracers)

are defined in the horizontal at differentlocations than the velocity componentvariables u and v. As shown in Figure4.1.1, a “u,v” grid point with a given i,jdesignation is located to the northeast ofthe tracer grid location having the samei,j values. The two-dimensional volumetransport streamfunction ^ (variable p inthe emode.h file) is also aligned on thehorizontal tracer grid.In the GFDL Modular Ocean Model,horizontal grid locations are carried interms of degrees of latitude and longitude.Variable xt(i) represents the longitudeof the “i-th” point on the tracer grid, andvariable xu(i) represents the longitude ofthe “i-th” point on the u,v grid. Similarly,variable yt(j) represents the latitude ofthe “j-th” model row on the tracer grid,while the latitude of the “j-th”u,v j-row isstored in variable yu(j).In the horizontal, tracer grid points lie inthe exact center of tracer grid box cells.The four points of a tracer grid cell lie atu,v grid point locations. The distance (incentimeters) across a tracer grid box cellin the north-south (meridional) direction atlocation i,j is stored in variable dyt(j),while the distance in the east-west (zonal)direction at the Equator is stored invariable dxt(i)(see the grdvar.h file).As depicted in Figure 4.1.1, the GFDLModular Ocean Model allows for variablegrid spacing in the horizontal. Therefore,u,v grid points do not necessarily lie inthe center of the four surrounding tracergridpoints. Distances across u,v grid cellsare given by variables dyu and dxu, andcorrespond to the distances (incentimeters) between tracer grid points inthe north-south and east-west directions,respectively.


4-3

N

W E

S

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

T

TTu v

u v

u v

u v(i,j)

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u v

u vu v

yu(j)

yu(j+1)

yu(j-1)

yt( j )

yt( j+1)

xt(i) xt(i+1)xt(i-1)xu(i) xu(i+1)xu(i-1)

dxt( i )

dxu(i)

dyu(

j)

dyt

( j )

FIGURE 4.1.1 Skematic view of the GFDL Modular Ocean Model’s horizontal grid arrangement.

h

q

yt( j-1)

(i,j)


4-4

Since the dxu and dxu variables arecalculated as the distance (in centimeters)across a grid cell in the zonal direction atthe Equator, one must multiply thesevariables by the cosine of the latitude ofthe j-row of interest in order to determinethe true east-west distance between gridpoints at a given latitude. The cosines ofthe latitudes of tracer grid points (yt(j))are stored in variable cst(j), and thecosines of the u,v grid point latitudes(yv(j)) of tracer grid points are stored invariable csu(j) (see file grdvar.h).The following sentence attempts to tietogether some of the elements presentedthus far concerning the horizontal gridused in the GFDL MOM code:The east-west distance, in centimeters,across the southern face of the tracer gridbox cell centered at latitude yt(j) andlongitude xt(i) is equal to the distancefrom longitude xu(i-1) to xu(i) atlatitude yu(j), which can be computedas... csu(j-1) * dxu(i)With the above preliminaries out of theway we can now turn to the question ofhow a user specifies the horizontal griddesired for a particular application in theMOM code. The process begins whenprogram ocean calls subroutinegrids. Subroutine grids, andsupporting subroutines mesh andcmesh and function indp are found infile setgrid.F.

Upon entering subroutine grids, a call tosubroutine mesh is made in order todetermine tracer grid cell widths dxt anddyt. All other grid variables are calculatedwithin the subroutine grids fromthese two variables.

As alluded to earlier, the GFDL MOM codeprovides a means by which variable gridspacing can be incorporated into themodel. This is achieved in subroutinemesh. The nxres and nyres parametersare the number of subdomains of eitherconstant or smoothly varying grid celldimensions. For the simplest case of agrid that is uniform in terms of latitude andlongitude grid spacing, parameters nxresand nyres are set to be equal to one infile param.h. This was done for the sampleconfiguration of the MOM code.Variables used to determine the gridspacing (be the spacing either uniform orvariable) are set by the user in theblkdta.F file. These variables (namely,variables stlon, xmax, xmin, xwidand idir for longitudinal subdomains,and stlat, ymax, ymin, ywid andjdir for the latitudinal subdomains) aredescribed in the comments found in thecoord.h file.For each of the longitudinal and latitudinalsubdomains asked for (nxres andnyres) the user must specify the sum ofthe distances (in degrees) to be coveredby tracer grid box cells within thatsubdomain. Subdomain widths are givenby xwid(nx) and ywid(ny), where nygoes from 1 to nxres and ny from 1 tonyres. Variables xmax and xmin andymax and ymin are used to specify theminimum and maximum tracer grid boxcell widths within the given domain.Uniform spacing within a subdomain canbe achieved by setting the minimum andmaximum widths to be equal. Otherwise,smoothly varying grid cell widths will becomputeda, and the idir and jdirvariables are set to either +1 or -1 tosignify whether the grid box cells are to


4-5

increase or decrease as one movesacross the subdomain to higher i or jvalues. The calculations for the variabletracer grid cell widths are performed insubroutine cmesh.The model grid point located in the farsouthwest of the entire model domain is atracer grid point with an i,j location of(1,1). All model grid locations arecomputed using this position as a startingpoint.. This starting location for the modelgrid is given (in degrees) by variablesstlat and stlon.Variables xt, yt, xt and xu are eachmonotonically increasing in the MOMcode. In cases of an ocean model withpseudo-realistic geometry, the cyclicoption needs to be specified, and thus thesum of all the longitudinal subdomainsmust be greater than 360 degrees (as inthe sample case).Should a user have a special application inwhich it is desired to have latiitudal and/orlongitudinal grid spacing that varies in away not readily produced by the abovedescribed method, we suggest thefollowing get-around. Add a new ifdefoption to the code, such that when it isdefined subroutine grids calls a new, usersupplied routine, rather than subroutinemesh.. This new routine should returnvariables dxt and dyt. Once that isaccomplished, all other grid variables willbe computed, starting at the stlat andstlon tracer gridpoint location.

REFERENCES

Arakawa, A., and V. Lamb, “ComputationalDesign of the basic dynamical processesof the UCLA general circulation model”,Methods in Computational Physics, vol 17,Academic Press, (1977)


4-6


IN-1

INDEX

Aah variable ............................................... 3-4, 3-7

see also eddy coefficientsahisop variable ........................................... 3-7

see also eddy coefficients ........................... 3-7aidif variable .............................................. 3-5am variable ....................................................... 3-4

see also eddy coefficientsanonymous ftp, see ftp command .......... iiiaq variable ....................................................... 3-6

see also eddy coefficients .......................... 3-6

Bbiharmonic option ........................................... 3-4blkdta.F file ............... 2-5, 3-5, 3-6, 3-8, 3-11bottom drag ..................................................... 2-5, 3-5boundary conditions

lateral ...................................................................... 2-5vertical .................................................................... 2-4

Bryan, Kirk ...........................................................vi, viibvdc variable ................................................. 3-5

Ccalling tree diagrams .......................................... 2-6cdbot variable .............................................. 3-5

see also bottom dragcflcrt variable ......................................... 3-11checks.F file ......................................... 2-4, 3-2C-language preprocessor, see cpp commandclinic.F file ................................................. 2-2common blocks ..................................... 1-4, 1-5, 2-2congr5.F file ................................................. 3-3congrad5pt option ........................................... 3-3congrad9pt option ........................................... 3-3consistency checks ............................................. 2-4consthmix option .............................................. 3-4constvmix option ............................................... 3-5convective adjustment ....................... 3-5, 3-6, 3-7coord.h file .................................................... 2-4Cox, Michael ...................... vi, vii, 2-3, 2-5, 3-3, 3-7cpp command ............................... 1-3, 2-4, 3-1, 3-2Cray YMP ......................................................... 2-2, 3-8

getwa/putwa routines ................................... 3-10

SSD (solid state disk) .......................... 3-8, 3-10cray.run file ......................................... 1-2, 3-2crayio option ..................................................... 3-10cregin.h file ................................................. 2-5crit variable ................................................. 3-3ctcmix.h file ................................................. 3-6ctmngr.h file ................................................. 2-4customization options ........................ 1-5, 2-3, 3-1

see also ifdef directivescvbc.h file ....................................................... 2-4cvmix.h file .................................................... 3-5

Ddenscoef.F file ........................................... 2-6depths program .............................................. 2-6diagnostic timestep ................................... 3-8, 3-11diskless option .................................. 3-2, 3-8, 3-10Dixon, Keith .........................................................iv, vidncoef.h file ................................................. 2-6docmnt.F file ......................................... 1-5, 2-5

Eeddy coefficients .......................... 3-4, 3-5, 3-6, 3-7e-mail addresses .....................................................vieqstat program .............................................. 2-6equation of state ................................................... 2-6equivalence statements ................................... 2-5external mode

options ................................................................... 3-3

Ffilename suffix ................................... 1-4, 2-2filtering options ...................................................... 3-9finite impluse response filter, see firfil option

3-9fio option ....................................................... 3-2, 3-10firfil option .............................................................. 3-9fkph variable ................................................. 3-5

see also eddy coefficientsfkpm variable ................................................. 3-5

see also eddy coefficientsfonts ...............................................................................ivFortran-77 ................................................................ 2-2fourfil option .......................................................... 3-9fourier filter, see fourfil option ..................... 3-9fourth order mixing .............................................. 3-4ftp command ....................................................iii, iv

Ggetvar.F file ................................................. 2-6


IN-2

GFDL Ocean Group Technical Report #1 .viGFDL Ocean Group Technical Report #2 .ivgrid options .............................................................. 3-8

HHellerman and Rosenstein winds ................ 1-3horizontal mixing schemes, see lateral mixing

schemeshybrid mixing schemes ..................................... 3-7hypergrid option ................................................ 3-3

II/O options ............................................................. 3-10ifdef directives ................................ 1-3, 2-3

conflict checks ................................................... 3-2implicitvmix option .......................... 3-5, 3-6, 3-7include files ................................. 1-4, 1-5, 2-2indp function ......................................... 2-4, 2-6input/output, see I/O optionsinternet ....................................................................iii, iviounit.h file ................................................. 2-5iperim.F file ................................................. 2-4island calculations ............................................... 2-4isopycmix option .............................................. 3-7isopycnal mixing, see isopycmix ............... 3-7isopycnal mixing, see isopycmix option

Kkeepterms option ................................... 2-2, 3-10Kerr, Christopher ................................................... vii

Llalg option, see tcvmix optionlateral mixing schemes ............................. 3-4, 3-7leq option, see tcvmix optionLevitus climatology .............................................. 1-3

MMahlman, J. ............................................................. viimatrix.F file ................................................. 2-6Mellor, George ....................................................... 3-6Mellor-Yamada turbulence closure, see also

tcvmix option ............................................... 3-6memory size ........................................................... 2-3mergelib utility ................................ 1-2, 1-4metric nonlinear terms ..................................... 3-10Miyakoda, Kikuro ........................................... vii, 3-4modular design ...................................................... 2-2MOM_NEWS file .............................................v, 1-2multitasking option ................................ 3-8, 3-10

my_mom directory ...........................................iv

Nnamelist ..................................... 2-2, 2-4, 3-3, 3-5, 3-6naming conventions ........................................... 2-3ncon variable ......................................... 3-6, 3-7nearest model gridpoint .................................... 2-4newtonian damping .......................................... 3-10nlhmix option ....................................................... 3-4nmix variable ................................................. 2-4nohilats option .................................................. 3-10non-linear horizontal subgrid-scale mixing

see also nlhmix option .............................. 3-4ntasks variable ........................................... 3-8ntlev parameter ................................. 3-8, 3-10

Oocean program ................................................. 2-6ocean.F file .................................................... 2-5ocean.in file ................................................. 1-2odam.F file ....................................................... 2-6off-line analysis ..................................................... 2-5oldrelax option ................................................... 3-3Oort climatology .................................................... 1-3optimization options ............................................ 3-8organization of variables .................................. 2-2out-of-bounds references ................................ 2-2

PPacanowski, Ronald ..............................vi, 3-5, 3-6param.h file .................................................... 2-2parameter statements ....................................... 1-4pconst.h file ................................................. 2-3Philander, S.G. .............................................. 3-5, 3-6Poisson solvers, see external mode optionsppvmix option ............................................. 3-5, 3-6PREP_DATA directory ..........................iv, 1-3printout file ......................................... 1-2, 1-5

RREAD_ME file ............................................ 1-2, 1-3readrmsk option ................................................ 2-5readrmsk option ........................................... 2-5Redi, M. ..................................................................... 3-7REFERENCES ..........iv, vi, 1-3, 3-3, 3-4, 3-6, 3-7region masks

horizontal .............................................................. 2-5vertical ................................................................... 2-5

regional averages ................................................ 2-5relax.F file .................................................... 3-3


IN-3

resource requirements ...................................... 2-6restorst option .................................................... 3-10rigidlid option ....................................................... 3-3Rosati, Anthony .........................................iv, vi, 3-4runtime documentation ..................................... 2-5

Ssaving original files .............................................. 1-4scalar.h file ................................................. 2-5scale selective mixing ........................................ 3-4SCCS ................................................................. 1-5, 2-4Scripps topography ............................................. 1-3second order mixing ................................... 3-4, 3-5Semtner, A. .........................................................vi, viisetgrd.F file ................................................. 2-6setgrid.F file .............................................. 2-4setvbc.F file ............................................... 3-10size program .................................................... 2-6size.F file ..................................................... 3-10skipland option ........................................... 2-3, 3-8slabs ............................................................ 2-3, 2-4, 3-8slabs.h file .................................................... 2-3slmxr variable .............................................. 3-7Smagorinsky, J. .................................................... 3-4snaps variable .............................................. 2-5snapshot option ................................................. 2-5solid state disk, see Cray YMP ..................... 3-8sor variable .................................................... 3-3Source Code Control System, see SCCSsources and sinks ................................................ 2-4splitlib utility ........................ 1-2, 1-4, 1-6SSD, see Cray YMP ($nopage) ................... 3-8statement functions ............................................. 2-3streamfunction

volume transport ............................................... 3-3switch.h file ......................................... 2-4, 2-5symmetry option ............................................... 3-2

Ttcvmix option ............................................... 3-5, 3-6

leq option ........................................................... 3-6term balances ........................................................ 2-5testcfl option ....................................................... 3-11thick.h file .................................................... 2-6time manager ......................................................... 2-4timing option ........................................................ 3-8tmngr.F file .................................................... 2-4tracer averaging timestep ................................ 3-8tracer averaging timestep, see also regional

averages ($nopage) ................................... 3-8tracer.F file ......................... 2-2, 3-5, 3-6, 3-7tracing variables and options ......................... 2-2turbulent kinetic energy .................................... 3-6

UUNIX commands

cat ............................................................................ 1-5chmod .................................................................... 1-4cpp, see cpp command ............................... 1-3diff .................................................................... 1-6, 2-4ftp, see ftp command ................................... iiigrep ................................................................. 2-2, 3-7ls ...................................................................................vtar ............................................................................. 1-2

UPDOC ..................................................................... 2-3upgrade utility ........................... 1-3, 1-5, 1-6

Vvdclim variable ........................................... 3-5vdclim variable, see also eddy

coefficients ...................................................... 3-5version number ..................................................... 1-2vertical mixing schemes ........................... 3-5, 3-7vertical resolution ................................................. 3-6vvclim variable ........................................... 3-5vvclim variable, see also eddy

coefficients ...................................................... 3-5

Wwind stress .............................................................. 1-3

the g.f.d.l. modular ocean model users guide

Documents