Reliable supplies of clean, freshwater are essential to life andeconomic growth. It is not sur-

prising then that demands for waterincreased dramatically during the lastcentury as human populations grew,and energy consumption and industryexpanded. As demand approachessupply, societies will become vulnera-ble to even minor variations in the cli-mate and use of the land. Ironically,we now need to critically manage aresource that had almost no value lessthan a generation ago.

Scarce water resources can be man-aged objectively if decisions are basedon the best available science and real-istic computational models of complexwatersheds. Detailed physics-basedmodels, running much faster than realtime on high-performance computers,can be used to test hypotheses aboutthe performance of watersheds facinginevitable land use changes, climatechange, and increased climate variabil-ity. Decision makers can use suchmodels to evaluate management alter-natives or the effects of alternate cli-mate regimes and to support decisionsabout allocations of water betweenagriculture, ecosystems, industry, andmunicipalities.

Los Alamos National Laboratoryand the National Science FoundationScience and Technology Center forSustainability of Semi-Arid Hydrologyand Riparian Areas are developing ahigh-resolution, physics-based compu-

tational model, known as the LosAlamos Distributed Hydrology System(LADHS). The model can be used toassess water resources at scales thatare relevant to science and to decisionmakers. It is composed of four inter-acting components: a regional atmos-pheric model that is driven by globalclimate data, a land surface hydrologymodel, a subsurface hydrology model,and a river-routing model. When cou-pled together, these four componentsrepresent the complete hydrosphere.Our scientific and engineering goalsare to retain the essential physics of allthe separate components and toinclude realistic feedback amongthem. Because several alternativeapplication codes (legacy codes) existfor each of these components, two ofour key software goals are to linkexisting applications together withminimal code rewriting and to providea software environment that is flexibleenough to accept different alternatives.

We describe our progress in usingthe LADHS by means of a concreteexample: quantifying the water bal-ance of the Rio Grande Basin.

The Rio Grande Watershed

The Rio Grande is a major riversystem in the southwestern UnitedStates and northern Mexico. Ourinterest is in the upper Rio Grande,which extends from headwaters in the

San Juan and Sangre de CristoMountains of southern Colorado toFort Quitman, Texas (about 40 milesdownstream from El Paso and Juarez),where it runs dry (see Figure 1). Theupper basin covers about 90,000square kilometers and includes thecities of Santa Fe and Albuquerque,New Mexico, and the El Paso–Juarezmetropolitan area. The Rio Grandesystem provides water for flora, fauna,agriculture, domestic consumption,recreation, business, and industry.

Water moves through the basinalong multiple natural pathways, themost important of which are precipita-tion, surface runoff, infiltration,groundwater recharge and discharge,and evapotranspiration, as seen inFigure 2. Spring snowmelt and sum-mer monsoon storms are the mainsources of water in the basin(Costigan et al. 2000). The northernRio Grande and its tributaries aredominated by snowmelt runoff, butstreamflow in the southern tributariesis dominated by summer rain from theNorth American monsoon.

The atmosphere and river dis-charges are the main mechanisms fortransporting water out of the basin—indeed, out of any basin. Annual riverflows have averaged about a millionacre-feet per year in the upper RioGrande, but variability is quite high.The basin has also been subjected tolengthy drought periods, such as theone in the 1950s that caused a rapid

232 Los Alamos Science Number 28 2003

shift in forest and woodlandzones on the Pajarito Plateau(Allen and Breshears 1998). Wemay be entering another suchdrought period now.

Apart from its land, sky, andrivers, the other major featureof the Rio Grande Basin isgroundwater, which is the pri-mary source of water for metro-politan areas. Losses from theriver to the groundwater arelocalized, as are gains to theriver from the groundwater. Insome areas, streamflow is evensupported by groundwater.Typically, the groundwater isrecharged through mountainblocks and in streams alongmountain fronts.

Increasing demands fromcompeting uses may eventuallydeplete groundwater resourcesand affect surface-waterresources. Indeed, water avail-ability is already an importantissue throughout the basin.Sustainability of waterresources in the upper RioGrande Basin requires anunderstanding of the conjunc-tive use of ground and surfacewater, especially groundwaterrecharge from different sources.

The LADHS

Our computational approach is tolink a regional atmospheric process withsurface and subsurface hydrologicprocesses in a data flow that corre-sponds to regional water cycles. Thedetailed physics of the physical process-es are summarized in Table I, alongwith the resolutions that we employ inour model. The flow of data through themodel reflects mass and energyexchanges among the four domains inour representation of the hydrosphere.Fluxes are basically driven by dissipa-tive waves operating at different scales.

It should be noted that like everymajor river in the West, the Rio Grandeis highly regulated; thus, the measuredstreamflow reflects the operation ofdiversion and storage dams as well asnatural forces. Reservoirs and theiroperations are critical to determiningregional effects of climate variation,because management of the waterresource can alleviate or modify theimpact of variability through storageand operation (Lins and Stakhiv 1998).At present, the LADHS emphasizesinteractions among natural processes,although the system is modular enoughto accept components representinghuman demands and resources.

Regional Atmosphere. Theregional atmosphere compo-nent of our model is currentlyrepresented by the RegionalAtmospheric ModelingSystem (RAMS). It providesprecipitation, temperature,humidity, radiation, and winddata to the surface-waterhydrology component. RAMSsolves the Navier-Stokes equa-tions with finite-differencingmethods to estimate potentialtemperature, mixing ratio ofwater, atmospheric pressure,and horizontal and verticalcomponents of wind (Pielke etal. 1992, Cram et al. 1992).The model consists of mod-ules that allow for many possi-ble configurations of parame-terizations for processes suchas radiation calculations andcloud microphysics. RAMScan use telescoping, interac-tive, nested grids to represent alarge area with relativelycoarse resolution and smallerareas within this domain withgreater resolution. For eachtime step, the coarse-gridinformation is interpolated tothe fine grid and the fine-gridvariables are averaged back upto the coarse grid to provide

the two-way interaction. We can enternonstationary global climate effectsinto RAMS via global boundary condi-tions. These would be set by observedsea-surface temperatures and atmos-pheric fields or by output from a globalclimate model.

Land Surface. The Los AlamosSurface Hydrology (LASH) System isa grid-based water balance model(Xiao et al. 1996, Ustin et al. 1996)that represents land surface hydrologyand, in particular, the hydrology ofriver basins. It also represents someprocesses in high resolution to accountfor soil erosion, contaminant transport,

Number 28 2003 Los Alamos Science 233

Virtual Watersheds

Figure 1. The Upper Rio Grande BasinThe upper Rio Grande runs from southern Colorado tothe western-most tip of Texas. The black boundarydefines the basin. All ground and surface water withinthe basin eventually flows towards the river.

and biogeochemical cycling. Themodel simulates surface and subsur-face flows in two dimensions. Surfaceflows are routed using a diffusivewave approximation to the momentumequation with an explicit finite-differ-ence scheme solution (Julien et al.1995). Subsurface flow is routed usinga finite-difference form of Darcy’s lawto determine the amount of flowbetween adjacent elements. The soilprofile consists of two layers, plus athird if a saturated zone is present.Evapotranspiration, or the process bywhich plants extract water from a sub-surface layer and “secrete” it throughtheir leaves into the atmosphere, isbased on the incomplete cover modelpresented by Ritchie (1972).

River Routing. Our initialapproach was to use the NationalWeather Service’s Dynamic WaveOperational Model (Fread 1988) tomodel how rivers and channelswould flow, given our land contours,since we planned to simulate basinsunder natural (unregulated) flow con-ditions. However, those conditions donot provide the data needed by waterresource managers. We are evaluatingother codes for their ability toinclude reservoirs and dendriticdrainage patterns.

Subsurface Hydrology.Groundwater represents a major waterresource that is not included in currentclimate models. The Finite ElementHeat and Mass (FEHM) code is athree-dimensional multiphase flow codethat we use to model both the shallowsubsurface aquifers and regionalaquifers (Zyvoloski et al. 1997).FEHM solves mass- and energy-flowequations in a porous medium usingcontrol-volume finite elements.

So far, we have concentrated oncoupling RAMS and LASH together,because the land surface–atmosphereinterface controls most hydrologic

exchanges on time scales of less thana few years. LASH requires meteoro-logical data from RAMS, such as pre-cipitation, temperature, wind speed,short- and long-wave radiation, andair pressure, whereas RAMS mustreceive evapotranspiration and relatedquantities from LASH. However, bothRAMS and LASH are legacy codesthat were not designed to be coupledto other codes. The scale and size of thedata structures used by each code aredifferent; two- and three-dimensionalarrays must be exchanged; RAMS runsin a master/slave style and has a user-defined distribution of data thatdepends on the number of processors;and the two applications have differentgrid orientations.

The Parallel Applications Workspace(PAWS), developed at Los Alamos,provides a flexible software environ-ment for connecting these separateparallel applications. PAWS can alsoaccept any alternate application codeswe wish to incorporate into the model.

A central PAWS controller coordi-nates communications between appli-cations so that they can share paralleldata structures, such as multidimen-sional arrays. Applications can haveunequal numbers of processors, usedifferent parallel data layout strate-gies, and be written in different lan-guages. After the workspace is estab-lished before runtime, PAWS does notinterfere with processing. The PAWScontroller coordinates the creation ofconnections between components anddata structures.

Originally developed through theDOE Accelerated StrategicComputational Initiative and Office ofScience DOE 2000 AdvancedComputational Testing and SimulationToolkit, PAWS has been extended andgeneralized by the requirements ofLADHS. New capabilities includehandling multiple grid orientationsand data with strides greater than 1,transmitting local data within guard-cell-bound memory, interacting with a

234 Los Alamos Science Number 28 2003

Virtual Watersheds

Figure 2. The Hydrologic CycleA river basin is a dynamic region, with water entering and leaving along multiplenatural pathways. Precipitation (primarily rain, hail, or snow) brings fresh water intothe basin. The water can flow overland (surface runoff) and make its way to smallchannels, streams, and tributaries before becoming part of the river. Water alsoenters the ground, where it can flow beneath the land surface and eventually feedthe river, or it can recharge (resupply) aquifers. The major process that returnswater to the atmosphere is evapotranspiration, a dual process consisting of evapo-ration from surface areas, and transpiration, wherein plants absorb and subse-quently evaporate groundwater. The LADHS couples these processes, providing acomplete water balance for the river basin.

Soilhydrologicmodel

Evapotranspiration

Groundwaterhydrologic model

Regional atmosphericmodel

Precipitation

Channelgroundwater interaction

Overlandflow

Channelflow

Terrestrialhydrologic model

Infiltration

master/slave component model, andusing multiple communication strate-gies. These capabilities are also ofinterest to the Common ComponentArchitecture Forum, of which thePAWS project is a member andwhich is working on defining stan-dardized component interfaces forhigh-performance computing.

One of our next steps will be toimplement in PAWS the entireLADHS—RAMS, LASH, FEHM, andriver-routing applications.

Initial Studies and Results

In our initial studies, we have beenespecially interested in how the spatialextent and timing of precipitationinfluences soil moisture, a metric thatis of particular interest to farmers. Wehave chosen the 1992–1993 wateryear (October 1992–September 1993)as our test period and the northernhalf of the Rio Grande Basin (south-western Colorado and northern New Mexico) as our test area. The1992–1993 water year was an El Niñoyear with higher than normal precipi-tation in the Southwest, especiallyduring the winter season.

Precipitation is notoriously difficultto simulate because it is highly local-ized. Nonetheless, its timing andextent are critical to regional and localwater budgets. Our precipitation esti-mates are based on high-resolution

simulations using RAMS with threenested grids. The largest grid, 80 kilo-meters on a side, covers most of thewestern United States, along withparts of Canada, Mexico, and thePacific Ocean. This grid is necessaryto simulate the flow features in theregion. A medium-scale grid containsthe states of Utah, Arizona, Colorado,and New Mexico and has a horizontalgrid spacing of 20 kilometers. Giventhat resolution, large terrain features,such as mountain ranges, are resolvedwell enough to be recognized by themodel. A third grid, 5 kilometers on aside, is also used in many of the simu-lations to better resolve smaller terrainfeatures.

Our initial results indicate that theRAMS model can reproduce the pro-nounced year-to-year variabilityobserved in precipitation patternsacross the western United States(Costigan et al. 2000). Simulated andobserved monthly precipitation totalscompare fairly well, although they arefar from perfect (see Figure 3). In gen-eral, the 1992–1993 water year waswetter then normal, and our model hada tendency to overestimate precipita-tion at some high-elevation locations.

Figure 4 shows an example of out-put from the coupled land surface/atmosphere model, in which we simu-lated the effect of snow-water equiva-lent on soil moisture. Snow-waterequivalent is the amount of water con-tained in snow, and its extent is the

same as the snowpack. Snow accumu-lation is based on the RAMS definitionof snow, with snowmelt determined bytemperature. It is produced by RAMSat 5-kilometer resolutions, and theblocky nature of the snow distributionin Figure 4(a) is evident. LASH oper-ates at a much finer, 100-meter resolu-tion. RAMS and LASH were coupledby a statistical down-scaling techniquebased on kriging, which is an estima-tion procedure used in geostatistics(Campbell 1999). The highly resolvedland surface (modeled by 9,307,500grid cells) results in a smooth, detailedmap of soil moisture. That level ofdetail is important when simulatinglocal processes such as soil erosion andcontaminant transport.

Conclusion

Although we cannot experimentwith a system as large and valuable asthe hydrosphere of the Rio GrandeBasin, computer hardware and soft-ware have advanced until simulationsof river basins can be highly realistic.Gaps in the data and inadequacies incoupling the components of the modelare now the main limits on basin-scalesimulations. In some cases, couplingis simply a matter of scaling oneprocess to another while conservingmass and energy. In other cases, newscience is required. This is especiallytrue of “ecohydrology” and “agrohy-

Number 28 2003 Los Alamos Science 235

Virtual Watersheds

Table I. LADHS Physical Processes and Model Resolutions

Component Physics Characteristic Scales Model Resolution

Groundwater Darcy’s equation mm-m/day ~100 m

Unsaturated subsurface Multiphase flow mm-cm/min 100 m

Atmosphere Navier-Stokes equations mm-m/s 1–5 km

Overland flow Saint-Venant equations cm-m/s 100 m

Snowmelt Diffusion (heat and mass) m/hr 100 m

Stream Saint-Venant equations m/s By reach

Evapotranspiration Diffusion m/s 100 m

drology,” where the effects of riparianareas and farming on processes likeaquifer recharge and evapotranspira-tion must be quantified.

We also need new science to repre-sent the impacts of municipalities and

industry. Although large networks existfor observing some data, such as tem-perature and precipitation, they are theexception. Remote sensing, especiallysatellite based, and new geological andgeophysical characterization tech-

niques may eventually fill many datagaps. However, the theory of coupledbasin-scale modeling will need meth-ods of quantifying uncertainty becauseno data set will ever be exact.

As human activity pushes againstthe margins of available water supplies,we may soon need a crystal ball toassess the effects of even small increas-es in demand or small variations insupply. What does a crystal ball looklike? One version may be a large com-puter, a computational model, and ateam of scientists that can apply themodel and interpret the results. �

Further Reading

Allen, C. D., and D. D. Breshears. 1998.Drought-Induced Shift of a Forest-WoodlandEcotone: Rapid Landscape Response toClimate Variation. Proc. Natl. Acad. Sci.U.S.A. 95: 14839.

Campbell, K. 1999. Linking Meso-Scale andMicro-Scale Models: Using BLUP forDownscaling. 1999 Proceedings of theSection on Statistics and the Environment,American Statistical Association.

Costigan, K. R., J. E. Bossert, and D. L.Langley. 2000. Atmospheric/HydrologicModels for the Rio Grande Basin:Simulations of Precipitation Variability.Global Planet. Change 25: 83.

Cram, J. M., R. A. Pielke, and W. R. Cotton.1992. Numerical-Simulation and Analysis ofa Prefront Squall Line. 1. Observations andBasic Simulation Results. J. Atmos. Sci. 49(3): 189.

Fread, D. L. 1988. The NWS DAMBRK Model:Theoretical Background/UserDocumentation. National Weather Service,Office of Hydrology, Silver Spring, MD.

Julien, P. Y., B. Saghafian, and F. L. Ogden.1995. Raster-Based Hydrologic Modeling ofSpatially-Varied Surface Runoff. WaterResour. Bull. 31 (3): 523.

Lins, H. F.; and E. Z. Stakhiv. 1998. Managingthe Nation’s Water in a Changing Climate. J.Am. Water Resour. Assoc. 34 (6): 1255.

Pielke, R. A., W. R. Cotton, R. L. Walko, C. J.Tremback, M. E. Nicholls, M. D. Moran, D.A. Wesley, T. J. Lee, and J. H. Copeland.1992: A Comprehensive MeteorologicalModeling System—RAMS. Meteor. Atmos.Phys. 49: 69.

Ritchie, J. T. 1972. A Model for PredictingEvaporation from a Row Crop withIncomplete Cover. Water Resour. Res. 8: 1204.

Ustin, S. L., W. W. Wallender, L. Costick, R.

236 Los Alamos Science Number 28 2003

Virtual Watersheds

41N

39N

37N

35N

33N

31N

41N

39N

37N

35N

33N

31N

123W 121W 119W 117W 115W

Longitude (°)113W 111W 109W 107W 105W

Latit

iude

(°)

Latit

iude

(°)

< 50 mm< 125 mm< 250 mm

50 <125 <250 <

Figure 3. Precipitation from RAMSThe plots are a comparison of (a) observed data and (b) RAMS output for July 1993.The blue lines mark the approximate location of nested grid boundaries. Circles arecentered on the observation sites with their size representing the accumulated pre-cipitation (in millimeters) for the month. Model results were bilinearly interpolated tothe observation sites in order to facilitate comparisons. While not perfect, the RAMSestimate of the seasonal precipitation agrees with the measured data.

(a)

(b)

Lobato, S. N. Martens, J. Pinzon, and Q.Xiao. 1996. Modeling Terrestrial andAquatic Ecosystem Responses toHydrologic Regime in a CaliforniaWatershed. Chapter 6 in Status of the SierraNevada, Volume III Assessments,Commissioned Reports, and BackgroundInformation. Sierra Nevada EcosystemProject Final Report to Congress, WildlandResources Center Report No. 38, Universityof California, Davis.

Xiao, Q.-F., S. L. Ustin, and W. W. Wallender.1996. A Spatial and Temporal ContinuousSurface-Subsurface Hydrologic Model. J.Geophys. Res. 101: 29565.

Zyvoloski, G. A., B. A. Robinson, Z. V. Dash,and L. L. Trease. 1997. User’s Manual forthe FEHM Application a Finite-ElementHeat- and Mass-Transfer Code. LA-13306-M. Los Alamos, New Mexico: Los AlamosNational Laboratory.

Number 28 2003 Los Alamos Science 237

Virtual Watersheds

0

0.1–5

5–10

10–20

20–40

40–80

80–120

120–180

180–240

>240

Snow-waterequivalent

(cm)0.00–0.05

0.05–0.10

0.10–0.15

0.15–0.20

0.20–0.25

0.25–0.30

0.30–0.35

0.35–0.40

0.40–0.45

0.45–0.50

0.50–0.55

0.55–0.60

Volumetricsoil moisture

content

Figure 4. Examples of Snow Pack and Soil Moisture Results from LADHSPanel (a) shows the RAMS estimates of snow-water equivalent. Snow is mainly found in the San Juan and Sangre de CristoMountains during this October-November period. The snow distribution is not resolved very well because of the coarseness ofthe RAMS grid (5-km grid cells). (b) The plot shows the surface soil moisture estimates from LASH. Coupling between RAMSand LASH, which uses a finer grid (100-mm cells), smoothes the snow distribution. The distribution of soil moisture ranges fromvery dry in the San Luis Valley around Alamosa, Colorado, where there is little precipitation on an annual basis, to very wet con-ditions in higher-elevation zones where snow accumulation and melt usually occur.

Larry Winter is an applied mathematicianwith nearly 30 years of experience in computa-tional science and mathematics. He is the

deputy director of theNational Center forAtmospheric Research inBoulder, Colorado. Whileat Los Alamos, he was amember of theMathematical Modelingand Analysis Group in theTheoretical Division and

led the Geoanalysis Group and the ComputerResearch and Applications Group. Larry’s prin-cipal scientific interests are applications of sto-chastic partial differential equations andhigh-performance computing to the geo-sciences. He was one of the principal investiga-tors on the Los Alamos project to model thewater cycle of the upper Rio Grande Basinusing high-performance computing, a part ofthe scientific program of the National ScienceFoundation’s Science and Technology Centerfor Sustainable Semiarid Hydrology andRiparian Areas. He led the center’s effort todevelop high-performance computational mod-els of basin-scale water cycles.

Everett Springer graduated from theUniversity of Kentucky in 1975 with a bache-lor’s degree in forestry and received his Ph. D.in watershed science from Utah StateUniversity in 1983. Everett was a hydrologistwith the AgriculturalResearch Service of theU.S. Department ofAgriculture from 1982to 1985, when hebecame a staff memberin the EnvironmentalScience Group at LosAlamos. He is currentlya member of theAtmospheric, Climate and EnvironmentalDynamics Group. He has worked on character-izing and modeling vadose zone and surfacehydrology as well as contaminant transport.For the past 6 years, he has worked on apply-ing high-performance computing to simulatingcoupled hydrologic systems.

(a) (b)