overview of the arcgis wrf-hydro pre-processing tool · overview of the arcgis wrf-hydro...

Presented by:

Kevin Sampson, GISP

NCAR WRF-Hydro Development Team

Overview of the ArcGIS WRF-Hydro Pre-processing Tool

CUAHSI Training Workshop | The Community WRF-Hydro Modeling System | May 2-4 2017

Motivation

o Fast, efficient method for producing the ‘routing stack’ and convenience data.

o Consistent processing methodology between domains, regions, datasets.

o Remove the heavy GIS burden from modelers.

To provide WRF-Hydro with a complete set of routing grids that have been hydrologically processed according to user specifications.

Terrain Pre-processing Tools

o WRF-Hydro ArcGIS Pre-processing Toolset

• Fast, efficient method for producing the ‘routing stack’ and other convenience data

• Consistent processing methodology between domains, regions, datasets

WRF-Hydro & ArcGIS

o Desktop GIS Application Suite

o Site-licenses available at most US academic institutions

o Ecosystem of compatible hydrology tools

• Spatial Analyst

• ArcHydro

• TauDEM

o Extensible using Python API (arcpy)

o Handles everything from projections, to analysis, to mapmaking in one library.

o netCDF4 Python included as of 10.3

o https://github.com/NCAR/wrf_hydro_arcgis_preprocessor.git

Requirements

o ArcGIS for Desktop

• Version 10.3.1+ (minimum 10.2.1)

• DO NOT USE ArcGIS 10.4

• Basic, Standard, or Advanced license levels

• Spatial Analyst extension required

• Python 2.7.8, NumPy 1.7.1

• Both installed with ArcGIS Desktop ‘complete’ installation

Python Toolboxes

o Python script wrapped to act as an ArcGIS Toolbox

ToolboxScript (.pyt)

FunctionScript (.py)

o PYT file is the toolbox script containing multiple toolsets

• Functions called from separate script (wrf_hydro_functions.py)

o Parameter handling and validation

WRF-Hydro preprocessor toolbox as viewed from ArcCatalog.

AdvantagesEasy to modifyPortableMany tools organized

Python Toolboxes (.pyt)

o Toolboxes wrapped in Python classes

Pre-ProcessorInputs Outputs

Other parameters

© esri

netCDF file format

o network Common Data Form

• “.nc” extension

o Self describing

• Includes information about the data, coordinate system

o Machine independent

• Usable in many operating systems

o Used in Atmospheric Science

o Multidimensional

• x, y, z, t

y

x

Time

z

FullDom File (Routing Grid)

o netCDF WRF-Hydro input file

o Full High Resolution Domain file (FullDom_hires.nc)

o Stores all routing grids as 2-dimensional variables

o Stores CF-compliant spatial metadata

• grid_mapping

• Projection information

• Coordinate System variable

• ArcGIS-compliant projection information

o Easy to import into GIS Applications

o Ingested by WRF-Hydro

A Note on CF Metadata

o Climate and Forecast Conventions for netCDF data

• Like a standard

• Current version 1.6

• http://cfconventions.org/latest.html

o CF conventions for just about any type data

• Gridded

• Point

• Profile

• timeSeries

o CF-compliant netCDF files make them much easier to use in client applications

• Panoply, ArcGIS, QGIS

Process GEOGRID File

Inputs

o Required:

• WRF GEOGRID file (.nc)

• High-Resolution Elevation

• Elevation file (Esri GRID, GeoTIFF, etc.)

• Mosaic Datasets

o Parameters:

• Regridding Factor – nesting relationship of routing:land grids

• Minimum basin size (in routing grid cells)

• OVROUGHRTFAC - constant

• RETDEPRTFAC – constant

o Optional:

• Station Locations (.csv)

• Lake Polygons (polygon feature class or .shp)

Model Domain

• Area of interest

• Defines model domain

– Size

– Location

– Horizontal resolution

• Defined by geogrid file

– Example:

Input: WRF GEOGRID

o GEOGRID is used in the WRF-Hydro GIS Pre-processor to define the domain’s coordinate reference system, extent, resolution, and certain 2D variables:

• HGT_M (elevation)

• LU_INDEX (landuse)

o Currently supported GEOGRID coordinate systems

• MAP_PROJ = 1 (Lambert Conformal Conic)

• MAP_PROJ = 3 (Mercator)

• MAP_PROJ = 6 (Cylindrical Equidistant but NOT w/ rotated pole)

• MAP_PROJ = 2 (Polar Stereographic)

The purpose of the GEOGRID file is to define the simulation domain and interpolate various static geographical datasets to the model grid.

GEOGRID: Projected Coordinate System

"PROJCS['Lambert_Conformal_Conic',GEOGCS['GCS_Sphere',DATUM['D_Sphere',SPHEROID['Sphere',6370000.0,0.0]],PRIMEM['Greenwich',0.0],UNIT['Degree',0.0174532925199433]],PROJECTION['Lambert_Conformal_Conic'],PARAMETER['false_easting',0.0],PARAMETER['false_northing',0.0],PARAMETER['central_meridian',-105.0],PARAMETER['standard_parallel_1',30.0],PARAMETER['standard_parallel_2',50.0],PARAMETER['latitude_of_origin',39.9400138855],UNIT['Meter',1.0]];-36695400 -29251300 10000;-100000 10000;-100000 10000;0.001;0.001;0.001;IsHighPrecision"

esri PE string

Input: Elevation Raster

o Must be an ArcGIS-readable raster format

o Must contain valid coordinate reference system

o Must cover entire extent (and more) of your GEOGRID domain

o Elevation units must be converted to meters (m)

o Should be hydrologically corrected

• Not necessary but helps with channel placement, hydro enforcement, etc.

o Supported datasets include, but are not limited to:

• HydroSHEDS

• NHDPlus (converted from cm to m)

• EU-DEM

Input: Elevation Mosaics

NHD Plus v2 EU-DEM

HydroSheds

Input: Regridding Factor

1000m

100m

GEOGRID Resolution

Regridding Factor= Routing Resolution

1000m

100mRegridding Factor: 10

Raster resolution for terrain processing

1000m 30m

Model Resolution High Resolution

Terrain Pre-processing Workflow

o Resample high-resolution DEM and land use to target resolution and extent using Nearest Neighbor algorithm

o Void-fill the resampled DEM

o D8 Flow Direction

o Flow Accumulation

o Derive CHANNELGRID from flow accumulation raster using threshold of minimum basin size

o Derive Strahler stream order from CHANNELGRID

DEM Resampling Pit-filling Flow directionFlow

accumulationStream

DefinitionStream Order

Basic workflow for terrain pre-processing of WRF-Hydro routing grids.

Reproject & Resample Hydro DEM

o Project input DEM to model projection and domain

o Resample to routing grid resolution

• NEAREST NEIGHBOR resampling

• Finds the ‘nearest neighbor’ pixel in input DEM to each new pixel in the model grid.

• There are more sophisticated methods, but this is fastest and preserves values inside original range of values.

o Re-projection and Resampling can ‘break’ the input HydroDEM.

• Causing artificial ‘pits’.

• Filling in ‘burned in’ areas.

o Even though we start with a HydroDEM, we ‘break’ it, then re-condition it.

Process: Pit Filling

Spatial Analyst “Fill“ Tool

o Fill depressions so that water can roll downhill only. This also creates a smoother DEM than you might find in nature.

o This simple hydro-enforcement method can resolve most issues in a DEM.

o Optional z-limit (global variable) to limit fill depth.

o Problem: some pits are natural, some are not.

© Esri: http://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-fill-works.htm

Methods: Tarboton et al (1991)

Pit-Filling issues

o …some pits are natural, some are not.

Resampling can break hydrologic connectivity

o Coarsening a HydroDEMcan break hydrologic connectivity.

o Try not to get too much coarser than input HydroDEM, or perform extensive hydro-enforcement on your input DEM first.

o Here, a canyon outlet is filled, causing entire valley to fill during pit filling process.

Flow Direction & Flow Accumulation

o D8 Flow Direction

o Flow Accumulation

1

248

16

32 64 128

Methods: Jenson and Domingue (1988)

© Esri: http://desktop.arcgis.com/en/arcmap/latest/tools/spatial-analyst-toolbox/how-flow-accumulation-works.htm

Stream Definition

o Input Parameter: Number of pixels to define stream

• Yields a minimum ‘basin’ size

• Given in pixels (unitless), on the routing grid

• Affects density of generated channel network

1km2 0.1km2 0.01km2

An analytic method for determining an appropriate threshold value for stream network delineation is presented in Tarboton et al. (1991)

Stream definition

o Use flow accumulation threshold to define channels

o Option: use gaged basins as mask to assign CHANELGRID values

o If reach-based routing is selected, Stream to Feature used to create vector geometry of streams

• streams.shp shapefile written to output directory

Flow Accumulation Channelgrid

Input: Station Location File

o Create in Excel, Numbers, Word, etc.

o Direct output of attribute table from shapefile or feature class

o “LON” & “LAT” required

o If present, basins will be delineated using the points provided

• frxst_basns output variable will be created

• frxst_pts & basn_msk variables will be populated

o If masked to basins, CHANNELGRID will have values -1, 0, -9999

FID,LON,LAT,STATION,Name15,-105.92833,40.08139,Fraser_at_Granby,903330018,-105.9,40.12083,COLO_nr_GRANBY,901950020,-106.3333,39.8803,Blue_R_blw_Grn_Mtn,9057500

Process: Basin Delineation

o Snap points to streams

o ‘Walk’ down channel network a specified distance

• Default = 3 pixel widths (global variable)

o Delineate basin using ‘Watershed’ Spatial Analyst tool

Conejos_nr_Mogote

• Writes output file to: frxst_basns, basn_msk, &gw_basns_geogrid.txt

Basin Masking

o Option to ‘turn off’ channel networks outside gaged basins.

o If gages are provided and the option to mask CHANNELGRID to basins is selected.

• Channel pixels inside gaged basins = 0

• Channels outside = -1

Process: Reach-Based Routing

o CHANNELGRID raster is converted to a shapefile (streams.shp)

o Decomposes line geometry to nodes, and gathers elevation, Latitude and Longitude at each node

o LINKID grid in FullDom file is created and populated with link ID values

o Constructs a .nc file with necessary parameters for reach-based routing:

• Length, Slope, Order, Drop, X/Y, etc.

• Writes output file to Route_Link.nc

Reach-routing Table

Parameter Description

link Link ID (NHDFlowline_Network COMID)

from From Link ID (PlusFlow table FROMCOMID for every TOCOMID)

to To Link ID (PlusFlow table TOCOMID for every FROMCOMID)

lon longitude of the start node

lat latitude of the start node

alt Elevation in meters from the North American Vertical Datum 1988 (NADV88) at start node (MaxElevSmo)

order Stream order (Strahler)

Qi Initial flow in link (CMS)

MusK Muskingum routing time (s)

MusX Muskingum weighting coefficient

Length Stream length (m)

n Manning's roughness

So Slope (meters/meters from NHDFlowline_network.SLOPE)

ChSlp Channel side slope

BtmWdth Bottom width of channel

Kchan Channel conductivity (mm/hr)

x X-coordinate in projected coordinate system

y Y-coordinate in projected coordinate system

o CF-netCDF file containing reach-routing parameters

o Mix of NHD attributes and default values

Process: Stream Order

Stream Order Spatial Analyst tool

• Strahler stream order

• Writes output file to STREAMORDER variable

Route_Link.nc

o netCDF file to store link information.

• 1-Dimension (linkDim)

o CF-netCDF ‘timeSeries’ convention.

DefaultsQi: 0 cmsMusK: 3600 sMusX: 0.2n: 0.035ChSlp: 0.05BtmWdth: 5 mKchan: 0 mm/hr

Process: Latitude & Longitude Grids

o Each pixel on the routing grid will have lat/lon calculated

o Project domain grid to a sphere

o Uses $$XMAP & $$YMAP functions from deprecated arcgisscripting python library.

o Project back to WRF projected coordinate system.

Latitude Longitude

Process: Reservoir Routingo If the option is selected, a polygon shapefile or feature class is

required as input.

o Populates LAKEGRID variable

o Assigns lake ID values to pixels where lakes drain into channel.

o Constructs a LAKEPARM.nc file with necessary variables for reach-based routing:

• Lake area, max elevation, min elevation, base elevation, orifice elevation

LAKEGRID / LAKEPARM.nc

o Input: Reservoirs Shapefile or Feature Class (polygon)

o Size threshold: ‘Threshold’ global variable

• Default: 0.75 km2

o Polygons are resolved on the model grid

o Lake ID (provided) is used to identify lakes

• If no ID is provided, 1-n is used

Original polygons Lakes on the routing grid

Reservoirs and Channels

Lake Routing Table

Parameter Description

Discharge Default Discharge

LkArea Gridded lake area (sq. km)

LkMxH Maximum lake elevation (m ASL)

OrificeA Orifice cross-sectional area (sq. m)

OrificeC Orifice coefficient

OrificeE Orifice elevation (m ASL)

WeirC Weir coefficient

WeirH Weir Height (m ASL)

WeirL Weir length (m)

alt vertical distance above mean sea level in (m ASL)lake_id Lake ID

lat latitude of the lake centroid

lon longitude of the lake centroid

Lakes/Reservoir routing in the NWM

• Lake parameters are stored in the LAKEPARM.nc lake routing table

• Lakes are defined on the 250m routing grid, parameters in the table

Other Grids

o landuse

• GEOGRID LU_INDEX resampled (nearest neighbor) to routing grid

o OVROUGHRTFAC

• Constant 1.0

o RETDEPRTFAC

• Constant 1.0

o LKSATFAC

• Constant 1000.0

GIS Pre-processor Outputs

o Set of netCDF, shapefile, txt (ASCII raster), & log files

• 2-4 netCDF files

• 0-1 ASCII Raster (.txt)

• 0-1 Shapefiles

• 1 .log file

Other Utilities

o ExamineOutputs

• Extracts .zip output file to individual rastersfor viewing in Desktop GIS applications.

o ExportGrid

• Export any M-grid variable from the GEOGRID file to GRID or .TIF format

o ExportPRJ

• Reads GEOGRID file attributes and builds a projection file (.prj)

o GenerateLatLon

• Builds latitude and longitude GRID files from any raster input.

o DomainShapefile

• Creates a polygon shapefile defining the domain boundary.

Customization

o Multiple Toolboxes may be released for different users

• UCAR Internal toolbox takes advantage of available services

• Elevation mosaics and vector services (NHDPlus lakes, flowlines)

• Modular tool structure allows building rapid new toolsets

Tool Messages

Documentation & Test Data

o Detailed documentation

• 40+ page PDF

• Describes tool capabilities, requirements, parameters, and GIS methods used in the tool chain.

o Small GEOGRID domains for testing tool functionality

• Front Range (Lambert Conformal Conic)

• India (Mercator)

o Expected Output provided for comparison

o Required Elevation files (.tif) provided

o Optional stream gages & lakes provided (Front Range)

Bottlenecks / Constraints

o Project High-Resolution Dataset for large areas

• Can be avoided by pre-projecting/resampling high res data

o Flow Accumulation – slowest part of the process

o Not multi-threaded

• Processes runs on one core

• Process not well suited to parallelization

o Windows only GIS platform

THANK YOUksampson@ucar.edu

References

o Jenson, S. K., and J. O. Domingue. 1988. "Extracting Topographic Structure from Digital Elevation Data for Geographic Information System Analysis." Photogrammetric Engineering and Remote Sensing 54 (11): 1593–1600.

o Tarboton, D. G., R. L. Bras, and I. Rodriguez–Iturbe. 1991. "On the Extraction of Channel Networks from Digital Elevation Data." Hydrological Processes 5: 81–100.

WRF-Hydro GIS Preprocessing ToolsDemo