multidimensional data and gis steve kopp nawajish noman esri

Multidimensional Data Multidimensional Data and GISand GIS

Steve KoppSteve KoppNawajish NomanNawajish Noman

ESRIESRI

OverviewOverview

Introduction to Multidimensional DataIntroduction to Multidimensional Data What is netCDF?What is netCDF? Some netCDF data sourcesSome netCDF data sources Working with netCDF in ArcGISWorking with netCDF in ArcGIS

Multidimensional DataMultidimensional Data

Data cube (3D) or hypercube (4D,5D…)Data cube (3D) or hypercube (4D,5D…)

• Temperature varying with timeTemperature varying with time

• Temperature varying with time and altitudeTemperature varying with time and altitude

XY

TZ

XY

T

Gridded DataGridded Data

Raster

Point Features


Time = 1

Time = 2

Time = 3


141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

Time = 1

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

Time = 2

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Time = 3


141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

TimeTime = 1

Time = 2

Time = 3

Data Cube

Time Slices

Multidimensional DataMultidimensional Data141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411Y

X

Time142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Time = 1

Time = 2

Time = 3


141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

Altitude

Multidimensional Data FormatsMultidimensional Data Formats

• NetCDF (network Common Data Form) NetCDF (network Common Data Form) • HDF (4.x and previous releases), HDF-EOS, HDF5 HDF (4.x and previous releases), HDF-EOS, HDF5

(Hierarchical Data Format)(Hierarchical Data Format)• GRIB, GRIB II (GRIdded Binary)GRIB, GRIB II (GRIdded Binary)

What is NetCDF?What is NetCDF?NetCDF NetCDF ((netnetwork work CCommon ommon DData ata FForm) orm)

A platform independent format for representing multi-A platform independent format for representing multi-dimensional array-orientated scientific data. dimensional array-orientated scientific data.

Self Describing - Self Describing - a netCDF file includes information about the a netCDF file includes information about the data it contains.data it contains.

Direct Access - Direct Access - a small subset of a large dataset may be a small subset of a large dataset may be accessed efficiently, without first reading through all the accessed efficiently, without first reading through all the preceding data.preceding data.

Sharable - Sharable - one writer and multiple readers may simultaneously one writer and multiple readers may simultaneously access the same netCDF file.access the same netCDF file.

NetCDF is new to the GIS community but widely used by NetCDF is new to the GIS community but widely used by scientific communities for around many yearsscientific communities for around many years

Why NetCDF?Why NetCDF?Most commonly used format in the oceanographic and Most commonly used format in the oceanographic and atmospheric science for observational data and numerical atmospheric science for observational data and numerical modelingmodeling

• The National Center for Atmospheric Research (NCAR)The National Center for Atmospheric Research (NCAR)• University Corporation for Atmospheric Research (UCAR) University Corporation for Atmospheric Research (UCAR)

NOAA's Climate Diagnostics Center (CDC) NOAA's Climate Diagnostics Center (CDC) • Los Alamos National Laboratory (LANL)Los Alamos National Laboratory (LANL)• The National Center for Supercomputing Applications The National Center for Supercomputing Applications • US Air Force and NavyUS Air Force and Navy• Atmospheric Research in AustraliaAtmospheric Research in Australia• Australia Defense Australia Defense • UK Hydrographic OfficeUK Hydrographic Office• NATONATO• ......

What is a NetCDF file?What is a NetCDF file?NetCDF is a binary fileNetCDF is a binary file

A NetCDF file consists of:A NetCDF file consists of:Global Attributes:Global Attributes: Describe the contents of the fileDescribe the contents of the fileDimensions:Dimensions: Define the structure of the data Define the structure of the data

(e.g Time, Depth, Latitude, Longitude)(e.g Time, Depth, Latitude, Longitude)Variables:Variables: Holds the data in arrays shaped Holds the data in arrays shaped

by Dimensionsby DimensionsVariable Attributes:Variable Attributes: Describes the contents of Describes the contents of

each variableeach variableCDL (network CDL (network CCommon ommon DData form ata form LLanguage) description takes the anguage) description takes the

following formfollowing formnetCDF name { netCDF name {

dimensions: ... dimensions: ... variables: ... variables: ... data: ... data: ...

} }

Storing Data in a netCDF FileStoring Data in a netCDF File

141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

netcdf mynetcdf{dimensions:

X=4;Y=4;Time=UNLIMITED;

variables:float X(X);float Y(Y);int Time(Time);float Temperature(Time, Y, X);

data:X = 10, 20, 30, 40;Y = 110, 120, 130, 140;Time = 31, 59, 90;

}

Storing Data in a netCDF FileStoring Data in a netCDF File

141 241 341

131 231 331

121 221 321

441

431

421

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

netcdf mynetcdf{dimensions:



data:X = 10, 20, 30, 40;Y = 110, 120, 130, 140;Time = 31, 59, 90;

Temperature =111,211,311,411;

}

Time = 1

Y = 1

X = 1 to 4

Storing Data in a netCDF FileStoring Data in a netCDF Filenetcdf mynetcdf{dimensions:



data:X = 10, 20, 30, 40;Y = 110, 120, 130, 140;Time = 31, 59, 90;

Temperature =111,211,311,411,121,221,321,421,131,231,331,431,141,241,341,441;

}

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

Time = 1

Y = 1 to 4

X = 1 to 4

Storing Data in a netCDF FileStoring Data in a netCDF Filenetcdf mynetcdf{dimensions:



data:X = 10, 20, 30, 40;Y = 110, 120, 130, 140;Time = 31, 59, 90;

Temperature =111,211,311,411,121,221,321,421,131,231,331,431,141,241,341,441,112,212,312,412,122,222,322,422,132,232,332,432,142,242,342,442,113,213,313,413,123,223,323,423,133,233,333,433,143,243,343,443;

}

Y

X

Time

Time = 1 to 3

Y = 1 to 4

X = 1 to 4

NetCDF ConventionsNetCDF Conventions• CF Conventions CF Conventions (Recommended, if applicable)(Recommended, if applicable) • COARDS Conventions COARDS Conventions (1995 standard that CF Conventions extends (1995 standard that CF Conventions extends

and generalizes)and generalizes) • GDT Conventions GDT Conventions (1999 standard that CF Conventions extends and (1999 standard that CF Conventions extends and

generalizes)generalizes) • CDC Conventions CDC Conventions (for gridded data, compatible with but more (for gridded data, compatible with but more

restrictive than COARDS)restrictive than COARDS) • NCAR-RAF Conventions for Aircraft Data NCAR-RAF Conventions for Aircraft Data • AMBER Trajectory Conventions for molecular dynamics simulations AMBER Trajectory Conventions for molecular dynamics simulations • NUWG Conventions NUWG Conventions (1992-1995 effort to create some observational (1992-1995 effort to create some observational

data conventions)data conventions) • PMEL-EPIC Conventions PMEL-EPIC Conventions • GDV Conventions GDV Conventions (deprecated)(deprecated)

CF ConventionCF ConventionCClimate and limate and FForecast (CF) Conventionorecast (CF) Conventionhttp://www.unidata.ucar.edu/software/netcdf/docs/conventions.htmlhttp://www.unidata.ucar.edu/software/netcdf/docs/conventions.html

Initially developed forInitially developed for• Climate and forecast dataClimate and forecast data• Atmosphere, surface and ocean model-generated dataAtmosphere, surface and ocean model-generated data• Comparable observational datasetsComparable observational datasets

• The CF conventions generalize and extend the The CF conventions generalize and extend the COARDS (COARDS (CCooperative ooperative OOcean/cean/AAtmosphere tmosphere RResearch esearch DData ata SService) convention.ervice) convention.

• CF is now the most widely used conventions for CF is now the most widely used conventions for geospatial netCDF data. It has the best coordinate geospatial netCDF data. It has the best coordinate system handling.system handling.

Current and Future StandardsCurrent and Future Standards

Fill in details…Fill in details…

Galeon WCSGaleon WCS

OGC testbedOGC testbed

CF v2CF v2

NetCDF Data SourcesNetCDF Data Sources

• Community Climate Systems Model (CCSM) Community Climate Systems Model (CCSM) http://www.ccsm.ucar.edu, https://www.earthsystemgrid.org/http://www.ccsm.ucar.edu, https://www.earthsystemgrid.org/

• The CCSM is fully-coupled, global climate model that provides The CCSM is fully-coupled, global climate model that provides state-of-the-art computer simulations of the Earth's past, state-of-the-art computer simulations of the Earth's past, present, and future climate states. present, and future climate states.

• 100 yrs of climate change forecast data (2000-2099)100 yrs of climate change forecast data (2000-2099)• Control runs (1870-1999) and scenario runsControl runs (1870-1999) and scenario runs• Temperature, precipitation flux, surface snow thickness, snowfall Temperature, precipitation flux, surface snow thickness, snowfall

flux, cloud water content, etc.flux, cloud water content, etc.

• Program for Climate Model Diagnosis and Program for Climate Model Diagnosis and Intercomparison (PCMDI) Intercomparison (PCMDI) http://www-pcmdi.llnl.gov/http://www-pcmdi.llnl.gov/


• Vegetation and Ecosystem Modeling and Analysis Vegetation and Ecosystem Modeling and Analysis Project (VEMAP) Project (VEMAP)

http://dataportal.ucar.edu/vemap/main.htmlhttp://dataportal.ucar.edu/vemap/main.html• VEMAP was a large, collaborative, multi-agency program to VEMAP was a large, collaborative, multi-agency program to

simulate and understand ecosystem dynamics for the simulate and understand ecosystem dynamics for the continental United States. continental United States.

• The VEMAP Data Portal is a central collection of files maintained The VEMAP Data Portal is a central collection of files maintained and serviced by the NCAR Data Groupand serviced by the NCAR Data Group

• Climate data interval: Annual, monthly, and daily.Climate data interval: Annual, monthly, and daily.• Data type: Historical and model resultsData type: Historical and model results• Data: Temperature, irradiance, precipitation, humidity, incident Data: Temperature, irradiance, precipitation, humidity, incident

solar radiation, vapor pressure, elevation, land area, vegetation, solar radiation, vapor pressure, elevation, land area, vegetation, water holding capacity of soil, etc. water holding capacity of soil, etc.


• British Atmospheric Data Center (BADC) British Atmospheric Data Center (BADC)

http://badc.nerc.ac.uk/data/http://badc.nerc.ac.uk/data/• The role of the BADC is to assist UK atmospheric researchers to The role of the BADC is to assist UK atmospheric researchers to

locate, access and interpret atmospheric data.locate, access and interpret atmospheric data.• Many datasets are publicly available but datasets marked with Many datasets are publicly available but datasets marked with

key symbol have restricted access.key symbol have restricted access.• Datasets are organized by projects or organizations.Datasets are organized by projects or organizations.• Climatology Interdisciplinary Data Collection (CIDC) has monthly Climatology Interdisciplinary Data Collection (CIDC) has monthly

means of over 70 Climate Parameters.means of over 70 Climate Parameters.• Met Office - Historical Central England Temperature Data has Met Office - Historical Central England Temperature Data has

the monthly series, which begins in 1659, is the longest available the monthly series, which begins in 1659, is the longest available instrumental record of temperature in the world. The daily series instrumental record of temperature in the world. The daily series begins in 1772. begins in 1772.


• National Oceanic & Atmospheric Administration (NOAA)National Oceanic & Atmospheric Administration (NOAA)• National Digital Forecast Database (NDFD) National Digital Forecast Database (NDFD)

http://www.nws.noaa.gov/ndfd/http://www.nws.noaa.gov/ndfd/• Radar Integrated Display with Geospatial Element (RIDGE) Radar Integrated Display with Geospatial Element (RIDGE)

http://www.srh.weather.gov/ridge/http://www.srh.weather.gov/ridge/• Precipitation Analysis Precipitation Analysis

http://www.srh.noaa.gov/rfcshare/precip_download.phphttp://www.srh.noaa.gov/rfcshare/precip_download.php• Climate Diagnostics Center http://www.cdc.noaa.gov/Climate Diagnostics Center http://www.cdc.noaa.gov/

NetCDF in ArcGISNetCDF in ArcGIS• NetCDF data is accessed asNetCDF data is accessed as

• RasterRaster• FeatureFeature• TableTable

• Direct read (no scratch file)Direct read (no scratch file)• Exports GIS data to netCDFExports GIS data to netCDF

NetCDF ToolsNetCDF Tools

Toolbox: Multidimension ToolsToolbox: Multidimension Tools• Make NetCDF Raster LayerMake NetCDF Raster Layer• Make NetCDF Feature LayerMake NetCDF Feature Layer• Make NetCDF Table ViewMake NetCDF Table View• Raster to NetCDFRaster to NetCDF• Feature to NetCDFFeature to NetCDF• Table to NetCDFTable to NetCDF• Select by DimensionSelect by Dimension

Using NetCDF DataUsing NetCDF DataBehaves the same as any layer or table Behaves the same as any layer or table DisplayDisplay

• Same display tools for raster and feature layers will work on Same display tools for raster and feature layers will work on netCDF raster and netCDF feature layers.netCDF raster and netCDF feature layers.

• GraphingGraphing• Driven by the table just like any other chart.Driven by the table just like any other chart.

• AnimationAnimation• Multidimensional data can be animated through a dimension Multidimensional data can be animated through a dimension

(e.g. time, pressure, elevation)(e.g. time, pressure, elevation)

• Analysis ToolsAnalysis Tools• A netCDF layer or table will work just like any other raster layer, A netCDF layer or table will work just like any other raster layer,

feature layer, or table. (e.g. create buffers around netCDF feature layer, or table. (e.g. create buffers around netCDF points, reproject rasters, query tables, etc.)points, reproject rasters, query tables, etc.)

NetCDF Layer/Table PropertiesNetCDF Layer/Table Properties

Raster

Feature

Table

Making a NetCDF Raster LayerMaking a NetCDF Raster Layer

Usage: MakeNetCDFRasterLayer <in_netCDF_file> <variable> Usage: MakeNetCDFRasterLayer <in_netCDF_file> <variable> <x_dimension> <y_dimension> <out_raster_layer> {band_dimension} <x_dimension> <y_dimension> <out_raster_layer> {band_dimension} {Dimension {Value};Dimension {Value}...} {BY_VALUE | BY_INDEX}{Dimension {Value};Dimension {Value}...} {BY_VALUE | BY_INDEX}

• Drag and dropDrag and drop• Use the toolUse the tool

Changing Time SliceChanging Time Slice

Time = 1

(Selecting a different “dimension value”)

141 241 341

131 231 331

121 221 321

111 211 311

441

431

421

411

142 242 342

132 232 332

122 222 322

112 212 312

442

432

422

412

143 243 343

133 233 333

123 223 323

113 213 313

443

433

423

413

Y

X

Time

NetCDF and Spatial ReferenceNetCDF and Spatial Reference• Geographic Coordinate Systems (GCS)Geographic Coordinate Systems (GCS)

• X dimension units: X dimension units: degrees_eastdegrees_east• Y dimension units: Y dimension units: degrees_northdegrees_north

• Projected Coordinate Systems (PCS)Projected Coordinate Systems (PCS)• X dimension standard_name: X dimension standard_name: projection_x_coordinateprojection_x_coordinate• Y dimension standard_name: Y dimension standard_name: projection_y_coordinateprojection_y_coordinate• Variable has a Variable has a grid_mappinggrid_mapping attribute. attribute. • CF conventions currently supports only eight predefined CF conventions currently supports only eight predefined

coordinate systems coordinate systems

• UndefinedUndefined• If not GCS or PCSIf not GCS or PCS• ArcGIS writes (and recognizes) PE String as a variable attribute.ArcGIS writes (and recognizes) PE String as a variable attribute.

multidimensional data and gis steve kopp nawajish noman esri

Documents

x data

netcdf data sourcesworking

scientific data

observational data

data e

preceding data

netcdf filetime

yearswhy netcdf