lidar data products in indiana ispls workshop january 18 th, 2013 christopher morse nrcs indiana gis...

LiDAR Data Products in Indiana

ISPLS WorkshopJanuary 18th, 2013Christopher Morse

NRCS Indiana GIS CoordinatorSpecial thanks to:

Jim SparksPhil WorrallR. Wilkinson

Helping People Help the LandUSDA is an equal opportunity provider and

employer.

Data Overview and Access

Digital Elevation Data

A Short ReviewRaster data (evenly spaced, gridded data)Cells hold values for the height of a feature or site

referenced to a common vertical datumResolution refers to the size of the pixels in the data

A DEM with 30 meter resolution is composed of all cells being 30 meters x 30 meters in the x and y directions and each cell holds a single elevation value (z)

Elevation value (z) could be stored in a variety of units


employer.

Digital Elevation DataDefinitions paraphrased from Maune et al, 2nd edition DEM Users Manual(Terms often used interchangeably)

DEM = Digital Elevation Model; Typically Bare Earth or terrain DTM = Digital Terrain Model; Similar to DEM with the addition of some

elevations for significant topographic features on the land defined by mass points or break lines

DSM = Digital Surface Model; Similar to a DEM or DTM, but shows the tops of all surfaces including buildings, trees, and other features above the bare earth


employer.

Image from State of Indiana Orthophotography and LiDAR Program, presentation by R.N. Wilkinson


Definitions paraphrased from Maune et al, 2nd edition DEM Users Manual:

Mass Points are irregularly spaced vector points with an x,y, and z value.

Breaklines are linear features that describe changes in the terrain surface (roads, streams, building footprints, etc…)


employer.


Common Resolutions, locally (not all resolutions are available in all areas)USGS primarily 30 meter, 10 meter; some 3 meter


employer.


Common Resolutions, locally (not all resolutions are available in all areas)Indiana has 2005 elevation data at 5 foot resolution for all parts of the stateIndiana is generating new DTM data for 2011-2013 at 5 foot resolutionLocal governments may hold a variety of high resolution products


employer.


Why generate new DEMs at the same resolution?

FundsDramatically Improved Vertical Accuracy

USGS National Elevation Data (2003) = 2.44 meters RMSE

Indiana 2005 Data = 6 feet RMSEIndiana 2011-2013 Data = 18.5 cm RMSE

Vertical Accuracy is a critical factor when considering best supportable contour interval


employer.


What does that mean?Best supportable auto-generated contours:

USGS National Elevation Data = over 20 foot intervalIndiana 2005 Data = 20 foot intervalIndiana 2011-2013 Data = 2 foot interval

Auto generated results limited without use of breaklines


employer.

Image source: wikipedia.org/wiki/Contour_line


Indiana’s new DEMsDerived from LiDARActually a DTM due to the inclusion of breaklines for

some hydro featuresRivers greater than 100 feet wide and water bodies of 2

acres or greater digitized from accompanying imageryRivers digitized in direction of flow and water bodies with a

set elevation (water bodies will be “flat” and rivers will flow “downhill”)

No LiDAR points used within 1.5 meters of a digitized breakline

State Plane Coordinates (NAD 83 feet, NAVD 88)


Indiana’s new DEMs5 foot horizontal pixel resolutionSupports 2 foot contours (thus at least 18.5 cm

vertical RMSE accuracy met)Part of the IndianaMap

http://www.indianamap.orgDEMs Available for download from:

Indiana Spatial Data Portal: http://gis.iu.edu/


Resources on Digital Elevation DataUSGS National Elevation Dataset (NED) Info:http://ned.usgs.gov/about.asp

Text:Digital Elevation Model Technologies and Applications: The DEM Users Manual, 2nd Edition

Edited by David F. Maune, PhD, CPPublished by ASPRS, 2007


employer.

LiDAR Data

LiDAR = Light Detection And RangingUses an active sensor to emit energy (light)

and detect returned energyCan be collected day or night)

Image from State of Indiana Orthophotography and LiDAR Program, presentation by R.N. Wilkinson

LiDAR Data

Airborne and Terrestrial capabilitiesCombines GPS and an Inertial Measurement

device to compute x,y,z positionsEvery point recorded has an x,y,z, and

intensity value

LiDAR Data All reflections of emitted energy are returned,

generating a point cloud of the data The point cloud contains data points for scan hits at

multiple heights on objects, as well as some noise due to atmospheric conditions.

These hits are referred to as returns and are referenced in ascending order from highest elevation to lowest elevation for a set of returnsTop of a building or tree is the 1st returnCanopy of a tree or side of a building is 2nd or 3rd return,

and so on as the returned hits descend in elevation

LiDAR Data

All returns

1st return

2nd return

3rd return

4th return

Image from Lidar Technology Overview, presentation by USGS, June 2007

LiDAR Data

The vendor uses classification algorithms on the data

Vendor delivers a data product depending on the customer’s specifications

LiDAR Data

Indiana’s LiDAR Data (2011-2013)

Classified Point Data1.5 meter Nominal Pulse Spacing (the estimated average spacing of irregularly-spaced points in both the along-track and cross-track directions – FEMA)LAS files in 5000’ x 5000’ tilesData Delivered in appropriate State Plane Coordinate System (NAD 83 Feet, NAVD88)

LiDAR Data

Indiana’s LiDAR Data (2011-2013)Classification Scheme(This is not the same as 1st return, 2nd return)1 = Processed but Unclassified2 = Bare Earth/Ground7 = Noise9 = Water10 = Ignored Ground (breakline proximity)13 = Bridges (over 100 feet in length; foot bridges not included)

LiDAR Data

Some LiDAR ResourcesUSGS LiDAR Guidelines (replaces draft versions 13 and 14)http://pubs.usgs.gov/tm/11b4/

NDEP Guidelines for Digital Elevation Data, Version 1.0 (2004)(Currently in work for an updated release)http://www.ndep.gov/TechSubComm.html

Education (most class materials available freely)https://www.e-education.psu.edu/lidar/resources/l1.html

LiDAR DataLiDAR’s Limitations (in designs)Site changes – snapshot on day of flightGrade breaks – collection pattern is “random” and not based on changes in grade as a field surveyCritical elevations – may not detect control elevations such as building floor elevations, edges of concrete, property boundaries or culvert inlet/outlet elevations (requires local benchmarking at site and adjustment of data to benchmark)Vegetation – May affect readings, dependent on quality of the data, density of vegetation. Tillage may affect surface smoothness (can affect slope calculations)Water – LiDAR can penetrate water, but type of laser and water turbidity can affect this. Standing water can invalidate a local elevation estimate from LiDAR. If you believe a data result is due to influence of water, don’t use it for an elevation

LiDAR Data

LiDAR’s Uses (in designs)Planning – Visualization of data and its derivaties (hillshades and contours) can be very useful in planningPreliminary Design – LiDAR relative accuracy is typically very high for a site, so preliminary design for a number of uses can be done with CAD generated surfaces and later tied to a site’s elevations through adjustment to benchmarks obtained in a field survey (if the site has not undergone major change since the collect)

Indiana Data (2011-2013)

Acquiring Indiana LiDAR Data• IndianaMap & Indiana Spatial Data Portal

(www.IndianaMap.org) View and File-based access to point cloud and hydro-flattened DEM data

• Open Topography Server (UCSD)

• Key advantagesUser Defined Area of InterestMitigates need for local storage of unneeded dataOpens the door to Indiana data for all usersLeverage server side processing for extraction and

derivativesHelping People Help the Land

USDA is an equal opportunity provider and employer.

OT Links

• Open Topography Home Page:http://www.opentopography.org

Open Topography Data Page:http://opentopo.sdsc.edu/gridsphere/gridsphere?

cid=datasets

• Indiana’s LiDAR Data Home Page:*http://igic.org/projects/lidar/index.html

*Recommend you use this home page – Indiana news, tips, tricks, documentation, and instructions will be poster here.


employer.

Select a Region


employer.

Results


employer.

More


employer.

Get Data (Top)


employer.

Get Data (Mid)


employer.

Get Data (Bottom)


employer.

Choices

That’s a lot of boxes!

How do I get what I need?


employer.

Choices

• Understand what data you really needWhat task are you trying to accomplish?What data do you really need for that task?Decisions will depend on uses for LiDAR in which

you are engaged


employer.

Choices

• Understand what data you really needDo you need LiDAR points (LAS files) for particular

areas?Get it here

Do you need the bare earth points or the entire point cloud?Decision driven by your intended use

Do you need a DEM, TIN or derivatives?You may not want to get it here (yet) unless you need a

custom DEM or cannot process TINs or generate derivatives in your own applications


employer.

Ground Return (LAS, DEM, No TIN) Example

• Section 1 - BasicsSelect Area


employer.


• Section 1 - BasicsReview number of points and modify extent if

necessarySet Ground Classification

This choice drives the nature of the derived DEM/TIN laterSelect Coordinate System

Note that this choice will affect your units in later stepsHelping People Help the Land

USDA is an equal opportunity provider and employer.


• Section 2 – Point Data FormatPreference and capabilitiesLAS = Larger d/l, no decompression neededLAZ = smaller d/l, must decompressASCII = largest d/l, no decompression needed


employer.


• Section 3 – DEM GenerationGridding Parameters

Remember: Units match projection choice from step #1c.Resolution: At least the point spacing of the dataset (1.5 m

or about 5 feet)Radius: At least the resolution of the dataset. The larger the

radius, the more “smoothed” the DEM. Using 2x the resolution guarantees a “3-cell filter”


employer.


• Section 3 – DEM Generation (cont’d)Methods


employer.


• Section 3 – DEM Generation (cont’d)

Zmean Grid – A basic averaging methodCan be used when making any surface to average out data

irregularities that would be emphasized by Zmin or ZmaxIf you chose all points in step #1b, but you still want a DEM

of the “bare earth”, you can select Zmin instead, but it will have different results than Zmean. The differences may or may not be significant to your work, but only you can determine that.

Not clear if this implements any nearest neighbor weightings, but from the description it seems not.


employer.


• Section 3 – DEM Generation (cont’d)

Zidw Grid IDW = Inverse Distance Weighting – explicitly implements an

assumption that things that are close to one another are more alike than those that are farther apart. Is also an averaging method.

Resulting surface will not pass through the sample points.No option to control the power factor (possibly default of 2) IDW in general is not recommended for gridding Terrains per

Maune et al, Digital Elevation Models Technologies and Applications: The DEM Users Manual, 2nd Edition; page 10. It is not clear if the particular IDW algorithm and settings at Open Topography would contradict that.


employer.


• Section 3 – DEM Generation (cont’d) The Zmean to Zidw difference (2ft contours from 5 ft derived DEM,

smoothed with PAEK, 50 ft)


employer.


• Section 3 – DEM Generation (cont’d)Which method should I use?It depends on your needs

Zmin can give bare earth values when using ground returns or entire point cloud, but will always assign the lowest value in the search radius

Zmax can give a first return surface when using all returnsZmean is a basic average of points to simulate bare earth

when using ground returns (can be a very reasonable DEM) or of entire surface when using all returns

Zidw is like Zmean but more specialized to weight points that are nearer – may result in some sharp exaggerations or newly introduced inaccuracy. Typically not the best for gridding terrain data.


employer.


• Section 3 – DEM Generation (cont’d)Which method should I use?The good news

It’s multiple choice! Try them all at once.Alternately, you can generate your own DEM from the LiDAR

points you download using a variety of tools of your own and you do not need to generate a DEM here at all (however, this example is for a DEM, so this is just a reminder that this step is optional and specialized).

You could also download the final project DEMs from ISDP, and in the future, OT, without processing here at all and you can resample from those.


employer.


• Section 3 – DEM Generation (cont’d)Formats – ArcASCII Grid, GeoTIFF, IMG, or All


employer.


• Section 3 – DEM Generation (cont’d)Null Filling

Will fill in small blank areas in the DEM being generated at 3, 5, or 7 pixel filter sizes

Your choice depends on your project needs, but setting a value here will minimize tiny holes of no data in the resulting surface

Will not fix large holes from water bodies or buildings


employer.


• Section 4 – Derivative ProductsHillshade will use altitude of 45 degrees and

azimuth of 315 degreesSlope will be degrees, not percent


employer.


• Section 5 – Visualization ProductsOptionalCan generate files for use in Google EarthNot used in this example (will uncheck)


employer.


• Runtime


employer.


• Results


employer.


• ResultsData can be added straight to ArcMap, however

statistics will not be calculated by defaultYou can use tools in ArcGIS to calculate statisticsAlternately, you can force the statistics to

calculate under the Symbology tab by switching back and forth from None to Standard Deviations under the Stretch method drop down box


employer.


• Results (Zmean, elevation, hillshade)


employer.


• Results (Zidw, elevation, hillshade)


employer.

Limitations

• Area of interest scope is limited to 50 million points if you are not logged into OT

• You can increase this to 150 million points when logged into OT

• Point density and natural geography will cause the approximate area corresponding to that number of points to fluctuate

• Ground vs. All will return different point count


employer.

Limitations

• Working with LAS DataTypically requires a tool that supports LAS formatAlternately can convert to a 3d x,y,z format

• Raw LiDAR data can have unclassified points and gaps in coverage depending on the return or classification selected


employer.

Limitations

• Derivative products can have similar gaps, such as the DEM or hillshade generated at OT

• Break lines are not used to generate data from OT• This is due to the file being generated from ONLY

the LiDAR points by the OT site• DEMs Delivered from the vendor have been post

processed to fill in such areas and to use some breaklines (will be added to OT, currently hosted at ISDP)


employer.

Some Uses for OT Generated DEM

• Analysis in open terrain with few to no structures or water bodies

• Can compensate for gaps from buildings and water bodies in the OT generated DEM using some techniques in ArcGIS Spatial Analyst to further fill holes in the surface


employer.

Fill the blanks in the DEMRaster calculator (re-run until no NoData returns)

Arc10:Con(IsNull(Raster), FocalStatistics(Raster, NbrCircle(10, "CELL"), "MEAN"), Raster)

Where Raster is the name of the DEM layer, Circle is the search type, 10 is the radius in cells. These criteria can be modified.

Arc 9:Con(IsNull(Raster), FocalMean(Raster, rectangle,10,10), Raster)

Where Raster is the name of the DEM layer, rectangle is the search type, 10x10 is the rectangle size (in cells). These criteria can be modified.


employer.

Fill the blanks in the DEM

Model Builder (Arc 10)Same expression, constructed in model builder


employer.

Fill the blanks in the DEMPython (with Arc 10)

Add a while loop to continue processing as long as IsNull generates any True (1) results


employer.

Recommended Settings

• To get a “bare earth” dataset at Open TopographyReturn Classification = GroundCoordinate System = User preferencePoint Format = User PreferenceDEM Generation (optional)

Method = Zmean gridResolution = at least 1.5 m (UTM) or 5 ft (St. Plane)Radius = at least 3x the resolutionNull Filling = 7 (smooth out small gaps)Should produce essentially a bare earth DEM, gaps for water

and buildings will be present


employer.


• Example of a Zmean surface (no exaggeration)


employer.

Recommended Settings• To get a “1st return” dataset at Open Topography

Return Classification = AllCoordinate System = User preferencePoint Format = User Preference“DEM” Generation (optional)

Method = Zmax gridResolution = at least 1.5 m (UTM) or 5 ft (St. Plane)Radius = same as resolution (minimize radius to decrease

blending of vertical features)Null Filling = 7 (smooth out small gaps)Should produce a surface approximating a DSM, but will lack

true vertical definition of features, gaps for water may be present, may be affected by noise in the data


employer.


• Example of a Zmax surface


employer.

Questions?

Chris MorseUSDA-NRCS

NRCS Indiana GIS Coordinator317-295-5849

[email protected]


employer.

lidar data products in indiana ispls workshop january 18 th, 2013 christopher morse nrcs indiana gis...

Documents

data overview

digital elevation model

new dtm data

lidar data products

gridded data cells

land usda

equal opportunity provider

foot interval indiana