advanced custom raster processing using python · • working with raster data using python •...
TRANSCRIPT
Jie Zhang, Hao Hu
Advance Custom Raster Processing Using Python
• Introducing Python API in ArcGIS for Raster Processing and Analysis- ArcGIS API vs ArcPy- Python Raster Function
• Customize Raster Processing/Analysis with ArcPy- Raster Object- Write custom Pixel Block operation- Write custom Cell operation
• Python Raster Function- Anatomy/API- How to write/use/deploy
Agenda
Use Python API for Raster Processing/Analysis
ArcG
IS A
PI
ArcP
y
• ArcGIS API- Work with ArcGIS
Enterprise/ArcGIS Online- Remote processing
• ArcPy- Requires Desktop App
(ArcGIS Pro/ArcMap) - Or Available in Hosted
Notebook Server Advanced Runtime
- Local processing
https://developers.arcgis.com/python/
https://pro.arcgis.com/en/pro-app/arcpy
Python Raster Function
https://github.com/Esri/raster-functions
• Not Part of ArcPy or ArcGIS API for Python• A Python Script that defines a Custom Raster Function
Class• Allows ArcGIS to utilize Custom Raster Analysis through a
python adapter• Available support on Desktop app and Server• More details to follow…
Python Raster Function Script
Python runtime
Server
Desktop
Jie Zhang
ArcPy for Raster Processing and Analysis
ArcPy API for Raster Data Processing and Analysis
* Some functions and classes are available both under Image Analyst and Spatial Analyst https://pro.arcgis.com/en/pro-app/arcpy
ArcPy
RasterRasterInfo
Class
ListRastersNumpyArrayToRasterRasterToNumpyArrayDescribe- Raster Dataset - Raster Catalog- Raster Band
Functions
ClassPixelBlockPixelBlockCollectionRasterCollection
Function (70 Analytic Functions)AggregateAnomaly etc.Operators(21 Operators)ArithmeticBitwiseBooleanRelational
Image AnalystClassRasterCellIterator11 Parameter Classes
Function (59 Analytic Functions)AggregateAnomaly etc.Operators(21 Operators)ArithmeticBitwiseBooleanRelational
Spatial Analyst
Over 90 Image Analyst ToolsOver 200 Spatial Analyst Tools
(Geoprocessing tool is also accessible through arcpy API)
Geoprocessing Tools
Raster Object and Image Analyst Extension
https://pro.arcgis.com/en/pro-app/arcpy/image-analyst
• Raster- Use for Single Raster Dataset analysis- Enhanced to support Multidimensional Raster Dataset- Query metadata information with RasterInfo (e.g. band
count)
• RasterCollection- Use for Grouped Raster Analysis (e.g. sum)
• Function- Ready to use built-in functions for Data Processing
and Analysis
RasterRasterInfo
ClassClassPixelBlockPixelBlockCollectionRasterCollection
Function (70 Analytic Functions)AggregateAnomaly etc.Operators(21 Operators)ArithmeticBitwiseBooleanRelational
Image Analyst
Pixel Block and Pixel Block Collection
• Use to implement custom analysis algorithm on Raster Data• PixelBlockCollection
- Iterator of PixelBlock objects in a Raster or a List of Raster- Use stride to create overlap between adjacent PixelBlocks
• PixelBlock- Block of pixels read from a Raster with defined size- Also support Multidimensional block
Jie Zhang
Species Distribution Model
Hao HuRaster Cell Iterator
Raster Cell Iterator (RCI)• Visit each cell in a Raster object by row/column
indices and read/write cell values
• Support iterating multiple raster datasets and perform analysis at the same location
• Access to cell values at each location and neighboring locations is efficient
• Available through the Spatial Analyst module
arcpy.sa.RasterCellIterator(*args),
*args is a dictionary that defines rasters and padding in the following format:
{'rasters': [in_rasterobj1, in_rasterobj2, ...], 'padding': padding_factor}
0 1 2 3 4
14 23 32 42 50
14 24 33 43 51
14 23 32 42 51
14 23 32 43 51
14 23 30 39 48
0 1 2 3 4
General steps to use RCI
• Step 1. Create an empty output raster with RasterInfoobject
- RasterInfo contains properties information of a raster- Generated from an existing raster or construct from scratch
# Create an empty RasterInfo object myRasterInfo = RasterInfo() # Load raster info from a Python dictionary
myRasterInfoData = { 'bandCount': 1, 'extent': {'xmin': -107.0, 'ymin': 38.0, 'xmax': -104.0,
'ymax': 40.0, 'spatialReference': {'wkid': 4326}}, 'pixelSizeX': 0.01, 'pixelSizeY': 0.01, 'pixelType': 'U8',
}
# Load myRasterInfoData JSON string into myRasterInfomyRasterInfo.fromJSONString(json.dumps(myRasterInfoData))
# Create a new Raster object based on myRasterInfomyRaster = Raster(myRasterInfo)
# Read from a raster datasetlandu = Raster(r"C:\sample_data\landuse.tif")
# Get the raster info by calling getRasterInfo methodraster_info = landu.getRasterInfo()
# Create a temporary output raster based on the raster info of landuseoutput = Raster(raster_info)
Create RasterInfo from an existing raster
Create RasterInfo from scratch
General steps to use RCI
• Step 1. Create an empty output raster with RasterInfoobject
- RasterInfo contains properties information of a raster- Generated from an existing raster or construct from scratch
• Step 2. Define a Raster Cell Iterator to iterate over raster dataset(s)
Create RasterInfo from an existing raster
# Create a Raster Cell Iterator objectrci = RasterCellIterator({
'rasters':[input_ras, output_ras], 'padding’:1
})
General steps to use RCI
• Step 1. Create an empty output raster with RasterInfoobject
- RasterInfo contains properties information of a raster- Generated from an existing raster or construct from scratch
• Step 2. Define a Raster Cell Iterator to iterate over raster dataset(s)
• Step 3. Loop through the cells of raster(s) to perform read/write operations
- Access cell value of multiband raster- Access neighboring values- Implicit and explicit use of RCI
Create RasterInfo from scratch
# Persist output raster to diskoutput_ras.save(r"C:\output\out_ras.tif")
# Access cell value in multi-band rasterfor r, c in rci:
output_ras[r, c] = (input_ras[0, r, c] + \input_ras[1, r, c]) / 2
# Access neighboring valuesfor r, c in rci:
value_sum = 0for x in [-1,0,1]:
for y in [-1,0,1]: value_sum += input_ras[r+x, c+y]
output_ras[r, c] = value_sum / 9
# Implicitly loop through cellsfor r, c in input_ras:
input_ras[r, c] = r + c
General steps to use RCI
• Step 1. Create an empty output raster with RasterInfoobject
- RasterInfo contains properties information of a raster- Generated from an existing raster or construct from scratch
• Step 2. Define a Raster Cell Iterator to iterate over raster dataset(s)
• Step 3. Loop through the cells of raster(s) to perform read/write operations
- Access cell value of multiband raster- Access neighboring values- Implicit and explicit use of RCI
• Step 4. Save the analysis output # Persist output raster to diskoutput_ras.save(r"C:\output\out_ras.tif")
• Create a new raster with a specific pattern
Example applications with RCI
Create a heterogenous surface raster represented by a checkerboard pattern
• Custom Focal Operations
2 2 1
3 5 4
4 2 2
8
2 5 1
3 5 4
4 2 5
6
Input Output
Counting the number of neighboring cells that have a value different to the center cell
How RCI visit each cell in a Raster?
PixelBlock
128
128
……………………
……………………
0 1 2 3 4 5 6 7 … 126 127
0 1 2 3 4 5 6 7 …126 127
1 2 34 56 7 8
130
130
Padding
Hao Hu
Custom Raster Analysis with Raster Cell Iterator
Jie ZhangPython Raster Function
What’s a Python Raster Function?
• Transforming Rasters - image processing and analytic algorithms - in Python.
• Implement a raster function from the comfort of your Python module.
• Architecture: Module loaded by an adapter - Python-aware and a first-class participant in the function chain.
What’s a Python Raster Function?
• Extend Raster Model
Raster Dataset
Mosaic Dataset Item
Image Service
Mosaic Dataset
Hello, World!
The API
• customize our function object—a specific instance of our class—as soon as it's created.
• Define raster function name & description.
• __init__
The API
• Define all input parameters to the function.
• For each parameter, define:- Name (Identifier) - Display Name- Long Description- Data Type- Default Value- Required vs Optional
• getParameterInfo()
The API
• How are input rasters read—Padding, Mask, …?
• How’s the output raster constructed—inherit NoData, Metadata, …?
• Given: Nothing.
• Returns: dictionary containing configuration attribute values.
• getConfiguration()
The API
• Define a subset of input rasters.
• Pixels read from selected rasters.
• Given: properties of the requested pixel block, all scalar parameter values.
• Returns: names of selected rasters.
• selectRasters()
The API
• Defines the output raster.
• Invoked each time a dataset containing the Python raster function is initialized.
• Given: Raster info associated with all input rasters.
• Returns: Raster Info of the output raster.
• updateRasterInfo()
The API
• Workhorse of the raster function. Process Pixels.
• Given: • Expected pixel-block size+location• output raster properties (map space)• pixels+mask of selected input rasters
• Returns: Pixels+mask of requested pixel block.
• updatePixels()
The API
• Create or update dataset- or band-level metadata.
• Given:
• property names• band index• current key metadata values
• Returns: updated values of given properties
• updateKeyMetadata()
The API
• Given:• info on parent product, • context of execution.
• Returns:• OK to Run (Boolean)—Licensed to execute or not? • Expected product level and extension.
• isLicensed()
An Python Raster Function Example
Compound Topographic Index
Compound Topographic Index (CTI)• A steady state wetness index, a.k.a. Topographic Wetness index (TWI)
T.R. Matthews et al. 2015
• Used to study spatial scale effects on hydrological processes and to identify hydrological flow paths for geochemical modelling.
• Used to characterize biological processes such as annual production, vegetation patterns, and forest site quality.
Compound Topographic Index (CTI)
• Calculations
Compound Topographic Index
𝑪𝑪𝑪𝑪𝑪𝑪 = 𝐥𝐥𝐥𝐥 𝒇𝒇𝒇𝒇𝒇𝒇𝒘𝒘𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒂𝒇𝒇𝒂𝒂𝒂𝒂𝒂𝒂𝒇𝒇𝒂𝒂 + 𝟏𝟏 ∗𝒂𝒂𝒄𝒄𝒇𝒇𝒇𝒇𝒄𝒄𝒂𝒂𝒄𝒄𝒄𝒄
𝐭𝐭𝐭𝐭𝐥𝐥 𝒄𝒄𝒇𝒇𝒇𝒇𝒔𝒔𝒄𝒄
• Calculations
Compound Topographic Index
Slope
Flow Accumulation
CTI
CTI Function
CTI: Code
Jie Zhang
CTI Raster Function
Raster Function Template
Raster Function Templates - Basic workflows
• Create new function templates- Via Raster Function Template Editor- Layer > Symbology > Export As Raster Function Template- Function Raster Dataset > Functions > Save as- Symbology as a template
• Raster Layer in a Map• Image Analysis Window• Raster Functions Pane in Pro• Layer > Properties > Functions tab
Raster Function Templates – Advance Workflow
• On a Mosaic Dataset- Populating a mosaic using the Add Rasters tool- Mosaic dataset items
- Batch Edit Raster Functions or - Edit Raster Function Geoprocessing Tool.
- As Processing Templates
• On an Image Service—for server-side processing
• Learn more at https://github.com/Esri/raster-functions#raster-function-templates.
Jie Zhang
Apply Raster Function Template
Additional Consideration
Performance
• Do not use ArcPy module or array looping in python raster function code
• Use NumPy and SciPy.
• No significant improvement performance when using Cython.
• Leverage well-known options to optimize time-critical parts of your raster function.
Publishing & Deployment
• Python version• 3.6.9 - ArcGIS Pro 2.5/ArcGIS Server 10.8
• Manual Deployment of Python Script or binary to “trusted” Location- <ArcGIS Pro Install Location>\Pro\Resources\Raster\Functions- <ArcGIS Server Install
Location>\Server\framework\runtime\ArcGIS\Resources\Raster\Functions- Server deployment has to be on every server node
• Distributed Raster Analytics Support Python Raster Function- Learn more about Distributed Raster Analytics
https://doc.arcgis.com/en/imagery/workflows/resources/distributed-processing-with-raster-analytics.htm
GitHub/Wiki
• Where do functions and templates live? How to learn?- https://github.com/Esri/raster-functions- https://github.com/Esri/raster-functions/wiki
• Go ahead:- Browse samples to learn more. - Fork the repo. Experiment.- Log defects or enhancement requests as issues. - Send pull requests to contribute code.
PixelBlock vs RasterCellIterator vs Python Raster Function
• PixelBlock- Pixel Block level analysis- One time analysis- Available from arcpy
Image Analyst- No deployment required- Local processing
• RasterCellIterator- Raster Cell analysis- One time analysis- Available from arcpy
Spatial Analyst- No deployment required- Local processing
• Python Raster Function - Pixel Block level analysis- Reusable through raster
function template- Customizable license
restriction- Requires deployment- Support Local/Remote
processing, scalable- Support on-the-fly
visualization
• Working with Raster Data Using Python• Managing and Sharing Raster Data for Analysis• ArcGIS API for Python: Imagery and raster analysis on your Web GIS• Advanced Analysis of Satellite Imagery using Python
Related Technical Workshops
• Species Distribution Modelhttps://github.com/rsjiezhang/imagery-demo/tree/master/devsummit2020/AdvancedRasterProcessing/SpeciesDistributionModel
• Compound Topographic Indexhttps://github.com/Esri/raster-functions/blob/master/functions/CompoundTopographicIndex.py
• Raster Cell Iteratorhttps://github.com/hao9717/DevSummit20/blob/master/PythonCustomAnalysis/Notebook/RCI.ipynb
Demo Source code
Thank you!