catena distributed generic processing chain for optical satellite imagery processing peter reinartz,...
TRANSCRIPT
CATENADistributed Generic Processing Chain for Optical Satellite Imagery Processing
Peter Reinartz, Thomas KraußRemote Sensing Technology InstitutePhotogrammetry and Image Analysis
ESA Workshop on Models forScientific Exploitation of EO data
Frascati, 2012-10-11
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 2
Why processing chains for higher level optical data processing?
• Needed for:• Processing of large areas and large data volumes
e.g. Image2006/2009/2012, each time about 3500 scenesIRS/SPOT for whole Europe-38
• Processing of time seriese.g. CCI-Fire, Meris/ATSR/SPOT-VGTfor 1995-2009, about 130.000 scenes
• Requirements:• Fully automatic processing of• Mass data from• Many optical sensors/satellites• Modular and easy re-configurable
for many projectsImage 2006, ~3500IRS/SPOT scenes
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 3
ImageMatching
Sensor ModelRefinement
Ortho-rectification
Atmosphericcorrection
Automatic and operational processing chain for processing of mass data
Using global databases and reference data
Support of native satellite image formats from SPOT4/5, IRS-P6 Liss3/AWiFS, ALOS AVNIR/PRISM, Ikonos, Quickbird, RapidEye, WorldView, GeoEye, Cartosat, Pleiades, Meris, ATSR, VGT, Modis, …
Thematicprocessing
input output
extract ground controlpoints from globalReference databases
perform parameterestimation
Use globalDEM database
removeatmosphericinfluence
Reference image
Original image
Ortho image
DEM
CATENACATENACATENA – chain for fully automaticprocessing of optical satellite data
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 4
CATENA – Use cases
• Catena is– A chain of processing modules– Uses interface- and data standards– Usable as DIMS- or stand-alone-version
• Example use cases are– Orthorectification in Image2006, Image2009, UrbanAtlas, ...– DEM-Generation as service for Cartosat/Euromap– Stereoprocessing, Time series, CCI-Fire, ...
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 5
CATENA – Requirements
• Systems and Libraries:– Linux (tested on CentOS, Ubuntu)– XDibias (DLR in-house development)– Python, scipy, numpy– GDAL
• Modules:– Must not be interactive (automatic processing chain!)– Preferable: UNIX C/C++ source code, python code– Possible: Java, Fortran, any standard UNIX (script)
programming language– No commercial programming environments which
require any kind of licenses!
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 6
CATENA – Interfaces
• Input-Data:– Original Level-1-satellite data containing all metadata– Processed data including required metadata
• Modules:– Image data and metadata in standardized XDibias format– Modules wrap existing processors with configuration files and
any image format supported by GDAL
• Output:– Any image format supported by GDAL– Standardized export.xml containing meta- and processing info – JPG-Quicklooks, KML files, any other intermediate files
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 7
CATENA – Summary of Principles
• Standardized image- and metadata formats
• Standardized process flow organization• Processor follows ESA „Generic IPF Interface Specifications“
• Distributed computing and storage
• Standardized Development and Deployment process• Guidelines for module development, documentation and deployment• ISO9000 certification in process: external audit today (2012-10-11)
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 8
Work-space
CATENACATENA
Module 1
Originaldata
ReferenceDB 3
Ingestion
Ort
ho
Web-Interface
ReferenceDB 2
ReferenceDB 1Module 2
Module 3
Module 4
Export
Import
Module ...DE
M-G
ener
atio
n
CC
I-F
ire
Ch
ain
. . .
Delivery
Delivereddata
Processingcontrol system
Selectprocessing
chain and setparameters
Selectprocessing
chain and setparameters
Modules andorder defined
in chain
Modules andorder defined
in chainSome modulesneed additional
data
Some modulesneed additional
data
Each jobgets processed in
own space
Each jobgets processed in
own space
Standardizedimage andmeta data
Standardizedimage andmeta data
DIMS-PSM orstand-alone
DIMS-PSM orstand-alone
Cleaned upafter delivery
Cleaned upafter delivery
Processing chains
Atm
osp
her
ic C
orr
.
CATENA – System overview
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 9
CATENA – Grid computing
Referencedata
data
crontab
work
crontab
work
crontab
work
crontab
work
crontab
work
crontab
work
crontab
work
crontab
work . . .
Node n
Node 1
Node 2
Node 5 Node 4
Node 3
Node 6
Server
crontab
work
DB
webserver
scenedatabase • Simply add new node
by creating working directory and inserting CATENA into crontab
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 10
CATENA – Distributed Mass Storage
• Distributed mass storage with access from each processing node is needed for automatic processing of time series or bulk data for:
• Realized as easily expansible Scality storage ring:
CATENACATENA
Data
Data is storedautomatically inthree distributedcopies in the ring,read-access alsoin parallel fromthree storagenodes.
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 11
Web-Interface of stand-alone version
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 12
Examples
• Orthorectificationgeocoded, optionally atmospheric corrected satellite images for further thematic processing and emergency mapping
• DEM generationgenerate DEMs and Ortho images from (multi) stereo satellite data
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 13
Processing chain: Orthorectification
• Standard processing chain for most optical satellite data
• Satellites acquire oblique images
• Ephemeris and attitude not exactly known
• Correct these using ground control points• from already existing geocoded images
• Project satellite image on existing digital elevation model• from DEM database (e.g. SRTM)
• Resample satellite image in requested projection and resolution
CATENACATENA
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 14
Processing chain: OrthorectificationWorkflow
CATENACATENA
Original image Reference image
DEM
Ortho image
Matching
Control points
Improvementof orbit and attitude data
Generation of ortho image
Quality check
Manually measuredground control points
Delivery
Atmospheric Correction
Thematic processing
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 15
RMSE x (blue) and RMSE y (red) Coverage 1&2
0
5
10
15
20
met
er
Overall Geometric AccuracyOverall Geometric Accuracy
Overall (~4000 scenes) mean accuracy w.r.t. reference data set:RMSEx/y ~ 10 m ( CE64 ~14m)~0.5 pixel size of resampled images
Mean number of ICPs per scene for accuracy assessment:
IRS-P6: 5496 points / sceneSPOT 4/5: 1360 points / sceneResidual plots available
Cov
erag
e 1
Cov
erag
e 2
RMSE X Y X Y
Requirement: RMSE < 20m
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 16
• At least two images from same orbit
• Good relative orientation required,<0.5px, Bundle block adjustment
• Dense pixelwise Semi-Global Matching= Disparity map on original images
• Reprojection of DEM to target coordiante system, Interpolation and filling of holes
• Orthorectification of the original imagery
Metadata
Orientation
Matching
DEM Generation
Ortho-rectifi-cation
Images
DEM Ortho
Inp
ut
Pro
cess
ing
Ou
tpu
tProcessing chain: DEM generationWorkflow
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 17
Processing chain: DEM generationOrtho image and DEM, London
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 18
Processing chain: DEM generationLondon DSM from 5 WorldView-2 Images
• Very steep terrain
• Very detailed surface model• Film: http://www.dlr.de/dlr/desktopdefault.aspx/tabid-10212/332_read-921/
15°
0°
-15°
Processing chain: DEM generationK2 WorldView-2 Triple Stereo
Processing Chains for Optical Data • Thomas Krauß • 2012-07-27 • www.DLR.de • Slide 19
Processing Chains for Optical Data • Thomas Krauß • 2012-10-11 • www.DLR.de • Slide 20
Processing Chains for Optical DataSummary
• Processing chain CATENA developed at the Remote Sensing Technology Institute of DLR for fully automatic processing of mass data from many different optical satellites
• Already in use for many projects (Image2006-2012, UrbanAtlas, CCI-Fire, Cartosat-DEM-processor, Worldview-2 and Pleiades DEM generation, …)
• Based on the general processing chain infrastructure CATENA including:• Modular system of processing Modules connected to Chains• Distributed parallel grid computing• Distributed mass storage
• Easily expandable, e.g.:• A new processing chain for a new project• Adding normal Linux-PCs or virtual machines as new background
processing nodes
CATENACATENA
Contact: Thomas Krauß, DLR-IMF, [email protected]