european space agency (esa) landsat mss/tm/etm+/oli ... · landsat mss dataset improvements and...

European Space Agency (ESA) Landsat MSS/TM/ETM+/OLI Archive: 42 years of our history

Sébastien Saunier, Amy Northrop, Samantha Lavender, Luca Galli, Riccardo Ferrara, Stefano Mica, Roberto Biasutti, Philippe Goryl, Ferran Gascon, Marco Meloni, Bas Altena

MultiTemp 2017, Bruges, June 2017

Landsat MSS Dataset Improvements and Multi Temporal Analysis

This image cannot currently be displayed.

Contents

2

• From 1975, ESA has used its multi-mission ground systems to acquire, process and archive data from the Third Party Missions (TPM).

• Landsat data continuity missions are part of ESA’s TPM.

3

4

1972

1973

197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000200120022003200420052006200720082009201020112012

Landsat 1 - MSSLandsat 2 - MSSLandsat 3 - MSSLandsat 4 - MSSLandsat 5 - MSSLandsat 5 - TMLandsat 6 - ETMLandsat 7 - ETM+

Landat 8 - OLI/TIRS

ESA Operation in operational lifeOperational life

5

Landsat Sensor

RBV (LS-1-3)

MSS (LS-1-5)

TM (LS-4-5)

ETM+ (LS-7)

Spectral Bands (µm) 1) 0.48-0.57 VNIR

2) 0.58-0.68 VNIR 3) 0.70-0.83 VNIR P) 0.505-0.75 VNIR (available on LS-3)

4) 0.50-0.60 VNIR 5) 0.60-0.70 VNIR 6) 0.70-0.80 VNIR 7) 0.80-1.1 VNIR 8) 10.4-12.5 TIR (available on LS-3)

1) 0.45-0.52 VNIR 2) 0.52-0.60 VNIR 3) 0.63-0.69 VNIR 4) 0.76-0.90 VNIR 5) 1.55-1.75 VNIR 6) 10.4-12.5 TIR 7) 2.09-2.35 SWIR

1) 0.45-0.52 VNIR 2) 0.52-0.60 VNIR 3) 0.63-0.69 VNIR 4) 0.76-0.90 VNIR 5) 1.55-1.75 VNIR 6) 10.4-12.5 TIR 7) 2.09-2.35 SWIR P) 0.50-0.90 VNIR

Swath Width 185 km 185 km 185 km 185 km

Spatial resolution (Across Track / Along Track)

40 m (PAN) 80 m (VNIR)

59x79 m VNIR 240 m TIR

30 m VNIR/SWIR 120 m TIR

15 m PAN 30 m VNIR/SWIR 60 m TIR

Radiometric resolution Analogic 6 bit 8 bit 9 bit ( 8 bit transmitted)

Band to band registration 0.2 pixel (90 %) 0.2 pixel (90 %) Geodectic Accuracy (without Ground Control Point)

500 m (90 %) 400 m (90 %)

Data Rate 15 Mbits/s 85 Mbits/s 2*75 Mbits/s Average Power 50 W 332 W 590 W

Telescope Aperture 23 cm 40.6 cm 40.6 cm

Context

• The ESA collection of Landsat Historical data observed in Continental Europe, African Region and Arctic has become massive.

• Major efforts have been undertaken, to improve processing s/w and then produce the most up to date Level 1 products for the community.

• (As part of the International Ground Station (IGS), ESA is continuously maintaining efforts to acquire, process and distribute new real time Landsat 8 data

6

7

This archive was a black box

8

Status prior improvements / Solutions

9

Item Status Solution Reference

Format CEOS Format -Aligned to USGS: GeoTIFF and ancillary files within a zip archive

LS-DFCB

GEO Accuracy

Orbit and attitude data are mostly inaccurate preventing operational use of data. Image Geometry is distorted and “Panoramic Effect” observed

-Re evaluate / Re-estimate Orbit data -Improve GCP set, DEM, GCP Filtering -Improve Geometric Model (attitude model) -Use Multi Scene Refinement concept

ACS ATBD

Status prior improvements / Solutions 2/2

10

Item Status Solution Reference

RADIO Accuracy

Large variations in pre flight calibration Image Saturation

Ground Model Lifetime Calibration Update of calibration

Helder , Saunier

Image Quality Image Anomalies Detect and Flag with BQA ESA s LANDSAT Product Definition

Product generation

A lot of products has never been generated

Systematic generation and quality control

TPM website https://tpm-ds.eo.esa.int/collections/

Helder “Absolute Radiometric Calibration of Landsat Using a Pseudo Invariant Calibration Site” , IEEE Transactions on Geoscience and Remote Sensing Volume 51, Issue 3 (March 2013).

Saunier, “Landsat MSS, radiometric calibration processing improvement” in Proceeding of ESA Living planet Symposium 2016.

Difference with USGS products

• Geometry : Instead of systematically applying scene based ortho-rectification – it is in some cases more appropriate to calibrate the geo-referencing model by using data from the whole satellite pass. The distortions for the whole pass are modelled using a weighted, constrained, least squares fit with iterative outlier removal. Also, an accurate ortho rectification is applied for scenes partially occluded with cloud, and for scenes with large portion of water.

• Radiometric calibration: The analysis of Level 0 data, has revealed that MSS calibration was not optimal, in a way that for bright area target RED / NIR1 bands included a lot of saturation. For these regions, the calibration coefficients have been revised and most data are now almost free from saturations.

• BQA: Quality information attached to Level 1 MSS pixel has been developed for a better filtering of input pixels.

• Image Reparation: Pixels flag as corrupted for a certain class of anomaly (sticky bit) are repaired – Development / Validation is ‘on going’

11

Geometric Processing (1/2)

12

• Objective is simple – maximize number of Terrain Corrected products (L1T).

Geometric Processing (2/2)

Quicklooks for same pass v3.04 (top) and v3.06 (bottom)

– Note the scenes with large distortions for v3.04

13

Geometric Processing Accuracy

• Methodology has been validate for TM and Improved for MSS.

14

15 Calibration Stability of MSS data after alignment to TM Data (Helder, 2013)

Radiometric Processing 1/2

• Intensive work of the SDU / usgs to harmonized MSS calibration. Calibration methodology implemented at ESA.

16 Selected Landsat WRS scenes for calibration updates

Radiometric Calibration 2/2

• Objective is simple – maximize number of Saturation Free products.

17

18

Bit Value USGS Landsat Collection 1 Description

ESA MSS Description

0 1 Designated Fill Designated Fill

1 2 Dropped Pixel Dropped Pixel

2 4 Radiometric Saturation

Radiometric Saturation

3 8

4 16 Cloud Cloud

5 32 Cloud Confidence Cloud Confidence

6 64

7 128 Cloud Shadow Confidence

Cloud Shadow Confidence

8 256

9 512 Snow / Ice Confidence

Snow / Ice Confidence

10 1024

11 2048 Scan Line Anomaly (SLA)

striping / saturation /

hotspot / sticky bit

12 4096

13 8192

14 16584

15 32786 Land / Water mask

BQA – Pixel based information

19

– Scan Line Artefacts (SLA) – hot spot, sticky bit

– Dropped lines

– Saturation

– Cloud cover

– Land / Water mask (ESA CCI product)

Insight on the archive

• Analyze temporal coverage of MSS data for specific countries and considering winter and summer period.

• EU countries covered since the beginning and almost uniformly.

• Improvements of the coverage of Eastern Countries (Russian, Kazakstan), Northern Countries (Greenland) and Southern Countries (Western part of Africa ) and Islands.

22

23

24

“Proof of concept”

• Outline : Several multi temporal applications for which the use of MSS is valuable have been selected in order to demonstrate that data are fit for application purposes.

• Thematic Area : Phenology Monitoring, Snow / Ice Monitoring, and urban mapping

25

Phenology : Monitoring of Forest (Case 1) / Grass land (Case 2) • Approach : Collect a large MSS, TM, LS08 dataset

over forest (Case 1), grass land (Case 2). Apply preprocessing, compute NDVI, perform analysis.

• Case 1 Study site: – Bouconne, Buzet Forest

around Toulouse (France)

• Case 2 Study site: – The northern coast of East Anglia,

(Eastern North Sea coastline of the United Kingdom)

26

Phenology Case 1– Results 1/2

27

Phenology Case 1– Results 2/2

• Agreements between measurements from two different dataset, involving different measurements.

• Good shape of the average NDVI with more uncertainties at growing season

• Outliers removed because of – In some cases, very small NDI values on MSS1 / MSS2

– 1985/1987 Long and in late freezed period

– One LS08 NDVI (Summer 2016) very high compared to previous ones (0.76)

28

Phenology Case 2 – Results

• Strong Discrepancies between TM and MSS

• MSS NDVI appears lower than TM and its temporal variability is important.

• Variability of MSS calibration is an issue, strong changes are mostly due to weather conditions and year to year seasonal shift.

29

• Landsat 5 data , MSS (1990-1993)/ TM (1985-2010)

Snow/Ice : Equilibrium Line

• Approach : Collect a MSS, Sentinel 2 data acquired (if possible) in Autumn, Segment Equilibrium line, and assess if MSS data radiometry allows to perform this task.

30

Snow/Ice : Equilibrium Line

31

Snow/Ice : Equilibrium Line

• Results : – MSS2/3/4/5 data allows to search this line at large scale

– Problem is more the period of data, two observation per year and sometimes cloudy.

– Dataset, MSS 5, can be densified with TM data.

– Landsat and S2 Results are in agreement.

32

Urban Mapping

• Approach : Collect a large MSS, TM, LS08 dataset over Toulouse City, more than 100 products, create a movie and assess visually urban dynamic – Evaluate if details seen with MSS are sufficiently well defined to characterized changes.

• Results : – No geometric correction applied, except for one product

– The registration of images are with 1.5 pixels if one consider all the series.

– See next slides.

33

Conclusions

34

• The ESA MSS data reprocessing project has been presented. The work brings new knowledge regarding our data stored at ESA.

• We hope it will benefit to a larger user audience.

• The same approach has been successfully applied for TM / ETM+ data. However, MSS required very careful / specific treatment in term of product development, product processing and quality control.

• The reprocessing of MSS data will take place by the end of Summer.

• LS08 data Near Real time service proposed by ESA

• IDEAS+ QC activities performed on a monthly basis: – Absolute Geolocation Accuracy Assessment;

– Multi temporal Geolocation Accuracy Assessment ;

• Short term geolocation stability – data from the same path, observed on the same date are analysed

• Cross track variability – different regions in the image located at several locations in the field view are compared

– Band to band Geometric Registration Accuracy Assessment;

– Radiometric Calibration stability over PICS;

• Statistical comparison between results of the two different ROIs is systematically performed

• Influence of terrain content is better assessed.

– BQA band inspection, Image Artefact Analysis and Thermal band inspection.

• Monthly Performance Report is issued every month and available @ https://earth.esa.int/web/sppa/mission-performance/esa-3rd-party-missions/landsat-8/oli-tirs/cyclic-quality-reports

35

LS08 Processing and QC at ESA

Team Introductions

Processor development

Instrument Data quality Evaluation and Analysis Service eXpert Product Reprocessing Scalable Service

Tool verification

L1 QC

Expert support

Project management

Systematic QC tool development (AMALFI)

Project management

Data collection (L0 & AUX)

Bulk processing

Repatriation (consolidated L0 & processed L1)

Data consolidation

L0 QC

Processor integration

Data Access

• ESA’s TPM online dissemination service: https://tpm-ds.eo.esa.int/collections/

• ESA Landsat 8 Web Portal: https://landsat8portal.eo.esa.int/portal/

• ESA Landsat 8 RealTime Viewer http://landsat8-realtime.eo.esa.int/observer

37

Thank you for your attention

This image cannot currently be displayed.

• Estimate LS05 MSS Calibration parameters over Libya 4 PICS applying cross calibration methodology;

• Validation on a small dataset observed over Libya 4 and in Iceland;

• Validation on a large dataset observed in all available PICS test sites;

• Define most affected WRS scenes in the archive broken down into Desert / Ice-snow regions;

• For these scenes apply new calibration parameters depending for L1 product generation depending on the season (Summer / Winter)

39

MSS v3.05 IPF – Enhancements Calibration Coefficients

40

Oversampled LANDSAT 8

Surface Reflectance

Measurements

ECMWF Water vapor

Ozone

MSS Spectral Band Definition

MSS Top of

Atmosphere Measurement

SIMULATED Top of

Atmosphere Measurement

Radiative Transfer

Code (6s)

MSS Observation

date and geometrie

Data Collected for all ROIs

Statistical Comparison

Calibration Parameters

41 Author: SEBASTIEN

MSS v3.05 IPF – Enhancements Calibration Coefficients

Libya 4 dataset, RED band

Libya 4 dataset, NIR1 band

42

CPF patch with changes on Min / Max Radiance

Latest CPF patch with changes on Min / Max Radiance and

Radiometric Calibration Gain

MSS v3.05 IPF – Enhancements Calibration Coefficients

Timeline of Events

43

2014 2015 2016 2017

IPF, QC Tool & CPF

verification

v3.05 specification definition &

design

MSS stress test (v3.03)

v3.04 development &

verification

Processing

QC of full MSS archive

MSS L1 ACTIVITIES

TM/ETM+ L1 ACTIVITIES

MTI Delta

QUALITY CONTROL

TM BSK

CPF definition & preliminary verification

QC tool update

ETM+ KSE & MPS

TM MPS TM MTI

ETM+ MTI

Mid-orbit gap analysis

Regeneration of TM GCP AUX

IDEAS+ activity X-PReSS activity

Processing Processing

AUX SV generation

44

Iceland Results (snow – ice) Percentage of saturated pixels

for 3 L1 scenes: Green and Red bands Little reduction of saturation

Percentage of saturated pixels for 3 L1 scenes: NIR1 and NIR2 bands Significant improvement for NIR1

MSS v3.05 IPF – Enhancements Calibration Coefficients

MSS v3.05 IPF – Enhancements Calibration Coefficients • LS05 MSS CPF files have been updated (applicable for a

restricted number of WRS scenes): – LM5CPF19840406_19841108 – LM5CPF19841109_19940428

• Revised Band 2 and Band 3 scaling parameters improve product quality

• Minimum Radiance and Maximum Radiance have been changed for the RED and NIR1 bands

• The NIR1 calibration gain of the first CPF has been changed • Validation on a extended PICS dataset is now on going • Regarding methodology, improvements have been tested by

considering varying AOTs value depending on the location in the scene, it increases the accuracy of the results

45

BQA

• Additional logging – To ensure full traceability between L0 and L1 (log files not available to users)

• Full resolution jpeg QuickLook – In addition to existing medium res. png

• Quality Assurance Band (BQA) – 16 bit GeoTIFF file for all MSS L1 products – Sticky bit correction (TBC) – Updated L1 product quality classification

• Updated calibration coefficient – To reduce L1-introduced saturation

• Geometry – To provide more details on GCPs – To give greater confidence on accuracy

46

Bit Value USGS Landsat Collection 1 Description

ESA MSS Description

0 1 Designated Fill Designated Fill

1 2 Dropped Pixel Dropped Pixel

2 4 Radiometric Saturation

Radiometric Saturation

3 8

4 16 Cloud Cloud

5 32 Cloud Confidence Cloud Confidence

6 64

7 128 Cloud Shadow Confidence

Cloud Shadow Confidence

8 256

9 512 Snow / Ice Confidence

Snow / Ice Confidence

10 1024

11 2048 Scan Line Anomaly (SLA)

striping / saturation /

hotspot / sticky bit

12 4096

13 8192

14 16584

15 32786 Land / Water mask

MSS v3.05 IPF – Enhancements Geometry • The nominal geometrical correction procedure of the

ESA Landsat processor is to: – Automatically locate GCPs from a GCP database within a

scene by normalized absolute image correlation in orthorectified map geometry.

– Model the distortion with an affine transform using a weighted, constrained, least squares fit with iterative outlier removal.

– Apply the correction using DEM to produce L1T product

• We call this single scene refinement

47

MSS v3.05 IPF – Enhancements Geometry • The overall accuracy of the single scene fit is

estimated considering: – The number of found GCPs

– Their correlation SNR

– Their distribution over the scene

– The chi squared goodness of fit

– The number of outliers

48

MSS v3.05 IPF – Enhancements Geometry • If the estimated accuracy obtained by single scene fit

is not considered sufficient to produce an optimal quality L1T, several back up possibilities are available: – Label the L1T as suboptimal

– Use multiscene refinement to produce L1T

– Produce a L1G+ correcting geometry without DEM (not used for ETM)

– Produce a L1Gt using DEM and nominal geometry (used only for ETM)

– Produce a L1G with no geometric correction or DEM

49

MSS v3.05 IPF – Enhancements Geometry • In multiscene refinement:

– The affine transform distortions estimated by single scene fitting are used to estimate 2 distortions at 2 points along track per scene in a common SOM projection geometry.

– The distortions for the whole pass are modelled using a weighted, constrained, least squares fit with iterative outlier removal.

– A generic fourier series is now used to model the variation of distortions along track.

50

MSS v3.05 IPF – Enhancements Geometry • The main enhancements w.r.t. v3.04 (delta MSS processor)

are: – Stronger distortion constraints are used for single scene fitting

(this can avoid the production of very distorted scenes and help outlier removal in difficult situations)

– Use general Fourier series instead of b-splines for multiscene fitting (this works much better in particular when distortions have to be extrapolated for several scenes at start and end of pass)

– Add accuracy estimates which also consider distribution of used GCPs or multiscene refinement in MTL annotations, for L1G+ products also.

– Add other statistics to MTL annotations regarding GCP distribution – Use another orbit propagator (works better for longer

propagations)

51

MSS v3.05 IPF – Enhancements Geometry

Example of multiscene fit with estimated errors (L0 pass with several scenes covered by clouds)

– Note that for MSS the attitude information are missing

52

MSS v3.05 IPF – Enhancements Geometry • The multiscene refinement accuracy for each scene is

estimated considering: – The number of used single scene distortion estimates

– Their input single scene estimated accuracies

– Their distribution over the scene

– The chi squared goodness of fit

– The number of outliers

– The position of the scene within the pass

53

MSS v3.05 IPF – Enhancements Geometry • New MTL fields per product level

54

Level L1G/L1GT L1G+/L1T L1G+/L1T

Fit type None Multi Single MODEL_FIT_TYPE x x x GEOMETRIC_MAX_ERR x x GROUND_CONTROL_POINT_DISCARDED x x NUMBER_EMPTY_GROUND_CONTROL_POINT_WINDOWS x GROUND_CONTROL_POINT_WINDOW<n> x GROUND_CONTROL_POINT_RESIDUALS_SKEW_X x GROUND_CONTROL_POINT_RESIDUALS_SKEW_Y x GROUND_CONTROL_POINT_RESIDUALS_KURTOSIS_X x GROUND_CONTROL_POINT_RESIDUALS_KURTOSIS_Y x