…

…

…

…

…

…

…

…

…

…

Patrick Helber, Benjamin Bischke, Damian Borth, Andreas Dengel

Deep Learning Competence Center & Smart Data and Knowledge Services

German Research Center for Artificial Intelligence (DFKI)

Earth Observation From SpaceDeep Learning Based Satellite Image Analysis

…

…

…

…

Satellite Data Becomes Publicly Available

2

© ESA / Astrium

Global Challenges

3

Satellite Analysis -> Global Challenges

4

Infrastructure & Traffic MonitoringPollution Monitoring: Oil Spills and Pipelines

Land Change Detection, DeforestationAgriculture

Agriculture: Monitoring Farm Land

5

© ESA

Climate Change: Deforestation

6

Monitoring of Economic Factors

7

• Economic Growing and Shrinking

© DigitalGlobe

Natural Disaster: Flooding, Wildfire, Earthquake …

8

Earth Observing SatellitesOptical and Radar

9

Satellites – Optical View

• Sentinel-2

– 10 meters per pixel

– Two-satellite constellation

– 5 days revisit time

– Global land surface coverage• Onshore, large islands, inland and coastal waters

• Worldview, GeoEye, SkySat

– Up to ca. 0.30 meters per pixel

– High-resolution images

10

Author: Raman

Optical View: 10 meters/pixel

11

Optical View: High-resolution up to ca. 30cm/pixel

12

Radar Satellites – Sentinel 1

• Synthetic Aperture Radar (SAR)

• Sentinel-1

– 6 day repeat cycle

– Two satellite constellation

• For the first time continuous

radar scanning

13

© ESA

Radar-Based Earth Observation

14

Satellite Image ClassificationOptical Images

15

EuroSAT Publicly Released by DFKI

16

Ground Truth Data

• European Urban Atlas

– Detailed mapping for 695 cities distributed over 30 Europeancountries

– Released August 2016

– Covered time period: 2011-2013

17

P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” arXiv preprint arXiv:1709.00029, 2017.

EuroSAT - Distributed Over 30 Countries

• 10 classes

• 27,000 images

• 64 x 64 images

• 2,000 – 3,000 images per class

• 13 spectral bands• Spatial resolution: 10 m per pixel

18

P. Helber, B. Bischke, A. Dengel, and D. Borth, “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification,” arXiv preprint arXiv:1709.00029, 2017.

Land-Use and Land-Cover Classification

19

Industrial Residential Highway Annual Crop Permanent Crop Pasture

Sea & Lake RiverForestHerbaceous vegetation

Classification with ResNet and GoogLeNet

• Dataset split

– 80% Training and 20% Testing

• Convolutional Neural Networks

– ResNet-50

– GoogLeNet

20

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1–9, 2015.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.

Inception Module

ResNetGoogLeNet

Residual Building Block

Models Pre-trained on ImageNet

21

• Transfer learning

– Fine-tuning of pre-trained networks

• Pre-trained on ImageNet

– ILSVRC-2012

Band combination

RGB SWIR CI

Accuracy 0.9857 0.9705 0.9830

SWIR = Short-Wave-Infrared

CI = Color-Infrared

Classification Results

22

• Transfer learning

– Fine-tuning of pre-trained networks

• Pre-trained on ImageNet

– ILSVRC-2012

Band combination

RGB SWIR CI

Accuracy 0.9857 0.9705 0.9830

SWIR = Short-Wave-Infrared

CI = Color-Infrared

Spotting Land Use Changes

23

Residential Area Built Upin Dallas, USA

August 2015 March 2017

• Large-scale scanning and monitoring

• Time component• High frequency

• Near-real-time

• Future availability

• Innovative applications

• Building systems for future

real-time applications

Spotting Land Use Changes

October 2015 September 2016

Deforestation (Forest Clearing)in Villamontes, Bolivia

24

Industrial Buildings Demolished in Shanghai, China

December 2015 December 2016

Support Mapping Services

25

Melbourne, Australia Shanghai, China

Look More CloselyClassification with High-Resolution Optical Images

26

Functional Map of the World

• Four different satellites

• Different spatial and spectralresolutions

– 4-band and 8-band

• Temporal image sequences

• 63 land-use/-cover and building types

– Security & Defense

– Emergency Response

– Infrastructure

27

Christie, Gordon and Fendley, Neil and Wilson, James and Mukherjee, Ryan, “Functional Map of the World” CVPR, 2018.

Airport Car DealershipBorder Checkpoint

Police Station Shopping MallFlooded Road

Lighthouse DamSolar Farm

Task: Sequence Classification

• Sequences determine the underlying class

– Construction sites

– Flooded road

28

Classification and Fusion Approaches

• DenseNet-161

– CNN Features (I)

– Metadata Features (M)

– CNN + Metadata Features (IM)

• Sequence Classification

with LSTM

29

DenseNet

Approach LSTM-M CNN-I LSTM-I CNN-IM LSTM-IM

F1-Score 0.193 0.679 0.688 0.722 0.734

Christie, Gordon and Fendley, Neil and Wilson, James and Mukherjee, Ryan, “Functional Map of the World” CVPR, 2018.

CNN (Features)

Metadata Features

Misclassification

Hard to classify

• Port Shipyard

• Office building Fire station, police station

• Hospital Educational institution

• Police station Educational institution

• Multi unit residential Single unit residential

30

Shipyard Port

HospitalEducational Institution

Approach Port Shipyard Police station

Office Building

Hospital

F1-Score 0.193 0.351 0.329 0.225 0.458

Satellite Image SegmentationHigh-Resolution Optical Images (Aerial)

31

Fine-Grained Segmentation With 10cm/Pixel Images

32

Fine-Grained Cadastral Map for Land Change Detection

33

Work in progress!

Netherlands: High-Resolution RGB+NIR

34

Leverage Learning with Altitude Measurements

35

Beyond the Visible SpectrumMultispectral Signatures

36

Multispectral Classification

37

EuroSATRGB

FMoW

Multispectral Segmentation

• Damage Estimation Caused by Wildfires

– Dataset• Satellite: Sentinel-2

• Mapping: EMS Coperncicus

• 13 spectral bands

• 1.564 scenes from 5 different countries

• 256px images

– Band combinations• Visible spectrum: RGB

• Infrared spectrum: Red Edge

• Near-Infrared: NIR

38

RGB + Red Edge + NIR Spectral Information

39

SegNet

SegNet Input Pix. Accuracy mIOU

RGB 0.9878 0.7657

Red Edge 0.9885 0.7592

NIR 0.9815 0.4964

RGB + NIR 0.9864 0.7844

RGB + Red Edge 0.9855 0.7928 (+2,71)

RGB + Red Edge + NIR Prediction

loss

epoch

Qualitative Results

40

GT RGB RGB + Red Edge

RGB Red Edge10m 20m

Beyond the Visible SpectrumRadar Waves

41

Optical Satellite Images - Obstructed View

42

Optical Satellite Images - Obstructed View

43

Optical Satellite Images - Obstructed View

44

Optical Satellite Images - Obstructed View

45

Vision Independent of Clouds

46

Radar-Based Earth Observation Imagery

47

VVVH

© ESA

• Synthetic Aperture Radar (SAR)

• Satellite: Sentinel-1

• Dual-Polarization

– VV and VH

• Scenarios• Deforestation

• Find safest route for icebreakers

• Land monitoring

• Ship traffic monitoring

• Natural disasters like flooding and earthquake

Dual-Polarized SAR

48

Flooding

• Satellite: Sentinel-1

• Mapping: EMS Copernicus

• 9.426 radar image scenes

• 56 flooding events in 12 countries

• Globally spread• Australia, USA, France, UK, …

Segmentation with Dual-Polarized Radar Waves

• Convolutional Networks

– SegNet: Convolutional Encoder-Decoder Network

49

SegNet PredictionInput

VV + VH

Model Pix. Accuracy mIOU

FCN-6 0.9496 0.6772

VGG16 0.9413 0.6740

SegNet 0.9530 0.6541

Segmentation

Natural Disasters: Damage Maps

Flooding: Damage Estimation Fire: Damage Estimation

• Biomass by radar waves

50

• Government, Insurance, Investment

Copernicus Emergency Management Service (© European Union, 2012-2017)

Destroyed

Highly damaged

Slight damage

Multimedia Satellite Task 2018

• Emergency Response for Flooding Events

• Focus on Road Passibility (Impact of Infrastructure) (Road Access, blocked Road) with two subtasks

• Classification of Road-Access & Passability– in Social Multimedia

– in multiple Satellite Images (Radar, Optical)

• Registration open (http://www.multimediaeval.org)

51

Images in Tweets (Hurricane Harvey)

Satellite Images of Houston (US) for Hurricane Harvey

CombineResultvia Geo-Location

Thanks!

52

NVAIL partner

all networks trained on DGX-1