spacenet: accelerating machine learning for foundational ... · public sector © 2019, amazon web...

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T

SpaceNet: Accelerating Machine Learning for Foundational Mapping Challenges

Ryan LewisSVP IQT CosmiQ Works

Joe FlasherOpen Geospatial Data LeadAWS

Adam Van EttenResearch DirectorIQT CosmiQ Works

Todd BacastowDirector Maxar Technologies


Agenda

SpaceNet Introduction

Previous Challenge Results

Upcoming Challenges

Information Channels


4

© SpaceNet LLC 2019.

(1) Machine learning algorithms & (2) increased overhead data collection will

fundamentally disrupt geospatial analytics

Convergence of Two Tech Trends

4


3 Market Challenges

5

Lack of Curated, Labeled Data Sets for Geospatial Applications

Open Source, AI Models Designed for Different Problems

Open Software Tools for Geospatial Analysis Are Limited


6

SpaceNet’s Mission

SpaceNet is a nonprofit LLC focused on:

1. Data Developing Open Source Data Sets

2. Algorithms Fostering Applied Research for AI Software

3. Evaluation Benchmarking Performance for Applications


SpaceNet: 4 Pillars

7

Labeled Data Sets Competitions Algorithms Software Tools

• Images of 6 Cities

• 800,000+ Building Footprints

• 10,000 km Road Labels

• 4 Competitions on TopCoder

• $200,000 in Total Prizes

• 1,000+ Submissions Worldwide

• 18 Algorithms

o 13 Building Detection

o 5 Road Detection & Routing

• Ease Use of Imagery

• Simplify Evaluation

• Speed Up Model Deployment



SpaceNet: Opening the Floodgates for GEOINT R&D2016

SpaceNet 1: Building Footprint ExtractionCars Overhead With Context (COWC)IARPA Multi-View Stereo 3D Mapping

2017

SpaceNet 2: Multi-City Building FootprintsIARPA Functional Map of the WorldUSSOCOM Urban 3D Challenge

2018

SpaceNet 3: Road Network ExtractionSpaceNet 4: Off-Nadir Building FootprintsCrowdAI Mapping ChallengeDIUx xView Object Detection Challenge

2019

Microsoft US & Canadian Building FootprintsUpcoming: SpaceNet 5: Roads with travel time

8


Competitions to Date

9

SpaceNet 111/2016 – 1/2017

SpaceNet 26/2017 – 8/2017

SpaceNet 311/2017 – 2/2018

SpaceNet 410/2018 – 1/2019

Building Footprint

Detection

Rio De Janeiro

Building Footprint

Detection

Las Vegas, Paris, Khartoum, & Shanghai

Road Extraction &

Routing

Las Vegas, Paris, Khartoum, & Shanghai

Building Footprint

Detection (Off-Nadir)

Atlanta



Automated Overhead Imagery Analysis is Improving

10


11

Fei Fei Li’s (Founder of ImageNet) presentation at CVPR 2017

Community Acknowledgement



SpaceNet 4 Overview

13

Imagery Data Set

27 Collects Over Atlanta7O to 54O Off-Nadir655 km2 Covered0.5 m Resolution

Labels

126,747 Building FootprintsFrom 20 m2 to >2,000 m2

Urban, Industrial, & Suburban~3,000 km Road Network Labels

Algorithms

5 Open Sourced Solutions15 Computer Vision Models w/

Solution Explanations> 250 Competition Submissions


Why Off-Nadir Imagery14

Urgent Collections are Often Off-Nadir (below)State-of-the-art algorithms were untested on off-nadir imagery

Daiichi Power Plant | Fukushima, Japan

Look Angle: About 45º

Imagery Courtesy of DigitalGlobe


Barriers to Off-Nadir Imagery Analysis

15

Variable Shadows

Occluded Structures

Footprint Displacement

Resolution Degradation


Scoring: Building Footprint Extraction

16

1. Find predicted buildings with Intersection over Union (IoU) > 0.5

Truth Pred.

IoU = 0.75Success

IoU = 0.15Failure

2. Aggregate successes/failures across all collections in three look angle bands:

Ground truth

A. Nadir: 0-25B. Off-nadir: 26-40C. Very off-nadir: > 40


Algorithmic Challenges: Nadir Angle

17

40% drop in score for allalgorithms from 7º to 54º


18

Algorithmic Challenges?: Shadows

Image from the South(facing north)

Image from the North(facing south)


Top buildings:Not occluded

Bottom buildings:Occluded by trees

Algorithmic Challenges?: Occluded Buildings


Algorithmic Challenges: Building Size


Footprint Quality Threshold Matters

21

Truth Pred.

IoU = 0.75

IoU = 0.15


Research Publication from SpaceNet 4

22

Link: https://arxiv.org/abs/1903.12239


What’s Next: Returning to Road Networks

Scoring based on pixel masks does not always incentivize the desire outcomes


• Segmentation efforts have demonstrated some success in identifying road pixels from overhead imagery but do not always incentivize the desired outcome

• Evaluation metrics are pixel-based: (1) completeness, correctness, quality; and

• (2) relaxed F1

• correct value within 3 pixels

Wang et al 2016 (Quality = 0.86)http://www.mdpi.com/2220-9964/5/7/114

Zhang et al 2017 (relaxed F1 = 0.92)https://arxiv.org/pdf/1711.10684.pdf

Mhih and Hinton 2010 (relaxed F1 = 0.90)http://www.cs.toronto.edu/~fritz/absps/road_detection.pdf

Limitations of Current Segmentation Techniques


Scoring Routing Information: APLS

•Average Path Length Similarity (APLS) was developed for SN3

•Both Logical and Physical Topology Are Important for Road Detection

•Sum the Difference in Paths between Ground Truth & Proposals

•Betweenness Centrality is Fundamental to the APLS Metric


Ground Truth1. Parse road labels into 400m sections concurrent with SpaceNet imagery

2. Create ground truth masks by drawing a 2m buffer around road centerlines in road labels

3. Augment the training dataset by a factor of 3 via HSV rescaling and rotations [increases performance by 8% (Vegas) to 13% (Khartoum)]

4. Train a deep learning segmentation model (PSPNet, U-Net) using SpaceNet imagery & road masks

5. Perform post-processing to eradicate short segments and close small gaps

Predictions: Segmentation Model


A. Extract skeleton from proposal mask

B. Build a proposal graph from the skeleton

C. Simplify and smooth proposal graph

A B C

Predictions: Mask to Graph


Entrant Country Avg Score Las Vegas Paris Shanghai Khartoum

albu Russia 0.6663 0.7977 0.6040 0.6543 0.6093

cannab Russia 0.6661 0.7804 0.6446 0.6398 0.5996

pfr France 0.6660 0.8009 0.6008 0.6646 0.5975

selim_sef Germany 0.6567 0.7884 0.5991 0.6472 0.5922

fabastani USA 0.6284 0.7710 0.5474 0.6326 0.5628

ipraznik Germany 0.6215 0.7578 0.5668 0.6078 0.5537

tcghanareddy India 0.6182 0.7591 0.5710 0.6014 0.5415

hasan.asyari Norway 0.6097 0.7407 0.5557 0.5952 0.5472

aveysov Russia 0.5943 0.7426 0.5805 0.5751 0.4789

SpaceNet 3 Results

Winning entrant, albu, submitted a generalized model across all four cities


AOI_2_Vegas_img1011 – APLS = 0.512 AOI_2_Vegas_Img1045 – APLS = 0.988

Dave Lindenbaum, GTC 2018

Las Vegas Results

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I TDave Lindenbaum, GTC 2018

Khartoum Results

AOI_5_Khartoum_img404 - APLS = 0.385 AOI_5_Khartoum_img398 – APLS = 0.897

31


Ground Truth

Raw Proposal Mask

Extract Entire Khartoum Road Network•Combined BASISS & Albu’s Implementation (w/ Extra Post-Processing from CosmiQ Works)

o Image Size (Pixels) 55,420 x 161,258 (9 terapixels)

o Image Size (Km) 16 x 48

o File Size (GB) 89

o Nodes 195,938

o Edges 258,655

Total Processing Time = 6.3 Hours (Single GPU/CPU)


SpaceNet 5: Expanding Upon Routing

33

Challenge Participants Will Be Asked to Infer:

… From a Single Satellite Image

Road

Networks

Routing

Information

Travel

Times

The SpaceNet 5 Challenge is Scheduled to Launch in September 2019


APLS = 0.81

Multiclass Baseline

34

•Trained resnet34 + unet segmentation model

o Use 7– channel training masks

RGB Image Labels Projection


SpaceNet 6: Preliminary Planning

35

Model Deployability / Generalizability

New Applications Beyond Foundational Mapping

Sensor & Data Fusion


Information Channels

37

CosmiQ’s Repo:https://github.com/CosmiQ

SpaceNet’s Repo:https://github.com/SpaceNetChallenge

SpaceNet Competition Hosting Site

https://www.topcoder.com/spacenet

SpaceNet Data on AWShttps://registry.opendata.aws/spacenet/

CosmiQ’s Blog:

The DownLinQhttps://medium.com/the-downlinq

CosmiQ’sTwitter:

@CosmiQWorks

Title: Training_Data

Found on: Apple Podcasts, Google Play, Spotfiy, Stitcher,

& SoundCloud


Thank you!


Ryan [email protected]

spacenet: accelerating machine learning for foundational ... · public sector © 2019, amazon web...

Documents