© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
SpaceNet: Accelerating Machine Learning for Foundational Mapping Challenges
Ryan LewisSVP IQT CosmiQ Works
Joe FlasherOpen Geospatial Data LeadAWS
Adam Van EttenResearch DirectorIQT CosmiQ Works
Todd BacastowDirector Maxar Technologies
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Agenda
SpaceNet Introduction
Previous Challenge Results
Upcoming Challenges
Information Channels
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
4
© SpaceNet LLC 2019.
(1) Machine learning algorithms & (2) increased overhead data collection will
fundamentally disrupt geospatial analytics
Convergence of Two Tech Trends
4
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
3 Market Challenges
5
Lack of Curated, Labeled Data Sets for Geospatial Applications
Open Source, AI Models Designed for Different Problems
Open Software Tools for Geospatial Analysis Are Limited
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
6
SpaceNet’s Mission
SpaceNet is a nonprofit LLC focused on:
1. Data Developing Open Source Data Sets
2. Algorithms Fostering Applied Research for AI Software
3. Evaluation Benchmarking Performance for Applications
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
SpaceNet: 4 Pillars
7
Labeled Data Sets Competitions Algorithms Software Tools
• Images of 6 Cities
• 800,000+ Building Footprints
• 10,000 km Road Labels
• 4 Competitions on TopCoder
• $200,000 in Total Prizes
• 1,000+ Submissions Worldwide
• 18 Algorithms
o 13 Building Detection
o 5 Road Detection & Routing
• Ease Use of Imagery
• Simplify Evaluation
• Speed Up Model Deployment
© SpaceNet LLC 2019.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
SpaceNet: Opening the Floodgates for GEOINT R&D2016
SpaceNet 1: Building Footprint ExtractionCars Overhead With Context (COWC)IARPA Multi-View Stereo 3D Mapping
2017
SpaceNet 2: Multi-City Building FootprintsIARPA Functional Map of the WorldUSSOCOM Urban 3D Challenge
2018
SpaceNet 3: Road Network ExtractionSpaceNet 4: Off-Nadir Building FootprintsCrowdAI Mapping ChallengeDIUx xView Object Detection Challenge
2019
Microsoft US & Canadian Building FootprintsUpcoming: SpaceNet 5: Roads with travel time
8
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Competitions to Date
9
SpaceNet 111/2016 – 1/2017
SpaceNet 26/2017 – 8/2017
SpaceNet 311/2017 – 2/2018
SpaceNet 410/2018 – 1/2019
Building Footprint
Detection
Rio De Janeiro
Building Footprint
Detection
Las Vegas, Paris, Khartoum, & Shanghai
Road Extraction &
Routing
Las Vegas, Paris, Khartoum, & Shanghai
Building Footprint
Detection (Off-Nadir)
Atlanta
© SpaceNet LLC 2019.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Automated Overhead Imagery Analysis is Improving
10
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
11
Fei Fei Li’s (Founder of ImageNet) presentation at CVPR 2017
Community Acknowledgement
© SpaceNet LLC 2019.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
SpaceNet 4 Overview
13
Imagery Data Set
27 Collects Over Atlanta7O to 54O Off-Nadir655 km2 Covered0.5 m Resolution
Labels
126,747 Building FootprintsFrom 20 m2 to >2,000 m2
Urban, Industrial, & Suburban~3,000 km Road Network Labels
Algorithms
5 Open Sourced Solutions15 Computer Vision Models w/
Solution Explanations> 250 Competition Submissions
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Why Off-Nadir Imagery14
Urgent Collections are Often Off-Nadir (below)State-of-the-art algorithms were untested on off-nadir imagery
Daiichi Power Plant | Fukushima, Japan
Look Angle: About 45º
Imagery Courtesy of DigitalGlobe
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Barriers to Off-Nadir Imagery Analysis
15
Variable Shadows
Occluded Structures
Footprint Displacement
Resolution Degradation
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Scoring: Building Footprint Extraction
16
1. Find predicted buildings with Intersection over Union (IoU) > 0.5
Truth Pred.
IoU = 0.75Success
IoU = 0.15Failure
2. Aggregate successes/failures across all collections in three look angle bands:
Ground truth
A. Nadir: 0-25B. Off-nadir: 26-40C. Very off-nadir: > 40
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Algorithmic Challenges: Nadir Angle
17
40% drop in score for allalgorithms from 7º to 54º
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
18
Algorithmic Challenges?: Shadows
Image from the South(facing north)
Image from the North(facing south)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Top buildings:Not occluded
Bottom buildings:Occluded by trees
Algorithmic Challenges?: Occluded Buildings
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Algorithmic Challenges: Building Size
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Footprint Quality Threshold Matters
21
Truth Pred.
IoU = 0.75
IoU = 0.15
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Research Publication from SpaceNet 4
22
Link: https://arxiv.org/abs/1903.12239
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
What’s Next: Returning to Road Networks
Scoring based on pixel masks does not always incentivize the desire outcomes
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
• Segmentation efforts have demonstrated some success in identifying road pixels from overhead imagery but do not always incentivize the desired outcome
• Evaluation metrics are pixel-based: (1) completeness, correctness, quality; and
• (2) relaxed F1
• correct value within 3 pixels
Wang et al 2016 (Quality = 0.86)http://www.mdpi.com/2220-9964/5/7/114
Zhang et al 2017 (relaxed F1 = 0.92)https://arxiv.org/pdf/1711.10684.pdf
Mhih and Hinton 2010 (relaxed F1 = 0.90)http://www.cs.toronto.edu/~fritz/absps/road_detection.pdf
Limitations of Current Segmentation Techniques
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Scoring Routing Information: APLS
•Average Path Length Similarity (APLS) was developed for SN3
•Both Logical and Physical Topology Are Important for Road Detection
•Sum the Difference in Paths between Ground Truth & Proposals
•Betweenness Centrality is Fundamental to the APLS Metric
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Ground Truth1. Parse road labels into 400m sections concurrent with SpaceNet imagery
2. Create ground truth masks by drawing a 2m buffer around road centerlines in road labels
3. Augment the training dataset by a factor of 3 via HSV rescaling and rotations [increases performance by 8% (Vegas) to 13% (Khartoum)]
4. Train a deep learning segmentation model (PSPNet, U-Net) using SpaceNet imagery & road masks
5. Perform post-processing to eradicate short segments and close small gaps
Predictions: Segmentation Model
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
A. Extract skeleton from proposal mask
B. Build a proposal graph from the skeleton
C. Simplify and smooth proposal graph
A B C
Predictions: Mask to Graph
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Entrant Country Avg Score Las Vegas Paris Shanghai Khartoum
albu Russia 0.6663 0.7977 0.6040 0.6543 0.6093
cannab Russia 0.6661 0.7804 0.6446 0.6398 0.5996
pfr France 0.6660 0.8009 0.6008 0.6646 0.5975
selim_sef Germany 0.6567 0.7884 0.5991 0.6472 0.5922
fabastani USA 0.6284 0.7710 0.5474 0.6326 0.5628
ipraznik Germany 0.6215 0.7578 0.5668 0.6078 0.5537
tcghanareddy India 0.6182 0.7591 0.5710 0.6014 0.5415
hasan.asyari Norway 0.6097 0.7407 0.5557 0.5952 0.5472
aveysov Russia 0.5943 0.7426 0.5805 0.5751 0.4789
SpaceNet 3 Results
Winning entrant, albu, submitted a generalized model across all four cities
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
AOI_2_Vegas_img1011 – APLS = 0.512 AOI_2_Vegas_Img1045 – APLS = 0.988
Dave Lindenbaum, GTC 2018
Las Vegas Results
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I TDave Lindenbaum, GTC 2018
Khartoum Results
AOI_5_Khartoum_img404 - APLS = 0.385 AOI_5_Khartoum_img398 – APLS = 0.897
31
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Ground Truth
Raw Proposal Mask
Extract Entire Khartoum Road Network•Combined BASISS & Albu’s Implementation (w/ Extra Post-Processing from CosmiQ Works)
o Image Size (Pixels) 55,420 x 161,258 (9 terapixels)
o Image Size (Km) 16 x 48
o File Size (GB) 89
o Nodes 195,938
o Edges 258,655
Total Processing Time = 6.3 Hours (Single GPU/CPU)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
SpaceNet 5: Expanding Upon Routing
33
Challenge Participants Will Be Asked to Infer:
… From a Single Satellite Image
Road
Networks
Routing
Information
Travel
Times
The SpaceNet 5 Challenge is Scheduled to Launch in September 2019
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
APLS = 0.81
Multiclass Baseline
34
•Trained resnet34 + unet segmentation model
o Use 7– channel training masks
RGB Image Labels Projection
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
SpaceNet 6: Preliminary Planning
35
Model Deployability / Generalizability
New Applications Beyond Foundational Mapping
Sensor & Data Fusion
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Information Channels
37
CosmiQ’s Repo:https://github.com/CosmiQ
SpaceNet’s Repo:https://github.com/SpaceNetChallenge
SpaceNet Competition Hosting Site
https://www.topcoder.com/spacenet
SpaceNet Data on AWShttps://registry.opendata.aws/spacenet/
CosmiQ’s Blog:
The DownLinQhttps://medium.com/the-downlinq
CosmiQ’sTwitter:
@CosmiQWorks
Title: Training_Data
Found on: Apple Podcasts, Google Play, Spotfiy, Stitcher,
& SoundCloud
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R S U M M I T
Ryan [email protected]