presented by bingcai zhang, shannon mcdonald , ellen hein ... · • difficult to collect large...

© 2019 BAE Systems. All Rights Reserved. ClearFlite, GXP, GXP OpsView, GXP WebView, GXP Xplorer, SOCET GXP, and SOCET SET are registered trademarks of BAE Systems. This document gives only a general description of the product(s) or service(s) offered by BAE Systems. From time to time, changes may be made in the products or conditions of supply. EXPORT-CONTROLLED DATA. This document contains technical information whose export is governed by the U.S. Export Administration Regulations (EAR). This informationmust not be transferred to a foreign person without proper authorization of the U.S. Department of Commerce. Violations may result in administrative, civil or criminal penalties. E-20190317-65. NOT INTENDED FOR PUBLIC RELEASE.

Presented by Bingcai Zhang, Shannon McDonald , Ellen Hein, and Quoc Toan Luong

Machine Learning tools and techniques

GXP360º Professional Exchange and WorkshopMarch 18-22, 2019 | San Diego, California

© 2019 BAE Systems. All Rights Reserved. EAR CONTROLLED. E-20190317-65. NOT INTENDED FOR PUBLIC RELEASE. Machine Learning tools & techniques | Multiple presenters

Why Machine Learning

• Takes advantage of rapid advances from the research community• Combined with three decades of photogrammetry and computer vision expertise• Achieve near human level accuracy in the geospatial intelligence domain• Supports all types of geospatial imagery and sensor modalities including LiDAR 3-D point cloud

Machine Learning (DeepObject) supports detection of things from imagery and point clouds with unprecedented accuracy.



Five objectives for object detection

• Precision defined as (true positive) / (true positive + false positive)• Recall defined as (true positive) / (true positive + false negative)• Detection speed defined by number of pixels per second• Average positional accuracy defined by distances between centers of bounding box vs. object centers

• Precision object geolocation requires high positional accuracy• Number of positive training examples

• Objects may be rare (how many Su fighter jets in the world?)



Double Convolutional Neural Networks (CNN)

• First CNN with maximum translation invariance• Second CNN with minimum translation invariance• First CNN generates region proposals for second CNN• Second CNN detect objects with high positional accuracy• First CNN is fast with high recall• Second CNN is slow with high precision and high positional accuracy• First CNN and second CNN are trained separately and differently• When objects are very sparse, Double CNN can detect objects with high speed as well as high positional accuracy



Translation invariance vs. positional accuracy

• Network output varies little when input is translated• Training data augmentation• Pooling operations:

• Kernel size• MAX vs. AVE

• Kernel size of convolutional layers• Gradual increase in the receptive field size of neurons in successive convolution layers• Network depth

• Great if positional accuracy is not relevant• Bad when positional accuracy is important



Translation invariance vs. detection speed



Object sparsity vs. detection speed



Object sparsity vs. detection speed …2



Double CNN vs. Single CNN

• Single CNN is the second CNN only• 753,830,400 pixels, TITAN V GPU, Intel® Xeon® CPU E5-2650 v4 @2.20GHz

• Double CNN: 9 minutes

• Single CNN: 35 minutes • Speed up 2.89 times• 1.4 million pixels per second



Positional accuracy and orientation



Double CNN case study metrics

Precision 100.0%

Recall 100.0%

Average positional accuracy 4 pixels

Detection speed 1.4 million pixels per second

Positive training examples 152

Average orientation accuracy 6 degree



Positive training examples

• Difficult to collect large number of positive training examples• How many Su fighter jets in the world?

• To prevent overfitting, CNN model capacity is related to number of training examples• CNN model capacity is related to complexity of detection task• Simplicity learning reduces complexity of detection task, which needs smaller CNN model capacity, which requires smaller

number of training examples1. Color normalization2. Rotation normalization3. Scale normalization


Ellen HeinQuoc Toan Luong

Speeding Up DeepObject

13



Speeding up DeepObjectDeepObject efficiency at inference time

Problem definition:• Inference time refers to a phase when models predict output of real data• Majority of cost are in hardware and power• Optimizing models for this phase best reduces cost of product• Many deep learning tools and frameworks are optimized first for Unix

• Conversion potentially yields low cost upgrades• Enables leveraging of suite of utilities


© 2019 BAE Systems. All Rights Reserved. EAR CONTROLLED. E-20190317-65. NOT INTENDED FOR PUBLIC RELEASE. Machine Learning tools & techniques | Multiple presenters 15

17% improvement on inference time On average, 25% runtime improvement across all processes

Converting to Ubuntu

Experiment performed on machines with two NVidia Titan V cards


© 2019 BAE Systems. All Rights Reserved. EAR CONTROLLED. E-20190317-65. NOT INTENDED FOR PUBLIC RELEASE. Machine Learning tools & techniques | Multiple presenters 16

Moving past simple porting• NVidia deep learning inference optimizer• Lowers inference time and memory usage of

pre-trained models• Enables half precision (FP16) model

initialization to potentially gain a further 4x performance improvement

Leverage TensorRT



At higher batch sizes, TensorRT approaches 2.25x inference speed up

Experiment performed on NVidia Titan X with TensorRT 4

Inference Performance Improvement On Variable Batch Sizes



At higher batch sizes, TensorRT is over 4x efficient

Experiment performed on NVidia Titan X with TensorRT 4

Memory Efficiency On Variable Batch Sizes



The Numbers

• Format: average runtime per inference / GPU memory consumption• Experiment performed on NVidia Titan X with TensorRT 4

Batch Size Caffe TensorRT1 0.005273s/464MiB 0.001226s/261MiB8 0.008546s/571MiB 0.002839s/281MiB32 0.019517s/1004MiB 0.007036s/349MiB64 0.032707s/1505MiB 0.012237s/439MiB128 0.059624s/2329MiB 0.023177s/620MiB256 0.102249s/4202MiB 0.047192s/982MiB512 0.198635s/7947MiB 0.088389s/1705MiB



Past workMulticlass semantic image segmentation

20

Initially, we investigated multiclass semantic image segmentation using the PASCAL VOC 2012 dataset. It was determined that binary semantic segmentation was more than what was needed. Multiclass segmentation also has a noticeably lower accuracy, precision, and recall than binary semantic segmentation.

• Tests using multiclass semantic image segmentation only an accuracy of around 30-35%• Single class accuracy was greatly influenced by how populous a class was in the dataset; For example, the background

class had an accuracy of around 85-90%



Binary semantic image segmentationDeepGlobe building extraction dataset

21

The DeepGlobe Building Detection dataset is a dataset originally from SpaceNet consisting of over 10,000, 650x650, 16-bit, Red, Green, Blue (RGB) and multispectral images and over 200,000 polygons. The goal of this dataset is to extract building footprints from satellite images. The images are taken from 4 different regions: Las Vegas, Nevada, USA; Paris, France; Shanghai, China; and Khartoum, Sudan.

Shanghai, China Las Vegas, USA Khartoum, Sudan Paris, France



Binary semantic image segmentationLinkNet model

22

The LinkNet model architecture is a model created for pixel-wise semantic segmentation. More information on this model architecture can be found in the paper “LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation”, by A. Chaurasia and E. Culurciello.

Compared to the UNet model architecture, LinkNet is a quarter of the size and twice as fast.• Number of Parameters: 11,560,769• Model Size: 44.5 MB• Inference speed on 9 256x256 image chips: 1.1 seconds• Validation F1-Score: 77.6%



Binary semantic image segmentationInitial results 1

23

The DeepGlobe Building Extraction dataset never released the validation set labels, so there was a more limited amount of data available. The images below are from the original validation set (which does not have labels).

Las Vegas, USA Paris, France



Binary semantic image segmentationInitial results 2

24

The DeepGlobe Building Extraction dataset never released the validation set labels, so there was a more limited amount of data available. The images below are from the original validation set (which does not have labels).

Shanghai, China Khartoum, Sudan



Binary Semantic Image SegmentationPossible Applications

25

Possible relevant applications of binary semantic image segmentation:• Detecting areas where buildings are likely.• Classifying ground and not ground (future work).• Classifying land types like forest, water, urban, etc.• Detecting roads, airports, etc. to limit the area DeepObject ‘looks’.



We value your feedback

26

We improve because of your feedback!

Please take a moment to complete the brief paper survey that was provided to you at the beginning of this session.

If you did not get a survey, see your session monitor in the back of the room or your session instructor.

We appreciate your time and suggestions!


Bingcai ZhangTel: 858 592-5218E-mail: [email protected]

Shannon McDonald , Ellen Hein, and Quoc Toan Luong

Thank you!

mailto:[email protected]

presented by bingcai zhang, shannon mcdonald , ellen hein ... · • difficult to collect large...

Documents