we trained a neural network on satellite imagery to predict wealth from space

What does money look like from space?(here comes the neighborhood)

The Training ModelSatellite Imagery From GBDX

The Training ModelCensus Tracts (2013)

Quartiles

The Training Model

Quartile 0< $34,176

Quartile 1$34,177 - $49,904

Quartile 2$49,905 - $71,875

Quartile 3$71,876+

Satellite Imagery & Census Data

The Training Model

Centroids

The Training Model

Centroid Outlines

The Training Model


The Training Model (Resnet 50)

Satellite Imagery & Census Data Neural Network


Satellite Imagery & Census Data Output

=

Neural Network


Q0 = 91%Q1 = 5.64%Q2 = 2.55%Q3 = .41%

Satellite Imagery & Census Data Output

=

Neural Network


The model carrieswhat it has learned and

repeats the process.

Q0 = 91%Q1 = 5.64%Q2 = 2.55%Q3 = .41%


Input Normalization Normalization Normalization MergeActivationActivationConvolution Convolution Convolution

Neural Network

The data is fed into the model.


The Training Model (Resnet 50)Neural Network

A number of filters are applied to the image.



The resulting new values are normalized to be within learned mean and standard deviations of the dataset.



Separates out features from important and non important ones.



Repeat.



Results are merged with previous iteration.

The Training Model (Resnet 50)TensorBoard data from CMU

So now you have a Classifier

Centroid Outlines

Classified Approximate

What is the classifier seeing?

NYC Areas Classified as Q0

NYC Areas Classified as Q1 & Q2

NYC Areas Classified as Q3

Classification

Confidence 97% Q0 3% Q10% Q20% Q3

Classification

Confidence 92% Q0 7% Q10% Q20% Q3

Classification

Confidence 91% Q0 5% Q12% Q2.4% Q3

Classification

Confidence 2% Q02% Q16% Q290% Q3

Classification

Confidence 0% Q0.1% Q10% Q299% Q3

“Despite this encouraging process, there is still little insight into the internal operation and behavior of these complex models, or how they achieve such good performance. From a scientific standpoint, this is deeply unsatisfactory. Without clear understanding of how and why they work, the development of better models is reduced to trial-and-error.”

Visualizing and Understanding Convolutional Networks - Matthew D. Zeiler, Dept. of Computer Science, Courant Institute, New York University - Rob Fergus, Dept. of Computer Science, Courant Institute, New York University

Baseball Field Experiment

Confidence 91% Q0 5% Q12% Q2.4% Q3

Original Image


Confidence 79% Q0 (-12)12% Q1 (+7)7% Q2 (+5).8% Q3 (+.4)

Added Trees


Confidence 63% Q0 (-28)19% Q1 (+14)13% Q2 (+11)3% Q3 (+2.6)

Added More Trees


Confidence 68% Q0 (-23)17% Q1 (+12)11% Q2 (+9)2% Q3 (+1.6)

And Added More Trees


Confidence 44% Q0 (-47)12% Q1 (+7)23% Q2 (+21)18% Q3 (+17.6)

Added All The Trees

Tree Experiment


Original Image

Tree Experiment


Tree Experiment

Confidence 77% Q0 (-2)10% Q1 (0)6% Q2 (+1)5% Q3 (0)

Tree Experiment 2


Original Image

Tree Experiment 2


Tree Experiment 2

Confidence 87% Q0 (-2)11% Q1 (+1)0% Q2 (0)0% Q3 (0)

Trump Tower Experiment

Confidence 0% Q0.1% Q10% Q299% Q3

Original Image


Confidence 16% Q0 (+16)25% Q1 (+24.9)11% Q2 (+11)45% Q3 (-54)

Added Grass


Confidence 34% Q0 (+34)47% Q1 (+46.9)6% Q2 (+6)10% Q3 (-89)

Built a Wall

Conclusions:• It's possible to predict income levels

from space• Underlying data can provide valuable

assistance to complex neural networks• Human-based empirical inquiry has legs• Teasing out why it knows what it knows

is interesting

Questions:• What does this thing do when you point

it at other cities?• What are the similarities and differences

between cities from space?• Can we construct a model to account for

seasonal variance?• Can we construct a model to account for

architectural difference?

cmu.edu

Who

stamen.com amantiwari.comgbdx.geobigdata.io

Carnegie MellonStamen Design Aman TiwariDigital Globe

Thanks

(@stamen rules)

we trained a neural network on satellite imagery to predict wealth from space

Data & Analytics