deep learning explained - bgumcv172/wiki.files/2017-06...deep learning explained dolev pomeranz,...

Deep Learning ExplainedDolev Pomeranz, Chief Architect

Trax @ BGU

Intro to Trax

Traxs’ Business Application

Manual Audit

Slow and ExpensiveInconsistent untraceable

Fast and CheapConsistentTraceable

Trax Automatic Recognition Audit

‘Big Data’ for retail

AVAILABILITY SHARE OFSHELF

PRICING PROMOTIONALACTIVATIONS

COMPETITVEINSIGHTS

PLANOGRAMCOMPLIANCE

SHELF STANDARDS

Trax unlocks ‘Big Data’ for the retail industry

Scale of coverage Scale of the Data

• Visits• Scenes

• Images• Products

Welcome to Trax Universe

Trax’s Visual Challenges

# Classes

Fine-Grained Classification

Crowded Scene

Dynamic Classes

Trax AI Retail InfrastructureImages Actionable Insights

Trax AI Retail Infrastructure

Computational

Data Science

Models

Data Science

Engine

Deep Fine-Grained

Recognition Engine

Learning

Computational

Retail ‘Big Data’

Inside the Trax AI Retail Infrastructure

AI – Quest for learning

What can we learn

CV NLP SR RL

The AI revolution

AI & Machine learning

What is learning?

𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 → 𝐺𝑒𝑛𝑒𝑟𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛

The power of Math

Source http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html

→𝐺𝑒𝑛𝑒𝑟𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛 𝐿𝑜𝑐𝑎𝑙𝑖𝑡𝑦 → 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒

Introduction to ‘Neural networks’

Biologically inspired?

The real Neuron

Simple and Complex cells (1981 Nobel – Hubel, Wiesel)

Place and Grid cells (2014 Nobel - O'Keefe, Moser, Moser)

Concept cell – Luke Skywalker cell

History of Neural Networks

• 1940’s – Complex behavior from a network of simple units [Hebb]

• 1950’s – Perceptron [Rosenblatt]

– Can implement NAND (universal gate)

Neural Networks

• The Neuron

𝑧 ∈ ℝ ֜ 𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔

𝑧 = 𝑓 𝑊 ∙ 𝑋 + 𝑏

Neural Networks

Activation functions

𝑧 =1

1 + 𝑒− 𝑊∙𝑋+𝑏

𝑆𝑖𝑔𝑚𝑜𝑖𝑑

𝑧 = tanh 𝑊 ∙ 𝑋 + 𝑏

𝑇𝑎𝑛ℎ

𝑧 = max 0,𝑊 ∙ 𝑋 + 𝑏

𝑅𝑒𝑙𝑢

Neural Networks

• MLP - multilayer perceptron (fully connected)

Deep learning explained

Why should we explain?

Source: https://www.eff.org/ai/metrics

State of CV and AI just before DL

Andrej Karpathy blog

http://karpathy.github.io/ 2012/10/22etats/-fo-retupmoc-noisiv/

Deep Learning

Main properties

Deep & Huge

Architecture

Single

Internet & MobileAugmentation

Backpropagation

Transfer learning

Training

Data – Internet & Mobile

Pope Francis Pope Benedict

Data – Underappreciated

Data – Augmentation + Synthesis

https://www.cs.tau.ac.il/~wolf/papers/repcounticcv.pdfhttps://sites.google.com/site/mrsdproject201415teamg/document

/softwarehttps://github.com/udacity/self-driving-car-sim

Data – Transfer Learning

https://medium.com/merantix/applying-deep-learning-to-real-world-problems-

ba2d86ac5837

Model – Architecture

Convolutional layer

Representation learning

Pooling Layer

Model – Deep & Huge

Remember MLP

LeNet [1989]

AlexNet [2012]

(SuperVision)

GoogLeNet [2014]

(Inception)

http://www.topbots.com/a-brief-history-of-neural-network-architectures/

Model – Single VS Pipeline

• Traditional

• Deep

Training

Training – Backpropagation

How to minimize?

𝑓 𝑥 = 𝑦

argmin𝑥

𝑓 𝑥

Method 1 – Random search

Method 2 – Approx. Gradient Decent

𝑑𝑓 𝑥

𝑑𝑥= lim

ℎ→0

𝑓 𝑥 + ℎ − 𝑓 𝑥

𝑑𝑓 𝑥

𝑑𝑥≈𝑓 𝑥 + ℎ − 𝑓 𝑥 − ℎ

Approximating the gradient – numerical analysis

𝐸𝑟𝑟𝑜𝑟:

𝑂 ℎ2

𝐿 =1

𝑖=1

𝐿𝑖 𝑥𝑖 , 𝑦𝑖 ,𝑊 + 𝜆𝑅 𝑊

𝛻𝑊𝐿 =𝑑𝐿 𝑤1𝑑𝑤1

, ⋯ ,𝑑𝐿 𝑤𝑚𝑑𝑤𝑚𝐺𝑟𝑎𝑑𝑖𝑒𝑛𝑡

Back to Neural nets

AlexNet → 𝑚 = 65,000,000

Remember the huge data set

→ 𝑁 = 14,000,000

Running time

𝐼𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑆𝑝𝑒𝑒𝑑 = 𝐹𝑜𝑟𝑤𝑎𝑟𝑑 𝑝𝑎𝑠𝑠 𝑠𝑝𝑒𝑒𝑑 ∗ 𝑁𝑒𝑤 𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑠 ∗ 𝐷𝑎𝑡𝑎 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒

Summary so far

Random Approx. GD

Code Simple Simple

New weights

passes

Fast 𝑂(1) Slow 𝑂(𝑚)

Iteration speed Slow 𝑂(𝑁) Slow 𝑂(𝑚 ∗ 𝑁)

Gradient None Approx.

Can we do better?

Calculus recap

The sums, products, and compositions of analytic functions are analytic

Any analytic function is infinitely differentiable

Neural network is a composition of analytical functions

= tanh 𝑊2,1 ∙ tanh 𝑊1,1 ∙ 𝑋 + 𝑏1,1 , tanh 𝑊1,2 ∙ 𝑋 + 𝑏1,2 , ⋯ + 𝑏2,1

Gradients using the Chain rule

𝜕𝐿

𝜕𝑐

𝜕𝐿

𝜕𝑎=𝜕𝐿

𝜕𝑐

𝜕𝑎

𝜕𝐿

𝜕𝑏=𝜕𝐿

𝜕𝑐

𝜕𝑏

Assume known

• Analytic functions– Exploiting structure

• Chain rule – Gradient based only on following neighbor neurons

• Backward pass– Propagating the gradient, caching calculations

Dynamic programming

𝐵𝑎𝑐𝑘𝑝𝑟𝑜𝑝𝑎𝑔𝑎𝑡𝑖𝑜𝑛 = 𝐶ℎ𝑎𝑖𝑛 𝑅𝑢𝑙𝑒 + 𝐷𝑦𝑛𝑎𝑚𝑖𝑐 𝑝𝑟𝑜𝑔𝑟𝑎𝑚𝑚𝑖𝑛𝑔

𝑚𝑎𝑡ℎ 𝑐𝑜𝑚𝑝𝑢𝑡𝑒𝑟 𝑠𝑐𝑖𝑒𝑛𝑐𝑒

Training – SGD

Great improvement but still slow

Remember the huge data set

→ 𝑁 = 14,000,000

Training – SGD

Simple solution − Stochastic Gradient Decent

• Each epoch permutated the data– stochastic

• Each iteration take a small constant subset (i.e., mini-batch)– gradient approximation

Training – SGD

Algorithms comparison

Random Approx. GD Analytic SGD

Code Simple Simple Complex

New weights

passes

Fast 𝑂(1) Slow 𝑂(𝑚) Fast 𝑂(1)

Iteration speed Slow 𝑂(𝑁) Slow 𝑂(𝑚 ∗ 𝑁) Fast 𝑂(1)

Gradient None Approx. Exact

Training – Intermediate summary

Can we do even better?

Simple network classifying MNIST digits

https://cloud.google.com/blog/big-data/ 2017/01nrael/-wolfrosnet-dna-

peed-gninrael-tuohtiw-a-dhp

Working on mini-batches of 100 images

Using algebra matrix notations

Training – GPU

Someone already knows how to multiply matrices fast!

Training – GPU

One GPU to rule them all

Training – Summary

𝐼𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑆𝑝𝑒𝑒𝑑 = 𝐹𝑜𝑟𝑤𝑎𝑟𝑑 𝑝𝑎𝑠𝑠 𝑠𝑝𝑒𝑒𝑑 ∗ 𝑁𝑒𝑤 𝑤𝑒𝑖𝑔ℎ𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑠 ∗ 𝐷𝑎𝑡𝑎 𝑠𝑒𝑡 𝑠𝑖𝑧𝑒

BackpropagationGPU SGD

Recent research

One Model To Learn Them All

[16/6/2017] L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, J. Uszkoreit

See, Hear, and Read: Deep Aligned Representations

[3/6/2017] Y. Aytar, C. Vondrick, A. Torralba

Distill – Research Debt

http://distill.pub/2017/research-debt/

Open source

Open source libraries

This document and the information contained herein is confidential. You must hold this document and any oral information provided

in connection with this document in strict confidence; Any reproduction, retransmission, republication, translation, or other use of, all

or part of this document is expressly prohibited, unless prior written permission has been granted by Trax.

This document is provided for information purposes only for the exclusive use of the recipients to whom it is addressed and the

contents hereof are subject to change without notice. Whilst the information contained herein has been prepared in good faith, it is

not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law,

including implied warranties and conditions of merchantability, fitness for a particular purpose and/or in relation to the accuracy,

completeness or reliability of the information contained in this document or any oral information provided in connection herewith, or

any data it generates. Trax accepts no responsibility, obligation or liability (whether direct or indirect, in contract, tort or otherwise) in

relation to any of such information. Trax expressly disclaim any and all liability may be based on this document and any errors

therein or omissions therefrom; and no contractual obligations are formed either directly or indirectly by this document.

Trax, the Trax logo and other all other Trax trademarks, logos and service marks used in this document are the trademarks or

service marks of Trax and its affiliates. All other marks contained herein are the property of their respective owners. Trax has

intellectual property rights relating to technology that is described in this document.

We capture retail today,

So you can see the opportunity of tomorrow

deep learning explained - bgumcv172/wiki.files/2017-06...deep learning explained dolev pomeranz,...

Documents

fsps and dei explained -...

an advanced greedy square jigsaw puzzle solver dolev...

“helping job hunters navigate health insurance and find...

search queries explained – a deep dive into query rules,...

pomeranz no taxation without information...

dina pomeranz agosto 2011 - hbs.edu files/metodos-de... ·...

57 sweet memories for margaret pomeranz michael …october...

artificial intelligence - etc.at · artificial intelligence...

detecting malware with memory...

evading(deep(inspection(for fun(and(shell(...•...

deep learning neural networks and ai explained

deep learning explained to expect deep learning basics (6...

recurrent ladder networks...graphical models (see, e.g.,...

minerals explained v—some ore minerals · minerals...

copyright © hal pomeranz hal pomeranz hal@deer-run.com...

main laptop parts explained partiiop parts explained partii

kenneth pomeranz department of history, 200 … · kenneth...

cs7015 (deep learning) : lecture...

ПО БОЖЬЕМУ СЛЕДУ -...

deep belief networks learning advanced...