deep convolutional neural networks - computer...
TRANSCRIPT
-
Deep convolutional neural networks
June 2nd, 2015
Yong Jae LeeUC Davis
Many slides from Rob Fergus, Svetlana Lazebnik, Jia-Bin Huang
-
Announcements Post questions on Piazza for review-session Course evaluation
2
-
Overview Background Convolutional Neural Networks (CNNs) Understanding and Visualizing CNNs Applications Packages
-
Traditional Recognition Approach
Hand-designedfeature extraction
Trainableclassifier
Image/ VideoPixels
Features are not learned Trainable classifier is often generic (e.g. SVM)
ObjectClass
-
Traditional Recognition Approach Features are key to recent progress in recognition Multitude of hand-designed features currently in use
SIFT, HOG, .
Where next? Better classifiers? Or keep building more features?
Felzenszwalb, Girshick, McAllester and Ramanan, PAMI 2007
Yan & Huang (Winner of PASCAL 2010 classification competition)
-
Features are key to recent progress in recognition
SIFT [Loewe IJCV 04] HOG [Dalal and Triggs CVPR 05]
SPM [Lazebnik et al. CVPR 06] DPM [Felzenszwalb et al. PAMI 10]
Color Descriptor [Van De Sande et al. PAMI 10]
https://www.cs.ubc.ca/%7Elowe/papers/ijcv04.pdfhttp://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdfhttp://www.di.ens.fr/sierra/pdfs/cvpr06b.pdfhttp://www.cs.berkeley.edu/%7Erbg/latent/https://staff.fnwi.uva.nl/th.gevers/pub/GeversPAMI10.pdf
-
What about learning the features?
Learn a feature hierarchy all the way from pixels to classifier
Each layer extracts features from the output of previous layer
Layers have (nearly) the same structure
Train all layers jointly
Layer 1 Layer 2 Layer 3 Simple Classifier
Image/ VideoPixels
-
Shallow vs. deep architectures
Hand-designedfeature extraction
Trainableclassifier
Image/ VideoPixels
ObjectClass
Layer 1 Layer N Simple classifierObject Class
Image/ VideoPixels
Traditional recognition: Shallow architecture
Deep learning: Deep architecture
-
Biological neuron and Perceptrons
A biological neuron An artificial neuron (Perceptron) - a linear classifier
-
Simple, Complex and Hyper-complex cells
David H. Hubel and Torsten Wiesel
David Hubel's Eye, Brain, and Vision
Suggested a hierarchy of feature detectors in the visual cortex, with higher level features responding to patterns of activation in lower level cells, and propagating activation upwards to still higher level cells.
video
http://hubel.med.harvard.edu/https://www.youtube.com/watch?v=Yoo4GWiAx94
-
Hubel/Wiesel Architecture and Multi-layer Neural Network
Hubel and Weisels architecture Multi-layer Neural Network- A non-linear classifier
-
Multi-layer Neural Network A non-linear classifier Training: find network weights w to minimize the
error between true training labels and estimated labels
Minimization can be done by gradient descent provided f is differentiable
This training method is called back-propagation
http://en.wikipedia.org/wiki/Backpropagation
-
Neocognitron [Fukushima, Biological Cybernetics 1980]
Deformation-Resistant Recognition
S-cells: (simple)- extract local features
C-cells: (complex)- allow for positional errors
http://www.cs.princeton.edu/courses/archive/spr08/cos598B/Readings/Fukushima1980.pdf
-
Convolutional Neural Networks (CNN, Convnet) Multi-layer Neural network with
- Local connectivity- Shared weight parameters across spatial positions
Stack multiple stages of feature extractors Higher stages compute more global, more
invariant features Classification layer at the end
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86(11): 22782324, 1998.
LeNet-1 from 1993
http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
-
Feed-forward feature extraction: 1. Convolve input with learned filters2. Non-linearity 3. Spatial pooling 4. Normalization
Supervised training of convolutional filters by back-propagating classification error
Input Image
Convolution (Learned)
Non-linearity
Spatial pooling
Normalization
Convolutional Neural Networks (CNN, Convnet)
Feature maps
-
1. Convolution Dependencies are local Translation invariance Few parameters (filter weights) Stride can be greater than 1
(faster, less memory)
Input Feature Map
.
.
.
-
2. Non-Linearity
Per-element (independent) Options:
Tanh Sigmoid: 1/(1+exp(-x)) Rectified linear unit (ReLU)
Simplifies backpropagation Makes learning faster Avoids saturation issues Preferred option
-
3. Spatial Pooling Sum or max Non-overlapping / overlapping regions Role of pooling:
Invariance to small transformations Larger receptive fields (see more of input)
Max
Sum
-
4. Normalization Within or across feature maps Before or after spatial pooling
Feature MapsFeature Maps
After Contrast Normalization
-
Compare: SIFT Descriptor
Applyoriented filters
Spatial pool (Sum)
Normalize to unit length
Feature Vector
Image Pixels
Lowe[IJCV 2004]
-
Compare: Spatial Pyramid Matching
Filter with Visual Words
Multi-scalespatial pool (Sum)
Take max VW response
Global image
descriptor
Lazebnik, Schmid,
Ponce [CVPR 2006]
SIFT features
-
Convnet Successes
Handwritten text/digits MNIST (0.17% error [Ciresan et al. 2011]) Arabic & Chinese [Ciresan et al. 2012]
Simpler recognition benchmarks CIFAR-10 (9.3% error [Wan et al. 2013]) Traffic sign recognition
0.56% error vs 1.16% for humans [Ciresan et al. 2011]
But until recently, less good at more complex datasets Caltech-101/256 (few training examples)
-
ImageNet Challenge 2012
[Deng et al. CVPR 2009]
~14 million labeled images, 20k classes
Images gathered from Internet
Human labels via Amazon Turk
ImageNet Challenge: 1.2 million training images, 1000 classes
A. Krizhevsky, I. Sutskever, and G. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012
http://www.cs.toronto.edu/%7Efritz/absps/imagenet.pdf
-
ImageNet Challenge 2012 AlexNet: Similar framework to LeCun98 but:
Bigger model (7 hidden layers, 650,000 units, 60,000,000 params) More data (106 vs. 103 images) GPU implementation (50x speedup over CPU)
Trained on two GPUs for a week Better regularization for training (DropOut)
A. Krizhevsky, I. Sutskever, and G. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012
http://www.cs.toronto.edu/%7Efritz/absps/imagenet.pdf
-
Using CNN for Image Classification
car
AlexNet
Fixed input size: 224x224x3
-
ImageNet Challenge 2012 Krizhevsky et al. -- 16.4% error (top-5) Next best (non-convnet) 26.2% error
0
5
10
15
20
25
30
35
SuperVision ISI Oxford INRIA Amsterdam
Top-
5 er
ror r
ate
%
-
ImageNet Challenge 2012-2014Team Year Place Error (top-5) External data
SuperVision Toronto(7 layers)
2012 - 16.4% no
SuperVision 2012 1st 15.3% ImageNet 22k
Clarifai NYU (7 layers) 2013 - 11.7% no
Clarifai 2013 1st 11.2% ImageNet 22k
VGG Oxford (16 layers) 2014 2nd 7.32% no
GoogLeNet (19 layers) 2014 1st 6.67% no
Human expert* 5.1%
Team Method Error (top-5)
DeepImage - Baidu Data augmentation + multi GPU 5.33%
PReLU-nets - MSRA Parametric ReLU + smart initialization 4.94%
BN-Inception ensemble - Google
Reducing internal covariate shift 4.82%
Best non-convnet in 2012: 26.2%
http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
-
Understanding and Visualizing CNN Visualize input pattern using deconvnet Individual neuron activation Fooling CNNs
-
Map activation back to the input pixel space
What input pattern originally caused a given activation in the feature maps?
Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]
http://ftp.cs.nyu.edu/%7Efergus/papers/zeilerECCV2014.pdf
-
Layer 1
Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]
http://ftp.cs.nyu.edu/%7Efergus/papers/zeilerECCV2014.pdf
-
Layer 2
Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]
http://ftp.cs.nyu.edu/%7Efergus/papers/zeilerECCV2014.pdf
-
Layer 3
Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]
http://ftp.cs.nyu.edu/%7Efergus/papers/zeilerECCV2014.pdf
-
Layer 4 and 5
Visualizing and Understanding Convolutional Networks [Zeiler and Fergus, ECCV 2014]
http://ftp.cs.nyu.edu/%7Efergus/papers/zeilerECCV2014.pdf
-
Occlusion Experiment
Mask parts of input with occluding square
Monitor output
-
Input image
p(True class) Most probable class
-
Input image
p(True class) Most probable class
-
Input image
p(True class) Most probable class
-
Individual Neuron Activation
RCNN [Girshick et al. CVPR 2014]
http://www.cs.berkeley.edu/%7Erbg/papers/r-cnn-cvpr.pdf
-
Individual Neuron Activation
RCNN [Girshick et al. CVPR 2014]
http://www.cs.berkeley.edu/%7Erbg/papers/r-cnn-cvpr.pdf
-
Individual Neuron Activation
RCNN [Girshick et al. CVPR 2014]
http://www.cs.berkeley.edu/%7Erbg/papers/r-cnn-cvpr.pdf
-
Fooling CNNs
Intriguing properties of neural networks [Szegedy ICLR 2014]
http://arxiv.org/pdf/1312.6199v4.pdf
-
What is going on? Recall gradient descent training: modify the
weights to reduce classifier error
Adversarial examples: modify the image to increase classifier error
http://karpathy.github.io/2015/03/30/breaking-convnets/Explaining and Harnessing Adversarial Examples [Goodfellow ICLR 2015]
www
E
http://karpathy.github.io/2015/03/30/breaking-convnets/http://arxiv.org/pdf/1412.6572.pdf
-
Fooling CNNs
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images [Nguyen et al. CVPR 2015]
http://arxiv.org/pdf/1412.1897.pdf
-
Images that both CNN and Human can recognize
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images [Nguyen et al. CVPR 2015]
http://arxiv.org/pdf/1412.1897.pdf
-
Direct Encoding
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images [Nguyen et al. CVPR 2015]
http://arxiv.org/pdf/1412.1897.pdf
-
Indirect Encoding
Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images [Nguyen et al. CVPR 2015]
http://arxiv.org/pdf/1412.1897.pdf
-
Beyond classification Detection Segmentation Regression Pose estimation Matching patches Synthesis
and many more
-
R-CNN: Regions with CNN features Trained on ImageNet classification Finetune CNN on PASCAL
RCNN [Girshick et al. CVPR 2014]
http://www.cs.berkeley.edu/%7Erbg/papers/r-cnn-cvpr.pdf
-
Labeling Pixels: Semantic Labels
Fully Convolutional Networks for Semantic Segmentation [Long et al. CVPR 2015]
http://www.cs.berkeley.edu/%7Ejonlong/long_shelhamer_fcn.pdf
-
Labeling Pixels: Edge Detection
DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection [Bertasius et al. CVPR 2015]
http://arxiv.org/pdf/1412.1123.pdf
-
CNN for Regression
DeepPose [Toshev and Szegedy CVPR 2014]
http://arxiv.org/pdf/1312.4659v3.pdf
-
CNN as a Similarity Measure for Matching
FaceNet [Schroff et al. 2015]Stereo matching [Zbontar and LeCun CVPR 2015]Compare patch [Zagoruyko and Komodakis 2015]
Match ground and aerial images [Lin et al. CVPR 2015]FlowNet [Fischer et al 2015]
http://arxiv.org/pdf/1503.03832v1.pdfhttp://arxiv.org/pdf/1409.4326v1.pdfhttp://arxiv.org/pdf/1504.03641.pdfhttp://cs.brown.edu/people/hays/papers/deep_geo.pdfhttp://arxiv.org/pdf/1504.06852.pdf
-
CNN for Image Restoration/Enhancement
Super-resolution [Dong et al. ECCV 2014]
Non-blind deconvolution[Xu et al. NIPS 2014]
Non-uniform blur estimation[Sun et al. CVPR 2015]
http://personal.ie.cuhk.edu.hk/%7Eccloy/files/eccv_2014_deepresolution.pdfhttp://papers.nips.cc/paper/5485-deep-convolutional-neural-network-for-image-deconvolution.pdfhttp://arxiv.org/pdf/1503.00593.pdf
-
CNN for Image Generation
Learning to Generate Chairs with Convolutional Neural Networks [Dosovitskiy et al. CVPR 2015]
http://arxiv.org/pdf/1411.5928.pdf
-
Chair Morphing
Learning to Generate Chairs with Convolutional Neural Networks [Dosovitskiy et al. CVPR 2015]
http://arxiv.org/pdf/1411.5928.pdf
-
Transfer Learning Improvement of learning in a new task through the
transfer of knowledge from a related task that has already been learned.
Weight initialization for CNN
Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks [Oquab et al. CVPR 2014]
http://www.cv-foundation.org/openaccess/content_cvpr_2014/papers/Oquab_Learning_and_Transferring_2014_CVPR_paper.pdf
-
Convolutional activation features
[Donahue et al. ICML 2013]
CNN Features off-the-shelf: an Astounding Baseline for Recognition[Razavian et al. 2014]
http://arxiv.org/pdf/1310.1531.pdfhttp://arxiv.org/pdf/1403.6382.pdf
-
CNN packages Cuda-convnet (A. Krizhevsky, Google) Caffe (Y. Jia, Berkeley)
Replacement of deprecated Decaf Overfeat (NYU) Torch MatConvNet (A. Vedaldi, Oxford)
https://code.google.com/p/cuda-convnet/http://caffe.berkeleyvision.org/https://github.com/UCB-ICSI-Vision-Group/decaf-release/http://cilvr.nyu.edu/doku.php?id=code:starthttp://torch.ch/http://www.vlfeat.org/matconvnet/
-
Resources http://deeplearning.net/
https://github.com/ChristosChristofidis/awesome-deep-learning
http://cs231n.stanford.edu/syllabus.html
http://deeplearning.net/https://github.com/ChristosChristofidis/awesome-deep-learninghttp://cs231n.stanford.edu/syllabus.html
-
Questions?
See you Thursday!
61
Deep convolutional neural networksJune 2nd, 2015AnnouncementsOverviewTraditional Recognition ApproachTraditional Recognition ApproachFeatures are key to recent progress in recognitionWhat about learning the features?Shallow vs. deep architecturesBiological neuron and PerceptronsSimple, Complex and Hyper-complex cellsHubel/Wiesel Architecture and Multi-layer Neural NetworkMulti-layer Neural NetworkNeocognitron [Fukushima, Biological Cybernetics 1980]Convolutional Neural Networks (CNN, Convnet)Convolutional Neural Networks (CNN, Convnet)1. Convolution2. Non-Linearity3. Spatial Pooling4. NormalizationCompare: SIFT DescriptorCompare: Spatial Pyramid MatchingConvnet SuccessesImageNet Challenge 2012ImageNet Challenge 2012Using CNN for Image ClassificationImageNet Challenge 2012Slide Number 27Understanding and Visualizing CNNMap activation back to the input pixel spaceLayer 1Layer 2Layer 3Layer 4 and 5Occlusion ExperimentInput imageInput imageInput imageIndividual Neuron ActivationIndividual Neuron ActivationIndividual Neuron ActivationFooling CNNsWhat is going on?Fooling CNNsImages that both CNN and Human can recognizeDirect EncodingIndirect EncodingBeyond classificationR-CNN: Regions with CNN featuresLabeling Pixels: Semantic LabelsLabeling Pixels: Edge DetectionCNN for RegressionCNN as a Similarity Measure for MatchingCNN for Image Restoration/EnhancementCNN for Image GenerationChair MorphingTransfer LearningConvolutional activation featuresCNN packagesResourcesQuestions?