Download - Unsupervised feature learning for audio classification using convolutional deep belief networks
![Page 1: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/1.jpg)
Unsupervised feature learning for audio classification using convolutional deep belief net
works
Honglak Lee, Yan Largman, Peter Pham and Andrew Y. Ng
Presented by Bo Chen, 5.7,2010
![Page 2: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/2.jpg)
Outline
• 1. What’s Deep Learning?
• 2. Why use Deep Learning?
• 3. Foundations of Deep Learning
• 4. Convolutional Deep Belief Networks
• 5. Results
![Page 3: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/3.jpg)
Deep Architecture
• Deep architectures: compositions of many layers of adaptive non-linear components.
Difficulty: parameter searching (local minima)
• Deep belief nets: probabilistic generative models that are composed of multiple layers of stochastic, latent variables. (Hinton et al., 2006)
Deep Learning Wiki
![Page 4: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/4.jpg)
Why Use Deep Learning
• Insufficient depth can hurt Usually our experiences tell us that one-layer machine only gives us
a set of general dictionary elements, unless a huge number of dictionary elements.
• The brain has a deep architecture• Cognitive processes seem deep• Learn a feature hierarchies or the complicated fu
nctions that can represent high-level abstractions
For example, PixelsEdgletsMotifsPartsObjectsScenes
Some from Yoshua Bengio’s course notes and Yann Lecun, et.al.,2010
![Page 5: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/5.jpg)
One-layer dictionary
30 16x16 dictionary elementsand reconstructed images
250 16x16 dictionary elementsand reconstructed images
![Page 6: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/6.jpg)
Restricted Boltzmann Machine
Figure from R Salakhutdinov et. al.
Energy functionBinary-valued
Real-valued
Contrastive divergence is used to solve the problem. (Hinton et al., 2006)
![Page 7: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/7.jpg)
Deep Architectures
RBM in the different layers can be independently trained.
![Page 8: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/8.jpg)
Convolutional Network Architecture
Figure from Yann LeCun et. al, 1998
Intuitively, in each layer the weight matrix will catch the most consistent ‘structures’ through all of the images.
![Page 9: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/9.jpg)
3-dimensional Dictionary elements in the second layer
The dictionary element in the second layeris a 3-dimensional matrix.
D: the first-layer dictionary element E: the second-layer dictionary elementS: the convolution of the image and the first-layer elements.
![Page 10: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/10.jpg)
Convolutional RBM with Probabilistic Max-Pooling Layer
Max-pooling Layer
![Page 11: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/11.jpg)
Convolutional Deep Belief Networks
: the weight matrixConnecting poolingunit Pk to detection unit H’l.
![Page 12: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/12.jpg)
Results on Natural Images
![Page 13: Unsupervised feature learning for audio classification using convolutional deep belief networks](https://reader036.vdocuments.mx/reader036/viewer/2022081420/56815120550346895dbf3e11/html5/thumbnails/13.jpg)
Results Caltech101 Images