one-shot learning
TRANSCRIPT
![Page 1: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/1.jpg)
One Shot Learning
SK Telecom Video Tech. Lab.김지성 Manager
![Page 2: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/2.jpg)
Contents
Why do we need one-shot learning?
What is one-shot learning?
How to do “one-shot learning”
Recap
![Page 3: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/3.jpg)
종합기술원
미래기술원
Lab Introduction using Backpropagation
Video Tech. Lab
![Page 4: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/4.jpg)
One Shot Learning: Learning a class froma single labelled example
![Page 5: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/5.jpg)
Giraffe Classification
VS
어린이는 한 장의 기린 사진을 통해기린이라는 Class의 Concept을
학습 가능함
DNN은 한 class를학습하기 위해
수백장 or 수천장필요함
![Page 6: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/6.jpg)
Korean Food Classifier
![Page 7: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/7.jpg)
CNN Architecutre
…
1. Batch Size (ex. 256장)씩 학습한다. (120만장씩 90번)2. 최적화는 Gradient based Optimization
( Extensive, Incremental Training on Large DB )3. 새로운 Image에 대한 학습 반영이 느리다.4. 그러므로 One-Shot Learning에 적합하지 않다!
![Page 8: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/8.jpg)
Machine Learning Principle: Test and Train conditionsMust Match.
![Page 9: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/9.jpg)
Matching Networks Architecutre (deepmind, ‘16)
1. Test 할 때 1장만 보여주면서 학습할 때 수백장 보여주는게 좋을까?2. 학습할 때도 한 class당 한장씩 만 보여주어야 한다! (or 5장)3. g, f 는 입력을 embedding 하는 neural network (VGG, Inception)4. Embedding을 통해 나온 feature vector들 사이의 유사도를 구한다. 5. 유사도의 weighted sum을 통해 최종 Test label을 결정
![Page 10: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/10.jpg)
Matching Networks Architecutre
: Test Example
Attention by Softmax & Cosine distance
Candidate : Chihuahua
Test Input : Shepherd
Candidate : Retriever
Candidate : Siberian Husky
Candidate : Shepherd
fc7 : 4096 dim
CNN
CN
N
: Test Label: Training Set
![Page 11: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/11.jpg)
Matching Networks Architecutre
: Test Example
It’s differentiable! End to End Nearest Neighbour Classifier
Candidate : Chihuahua
Test Input : Shepherd
Candidate : Retriever
Candidate : Siberian Husky
Candidate : Shepherd
fc7 : 4096 dim
CNN
CN
N
: Test Label: Training Set
CN
N
CN
N
CN
N
Bi-LSTM
LSTM
![Page 12: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/12.jpg)
Matching Networks Architecutre
CN
NCN
NCN
NCN
N
Bi-LSTM
fc7 : 4096 dim
fc7 : 4096 dim
fc7 : 4096 dim
fc7 : 4096 dim
![Page 13: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/13.jpg)
Matching Networks Result
![Page 14: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/14.jpg)
Recap
Why do we need one-shot learning?If there is a few data for training/testing
What is one-shot learning?Learning a class from a single labelled example
How to do “one-shot learning”Start with Omniglot Example
import tensorflow as tf
![Page 15: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/15.jpg)
Face Recognition
![Page 16: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/16.jpg)
Appendix
![Page 17: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/17.jpg)
One-Shot Learning
How can we learn a novel concept from a few examples? (random guess will be 1/N)
![Page 18: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/18.jpg)
Korean
Problem Setting - Training
![Page 19: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/19.jpg)
Problem Setting - Training
Greek
![Page 20: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/20.jpg)
Problem Setting - Testing
1 Angelic
2 Oriya
3 Tibetan
4 Keble
5 ULOG
1 Angelic ?
2 Oriya
3 Tibetan ?
4 Keble
5 ULOG? = 1/N
한번은 정답을 알려준다. (One-shot)
새로운 Character에 대해 Test (5지 선다)
![Page 21: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/21.jpg)
Recent Papers about One Shot Learning
One-shot Learning with Memory-Augmented Neural Networksa) Submitted on : 16.05.19b) Written by : Deep Mindc) Link : https://arxiv.org/abs/1605.06065
Matching Networks for One Shot Learninga) Submitted on : 16.06.13b) Written by : Deep Mindc) Link : https://arxiv.org/abs/1606.04080
![Page 22: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/22.jpg)
One Shot Learning withMemory AugmentedNeuralNetwork
![Page 23: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/23.jpg)
MANN Architecture
RAM
CPU
![Page 24: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/24.jpg)
Neural Turing Machine vs Memory Augmented Neural Network
1. MANN은 NTM의 변형이다.2. Controller는 Feed Forward NN or LSTM을 사용하였다.3. 기존 NTM은 복사, 정렬등의 알고리즘을 학습하는데 사용하였지만 이
논문에서는 One-shot Learing에 사용4. 빠른 학습을 위해 Memory에 Write할 때 가장 적게 사용된 메모리 or
가장 최근에 사용된 메모리에 Write함 (LRUA)5. 그렇다면 기본적인 질문?! 왜 Augmented Memory가 필요할까?
![Page 25: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/25.jpg)
Why do we need Differentiable Memory?
잘 알려진 것 처럼 CNN is good for spatial structure (Image, … )
RNN is good for temporal structure (Audio, …)
![Page 26: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/26.jpg)
Augmenting Neural Nets with a Memory Module
CNN + RNN is good for spatiotemporal structure (Video, …)
그렇다면 아주 간단한 Question & Answering 문제에서도CNN 및 RNN or 그 조합이 잘 동작하는가?
![Page 27: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/27.jpg)
Augmenting Neural Nets with a Memory Module
![Page 28: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/28.jpg)
Augmenting Neural Nets with a Memory Module
bAbI task Example Story
Sam moved to the garden.Mary left the milk.John left the football.Daniel moved to the garden.Sam went to the kitchen.Sandra moved to the hallway.Mary moved to the hallway.Mary left the milk.Sam drops the apple there.
Q : Where was the apple after the garden?
![Page 29: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/29.jpg)
Augmenting Neural Nets with a Memory Module
bAbI task Example Story
Sam moved to the garden.Mary left the milk.John left the football.Daniel moved to the garden.Sam went to the kitchen.Sandra moved to the hallway.Mary moved to the hallway.Mary left the milk.Sam drops the apple there.
Q : Where was the apple after the garden?
![Page 30: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/30.jpg)
Augmenting Neural Nets with a Memory Module
bAbI task Example Story
Sam moved to the garden.Mary left the milk.John left the football.Daniel moved to the garden.Sam went to the kitchen.Sandra moved to the hallway.Mary moved to the hallway.Mary left the milk.Sam drops the apple there.
Q : Where was the apple after the garden?
![Page 31: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/31.jpg)
Augmenting Neural Nets with a Memory Module
bAbI task Example Story
Sam moved to the garden.Mary left the milk.John left the football.Daniel moved to the garden.Sam went to the kitchen.Sandra moved to the hallway.Mary moved to the hallway.Mary left the milk.Sam drops the apple there.
Q : Where was the apple after the garden?
![Page 32: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/32.jpg)
Augmenting Neural Nets with a Memory Module
즉! Data가 out-of-order인 경우 기존 CNN, RNN으로는 잘 해결되지 않는다.현재 아래 구조의 data들에 대해서는 아직 다양한 노력 中
1. Out-of-order access (순차적이지 않은 접근)2. 장기 기억3. 정렬되지 않은 data
![Page 33: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/33.jpg)
Differentialble Memory : MemN2N
1. 단어를 one-hot vector로 변형한 뒤 A, B, C matrix를 통해 문장을 메모리화2. Embedding 된 질문 문장(u)과의 inner product를 통해 Attention 결정3. 구해진 attention과 입력 Output Memory와의 weighted sum을 통해
output matrix o 구함4. 최종적으로 Softmax(W(u+o))를 고려해 답을 도출
![Page 34: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/34.jpg)
MANN - Task Setup
1. : Training Set2. 입력이미지는 105 x 105 20 x 20 400 x 1 로 flatten3. Time off-set labels 을 사용 (x_t, y_t-1) 하여 현재 결과가 다음
입력으로 들어가도록 구성
![Page 35: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/35.jpg)
MANN - Network Strategy
1. Episode = Task 학습에 사용한 Sequence (x_t, y_t-1)2. 1 episode의 길이는 5 classes 일 경우에는 50, 10 classes일
경우에는 1003. 학습은 이런 episode를 10만번 수행 하여 결과 도출
![Page 36: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/36.jpg)
MANN - Network Strategy
1. Time off-set labels 을 사용 (x_t, y_t-1) 하여 현재 결과가 다음입력으로 들어가도록 구성
2. External Memory에는 (image, label) 쌍으로 저장해 놓는다.3. External Memory에 새로 입력된 이미지와 유사도가 큰 이미지가
있으면 Read해서 결과를 도출한다.
![Page 37: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/37.jpg)
Omniglot classification - LSTM (5 way)
![Page 38: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/38.jpg)
Omniglot classification - MANN (5 way)
![Page 39: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/39.jpg)
Omniglot classification - LSTM (15 way)
![Page 40: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/40.jpg)
Omniglot classification - MANN (15 way)
![Page 41: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/41.jpg)
Human vs MANN
1. 사람에게도 동일하게 한장의 사진을 보여준 뒤 1-5 사이의 숫자를선택하게 하고 매 Instance 마다 정답을 알려줄 경우 1-shot의경우 57.3% 정확도
2. 하지만 MANN은 82.8 % 정확도3. 앞에서 설명한 Matching Net은 98.1% 정확도
![Page 42: One-Shot Learning](https://reader034.vdocuments.mx/reader034/viewer/2022042619/587dd4981a28ab5b5b8b6025/html5/thumbnails/42.jpg)
Summary
1. Internal Memory (CNN, RNN)는 새로운 information을Internal Memory(Weights) 에 반영하는 속도가 느림.
2. External Memory (MemN2N, NTM)을 사용하면 빠르게새로운 정보를 저장하고 검색하는 것이 가능함. (MANN)
3. Deep Neural Feature에 대한 Metric Learning 과 External Memory의 아이디어를 결합하여 Matching Network를제안함
4. 학습에 필요한 이미지가 충분하지 않을 경우 큰 장점이있지만 반대의 경우에는 State-of-the Art 대비 열세