![Page 1: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/1.jpg)
Self-Supervised
Representation Learning
May 1, 2020
Seokho Moon
![Page 2: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/2.jpg)
논문, 블로그, 발표자료및다른연구원분들(도형록, 강현구등)의
도움을받아작성되었음을미리알려드리며
감사의말을전합니다.
https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html
https://hoya012.github.io/blog/Self-Supervised-Learning-Overview/
PR-208: Unsupervised Visual Representation Learning Overview: Toward Self-Supervision
[Self-supervised learning] AI프렌즈세미나발표자료 -서정훈
![Page 3: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/3.jpg)
3
1. Introduction
1. Background
2. Brief framework
2. Pretext tasks
Exemplar
Context Prediction(Relative Patch Location)
Jigsaw Puzzle
Image Colorization
Count
Rotation
3. Downstream task evaluation
Transfer learning
4. SimCLR
Paper explanation
Results
5. Conclusion
Comments
![Page 4: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/4.jpg)
4
![Page 5: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/5.jpg)
5
![Page 6: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/6.jpg)
6
![Page 7: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/7.jpg)
7
![Page 8: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/8.jpg)
8
![Page 9: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/9.jpg)
9
![Page 10: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/10.jpg)
11
![Page 11: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/11.jpg)
12
![Page 12: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/12.jpg)
13
![Page 13: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/13.jpg)
14
![Page 14: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/14.jpg)
15
![Page 15: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/15.jpg)
16
![Page 16: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/16.jpg)
17
![Page 17: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/17.jpg)
18
![Page 18: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/18.jpg)
19
Pretext task
![Page 19: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/19.jpg)
20
Pretext task
![Page 20: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/20.jpg)
21
![Page 21: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/21.jpg)
22
![Page 22: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/22.jpg)
23
![Page 23: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/23.jpg)
24
![Page 24: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/24.jpg)
25
![Page 25: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/25.jpg)
26
![Page 26: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/26.jpg)
27
![Page 27: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/27.jpg)
28
![Page 28: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/28.jpg)
29
![Page 29: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/29.jpg)
30
![Page 30: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/30.jpg)
31
![Page 31: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/31.jpg)
32
![Page 32: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/32.jpg)
33
Linear Classifier
![Page 33: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/33.jpg)
34
![Page 34: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/34.jpg)
35
![Page 35: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/35.jpg)
36
![Page 36: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/36.jpg)
37
![Page 37: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/37.jpg)
38
![Page 38: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/38.jpg)
39
![Page 39: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/39.jpg)
40
![Page 40: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/40.jpg)
41
![Page 41: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/41.jpg)
42
![Page 42: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/42.jpg)
43
![Page 43: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/43.jpg)
44
![Page 44: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/44.jpg)
45
![Page 45: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/45.jpg)
46
![Page 46: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/46.jpg)
47
![Page 47: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/47.jpg)
48
![Page 48: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/48.jpg)
49
![Page 49: Self-Supervised Representation Learningdmqm.korea.ac.kr/uploads/seminar/200501_Open_Seminar... · 2020. 5. 1. · Yann LeCun 201914 O I now call it "self-supervised learning", because](https://reader033.vdocuments.mx/reader033/viewer/2022060903/609ef72479a5793cba54f531/html5/thumbnails/49.jpg)
Thank you