semantic image search - cloud object storageimag… · reverse image search in postgresql using...
TRANSCRIPT
![Page 1: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/1.jpg)
Semantic Image SearchAlex Egg
![Page 2: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/2.jpg)
InspirationKrizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
![Page 3: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/3.jpg)
Deep CNNc) First Pooling Layer
d) second to last FC Layer
Decaf
C1 C2 C3 C4 C5 FC6 FC7 FC8
Feature Generator Classifier
AlexNet 2012
![Page 4: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/4.jpg)
Nearest Neighbor SearchImages in 2D space
![Page 5: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/5.jpg)
Semantic SearchReverse Image Search: Image A -> Identical image of A
Semantic Image Search: Image A -> Any images containing A
The word "semantic" refers to the meaning or essence of something
● Text Semantics: Sentiment & Meaning● Image Semantics: Object Quantification
![Page 6: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/6.jpg)
Theoretical Implementation1. Setup DCNN (Image Feature Generator)2. Setup Database3. Index feature vectors in database4. Query database using 1-Nearest Neighbor Search
![Page 7: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/7.jpg)
![Page 8: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/8.jpg)
Theoretical Impl. Problems
● Accuracy: Distance measure breakdown in high dimensions
● Scalability: Storage & Nearest Neighbor intractabilityNaive Solution
![Page 9: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/9.jpg)
Problem: Accuracy● AlexNet: 4096D● VGG: 4096D● Inception V3: 2048D● Inception V4: 1536D
After about 10D all points are equally far away -- distance measure break down: Curse of Dimensionality
![Page 10: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/10.jpg)
Problem: Scalability Storage Complexity: 4096x32 bits float = 16.38 KB/vector x 1M images = 16.38 GB
Computational Complexity: 1NN = O(n); 1ms*1e6=16.6m
We can mitigate most of these problems using a modern database system.
![Page 11: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/11.jpg)
Solutions● Scalability: Reduce search
space ● Accuracy: Reduce
Dimensionality
![Page 12: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/12.jpg)
Dimensionality Reduction● PCA: Directions of projection are data-dependent● Random Projections: Direction of projections are data-independent
1. Data is so high dimensional that it is too expensive to compute PCA2. You don’t have access to the data all at once, as in streaming
![Page 13: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/13.jpg)
![Page 14: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/14.jpg)
/\
2
![Page 15: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/15.jpg)
/\ /\/\
4
![Page 16: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/16.jpg)
/\ /\/\/\/\
6
![Page 17: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/17.jpg)
/\ /\/\ /\/\/\/\
8
![Page 18: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/18.jpg)
1
2
3
4
5
6
7
8
/\ /\/\ /\/\/\/\
8
![Page 19: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/19.jpg)
000
001
010
011
100
101
110
111
/\ /\/\ /\/\/\/\
8
![Page 20: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/20.jpg)
Johnson-Lindenstrauss LemmaThe Johnson-Lindenstrauss Lemma: “A set of p points in high-dimensional space can be linearly embedded in m > (12 log p) dimensions without distorting the distance between any two points by more than a factor of (1 ± ε).
m > (4 log 1e6)
m > 55
● 2^5 = 32 splits● 2^6 = 64 splits
![Page 21: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/21.jpg)
Hash TableBinary Tree Search is O(log n)
Hash Table lookup is O(1)
Hashing Function: h(v) = sgn(v ⦁ r),
that is h(v) = ±1 depending on what side of the hyperplane v lies.
Bad Hashing Function (Maximizes Collisions)
CNN Image Vector 4096D
4 Random Projections
![Page 22: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/22.jpg)
Locality Sensitivity Hashing (LSH)Keep splitting until node are small enough
Median splits give nicely balances trees
Build a forest
![Page 23: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/23.jpg)
![Page 24: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/24.jpg)
/\
2
![Page 25: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/25.jpg)
/\ /\/\
4
![Page 26: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/26.jpg)
/\ /\/\/\/\
6
![Page 27: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/27.jpg)
/\ /\/\ /\/\/\/\
8
![Page 28: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/28.jpg)
Forrest
![Page 29: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/29.jpg)
Smart Implementation 1. Same DCNN Feature Extractor2. Database to store hashtable instead of vectors3. Index Features in Database4. Approximate Nearest Neighbors using LSH
![Page 30: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/30.jpg)
Query: Giraffe & ZebraResults: Giraffes and/or Zebras in various colors, varieties & orientations
![Page 31: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/31.jpg)
Query: Human FaceResults: Human Faces in various colors, varieties & orientations
![Page 32: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/32.jpg)
Query: CatResults: Cats in various colors, varieties & orientations
![Page 33: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/33.jpg)
Query: Polar BearResults: Polar bears in various orientations + white sheep
![Page 34: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/34.jpg)
Query: Grizzly BearResults: Bears in various colors, varieties & orientations
![Page 35: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/35.jpg)
Query: Orange CatResults: Cats in various colors, varieties & orientations
![Page 36: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/36.jpg)
Query: GiraffeResults: Giraffes various colors, varieties & orientations
![Page 37: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/37.jpg)
Future Work● Text -> Image Search: Type in text phrase, then convert into a point in the
same high-dimensional space as the images● Deployment w/ kubernetes
![Page 38: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/38.jpg)
Appendix
![Page 39: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/39.jpg)
History2005: No source control!
2010: Source control & continuous builds, yay! .. but not for ML :(
2017: Great tools! .. still have a way to go for ML.
![Page 40: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/40.jpg)
Distance MeasuresDistance between two points in N dimensional space:
Euclidian
Cosine
Manhattan
![Page 41: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/41.jpg)
Naive Implementation● Feature Generator: TensorFlow Serving VGG16/FC6● Database: PostgreSQL● Frontend: Flask● Deployment: Docker & Kubernetes●
![Page 42: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/42.jpg)
VGG16Tensorflow Implementation
103 images/s on CPU
711 images/s on Tesla V100 GPU
![Page 43: Semantic Image Search - Cloud Object StorageImag… · Reverse image search in PostgreSQL using vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search](https://reader036.vdocuments.mx/reader036/viewer/2022071014/5fcd100d8abf7d3baa10f2b3/html5/thumbnails/43.jpg)
Reverse image search in PostgreSQLusing vector operations: rows are CNN image vectors. Euclidian Distance exhaustive search against query image. This is as opposed to the naive solution of doing exhaustive euclidian search in memory.