algorithms and applications: image formation process part 2 ...ds24/lehre/cv1_ws_2013/vl5.pdfroadmap...
TRANSCRIPT
![Page 1: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/1.jpg)
Computer Vision I -Algorithms and Applications:
Image Formation Process – Part 2
Carsten Rother
Computer Vision I: Image Formation Process
17/11/2013
![Page 2: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/2.jpg)
Roadmap this lecture• Geometric primitives and transformations (sec. 2.1.1-2.1.4)
• Geometric image formation (sec 2.1.5, 2.1.6)• Pinhole camera• Lens effects
• Photometric image formation process (sec 2.2)
• The Human eye
• Camera Types and Hardware (sec 2.3)
• Appearance based matching
17/11/2013Computer Vision I: Image Formation Process 2
![Page 3: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/3.jpg)
Reminder: Pinhole Camera (Geometry)
17/11/2013Computer Vision I: Image Formation Process 3
• Camera matrix P has 11 DoF
• Intrinsic parameters
• Principal point coordinates (𝑝𝑥, 𝑝𝑦)
• Focal length 𝑓
• Pixel magnification factors 𝑚
• Skew (non-rectangular pixels) 𝑠
• Extrinsic parameters
• Rotation 𝑹 (3DoF) and translation 𝐂 (3DoF) relative to world coordinate system
𝒙 = 𝑲 𝑹 (𝑰𝟑×𝟑 | − 𝑪) 𝑿~
~
𝑲 =𝑓 𝑠 𝑝𝑥0 𝑚𝑓 𝑝𝑦0 0 1
𝒙 = 𝑷 𝑿
![Page 4: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/4.jpg)
Reminder: Lens effects
17/11/2013Computer Vision I: Image Formation Process 4
![Page 5: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/5.jpg)
Roadmap this lecture• Geometric primitives and transformations (sec. 2.1.1-2.1.4)
• Geometric image formation (sec 2.1.5, 2.1.6)• Pinhole camera• Lens effects
• Photometric image formation process (sec 2.2)
• The Human eye
• Camera Types and Hardware (sec 2.3)
• Appearance based matching
17/11/2013Computer Vision I: Image Formation Process 5
![Page 6: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/6.jpg)
Image formation Process
17/11/2013Computer Vision I: Image Formation Process 6
![Page 7: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/7.jpg)
Model of the Surface
17/11/2013Computer Vision I: Image Formation Process 7
BRDF (Bi-diretcional Reflectance Function)
light
Rendering equation:
Outgoing energy: 𝒕 – timex – position𝜆 – wavelength𝜔𝑖 / 𝜔𝑜 - incoming / outgoing direction
𝒙
Outgoing energy (shining surface)
BRDF function (4DoF only 𝜔𝑖 , 𝜔𝑜
considered)
Incoming light
Fore-shorting angle(always ≥ 0)
𝜔𝑖 𝜔𝑜
![Page 8: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/8.jpg)
Example: diffuse illumination
17/11/2013Computer Vision I: Image Formation Process 8
DIFFUSE
𝑓𝑟 same for all 𝜔𝑜
Single Light source:
Shading effects come from: max(0, 𝑤𝑖 𝒏)
Rendering equation
constant0
constant (fixed 𝜔𝑖)
0 constant
![Page 9: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/9.jpg)
Examples: BRDFs
17/11/2013Computer Vision I: Image Formation Process 9
DIFFUSE SPECULAR DIFFUSE + ROUGH SPECULAR
0
Rendering equation (single light source)
constant (fixed 𝜔𝑖)
Single light source (top, right)
𝑓𝑟 𝑓𝑟 𝑓𝑟
![Page 10: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/10.jpg)
Inside a graphics engine
• Rendering equation
• Ray tracing
• Radiosity
17/11/2013Computer Vision I: Image Formation Process 10
![Page 11: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/11.jpg)
Example
17/11/2013Computer Vision I: Image Formation Process 11
Texture only
Direct light Global (direct + indirect) light
3D scene in blender
![Page 12: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/12.jpg)
How to model light?
17/11/2013Computer Vision I: Image Formation Process 12
• Point light source
• Area light source
• Non-local illumination situation can be approximated with spherical harmonics
First 9 spherical harmonics (gray scale)
They can produce this
![Page 13: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/13.jpg)
Capturig BRDF / BTF capture
17/11/2013Computer Vision I: Image Formation Process 13
[Christopher Schwartz, Ralf Sarlette, Michael Weinmann and Reinhard Klein]
BTF – Bidirectional Texture function. At every spatial location 𝒙 a different BRDF
![Page 14: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/14.jpg)
Fabricating BRDFs
17/11/2013Computer Vision I: Image Formation Process 14
[Levin et al. Siggraph 2013]
![Page 15: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/15.jpg)
Single Image
17/11/2013Computer Vision I: Image Formation Process 15
[Barron and Malik 2012]
![Page 16: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/16.jpg)
Single Image … far from being solved
17/11/2013Computer Vision I: Image Formation Process 16
[Barron and Malik 2012]
![Page 17: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/17.jpg)
Reconstruction and recognition
17/11/2013Computer Vision I: Image Formation Process 17
[Vinett, Rother, Torr, NIPS ‘13]
![Page 18: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/18.jpg)
Roadmap this lecture• Geometric primitives and transformations (sec. 2.1.1-2.1.4)
• Geometric image formation (sec 2.1.5, 2.1.6)• Pinhole camera• Lens effects
• Photometric image formation process (sec 2.2)
• The Human eye
• Camera Types and Hardware (sec 2.3)
• Appearance based matching
17/11/2013Computer Vision I: Image Formation Process 18
![Page 19: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/19.jpg)
The Human eye
17/11/2013Computer Vision I: Image Formation Process 19
• The retina contains different types of sensors:cones “Zapfen” (colors, 6 million) androds “Staebchen” (gray levels, 120 million)
• The resolution is much higher In fovea centralis
• Light first passes through the layer of neurons before it reaches photo sensors (smoothing). Only in the “fovea centralis” the light hits directly the photo sensors
• Signal goes out the other way: Retina → Ganglion cells (1 million) → Optic nerve → 1 MPixel Camera ?
![Page 20: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/20.jpg)
Spatial resolution
17/11/2013Computer Vision I: Image Formation Process 20
.
![Page 21: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/21.jpg)
Spatial resolution
Image Processing: Human seeing 5
.
2MP Camera, far from the screen
![Page 22: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/22.jpg)
Spatial resolution
Image Processing: Human seeing 5
.
5MP Camera, close to the screen
![Page 23: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/23.jpg)
Spatial resolution (secrets)
Image Processing: Human seeing 8
• The resolution is much higher In fovea centralis• The Information is pre-processed by Ganglion cells
(Compare: 3072×2304=7MPixel, 2.4 MB RGB JPEG lossless)• No still image, but a „Video“ (super-resolution)• Scanning technique − Saccades
![Page 24: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/24.jpg)
Eye Saccades
17/11/2013Computer Vision I: Image Formation Process 24
Eyes never move uniformly, but jump in saccades(approximately 15-100 ms duration between fixation points)
Saccades are driven by the “importance” of the scene parts(eyes, mouth etc).
![Page 25: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/25.jpg)
The Human eye
17/11/2013Computer Vision I: Image Formation Process 25
What is light?Spectrum, i.e. a function of the wavelength
Spectral resolution of the eye is relatively bad due to projection
Different colors can be computed by adding /subtracting signal of rods
cones “Zapfen”
![Page 26: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/26.jpg)
Roadmap this lecture• Geometric primitives and transformations (sec. 2.1.1-2.1.4)
• Geometric image formation (sec 2.1.5, 2.1.6)• Pinhole camera• Lens effects
• Photometric image formation process (sec 2.2)
• The Human eye
• Camera Types and Hardware (sec 2.3)
• Appearance based matching
17/11/2013Computer Vision I: Image Formation Process 26
![Page 27: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/27.jpg)
Camera types
• RGB cameras
• Depth cameras:
• Passive RGB stereo
• Active Structured light
• Active Time of Flight
• Lightfield cameras
17/11/2013Computer Vision I: Image Formation Process 27
![Page 28: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/28.jpg)
Digital RGB cameras
17/11/2013Computer Vision I: Image Formation Process 28
• A digital camera replaces film with a sensor array
• Each cell in the sensor array is light-sensitive diode that converts photons to electrons
• Two common types:• Charge Coupled Device (CCD)• Complementary metal oxide semiconductor (CMOS)
![Page 29: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/29.jpg)
CCD versus CMOS
17/11/2013Computer Vision I: Image Formation Process 29
• CCD: move charge from pixel to pixel and then converts to voltage and digital signal• Negative: more expensive
• CMOS: converts to voltage inside the pixel• Negative: slower to read out image ("rolling shutter" effect)
See details on: http://www.dalsa.com/shared/content/pdfs/CCD_vs_CMOS_Litwiller_2005.pdf
![Page 30: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/30.jpg)
Color - Filters
17/11/2013Computer Vision I: Image Formation Process 30
![Page 31: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/31.jpg)
Problem with de-mosaicing: color moiré
17/11/2013Computer Vision I: Image Formation Process 31
![Page 32: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/32.jpg)
Cause of color moiré
17/11/2013Computer Vision I: Image Formation Process 32
![Page 33: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/33.jpg)
General problems with RGB cameras• Noise
• low light is where you most notice noise
• light sensitivity (ISO) / noise tradeoff
• stuck pixels
• Resolution: Are more megapixels better?
• requires higher quality lens
• noise issues
• In-camera processing
• oversharpening can produce halos
• RAW vs. compressed
• file size vs. quality tradeoff
• Blooming
• charge overflowing into neighboring pixels
• More info online:
• http://electronics.howstuffworks.com/
• http://www.dpreview.com/
• http://www.dxomark.com/
17/11/2013Computer Vision I: Image Formation Process 33
![Page 34: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/34.jpg)
In-camera processing
17/11/2013Computer Vision I: Image Formation Process 34
![Page 35: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/35.jpg)
Historical context of cameras
17/11/2013Computer Vision I: Image Formation Process 35
![Page 36: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/36.jpg)
Camera types
• RGB cameras
• Depth cameras:
• Passive RGB stereo
• Active Structured light
• Active Time of Flight
• Lightfield cameras
17/11/2013Computer Vision I: Image Formation Process 36
![Page 37: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/37.jpg)
Passive Depth Camera –RGB stereo
17/11/2013Computer Vision I: Image Formation Process 37
Easy to match hard to match
![Page 38: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/38.jpg)
Active Depth Camera – structured light
17/11/2013Computer Vision I: Image Formation Process 38
IR projector
IR camera
![Page 39: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/39.jpg)
Depth Camera – structured light
17/11/2013Computer Vision I: Image Formation Process 39
• We have to perform matching (as with passive stereo camera) but now we have a very textured image
Reference image from IR projector
• Only works well when external IR light is not too string (not under sunlight)
![Page 40: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/40.jpg)
Depth Camera – structure Light
17/11/2013Computer Vision I: Image Formation Process 40
![Page 41: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/41.jpg)
Halfway
3 Minutes Break
Question?
17/11/2013Computer Vision I: Image Formation Process 41
![Page 42: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/42.jpg)
Depth camera – time of flight
17/11/2013Computer Vision I: Image Formation Process 42
Intensity image Depth ImagePMD Camera
![Page 43: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/43.jpg)
Principle Time of Flight
17/11/2013Computer Vision I: Image Formation Process 43
Modulated Light Source
Sensor Pixel
Cam Lens
[slide credits: Rahul Nair, Daniel Kondermann]
![Page 44: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/44.jpg)
Principle Time of Flight
17/11/2013Computer Vision I: Image Formation Process 44
[slide credits: Rahul Nair, Daniel Kondermann]
Modulated Light Source
Sensor Pixel
Cam Lens
![Page 45: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/45.jpg)
Principle Time of Flight
17/11/2013Computer Vision I: Image Formation Process 45
[slide credits: Rahul Nair, Daniel Kondermann]
Modulated Light Source
Sensor Pixel
Cam Lens
![Page 46: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/46.jpg)
Principle Time of Flight
17/11/2013Computer Vision I: Image Formation Process 46
[slide credits: Rahul Nair, Daniel Kondermann]
Modulated Light Source
Sensor Pixel
Cam Lens
![Page 47: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/47.jpg)
Principle Time of Flight
17/11/2013Computer Vision I: Image Formation Process 47
[slide credits: Rahul Nair, Daniel Kondermann]
Modulated Light Source
Sensor Pixel
Cam Lens
![Page 48: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/48.jpg)
Principle Time of Flight
17/11/2013Computer Vision I: Image Formation Process 48
[slide credits: Rahul Nair, Daniel Kondermann]
Modulated Light Source
Sensor Pixel
Cam Lens
![Page 49: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/49.jpg)
Principle Time of Flight
17/11/2013Computer Vision I: Image Formation Process 49
Measure:
• Phase shift
• Amplitude and offset
Output:
• Depth Image (from phase shift)(wavelength determines the range of depth values; around 3.5 meter)
• Intensity image (from amplitude and offset)
Sensor Pixel
∆Φ
offset
amplitude
[slide credits: Rahul Nair, Daniel Kondermann]
3.5meter
∆Φ means d= 0.3m or 3.8m or 7.3m or …(take differently modulated frequences)
![Page 50: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/50.jpg)
Depth camera – time of flight
One of the biggest problems is multi-path
17/11/2013Computer Vision I: Image Formation Process 50
![Page 51: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/51.jpg)
Lightfield cameras
Capture all light:
17/11/2013Computer Vision I: Image Formation Process 51
Refocus and change perspective with one image
http://www.lytro.com/camera/
![Page 52: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/52.jpg)
Roadmap this lecture• Geometric primitives and transformations (sec. 2.1.1-2.1.4)
• Geometric image formation (sec 2.1.5, 2.1.6)• Pinhole camera• Lens effects
• Photometric image formation process (sec 2.2)
• The Human eye
• Camera Types and Hardware (sec 2.3)
• Appearance based matching
17/11/2013Computer Vision I: Image Formation Process 52
![Page 53: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/53.jpg)
Roadmap: matching 2 Images (appearance & geometry)
17/11/2013Computer Vision I: Image Formation Process 53
v
• Find interest points
• Find orientated patches around interest points to capture appearance
• Encode patch appearance in a descriptor
• Find matching patches according to appearance (similar descriptors)
• Verify matching patches according to geometry
![Page 54: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/54.jpg)
Roadmap: matching 2 Images (appearance & geometry)
17/11/2013Computer Vision I: Image Formation Process 54
v
• Find interest points
• Find orientated patches around interest points to capture appearance
• Encode patch appearance in a descriptor
• Find matching patches according to appearance (similar descriptors)
• Verify matching patches according to geometry
![Page 55: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/55.jpg)
Reminder Lecture 3: Harris Corner Detector
17/11/2013Computer Vision I: Image Formation Process 55
Auto-correlation function:
Harris measure:
![Page 56: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/56.jpg)
Roadmap: matching 2 Images (appearance & geometry)
17/11/2013Computer Vision I: Image Formation Process 56
v
• Find interest points
• Find orientated patches around interest points to capture appearance
• Encode patch appearance in a descriptor
• Find matching patches according to appearance (similar descriptors)
• Verify matching patches according to geometry
![Page 57: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/57.jpg)
How to deal with orientation
17/11/2013Computer Vision I: Image Formation Process 57
Orientate with image gradient:
![Page 58: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/58.jpg)
Choose a patch around each point
17/11/2013Computer Vision I: Image Formation Process 58
How to deal with scale?
![Page 59: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/59.jpg)
Choose a patch around each point
17/11/2013Computer Vision I: Image Formation Process 59
How to deal with scale?
![Page 60: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/60.jpg)
Choose a patch around each point
17/11/2013Computer Vision I: Image Formation Process 60
How to deal with scale?
![Page 61: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/61.jpg)
Scale selection (illustration)
17/11/2013Computer Vision I: Image Formation Process 61
𝛻2 𝐺(𝜎) ∗ 𝐼 =𝜕2 (𝐺(𝜎) ∗ 𝐼)
𝜕𝑥2+𝜕2 (𝐺(𝜎) ∗ 𝐼)
𝜕𝑦2
𝒇 is Laplacian of Gaussian (LoG) operator.Measures an average edge-ness in all directions
(details on page 191)
![Page 62: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/62.jpg)
Scale selection (illustration)
17/11/2013Computer Vision I: Image Formation Process 62
![Page 63: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/63.jpg)
Scale selection (illustration)
17/11/2013Computer Vision I: Image Formation Process 63
![Page 64: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/64.jpg)
Scale selection (illustration)
17/11/2013Computer Vision I: Image Formation Process 64
![Page 65: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/65.jpg)
Scale selection (illustration)
17/11/2013Computer Vision I: Image Formation Process 65
We could match-up these curves and find unique corresponding points
![Page 66: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/66.jpg)
Scale selection (illustration)
17/11/2013Computer Vision I: Image Formation Process 66
Simpler: Find maxima /minima in image
![Page 67: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/67.jpg)
Extensions: general affine transformations
17/11/2013Computer Vision I: Image Formation Process 67
![Page 68: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/68.jpg)
Roadmap: matching 2 Images (appearance & geometry)
17/11/2013Computer Vision I: Image Formation Process 68
v
• Find interest points
• Find orientated patches around interest points to capture appearance
• Encode patch appearance in a descriptor
• Find matching patches according to appearance (similar descriptors)
• Verify matching patches according to geometry
![Page 69: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/69.jpg)
v
SIFT feature
17/11/2013Computer Vision I: Image Formation Process 69
[Lowe 2004]
• 4*4=16 cells • Each cell has an 8 bin histogram
(smoothed across cells)• In total: 16*8 values, i.e. 128D vector
64 pixels
64
pix
els
A cell has 16x16 pixels (here 8x8 for illustration only)
(blue circle shows center weighting)
![Page 70: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/70.jpg)
SIFT feature is very popular
17/11/2013Computer Vision I: Image Formation Process 70
• Fast to compute
• Can handle large changes in viewpoint well (up to 60𝑜 out of plane rotation)
• Can handle photometric changes (even day versus night)
![Page 71: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/71.jpg)
Many other feature descriptors
• MOPS [Brown, Szeliski and Winder 2005]
• SURF [Herbert Bay et al. 2006]
• DAISY [Tola, Lepetit, Fua 2010]
• Shape Context
• ….
17/11/2013Computer Vision I: Image Formation Process 71
DAISY
![Page 72: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/72.jpg)
Roadmap: matching 2 Images (appearance & geometry)
17/11/2013Computer Vision I: Image Formation Process 72
v
• Find interest points
• Find orientated patches around interest points to capture appearance
• Encode patch appearance in a descriptor
• Find matching patches according to appearance (similar descriptors)
• Verify matching patches according to geometry
![Page 73: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/73.jpg)
Appearance-based matching
17/11/2013Computer Vision I: Image Formation Process 73
![Page 74: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/74.jpg)
Appearance-based matching
17/11/2013Computer Vision I: Image Formation Process 74
N patches(e.g. N = 1000)
Goal:1) Find for each patch in left image the closest in right image 2) Accept all matches where descriptors are similar enough
N patches(e.g. N = 1000)
Methods:• Naïve: 𝑁2 tests (here 1 Million)• Hashing (locality sensitive hashing)• Kd-tree; on average NlogN tests (here 10,000)
Hashing
(HashingFunction)
Index for patch DB
![Page 75: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/75.jpg)
Subtask: Search for one patch
17/11/2013Computer Vision I: Image Formation Process 75
?
Database imageQuery patch
![Page 76: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/76.jpg)
Nearest Neighbor Search• Tracking in Video
17/11/2013Computer Vision I: Image Formation Process 76
The video is the Database
• Whole image has one descriptor
• Database is an image collection
• Image retrieval
![Page 77: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/77.jpg)
Kd-tree (d stands for dimension)
17/11/2013Computer Vision I: Image Formation Process 77
• Build the tree over database image:
1) Cycle over dimensions: x,y,z,x,y,z,….
2) Put in axis-aligned hyper-planes(split at median of point set)
• Result: balanced tree
• Nearest Neighbour search (to come)
Example in 3D
[Invented by Jon Louis Bentley 1975]
![Page 78: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/78.jpg)
4
Examples
17/11/2013Computer Vision I: Image Formation Process 78
1
2
3
5
6
![Page 79: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/79.jpg)
Examples
17/11/2013Computer Vision I: Image Formation Process 79
1
2
3
5
4,5,61,2,3
Dimension 1
0
4
5 10
8
Dim
ensi
on
2
d1>5
Kd-tree
46
![Page 80: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/80.jpg)
Examples
17/11/2013Computer Vision I: Image Formation Process 80
1
2
3
5
1 2,3
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5 4,6
Kd-tree
46
![Page 81: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/81.jpg)
Examples
17/11/2013Computer Vision I: Image Formation Process 81
1
2
3
5
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
Kd-tree
46
d1>5.5
4 6
![Page 82: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/82.jpg)
Examples: nearest neighbor search
17/11/2013Computer Vision I: Image Formation Process 82
1
2
3
5
4
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
query
Kd-tree
6
d1>5.5
4 6
![Page 83: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/83.jpg)
Examples: nearest neighbor search
17/11/2013Computer Vision I: Image Formation Process 83
1
2
3
5
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
visited
current best
search radius
query
Kd-tree
46
d1>5.5
4 6
![Page 84: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/84.jpg)
Examples: nearest neighbor search
17/11/2013Computer Vision I: Image Formation Process 84
1
2
3
5
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
visited
current best
search radius
“Radius not intersected”
query
Kd-tree
46
d1>5.5
4 6
![Page 85: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/85.jpg)
Examples: nearest neighbor search
17/11/2013Computer Vision I: Image Formation Process 85
1
2
3
5
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
visited
current best
search radius
“Radius intersects so go down the subtree”
query
Kd-tree
46
d1>5.5
4 6
![Page 86: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/86.jpg)
Examples: nearest neighbor search
17/11/2013Computer Vision I: Image Formation Process 86
1
2
3
5
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
visited
current best
search radius “better
solution found” query
Kd-tree
46
d1>5.5
4 6
![Page 87: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/87.jpg)
Examples: nearest neighbor search
17/11/2013Computer Vision I: Image Formation Process 87
1
2
3
5
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
visited
current best
search radius
query
“no need to visit these subtrees”
Kd-tree
46
d1>5.5
4 6
![Page 88: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/88.jpg)
Examples: nearest neighbor search
17/11/2013Computer Vision I: Image Formation Process 88
1
2
3
5
1
d1>5
d2>4.8 d2>6.5
0
4
8
Dim
ensi
on
2
Dimension 15 10
5d1>1
2 3
visited
current best
search radius
query
Done since root-node is marked in both ways
Kd-tree
46
d1>5.5
4 6
![Page 89: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/89.jpg)
Nearest neighbour search – pseudo code
17/11/2013Computer Vision I: Image Formation Process 89
Input: query point
Step 1: Find leave node (bucket) with query point Step 2: Make hyper-sphere with radius
(current best and query point)Step3: go up the tree and see if hyper-plane intersects hyper-sphere
Step 3a: no intersection: mark tree branch as visitedsince no better point can be found there. If node is root node then stop.
Step 3b: intersection: go down the branch to find potentially a better point. If so, mark as
current best and go to Step 2.
Current best
query
Hyper-plane positions
Nothing better possible
Pseudo code
On average O(log N)
![Page 90: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/90.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 90
From Andrew Moore: http://www.cs.cmu.edu/~awm/animations/kdtree/
![Page 91: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/91.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 91
![Page 92: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/92.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 92
![Page 93: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/93.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 93
![Page 94: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/94.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 94
![Page 95: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/95.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 95
![Page 96: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/96.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 96
![Page 97: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/97.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 97
![Page 98: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/98.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 98
![Page 99: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/99.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 99
![Page 100: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/100.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 100
![Page 101: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/101.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 101
![Page 102: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/102.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 102
![Page 103: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/103.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 103
![Page 104: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/104.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 104
![Page 105: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/105.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 105
![Page 106: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/106.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 106
![Page 107: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/107.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 107
![Page 108: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/108.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 108
![Page 109: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/109.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 109
![Page 110: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/110.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 110
![Page 111: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/111.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 111
![Page 112: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/112.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 112
![Page 113: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/113.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 113
![Page 114: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/114.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 114
![Page 115: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/115.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 115
![Page 116: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/116.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 116
![Page 117: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/117.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 117
![Page 118: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/118.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 118
![Page 119: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/119.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 119
![Page 120: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/120.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 120
![Page 121: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/121.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 121
![Page 122: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/122.jpg)
Example with many points in 2D
17/11/2013Computer Vision I: Image Formation Process 122
![Page 123: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/123.jpg)
v
Roadmap: matching 2 Images (appearance & geometry)
17/11/2013Computer Vision I: Image Formation Process 123
• Find interest points (including different scales)
• Find orientated patches around interest points to capture appearance
• Encode patch in a descriptor
• Find matching patches according to appearance (similar descriptors)
• Verify matching patches according to geometry
![Page 124: Algorithms and Applications: Image Formation Process Part 2 ...ds24/lehre/cv1_ws_2013/VL5.pdfRoadmap this lecture •Geometric primitives and transformations (sec. 2.1.1-2.1.4) •Geometric](https://reader033.vdocuments.mx/reader033/viewer/2022052013/60298b4122a1ff797531994f/html5/thumbnails/124.jpg)
Reading for next class
This lecture:
• Photometric image formation (sec 2.2)
• Camera Types and Hardware (sec 2.3)
• Appearance matching: (sec. 4.1.2-4.1.3)
Next lecture:
• Two-view Geometry (Hartley Zissermann)
17/11/2013Computer Vision I: Basics of Image Processing 124