covariant derivatives and vision
TRANSCRIPT
![Page 1: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/1.jpg)
1
Covariant Derivatives and Vision
Todor GeorgievAdobe Photoshop
Presentation at ECCV 2006
![Page 2: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/2.jpg)
2
Original
![Page 3: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/3.jpg)
3
Selection to clone
![Page 4: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/4.jpg)
4
Poisson cloning from dark area
![Page 5: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/5.jpg)
5
Selection to clone
![Page 6: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/6.jpg)
6
Poisson cloning from illuminated area
![Page 7: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/7.jpg)
7
Poissoncloning
Covariantcloning(see next)
![Page 8: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/8.jpg)
8
Poisson cloning can be viewed as an approximation to covariant cloning.
Outline of our theory:
![Page 9: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/9.jpg)
9
![Page 10: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/10.jpg)
10
Thanks to Jan Koenderink
![Page 11: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/11.jpg)
11
• The image is just a record of pixel values.
• We do not see pixel values directly.
• What we see is an illusion generated fromthe above record through internal adaptation.We can not compare pixels.
Retina / Cortex Adaptation
![Page 12: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/12.jpg)
12
Models of Image Space:
![Page 13: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/13.jpg)
13
• A pair (location, intensity)
• Multiple copies of the intensity line.
• We can compare intensities. The image is a functionthat specifies an intensity at each point.
(1) Cartesian Product
![Page 14: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/14.jpg)
14
• Two spaces and a mapping (vertical projection)
Total space EBase space B
• Fiber is the set of points that map to a single point. We will use vector bundles, where fibers are vectorspaces.
(2) Fiber Bundle
![Page 15: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/15.jpg)
15
• Mapping from base B to total space E
Sections replace functions
• We can not compare intensities. Horizontalprojection is not defined. We have forgotten it.
• Perceptually correct model of the image Image = graph of a section
Section in a Fiber Bundle
![Page 16: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/16.jpg)
16
Derivatives in a Fiber Bundle
Definition:
Derivative is a mapping from one section to another that satisfies the Leibniz rule relative to multiplication by functions:
In the Cartesian product space this definition is equivalent to the conventional derivative.
![Page 17: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/17.jpg)
17
If we express a section as a linear combination of some basis sections
then the derivative will be:
Derivatives in a Fiber Bundle
![Page 18: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/18.jpg)
18
Derivatives in a Fiber Bundle
If we represent the section in terms of the functionsthat define it in a given basis (not writing the
basis vectors), the last equation can be written as:
The functions are called “color channels” in Photoshop, and D is called “Covariant Derivative”. It corresponds to the derivative in the Cartesian product model.
![Page 19: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/19.jpg)
19
The covariant derivative is a rigorous mathematical tool for perceptual pixel comparison in the fiber bundle modelof image space. It replaces the conventional derivative of the Cartesian product model as:
![Page 20: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/20.jpg)
20
![Page 21: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/21.jpg)
21
- Reconstructing images with the covariant Laplace equation
,based on adaptation vector field, A.
- Reconstructing surfaces based on gradient field.Recent work by R. Raskar et. al. Covariant Laplace should produce better results than Poisson.
How can we know A?It can be extracted based on the idea of covariantlyconstant section, next:
![Page 22: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/22.jpg)
22
Assume perceived gradient of image g(x, y) is zero.This means complete adaptation:
Substitute in covariant Laplace:
Covariantcloning
Poisson equation
![Page 23: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/23.jpg)
23
Poissoncloning
Covariantcloning
![Page 24: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/24.jpg)
24
Poisson
Covariant
![Page 25: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/25.jpg)
25
Detailed Example:
![Page 26: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/26.jpg)
26
Original Damaged Area
![Page 27: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/27.jpg)
27
Laplace
![Page 28: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/28.jpg)
28
Poisson
![Page 29: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/29.jpg)
29
Laplace
![Page 30: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/30.jpg)
30
Inpainting
Thanks to Guillermo Sapiro and Kedar Patwardhan
![Page 31: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/31.jpg)
31
Poisson
![Page 32: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/32.jpg)
32
Covariant
![Page 33: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/33.jpg)
33
Inpainting
Thanks to Guillermo Sapiro and Kedar Patwardhan
![Page 34: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/34.jpg)
34
Structure and Texture Inpainting
Bertalmio – Vese – Sapiro – Osher
![Page 35: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/35.jpg)
35
Covariant Inpainting
![Page 36: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/36.jpg)
36
Day
![Page 37: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/37.jpg)
37
Night
![Page 38: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/38.jpg)
38
Covariant cloning from day
![Page 39: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/39.jpg)
39
Poisson cloning from day
![Page 40: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/40.jpg)
40
Thanks to R. Raskar and J. Yu
Cloning from night to day
![Page 41: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/41.jpg)
41
Gradient Domain HDR Compression
Changing the lighting conditions. The visual system is robust. It compensates for the changes in illuminationby adaptation vector field A:
Simplest energy invariant to those transforms is:
![Page 42: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/42.jpg)
42
Gradient Domain HDR Compression
Euler-Lagrange equation for the above energy:
Exactly reproduces the result of the Fattal-Lischinski-Werman paper. They assume log; we derive log.
Any good visual system needs to be logarithmic!
![Page 43: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/43.jpg)
43
Conclusion:
The covariant (adapted) derivative providesa way to perform perceptual image processingaccording to how we see images as opposed to -how images are recorded by the camera.
Useful for Poisson editing, inpainting or any PDE,HDR compression, surface reconstruction from gradients, night/day cloning, graph cuts, bilateraland trilateral filters in terms of jet bundles, and practically any perceptual image editing.
![Page 44: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/44.jpg)
44
Bilateral interpreted in 3D image space
The image is a distribution in 3D:
or -perceptual?
Integrate the following 3D filter expression over z
and evaluate it on the original surface. Result:
(1)
Appendix:
![Page 45: Covariant Derivatives and Vision](https://reader031.vdocuments.mx/reader031/viewer/2022021421/586b82911a28abd80f8b93b9/html5/thumbnails/45.jpg)
45
(1)
Same procedure on the logarithmic expression
produces (using “delta function of function” formula):
(2)
Now, bilateral filter is exactly expression (2) dividedby expression (1). Paris-Durand paper derives a similarresult (based on intuition) and a speed up algorithm.
Bilateral