computer vision: shape from specularities and motion
TRANSCRIPT
![Page 2: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/2.jpg)
Introduction
• Introduction - Image Geometry
• Photometric Stereo (1)
• Structured Highlights (1)
• Stereo Techniques (2)
• Motion Techniques (3)
• Solder Joint Inspection (1)
![Page 3: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/3.jpg)
Specular Surface
• Angle of Incidence = Angle of Reflection
![Page 4: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/4.jpg)
Image Geometry
________________________
![Page 5: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/5.jpg)
Image Formation
• Geometry - determines where in the image plane the projection of a point in a scene will be located
• Physics of Light - determines the brightness of a point in the image plane as a function of scene illumination and surface properties
![Page 6: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/6.jpg)
Image Formation
![Page 7: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/7.jpg)
Image Formation
• The LINE OF SIGHT of a point in the scene is the line that passes through the point of interest and the centre of projection
• The above model leads to image inversion, to avoid this, assume the image plane is in front of the centre of projection
![Page 8: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/8.jpg)
Image Formation
![Page 9: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/9.jpg)
Perspective Projection
• (x’,y’) may be found by computing the co-ordinates of the intersection of the line of sight, passing thru’ (x,y,z) and the image plane
• By two sets of similar triangles :
x’=fx/z and y’=fy/z
![Page 10: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/10.jpg)
Image Irradiance (Brightness)
• The irradiance of a point in the image plane E(x’,y’) is determined by the amt of energy radiated by the corresponding scene in the direction of the image point : E(x’,y’)=L(x,y,z)
• Two factors determine radiance emitted by a surface patch I) Illumination falling on scene patch
- determined by the patch’s position relative to the distribution of light sources
II) Fraction of incident illumination reflected by patch
- determined by optical properties of the patch
![Page 11: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/11.jpg)
Image Irradiance
• (iiis the direction of the point source of scene illumination
• (eeis the direction of the energy emitted from the surface patch
• E(iiis the energy arriving at a patch
• L(eeis the energy radiated from the patch
![Page 12: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/12.jpg)
Image Irradiance
• The relationship between radiance and irradiance may be defined as follows :
L(eef(iieeE(ii
where f(iiee is the bidirectional reflectance distribution function (BRDF)
• BRDF - depends on optical properties of the surface
![Page 13: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/13.jpg)
Types of Reflectance
• Lambertian Reflectance
• Specular Reflectance
• Hybrid Reflectance• Electron Microscopy Reflectance (not covered)
![Page 14: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/14.jpg)
Lambertian Reflectance
• Appears equally bright from all viewing directions for a fixed illumination distribution
• Does not absorb any incident illumination
• BRDF is a constant (1/)
![Page 15: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/15.jpg)
Lambertian Reflectance - Point Source
• Perceived brightness illuminated by a distant point source
L(ee Cos s -- Lambert Cosine Rule
• this means, a surface patch captures the most illumination if it is orientated so that the surface normal of the patch points in the direction of illumination
![Page 16: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/16.jpg)
Lambertian Reflectance - Uniform Source
• Perceived brightness illuminated by a uniform source
L(ee
• this means, no matter how a surface is illuminated, it receives the same amount of illumination
![Page 17: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/17.jpg)
Specular Reflectance
• Reflects all incident illumination in a direction that has the same angle with respect to the surface normal, but on the oppside of the surface normal
• light in the direction (iiis reflected to (ee(ii+
• BRDF is (e -i(e-i- / Sin i Cos i
![Page 18: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/18.jpg)
Specular Reflectance
• Perceived brightness is
L(eeee
• this means, the incoming rays of light are reflected from the surface like a perfect mirrior
![Page 19: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/19.jpg)
Hybrid Reflectance
• Mixture of Lambertian and Specular reflectance
• BRDF is (e -i(e-i- / Sin i
Cos i
• where is the mixture of the two reflectance functions
![Page 20: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/20.jpg)
Surface Orientation
• If (x,y,z) is a point on a surface and (x,y) is the same point on the image plane, with distance z from the camera (depth), then a nearby point is
(x+x, y+y)
• the change in depth can be expressed as
z = (z/x)x + (z/y)y
![Page 21: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/21.jpg)
Surface Orientation
• The size of the partial derivaties of z with respect to x and y are related to the orientation of the surface patch.
• The gradient of (x,y,z) is the vector (p,q) which is given by
p = (z/x)q (z/y)
![Page 22: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/22.jpg)
Reflectance Map
• For a given light source distribution and a given surface material, the reflectance of all surface orientations of p and q can be catalogued or computed to yield the reflectance map R(p,q) which leads to the image irradiance equation
E(x,y) = R(p,q)
![Page 23: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/23.jpg)
Reflectance Map
• i.e., that the irradiance at a point in the image plane is equal to the reflectance map value for surface orientation p and q in the corresponding point in a scene
• in other words, given a change in surface orientation, the reflectance map allows you to calculate a change in image intensity.
![Page 24: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/24.jpg)
Shape from Shading
• the oppside problem, we know E(x,y) = R(p,q), so we need to calculate p and q for each point (x,y) in the image
• Two unknows, one equation, therefore, a constraint must be applied.
![Page 25: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/25.jpg)
Shape from Shading
• Smoothness constraint
• Objects are made of a smooth surface, which depart from smoothness only along their edges
• may be expressed as
ydxdqqppe yxyxs ))()(( 2222
![Page 26: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/26.jpg)
Shape from Shading
![Page 27: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/27.jpg)
Photometric Stereo
• Asssume a scene with Lambertian reflectance
• Each point (x,y) will have brightness E(x,y) and possible orientations p and q for a given light source
• if the same surface is illuminated by a point source in a different location, the reflectance map will the different
![Page 28: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/28.jpg)
Photometric Stereo
• Using this method, surface orientation may be uniquely identified
• In reality, not all incident light is radiated from a surface, this is accounted for by adding an albedo factor () into the image irradiance eqn.
• E(x,y) = R(p,q)
![Page 29: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/29.jpg)
Photometric Stereo
________________________
![Page 30: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/30.jpg)
Determining Surface Orientations of Specular Surfaces by Using the
Photometric Stereo Method
Katsusi IkeuchiMinistry of International Trade in Industry, Japan
![Page 31: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/31.jpg)
Introduction
• Photometric stereo may be used to determine the surface orientation of a patch
• for diffuse surfaces, point source illumination is used
• for specular surfaces, a distributed light source is required
![Page 32: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/32.jpg)
Image Radiance
• For a specular surface and an extended light source :
Le(eeLiee
• Relationship between reflected radiance and image irradiance
• Ep = {(/4)(d/fp)2Cos4}Le fp = focal length
d = diameter of aperture
= off-axis angle
![Page 33: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/33.jpg)
Image Radiance
• from this a brightness distribution may be derived
• and from that an inverse transformation
![Page 34: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/34.jpg)
System Implementation
• Two Stage Process– Off-Line Job– On-Line Job
![Page 35: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/35.jpg)
Off-Line Job
• Light Source : Three linear lamps, placed symmetrically 120 degrees apart
• Lookup Table : Could use 3D table, but observed triples often contain errors
• Instead use 2D lookup Table - each element has two alternatives
• Each alternative consists of a surface orientation and an instensity
![Page 36: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/36.jpg)
Off-Line Job
![Page 37: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/37.jpg)
On-Line Job
• Normalization is required to cancel the effect of albedo
• Brightness calibration is required also
• The correct alternative of the two solutions is found by comparing the distance between the actual third image brightness and the element of the matrix
![Page 38: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/38.jpg)
Results
• Works well in a contrainted environment
• has problems if the surface is not smooth
![Page 39: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/39.jpg)
Extracting the Shape and Roughness of Specular Lobe Objects Using Four Light
Photometric Stereo
Fredric Solomon
Katsushi IkeuchiCarnegie Mellon
![Page 40: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/40.jpg)
Structured Highlights
__________________________
![Page 41: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/41.jpg)
Structured Highlight Inspection of Specular Surfaces
Arthur C. Sanderson
Lee E. Weiss
Shree K. NayarCarnegie Mellon
![Page 42: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/42.jpg)
Introduction
• Structured Highlight approach yields 3D images from point sources and images
• ‘Highlight’ - light source reflected on a specular surface
![Page 43: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/43.jpg)
Introduction
• Angle of Incidence = Angle of Reflection• A fixed camera will image a reflected light ray
(highlight) only if it is positioned and orientationed correctly
![Page 44: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/44.jpg)
Introduction
• Once a highlight is observed, if the direction of the incident ray is known, the orientation of the surface element may be found
• A spherical array of fixed point light sources is used to ensure all positions and directions are scanned
![Page 45: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/45.jpg)
Lambertian Reflectance
• The reflectance relationship for a Lambertian model of image E(x,y)
E(x,y) = A (n . s) n = surface normal (unit vector)
s = source direction (unit vector)
A = constant related to illumination intensity and surface albedo
![Page 46: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/46.jpg)
Hybrid Reflectance
• The reflectance relationship for a hybrid model of image E(x,y)
E(x,y) = A k (n . s) + (a/2)(1-k) .
[2(n . z)(n . s)-(z . s)] z = viewing direction (unit vector)
k = relative weight of specular and Lambertian components
n = sharpness of the specularity
![Page 47: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/47.jpg)
Structured Hightlight Inspection
• Using the above equation, the slope of any point may be calculated
• Surface orientation may be determined by the sources that produce local peaks in the reflectance map.
![Page 48: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/48.jpg)
Camera Models
• Perspective Camera Model
• Orthographic Projection Model
• “Fixed” Camera Model
![Page 49: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/49.jpg)
Perspective Camera Model
• All reflected rays pass though a focal point
• this model provides very accurate measurements, but requires extensive calibration procedures
![Page 50: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/50.jpg)
Orthographic Projection Model
• the focal point is assumed to be an infinite distance from the camera and all the reflected rays are perpendicular to the image plane
![Page 51: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/51.jpg)
“Fixed” Camera Model
• all rays are emitted from a single point on the reflectance plane and all surface normal estimates are computed to that reference point
![Page 52: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/52.jpg)
Camera Models - Accuracy
• Perspective Camera Model– Most accurate
• “Fixed” Camera Model– Next most accurate
• Orthographic Projection Model– Most sensitive to error
![Page 53: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/53.jpg)
SHINY - Structured Highlight INspection sYstem
• Highlightrs are extracted from images and tablulated
• Surface normals are computed based on lookup tables dervied from calibration experiments
• Reconstruction is done using interpolation followed by smoothing
![Page 54: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/54.jpg)
Stereo Hightlight Algorithm
• The assumption of a distant source to uniquely identify the angle of incidence of illumination is an approximation
• To improve this, a second camera is used with stereo matching for greater accuracy
![Page 55: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/55.jpg)
Results
• With two cameras need to resolve stereo matching ambiguities, therefore, need further constraints
• This technique is slow (1988)
![Page 56: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/56.jpg)
Stereo Techniques
________________________
![Page 57: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/57.jpg)
Stereo in the Presence of Specular Reflection
Dinkar N. Bhat
Shree K. NayarColumbia University
![Page 58: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/58.jpg)
Introduction
• Stereo is a direct method of obtaining the 3D structure of the visual world
• But, it suffers from the fact that the correspondence problem is inherently underconstrained
![Page 59: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/59.jpg)
Correspondence Problem
• the most common constraint is that intensities of corresponding points in images are identical
• The assumption is not valid for specular surfaces (since intensity is dependant on viewing direction)
![Page 60: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/60.jpg)
Specular Reflection
• When a specular surface is smooth, the distribution of the specular intensity is concentrated
• As the surface becomes rougher, the peak volume of the specular intensity decreases and the distribution widens
![Page 61: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/61.jpg)
Specular Reflection
Smooth Surface Rough Surface
![Page 62: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/62.jpg)
Implications for Stereo
• The total image intensity of any point is the sum of the diffuse and specular intensity conponents
• Since the change in diffuse components is very small relative to the changes in specular components, it follows that the overall change in intensity is approximately equal to the specular intensity differences
Idiff ~= | Is1 - Is2|
![Page 63: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/63.jpg)
Implications for Stereo
• This approximation will assist in determining an optimal binocular stereo configuration, which minimises specular correspondence problems but maximises precision in depth estimation
![Page 64: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/64.jpg)
Binocular Stereo Configuration
![Page 65: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/65.jpg)
Vergence
• When cameras are orientated such that their optical axes intersect at a point in space, this point is refered to as the point vergence
• Depth accuracy is directly proportional to vergence (…which conflicts with the requirement to minimize intensity differences)
![Page 66: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/66.jpg)
Binocular Stereo
• Determining the maximum acceptable vergence can be formulated as a constrained optimization problem
fobj = v1 . v2
c1 : Idiff < a specified threshold
c2: the cameras lie in the X-Z plane
![Page 67: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/67.jpg)
Experiments
• Two uniformly rough cylindrical objects wrapped, one is gift wrapper and the other in xerox paper
• Similar patterns were marking on both
![Page 68: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/68.jpg)
![Page 69: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/69.jpg)
Trinocular Stereo
• Required in environments which are less structured and where surface roughness cannot be estimated
• Allows intensity difference at a point to be constrained to a threshold in at least one of the stereo pairs
![Page 70: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/70.jpg)
Trinocular Stereo
![Page 71: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/71.jpg)
Experiments
• The experiments done indicate that the reconstruction algorithm works resonably well in an unconstrained environment
![Page 72: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/72.jpg)
Retrieving Shape Information from Multiple Images of a
Specular SurfaceHoward SchultzUniversity of Massachusetts
![Page 73: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/73.jpg)
Introduction
• This research extends a diffuse mutli-image shape-from-shading technique to perform in the specular domain
![Page 74: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/74.jpg)
Viewing Geometry
• Assumes an ideal camera with focal length f viewing a surface
• The camera focal point is located at P and O is a point on the surface
• From Snell’s Law an equation can be derived relating the objects position in space to its image on the image plane
![Page 75: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/75.jpg)
Viewing Geometry
![Page 76: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/76.jpg)
Image Synthesis
• the specular surface stereo method requires a model that predicts accurately the irradiance at each pixel
• Use Idealized Image Synthesis Model• this will allow us to determine that the irradiance
is directly proportional to the product of the radiance and the reflection co-efficient
![Page 77: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/77.jpg)
Specular Surface Stereo
• Starting at a known evelation, an iterative process is used to determine shape
• Two-step process, determine orientation and propagation
![Page 78: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/78.jpg)
Surface Orientation
• Identify the pixels that view the surface point (by calculating an inverse of a projective transform)
• A value of (p,q) is found such that the predicted irradiance at E(p,q) match the observed values
![Page 79: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/79.jpg)
Surface Propagation
• if a point is known on a surface, it is possible to recover shape by propagation
• If (x,y) has elevation h and gradient (p,q) then (x+x, y+y) has elevation
h’ = h +px +qy
![Page 80: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/80.jpg)
Obtaining Seed Values
• if there are surface features with diffuse proprties (e.g. scratchs or rough spots), use feature matching methods
• if surface is smooth, use a laser range finder
![Page 81: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/81.jpg)
Results
• Tests were done on four _simulated_ images to determine the feasibilty of the method, the results were 99% accurate
• Using this method in the ‘real world’ would require more constraints
![Page 82: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/82.jpg)
Motion Techniques
________________________
![Page 83: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/83.jpg)
A Theory of Specular Surface Geometry
Michael Oren
Shree K. NayarColumbia University
![Page 84: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/84.jpg)
Introduction
• Develops a 2D profile recovery technique and generalize to 3D surface recovery
• Two major issues associated with
specular surfaces– detection– shape recovery
![Page 85: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/85.jpg)
Introduction
• Specular surfaces introduce a new kind of image feature, a virtual feature
• A virtual feature is the reflection by a specular surface of another scene point which travels over the surface when the observer moves.
![Page 86: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/86.jpg)
Curve Representation
• Cartesian co-ordinates result in complex equations describing specular motion
• Using the Legendre transform to represent the curve as an envelope of tangents
![Page 87: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/87.jpg)
Curve Representation
![Page 88: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/88.jpg)
2D Caustics
• When a camera moves around an object the virtual features move on the specular surface, producing a family of reflected rays (the envelope defined by this family is called the caustic)
• On the other hand, the caustic of a real feature is one single point (the actual position of the feature in the scene where all the reflected rays intersect)
![Page 89: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/89.jpg)
Test Image
![Page 90: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/90.jpg)
2D Caustics
• Using this, feature classification is simply a matter of computing a caustic and determining whether it is a point or a curve
• Features are tracked from one frame to the next using a sum of square difference (SSD) correlation operator
![Page 91: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/91.jpg)
2D Profile Recovery
• The camera is moved in the plane of the profile and the features are tracked
• An equation may be derived relating the caustic to the surface profile, allowing the recovery of the 2D profile from the image.
![Page 92: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/92.jpg)
3D Surface Recovery
• The 3D camera motion problem will result in an arbitrary space curve rather than a family of curves as in the 2D case
• The 3D problem cannot but reduced to a finite number of 2D profile problems
![Page 93: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/93.jpg)
3D Surface Recovery
• The concept behind the derevation of the 3D caustic curve is to decompose the caustic point position at any given instant into two orthogonal components
• As the camera moves along the specular object, a virtual feature travels along the 3D profile on the objects surface.
• It is possible to develop an equation which relates the trajectory of the virtual feature to the surface profile
![Page 94: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/94.jpg)
Results
• The 2D testing involved tracking two features on two different specular surfaces, in both experiments the profile was accurately estimated
• The 3D testing involved tracking a highlight on a specular surface, the recovered curve is in strong agreement with the actual surface
![Page 95: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/95.jpg)
Epipolar Geometry
________________________
![Page 96: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/96.jpg)
Epipolar Geometry
• two cameras are displaced from each other by a baseline distance
• Object point X forms two distinct image points x and x’
![Page 97: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/97.jpg)
Epipolar Geometry
• Assume images formed in front of camera to avoid inversion problem
• point (x’, y’) in the images plane from a real point (x, y, z) may be calculated as
x’ = fx/z and y’ = fy/z
• the displacement between the locations of image point is called the disparity
![Page 98: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/98.jpg)
Epipolar Geometry
• the plane passing through the two camera centres and the object point is called the epipolar plane
• the intersection of the image plane and the epipolar plane is called the epipolar line
![Page 99: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/99.jpg)
Generalizing Epipolar-Plane Image Analysis on the Spatiotemporal Surface
H. Harlyn Baker
Robert C. BollesSRI International
![Page 100: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/100.jpg)
Introduction
• The technique of Epipolar-Plane Image Analysis involves obtaining depth estimates for a point by taking a large number of images
• This gives a large baseline and higher accuracy
• It also minimises the correspondence problem
![Page 101: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/101.jpg)
Epipolar-Plane Image Analysis
• this technique imposes the following constraints– the camera is moving along a linear path– it acquires images at equal spacing as it is
moved– the camera’s view is orthogonal to the direction
of travel
![Page 102: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/102.jpg)
Epipolar-Plane Image Analysis
• the traditional notion of epipolar lines is generalized to an epipolar plane
• using this, plus the fact that the camera is always moving along a linear path and we may conclude the a given scence feature will always be restricted to a given epipolar plane
![Page 103: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/103.jpg)
Epipolar-Plane Image Analysis
![Page 104: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/104.jpg)
The Spatiotemporal Surface
• As images are collected, they are stacked up into a spatiotemporal surface
• as each new image is obtained its spatial and temporal edge contours sre constructed
• using a 3D Laplacian of a 3D Gaussian
![Page 105: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/105.jpg)
The Spatiotemporal Surface
![Page 106: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/106.jpg)
3D Surface Estimation and Model Construction From Specular Motion
in Image Sequences Jiang Yu Zheng
Norihiro AbeKyushu Institiute of Technology
Yoshihiro Fukagawa
Torey Corporation
![Page 107: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/107.jpg)
Introduction
• This technique reconstructs 3D models of complex objects with specular surfaces
• The process involves rotating the object under inspection
![Page 108: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/108.jpg)
System Setup
![Page 109: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/109.jpg)
Projected Highlights
• An extended light source project highlight stripes onto the object
• The stripes gradually shift across the object surface and pass most point once
• The specular motion is captured in epipolar-plane images
![Page 110: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/110.jpg)
Feature tracking
• We know how to detect corners and edge of surface patterns
• The motion type of highlights in EPI can be used to determine five categories of shape– convex corner
– convex
– planer
– concave
– concave corner
![Page 111: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/111.jpg)
EPI-Plane Images
• During the rotation, highlights will split and merge, appear and dissapear, etc.
![Page 112: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/112.jpg)
Results
• Using EPIs results in very accurate reconstruction of surface shapes
![Page 113: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/113.jpg)
Solder Joint Inspection
____________________________
![Page 114: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/114.jpg)
Visual Inspection System for the Classification of Solder Joints
Tae-Hyeon Kim
Young Shik Moon
Sung Han ParkHanyang University
Kwang-Jin YoonLG Industrial Systems
![Page 115: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/115.jpg)
Introduction
• Uses three layers of ring shaped LED arrays, with different illumination angles
• Solder Joint are segemented and classified using either their 2D features or their 3D features
![Page 116: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/116.jpg)
Classification of Joints
![Page 117: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/117.jpg)
Preprocessing
• Objective is to identify and segement the soldered regions
• Solder is isolated both vertically and horozontally
![Page 118: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/118.jpg)
Feature Extraction - 2D
• Average gray level value of I1 and I3
X1 = 1/N * IK(x,y)
• Percentage of highlights of I1 and I2
X2 = 1/N * U(x,y) * 100
U(x,y) = thresholded image of I1
![Page 119: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/119.jpg)
Feature Extraction - 3D
• Shape recovery is done using a hybrid reflectance model for all samples not in the confidence interval
• A reflectance map is built up representing intensity values as a function of orientation for each illumination angle
• For each point, three intensity values are recovered and from these and the reflectance map, the orientation is estimated
![Page 120: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/120.jpg)
Classification -2D
• Uses 3-Layer backpropagation neural network
• Four input nodes for four features
• Five hidden layer nodes
• Four output nodes for four solder types
![Page 121: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/121.jpg)
Classification - 3D
• Bayes Classifier assuming Gaussian Distribution
![Page 122: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/122.jpg)
Inspection System
![Page 123: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/123.jpg)
Results
Features Class Number Correct % Incorrect2D Good 52 98 22D Excess 57 100 02D Insuff. 44 100 02D None 50 100 02D Total 203 99.5 0.52D+3D Good 52 100 02D+3D Excess 57 100 02D+3D Insuff. 44 100 02D+3D None 50 100 02D+3D Total 203 100 0
![Page 124: Computer Vision: Shape from Specularities and Motion](https://reader036.vdocuments.mx/reader036/viewer/2022070316/5560c953d8b42a08088b485b/html5/thumbnails/124.jpg)
Results
Features Time (s)2D 1.863D 19.83