ter haar romeny, icpr 2010 introduction to scale-space and deep structure
TRANSCRIPT
ter Haar Romeny, ICPR 2010
Introduction toScale-Space
and Deep Structure
ter Haar Romeny, ICPR 2010
Importance of Scale
Painting by Dali
• Objects exist at certain ranges of scale.
• It is not known a priory at what scale to look.
ter Haar Romeny, ICPR 2010
At the original scale of a dithered image we cannot calculate a derivative.We need to observe the image at a certain scale.
BLUR
ter Haar Romeny, ICPR 2010
Solution?
Look at all scales simultaneously
Scale
x
y
Scale Space
ter Haar Romeny, ICPR 2010
Scale Space in Human Vision
The human visual system is a
multi-scale sampling device
The retina contains receptive
fields; groups of receptors
assembled in such a way that they
form a set of apertures of widely
varying size.
ter Haar Romeny, ICPR 2010
Practical Implementation
Convolve the image with a Gaussian Kernel
2
221
2/22
)2(1),(
D
D
xx
exG
ter Haar Romeny, ICPR 2010
We can calculate derivatives and combinations of them at all scales
22yx LL
yyxx LL
Gradient Magnitude
Laplacian
Original Image
ter Haar Romeny, ICPR 2010
Main Topic
In this presentation we will show how
we can exploit the deep structure of
images to define invariant interest
points and features which can be used
for matching problems in computer
vision.
We consider only grey-value images.
ter Haar Romeny, ICPR 2010
Interest Points
The locations of particularly characteristic points
are called the interest points or key points.
These interest points have to be as invariant as
possible, but at the same time they have to carry
a lot of distinctive information.
ter Haar Romeny, ICPR 2010
Why interest points in scale-space?
Information in interest points is defined
by their neighborhood. But how big
should we choose this neighborhood?
• Let’s take the corners of the mouth as interest points.
• The red circles are the areas in which the information is gathered.
• If we make the picture bigger, the size of the neighborhood is too small.
• The neighborhood should scale with the image
ter Haar Romeny, ICPR 2010
When the interest points are detected in
scale space they do not only have spatial
coordinates x and y, but also a scale .
This scale tells us how big the
neighborhood should be.
Why interest points in scale-space?
ter Haar Romeny, ICPR 2010
Which interest points to use?
Our interest points have to be detected in scale space.They also have to…
…contain a lot of information…be reproducible…be stable…be well understood
ter Haar Romeny, ICPR 2010
We suggest Top-Points
The points we introduce have these
desired properties.
ter Haar Romeny, ICPR 2010
Critical Points, Paths and Top-Points
Maxima
Minimum
SaddlesL=0Critical Points
ter Haar Romeny, ICPR 2010
Critical Points, Paths and Top-Points
Maxima
Minimum
SaddlesL=0Critical Points
Det(H)=0Top-Points
ter Haar Romeny, ICPR 2010
Possible to calculate them for every function of the image L(x,y,)
Original Gradient Magnitude
Laplacian Det(H)
ter Haar Romeny, ICPR 2010
Detecting critical paths
Since for a critical path
L=0, intersection of
level surfaces Lx=0 with
Ly=0
will give the critical
paths.
ter Haar Romeny, ICPR 2010
Detecting Top-Points
Since for a top-point both L=0 and det[H]=Lxx Lyy-Lxy
2=0, we can find them by intersecting the paths with the level surface Det[H]=0
ter Haar Romeny, ICPR 2010
ter Haar Romeny, ICPR 2010
Original image
Top-points and features
Reconstruction
ter Haar Romeny, ICPR 2010
Metameric class
OriginalBy adjusting boundary and smoothness constraints we can improve the visual performance.
For this 300x300 picture 1000 top-points with 6 features were used.
ter Haar Romeny, ICPR 2010
Localization of Top-Points
For points close to top-points it is
possible to calculate a vector pointing
towards the position of the top-point.
x
y
Approximated Top-PointsDisplacement VectorsReal Locations
ter Haar Romeny, ICPR 2010
Stability of Top-Points
We can calculate the variance
of the displacement of top-
points under noise.
We need 4th order derivatives
in the top-points for that.
ter Haar Romeny, ICPR 2010
ter Haar Romeny, ICPR 2010
Thresholding on stability
Stable Paths Unstable Paths
ter Haar Romeny, ICPR 2010
Invariance of top-points
Top-points are invariant to certain transformations.
By invariant we mean that they move according to the
transformation.
Allo
wed
Tra
ns.
ter Haar Romeny, ICPR 2010
Differential invariants
We use the complete set
of irreducible 3rd order
differential invariants.
These features are
rotation and scaling
invariant.
ter Haar Romeny, ICPR 2010
The task
We have a scene and from
that scene we want to
retrieve the location of the
query object.
ter Haar Romeny, ICPR 2010
The top-points and differential invariants are
calculated for the query object and the scene.
ter Haar Romeny, ICPR 2010
Distance between feature vectors
A sensible distance between feature vectors is essential.
We have used Euclidean distance on ‘normalized’
differential invariants.
We tried Mahalanobis distance obtained from a training
set.
ter Haar Romeny, ICPR 2010
Similarity measure
We can calculate the propagation of noise in scale
space*.
This enables us to calculate a covariance matrix for
each feature vector.
The dissimilarity (“distance”) measure is expressed
as:
*Topological and Geometrical Aspects of Image Structure, Johan Blom
ter Haar Romeny, ICPR 2010
We now compare the
differential invariant features.
compare
distance = 0.5distance = 0.2distance = 0.3
ter Haar Romeny, ICPR 2010
The vectors with
the smallest
distance are paired.
smallest distance
distance = 0.2
ter Haar Romeny, ICPR 2010
A set of coordinates is
formed from the
differences in scale
(Log(o1)- Log(s2)) and in
angles (o1- s2).
(1, 1)
ter Haar Romeny, ICPR 2010
Dq
Important Clusters
For these clusters we calculate the mean and
Clustering (,)
If these coordinates are plotted
in a scatter plot clusters can be
identified.
• In this scatter plot we find two dense clusters
ter Haar Romeny, ICPR 2010
The stability criterion removes much of the scatter:
ter Haar Romeny, ICPR 2010
Rotate and scale according to the cluster means.
ter Haar Romeny, ICPR 2010
The translations we find correspond to the location of the objects in the scene.
ter Haar Romeny, ICPR 2010
In this example we have two clusters of correctly matched points.
C1
C2
ter Haar Romeny, ICPR 2010
The transformation of each object in the scene
matching to the query object is known from the
clustering.
ter Haar Romeny, ICPR 2010
We can transform the outline of the query object and project it on the scene image.
ter Haar Romeny, ICPR 2010
VideoGoogle?