ee 576 - image featuresisl.ee.boun.edu.tr/courses/ee576/lectures/sunum/featurespres.pdf · outline...

Outline

Introduction

EE 576 - Image Features

H.I. Bozma

Electric Electronic EngineeringBogazici University

March 11, 2020

H.I. Bozma EE 576 - Image Features

Outline

Introduction

IntroductionLocal FeaturesLocal Structure MatrixHarris Corner DetectorKLT DetectorGlobal Features


Outline

Introduction

Local Features

Local Structure Matrix

Harris Corner Detector

KLT Detector

Global Features

Features

◮ Local: Derived directly parts of the image◮ Results of filtering operations (edges)◮ Corners◮ KLT◮ SIFT

◮ Global: Derived from the whole of the image◮ Histogram◮ Bag-of-words◮ GIST


Outline

Introduction

Local Features



KLT Detector

Global Features

Edges

Changes in image brightness is to capture important events andchanges in properties of the world. In particular, edges encode:

◮ Changes in material properties

◮ Discontinuities in depth,

◮ Discontinuities in surface orientation,

◮ Variations in scene illumination.


Outline

Introduction

Local Features



KLT Detector

Global Features

Edges

◮ Intensity: I =√

R2x + R2

y

◮ Angle: θ = tan−1 Ry

Rx


Outline

Introduction

Local Features



KLT Detector

Global Features

Sobel


Outline

Introduction

Local Features



KLT Detector

Global Features

Canny

1. Apply Gaussian filter to smooth the image in order to removethe noise

2. Find the intensity gradients of the image

3. Apply non-maximum suppression to get rid of spuriousresponse to edge detection

4. Apply double threshold to determine potential edges

5. Suppress all the edges that are weak and not connected tostrong edges.


Outline

Introduction

Local Features



KLT Detector

Global Features

Hough Transform

Assume edge detection & thresholding of the edge magnitudeConsider a pixel x = [x1, x2]

A straight line passing thruax1 + b = x2There are many lines passing thru x

∃ line, many x that satisfy the line equation


Outline

Introduction

Local Features



KLT Detector

Global Features

Hough Transform (cont)

Consider x as parameters and a, b as variables.

b = −x1a+ x2 (1)

A line in (a, b) space parametrized by xAnother x gives another line


Outline

Introduction

Local Features



KLT Detector

Global Features

Hough Transform (cont)


Outline

Introduction

Local Features



KLT Detector

Global Features

Corners

◮ Corners – Image features characterized by their high intensitychanges in the horizontal and vertical directions.

◮ Object is square =⇒ Its four corners are very good interestpoints.

◮ Corners are used in shape analysis and motion analysis.


Outline

Introduction

Local Features



KLT Detector

Global Features

Corner

Figure: Left: An edge, Right: A corner.


Outline

Introduction

Local Features



KLT Detector

Global Features

Edges vs Corners

◮ Edges are locations where the variation of I in a certaindirection is high, while the variation in the orthogonaldirection is low. For example, in an edge oriented along the x2axis, ∂I

∂x1is high while ∂I

∂x2is low.

◮ Corners are local image features characterized by locationswhere variations of intensity function in both x1 and x2directions are high – both ∂I

∂x1and ∂I

∂x2are high.


Outline

Introduction

Local Features



KLT Detector

Global Features

Corner Detection

◮ Directly from the grayscale images or

◮ As the intersection of digital contours (usually lines) if thesecontours have been found.


Outline

Introduction

Local Features



KLT Detector

Global Features

Corner Detectors

The two most commonly used corner detectors to-date are:

◮ the Harris corner detector

◮ the Kanade-Lucas-Tomasi (KLT) corner detector


Outline

Introduction

Local Features



KLT Detector

Global Features

Local Structure Matrix C

A common property : Both operate with C

C =

[

∂2I∂2x1

∂2I∂x2∂x1

∂2I∂x2∂x1

∂2I∂2x2

]


Outline

Introduction

Local Features



KLT Detector

Global Features

Practical Computation of C

The partial derivatives – Replaced, respectively, by theconvolutions with a 1D horizontal and vertical discrete masks.

◮ The derivatives of the intensity function are first calculated ineach point.

◮ If necessary, the image is smoothed before taking thederivatives

◮ Then, C is obtained.

◮ Finally, each of the entries is smoothed (integrated) byGaussian filter of selected size σ.

◮ Often, a simple box (averaging) filter is used instead of theGaussian.


Outline

Introduction

Local Features



KLT Detector

Global Features

Eigenvalues of C

◮ λ1 and λ2 be the two eigenvalues of the matrix C.

◮ As C is symmetric and positive semi-definite, both theeigenvalues are non-negative.


Outline

Introduction

Local Features



KLT Detector

Global Features

Eigenvalues and Eigenvectors

◮ In a uniform and homogeneous region, λ1, λ2 = 0,

◮ At a step edge, λ1 > λ2 = 0. The corresponding eigenvectorfor λ1 is associated with the direction that is orthogonal tothe edge.

◮ At a corner, λ1 ≥ λ2 > 0. The larger are the values of λ1 andλ2, the higher are the contrasts of the edges orthogonal to thedirections of the corresponding eigenvectors.

◮ Hence, the eigenvectors encode the edge directions whileeigenvalues encode the edge magnitudes.


Outline

Introduction

Local Features



KLT Detector

Global Features

Corner Detection II

In summary, a corner should be marked at a location where thesmaller eigenvalue λ2 is large enough.


Outline

Introduction

Local Features



KLT Detector

Global Features

Algorithm

In Harris corner detector, the local structure matrix is smoothed bya Gaussian filter.

CH = wG (σ) ∗[

∂2I∂2x1

∂2I∂x2∂x1

∂2I∂x2∂x1

∂2I∂2x2

]

wG is an isotropic Gaussian filter with standard deviation σ

The operation ∗ denotes convolution.


Outline

Introduction

Local Features



KLT Detector

Global Features

◮ A measure of the corner response at each pixel coordinates xis defined by:

h(x) = det(CH(x))− κtrace(CH(x))2

◮ κ is a parameter. Note that κ < 0.25, then h(x) > 0

◮ A corner is detected if h(x) > τ .


Outline

Introduction

Local Features



KLT Detector

Global Features

Additional Considerations

◮ To prevent too many corner features occurring togetherclosely, a non-maximal suppression process on the cornerresponse image

◮ Thresholding process.


Outline

Introduction

Local Features



KLT Detector

Global Features

Harris Detector Parameters

1. κ

2. The radius, d, of the neighbourhood region for suppressingweak corners,

3. The threshold value τ


Outline

Introduction

Local Features



KLT Detector

Global Features

Mathematical Formulation

Let λ1 = λ, then λ2 = αλ where 0 ≤ α ≤ 1

h(x) = λ1λ2 − κ(λ1 + λ2)2

= αλ2 − κ(1 + α)2λ2


Outline

Introduction

Local Features



KLT Detector

Global Features

◮ If h(x) ≥ 0 → κ ≤ α

(1+α)2.

◮ Two equally strong edges → α = 1 → κ = 14 .

◮ If one of the edges is very weak – α << 1, κ < α << 1.


Outline

Introduction

Local Features



KLT Detector

Global Features

κ Parameter

◮ κ - A measure of the lower bound for the ratio between themagnitude of the weaker edge and that of the stronger edge.

◮ Larger value of κ – A less sensitive detector and yields lesscorners

◮ Smaller value of κ – A more sensitive detector and yields morecorners.


Outline

Introduction

Local Features



KLT Detector

Global Features

Corners detected by the Harris corner detector

Figure: σ = 1, τ = 1500 (threshold), and d = 5 (neighbourhood radiusfor non-maximal suppression). Left: κ = 0.1, Right= κ = 0.15.


Outline

Introduction

Local Features



KLT Detector

Global Features

Two Parameters

1. The threshold on λ2, denoted by τ ,

2. The linear size of a square window (neighbourhood) D.


Outline

Introduction

Local Features



KLT Detector

Global Features

KLT Algorithm

1. Compute ∂I∂x1

and ∂I∂x2

on the image.

2. For each image point p:

2.1 Form the matrix C over a D × D neighbourhood N of p asfollows:

CK =∑

y∈N

[

∂2I

∂2x1(y) ∂

2I∂x2∂x1

(y)∂2I

∂x2∂x1(y) ∂

2I∂2x2

(y)

]

2.2 Compute λ2 - the smaller eigenvalue of C;2.3 If λ2 > τ , save the p into a list, L.

3. Sort L in decreasing order of λ2

4. Scan the sorted list from top to bottom. For each currentpoint, p, delete all points appearing further in the list whichbelong to the neighbourhood of p.


Outline

Introduction

Local Features



KLT Detector

Global Features

Output of KLT Corner Detector

◮ A list of corner points that have λ2 > τ

◮ the neighbourhood N of these points do not overlap.


Outline

Introduction

Local Features



KLT Detector

Global Features

◮ No simple criteria for determining the size of d – commonlytaken to be 2 ≤ d ≤ 10.

◮ The summation operation over the neighbourhood N →Uniform smoothing for the image gradients around eachimage point.

◮ The parameters τ and d for the KLT corner detector –similar to the parameter κ and the parameter d used in thenon-maximal suppression of the Harris corner detector.


Outline

Introduction

Local Features



KLT Detector

Global Features

Corners detected by the KTL detector


Outline

Introduction

Local Features



KLT Detector

Global Features

Corners detected by Left: Harris, Right: the KTL detector.


Outline

Introduction

Local Features



KLT Detector

Global Features

SIFT Keypoint Detector [1]

◮ SIFT features → Represent local image measurements:◮ Relatively invariant to image translation, scaling and rotation◮ Partially invariant to changes in illumination and local image

deformations.


Outline

Introduction

Local Features



KLT Detector

Global Features

SIFT Algorithm

A number of Gaussian filters whose standard deviations {σ1, σ2}differ by a fixed scale factor. That is, σj+1 = kσjk - A constant scalar set to

√2

The input image I → Convolved with a number of Gaussian filtersThe convolutions yield a small number of smoothed images,denoted by G (x ;σ1),G (x ;σ2), · · ·


Outline

Introduction

Local Features



KLT Detector

Global Features

Smoothing with varying σ


Outline

Introduction

Local Features



KLT Detector

Global Features

SIFT Algorithm - 2

The adjacent smoothed images – Subtracted to yield a smallnumber (3 or 4) of DoG (Difference-of-Gaussian) imagesD(x ; j) = G (x ;σj+1)− G (x ;σj)Repeat on the subsampled smoothed images → A number of DoGimages over the scale space.Each point in these DoG images is then examined.Keypoint: Marked at a location where the point is a localminimum or maximum of its 8 neighbours on the same scale andof its 9 neighbours on the scales above and below.


Outline

Introduction

Local Features



KLT Detector

Global Features

DOGs


Outline

Introduction

Local Features



KLT Detector

Global Features

SIFT Algorithm - 3

Examine the keypoints for possible elimination via checking forpoints that have poorly determined locations but have high edgeresponses.


Outline

Introduction

Local Features



KLT Detector

Global Features

SIFT Algorithm - 4 Elimination Stage

◮ The 2× 2 Hessian matrix (The second partial derivatives) ofthe local image intensity around each keypoint is computed as:

H =

[

Dx1x1 I Dx2x1 I

Dx1x2 I Dx2x2 I

]

◮ For poorly defined peaks in the DoG function, the principalcurvature across the edge would be much larger than theprincipal curvature along it which can be computed via solvingfor the eigenvalues of H.

◮ The eigenvalues of H are proportional to the principalcurvatures of D. It turns out that the ratio r of the twoeigenvalues is sufficient for SIFT’s purposes - where r isdefined as the ratio of the larger eigenvalue to that of thesmaller eigenvalue.


Outline

Introduction

Local Features



KLT Detector

Global Features

SIFT Algorithm - 4 Elimination Stage (cont)

◮ The trace of H Dx1x1 I + Dx2x2 I , gives us the sum of the twoeigenvalues

◮ The determinant Dx1x1 IDx2x2 I − Dx2x1 IDx1x2 I yields theproduct.

◮ R = Tr(H)2

Det(H) = (r+1)2

r

◮ R is minimum when the eigenvalues are equal to each other→ The higher the absolute difference between the twoeigenvalues, which is equivalent to a higher absolute differencebetween the two principal curvatures of D, the higher thevalue of R.

◮ Hence, R >(rth+1)2

rth=⇒ Poor localization and hence reject.


Outline

Introduction

Local Features



KLT Detector

Global Features

SIFT Algorithm - 5

• A set of keypoint locations• The surrounding intensity and gradient information around eachkeypoint – Examined for the coding of the SIFT descriptor at thatkeypoint.• Histograms computed in a 16 x 16 region around the keypointsuch that each histogram contains samples from a 4 x 4 subregionof the original neighborhood region.• The image gradients are computed and classified into 8orientations for a 4× 4 histogram array giving a SIFT descriptor of128 elements long for each keypoint


Outline

Introduction

Local Features



KLT Detector

Global Features

Encoding Keypoints


Outline

Introduction

Local Features



KLT Detector

Global Features

Comparative Analysis

◮ Unlike the Harris and KLT corner detectors, the SIFTkeypoints do not often coincide at corner features.

◮ However, the SIFT descriptor of each keypoint does provideuseful information for wide baseline stereo matching and forobject recognition.


Outline

Introduction

Local Features



KLT Detector

Global Features

Histogram

Figure: Images of an object from two different viewpoints and theirhistograms.


Outline

Introduction

Local Features



KLT Detector

Global Features

Bag-of-Words [2]

◮ Image representation - A collection of words, disregardinggrammar and even word order.

◮ Words – Obtain many samples and then cluster to get asmaller set

◮ Frequency of occurrence of each word - Used as a feature fortraining a classifier

◮ To vector quantize the descriptors into clusters which will bethe visual ‘words’ for text retrieval → Vocabulary

◮ An image → As an histogram of words


Outline

Introduction

Local Features



KLT Detector

Global Features

Bag-of-Words

→


Outline

Introduction

Local Features



KLT Detector

Global Features

Spatial Envelope [3]

naturalnessopennessroughnessexpansionruggednessSpectral Spatial domain


Outline

Introduction

Local Features



KLT Detector

Global Features

Hull Census Transform (HCT)


Outline

Introduction

Local Features



KLT Detector

Global Features

Bubble Space [4]

Robot base

x = [c α]T

Base space X = R2 × S1

Pan-tilt movement:

f = [f1f2]T

Camera state space F ⊂ S2

Bubble space B = X × FRobot’s base + its viewing directions


Outline

Introduction

Local Features



KLT Detector

Global Features

D. G. Lowe, “Object recognition from local scale-invariantfeatures,” in ICCV, 1999, pp. 1150–1157.

J. Sivic and A. Zisserman, “Efficient visual search of videoscast as text retrieval,” Pattern Analysis and Machine

Intelligence, IEEE Transactions on, vol. 31, no. 4, pp.591–606, April 2009.

A. Torralba, K. P. Murphy, W. T. Freeman, and M. A. Rubin,“Context-based vision system for place and objectrecognition,” Computer Vision, IEEE Int. Conf. on, vol. 1, p.273, 2003.

O. Erkent and H. I. Bozma, “Bubble Space and PlaceRepresentation in Topological Maps,” The Int. J. of Rob. Res.,vol. 32, no. 6, pp. 671 – 688, 2013.


ee 576 - image featuresisl.ee.boun.edu.tr/courses/ee576/lectures/sunum/featurespres.pdf · outline...

Documents