spatial alignment

Spatial Alignment

Spring 2009

Ben-Gurion University of the Negev

Sensor Fusion Spring 2009

Instructor

• Dr. H. B Mitchell

email: [email protected]


Spatial Alignment

Process of geometrically aligning images of the same scene acquired

At different times (multi-temporal fusion) With different sensors (multi-modal fusion) From different viewpoints (multi-view fusion)


Example


Spatial Alignment Algorithms

Classify them by the nature of the images to register Monomodal registration Multimodal registration


Spatial Alignment

A: reference image B: floating image Spatial alignment finds transformation T which maps

each pixel (x,y) in B into a location in A:

A B

(x,y)


Transformations

Some common transformations:

Local Global transformations


Global Transformations

Translation Similarity Affine Perspective Polynomial


Spatial Alignment

The location does not correspond to a pixel location in A.

In order to convert the gray-level

into a digital image which is defined at the same pixel locations as A we require an interpolation/resampling

Symbolically write it as


Spatial Alignment

Spatial alignment of B to A gives us But we require This is found by going in the reverse direction, i.e. from A

to B

B to A

A to B


Spatial Alignment. Nearest Neighbor

Simplest resample/interpolation algorithm is nearest neighbor.

We have Then find if is nearest pixel coordinates

to then

B to A

A to B


Spatial Alignment. Bilinear Interpolation

Bilinear interpolation is also very simple.


Mutual Information

In multi-modal spatial alignment we find the geometric transformation T by matching the picture A and the transformed image T(B) using a similarity measure S.

We require a similarity measure S(A,T(B)) which Depends on the intrinsic structure of the scene and is

independent of the image gray-levels Falls monotonically as we move away from the true alignment


Mutual Information

The mutual information MI(A,T(B)) has been found to work very well for this purpose

MI depends only on the distribution of pixel gray-levels and not on the gray-levels themselves.

MI is defined as

where


Mutual Information. Histogram

The simplest method for calculating

is to use histograms: Let and . Divide into M histogram

bins and into N histogram bins. Then

Are these the same?What is the #pixels?


Mutual Information. Histogram Histogram method is widely used. However its

disadvantages are: Probability densities are discontinuous Requires an optimal choice of bin widths. If the bin width is too

small then the density estimate is noisy. If the bin is too wide then the density estimate is shows no detail, i.e. too smooth.


Mutual Information. Histogram Empirical formula for optimal number of equi-spaced bins in range [0,1]:

Birge and Rozenholc. How many bins should be put in a regular histogram? ESAIM: Probability and Statistics (2006)


Mutual Information. Parzen Windows

Parzen windows replace discrete histogram bins with continuous bins.

If A contains K samples the estimated probability density is

Often use a Gaussian function for Ker with a bandwidth Simple rule of thumb estimates for is


Mutual Information. Iso-Lines

Iso-lines is a new method to calculate Ref: Rajwade et al Probability density estimation using

iso-contours and iso-surfaces. PAMI (2009) .

Suppose in triangles gray-levels vary as

We consider whether triangle contains a point which has a quantized gray-level in A and in B. If such a point exists then it contributes a vote of one to


Mutual Information. PVI

Histogram, Parzen windows and Iso-Lines all require T(B) i.e. require the transformation T.

One method which does not require the transformation T is PVI (Partial Volume Interpolation).

Suppose the pixel in B transforms to in A. If have quantized gray-levels and has

quantized gray-level , then

receives a vote


Mutual Information. Artifacts

Assumed the MI falls monotonically to zero as we move away perfect alignment.

In practice this is not true. The reason is due to inaccuracies in estimating marginal densities and

joint density . The artifacts are due to

Interpolation effects. Empirically best interpolation algorithm for MI calculation is nearest neighbor since this does not introduce new gray-levels

Small size effects Changes in the overlap area


Mutual Information. Interpolation Artifacts


Mutual Information. Small Size Effects

If we perform MI on small image patches then find a “small-size” effect which occurs when the patch is too small to contain significant structure.

Suggested method for identifying patches with no significant structure is Moran’s autocorrelation coefficient. Project


Mutual Information. Overlap Effects

The MI depends on the statistics of the overlap area. As the overlap area changes we find MI changes slightly. However this tends to smear out optimum peak.

Solution is to use a “normalized” MI:

NMI = MI/(H(a)+H(b))

NMI = MI/(H(a)H(b)) etc


Mutual Information. Hierarchical Scheme

The MI scheme assumes we transform the image B using some global transformation T.

Often the transformation cannot be described with a single global transformation.

In this case we often use a collection of local transformation which we calculate in a hierarchical scheme.


Hierarchical Spatial Alignment

In hierarchical spatial alignment we progressively sub-divide the image into smaller and smaller sub-images. The process is as follows

Register B to image A using global transformation T Divide A and B’=T(B) into four equal parts Register each sub-image with

corresponding sub-image using transformations Combine the transformations into a single

transformation T using an interpolation algorithm. Apply T to obtaining .

Continue


Hierarchical Spatial Alignment


Hierarchical Spatial Alignment. TPS

Common to integrate using a thin-plate spline interpolation algorithm

Project.

spatial alignment

Documents

b sensor fusion spring

examplesensor fusion

multimodal spatial alignment

spatial alignment algorithms

spatial alignmenta

spatial alignmentprocess

spatial alignmentspring

bspatial alignment