A Study of Image Fusion Using Wavelet-based Weighted-Average Technique

with Automatic Weight Adjustment

Wattanit Hotrakool Department of Electronic and Electrical Engineering, University of Sheffield

I. INTRODUCTION

This coursework is given as a part of study in Computer Vision program. The aims of this coursework is to study

and implement the Wavelet-based image fusion technique, which is one of the most popular image fusion technique

for fusing two or more images together with very high quality. The objectives of this coursework are 1) to

understand the Discrete Wavelet Transform, 2) to implement the image fusion technique using Wavelet

transformation to fuse various types of images, and 3) to study the effect of different parameters and decision rules.

II. BACKGROUND

Image fusion is an attempt to combine information from two or more images of the same scene together into one

single image. The image that is the outcome of image fusion will be richer in information than any of the input

images. The image fusion can be used in the wide varieties of applications such as combining information from

many sensors or cameras together in surveillance mission, or remote sensing, to combine information from many

devices for better diagnosis in medical imaging, and to improve the quality of images by solving out-of-focus

problem or lighting problem for landscape pictures.

There are many kinds of input images for image fusion, for examples, multi-focus images, multi-sensors images,

and multi-exposure images, etc. The multi-focus images are the images that their focuses are on the different subject

of the same scene. The multi-sensors images are the images which taken from the same scene using a different kind

of device. CT - MRI images and Normal - IR camera are the examples of this kind of images. The multi-exposure

images are the images taken from the same scene with different amounts of light, so some image can be dark

whereas some can be bright.

There are many proposed technique for image fusion. These techniques can be divided into 2 categories which are

1) image fusion in a spatial domain. The examples of these techniques are the intensity weighted-average, and the

principle component analysis [1]. 2) Image fusion using transformation domain, there are many different approaches

in this category. The popular transformation approaches are the multi-resolution decomposition (e.g. Gaussian

pyramid, Laplacian pyramid), and the image fusion using the discrete wavelet transformation [2].

Normally, the wavelet-based fusion is the most popular among the above mentioned techniques because it

contains both multi-resolution property and also contains both structural and detail information of images. Therefore

the fused image that is the output of this technique mostly be higher quality than any other techniques in most

situations.

A. Discr

The 2-

process to

that repre

into serie

The wa

images. I

both frequ

wavelet d

frequency

B. Fusio

To do t

the image

basic tech

window.

input with

coefficien

and wave

L

rete Wavelet Tra

-dimensional

o transform im

esents the wav

s of low-pass

Figure 1. Proced

(

avelet coeffic

In contrast wi

uency (detail)

domain shoul

y domain.

on Rule

the image fus

es will be fuse

hniques to be

Weighted-ave

h the highest v

nt to small, loc

elet-based fusi

Image

Low frequency

High frequency

LP Filter

HP Filter

ansform

discrete wav

mages from a

velet coefficien

and high-pass

dure of 2-D wave

(a) Spacial dom

ient clearly sh

th Fourier Tra

) and structure

ld give better

sion, both ima

ed. There are

e used as a de

erage is to tak

value to be the

cal regions of

on.

LP Filter

LP Filter

HP

HP

III

elet transform

spatial domai

nts of the ima

s filters as show

let transform

main

Figure 3. The

hows the both

ansform that

e of the image

r details than

ages are fused

many propose

ecision rule a

ke the average

e output. And

f the images. A

Low –High frequency

High –Low frequency

Low –Low frequency

High –High frequency

I. METHODOL

m (2D-DWT)

in into a wave

ages. The wave

ws in figure.

example of wavel

h low frequen

shows only fr

s in the same

in the spatia

d together unde

ed techniques

are: 1) weight

of all images

d the sliding w

All of these te

LOGY

, sometimes

elet domain [3

elet decompos

Figure

let decomposition

ncy componen

requency info

time. Because

al domain, an

er some decis

for decision r

ted-average 2)

to be an outp

window is to do

chniques can

LL2

HL

called wavele

3]. The wave

sition can be d

2. The wavelet co

(b) Waveln

nts and high f

ormation, wav

e of this advan

nd be more s

sion rules, the

rule. Among a

) maximum c

put. Maximum

o either weigh

be apply to bo

HL1

2 LH2

2 HH2

et decomposi

elet domain is

done by passin

oefficient domain

let domain

frequency com

elet transform

ntage, the ima

structural con

algorithm to

all techniques

oefficient, an

m coefficient is

hted-average o

oth intensity-b

LH1

HH1

ition, is the

the domain

ng an image

n

mponents of

m can shows

age fusion in

nsistent than

decide how

, the 3 most

d 3) sliding

s to take the

or maximum

based fusion

In this coursework we do the wavelet-based image fusion, therefore, the decision rule chose for this coursework

will be applied to fuse the wavelet coefficient of images instead of intensities. According to some research on image

fusion [4], [5], showing that the weighed-average are the best fusion technique among all basic fusion technique in

general case. So the wavelet weighted-average technique is chosen as the fundamental fusion rule in this

coursework.

C. Automatic weight adjustment

Because the performance and quality of the fused image that is a result from wavelet-based weighted-average

image fusion technique is largely depends on weights of each instance. The automatic weight adjustment technique

is proposed in order to find the appropriate weights of any pair of input images, to make the most reasonable fused

image. The term reasonable image here is defined as the image which contains appropriate amount of information

from all its inputs. In this case the reasonable image may not contain some information that may overpower the

smaller details from other image, trying to preserve small details as much as possible. The reasonable image in this

coursework is defined as the output image that has the highest value of structural similarity index (SSIM) that is = ∈ ( ( , ) + ( , )) where A, B are input image, O is output image, and I is a set of output from all weight coefficient. The structural

similarity index used here is the statistical index used for measure the similarity between any two images. The result

from SSIM is in range of 0 (totally different) to 1 (totally the same) [6]. The SSIM has more advantages than the

traditional way to measure image similarity such as Mean-squared Error (MSE). One of the advantages is SSIM

result in more consistent with visual inspection, i.e. likely the same with the human eyes. The SSIM is defined as ( , ) = (2 + )(2 + )( + + )( + + ) where is the average of x, is the average of y, is the variance of x, is the variance of y, is the

covariance of x and y, = ( ) and = ( ) where L is the dynamic range of images (in this case is 28) and

k1 and k2 are 0.01 and 0.03 accordingly.

III. DESIGN

In this coursework, the image fusion is implemented in MATLAB program. The structure of program is shown in

figure . The program consists of two main parts; 1) the image fusion algorithm, and 2) weight adjustment algorithm.

The image fusion algorithm is just like other normal wavelet weighted-average. The weigh adjustment algorithm is

composed of the computing of SSIM over the entire range of weight and the weight selection that select the

appropriate weight based on the computed SSIM. This weight will be assigned to the weighted-average fusion rule.

However, since this technique rely on the similarity between input and output image, so the inverse discrete

wavelet transform (IDWT) is needed to compute in every loop of weight-varying process. Hence made the overall

speed of algorithm reduced.

In the design of this algorithm, the mechanism to handle with various input colour space was also included. So

every type of input image, especially the index colour, will be converted to Grayscale before combining and before

computing SSIM.

Figure 5. image B,

Fig

(a)

(c) Result of multi-fo(c) fused image, (

(a)

gure 4. Algorithm

focus images (a) in(d) plot between w

m of proposed wav

(b)

(d)

nput image A, (b)weight and SSIM

(b)

velet weighted-av

IV. RESULT

) input M index

Figu

verage with autom

T

(a)

(c)ure 6. Result of me

B, (c) fused imag

(a)

matic weight adjus

edical images (a) ge, (d) plot betwe

stment

(b)

(d) input image A, (ben weight and SS

(b)

b) input image SIM index

Figure 7. Rimage B,

As seen

they quit

sometime

medical i

However

informati

This te

together t

to be regi

and make

During

running [

will sligh

The wa

based we

proposed

images. T

for each p

[1] MohamAnaly

[2] Guoliaon Me

[3] Gonza[4] Anjali

Techn[5] Anjali

Comp[6] Zhou W

Structu[7] Xing S

Fusion

(c) Result of multi-se(c) fused image, (

n from the ab

te look like i

es they didn’t

image and sce

these results

on.

echnique is als

therefore some

istered prior to

e entire algorit

g this evaluati

7]. However,

htly change; he

avelet-based i

eighted-averag

under the ass

Therefore the

pairs of input i

med R. Metwalli,ysis and High-Passang Tang, Jiexin Pechatronics and Aalo Pajares and Gi Malviya, and S.

nologies in Commi Malviya, and S. uter Science, 200Wang, Alan Conrural Similarity,” ISu-xia, Guo Pei-yn” International C

ensors images (a) (d) plot between w

ove result, the

images taken

contain so mu

enery image t

are said to be

so has drawba

e sharp details

o the algorithm

thm to be failu

ion, the wave

in case that w

ence the SSIM

mage fusion i

ge fusion techn

sumption that

computation o

images.

, Ayman H. Nasr,s Filter,” IEEE, 2Pu and Xinhan H

Automation, 2006.onzalo Pajares, “AG. Bhirud “Multiunication and CoG. Bhirud “Wave

09. rad Bovik, HamidIEEE Transl. IMA

yuan, and Chen TConference on Me

(d) input image A, (bweight and SSIM

e results from

physically. E

uch informatio

took a large p

e more beauti

ack as follows

s will be reduc

m. If the input

ure.

elet decompos

wavelet decomp

M and the final

V

is a very high

nique with au

the best fused

of SSIM is int

, Osama S. Farag 009.

Huang, “Image Fus. A wavelet-based i-Focus Image Fu

omputing, 2009. elet Based Multi-F

d Rahim Sheikh, aAGE PROCESSINian-hua, “Study oeasuring Technol

b) input M index

Figuima

V. DISCUSSIO

m this techniqu

Even though

on. This can b

portion of inf

ful than the fu

s; 1) the high

ced, especially

t images are no

sition of level

mposition level

weight may a

VI. CONCLUSI

h quality imag

utomatic weigh

d output should

troduced to m

REFERENCE

Allah, and S. El-R

sion Based on Mu

image fusion tutou sion of Digital Im

Focus Image Fusi

and Eero P. SimoNG, vol. 13, no. 4on Optimal Wavellogy and Mechatr

(c)ure 8. Result of muage B, (c) fused im

ON

ue should be s

these images

be seen from f

formation from

fused image th

frequency inf

y in multi-focu

ot registered, t

l 3 is used fo

is changed, th

also slightly ch

ION

ge fusion techn

hted adjustme

d be the outpu

measure this sim

S

Rabaie, “Image F

ulti-wavelet Trans

orial,” Pattern Recmages,” Internati

ion,” Internation

oncelli, “Image Qu4, April 2004. let Decompositio

ronics Automation

ulti-temporal imamage, (d) plot bet

aid to be “mo

are very com

figure , and fi

m the image t

hat is flooded

formation of b

us image, and

the computati

or limiting th

he result and q

hanges.

nique. In this

nt algorithm i

ut that most si

milarity and f

Fusion Based on P

sform,” Proc. in I

cognition 37, 200ional Conference

al Conference on

uality Assessmen

n Level in Infraren, 2010.

(d) ages (a) input imagtween weight and

ore natural”, w

mfort to see

igure , that res

that is more i

with large, sa

both images a

d 2) the input i

on of SSIM w

e energy affe

quality of the

coursework t

is studied. Th

imilar to both

find the best w

Principal Compon

IEEE Internationa

04, pp. 1855 – 187on Advances in R

n Methods and Mo

t: From Error Vis

ed and visual Ligh

ge A, (b) input d SSIM index

which means

for human,

sults of both

informative.

ame priority

are averaged

images need

would be fail

ected during

fused image

the wavelet-

is algorithm

of the input

weight value

nent

al Conference

72. Recent

odels in

sibility to

ht Image

APPENDIX

MATLAB CODE FOR IMAGE FUSION

function [I, OW, OWTable] = DoFusion(A, B, varargin) % Fuse 2 images together into one single image using automatic weight % adjustment and structural similarity index % % Usage: [I, OW, OWTable] = DoFusion(A, B) % [I, OW, OWTable] = DoFusion(A, B, s) % [I, OW, OWTable] = DoFusion(A, B, s, step) % % A, B are input images , can be pre-read matrices or file names. % s is a wavelet decomposition level, default s = 3 % step is a step size of vary weight , default step = 0.05 % I is a output fused image % OW is a value of optical weight using for image I % OWTable is a table to store value of all weight-SSIM pair % % Author: Wattanit Hotrakool % The University of Sheffield % Registration No: 100135895 % % Date 16/11/2010 % Read input images img1 = input_translation(A); img2 = input_translation(B); % Get the decomposition level s = get_decomposite_level(varargin); % Get the step size step = get_stepsize(varargin); % Perform the wavelet transform img1_wl = F2DWT(img1,s); img2_wl = F2DWT(img2,s); % Declare a OWTable OWTable = zeros(1+1/step,2); idx = 1; for Weight = 0:step:1 % Compute the SSIM for each value of weight per each iteration out_wl = Weight*abs(img1_wl) + (1-Weight)*img2_wl; tempout = I2DWT(out_wl,s); % Compute the SSIM SSIM = FindSSIM(tempout,img1)+FindSSIM(tempout,img2); % Record the weight and SSIM OWTable(idx,1) = Weight; OWTable(idx,2) = SSIM; idx = idx+1;

end % Find the weight corresponding to maximum SSIM [~,i] = min(OWTable(:,2)); FuseWeight = OWTable(i,1); % Use optimum weight to fused and reconstruct image. out_wl = FuseWeight *img1_wl + (1-FuseWeight)*img2_wl; out = I2DWT(out_wl,s); I = uint8(out); OW = FuseWeight; figure(10), imshow(A),title('Input A'); figure(20),imshow(B),title('Input B'); figure(30), imshow(I),title('Fused Image'); end function out = input_translation(in) % load input image into a matrix. % If the input image is specified as a file name, load image using % imread command. % If the input image is in index color space, convert the image into % grayscale. if (ischar(in)) [out,map] = imread(in); if size(map) ~= 0 out = ind2gray(out,map); end else out = in; end out = im2uint8(out); end function s = get_decomposite_level(inarg) % Set the wavelet decomposite level nofarg = size(inarg,2); s = 3; if nofarg >= 1; s = inarg{1}; end; end function step = get_stepsize(inarg) % Get the step size nofarg = size(inarg,2); step = 0.05; if nofarg >= 2; step = inarg{2}; end; if step > 1 || step < 0; step = 0.05; end; end

function SSIM = FindSSIM(im1,im2) % Compute the value of structural similarity (SSIM) % If the input image is in RGB space, convert into grayscale. if isrgb(im1); im1 = rgb2gray(im1); end if isrgb(im2); im2 = rgb2gray(im2); end ux = mean2(im1); uy = mean2(im2); sdx = std2(im1); sdy = std2(im2); varx = sdx^2; vary = sdx^2; corr = corr2(im1,im2); cov = corr*sdx*sdy; L = 255; k1 = 0.01; k2 = 0.03; c1 = (k1*L)^2; c2 = (k2*L)^2; SSIM = (2*ux*uy+c1)*(2*cov+c2)/((ux^2)+(uy^2)+c1)*(varx+vary+c2); end function [result] = isrgb(img) % Check whether the image is in RGB or not. sz = size(size(img)); result = 0; if sz(2) == 3; result = 1; end end