A Study of Image Fusion Using Wavelet-based Weighted-Average Technique
with Automatic Weight Adjustment
Wattanit Hotrakool Department of Electronic and Electrical Engineering, University of Sheffield
I. INTRODUCTION
This coursework is given as a part of study in Computer Vision program. The aims of this coursework is to study
and implement the Wavelet-based image fusion technique, which is one of the most popular image fusion technique
for fusing two or more images together with very high quality. The objectives of this coursework are 1) to
understand the Discrete Wavelet Transform, 2) to implement the image fusion technique using Wavelet
transformation to fuse various types of images, and 3) to study the effect of different parameters and decision rules.
II. BACKGROUND
Image fusion is an attempt to combine information from two or more images of the same scene together into one
single image. The image that is the outcome of image fusion will be richer in information than any of the input
images. The image fusion can be used in the wide varieties of applications such as combining information from
many sensors or cameras together in surveillance mission, or remote sensing, to combine information from many
devices for better diagnosis in medical imaging, and to improve the quality of images by solving out-of-focus
problem or lighting problem for landscape pictures.
There are many kinds of input images for image fusion, for examples, multi-focus images, multi-sensors images,
and multi-exposure images, etc. The multi-focus images are the images that their focuses are on the different subject
of the same scene. The multi-sensors images are the images which taken from the same scene using a different kind
of device. CT - MRI images and Normal - IR camera are the examples of this kind of images. The multi-exposure
images are the images taken from the same scene with different amounts of light, so some image can be dark
whereas some can be bright.
There are many proposed technique for image fusion. These techniques can be divided into 2 categories which are
1) image fusion in a spatial domain. The examples of these techniques are the intensity weighted-average, and the
principle component analysis [1]. 2) Image fusion using transformation domain, there are many different approaches
in this category. The popular transformation approaches are the multi-resolution decomposition (e.g. Gaussian
pyramid, Laplacian pyramid), and the image fusion using the discrete wavelet transformation [2].
Normally, the wavelet-based fusion is the most popular among the above mentioned techniques because it
contains both multi-resolution property and also contains both structural and detail information of images. Therefore
the fused image that is the output of this technique mostly be higher quality than any other techniques in most
situations.
A. Discr
The 2-
process to
that repre
into serie
The wa
images. I
both frequ
wavelet d
frequency
B. Fusio
To do t
the image
basic tech
window.
input with
coefficien
and wave
L
rete Wavelet Tra
-dimensional
o transform im
esents the wav
s of low-pass
Figure 1. Proced
(
avelet coeffic
In contrast wi
uency (detail)
domain shoul
y domain.
on Rule
the image fus
es will be fuse
hniques to be
Weighted-ave
h the highest v
nt to small, loc
elet-based fusi
Image
Low frequency
High frequency
LP Filter
HP Filter
ansform
discrete wav
mages from a
velet coefficien
and high-pass
dure of 2-D wave
(a) Spacial dom
ient clearly sh
th Fourier Tra
) and structure
ld give better
sion, both ima
ed. There are
e used as a de
erage is to tak
value to be the
cal regions of
on.
LP Filter
LP Filter
HP
HP
III
elet transform
spatial domai
nts of the ima
s filters as show
let transform
main
Figure 3. The
hows the both
ansform that
e of the image
r details than
ages are fused
many propose
ecision rule a
ke the average
e output. And
f the images. A
Low –High frequency
High –Low frequency
Low –Low frequency
High –High frequency
I. METHODOL
m (2D-DWT)
in into a wave
ages. The wave
ws in figure.
example of wavel
h low frequen
shows only fr
s in the same
in the spatia
d together unde
ed techniques
are: 1) weight
of all images
d the sliding w
All of these te
LOGY
, sometimes
elet domain [3
elet decompos
Figure
let decomposition
ncy componen
requency info
time. Because
al domain, an
er some decis
for decision r
ted-average 2)
to be an outp
window is to do
chniques can
LL2
HL
called wavele
3]. The wave
sition can be d
2. The wavelet co
(b) Waveln
nts and high f
ormation, wav
e of this advan
nd be more s
sion rules, the
rule. Among a
) maximum c
put. Maximum
o either weigh
be apply to bo
HL1
2 LH2
2 HH2
et decomposi
elet domain is
done by passin
oefficient domain
let domain
frequency com
elet transform
ntage, the ima
structural con
algorithm to
all techniques
oefficient, an
m coefficient is
hted-average o
oth intensity-b
LH1
HH1
ition, is the
the domain
ng an image
n
mponents of
m can shows
age fusion in
nsistent than
decide how
, the 3 most
d 3) sliding
s to take the
or maximum
based fusion
In this coursework we do the wavelet-based image fusion, therefore, the decision rule chose for this coursework
will be applied to fuse the wavelet coefficient of images instead of intensities. According to some research on image
fusion [4], [5], showing that the weighed-average are the best fusion technique among all basic fusion technique in
general case. So the wavelet weighted-average technique is chosen as the fundamental fusion rule in this
coursework.
C. Automatic weight adjustment
Because the performance and quality of the fused image that is a result from wavelet-based weighted-average
image fusion technique is largely depends on weights of each instance. The automatic weight adjustment technique
is proposed in order to find the appropriate weights of any pair of input images, to make the most reasonable fused
image. The term reasonable image here is defined as the image which contains appropriate amount of information
from all its inputs. In this case the reasonable image may not contain some information that may overpower the
smaller details from other image, trying to preserve small details as much as possible. The reasonable image in this
coursework is defined as the output image that has the highest value of structural similarity index (SSIM) that is = ∈ ( ( , ) + ( , )) where A, B are input image, O is output image, and I is a set of output from all weight coefficient. The structural
similarity index used here is the statistical index used for measure the similarity between any two images. The result
from SSIM is in range of 0 (totally different) to 1 (totally the same) [6]. The SSIM has more advantages than the
traditional way to measure image similarity such as Mean-squared Error (MSE). One of the advantages is SSIM
result in more consistent with visual inspection, i.e. likely the same with the human eyes. The SSIM is defined as ( , ) = (2 + )(2 + )( + + )( + + ) where is the average of x, is the average of y, is the variance of x, is the variance of y, is the
covariance of x and y, = ( ) and = ( ) where L is the dynamic range of images (in this case is 28) and
k1 and k2 are 0.01 and 0.03 accordingly.
III. DESIGN
In this coursework, the image fusion is implemented in MATLAB program. The structure of program is shown in
figure . The program consists of two main parts; 1) the image fusion algorithm, and 2) weight adjustment algorithm.
The image fusion algorithm is just like other normal wavelet weighted-average. The weigh adjustment algorithm is
composed of the computing of SSIM over the entire range of weight and the weight selection that select the
appropriate weight based on the computed SSIM. This weight will be assigned to the weighted-average fusion rule.
However, since this technique rely on the similarity between input and output image, so the inverse discrete
wavelet transform (IDWT) is needed to compute in every loop of weight-varying process. Hence made the overall
speed of algorithm reduced.
In the design of this algorithm, the mechanism to handle with various input colour space was also included. So
every type of input image, especially the index colour, will be converted to Grayscale before combining and before
computing SSIM.
Figure 5. image B,
Fig
(a)
(c) Result of multi-fo(c) fused image, (
(a)
gure 4. Algorithm
focus images (a) in(d) plot between w
m of proposed wav
(b)
(d)
nput image A, (b)weight and SSIM
(b)
velet weighted-av
IV. RESULT
) input M index
Figu
verage with autom
T
(a)
(c)ure 6. Result of me
B, (c) fused imag
(a)
matic weight adjus
edical images (a) ge, (d) plot betwe
stment
(b)
(d) input image A, (ben weight and SS
(b)
b) input image SIM index
Figure 7. Rimage B,
As seen
they quit
sometime
medical i
However
informati
This te
together t
to be regi
and make
During
running [
will sligh
The wa
based we
proposed
images. T
for each p
[1] MohamAnaly
[2] Guoliaon Me
[3] Gonza[4] Anjali
Techn[5] Anjali
Comp[6] Zhou W
Structu[7] Xing S
Fusion
(c) Result of multi-se(c) fused image, (
n from the ab
te look like i
es they didn’t
image and sce
these results
on.
echnique is als
therefore some
istered prior to
e entire algorit
g this evaluati
7]. However,
htly change; he
avelet-based i
eighted-averag
under the ass
Therefore the
pairs of input i
med R. Metwalli,ysis and High-Passang Tang, Jiexin Pechatronics and Aalo Pajares and Gi Malviya, and S.
nologies in Commi Malviya, and S. uter Science, 200Wang, Alan Conrural Similarity,” ISu-xia, Guo Pei-yn” International C
ensors images (a) (d) plot between w
ove result, the
images taken
contain so mu
enery image t
are said to be
so has drawba
e sharp details
o the algorithm
thm to be failu
ion, the wave
in case that w
ence the SSIM
mage fusion i
ge fusion techn
sumption that
computation o
images.
, Ayman H. Nasr,s Filter,” IEEE, 2Pu and Xinhan H
Automation, 2006.onzalo Pajares, “AG. Bhirud “Multiunication and CoG. Bhirud “Wave
09. rad Bovik, HamidIEEE Transl. IMA
yuan, and Chen TConference on Me
(d) input image A, (bweight and SSIM
e results from
physically. E
uch informatio
took a large p
e more beauti
ack as follows
s will be reduc
m. If the input
ure.
elet decompos
wavelet decomp
M and the final
V
is a very high
nique with au
the best fused
of SSIM is int
, Osama S. Farag 009.
Huang, “Image Fus. A wavelet-based i-Focus Image Fu
omputing, 2009. elet Based Multi-F
d Rahim Sheikh, aAGE PROCESSINian-hua, “Study oeasuring Technol
b) input M index
Figuima
V. DISCUSSIO
m this techniqu
Even though
on. This can b
portion of inf
ful than the fu
s; 1) the high
ced, especially
t images are no
sition of level
mposition level
weight may a
VI. CONCLUSI
h quality imag
utomatic weigh
d output should
troduced to m
REFERENCE
Allah, and S. El-R
sion Based on Mu
image fusion tutou sion of Digital Im
Focus Image Fusi
and Eero P. SimoNG, vol. 13, no. 4on Optimal Wavellogy and Mechatr
(c)ure 8. Result of muage B, (c) fused im
ON
ue should be s
these images
be seen from f
formation from
fused image th
frequency inf
y in multi-focu
ot registered, t
l 3 is used fo
is changed, th
also slightly ch
ION
ge fusion techn
hted adjustme
d be the outpu
measure this sim
S
Rabaie, “Image F
ulti-wavelet Trans
orial,” Pattern Recmages,” Internati
ion,” Internation
oncelli, “Image Qu4, April 2004. let Decompositio
ronics Automation
ulti-temporal imamage, (d) plot bet
aid to be “mo
are very com
figure , and fi
m the image t
hat is flooded
formation of b
us image, and
the computati
or limiting th
he result and q
hanges.
nique. In this
nt algorithm i
ut that most si
milarity and f
Fusion Based on P
sform,” Proc. in I
cognition 37, 200ional Conference
al Conference on
uality Assessmen
n Level in Infraren, 2010.
(d) ages (a) input imagtween weight and
ore natural”, w
mfort to see
igure , that res
that is more i
with large, sa
both images a
d 2) the input i
on of SSIM w
e energy affe
quality of the
coursework t
is studied. Th
imilar to both
find the best w
Principal Compon
IEEE Internationa
04, pp. 1855 – 187on Advances in R
n Methods and Mo
t: From Error Vis
ed and visual Ligh
ge A, (b) input d SSIM index
which means
for human,
sults of both
informative.
ame priority
are averaged
images need
would be fail
ected during
fused image
the wavelet-
is algorithm
of the input
weight value
nent
al Conference
72. Recent
odels in
sibility to
ht Image
APPENDIX
MATLAB CODE FOR IMAGE FUSION
function [I, OW, OWTable] = DoFusion(A, B, varargin) % Fuse 2 images together into one single image using automatic weight % adjustment and structural similarity index % % Usage: [I, OW, OWTable] = DoFusion(A, B) % [I, OW, OWTable] = DoFusion(A, B, s) % [I, OW, OWTable] = DoFusion(A, B, s, step) % % A, B are input images , can be pre-read matrices or file names. % s is a wavelet decomposition level, default s = 3 % step is a step size of vary weight , default step = 0.05 % I is a output fused image % OW is a value of optical weight using for image I % OWTable is a table to store value of all weight-SSIM pair % % Author: Wattanit Hotrakool % The University of Sheffield % Registration No: 100135895 % % Date 16/11/2010 % Read input images img1 = input_translation(A); img2 = input_translation(B); % Get the decomposition level s = get_decomposite_level(varargin); % Get the step size step = get_stepsize(varargin); % Perform the wavelet transform img1_wl = F2DWT(img1,s); img2_wl = F2DWT(img2,s); % Declare a OWTable OWTable = zeros(1+1/step,2); idx = 1; for Weight = 0:step:1 % Compute the SSIM for each value of weight per each iteration out_wl = Weight*abs(img1_wl) + (1-Weight)*img2_wl; tempout = I2DWT(out_wl,s); % Compute the SSIM SSIM = FindSSIM(tempout,img1)+FindSSIM(tempout,img2); % Record the weight and SSIM OWTable(idx,1) = Weight; OWTable(idx,2) = SSIM; idx = idx+1;
end % Find the weight corresponding to maximum SSIM [~,i] = min(OWTable(:,2)); FuseWeight = OWTable(i,1); % Use optimum weight to fused and reconstruct image. out_wl = FuseWeight *img1_wl + (1-FuseWeight)*img2_wl; out = I2DWT(out_wl,s); I = uint8(out); OW = FuseWeight; figure(10), imshow(A),title('Input A'); figure(20),imshow(B),title('Input B'); figure(30), imshow(I),title('Fused Image'); end function out = input_translation(in) % load input image into a matrix. % If the input image is specified as a file name, load image using % imread command. % If the input image is in index color space, convert the image into % grayscale. if (ischar(in)) [out,map] = imread(in); if size(map) ~= 0 out = ind2gray(out,map); end else out = in; end out = im2uint8(out); end function s = get_decomposite_level(inarg) % Set the wavelet decomposite level nofarg = size(inarg,2); s = 3; if nofarg >= 1; s = inarg{1}; end; end function step = get_stepsize(inarg) % Get the step size nofarg = size(inarg,2); step = 0.05; if nofarg >= 2; step = inarg{2}; end; if step > 1 || step < 0; step = 0.05; end; end
function SSIM = FindSSIM(im1,im2) % Compute the value of structural similarity (SSIM) % If the input image is in RGB space, convert into grayscale. if isrgb(im1); im1 = rgb2gray(im1); end if isrgb(im2); im2 = rgb2gray(im2); end ux = mean2(im1); uy = mean2(im2); sdx = std2(im1); sdy = std2(im2); varx = sdx^2; vary = sdx^2; corr = corr2(im1,im2); cov = corr*sdx*sdy; L = 255; k1 = 0.01; k2 = 0.03; c1 = (k1*L)^2; c2 = (k2*L)^2; SSIM = (2*ux*uy+c1)*(2*cov+c2)/((ux^2)+(uy^2)+c1)*(varx+vary+c2); end function [result] = isrgb(img) % Check whether the image is in RGB or not. sz = size(size(img)); result = 0; if sz(2) == 3; result = 1; end end