GRAPH-BASED FUSION FOR CHANGE DETECTION IN MULTI-SPECTRAL IMAGES

David Alejandro Jimenez-Sierra†, Hernan Darıo Benıtez-Restrepo†, Hernan Darıo Vargas-Cardona†

Jocelyn Chanussot?

† Pontificia Universidad Javeriana CaliDepartamento de Electronica y Ciencias de la Computacion{davidjimenez, hbenitez, hernan.vargas}@javerianacali.edu.co

? Grenoble Images Parole Signals Automatique Laboratory (GIPSA-Lab)Grenoble Institute of Technology

jocelyn.chanussot@gipsa-lab.grenoble-inp.fr

ABSTRACTIn this paper we address the problem of change detection

in multi-spectral images by proposing a data-driven frame-work of graph-based data fusion. The main steps of the pro-posed approach are: (i) The generation of a multi-temporalpixel based graph, by the fusion of intra-graphs of each tem-poral data; (ii) the use of Nystrom extension to obtain theeigenvalues and eigenvectors of the fused graph, and the se-lection of the final change map. We validated our approach intwo real cases of remote sensing according to both qualitativeand quantitative analyses. The results confirm the potential ofthe proposed graph-based change detection algorithm outper-forming state-of-the-art methods.

Index Terms— Change detection, data fusion, graph,multi-spectral, multi-temporal, remote sensing.

1. INTRODUCTION

Change detection (CD) refers to the task of analyzing two ormore images acquired over the same area at different times(i.e., multitemporal images) in order to detect zones in whichthe land-cover type changed between the acquisitions [1].CD permits to quantify the magnitude of natural disasters (i.efloodings) and changes generated by human activity. Thisanalysis provides fundamental data for environmental protec-tion, sustainable development, and maintenance of ecologicalbalance [2]. One of the most known sources of data forchange detection are the Multi-spectral (MS) images thatcontain information from both spatial and spectral domain(i.e. Landsat series of satellites). Giving two or more co-registered images, pixel based approaches carry out changedetection by probabilistic thresholding and machine learningmethods [3, 4]. Even though threshold methods are efficientand useful, they are sensitive to MS image noise and require ahigh accuracy in the estimation of the difference image prob-abilistic distribution. These issues make threshold methodsprone to artifacts in the final change map [5–9]. Machine

learning approaches are divided into two categories: clas-sification and clustering. Classification methods require amultitemporal reference, which is difficult to extract fromthe raw data, therefore, these methods are not a practicalsolution [10]. Clustering techniques [11–15] are affected byparameter initialization, which may generate local minimain the learning stage. In addition, the intrinsic brightnessdistortion in MS images yields inaccurate change maps [4].

In order to reduce the effect of intra-class small variabilityand artifacts presented in MS images, we proposed a graph-based data fusion approach applied to CD. Our method ex-tracts features from the dataset by finding eigenvectors of thenormalized graph Laplacian and applying a mutual informa-tion based criteria to extract the relevant eigenvector that cap-tures the change map. We validate our approach in two realcases: (i) a flooding, and (ii) a fire incident. Results show thatour model reduces the effects of artifacts in the final changemap, and it achieves low rates of false alarms in comparisonto probabilistic threshold methods [5–7].

2. GRAPH BASED DATA FUSION

2.1. Graph

A graph is a non linear structural representation of data, de-fined by G = (V,E), where G is the graph, V is a set ofnodes, and E refers to the arcs or edges that explain the di-rected or undirected relationship between nodes. The edgeshave associated a weight wi,j , that quantifies how strong therelationship is between nodes. The common measure usedfor each weight is a Gaussian kernel with standard deviationσ [16].

2.2. Graph-based fusion for change detection (GBF-CD)

2.2.1. Nystrom extension

Given the high number of pixels in an MS image, the com-putational cost of calculating the full matrix W ∈ RN×N is

arX

iv:2

004.

0078

6v1

[cs

.CV

] 2

Apr

202

0

extremely high (i.e an image with size 1280 × 960 is equiv-alent to N = 1228800). Therefore, an approximation of thismatrix is computed through the Nystrom extension [17]:

W = κG

([dAA dAB

dAB> C

]),

where κG is a Gaussian kernel, dAA ∈ Rns×ns , dAB ∈Rns×(N−ns) and C ∈ R(N−ns)×(N−ns). This method ap-proximates C by choosing ns samples from the dataset ofsize N (ns � N ), then W ≈ W = κG

([dAA;dAB

])>.

Thus, the eigenvectors of the matrix W, can be spanned byeigenvalues and eigenvectors of κG(dAA). Solving the diag-onalization of κG(dAA) (eigenvalues λ and eigenvectors U:κG(dAA) = U>ΛU), the eigenvectors of W can be spannedby U =

[U;κG(dAB)

>UΛ−1]>

. Since the approximatedeigenvectors U are not orthogonal, as explained in [17], toobtain orthogonal eigenvectors we use S = κG(dAA) +

κG(dAA)−12κG(dAB)κG(dAB)

>κG(dAA)−12 .Then, by

diagonalization of S (S = UsΛsUs) the final approximatedeigenvectors of W are given by:

U =

[κG(dAA)

κG(dAB)>κG(dAA)−

12

]UsΛs

− 12 .

2.2.2. Fusion step

Based on the methodology introduced in [18], where a nodeis understood as a pixel (i.e. image from different bands ortimes, also a mix of both) and it is assumed that all of themodalities are co-registered, the fusion of multi-temporal datais carried out by the procedure described in Figure 1, in whichfor each instance of time (Xk) the Algorithm 1 is performed.

In short, the algorithm output for one instance of time Xk

corresponds to the approximate normalized adjacency matrix

Algorithm 1: GBF for temporal data

Input: Temporal images Xk ∈ Rm×n, number ofsamples ns

Output: Fused graph WF ∈ R(ns+c)×ns

Initialize: k = 1, N = m ∗ nwhile k ≤ 2 do

1) Take ns samples from Xk

XkAA = sampler(Xk, ns), ∈ Rns

2) Find the complement Xk ∈ Rc of Xk

AA in Xk.3) For each set Xk

AA and Xk

perform the pairwisedistance between samples-samples(dAA

k ∈ Rns×ns ) and samples-complement(dAB

k ∈ Rc×ns ).dAA

k ={∥∥∥xkAAi

− xkAAj

∥∥∥2

}nsns

i j, ∀i 6= j

dABk =

{∥∥∥xki − xkAAj

∥∥∥32

}c ns

i j

, ∀i 6= j

4)Apply the normalized graph laplacian(D−

12 WD−

12 ) on the distances by using the code

in [17]5) Apply a Gaussian kernel (κG(.)) on thenormalized distances and build the approximatednormalized laplacian matrix based on theNystrom approximation.Wk

N =[κG(dAA

k);κG(dABk)]>

,k+ = 1

endWF = min(wk

Nij), with i = 1, .., c; j = 1, .., ns.

(WkN ) [17]. Then, the fusion step consist of capturing the

unique information given by each graph (WkN ) into one fused

graph (WF). In order to achieve this fusion, we maximizethe distance between pixels (i.e. choosing those pixels that

Before event

After event

𝑿𝑘 ∈ ℝ𝑚 ×𝑛

Take 𝑛𝑠 samples

(vector 𝑋𝐴𝐴𝑘 ) and

its complement

(vector 𝑋𝑘) for

each 𝑋𝑘

Pairwise distances

between samples-samples

𝑋𝐴𝐴𝑖𝑘 − 𝑋𝐴𝐴𝑗

𝑘

2 𝑖 𝑗

𝑛𝑠 𝑛𝑠

and

samples-complement

𝑋𝑘𝑖 − 𝑋𝐴𝐴𝑗𝑘

2

3

𝑖 𝑗

𝑐 𝑛𝑠

𝑋𝐴𝐴𝑘 ∈ ℝ𝑛𝑠

𝑋𝑘 ∈ ℝ𝑐

Perform normalized

graph Laplacian

𝑾𝑁 = 𝑫−𝟏

𝟐𝑾𝑫−𝟏

𝟐

And compute

Gaussian kernel

(𝜅𝐺)

𝒅𝑨𝑨𝑘 ∈ ℝ𝑛𝑠× 𝑛𝑠

Minimise similarity

between graphs

(Fusion step)

𝑾𝑭 = 𝑚𝑖𝑛 ෝ𝑤𝑖 𝑗𝑘 ;

k= [1,2]𝑖 = 1,2, … , 𝑐 ; 𝑗 = 1,2, … , 𝑛𝑠

𝑾𝑁𝑘 ∈ ℝ(𝑛𝑠+ 𝑐) × 𝑛𝑠

𝑾𝑭 ∈ ℝ(𝑛𝑠+ 𝑐) × 𝑛𝑠

𝒅𝑨𝑩𝑘 ∈ ℝ𝑐 × 𝑛𝑠

Loop for k = [1,2] (Instances of time)

Fig. 1. Graph-based fusion. Where, k is the time of the event 1 (pre) and 2 (post), Xk is an image that represents a event, XkAA

represents the samples from Xk, Xk

is the complement, dAAk is the pairwise distance between the samples in Xk

AA, dABk

is the pairwise distance between XkAA and X

k, D = Diag(d1, d2, . . . , dns) with di =

∑ns

j wkij is the approximated degree

matrix , and WkN is the normalized laplacian calculated by using the Nystrom approximation.

Apply Nyström

extensión

𝑾𝑭 ≈ 𝑼𝑫𝑼⊤

Get binarized prior

from the given input

images𝐼𝑏𝑒𝑓𝑜𝑟𝑒 − 𝐼𝑎𝑓𝑡𝑒𝑟

𝐼𝑏𝑒𝑓𝑜𝑟𝑒 + 𝐼𝑎𝑓𝑡𝑒𝑟

𝐼𝑃𝑟𝑖𝑜𝑟

Compute the new image from

each column vector

( 𝑢𝑖∈ ℝ(𝑛𝑠+ 𝑐) × 1 𝑖 = 1,2, …𝑛𝑠from 𝑼 weighted by the square

root of his eigenvalue (𝑑𝑖)

Calculate the mutual

information from the

given image ( 𝑋 = 𝐼𝑢𝑖 )

and the prior (𝑌 = 𝐼𝑃𝑟𝑖𝑜𝑟 )

𝑀𝐼 𝑋 , 𝑌 = 𝐸𝑃𝑋𝑌 log𝑃𝑋𝑌

𝑃𝑋𝑃𝑌

𝑀𝐼

𝑾𝑭 ∈ ℝ(𝑛𝑠+ 𝑐) × 𝑛𝑠

𝑰 𝒖𝒊 = 𝑢𝑖 𝑑𝑖

𝑰 𝒖𝒊 ∈ ℝ𝑚 × 𝑛

Select the

eigenvector ( 𝑢𝑖) and

its respective

eigenvalue ( 𝑑𝑖) that

maximise MI

(Change map)

𝐶ℎ𝑎𝑛𝑔𝑒 𝑀𝑎𝑝

𝑼 ∈ ℝ(𝑛𝑠+ 𝑐) × 𝑛𝑠

𝑫 ∈ ℝ𝑛𝑠 × 𝑛𝑠

Loop for 𝑖 = 1,2, … , 𝑛𝑠 eigenvectors

Fig. 2. Change detection. Where, WF is the fused graph, U is the approximate eigenvectors and D is the eigenvalues.

preserve most of the information):

WF = min(wkNij

),with k = [1, 2],

where wi,j represents the weight of the node for each in-stance of time (i = 1, 2, . . . , c; j = 1, 2, . . . , ns). In thissense, the learning of this approach is data driven (uses a fewns samples to learn) and it will be restarted from scratch foreach set of data.

3. APPLICATION OF GBF-CD FOR CHANGEDETECTION

3.1. Change detection scheme based on multi-temporalgraph

To obtain the change map from the multi-temporal graph (sec-tion 2), we apply the scheme detailed in Figure 2. Here, thepurpose is to attain the best match that reflects the changeproduced by any source. To do this, we use the eigenvectorsfrom the GBF-CD as descriptors of the change. Nevertheless,the number of eigenvectors is equal to the samples (ns) takenfrom the instances of time. Hence, we estimate the mutual in-formation to identify the relevant eigenvector that captures theglobal change. The output in Figure 2 is a vector that containsthe mutual information between the prior knowledge (differ-ence image) and the change map generated by the eigenvec-tors of the GBF-CD. Finally the change map detected is theeigen-image (Iui

) that maximizes the mutual information.

4. EXPERIMENTAL RESULTS AND DISCUSSION

4.1. Databases

Due to space limitations, we tested our approach on twodatasets.: Dataset A: NIR band images (Figure 3 (a-b)) wereacquired by the Thematic Mapper (TM) MS sensor of theLandsat-5 satellite. The scene represents an area includingLake Mulargia (Sardinia Island, Italy). The images consistof 573 × 479 pixels. The dates of acquisition were Septem-ber 1995 (before the event) and July 1996 (after the event).

Dataset B: RED band images (Figure 3 (c-d)) were acquiredby the Operational Land Image MS sensor of the Landsat-8satellite. The area includes Lake Omodeo and a portion ofTirso River (Sardinia Island, Italy). The images consist of965×742 pixels. The dates of acquisition were July 25, 2013(before the event) and August 10, 2013 (after the event).

Before

(a) Mulargia lake

After

(b) Flooded Mulargialake

Before

(c) Omodeo lake

After

(d) Fire near Omodeolake

Fig. 3. Satellite images from the NIR band for the flood event(Dataset A) and from the red band for fire event (Dataset B).

4.2. Experimental set-up

We compare the proposed GBF-CD with state of the art meth-ods: Rayleigh-Rice (rR) [6], Rayleigh-Rayleigh-Rice (rrR)[7], and the classical KittlerIllingsworth (KI) [5]. We evaluaterelevant metrics in change detection such as: missed alarms(MA), false alarms (FA), precision (P), recall (R), Cohen’skappa (K) and overall error (OE).

The number of samples (ns) was fixed at 92 and the stan-dard deviations (σ) for the kernels were σ1

lake = 2.5299 ×10−10, σ2

lake = 1.5561× 10−10, σ1fire = 2.793× 10−11 and

σ2fire = 1.6533×10−10, where the superscripts 1, 2 stands for

pre and post event respectively. We set these values through

exhaustive grid-search using MatLab R©2017a.

Table 1. Model Performance for dataset A.Method MA (%) FA (%) P R K OE (%)KI [5] 10.2425 1.0490 0.7229 0.8975 0.7941 1.3211

rR-EM [6] 5.7245 4.0147 0.4173 0.9427 0.5605 4.0653rrR-EM [7] 10.1440 1.0637 0.7203 0.8985 0.7928 1.3324GBF-CD 4.8504 0.3120 0.9029 0.9515 0.9242 0.4463

Table 2. Model Performance of dataset B.Method MA (%) FA(%) P R K OE (%)KI [5] 0 3.4291 0.5903 1 0.7262 3.2676

rR-EM [6] 0.0029 3.7382 0.5693 0.9999 0.7080 3.5623rrR-EM [7] 0.0029 2.1449 0.6973 0.9999 0.8112 2.0440GBF-CD 14.4217 0.1226 0.9718 0.8557 0.9059 0.7960

Table 1 shows the results for dataset A. We observe ourapproach outperforms the comparison methods for all met-rics. Similarly, Table 2 tabulates the outcomes for dataset B.Although, the KI method achieves a perfect score for MAand recall (R), the GBF-CD outperforms the state-of-the-artmethods in FA, P, K and OE.

Also, Figure 4 shows the behavior of each method interms of MA (blue points), FA (red points) and correctchanged pixels (green points). These results are remark-able, because we can see that probabilistic methods have aconsiderable number of FA in both datasets. Conversely,the GBF-CD deals well with this issue. FA is mostly gen-erated by the nearly similar intensity of pixels between realchanges regions and effects produced by the reflectance (i.e.weather variations, cloud density, daylight differences whenthe image was captured). The GBF-CD has some limitations.Firstly, for dataset A (see subfigure 4 (d)), we can observethat border of the change map is mainly composed by redpoints (FA). This is due to the neigboring pixels of the borderhave a similar intensity. Also, for dataset B (see figure 4(h)), the GBF-CD is unable to detect minor changes in theedge of lake Omodeo and the artifact located in the upper-leftcorner. For this reason, there are some missed alarms (MA)represented by the blue points.

Further work includes: (i) to decrease the dependence ofthe results with respect to the number of selected samples forthe Nystrom extension, (ii) to select an alternative metric in-stead of Euclidean distance (ED) to increase the differencebetween intensities in MS images and avoid raising the ED tothe power of three, (iii) to explore other kernel types.

5. CONCLUSIONS

In this paper, we introduced a change detection methodology(GBF-CD) based on graphs data fusion. Our main contri-bution is a “data-driven” framework. Our method modelsthe dataset by finding eigenvectors of the normalized graphLaplacian and applying a mutual information based criteria to

To ensure the reproducibility of the proposed method, the code is pub-licly available at: https://github.com/DavidJimenezS

obtain the relevant eigenvector that captures the change map.Experimental results showed that GBF-CD outperformedprobabilistic threshold methods when we evaluated severalmetrics (MA, FA, R, P, K, OE) in two real cases of changedetection in remote sensing images.

According to the previous results and analysis, we con-clude that the GBF-CD is a promising and robust approachfor detecting changes in remote sensing images.Acknowledgments This work was funded by the OMICASprogram: “Optimizacion Multiescala In-silico de CultivosAgrıcolas Sostenibles (Infraestructura y validacion en Arroz yCana de Azucar)”, sponsored within the Colombian ScientificEcosystem by The WORLD BANK, COLCIENCIAS, ICE-TEX, the Colombian Ministry of Education and the Colom-bian Ministry of Industry and Turism under GRANT ID:FP44842-217-2018.

6. REFERENCES

[1] Mauro Dalla Mura, Saurabh Prasad, Fabio Pacifici,Paulo Gamba, Jocelyn Chanussot, and Jon Atli Benedik-tsson, “Challenges and opportunities of multimodalityand data fusion in remote sensing,” Proceedings of theIEEE, vol. 103, no. 9, pp. 1585–1601, 2015.

[2] Dana Lahat, Tulay Adali, and Christian Jutten, “Multi-modal data fusion: an overview of methods, challenges,and prospects,” Proceedings of the IEEE, vol. 103, no.9, pp. 1449–1477, 2015.

[3] Amir Yavariabdi and Huseyin Kusetogullari, “Changedetection in multispectral landsat images using multiob-jective evolutionary algorithm,” IEEE Geoscience andRemote Sensing Letters, vol. 14, no. 3, pp. 414–418,2017.

[4] Mi Song, Yanfei Zhong, and Ailong Ma, “Change detec-tion based on multi-feature clustering using differentialevolution for landsat imagery,” Remote Sensing, vol. 10,no. 10, pp. 1664, 2018.

[5] Josef Kittler and John Illingworth, “Minimum errorthresholding,” Pattern recognition, vol. 19, no. 1, pp.41–47, 1986.

[6] Massimo Zanetti, Francesca Bovolo, and Lorenzo Bruz-zone, “Rayleigh-rice mixture parameter estimation viaem algorithm for change detection in multispectral im-ages,” IEEE Transactions on Image Processing, vol. 24,no. 12, pp. 5004–5016, 2015.

[7] Massimo Zanetti and Lorenzo Bruzzone, “A theoret-ical framework for change detection based on a com-pound multiclass statistical model of the difference im-age,” IEEE Transactions on Geoscience and RemoteSensing, vol. 56, no. 2, pp. 1129–1143, 2017.

(a) KI (b) rR (c) rrR (d) GBF-CD

(e) KI (f) rR (g) rrR (h) GBF-CD

Fig. 4. Change map detected with respect to missed alarms (MA), false alarms (FA) and corrrect changed pixels (C).

[8] Ammar Mian, Guillaume Ginolhac, Jean-Philippe Ovar-lez, and Abdourrahmane Mahamane Atto, “New robuststatistics for change detection in time series of multivari-ate sar images,” IEEE Transactions on Signal Process-ing, vol. 67, no. 2, pp. 520–534, 2018.

[9] Redha Touati, Max Mignotte, and Mohamed Dahmane,“Multimodal change detection in remote sensing imagesusing an unsupervised pixel pairwise based markov ran-dom field model,” IEEE Transactions on Image Pro-cessing, 2019.

[10] Begum Demir, Francesca Bovolo, and Lorenzo Bruz-zone, “Classification of time series of multispectral im-ages with limited training data,” IEEE Transactions onImage Processing, vol. 22, no. 8, pp. 3219–3233, 2013.

[11] Ashish Ghosh, Niladri Shekhar Mishra, and SusmitaGhosh, “Fuzzy clustering algorithms for unsupervisedchange detection in remote sensing images,” Informa-tion Sciences, vol. 181, no. 4, pp. 699–715, 2011.

[12] Turgay Celik, “Change detection in satellite images us-ing a genetic algorithm approach,” IEEE Geoscienceand Remote Sensing Letters, vol. 7, no. 2, pp. 386–390,2010.

[13] Maoguo Gong, Zhiqiang Zhou, and Jingjing Ma,“Change detection in synthetic aperture radar imagesbased on image fusion and fuzzy clustering,” IEEETransactions on Image Processing, vol. 21, no. 4, pp.2141–2151, 2011.

[14] Vladimir A Krylov, Gabriele Moser, Sebastiano B Ser-pico, and Josiane Zerubia, “False discovery rate ap-proach to unsupervised image change detection,” IEEE

Transactions on Image Processing, vol. 25, no. 10, pp.4704–4718, 2016.

[15] Zhunga Liu, Gang Li, Gregoire Mercier, You He, andQuan Pan, “Change detection in heterogenous remotesensing images via homogeneous pixel transformation,”IEEE Transactions on Image Processing, vol. 27, no. 4,pp. 1822–1834, 2017.

[16] Rosa Mara Mnedez Parra Alfredo Caicedo Barrero, Gra-ciela Wagner de Garca, “Introduccion a la teorıa degrafos,” vol. 1, pp. 1–12. Elizcom s.a.s, 2010.

[17] Charless Fowlkes, Serge Belongie, Fan Chung, and Ji-tendra Malik, “Spectral grouping using the nystrommethod,” IEEE Transactions on Pattern Analysis andMachine Intelligence, vol. 26, no. 2, pp. 214–225, 2004.

[18] Geoffrey Iyer, Jocelyn Chanussot, and Andrea LBertozzi, “A graph-based approach for feature extrac-tion and segmentation of multimodal images,” in 2017IEEE International Conference on Image Processing(ICIP). IEEE, 2017, pp. 3320–3324.