Multi-Sensor Fusion

Wangyan Li

liwangyan618907@gmail.com

November 1, 2015

Outline

Introduction

Multi-Sensor Fusion

Fusion with unknown correlation

Fast Covariance Intersection

Outline

Introduction

Multi-Sensor Fusion

Fusion with unknown correlation

Fast Covariance Intersection

Blind men and elephant

Five senses

Outline

Introduction

Multi-Sensor Fusion

Fusion with unknown correlation

Fast Covariance Intersection

Origin

Multi-sensor fusion is also known as multi-sensor data fusion[1, 2], which is an emerging technology originally catered for themilitary needs, such as, battlefield surveillance, automated targetrecognition, remote sensing, and guidance and control ofautonomous vehicles. In recent years, the multi-sensor fusiontechnology has been extensively applied to a variety of civilianapplications, such as monitoring of complex machinery, medicaldiagnosis, and smart buildings, robotics, video and imageprocessing, and intelligent system design.

[1] D. Hall and J. Llinas, “An introduction to multisensor datafusion,” Proceedings of IEEE, vol. 85, no. 1, pp. 6–23, Jan.1997.

[2] B. Khaleghi, A. Khamis, F. O. Karray, and S. N. Razavi,“Multisensor data fusion: A review of the state-of-the-art,”Information Fusion, vol. 14, no. 1, pp. 28–44, Jan. 2013.

Definition

The essence of multi-sensor fusion techniques is to combine datafrom multiple sensors, and related information from associateddatabases, to achieve improved accuracies and more specificinferences than could be achieved by the use of a single sensoralone [1].

[1] D. Hall and J. Llinas, “An introduction to multisensor datafusion,” Proceedings of IEEE, vol. 85, no. 1, pp. 6–23, Jan.1997.

History

At the beginning of 1970s, theU.S. navy merged data aboutSoviet naval movements atdata fusion centers, the resultturned out to be muchaccurate than using data fromsingle sonar [1].

[1] N. Friedman, Seapower asStrategy: Navies andNational Interests. NavalInstitute Press, 2001.

Fusion with no correlation

In 1971, the multi-sensor fusion problem was first studied in [1],where the estimation errors in two track files from differentsensors, but corresponding to the same target, were assumed to beindependent. Later, more general cases can been found in [2, 3],including the federated square root filter [3].

[1] R. Singer and A. Kanyuck, “Computer control of multiple sitetrack correlation,” Automatica, vol. 7, no. 4, pp. 455 – 463,1971.

[2] A. Willsky, M. Bello, D. Castanon, B. Levy, and G. Verghese,“Combining and updating of local estimates and regional mapsalong sets of one-dimensional tracks,” IEEE Transaction onAutomatic Control, vol. 27, no. 4, pp. 799–813, Aug. 1982.

[3] N. Carlson, “Federated square root filter for decentralizedparallel processors,” IEEE Transactions on Aerospace andElectronic Systems, vol. 26, no. 3, pp. 517–525, May 1990.

Table: Fusion with no correlation

Types of Estimation Errors Fusion Rules Comments

IndependentPf = [

n∑

i=1

(Pi )−1]−1

x̂f = Pf [n∑

i=1

(Pi )−1x̂i ]

Optimal

Fusion with correlation

Further, in 1986, [1] came up with a fusion equation with theconsideration of cross-covariance between local estimates. In 1994,[2] generalized the results in [1] and an optimal fusion equationwas deduced in the sense of maximum likelihood estimate, but theposterior probability density function should be of the standardnormal density function.

[1] Y. Bar-Shalom and L. Campo, “The effect of the commonprocess noise on the two-sensor fused-track covariance,” IEEETransactions on Aerospace and Electronic Systems, vol.AES-22, no. 6, pp. 803–805, Nov. 1986.

[2] K. H. Kim, “Development of track to track fusion algorithms,”in Proceedings of American Control Conference, vol. 1, 1994,pp. 1037–1041.

Fusion with correlation

Ten year later, this limitation has been overcome by [1], where anoptimal information fusion criterion was re-derived in the linearminimum variance sense with the help of Lagrange multiplier, fromthe mathematical point of view, this fusion rule is equivalent to thebest linear unbiased estimation fusion rule in [2], or weighted leastsquares fusion rule in [3].

[1] S. Sun and Z. Deng, “Multi-sensor optimal information fusionKalman filter,” Automatica, vol. 40, no. 6, pp. 1017–1023,Jun. 2004.

[2] X. Li, Y. Zhu, J. Wang, and C. Han, “Optimal linear estimationfusion—part I: Unified fusion rules,” IEEE Transactions onInformation Theory, vol. 49, no. 9, pp. 2192–2208, Sep. 2003.

[3] Z. Deng, Y. Gao, L. Mao, Y. Li, and G. Hao, “New approachto information fusion steady-state Kalman filtering,”Automatica, vol. 41, no. 10, pp. 1695–1707, Oct. 2005.

Table: Multi-sensor Fusion Rules

Types of Estimation Errors Fusion Rules Comments

IndependentPf = [

n∑

i=1

(Pi )−1]−1

x̂f = Pf [n∑

i=1

(Pi )−1x̂i ]

Optimal

Correlated Pf = (eTΣ−1e)−1

x̂f = Pf (eTΣ−1x̂)

* Optimal

* e = [I , · · · , I ]T , Σ = (Pij), i , j = 1, · · · , n, and x̂ =[x̂T1 , · · · , x̂Tn ]T

Remark

It’s worth noting that the above results are obtained within theKalman filtering framework (or Bayesian framework), which is onlya small fraction of the whole family of multi-sensor fusion, theother frameworks may include but not limit to evidential beliefreasoning, fuzzy reasoning, possibilistic, hybridization, and randomset theoretic, the interested readers may consult survey paper [1]for more details.

[1] B. Khaleghi, A. Khamis, F. O. Karray, and S. N. Razavi,“Multisensor data fusion: A review of the state-of-the-art,”Information Fusion, vol. 14, no. 1, pp. 28–44, Jan. 2013.

Whole picture

Figure: Multisensor data fusion [1]

[1] B. Khaleghi, A. Khamis, F. O. Karray, and S. N. Razavi,“Multisensor data fusion: A review of the state-of-the-art,”Information Fusion, vol. 14, no. 1, pp. 28–44, Jan. 2013.

Revisit

I Independent Estimates [1]

I Correlated Estimates [2]

I Unknown Correlation

[1] Y. Bar-Shalom and L. Campo, “The effect of the commonprocess noise on the two-sensor fused-track covariance,” IEEETransactions on Aerospace and Electronic Systems, vol.AES-22, no. 6, pp. 803–805, Nov. 1986.

[2] S. Sun and Z. Deng, “Multi-sensor optimal information fusionKalman filter,” Automatica, vol. 40, no. 6, pp. 1017–1023,Jun. 2004.

Outline

Introduction

Multi-Sensor Fusion

Fusion with unknown correlation

Fast Covariance Intersection

Causes

I lack of knowledge of the true system

I too difficult to describe

Lack of knowledge of the true system

I correlation can not been identified, e.g., correlations from theobservation noises, as yet unidentified, may occur during themoving process of vehicle which equipped a suite ofnavigation sensors [1].

I the filter implementation is an approximation, it’s oftenassumed that the prior estimation error and the newmeasurement error are uncorrelated, which may introducecertain degree of unknown correlations in final implementationprocess [2].

[1] S. J. Julier and J. K. Uhlmann, “A non-divergent estimationalgorithm in the presence of unknown correlations,” inProceedings of American Control Conference, vol. 4, 1997, pp.2369–2373.

[2] J. K. Uhlmann, “Dynamic map building and localization: Newtheoretical foundations,” Ph.D. dissertation, University ofOxford, 1995.

Too difficult to describe

I involve too many variables, e.g., thousands of states may beused in the applications of map building, which meansmaintaining an accurate covariance is impractical [1].

I data incest or double counting problem [2], where informationis inadvertently used several times due to the presence ofnetwork loops in distributed fusion settings.

I difficult to calculate, e.g., how to acquire reliable correlationsin nonlinear estimates is still an open question.

[1] S. J. Julier and J. K. Uhlmann, “A non-divergent estimationalgorithm in the presence of unknown correlations,” inProceedings of American Control Conference, vol. 4, 1997, pp.2369–2373.

[2] A. Makarenko, A. Brooks, T. Kaupp, H. Durrant-Whyte, andF. Dellaert, “Decentralised data fusion: A graphical modelapproach,” in Proceedings of the 12th International Conferenceon Information Fusion, Jul. 2009, pp. 545–554.

Summarize

To summarize, it’s widely believed that the unknown correlationsare ubiquitously existing in a diverse range of multi-sensor fusionproblems. Neglect the effect of unknown correlations can result ingrave consequence of performance deterioration even divergence.As such, it has attracted and sustained attention from researchersfor decades. However, due to its intricate unknown nature, it’s noteasy to come up with a satisfied scheme to address the fusionproblems with unknown correlations.

Naive approach

If we ignore the correlations directly, which is the Naive fusion [1],it may lead to divergence of the filter.

P−1 = P−11 + P−12

P−1x̂ = P−11 x̂1 + P−12 x̂2

[1] K. Chang, C.-Y. Chong, and S. Mori, “Analytical andcomputational evaluation of scalable distributed fusionalgorithms,” IEEE Transactions on Aerospace and ElectronicSystems, vol. 46, no. 4, pp. 2022–2034, Oct. 2010.

A normal suboptimal approach

To compensate this kind of divergence, a normal suboptimalapproach is to increase the system noise artificially. However, thisheuristic requires substantial certain expertise and compromises theintegrity of the Kalman filter framework [1].

[1] W. Niehsen, “Information fusion based on fast covarianceintersection filtering,” in Proceedings of the 5th InternationalConference on Information Fusion, 2002, pp. 901–904.

Covariance Intersection (CI)

Among existing fusion solutions for unknown correlation, it was thecovariance intersection algorithm [1], which invented by S. J. Julierand J. K. Uhlmann in 1997, that attracted much interest.

P−1 = ωP1−1 + (1− ω)P−12

P−1x̂ = ωP−11 x̂1 + (1− ω)P−12 x̂2

[1] S. J. Julier and J. K. Uhlmann, “A non-divergent estimationalgorithm in the presence of unknown correlations,” inProceedings of American Control Conference, vol. 4, 1997, pp.2369–2373.

Table 1: Multi-sensor Fusion RulesTypes of Estimation Errors Fusion Rules Comments

No Correlations(Independent)

Pf = (

n∑

i=1

P−1i )−1

x̂f = Pf (n∑

i=1

P−1i x̂i)

Optimal

Known Correlations(Correlated)

Pf = (eT Σ−1e)−1

x̂f = Pf (eT Σ−1x̂)* Optimal

Unknown Correlations

Pf = (

n∑

i=1

ωiP−1i )−1

x̂f = Pf (

n∑

i=1

ωiP−1i x̂i)

** Suboptimal

* e = [I, · · · , I]T , Σ = (Pij), i, j = 1, · · · , n, and x̂ = [x̂T1 , · · · , x̂T

n ]T .** Covariance intersection rule, where ωi ∈ [0 , 1],

∑ni=1 ωi = 1, and ωi =

arg minωi∈[0 ,1] tr{Pf}.

Advantages

1. The identification and computation of the cross-covariancesare completely avoided.

2. It yields the consistent fused estimator and a non-divergentfilter is obtained.

3. The accuracy of the CI fuser is higher than that of each localestimator.

4. It gives a common upper bound of actual estimation errorvariances, which is independent of unknown cross-covariances,so it has robustness with respect to unknowncross-covariances.

Theoretic

1. split covariance intersection (generalized CI): incorporatingknown independent information

2. an n2 dimensional space

3. an information-theoretical justification

4. Chernoff fusion: includes covariance intersection as a specialcase

5. ellipsoidal intersection: leaded to a higher accuracy to thefused estimate

Application

Figure:simultaneouslocalization andmapping

Figure: vehiclelocalization

Figure: Targettracker

Figure: NASAMars rover

Drawbacks

P−1new = ω1P−11 + ω2P

−12 + · · ·+ ωnP

−1n ,

x̂new = Pnew (ω1P−11 x̂1 + ω2P

−12 x̂2 + · · ·+ ωnP

−1n x̂n),

where ωi ∈ [0 , 1],∑n

i=1 ωi = 1. Note that, different choices of ωi

can be used to optimize the fused value with regard to a variety ofcriteria, for instance, the weight chosen herein is to minimize thetrace of Pnew for the purpose of the non-divergence of the systemestimates, that is,

ωi = arg minωi∈[0 ,1]

tr{Pnew}.

I n = 2; gold section method (i.e. 0.618 section) or Fibonaccimethod

I n ≥ 3; exactly a nonlinear optimization problem withconstraints in Euclidean space Rn; CVX

Fast covariance intersectionI Sequential covariance intersection [1]I Suboptimal non-iterative algorithm [2]I Ellipsoidal intersection [3]I Close-form optimization of covariance intersection [4]

[1] Z. Deng, P. Zhang, W. Qi, J. Liu, and Y. Gao, “Sequentialcovariance intersection fusion Kalman filter,” InformationSciences, vol. 189, pp. 293–309, Apr. 2012.

[2] W. Niehsen, “Information fusion based on fast covarianceintersection filtering,” in Proceedings of the 5th InternationalConference on Information Fusion, 2002, pp. 901–904.

[3] J. Sijs and M. Lazar, “State fusion with unknown correlation:Ellipsoidal intersection,” Automatica, vol. 48, no. 8, pp.1874–1878, Aug. 2012.

[4] M. Reinhardt, B. Noack, and U. D. Hanebeck, “Closed-formoptimization of covariance intersection for low-dimensionalmatrices,” 2012, pp. 1891–1896.

Outline

Introduction

Multi-Sensor Fusion

Fusion with unknown correlation

Fast Covariance Intersection

Sequential covariance intersection

Suboptimal non-iterative algorithm

P−1new = ω1P−11 + ω2P

−12 + · · ·+ ωnP

−1n ,

x̂new = Pnew (ω1P−11 x̂1 + ω2P

−12 x̂2 + · · ·+ ωnP

−1n x̂n),

ωi =

1tr(Pi )∑

i∈N1

tr(Pi )

.

Ellipsoidal intersection 1

Ellipsoidal intersection 2

Close-form optimization of covariance intersection

Different from the above methods trying to approximate theoptimal value, [1] can give an exact solution of the optimal weightsin case of low-dimensional covariance matrices by the close-formoptimization of covariance intersection, which reduced thenonlinear optimization problem into the polynomial root-findingproblem.

[1] M. Reinhardt, B. Noack, and U. D. Hanebeck, “Closed-formoptimization of covariance intersection for low-dimensionalmatrices,” 2012, pp. 1891–1896.

How to Cite the Paper

[1] W. Li, Z. Wang, G. Wei, L. Ma, J. Hu, and D. Ding, “A surveyon multi-sensor fusion and consensus filtering for sensornetworks,” Discrete Dynamics in Nature and Society, vol.2015, Article ID 683701, 12 pages, 2015.

How to Cite the Paper

@article{Li2015ddns,

year = {2015},

month = {Article ID 683701, 12 pages,},

volume = {2015},

author = {Wangyan Li and Zidong Wang and Guoliang Wei and Lifeng Ma and Jun Hu and Derui Ding},

title = {A Survey on Multi-Sensor Fusion and Consensus

Filtering for Sensor Networks},

journal ={Discrete Dynamics in Nature and Society}

}

Thank You