IEEE UKRI Talk, De Montfort Univ., April 2, 04
1
Open Issues in Constrained Blind Source Separation
Jonathon ChambersCardiff Professorial Research Fellow
Cardiff School of EngineeringCardiff University, Wales, U.K.
E-mail: [email protected]
IEEE UKRI Talk, De Montfort Univ., April 2, 04
2
Summary of Talk
• Acknowledgement
• Historical background & motivation
• BSS with matrix constraints
• Penalty functions in FD-BSS
• Exploiting periodicity in BSS
• Future application-driven challenges
IEEE UKRI Talk, De Montfort Univ., April 2, 04
3
Acknowledgements Jonathon Chambers wishes to express his sincere thanks
for the support of Professor Andrzej Cichocki, Riken Brain Science Institute, Japan
The invitation from the organising committee of the workshop to give this talk.
His co-researchers: Drs Saeid Sanei, Maria Jafari and Wenwu Wang.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
4
LMS Algorithm
B. Widrow, and M.E. Hoff, Jr.,
“Adaptive switching circuits,” IRE Wescon Conv. Rec., pt. 4, pp. 96-104, 1960.
LMS Update
t)(xe(t) t)(w 1)t(w
IEEE UKRI Talk, De Montfort Univ., April 2, 04
5
Historical Background
• The field of conventional adaptive signal processing has been greatly enhanced by the exploitation of constrained optimisation
• Constraints on the error, and/or structure or some norm of the weights via, for example, Lagrange multipliers and/or Karush-Khun-Tucker conditions
IEEE UKRI Talk, De Montfort Univ., April 2, 04
6
Historical BackgroundCertain key papers:
• O.L. Frost, III, “An algorithm for linearly constrained adaptive array processing,” Proc. IEEE, Vol. 60(8), pp. 926-925, 1972
• R.P. Gitlin et al. “The tap-leakage algorithm: an algorithm for the stable operation of a digitally implemented fractionally spaced equalizer,” Bell Sys. Tech. Journal, Vol. 61(8), pp. 1817-1839, 1982.
• D.T.M. Slock, “Convergence behavior of the LMS and Normalised LMS Algorithms,” IEEE Trans. Signal Processing, Vol. 41(9), pp. 2811-2825, 1993.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
7
Historical Background Cont.• S.C. Douglas, “A family of normalized LMS algorithms,”
IEEE Signal Processing Letters, Vol. 1(3), pp. 49-51, 1994.
• S.C. Douglas, and M. Rupp, “A posteriori updates for adaptive filters,” Asilomar Conference on Signals, Systems and Computers, Vol. 2, pp 1641-1645, 1997.
• T. Gänsler, et al., “A robust proportionate affine projection algorithm for network echo cancellation,” Proc. ICASSP 2000, Vol. 2, pp. 793-796, 2000.
• O. Vainia, “Polynomial constrained LMS adaptive algorithm for measurement signal processing,” Proc. IECON 2002, Vol. 2, pp. 1479-1482, 2002.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
8
Motivation
“In many applications of Independent Component Analysis (ICA) and Blind Source Separation (BSS) estimated source signals and the mixing or separating matrices have some special structure or some constraints are imposed for the matrices…”, Cichocki and Georgiev, 2003
IEEE UKRI Talk, De Montfort Univ., April 2, 04
9
H
s1
sN
W
x1
xM
Y1
YN
Unknown
Known
Independent?
Adapt
Mixing Process
Unmixing Process
Fundamental Model for Instantaneous Blind Source
Separation
IEEE UKRI Talk, De Montfort Univ., April 2, 04
10
Certain BSS Books
• Andrzej Cichocki and Shun-Ichi Amari, Adaptive Blind Signal and Image Processing, Wiley, 2002
• Simon Haykin Unsupervised Adaptive Filtering, Vols. I and II, Wiley, 2000
• Aapo Hyvärinen, Juha Karhunen and Erkki Oja, Independent Component Analysis, Wiley, 2001
• Te-Won Lee, Independent component analysis: theory and applications, Kluwer, 1998
IEEE UKRI Talk, De Montfort Univ., April 2, 04
11
BSS References
• A. Mansour and M. Kawamoto, “ICA Papers Classified According to their Applications and Performances”, IEICE Trans. Fundamentals, Vol. E86-A, No. 3, March 2003, pp. 620-633.
• In 2002, 800 different papers have been published, these are downloadable at http://ali.mansour.free/REF.htm
IEEE UKRI Talk, De Montfort Univ., April 2, 04
12
With a symmetric mixing matrix [C&G,2003]:-
Txxx
Txx
Tx
-xx
-xx
-1opt
VV t)}(xt)(xE{ R
where
VV R H W 21
21
BSS With Matrix Constraints
IEEE UKRI Talk, De Montfort Univ., April 2, 04
13
Stable Frobenius norm of the separating matrix
Theorem [C&G 2003]: The learning rule
W(t)t)I(t)(yF(t) dt
dW(t)N
where β > 0 is a scaling factor and
γ(t) = trace(WT(t)F(y(t))W(t)) > 0,
stabilizes the Frobenius norm of W(t) such that
1T2
Ft)W(t)(W traceW(t)
BSS With Matrix Consts. Cont.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
14
Consequence: The modified NG descent learning
algorithm, with a forgetting factor, described as
t)W(t)(t)W(t)(WW
J(W)(t)-
dt
dW(t) T
with γ(t) = -trace(WT(t)[J(W)/ W]WT(t)W(t)) > 0
has a W(t) with bounded Frobenius norm throughout
the learning process.
BSS With Matrix Consts. Cont.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
15
Prof. Amari’s “Leaky” NG Algorithm becomes
W(t)t)(yt))(yf(I)(
W(t)(t)(t)-1 1)W(tT
N
t
where 0 << (1-βγ(t)η(t)) < 1 is the leakage factor
BSS With Matrix Consts. Cont.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
16
Introducing a semi-orthogonality constraint so that it is possible to extract an arbitrary group of sources, say e, 1 e N.
Assuming pre-whitened data
Qx x and I }xxE{R N
T
XX
and the mixing matrix A = QH, the demixing
matrix We should satisfy WeA = [Ie,0N-e]
BSS With Matrix Consts. Cont.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
17
A natural gradient algorithm to find We becomes:-
t)(Wy(t)][y(t)ft)(xf[y(t)](t)
- t)( W )1t(W
eTT
ee
With initial conditions which satisfy
eTee I )0(W)0(W
BSS With Matrix Consts. Cont.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
18
)(1 ts
)(2 ts
)(2 tx
)(1 txM icrophone1
M ic rophone2
S peaker1
S peaker2
Real Convolutive Mixing Env. – Cocktail Party Problem
IEEE UKRI Talk, De Montfort Univ., April 2, 04
19
)(
)(
)(
)(
tx
tx
ts
ts
tHtH
tHtH
MNMNM
N
11
1
111
)()(
)()(
ConvolutionsHx *
N
j
P
pjij ptsphtx
1
1
0
)()()(
Compact form:
Expansion form:
Convolutive BSS – Model
IEEE UKRI Talk, De Montfort Univ., April 2, 04
20
Taxonomy of Existing Sols. To Convolutive BSS
• Performing blind separation in the time domain by extending the existing instantaneous methods to conv. case
• Transforming the convolutive BSS problem into multiple instantaneous (complex) problems in the frequency domain
• Decomposing the system into smaller problems using, for example, a subband approach
• Hybrid frequency and time domain approaches
IEEE UKRI Talk, De Montfort Univ., April 2, 04
21
)(
)(
)(
)(
NMNM
N
M s
s
HH
HH
x
x
1
1
1111
)()(
)()(
sHx *DFT
Convolutive BSS problem
Multiple complex-valued instantaneous BSS problems
Transform Convolutive BSS into the Frequency Domain
IEEE UKRI Talk, De Montfort Univ., April 2, 04
22
In the frequency domain:-
Mathematical Formulation
NTN1
NTN1
C t),(Xt),...,,(X t),X(
and
C t),(St),...,,(S t),S(
where
t),V( t),)S(H( t),X(
IEEE UKRI Talk, De Montfort Univ., April 2, 04
23
De-mixing Operation
independ.
mutually become t),(Yt),...,,(Y
such that determined )in W( Parameters
N1
TN1
NN
t)],(Yt),...,,([Y t),Y( and
C) W(where
t),X() W( t),Y(
IEEE UKRI Talk, De Montfort Univ., April 2, 04
24
Constrained Optimisation and Joint Diagonalisation
0 )wg( s.t. )wf(min :becomes Prob1 and,
]w,...,w,...,w,...,w,w,...,[w
vec(W) w form,in vector or,
1 r and RC:f
RC:W)](gW),...,(gW),([g W)(g
where
Prob1 - 0 W)(g s.t. f(W)min
NMNM1MN212N111
1MN
rMNTr21
C
IEEE UKRI Talk, De Montfort Univ., April 2, 04
25
Joint Diagonalisation Criterion
2
FYY
T
1
K
1tW
HVXY
t)],([R-t),(R (W) where
t),(W)(min arg ))(W(
-:FunctionCost
)(Wt),(R-t),(R) W( t),(R
diagF
FJ
Exploiting the non-stationarity of speech signals measured by the cross-spectrum of the output signals,
IEEE UKRI Talk, De Montfort Univ., April 2, 04
26
Exterior Penalty Function Approach
.set for the functionspenalty
exterior of sequence a form willW)(Then
.or 2, 1, b R,C:W)(W)( if And
1. let and ,q as ,
and 0such that be let , qFor
0}.W)(|C{W : and
continuous is RC:g(W) If :
q
MN
bqq
q
q1qq
MN
rMN
W
U
gU
N
gW
Lemma
IEEE UKRI Talk, De Montfort Univ., April 2, 04
27
Exterior Penalty Function Approach
Typical exterior penalty functions, and the shadow area represents the feasible set.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
28
Proposed General Cost Function
.)W(min argbecomes problem separation theThus,
functions. weightingnonneg theare ],...,[ and
))]W(()),...,W(([ )) U(W(where
))U(W()W( )W(
W
Tr1
Tr1
T
new
new
J
UU
JJ
With a factor vector κ to incorporate exterior penalty functions, our cost function becomes:-
IEEE UKRI Talk, De Montfort Univ., April 2, 04
29
Numerical Experiments• Use an exterior penalty function
2
F
H I-)()WW(
• Employ a variant of gradient adaptation
• Utilize the filter length constraint to address the permutation problem (Parra & Spence)
• System with two inputs and two outputs (TITO!)
• H(z) = [{1 1.9 -0.75}, z-5{0.5 0.3 0.2}; z-5{-0.7 -0.3 -0.2}, {0.8 -0.1}]; D = 7, T = 1024, K = 5.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
30
Convergence Performance of the New Criterion as a function of κ
IEEE UKRI Talk, De Montfort Univ., April 2, 04
31
Room Environment Experiment• Use roommix function due to Westner
• Room 10x10x10m3 cube
• Wall reflections calculated up to fifth order, atten. factor 0.5
• SIR is plotted as a function of length of the separating system
ji
2
j
2
ij
i
2
i
2
ii
)(s)(H
)(s)(H10log s]SIR[H,
IEEE UKRI Talk, De Montfort Univ., April 2, 04
32
Room Environment
IEEE UKRI Talk, De Montfort Univ., April 2, 04
33
Room Environment SIR
IEEE UKRI Talk, De Montfort Univ., April 2, 04
34
S1
S2
x1
x2
FDICA
1
2
P
)(ˆ1 S
)(ˆ2 S
S1×0.5
S2×1
S2 ×0.3
S1 ×1.2
S2×0.6
S1×0.4
Permutation Problem in FD-CBSS
IEEE UKRI Talk, De Montfort Univ., April 2, 04
35
Summary of Existing Solutions to Permut. Problem in FD-CBSS
• Constraints on the filter models in the frequency domain
• Using special structure contained in signals
• Merging beamforming view to align solutions
• Exploiting the continuity of the spectra of the recovered signals – could coupled hidden Markov Models be used?
• What happens when the sources move, enter/re-enter the environment? What is the way forward?
IEEE UKRI Talk, De Montfort Univ., April 2, 04
36
Exploiting Source (Pseudo) -Periodicity
• W. Wang, M.G. Jafari, S. Sanei, and J.A. Chambers, “Blind source separation of convolutive mixtures of cyclostationarity”, to appear in the Special Issue on BSS, International Journal of Adaptive Control and Signal Processing, Guest Editor: Mike Davies, Queen Mary’s College, University of London
• H. Swada, R. Mukai, S. Araki, and S. Makino, “A robust and precise method for solving the permutation problem of frequency-domain blind source separation”, ICA 2003, Nara, Japan, 2003, pp. 505-510.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
37
A natural gradient update exploiting cyclostationarity
kkkkkkk yT WRyyfIWW ~21
• The Cyclostationary NGA uses the update equation
where
and p is the cycle frequency of the p-th source
m
p
kjHm
p
pp ekkEkk11
~ yyRR yy
IEEE UKRI Talk, De Montfort Univ., April 2, 04
38
A natural gradient update exploiting periodicity
• The Periodic NGA type update equation
where
kk,TTk
k,TTkkkk
iiy
iiHy
WySR
ySRIWW
,
,21
1
iyi
iH
iy
Tkdiagsignk,T
TkkTk
, and
,
RyS
yyR
IEEE UKRI Talk, De Montfort Univ., April 2, 04
39
Emerging Applications Biomedical:-
ECG, EEG, MEG and their integration
Microarray time courses
Measurements from the nano-lab
http://www.nmrc.ie/research/transducers-group/trends.htmlhttp://www.nanospace.systems.org/ns_2000/NS00_Sessions.htmhttp://nanomed.ncl.ac.uk/m2l.htm
Star Trek: The Tri-corder
IEEE UKRI Talk, De Montfort Univ., April 2, 04
40
Emerging Applications • T. Bowles, J. Chambers, and A. Jakobsson, “Advanced spectral estimation for the identification of cell-cycle regulated genes”, IEEE EMBS UK and RI Postgraduate Conf in Biomedical Engineering and Medical Physics, 2003.
• X. Liao, and L. Carin, “Constrained independent component analysis of DNA microarray signals”, IEEE Workshop on Genomic Signal Processing and Statistics, 2002.
• S-I, Lee, and S. Batzoglou, “Discovering biological processes from microarray data using independent component analysis”, Dept EE/CS, Stanford Univ.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
41
Summary • The exploitation of constrained optimisation has been fundamental to the development and application of adaptive signal processing; this process is, however, very much in its infancy in blind source separation (BSS).
• Utilisation of certain a priori knowledge on the mixing matrices and the properties of the sources is likely to yield solutions to real-life SP problems.
• As such, the challenge for DSP engineers in the 21st Century, is to advance the application of BSS methods in line with methods from adaptive signal processing.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
42
Other References • A. Cichocki, and P. Georgiev, “Blind source separation algorithms with matrix constraints”, IEICE Trans. Fundamentals, Vol. E86-A(3), March 2003, pp. 522-531.
• J.G. McWhirter, “Mathematics and signal processing”, Mathematics Today, April 2003, pp 47-54.
• W. Wang, S. Sanei, and J. Chambers, “Penalty function based joint diagonalization approach for convolutive blind source separation”, submitted to IEEE T-SP, Sept 2003.
IEEE UKRI Talk, De Montfort Univ., April 2, 04
43
Close ???
Mark Twain
“A man who swings a cat by its tail learns things he can learn no other way”