lecture 9: minimum mean-square estimation · tu berlin| sekr. hft 6|einsteinufer 25|10587berlin...

TU Berlin | Sekr. HFT 6 | Einsteinufer 25 | 10587 Berlin

www.mk.tu-berlin.de

Faculty of Electrical Engineering and Computer Systems Department of Telecommunication Systems Information and Communication Theory Prof. Dr. Giuseppe Caire Einsteinufer 25 10587 Berlin Telefon +49 (0)30 314-29668 Telefax +49 (0)30 314-28320 [email protected] Sekretariat HFT6 Patrycja Chudzik Telefon +49 (0)30 314-28459 Telefax +49 (0)30 314-28320 [email protected]

Firma xy Herrn Mustermann Beispielstraße 11 12345 Musterstadt

Berlin, 1. Month 2014

Subject: Text…& Prof. Dr. Giuseppe Caire Lecture 9:

Minimum Mean-SquareEstimation

Copyright G. Caire (Sample Lectures) 274


www.mk.tu-berlin.de




Subject: Text…& Prof. Dr. Giuseppe Caire

Vector spaces, normed spaces and inner products

Definition 39. A set G with a commutative operation + : G ⇥ G ! G is anadditive group if

1. it is closed under + (i.e., a+ b 2 G for all a, b 2 G).

2. it has an additive identity (i.e., there exists an element 0 2 G such that 0+a =

a for all a 2 G).

3. Every element has an additive inverse (i.e., for all a 2 G there exist anelement �a such that a+ (�a) = 0).

⌃



www.mk.tu-berlin.de





Definition 40. A vector space V over R is an additive group such that

1. xv 2 V for all v 2 V and x 2 R.

2. 0v = 0 for all v 2 V .

3. 1v = v for all v 2 V .

⌃



www.mk.tu-berlin.de





Definition 41. A norm is a function k · k : V ! R+ that satisfies the followingproperties:

1. kvk = 0 if and only if v = 0.

2. kv + uk kvk+ kuk (triangle inequality).

3. kxvk = |x|kvk for all v 2 V and x 2 R.

⌃



www.mk.tu-berlin.de





Definition 42. A normed vector space is a vector space V with a norm k · k. ⌃

Notice: a norm is a “distance” function.

• For example, one can check that the norm defined as

kvk2 =

vuutnX

i=1

v2i

where V = Rn is the standard Euclidean n-dimensional vector space over R,defines a distance in the usual sense (length of the vector joining two pointsin Rn).

• Let v,u 2 Rn, then

kv � uk2 =

vuutnX

i=1

(vi

� ui

)

2

is the Euclidean distance between the points (vectors) v and u.



www.mk.tu-berlin.de





Definition 43. Given a vector space V over R, an inner product is a function(·, ·) : V ⇥ V ! R with the following properties:

1. (v,u) = (u,v) (symmetry).

2. (xv,u) = x(v,u), for all v,u 2 V and x 2 R (scaling).

3. (v1 + v2,u) = (v1,u) + (v2,u) (linearity).

4. (v,v) � 0, with equality if and only if v = 0.

A vector space with an inner product is called inner product space. ⌃



www.mk.tu-berlin.de





Theorem 41. Cauchy-Schwarz inequality:

(v,u)2 (v,v)(u,u)

with equality if and only if av = bu, with a, b 2 R not both zero.

Theorem 42. Let V be an inner product space. Then, the following is a norm(called 2-norm, or standard Euclidean norm):

kvk2 =p(v,v)



www.mk.tu-berlin.de





Least Squares approximation

• Consider the following problem. Let x denote a point (vector) in some vectorspace V over R and let y1, . . . ,ym

be a given collection of vectors:we wish to find the “best” approximation of x by a linear combination of thevectors {y

i

}.

• We have to give a rigorous meaning to the term “best”: if V is an inner productspace, we shall consider the minimum distance approximation, that is, welook for

bx =

mX

i=1

y

i

ai

such thatkx� b

xk22 = (x� bx,x� b

x)

is minimum.

• This approximation is called (linear) “Least-Squares” (some people call it“linear regression”).



www.mk.tu-berlin.de





• A brute-force approach: we can write

kx� bxk22 = kxk22 � 2(x, bx) + kbxk22

= kxk22 � 2

mX

i=1

(x,yi

)ai

+

mX

i=1

mX

j=1

ai

(y

i

,yj

)aj

= kxk22 � 2r

Txy

a+ a

TG

y

a

where we define the “cross-correlation vector”

r

xy

= ((x,y1), . . . , (x,ym

))

T

and the matrix of inner products (Gram matrix)

G

y

=

2

6664

(y1, y1) (y1, y2) · · · (y1, ym

)(y2, y1) (y2, y2)

......

(ym

, y1) (ym

, y2) · · · (ym

, y

m

)

3

7775



www.mk.tu-berlin.de





• Notice that Gy

is symmetric and positive semidefinite.

• Taking the gradient of the distance function with respect to a, we obtain theequation

G

y

a = r

xy

• Assuming for simplicity that G

y

is invertible (otherwise, we can eliminatesome linearly dependent y

i

and obtain the same subspace), we obtaina = G

�1y

r

xy

.

• OBSERVATION: notice that the solution bx satisfies the following orthogonality

condition:(x� b

x,yi

) = 0, 8 i = 1, . . . ,m



www.mk.tu-berlin.de





Linear Minimum Mean-Square Error estimation

• We have two jointly distributed random vectors X and Y.

• We observe Y and we with to “guess” the value of X in some optimal sense.

• Analogously to what done before, we define the following error function:Mean-Square-Error (MSE)

mse = E��X� b

X

��2

2

�

• We seek an estimator bX in the form of an affine function of the observation

Y, that is,bX = AY + b



www.mk.tu-berlin.de





• First, notice that for any mean vectors m

x

and m

y

and any estimator bX, we

can always reduce the problem to a zero-mean case by considering X

0=

X �m

x

, Y0= Y �m

y

. If bX

0 is the minimum MSE (MMSE) estimator for X0

given Y

0, thenbX = m

x

+

bX

0

is the optimal estimator for X given Y.

• We restrict to the zero-mean case, and seek bX in the form AY. The

orthogonality principle yields the condition

(X� bX,Y) = E

h(X� b

X)

TY

i= tr

⇣Eh(X� b

X)Y

Ti⌘

= 0

that, explicitly, writes

tr�E⇥XY

T⇤�AE

⇥YY

T⇤�

= 0



www.mk.tu-berlin.de





• We can solve for A as follows:

AE⇥YY

T⇤= E

⇥XY

T⇤

) A = E⇥XY

T⇤ �E⇥YY

T⇤��1

• In conclusions, in the general case (non-zero mean), we let

⌃

xy

= cov(X,Y) = E[(X�m

x

)(Y �m

y

)

T], ⌃

y

= cov(Y)

and we obtain the linear MMSE estimator (Wiener filter) of X from Y as

bX = m

x

+⌃

xy

⌃

�1y

(Y �m

y

)



www.mk.tu-berlin.de





• The error covariance matrix is given by

cov(X� bX) = E

h(X� b

X)(X� bX)

Ti

= ⌃

x

�⌃

xy

⌃

�1y

⌃

Txy

• The total MMSE, is given by E[kX� bXk22] = tr(cov(X� b

X)).

Notice: The estimation error vector X� bX is uncorrelated with the observation

vector Y.



www.mk.tu-berlin.de





Minimum Mean-Square Error estimation: the general case

• With the same setting as before, we now seek an estimator bX = g(Y), in the

space of all (measurable) functions of the observation Y.

Theorem 43. The MMSE estimator of X given Y is the conditional mean

bX = E[X|Y]

Proof.We use the orthogonality principle: the optimal estimator b

X must satisfy

Eh(X� b

X)

Tg(Y)

i= 0, for all functions g



www.mk.tu-berlin.de





• Hence, the (general) MMSE estimator of X given Y coincides with the linearMMSE estimator (Wiener filter) in the Gaussian case:

bX = E[X|Y] = m

x

+⌃

xy

⌃

�1y

(Y �m

y

)

• MMSE decomposition:

X =

bX+ (X� b

X) =

bX+V

where the MMSE estimator bX and the estimation error vector V are

independent,

bX ⇠ N (m

x

,⌃xy

⌃

�1y

⌃

yx

), V ⇠ N (0,⌃x|y)



www.mk.tu-berlin.de




Subject: Text…& Prof. Dr. Giuseppe Caire End of Lecture 9


lecture 9: minimum mean-square estimation · tu berlin| sekr. hft 6|einsteinufer 25|10587berlin...

Documents