gaussian channel - information theory 2013, lecture 6gaussian channel information theory 2013,...

Gaussian channelInformation theory 2013, lecture 6

Jens Sjölund

8 May 2013

Jens Sjölund (IMT, LiU) Gaussian channel 1 / 26

Outline

1 Definitions

2 The coding theorem for Gaussian channel

3 Bandlimited channels

4 Parallel Gaussian channels

5 Channels with colored Gaussian noise

6 Gaussian channels with feedback


Introduction

DefinitionA Gaussian channel is a time-discrete channel with output Yi , input Xiand noise Zi at time i such that

Yi = Xi +Zi , Zi ∼N (0,N )

where Zi is i.i.d and independent of Xi .

The Gaussian channel is the most important continuous alphabetchannel, modeling a wide range of communication channels.The validity of this follows from the central limit theorem


The power constraint

If the noise variance is zero or the input is unconstrained, thecapacity of the channel is infinite.The most common limitation on the input is a power constraint

EX2i ≤ P


Information capacityTheoremThe information capacity of a Gaussian channel with power constraintP and noise variance N is

C = maxf (x):EX2≤P

I(X;Y ) =12log

(1+

PN

)Proof.Use that EY 2 = P +N =⇒ h(Y ) ≤ 1

2 log2πe(P +N ) (Theorem 8.6.5)

I(X;Y ) = h(Y )− h(Y |X) = h(Y )− h(X +Z |X) = h(Y )− h(Z |X)

= h(Y )− h(Z) ≤ 12log2πe(P +N )− 1

2log2πeN

=12log

(P +NN

)=12log

(1+

PN

)which holds with equality iff X ∼N (0, P )


Outline

1 Definitions







Statement of the theorem

Theorem (9.1.1)The capacity of a Gaussian channel with power constraint P and noisevariance N is

C =12log

(1+

PN

)Proof.See the book. Based on random codes and joint typicality decoding(similar to the proof for the discrete case).


Handwaving motivation

Volume of n-dimensional sphere ∝ rn

Radius of received vectors√∑n

i=1Y2i ≤

√n(P +N )

Using decoding spheres of radius√nN the maximum number of

nonintersecting decoding spheres is therefore no more than

M =

(√n(P +N

)n(√nN

)n =(1+

PN

)n/2= 2

n2 log(1+ P

N )

(note the typo in eq. 9.22 in the book). This corresponds to a rate

R =logMn

=log

(2n2 log(1+ P

N ))

n=12log

(1+

PN

)= C


Outline

1 Definitions







Bandlimited channels

A bandlimited channel with white noise can be described as

Y (t) = (X(t) +Z(t)) ∗ h(t)

where X(t) is the signal waveform, Z(t) is the waveform of the whiteGaussian noise and h(t) is the impulse response of an ideal bandpassfilter, which cuts all frequencies greater than W .

Theorem (Nyquist-Shannon sampling theorem)Suppose that a function f (t) is bandlimited to W. Then, the function iscompletely determined by samples of the function spaced 1

2W apart.


Bandlimited channels

Sampling at 2W Hertz, the energy per sample is P /2W . Assuming anoise spectral density N0/2 the capacity is

C =12log

(1+

P /2WN0/2

)=12log

(1+

PN0W

)bits per sample

=W log(1+

PN0W

)bits per second

which is arguably one of the most famous formulas of informationtheory.


Outline

1 Definitions







Parallel Gaussian channels

Consider k independent Gaussian channels in parallel with a commonpower constraint

Yj = Xj +Zj , j = 1,2, . . . , k

Zj ∼N (0,Nj )

Ek∑j=1

X2j ≤ P

The objective is to distribute the total power among channels so as tomaximize the capacity.


Information capacity of parallel Gaussian channel

Following the same lines of reasoning as in the single channel caseone may show

I(X1, . . . ,Xk ;Y1, . . . ,Yk) ≤∑j

(h(Yj )− h(Zj )

)≤

∑j

12log

(1+

PjNj

)

where Pj = EX2j and

∑Pj = P . Equality is achieved by

(X1, . . . ,Xk) ∼N

0,P1 0 . . . 00 P2 . . . 0...

.... . .

...0 0 . . . Pk


Maximizing the information capacity

The objective is thus to maximize the capacity subject to the constraint∑Pj = P , which can be done a Lagrange multiplier λ

J(P1, . . . , Pk) =∑j

12log

(1+

PjNj

)+λ

∑j

Pj − P

∂J∂Pi

=1

2(Pi +Ni)+λ = 0

Pi = −12λ−Ni

∆= ν −Ni

however Pi ≥ 0, so the solution needs to modified to

Pi = (ν −Ni)+ =

ν −Ni if ν ≥Ni0 if ν < Ni


Water-filling

The gives the solution

C =k∑j=1

12log

(1+

(ν −Nj )+

Nj

)

where ν is chosen so that∑kj=1(ν −Nj )+ = P (typo in the summary).

As the signal power increases, power is alloted to the channels wherethe sum of noise and power is the lowest. By analogy to the way waterdistributes itself in a vessel, this is referred to as water-filling.


Outline

1 Definitions







Channels with colored Gaussian noise

Consider parallel channels with dependent noise.

Yi = Xi +Zi , Zn ∼N (0,KZ )

This can also represent the case of a single channel with Gaussiannoise with memory. For a channel with memory, a block of nconsecutive uses can be considered as n channels in parallel withdependent noise.


Maximizing the information capacity (1/3)

Let KX be the input covariance matrix. The power constraint can thenbe written

1n

∑i

EX2i =

1n

tr(KX) ≤ P

As for the independent channels,

I(X1, . . . ,Xk ;Y1, . . . ,Yk) = h(Y1, . . . ,Yk)− h(Z1, . . . ,Zk)

The entropy of the output is maximal when Y is normally distributed,and since the input and noise are independent

KY = KX +KZ =⇒ h(Y1, . . . ,Yk) =12log((2πe)n|KX +KZ |)


Maximizing the information capacity (2/3)

Thus, maximizing the capacity amounts to maximizing |KX +KZ |subject to the constraint tr(KX) ≤ nP . The eigenvalue decomposition ofKZ gives

KZ =QΛQt , where QQt = I

then,

|KX +KZ | = |Q||QtKXQ+Λ||Qt | = |QtKXQ+Λ| ∆= |A+Λ|

andtr(A) = tr(QtKXQ) = tr(QtQKX) = tr(KX) ≤ nP


Maximizing the information capacity (3/3)From Hadamard’s inequality (Theorem 17.9.2)

|A+Λ| ≤∏i

(Aii +λi)

with equality iff A is diagonal. Since∑i

Aii ≤ nP , Aii ≥ 0

it can be shown using the KKT conditions that the optimal solution isgiven by

Aii = (ν −λi)+∑i

(ν −λi)+ = nP


Resulting information capacity

For a channel with colored Gaussian noise

C =1n

n∑i=1

12log

(1+

(ν −λi)+

λi

)where λi are the eigenvalues of KZ and ν is chosen so that∑ni=1(ν −λi)+ = nP (typo in the summary).


Outline

1 Definitions







Gaussian channels with feedback

Yi = Xi +Zi , Zi ∼N (0,K (n)Z )

Because of the feedback, Xn and Zn are not independent; Xi iscasually dependent on the past values of Z.


Capacity with and without feedback

For memoryless Gaussian channels, feedback does not increasecapacity. However, for channels with memory, it does.

Capacity without feedback

Cn = maxtr(KX )≤nP

12n

log(|KX +KZ ||KZ |

)=

12n

∑i

log

1+ (λ−λ(n)i )+

λ(n)i

Capacity with feedback

Cn,FB = maxtr(KX )≤nP

12n

log(|KX+Z ||KZ |

)


Bounds on the capacity with feedback

Theorem

Cn,FB ≤ Cn +12

Theorem (Pinsker’s statement)

Cn,FB ≤ 2Cn

In words, Gaussian channel capacity is not increased by more thanhalf a bit or by more than a factor 2 when we have feedback.


gaussian channel - information theory 2013, lecture 6gaussian channel information theory 2013,...

Documents