0912712016

Last time : Tuesday

IN ;D §I( ×,

HAIf XHZ f

If X - Y . 2- 3

How to remember :

aHHH- nx.

o.cotun H\gut, .#

- Hk , 'll - HH ,H - HIYHTHCXKHHHHH

=Ilx;'ll - Ilxnlz )

=I(Y;Z ) - Ilyizk )= Ilx ;⇒ - Ilxizly )

It XIY ) is concave in Px →

Hogto show this

using Ilxiy )

ZIH, YIZ) when X - Y - Z ?

ILX ; Y) "

convex in Pyl :w~ Berth W - * Y

IN ;y ) ? Il X ; YIW)

Pxiw.FR fix Pyk

Pxlw -0=4 mipxkhgmpu.,

' '

agmexnaoggbinotion

Fano 's Inequality

Suppose you can estimate X from Y with high probability (HCXIY) is small

.

)

Suppose X - Y - I and P[ XFXT = Pe

HWH ){ Hlpeltpelgtxlwhen XFIM#

define E - txtx " ¢ when XX

HLX,

EII ) - HIEIIHHHIE,

I )

=Hlx1IltHtElx⇐HWII ) - HLEIIITHIXIE ,

I ) dear being slightly loose could write 1g¢XH)

e HIEIT PEENlP[E=D log1×1

Tie , 6

:s

uniform bound pfano's bound

^

hgtxlt←

decreasing

⇐ 1×1

. )p

e

Asymptotic Equipartition Property ( A. EP )

LLN. A X

, , ... ,Xn ii .d

nt±fXi±E[x]A .tt ntlg

.gr#x.....xItHWin other words? Pa..

.×|X , ,. .,xn ) - In

" " '

"

almost all events are almost equally surprising?

Types of Convergences :

Convergence in probability : 2n±sZ : Fe >o IPFZN-21 >e]→0 a . n→x

Almost sure convergence

: Znt Z : lP[ ftp.ztZ ]=OLz convergence

: Zn ± Z : Ef zn . zp]→opro#A AEP :

#g¥.gg#=nttgIHPxTxD=ntFzlgtIx.

,

p÷un Elgtxtx,- Hw

Markov Inequality If XZO up 1

proof :

P[X3c] , EEXT

E[4xxDfe[×T] >{ x }c} x

Chebgcher 's Inequality. P[ IX - EEDI >c]{ ¥pn¥ : Let Y=l*EEx]P Y }o

P[Yzd]{ EY =k÷=9P[ HEGAN ]

=lP[ HEIDI 3rd ]

proof of LLN : Assume : X has finite variance .

Use Chebychev 's Inequality .

It HE xi . EE ]1se]$ tinned → o

Typical Sets : Let Xn=(X, , ..

.

, Xn ) ( write Xsi = ( x ; ,. . .

,X ;))

A 'T - { xn :lHgp,÷µ- HWIEE }→

( implicitly . function of Px and Pxn=IP× )

Thm : 1) the Ad"

: Pxnlxnle [ In't "* 'd

,

z.nl#h ]

21 Pxnlain 't Plxneae"

] → 1 TEE

3) IAYY , znltkxhel

HIAHYZIHI " " " 4for n large enough .

pr¥ :

1.1 By dfn of AH'

2,1,

ByAEP theorem

' =

,I×Pxn*known Pxnk

" )

3xI*.

I" ( " " +4

= IAYY Ink " ' " '

4) For nlarge enough :p×FµqPxnlxY= Pxnlaet

"13 te

'

÷zµp" " ' ' '=ldnlznlt " "

Lossless Data Compression :

Xi , .. .

, Xn ~ iii.

d.

P

; Metta .is"Y=[2nY

Block encoding ×Lf#-→|DeI#tk bits per symbol

Goal : P[xtI]ce

Achievability : R is achievable if the >o,

exists n,

encoder,

deader such that P[ Xntlixe

Theorem : Minimum achievable rate for lossless compression is HK)

4

rxn tent lane IY "

µhogget"Iii¥x .2

Aslong as R > HWTEZ is achievable

→ You can turn this into O . error scheme by putting an indicator bit : If typical indicator bit is 1

and we do AEP encoding .If indicator is O then we encode the whole sequence xn with nlgl XI

and Avenge kyth : MAID ( nl HKH )) + ( t - Pla's"D(nlykl)What about a converse result ?

Given n,

enc,

kc.

st.

MXNFIDK for E arbitrarily small.

xtm . inDato Pnc

. ing .

nR3 Htm ) } Ilxn;m)3hIkn;xY=

HIM - Htxnkn) - NHHI - Htxnlx ") > nHW - HH - enlgtxl

⇒ Rs,

HW - elylxl - ¥ as e→o R > HE