basic lecture information theory and coding

8/13/2019 basic lecture information theory and coding

1/68

Dr. M. Arif Wahla

EE Dept

[email protected]

Military College of Signals

National University of Sciences & Technology (NUST), PakistanClass webpage:

http://learn.mcs.edu.pk/course/view.php?id=544

Information & Coding Theory

Course Outline/ Introduction


2/68

Founded by Claude E. Shannon (1916-2001)

The Mathematical Theory of Communication, 1948

Study fundamental limits in communications: transmission, storage,

etc

1:31 AM Course Outline 2

Information Theory


3/68



4/68

Information is uncertainty: modeled as random

variablesInformation is digital: transmission should be 0s

and 1s (bits) with no reference to what they

represent


Two Key Concepts


5/68

Source coding theorem

fundamental limit in data compression (zip,MP3, JPEG, MPEG)

Channel coding theorem

fundamental limit for reliable communicationthrough a noisy channel (telephone, cell phone,

modem, data storage, etc)


Two Fundamental Theorems


6/68

Information & Coding Theory

The fundamental problem of communication is that ofreproducing at one point either exactly or

approximately a message selected at another point.

(Claude Shannon, 1948)



7/68

Course Outline



8/68

This course will provide the students an introduction to

classical information theory and coding theory. The maincourse objective is to introduce the students to well-known information theoretic tools that can be used tosolve engineering problems.

The course will begin by describing basic communicationsystems problems where information theory may be applied.

An explanation of information measurement andcharacterization will be given. Fundamentals of noiseless

source coding and noisy channel coding will be taught next.Finally, some key information theory principles applied tocommunication security systems will be covered.


Course Objective (3+0)


9/68

Information theory is concerned with the fundamental limits of

communication.

What is the ultimate limit to data compression? e.g. how many bits

are required to represent a music source.

What is the ultimate limit of reliable communication over a noisy

channel, e.g. how many bits can be sent in one second over a

telephone line.


Course Outline -I


10/68

Coding theory is concerned with practical techniques to realize the

limits specified by information theory

Source coding converts source output to bits.

Source output can be voice, video, text, sensor output

Channel coding adds extra bits to data transmitted over the channel

This redundancy helps combat the errors introduced in transmitted bits

due to channel noise


Course Outline -II


11/68

Introduction

Communications Model

Information Sources

Source Coding

Channel Coding

Information Measurement

Definition and Properties of Entropy Uniqueness of the Entropy Measure

Joint and Conditional Entropy

Mutual Information and Conditional Mutual Information

Information Divergence Measures


Main Topics to be Covered


12/68


13/68


14/68


15/68


16/68

Applied Coding and Theory for Engineers

Richard B. Wells, Prentice Hall, 1999.

A Mathematical Theory of Communication,

Claude E. Shannon,

Bell System Technical Journal, 1948

available for free on line


Recommended Text Books & Study Material


17/68

ScheduleClass Meetings

Wednesday (5pm-8pm) 3L

Consultancy Hours

Wednesday (4pm-5pm), (8pm-8:30pm)

Other times by appointment (phone or email)

1:31 AM 17


18/68


19/68


20/68

Introduction to Information Theory

1:31 AM 21Course Outline


21/68

IT is about asking what is the most efficient path

from one point to another, in terms of some way ofmeasuring things.

1:31 AM 22

What is Information Theory (IT)?


h i f i h ( )?


22/68

Politics

Ask not what your country can do for you, but what you can do for

your country - John F. Kennedy

What makes the this political statements powerful (or at least

famous)?

force is efficiency of expression, there is an interpolationof many feelings,

attitudes and perceptions; there is an efficient encoding of emotional and

mental information.

1:31 AM 23

What is Information Theory (IT)?


f i h


23/68

Two important questions in engineering:

- What to do if information gets corrupted by errors?

- How much memory does it require to store data?

Both questions were asked and to a large degreeanswered by Shannon in his 1948 article:

use error correction and data compression.

1:31 AM 25

Information Theory

Claude Elwood Shannon (19162001), American electrical

engineer and mathematician, has been called the father of

information theory, and was the founder of practical digital

circuit design theory.


bl i C i i


24/68

Speed

Minimise length of transmitted data

Accuracy

Minimise and eliminate noise

Security

Ensure data is not changed or intercepted whilst in transit

1:31 AM 26

Problems in Communications


S l i


25/68

Speed

Minimise length of transmitted data

Use Data Compression

AccuracyMinimise and eliminate noise

Use Error Detection / Correction Codes

Security Ensure data is not changed or intercepted whilst in transit

Use Data Encryption / Authentication

1:31 AM 27

Solutions


C i i M d l


26/68

1:31 AM 28

Communications Model

Source Destination

signal

noise

Transmitter Receiver

received

signal

data data

Evesdropper



27/68

E D t ti /C ti C d


28/68

Error detectionis the ability to detect errors that are

made due to noise or other impairments in the course ofthe transmission from the transmitter to the receiver.

Error correctionhas the additional feature that enables

locating the errors and correcting them.

Examples: Compact Disc, DVD, GSM

Algorithms: Check Digit, Parity Bit, CRC, HammingCode, Reed-Solomon Code, Convolutional Codes, TurboCodes and LDPC Codes

1:31 AM 30

Error Detecting/Correcting Codes



29/68

Wh t i i f ti ?


30/68

information: [m-w.org]

1: the communication or reception of knowledge or

intelligence

1:31 AM 32

What is information?



31/68

Wh t i i f ti ?


32/68

Intuitively, an information source having more symbols

should have more information

For instance, consider a source, say S1, that wants to

communicate its direction to a destination using the

following symbols:

North (N), South (S), East (E), West (W)Another source, say S2, can communicate its direction

using:

North (N), South (S), East (E), West (W), Northwest (NW),

Northeast (NE), Southwest (SW), Southeast (SE)

Intuitively, all other things being equally likely, S2has

more information than S1

1:31 AM 34

What is an information source?


Mi i b f bit f


33/68

Before we formally define information, let us try toanswer the following question:

What is the minimum number of bits/symbolrequired to communicate an information source

having nsymbols?

1:31 AM 35

Minimum number of bits for a source



34/68

Minim m n mber of bits for a so rce


35/68

Let there be a sourceXthat wants to communicate

information of its direction to a destination

i.e., n=4 symbols: North (N), South (S), East (E), West (W)

According to our previous definition, log2(4)=2 bits are

required to represent each symbol

N: 00, S: 01,E: 10, W: 11

If 1000 symbols are generated byX, how many bits are

required to transmit these 1000 symbols?

1:31 AM 37





36/68

Let there be a sourceXthat wants to communicate informationof its direction to a destination

i.e., n=4 symbols: North (N), South (S), East (E), West (W) According to our previous definition, log2(4)=2 bits are

required to represent each symbol

N: 00, S: 01,E: 10, W: 11

If 1000 symbols are generated byX, how many bits arerequired to transmit these 1000 symbols?

2000 bits are required to communicate 1000 symbols2 bits/symbol

1:31 AM 38




37/68



38/68

Are 2 bits/symbol the minimum number of bits/symbol

required to communicate an information source having

n=4 symbols?

The correct answer isNO!

Lets see an example to emphasize this point

1:31 AM 40





39/68

So far in this example, we implicitly assumed that all

symbols are equally likely to occur

Lets now assume that symbols are generated according to

a probability mass functionpX

1:31 AM 41


N

0.6

0.3

S E

0.05

W

pX

X




40/68

Let us map the symbols to the following bit sequences:N: 0

S: 01

E: 011

W: 0111

1:31 AM 42


N

0.6

0.3

S E

0.05

W

pX

X



41/68


42/68



43/68

1:31 AM 45


Now if 1000 symbols are generated byX, how many bits are

required to transmit these 1000 symbols?

600 Ns, 300 Ss, 50 Es and 50 Ws

Total bits=6001+3002+503+504=1550

1550 bitsare required to communicate 1000 symbols

1.55 bits/symbol

N

0.6

0.3

S E

0.05

W

pX

X


N: 0

S: 01

E: 011

W: 0111



44/68

1:31 AM 46


1550 bitsare required to communicate 1000 symbols

1.55 bits/symbol

N: 0

S: 01

E: 011

W: 0111

N

0.6

0.3

S E

0.05

W

pX

X

The bit mapping defined in this example is generally called a code

And the process of defining this code is called source coding or

source compression

The mapped symbols (0, 01, 011 and 0111) are called codewords




45/68

Coming back to our original question:

Are 1.55 bits/symbol the minimum number of bits/symbol


n=4 symbols?

1:31 AM 47





46/68

Are 1.55 bits/symbol the minimum number of bits/symbol


n=4 symbols?

The correct answer is I dont know!

To answer this question, we first need to know the

minimum number of bits/symbol for a source with 4symbols

1:31 AM 48




47/68


48/68


49/68


50/68

Information content of a source


51/68

If we assume equally-likely symbols, we will always be

able to communicate all the symbols of the source using

log2(n) bits/symbol

In other words, this is the maximum number of bitsrequired to communicate any discrete source

But if a sources symbols arein fact equally likely, what is

the minimum number of bits required to communicate this

source?

1:31 AM 53

Information content of a source



52/68

Information content of uniform sources


53/68

The minimum number of bits required to represent a

discrete uniform source is log2(n) bits/symbol

For any discrete source where all symbols are not equally-

likely (i.e., non-uniform source), log2(n) represents themaximum number of bits/symbol

Among all discrete sources producing a given number of

n symbols, a uniform source has the highest information

content

1:31 AM 55




54/68



55/68

Two uniform sources S1and S1

n1and n2respectively represent the total number of

symbols for the two sources with n1> n2

Which source has higher information content?

1:31 AM 57




56/68



57/68

Thus if there are multiple sources with equally-likely

symbols, the source with the maximum number of

symbols has the maximum information content

In other words, for equally likely sources, a functionH(.)that quantifies information content of a source should be

an increasing function of the number of symbols

Lets call this functionH(n)

Any ideas whatH(n) should be?

1:31 AM 59




58/68


59/68


60/68


61/68


62/68


63/68

Information content of a non-uniform source


64/68

For a given symbol i, the information content of that

symbol is given by:

H(pX=i)=log2(1/pX=i)

So what is the expected or average value of the informationcontent of all the symbols ofpX?

1:31 AM 66Introduction to Information Theory


65/68



66/68

The information content of a discrete source with symboldistributionpXis:

1:31 AM 68

This is called the entropyof the source

and represents the minimum expected number of

bits/symbol required to communicate this source




67/68

Before finishing our discussion on information sources,

apply the formula for entropy on a uniform source:

1:31 AM 69

1 2

1/n

n

pX

X



68/68

basic lecture information theory and coding

Documents