unencrypted 4

7/29/2019 Unencrypted 4

1/58

Information Theory and Coding 314Dr. Roberto Togneri

Dept. of E&E EngineeringThe University of Western Australia

Section 4Error-Control Codes

(Channel Coding)

DISCLAIMER

Whilst every attempt is made to maintain the accuracy and correctness of these notes,the author, Dr. Roberto Togneri, makes no warranty or guarantee or promise expressor implied concerning the content


2/58

ITC314 Information Theory and Coding 314 623.314

22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-2

4. ERROR-CONTROL CODES (CHANNEL CODING)

4.1 Introduction

In the previous chapter we considered coding of information sources for transmission through anoiseless channel. In that case efficient compact codes was our primary concern. In general,though, the channel is noisy and errors will be introduced at the output. In this chapter we will beconcerned with codes which can detect or correct such errors. We restrict our attention to the classof binary codes and BSC channels. Our model for a noiseless channel was:

SourceEncoder

SourceDecoder

Modulator

Digital ChannelDemodulator

message

no errors

message

In the presence of noise we include an additional channel decoder/encoder. This allows us toseparate source representation (using compact codes) from error protection (using error-controlcodes).

Properties of Channel Coding

1.The channel encoder takes the binary stream output of the source encoder, and maps each fixed-block messageor informationwordof length L bits, to a fixed-block codewordof length N. Thuschannel coding usesblock codeswith N >L. The difference, N-L, represents thecheck-bitsused

for channel coding.

2.We restrict ourselves to binary channels. Hence the source encoding uses a binary code and thedata entering and leaving the channel encoder, channel decoder and channel is binary.

3.With L bits of information we need M =2L codewords, each N-bits in length. Thus we willreceive either one of the M =2L codewords or any one of the remaining 2N-2L words whichrepresent the codewords with detectable bit errors. It is this mapping operation and the extra N-L check bits which can be exploited for error-control.

Our model for a noisy channel is:

SourceEncoder

SourceDecoder

Modulator

Digital ChannelDemodulator

ChannelEncoder

ChannelEncoder

L bitmessage

N bitcodeword

N bitword

L bitmessage

NOISE

bit errors no errors

feedbackchannel error detected?


3/58



NOTE

1.With error correction the error is detected and corrected at the receiver. A communication systemwhich implements this form of channel coding is known as a Forward-Error-Correction (FEC)error-control system.

2.With error detection the error is detected and the transmitter is required retransmit the message.This requires a reliable return feedback channel from receiver to transmitter. The simplestimplementation is a stop-and-wait strategy whereby the receiver acknowledges (ACK) eachcorrect reception and negatively acknowledges (NACK) each incorrect reception. The transmitterthen retransmits any message that was incorrectly received A communication system whichimplements this form of channel coding is known as a Automatic-Repeat-Request (ARQ) error-control system.

3. In packet-switched data networks the, possibly, source-coded messages are transmitted in N-bit

codeword blocks called frames, packetsor segments (depending on which network layer handlesthe data) which incorporate not only the message but a control header and/or trailer. ARQ errorcontrol is used based on systematicerror-detecting channel codes with the check-bits included inthe appropriate checksum field of the header or trailer.

4.2 Information Rate (fixed-rate source coding)

It should be obvious that since N >L the rate of data entering the channel is greater than the rate ofdata entering the channel encoder. Let:

(bps)ratebitchannel1

(bps)ratebitcodingsourcefixed1

c

s

=

=

c

s

n

n

It should be evident that: [ ] [ ]sccs nn /LNor/LN == Equation 4.1

NOTE: If the source coder and channel bit rates are such that sc nn /L is not on integer, then a

dummy bit will occasionally have to be transmitted over the channel.

Channel Coding Information Rate

c

s

s

c

n

nR ===

N

L

Equation 4.2

Channel Code Redundancy:L

N1==

RCred

Equation 4.3

Excess Redundancy: L

L-N

1

1

== RE

red

Equation 4.4


4/58



Example 4.1

(a)_The source coder rate is 2 bps and the channel bit rate is 4 bps. Say L = 3, then

6]2/4[3]/[LN === sc nn and the channel coder will generate N-L =3 check bits for every L =3-bit message block and hence produce a N =6-bit codeword. The check bits can be used for error-

control (e.g. even parity-check)

(b)_

The source coder rate is 3 bps and the channel bit rate is 4 bps. Now L3

4N

= .

If L =3 ! N =4 (1 check bit for every 3-bit message block to produce a 4-bit codeword)

If L =5 ! 63

26N = (1 check bit for every 5-bit message block to produce a 6-bit codeword)

BUT there is a mismatch between the source code and channel data rates. Express the fraction as the

ratio of the two small integers dandn:nd=

32 thusn=3 andd=2. To synchronise the data rates

we need to transmitd=2 dummy bits for everyn=3 codewords / message blocks:

L =5, n=3" X X X X X X X X X X X X X X X =15 bitsN =6, d=2" + + + + + + + + + + + + + + + + + + ** =20 bits

Thus the average rate will be the correct (20/15) =(4/3). Both the check-bits +and dummy bits *can be used for error-control.

4.3 Decis ion Rules and Channel Decoding

With a noiseless channel we know with probability 1 which input was transmitted upon reception ofthe output. However, when the channel is noisy there is usually more than one possible inputsymbol that may have been transmitted given the observed output symbol. In the presence of noisewe must use a decision rule.

Decision Rule

Consider a channel with anr-symbol input alphabet:

A ={ai}, i =1,2 ..., r

and ans-symbol output alphabet:

B ={bj}, j =1,2 ..., s.

The decision ruled(bj) is any function specifying a unique input symbol for each output symbol, bj.

The decision ruled(b) is any function specifying a unique input word for each output word, b.


5/58



Example 4.2

Consider the channel matrix:

3

2

1

INPUT

a

a

a

4.03.03.0

5.03.02.0

2.03.05.0

OUTPUT

321

!! "!! #$bbb

Possible decision rules are:

Rule 1:"

33

22

11

)(

)(

)(

abd

abd

abd

===

and Rule 2"

23

22

11

)(

)(

)(

abd

abd

abd

===

With the second decision rule if 2311 bbbbb= then 22112311 )()( aaaabbbbdbd ==

The aim of a good decision rule is to minimise the error probability of making a wrong decision.

4.3.1 Minimum-Error Probability Decoding

We choose*)( abd j =

wherea* is given by:

P a b P a b ij i j( * / ) ( / ) Equation 4.1

Now From Bayes' law we have:

)(

)()/(

)(

*)(*)/(

j

iij

j

j

bP

aPabP

bP

aPabP

Equation 4.2SinceP(bj) is independent of ai inEquation 4.2we have:

)()/(*)(*)/( iijj aPabPaPabP

Equation 4.3

Thus the minimum-error probability decoding requires not only the channel matrix probabilities butalso the a priori probabilitiesP(ai). In some cases we do not have knowledge ofP(ai).


6/58



4.3.2 Maximum-Likelihood Decoding

We chose*)( abd j =

wherea* is given byP b a P b a ij j i( / *) ( / )

Equation 4.4

Thus maximum-likelihood decoding depends only on the channel. This is an advantage in situationswhere theP(ai) are not known. Although not minimum-error, maximum - likelihood decoding does

result in minimum-error decoding when the inputs are equiprobable.

4.3.3 Error probability

If we choose the input according to the decision ruled(bj) then the probability of getting it right is:

P a d b bj j( * ( ) / )=

and the probability of error:

P E b P a bj j( / ) ( * / )= 1

P(E/bj) is the conditional error probability given the outputbj.

MaximisingP(a*/bj) will minimise the error, hence the name minimum-error probability decoding!

The overall probabilityPE is the average ofP(E/bj) over all thebj.

=B

jjE bPbEPP )()/(

We can equivalently define:

=A

iiE aPaEPP )()/(

Equation 4.5and

= cBb ii abPaEP )/()/(Equation 4.6

whereb Bc is the summation over members of theB alphabet for whichd(bj) ai.


7/58



Example 4.3

Refer to same channel ofExample 4.2. Also assume we are given thatP(a1) =0.3, P(a2) =0.3 andP(a3) =0.4.

Maximum Likelihood DecodingThis requires finding the maximumP(bj/a*) for the given bj, that is the maximum value along the

column of the channel matrix )/( ij abPP corresponding tobj:

)/( ij abP

3

2

1INPUT

a

a

a

4.03.03.0

3.02.0

2.0

OUTPUT

321

0.5

0.30.5

!! "!! #$bbb

!

23

12

11

)(

)(

)(

abd

abd

abd

===

Note thatd(b2) =a2 =a3 are also correct. We calculate the error probability using:

==cBb

i

A

i

A

iiE abPaPaPaEPP

)/()()()/(

To derive

=CBb

ii abPaEP )/()/( we sum across theai row excluding thebj column ifd(bj) =ai:

3

2

1INPUT

a

a

a

% % &'(

!! "!! #$

61.0)()/(

40.04.00.1

15.03.05.0

06.03.02.0

4.03.03.0

3.02.0

2.0

OUTPUT

321

=

===

EP

iaPiaEP

0.5

0.30.5

bbb

ThusPE =0.61. Note thatP(E/a3) =1.0 is an obvious result since the decision rule never decides a3.

Minimum Error DecodingThis requires finding the maximumP(bj/a*)P(a*) for the givenbj, that is we need to know the inputprobabilities. We form the augmented channel matrix which is pre-multiplied by P(ai) and then find

the maximum value along the column of the augmented channel matrix )()/( iij aPabPP corresponding tobj:

)()/( iij aPabP

3

2

1INPUT

aa

a

0.160.12

0.15

12.015.009.006.0

06.009.0

OUTPUT

321

!!! "!!! #$bbb

!

33

32

11

)(

)(

)(

abd

abd

abd

=

==

And we calculate the error probability as before:

3

2

1INPUT

a

a

a

% % &'(

!! "!! #$

57.0)()/(

12.04.03.0

30.03.00.1

15.03.05.0

3.0

5.03.02.0

2.03.0

OUTPUT

321

=

===

E

P

i

aP

i

aEP

0.40.3

0.5

bbb

ThusPE =0.57 which is lower than maximum likelihood decoding.


8/58



Decision Rule for modern BSCs and typical source coders

The following will typically be the case:1.q

0 whenever ab

2. d(a,b) =d(b,a)

3. d(a,b) +d(b,c) d(a,c) (triangle inequality)

Example 4.4

Letn=8 and consider:

11001010

0100100010001011

=

==

c

ba

!

2),(

2),(

4),(

=

==

cad

cbd

bad

, and ),(),(),( cadcbdbad +


9/58



4.4.1 Decoding Rule using the Hamming Distance

As stated, for modern communications systems the maximum likelihood decoding rule will yieldthe same result as the minimum error decoding rule. When considering the decision rule for

deciding which N-bit codeword a was sent given the received N-bit word b, then a is chosen tomaximiseP(b/a). Say we transmit the N-bit codeword a and observe the N-bit word bat the output(of the BSC). Assume d(a, b) =D, then this means that a and b differ in exactly D places, thus Dbits are in error (with probabilityq) and N-D bits are OK (with probabilityp=1 - q). Thus:

( ) ( ) ( ) DND/ = pqabP

Sinceq t

Property 4.1

ProofLet a* be the codeword that is sent and b the received word. Assume there are t bit errors intransmission thend(a*,b) =t. To detect thatb is in error then it is sufficient to ensure that bdoes notcorrespond to any of theai codewords, i.e. d(b, ai) >0 for all i. Using the triangle inequality we have

for any other codewordai: )*,()*,(),()*,(),()*,( badaadabdaadabdbad iiii +To ensure d(b, ai) > 0 we must have: )*,()*,(0)*,()*,( badaadbadaad ii >> and sinced(a*,b) =t we get the final result that:

taad i >)*,( !d(K) >tand this is the condition for detecting up to all t-bit errors

Error CorrectionA block code K is said to correct t errors provided that for each code word a and each word bobtained by corrupting t bits in a, the maximum likelihood decoding leads uniquely to a. Thisproperty is important in situations where the communication system uses FEC error-control coding.

Error Correcting Property

A code corrects all terrors if and only if its minimum distance is greater than 2t:d(K) > 2t

Property 4.2


11/58



ProofLet a* be the codeword that is sent and b the received word. Assume there are t bit errors intransmission thend(a*,b) =t. To detect that b is in error and ensure maximum likelihood decodinguniquely yieldsa* (error can be corrected) then it is sufficient that d(a*,b) d(a*,b) we must have:

)*,(2)*,()*,()*,()*,( badaadbadbadaad ii >>and sinced(a*,b) =t we get the final result that:

taad i 2)*,( > !d(K) >2tand this is the condition for correcting up to all t-bit errors

Example 4.6

Even-parity check Code

The even-parity check code has N =L +1 by adding a parity-check bit to the L-bit message. For thecase L =2 we have:Message Codeword

00 00001 01110 10111 110

By comparing the Hamming distance between different codewords we see thatd(K) =2 (i.e. no twocodewords are less than 2 distance from each other) and by Property 4.1d(K) =2 >t =1 and as

expected this code will be able to detect all single-bit errors.

Repetition CodeThe repetition code is defined only for L =1 and performs error correction by repeating the messagebit (N-1) times (for N odd) and then using a majority vote decision rule (which is the same as theHamming distance decoding rule). Consider the N=3 repetition code:

Message Codeword0 0001 111

We see thatd(K) =3. FromProperty 4.1d(K) =3 >t=2 and this code can detect all 2-bit errors.Furthermore fromProperty 4.2d(K) =3 >2(t=1) and this code can correct all 1-bit errors. TheHamming decoding rule will perform 1-bit error correction. To see this and the majority votingoperation:

Non codeword Closestcodeword

Majoritybit

Action

001 000 0 message 0010 000 0 message 0011 111 1 message 1100 000 0 message 0101 111 1 message 1

110 111 1 message 1Note: Using majority voting of the received bits is a much simpler implementation for decoding thefinal message than using the Hamming distance decoding rule.


12/58



Code fromExample 4.5Message Codeword

00 00001 00110 01111 111

We see that d(K) =1 (e.g. distance between codewords 000 and 001) which would imply this codecannot not even do single-bit error detection let alone error correction. In Example 4.5 theHamming distance decoding rule was either performing 1-bit error detection or 1-bit errorcorrection. The explanation is that this code can detect most but not all 1-bit errors, specifically ifcodeword 000 is sent and there is a bit error in b3 we receive 001 and decode this to codeword 001(this is known as an undetected error as there is no way to detect such errors).

Code K6Code K6 will be used in a later example and is defined as follows

Message

(L =3)

Codeword

(N =6)000 000000001 001110010 010101011 011011100 100011101 101101110 110110111 111000

By comparing codewords we have thatd(K) =3 and hence this code can correct all 1-bit errors. Forexample, ifa=010101 is sent and b=010111 is received than the closest codeword is 010101 andthe error in b5 is corrected. This will always be the case no matter which codeword was sent andwhich bit is in error. It should be noted that there are only M =2L =23 =8 valid codewordscompared to 2N =26 =64 possible received words.

4.4.3 Upper boundB(N,d(K)) on number of codewords

Thed(K) for a particular block code K specifies the codes error correction/detection properties. Say

we want to design a code of length N and minimum distance d(K). What is the bound,B(N,d(K)) onM (and hence L), for any given (N, d(K)), i.e. findB(N,d(K) such that M B(N,d(K))?

This is sometimes known as the main coding theory problem (of finding good codes).Some results for B(N,d(K)) are:

B(N, 1) =2N

B(N, 2) =2N-1

B(N, 3) =2

N +1

N

B(N, 4) =2

N

N-1


13/58



In general, we have:

B(N, 2t+1) =B(N+1, 2t+2) for t=0,1,2,

Equation 4.7

thus if the we know the bound for evend(K) we can calculate the bound for oddd(K).

For evend(K) one bound (of the apparently many) is the Plotkin bound for evend(K):

N-(K)2

(K)2=(K))(N,

d

ddB providedd(K) is even and 2d(K) >N d(K)

Equation 4.8

Example 4.7

Our channel coder encodes messages of L =2 bits (M =4). What is the minimum block length, N,and possible code for the following requirements:

1-bit error detection

We needd(K) >t=1!d(K) =2, so M =4B(N,2) =2N-1! N =3 (smallest N)We need to design a code with L =2, N =3 and ensured(K) =2, so here it is:

Info Code00 00001 01110 10111 110

The even-parity check code!

1-bit error correction

We needd(K) >2(t=1) ! d(K) =3, so M =41N

2)3,N(N

+=B ! N =5 (smallest N)

We need to design a code with L =2, N =5 and ensured(K) =3, so here it is:Info Code00 0000001 0101110 1011111 11100

2-bit error detection

We need d(K) >(t =2) ! d(K) =3 ! same code as 1-bit error correction except that thedecoding rule is simply to indicate an error ifbai for all i.

2-bit error correction

We needd(K) >2(t=2) =4 ! d(K) =5, soB(N,5) =B(N+1,6) =N11

12

)1N()K(2

)K(2

=

+dd

using the Plotkin bound, and M =4N11

12

! N =8. We need to design a code with L =2, N =

8 and ensured(K) =5. Fun!


14/58



4.4.4 Issues

The problem with using the Hamming distance or encoding and decoding is that with L bitinformation words we generate 2L code words which we must store, and perform message word to

code word comparisons, which is an expensive operation. This represents an exponential increase inmemory and computational power with L. Only in the specific case of the repetition code is anefficient implementation (using majority voting) possible. Later we will develop a way ofsystematically encoding and decoding codes in an efficient manner by considering the importantclass ofbinary linear codes.

4.5 Block Error Probability and Information Rate

Let: Kn = block code of lengthn(code withn N-bit codewords)Pe(Kn) = block error probability for code KnPeb(Kn) = bit error probability for code KnR(Kn) = information rate for code Kn

The bit error probability Peb(Kn) indicates the probability of bit errors between the transmittedmessage and decoded message. This measures the true errors in using Kn. The block errorprobability Pe(Kn) indicates the probability of decoding to the incorrect codeword block. It shouldbe noted that Pe(Kn) Peb(Kn) since even with an incorrect codeword block not all of the bits willbe in error.

Example 4.8

Consider a channel coding with L=3 and N=6. The 3-bit information sequence 010 is encoded as010111 and sent through the channel. However due to a bit errors the received word is 011101 andthe channel decoder decodes this as codeword 011111 and message 011. Although the block errordecoding is 100% in error (i.e. Pe(Kn) =1.0) the first 2 bits are OK so the bit error decoding is only33% incorrect (i.e. Peb(Kn) =0.33).

When comparing different codes it is normal to usePe(Kn) rather thanPeb(Kn), sincePe(Kn) is easierto calculate thanPeb(Kn) andPe(Kn) represents the worst-case performance.

What is the relationship betweenPe(Kn) andR(Kn)?

Example 4.9

Consider a BSC withq=0.001 thenPeb(Kn) =0.001. How do we improve this and what is the pricethat we pay?

(n =3) repetition codeInfo Code

0 0001 111

We note that since L =1 thenPeb(Kn) =Pe(Kn). How do we calculatePe(Kn)? The majority vote andHamming distance decoding rule will fail to yield the correct codeword if: all n=3 bits are in error q3


15/58



2 out ofn=3 bits are in error pqpq 22 323 =

Hence:Pe(Kn) =q

3 +3q2p=3 x 10-6much better thanq=1 x 10-3, but there is a cost sinceR(Kn) =1/3

(n =5) repetition code

Info Code0 000001 11111

The majority vote and Hamming distance decoding rule will fail to yield the correct codeword if: all n=5 bits are in error q5

4 out ofn=5 bits are in error pqpq 44 545 =

3 out ofn=5 bits are in error 2323 1035

pqpq =

Hence: Pe(Kn) =q5 +5q4p+10q3p2 =10-8, but nowR(Kn) =1/5 !

With repetition codesn

R n1

)K( = and there is an exchange of message rate for message reliability.

But is there a way of reducingPe(Kn) without a corresponding reduction inR(Kn)? Yes!

Example 4.10

Consider a BSC withq=0.001 and the restriction thatR(Kn) =0.5. We design two codes K4 and K6in an attempt to lower Pe(Kn) and still keepR(Kn) =0.5

Code K4 (N =4, L =2)We transmita1 once and transmita2 three times to produce the following code:

I nfo Code00 000001 011110 100011 1111

We perform conditional error correction by assumingb1 is correct (a1 =b1) and then do a majorityvote on the remainingb2b3b4 bits to obtaina2. This decoding will be correct when: all n=4 bits are correct p4

b1 is correct and there is one error in the remaining 3 bits qpqp 33 313 =

Hence: 1 - Pe(Kn) =p4 +3p3q!Pe(Kn) =1 - p

4 - 3p3q=0.001

But this is indeed better if we consider Peb(Kn). In the following, let c mean correct and i meanincorrect. We can derive :Pci =Pr(a1 =c, a2 =i) q(3p

2q+p3) =! first bit only is incorrect (only 1 of the 2-bits is wrong)

Pic =Pr(a1 =i , a2 =c) =p(q3 +3pq2)! second bit only is incorrect (only 1 of the 2-bits is wrong)

Pii =Pr(a1 =i , a2 =i) =q(q3 +3pq2)! both bits are incorrect

Pcc =Pr(a1 =c , a2 =c) =p(3p2q+p3)! both bits are correct (decoding is correct)


16/58



And over a message with L =2 bits we can show that:

( ) ( )3223 32

322

)(0)(1)(2)K( pqp

qpqqq

pPPPPP ccicciiineb +++

=

+++=

Hence for q=0.001: Peb(Kn) =0.0005 1 bit errors (Show This!).Hence: Pe(Kn) =1 - p

6 - 6p5q=0.000015


17/58



channel by introducing redundancies. Shannon's Fundamental Theorem will tell us that for largeenough N, ifRC we can encode for error-free transmission:Shannons's Fundamental Theorem

Every binary symmetric channel of capacity C >0 can be encoded with an arbitrary reliability andwith information rate, R(Kn) C, arbitrarily close to C. That is, there exist codes K1, K2, K3 .such thatPe(Kn) tends to zero andR(Kn) tends toC with increasingn:

CRPn

en

==

)(Klim0,)(Klim nn 0limand0,)(Ki.e. 111 =>= nn CR

That is, IfR(Kn) C, thenPe(Kn) 0 for large enoughn, and as a bonusR(Kn) C!

4.6.1 Engineers Proof

The formal proof of Shannon's Theorem is quite lengthy and involved. We only present the salientfeatures of the proof here (which is why this is an engineers proof# ).

Assume an arbitrarily small number >1 0and that we have to design a code Knof lengthnsuchthat ( ) .K 1= CR n To do this we need an information length of )(L 1= Cnand hence:

11)()(K =

= C

n

CnR n

This means we need)1(2M

=

Cncodewords.

Shannon's proof makes use of random codes. Given any number nwe can pick M out of the 2n

binary words in a random way and we obtain a random code Kn. If)1(2M

=

Cnthen we know

that ( ) 1K = CR n butPe(Kn) is a random variable.

Denote:

))(KE(~

nee PP =as the expected value ofPe(Kn) for a fixed value ofn, but a completely random choice of Mcodewords. The main, and difficult part, of the proof (which we conveniently omit) is to show that:

as0~ nPeThus given arbitrarily small numbers 1>0 and 2>0 we can findnsuch that 2

~


18/58



Converse of Shannon's Theorem

In every binary symmetric channel of capacity C whenever codes Kn of lengths nhave information

rates at leastC +1 (1 >0) then the codes tend to be totally unreliable:( ) ( ) 1KK lim1 =+ nenn PCR

That is, IfR(Kn) >C, thenPe(Kn) 1 for large enoughn.

Example 4.11

Consider the BSC fromExample 4.10. With q =0.001 we know that:9886.0loglog1 =++= qqppC

We can now answer the4.5.1Issuesquestions:1.SinceR=0.5


19/58



4.7.2 Binary Linear Codes (Take 1)

Change in notation

n=N length of block code (entering the channel)k=L number of information / message bits in each block (entering channel encoder).

An important class of binary codes are the binary linear codes which can be described by systems oflinear equations as follows:

Binary linear codes (one definition)

Denote by xi the ith bit of the code word. Assume we have a message or information word ofkbits

which is encoded into a codeword ofn bits. In a binary linear code all the 2k codewords satisfy n-klinear equations inxi for i =1,2, n (i.e. n-kequations withnunknowns).

Furthermore, we can rearrange the equations to be homogenous (i.e. the RHS is zero)

Binary linear codes (another definition)

Every homogenous system of linear equations defines a linear code. We can also show that everylinear code can be described by a homogenous system of linear equations.

NOTE

Since there aren-kequations innunknowns then we haven - (n-k) =k independent variables andn-k dependent variables. Since the variables are binary this means we have 2k solutions which is asexpected.

Example 4.13

n-length repetition code (n odd)

Info Code:x1x2x3xn

0 0000

1 1111

Withk=1 then we needn-k=n-1 equations in thenunknowns (x1x2 x3xn) which fully describesthe repetition code:

1

13

12

xx

xx

xx

n =

==

)!

0

0

0

1

31

21

=+

=+=+

nxx

xx

xx

)


20/58



n-length even-parity check codeConsider the case ofn=3 (i.e. k=2):

Info Code:x1x2x3

00 00001 011

10 10111 110

In general n =k -1 and we expect n-k =1 equation which will fully specify the even-parity checkcode, and of course its the equation used to set the check-bit, x3, itself:

121 +++= nn xxxx * ! 021 =+++ nxxx *

4.7.3 Parity Check Matrix

Given a linear code K described by a system of (n-k) homogenous equations in (n) unknowns weconstruct the (n-k) x (n) matrix, H, with the co-efficients of the equations. That is, the ith row ofHexpresses theith equation. H is known as the parity-check matrix.

Parity Check Matrix and Parity Check equations

The binary matrix H is called a parity check matrix of a binary linear code K of length (n) providedthat the code words K are precisely those binary wordsxT =[x1x2 xn] which fulfil:

0

0

0

2

1

or

x

x

x

n

0HxH =

=

))andH has (n-k) rows and (n) columns

The system of (n-k) homogenous equations are known as the parity-check equations

NOTE: The systemHx =0will have 2k solutions.

Example 4.14

H for Repetition Code

0

0

0

1

31

21

=+

=+=+

nxx

xx

xx

) ! 0Hx =

=

n

n

n

x

x

x

x

x

n

1

3

2

1

1000001

0100001

0001001

0000101

0000011

1)

!!!!! "!!!!! #$

*

*

))+)))))

*

*

*

Hence, H is a (n-1) x (n) matrix


21/58



H for Even-parity check code

021 =+++ nxxx * ! [ ] 0Hx =

=

n

n

n

xx

x

x

1

2

1

11111 )!!! "!!! #$

*

Hence, H is a (1) x (n) matrix

4.7.4 Binary Linear Codes (Take 2)

If the wordx is a codeword thenHx =0, and ify is also a codeword thenHy =0. But ifz=x+ythenHz =Hx +Hy =0, z is also a codeword. This gives us the proper definition of a binarylinear code.

Binary Linear Code (proper definition)

A binary block code is said to be linear provided that the sum of any arbitrary two code words isalso a code word.

Example 4.15

Repetition CodeInfo Code:

x1x2x3xn

0 00001 1111

There are only two codewords and 0000 +1111 =1111 which is of course a codeword, sothe repetition code is a binary linear code.

Even-parity check codeThe even-parity check code is a binary linear code.

Code K4 fromExample 4.10 Info Code00 000001 011110 100011 1111

By adding any two codewords in can be shown that K4 is a binary linear code:0111 +1000 =11111111 +1000 =0111

Code K6 fromExample 4.10Showthat K6 is a binary linear code


22/58



Code fromExample 4.5Info Code00 00001 00110 01111 111

Is this code linear? Consider:001 +011 =010! this is not a binary linear code

4.7.5 Rectangular Code

As we shall see describing binary linear codes by the parity-check matrix will facilitate both theanalysis and design of codes. However in some cases special codes can be constructed which can beanalysed in a different and more intuitive way. One such class of code is the r x s rectangular codewhich is used when transmitting row by row or two dimensional data blocks. This is a binary linear

code of lengthn=rs, whose codewords are considered asr x smatrices. Each row and column hasr-1 ands-1 information bits respectively.

3 x 4 rectangular coder =3 ands =4"n =3 x 4 =12 andk =(3-1) x (2-1) =6. To analyse this code we represent thecodeword (x1, x2, , x12) in the following matrix form:

r =3

s=4

1211109

8765

4321

xxxx

xxxx

xxxx

The information bits are:x1 x2 x3 x5 x6 x7

The check bits are:x4 x8 x9 x10 x11 x12

Each check-bit is an even-parity check over the corresponding row or column. If one includes thecheck-bits themselves then there are r +s rows and columns to check over and hence r +s =7parity-check equations:

checks-parityrow

0

0

0

1211109

8765

4321

=+++

=+++=+++

xxxx

xxxx

xxxx

checks-paritycolumn

0

0

0

0

1284

1173

1062

951

=++=++=++

=++

xxx

xxx

xxx

xxx

We expectn-k=6 parity-check equations so one of the equations is in fact redundant. However all 7equations are used to derive the parity-check matrix description:


23/58



0Hx =

=

12

11

6

2

1

100010001000

010001000100

001000100010

000100010001

111100000000

000011110000

000000001111

x

x

x

x

x

)

)

rowparity-checks

column

parity-checks

4.7.6 Systematic Binary Linear Codes

Systematic (Separable) Binary Linear Codes

Block codes in which the message or information bits are transmitted in unaltered form are called

systematic codes. Specifically for binary linear codes, consider the (k) length messagemT =[m1, m2,, mk] (mi is thei

th bit if the message). The (n) length codeword is represented by:

xT =[mT |bT] =m1, m2, , mk, b1, b2, , bn-k wherebi is theithcheck bit

That is, the codeword is formed by appending the (n-k) check bits to the (k) information bits

NOTE: Unless otherwise stated, all codes we develop will be systematic codes

Communication Systems by S. Haykin uses a reversed vector representation

The text by S. Haykin adopts the notation: b0, b1, , bn-k-1, m0, m1, , mk-1which should compared to our notation: m1, m2, , mk, b1, b2, , bn-k

The notation adopted by S. Haykin assumes bits run to the right with mk-1 being the LSB (leastsignificant bit) of the message. The notation adopted here assumes bits run to the left with m1being the LSB of the message. Thus the bit sense is reversed and the subscript notation is different.

Comparing our notation to that of S. Haykin:(LSB) m1 =mk-1, m2 =mk-2,, bn-k-1 =b1, bn-k =b0 (MSB)

Example 4.16

The repetition code is obviously systematic

n-length even-parity check codeConsider the case ofn=3 (i.e. k=2 andn-k=1):

Infom1m2

Codex1x2x3

00 00 001 01 110 10 111 11 0

same

This is a systematic code sincex1x2 =m1m2 andx3 =b1 is the check-bit


24/58



Code K4 fromExample 4.10We have thatn=4, k=2 andn-k=2:

Infom1m2

Codex1x2x3x4

00 00 0001 01 11

10 10 0011 11 11

same

This is a systematic code sincex1x2 =m1m2 andx2x4 =b1b2 are the check-bits

Code K6 fromExample 4.10Showthat K6 is a systematic code

By makingm1, m2, , mk the independent variables and theb1, b2, , bn-k the dependent variablestheb1, b2, , bn-k can be expressed as explicit functions of them1, m2, , mk variables only. Hence:

Result 4.1

In encoding, the (n-k) check-bits are appended to the (k) length message to form the resultant (n)-length codeword.In decoding, after the correct codeword has been chosen (i.e. errors corrected), the (n-k) bitsappended to the (n)-length codeword are stripped to form the (k)-length message.

4.7.7 Systematic form for H

Define:b as the (n-k) column vector of check bitsmas the (k) column vector of information bits

cas the (n) column codeword vector =

b

mx

P as the (n-k) x (k) coefficient matrixG as the (n) x (k) generator matrixHas the (n-k) x (n) parity-check matrix

Since the check-bits are generated from the information bits we represent this operation by usingP :

b =Pm

Equation 4.9

and

=

P

IG k , whereIk is the (k) x (k) identity matrix, generates the codeword:

=

=

==

b

m

Pm

mm

P

IGmc k

Equation 4.10


25/58



and ifH =[P | In-k ], whereIn-k is the (n-k) x (n-k) identity matrix, then:

0bbbPmb

mI|[PHc =+=+=

= ]n-k

Exercise: Show that HG =0

Systematic Form for H

H =[ P | In-k ]Equation 4.11

NOTE

1.The systematic form for H makes it a trivial operation to extract P and hence generate the checkbits from the information bits.

2. IfH is in non-systematic form, the resulting parity-check equations will need to be manipulated,or H can be directly manipulated by standard row operations into systematic form.

Communication Systems by S. Haykin uses a reversed matrix representation

The parity-check matrix systematic structure adopted by S. Haykin is different than that describedhere and both the rows and columns of H are reversed. Thus a parity-check matrix, H, from S.Haykin:

=

mnmm

n

n

aaa

aaa

aaa

*)+))

*

*

21

22221

11211

H is equivalent to our:

=

11121

21222

12

aaaaaa

aaa

n

n

mmmn

**

))+)

*

H

Example 4.17

Repetition code

0Hc =

=

1

2

2

1

1

1000001

0100001

0001001

0000101

0000011

n

n

b

b

b

b

m

)

*

*

))+)))))*

*

*

n-1

P In-1

m

b


26/58



[ ] Pmb =

=

=

1

1

2

2

1

1

1

1

1

m

b

b

b

b

n

n

))

[ ] Gmc =

=

=

1

1

2

2

1

1

1

1

1

1

1

m

b

b

b

b

m

n

n

))

I1

P

Even-parity check code

[ ] 0Hc =

=

1

1

2

2

1

11111

b

m

m

m

m

n

n

)*

P I1

n-1

[ ] [ ] Pmb =

==

1

2

2

1

1 1111

n

n

m

m

m

m

b )*

Gmc =

=

=

1

2

3

2

1

1

1

2

2

1

11111

10000

00100

00010

00001

n

nn

n

m

m

m

m

m

b

m

m

m

m

)

*

*

))+)))*

*

*

)

P

In-1


27/58



4.7.8 Issues

Given the 2k solutions to the systemHx =0which we use as the codewords, and the corresponding2k information words or messages, we can generate a table of codewords with 2k entries. Encodingwill then involve a table lookup, and decoding will involve searching the same table for the closest

matching codeword (Hamming distance decoding rule). Both operations involve an exponentialincrease of complexity withk. However there is a better way: systematic binary linear codesmake encoding a simple logic operation on thek-bit message. syndrome decoding involves a simple logic operation on the n-bit received word for error

detection. Error correction will involve a lookup operation of a 2n-k table which becomes asimple logic operation on then-bit received word for the case of single-bit error correction.

It should be noted that lookup of a 2n-k table is much less expensive than finding the closest matchin a 2k table. Not only is a table lookup operation more efficient to implement (e.g. as an indexingoperation) but (n-k)


28/58



4.8.2 Syndrome

Let rbe the received word:

r =ci +e

whereci is the ith codeword that was transmitted ande is the error pattern of the transmission error.

Error correction involves identifying the error pattern, e, given only r. Once we know ewe obtainthe correct codeword by: ci =r +e. But how do we getefromr? Consider:

Hr =Hci +He=0+He=s

The (n-k) column vector, s, is called thesyndrome.

Now let:]|||||[ 1321 nn hhhhhH = *

wherehi corresponds to theith column ofH. And let:

=

ne

ee

)

2

1

e

whereei is theith bit ofe. Consider:

shhhhhhhhHe =+++=

= ][][][]|||||[ 22112

1

1321 nn

n

nn eee

e

e

e

*)

*

Consider a bit error in bitsj, kand l of the codeword thenej =ek =el =1 and all other ei =0. Hence itfollows that:

][][][ lkj hhhs ++=

That is, the syndrome is the sum of the columns inH which correspond to the bits in error. Thus weget the following result:

Result 4.2

The syndrome, s, only depends on the error pattern, e, and the syndrome is the sum of the columns

ofH which correspond to the bits in error, that is == 1s.t.for][ ii eihs .Notice that although s is calculated fromr it is related to e. But does knowing s allow us to get e?Unfortunately there are many error combinations which can yield the same syndrome (i.e. we cantsimply doe=H-1ssinceH is non-invertible). We make the following observations:

1.Consider the two error patterns, ei and ej, which yield the same syndrome, thenHei +Hej =s+s =0, henceci =(ei +ej) is by definition a codeword. Equivalently, ifei yields sthenei +calso yields the sames for any and all codewordsc.

Result 4.3

All error patterns that have the same syndrome differ by a codeword and a non-zero codewordadded to an error pattern will result in a different error pattern with the same syndrome.


29/58



2.The collection of all the error patterns that give the same syndrome is called acoset of the code.Since there are 2n-k different syndromes then there will be 2n-k different cosets for the code. FromResult 4.3since there are 2k codewords there will be 2k distinct error patterns in each coset.[Note that with 2n-k cosets and 2k error patterns per coset we will have 2n-k x 2n-k =2n errorpatterns and hence all possible error pattern combinations will have been accounted for]. Thusfor a given syndrome there will be 2k distinct error patterns in the coset. Which error pattern is

the right one?

3.Based on the Hamming distance decoding rule in the particular coset associated with thesyndrome calculated, it is reasonable to select the error pattern with the minimum number of biterrors (i.e. minimum number of bits that are 1), as this is the most likely error pattern to occur.The error pattern with the minimum number of bits being 1 in a particular coset is known as thecoset leader. Hence we choose the coset leader and add that to r to yield the most likelycodeword that could have been sent.

4. If there is more than one candidate coset leader this implies that for that particular syndrome the

error cannot be corrected since there is more than one error pattern which is equally likely.

4.8.3 Encoding of Systematic Codes

IfH is in systematic form (seeEquation 4.11), then the co-effiicient matrix, P, is trivially extracted.IfH is not in systematic form, it will need to be converted to systematic form by the appropriate rowoperations. Alternatively, the system of equationsHx =0can be converted directly to the formb =Pmby the appropriate algebraic operations.

Encoding the (k) bit information word, m, to the (n)-bit codeword, c, proceeds as follows:

1.Generate the check-bit vector, b =Pm

2.Form

=b

mc

Example 4.19

Consider the encoding process for the code fromExample 4.18with parity-check matrix:

=

01110

10010

01001

H

SinceH is not in systematic form we perform the following row operations in the order stated swap rows 2 and 3 swap rows 1 and 2 add row 2 to row 1

and hence:

=

=

10

01

11

10010

01001

00111

PHsys

Thus we get the check-bit generation equations:


30/58



=

=

2

1

3

2

1

10

01

11

m

m

b

b

b

Pmb

23

12

211

mb

mb

mmb

==

+=

25

14

213

xx

xx

xxx

==

+=

The code table is now derived by generating the check-bits from the message and appending thecheck-bits to the message to form the codeword:

Infox1x2

Codex1x2x3x4x5

0 0 0 0 0 0 00 1 0 1 1 0 11 0 1 0 1 1 01 1 1 1 0 1 1

4.8.4 Syndrome Decoding

Decoding the (n)-bit received word, r, to retrieve the (k)-bit information word, m, proceeds asfollows:

1. Calculate the syndromes=Hr2. For error detection:

ifs0 then there is an error and an indication has to be sent to retransmit the message ifs=0 then there is no error andc=r, hence go to step 4.

3. For error correction: Determine the unique coset leader, ec, of the coset corresponding to s and then correct the

most likely error pattern by c=r +ec If there is more than one candidate coset leader then only error detection is possible and anindication has to be sent to retransmit the message.

4. Extract the message, m, by stripping off the (n-k) check bits appended toc.

But how do we determine the coset leader? FromResult 4.2andResult 4.3at least two approachessuggest themselves:1. Find the minimum number of columns in H that when added together yield the syndrome. The

locations of the columns specify the bit positions of the coset leader, ec.2. Find one combination, any combination, of columns in H that when added together yields the

syndrome. Define the error pattern, e, such that ei =1 if column i was used in the combinationand then form the coset by adding each 2k codeword to eand then locate the coset leader as theerror pattern with the minimum number of bits being 1.

Since the above operations are expensive to perform for each received word, the usual practice is toprime the decoder with the pre-determined coset leader for each of the possible 2n-k syndromes. Thecoset leader is then found by usingsto index (lookup) the 2n-k table of coset leaders.

In the case of single-bit errors, fromResult 4.2 ifsmatches the ith column ofH then bit i is in error(the coset leader hasei =1, and all other bits are 0) and the codeword, c, is obtained by inverting theith bit ofr. Thus there is no need to explicitly determine the coset leader.


31/58



Example 4.20

Consider the decoding process for the code fromExample 4.18.SayrT =[01111] is received. The syndromes is calculated:

=

==01

0

1

11

1

0

1001001001

00111

Hrs

=

0

1

0

s matches only the 4th column ofH, " bit 4 is in error (i.e. ecT =[00010]) and hence

cT =[01101] which is a valid codeword. The message sent is thenmT =[01].

Say rT = [00111] is received. The syndrome

==

1

1

1

Hrs does not match any column of H.

Looking for the minimum number of columns which when added yieldswe see that:

shh =

=

+

=+

1

1

1

0

1

0

1

0

1

42 and shh =

=

+

=+

1

1

1

1

0

0

0

1

1

51

Hence there are two candidate coset leaders representing double-bit errors:ec1

T =[01010] andec2T =[10001] and thus the error cannot be corrected. Indeed we see that

c1T

=(ec1 +r)

T

=[01101] andc2T

=(ec2 +r)

T

=[10110] are equally distant fromr

T

=[00111].

4.8.5 Encoding and Decoding Implementations

EncodingGeneration of the (n-k) check-bits from themmessage bits, b =Pm, can easily be implemented inhardware using XOR logic gates or making use of XOR assembly language instructions forimplementation in software or embedded systems.

XORlogic gate

array

km

m

m

)

2

1

km

m

m

)2

1

knb

b

b

)

2

1

+

n

n

k

k

c

c

c

c

c

c

1

1

2

1

)

)


32/58



DecodingAssuming an FEC system, for each syndrome, s, the corresponding unique coset leader, ec, isderived and stored. Thus syndrome decoding becomes a table lookup operation using s as the indexinto a table of size 2n-k. The generation of the syndrome can be implemented using XOR logic gatesor special XOR assembly instructions. The table lookup operation can be implemented using a (n-k)-to-2n-k decoder and inverters can be used to correct the required bits in error.

XORlogic gate

array

nr

r

r

)

2

1

kns

s

s

)

2

1

n-kto2n-k

decoder

inverterlogicenable

inverterlogicenable

inverterlogicenable

inverterlogicenable

2n-k

inputORgateor

buffer

nc

c

c

)

2

1

)

4.9 Hamming Weight

Hamming weight / Minimum weight

The Hamming weight of a word, w(x), is defined as the number of bits distinct from 0. For eachnon-trivial code K, the smallest Hamming weight of a code word distinct from 0T =[000...0] iscalled the minimum weight of K, w(K).

Example 4.21

Hamming weight examples:w(xT =[111]) w(111) =3

w(101000) =2w(110110) =4

The minimum weight of the codes considered so far are:w(K) =n for then-length repetition codew(K) =2 for then-length even-parity check codew(K4) =1 for code K4 fromExample 4.10w(K6) =3 for code K6 fromExample 4.10

For the r x s rectangular code, K, it is not necessary to list the codewords in order to find theminimum weight. All that is needed is to use the row and column parity-check equations andattempt to find the non-zero codeword solution which uses the least number of bits equal to 1. It can

thus be shown thatw(K) =4 for any r x srectangular code.


33/58



Relationship between Hamming distance and Hamming weight for binary linear codes

For each non-trivial binary linear code the minimum distance, d(K), is equal to the minimumweight, w(K). That isd(K) =w(K)

Property 4.3

FromProperty 4.1andProperty 4.2we can now say:

Error Detection / Correction for binary linear codes

A binary linear code corrects (or detects) all terrors if and only if its minimum weight is larger than2t (or larger than t, respectively)

Property 4.4

Example 4.22

w(K) =n for then-length repetition code hence:" detect all (n-1) bit errors and correct all (n-1)/2 errors

w(K) =2 for then-length even-parity check code" detect all 1 bit errors

w(K4) =1 for code K4 fromExample 4.10" no error protection at all (code K4 corrects single-bit errors conditional on x1 being correct, but

ifx1 is corrupted this is not detected, hence code K4 has no unconditional error protection)

w(K6) =3 for code K6 fromExample 4.10

" corrects all 1 bit errors and detects all 2-bit errors

w(K) =4 for r x srectangular code" corrects all 1-bit errors and detects all 3-bit errors

4.10 Designing the Parity-Check matrix

Let x be the codeword with a Hamming weight equal to the minimum weight, w(K), of the code K.

IfH is a parity-check matrix for K, then we knowHx=0. Now by definitionxhas exactlyw(K) bitswhich are 1, the remaining bits being 0. Thus the operation Hx =0 effectively sums w(K) columnsofH to 0. This gives us the following results:

Result 4.4

For a linear code K with parity-check matrix, H, if the minimum number of columns ofH that sumto zero isn, thenw(K) =n.

Result 4.5

For a linear code K with parity-check matrix, H, if no combination ofnor less columns ofH sum tozero, thenw(K) >n.


34/58



4.10.1 Structure ofHfor single-error detection

For single-bit error detectionw(K) =2 or w(K) >1. FromResult 4.5this means no single column ofH must sum to zero, i.e. no column ofH must be a zero column. This means w(K) is at least 2 andcan detect all single-bit errors.

Property of H for single-error detection

A binary linear code K detects single-bit errors if and only if every parity-check matrix of K hasnon-zero columns (i.e. no column is the all-zeros column).

4.10.2 Structure ofHfor single error correction

For single-bit error correction w(K) =3 or w(K) >2. FromResult 4.5this means no single column(i.e. n=1) ofH must sum to zero and no two columns (i.e. n=2) ofH must sum to zero. The firstcondition implies non-zero columns and the second condition implies no two columns of H areidentical (if and only if two columns are identical will they sum to zero), that is, the columns arepairwise distinct.

Property of H for single-error correction

A binary linear code K corrects single-bit errors if and only if every parity-check matrix of K hasnon-zero, pairwise distinct columns (i.e. no column is the all-zeros column and no two columns arethe same).

If a binary matrix withr

rows has to have nonzero, pairwise distinct columns this means that thenumber of columns, c


35/58



Since the columns are non-zero, pairwise distinctw(K) =3

Infox1x2

Codex1x2x3x4x5

0 0 0 0 0 0 00 1 0 1 0 1 1

1 0 1 0 1 1 01 1 1 1 1 0 1

The minimum weight of the code is, as expected, w(K) = 3. Hence this is a single-bit errorcorrecting code. Note that the code fromExample 4.18is also another solution.

(b)It is now required to design the most efficient code for 2-bit error correction with k =2. Thisrequires a code with at least w(K) = 5. To determine the smallest n that permits this, the upperbound expressionB(n,5) that was derived inExample 4.7 for the same case of M =4, d(K) =5 andL =2, was N n=8, andn-k=6 check bits, thusH is a 6x8 matrix.

To design the matrix we note that to be in systematic form, predefines the 6x6 identity matrix, I6.How do we choose the remaining 2 columns (which defineP)? Sincew(K) =5 we must ensure thatno combination of 4 or less columns will add to zero. Here is one solution:

=

10000001

01000001

00100010

00010011

00001010

00000111

H

P I6

18

17

26

215

24

213

xx

xx

xx

xxx

xx

xxx

=

==

+==

+=

Infox1x2

Codex1x2x3x4x5x6x7x8

0 0 0 0 0 0 0 0 0 00 1 0 1 1 1 1 1 0 01 0 1 0 1 0 1 0 1 11 1 1 1 0 1 0 1 1 1

From which we see that the minimum weight of the code is indeed w(K) =5. Also 4182 ==R which

is not that good. We may need to increasekto achieve better efficiency.


36/58



4.11 Perfect Codes for Error Correction

A binary linear (n,k)-code requires 2k codewords of length n. Thus there are 2n possible words oflengthn and 2k of these are codewords and 2n - 2k of these are non codewords. If the code correctsup to terrors, the non codewords are either:

Type 1: non codewords of distancet or less from precisely one of the codewords, that is, the t orless bits in error can be corrected.

Type 2: non codewords of distance greater than t from two or more codewords, that is, the errorscan only be detected.

If all the non codewords are ofType 1 then the code is a perfect code.

Properties of Perfect Codes1. Perfect codes will always be able to correct the most likely errors for any word received, there is

no need to handle detected errors (there will always be a unique coset leader).2. Perfect codes guarantee maximum information rate for t-bit error correction (this is intuitively

obvious since the unnecessaryType 2 non-codewords dont exist and fewer words meansmallern).3. A perfect code correcting up to t errors will also have the smallest allowable minimum distance

of 2t+1.

Conceptual definition of a perfect code

A linear code of length n is called perfect for t errors provided that for every word r of length n,there exists precisely one code word of Hamming distancetor less fromr.

Operational definition of a perfect code

With onlyType 1non-codewords, r, thens=Hr, must be able to correct up to t-bit errors. Thus the2n-k possible values of s must be the same as the total number of different combinations of0, 1, 2, , t-1, tbit errors of ann-bit word. Thus a linear (n,k) code is perfect for terrors if:

=

=

t

i

kn

i

n

0

2

Equation 4.12

Example 4.24Consider the code fromExample 4.23(a):

Info Code0 0 0 0 0 0 00 1 0 1 0 1 11 0 1 0 1 1 01 1 1 1 1 0 1

Consider the 5-bit word 00111 It is not of distance (t=1) or less from precisely one codeword It is, in fact, of distance 2 from two codewords: 01011 and 10110! NOT a perfect code

Furthermore:

615

058222

1

0

325 =

+

=

===

=

=

t

i

knin


37/58



4.12 Hamming Codes - Perfect Codes for Single Error Correction

The Hamming codes we now consider are an important class of binary linear codes. Hammingcodes represent the family of perfect codes for single-error correction.

For single-error correctiont=1, and since the code is perfect fromEquation 4.12:

121

0

+=

=

=

=

ni

nt

i

kn = ! n=2m-1 wherem=n-k and k=n-m=2m-m-1

Properties of a (n,k) Hamming Code

Block length: n=2m-1 m 3

Information length : k=n- m=2m-m-1

Min. distance: d(K) =3

Error: single bit correction.

NOTE

1. For single-error correction, H must have non-zero, pairwise distinct columns. For mrows wecan have, at most, 2m-1 non-zero and distinct columns for H (i.e n 2m- 1 ).

2. Since

n

mn

n

kR

== , the more columns we have (larger n) per fixed m the better our

information rate. Thus the condition for perfect codes, n=2m-1, implies maximum informationrate.

Hamming Code (one possible definition)

A binary linear code is called a Hamming code provided that it has, for some number m, a paritycheck matrix H ofmrows and 2m-1 columns such that each non-zero binary word of length mis acolumn ofH.

4.12.1 Syndrome Decoding for Hamming Codes

Since the Hamming codes are perfect codes for single-error correction, all 2n possible receivedwords, r, are either a codeword or a codeword corrupted by 1 bit. Thus for each possible syndrome,s, there must be only one coset leader. From the syndrome decoding condition for single-errorcorrecting codes we have the following decoding procedure for Hamming codes:

1.Calculate the syndrome: s=Hr2. Ifs is zero there is no error.3. Ifs is non-zero, let i be the column of H which matches s, then correct the ith bit of the received

word to form the codeword, c.4.Decode to the information word, m, by stripping off the (n-k) check bits appended toc.


38/58



Example 4.25

Consider the following (7,4) Hamming code parity-check matrix:

=

1010101

1100110

1111000

H "3

4and7

====

knm

kn

Design the encoding logic, list the codewords and perform syndrome decoding on:0110010 0110101 1101100 0001111

Code TableWe need to convert H to systematic form:

=

1001011

0101101

0011110

sysH !

4217

4316

4325

xxxx

xxxx

xxxx

++=++=++=

messagex1x2x3x4codewordx1x2x3x4x5x6x7

messagex1x2x3x4codewordx1x2x3x4x5x6x7

0000 0000000 1000 10000110001 0001111 1001 10011000010 0010110 1010 10101010011 0011001 1011 10110100100 0100101 1100 11001100101 0101010 1101 11010010110 0110011 1110 11100000111 0111100 1111 1111111

Encoding Logic

x4 x3 x2 x1

messagex4x3x2x1

x7 x6 x5

codewordx7x6x5x4x3x2x1

UP for first 4bits

DOWN forlast 3 bits

N.B. Bits are serially shifted to the right with the LSB first (right-hand side) and MSB last (left-hand side), thus the message and codeword bits are reversed.


39/58



Syndrome DecodingWe calculates=Hr (using the original non-systematic form ofH):

rT sT i cT mT

0110010 111 7 0110011 01100110101 011 3 0100101 0100

1101100 010 2 1001100 10010001111 000 - 0001111 0001

NOTE

1.The first three Hamming codes are: m n k R

3 7 4 4/7 =0.57 4 15 11 11/15 =0.73 5 31 26 26/31 =0.84

2.The information rate for the Hamming codes go as12

1

=

=m

m

n

mnR but althoughR tends

to 1 rapidly for larger mwe correct only single errors for increasingly large blocks less errorprotection.

Block Error Probability for Hamming Codes

22121

120

121)(

=

mm

qppmPmm

e

4.13 Types of Errors - Probabil ity of Undetected Error

Assume the codeword ci was transmitted and r is received. The following types of errors can bepresent when decoding a word:1. No error (s=0andr =ci) " ideal

2. Detected error (s0andranyc):a) correctly correct (coset leader r =ci" non-fatal)b) cannot correct (two or more coset leaders " non-fatal)c) incorrectly correct (coset leader r =cjci" fatal)

3. Undetected error (s=0andr =cjci) " fatal

In FEC error-control both 2(c) and 3 are fatal (and also 2(b) if there is no ARQ capability). ForARQ error-control only 3 is fatal since the case of detected errors in 2 are dealt with in the sameway (by re-transmission). Since most communication systems use ARQ, it is the undetected errorsthat are a universal problem.


40/58



4.13.1 Probability of undetected error

An undetected error occurs if the received word, r, is a different codeword, (r =cj), then wastransmitted, ci. Thus:

e=ci+r =ci +cj (eis the sum of two codewords!eis a codeword itself)

Condition for an undetected error

An undetected error occurs if and only ifeis a nonzero code word.

The probability of undetected error, Pund(K), is the sum of the probability of occurrence of the errorpatterns which yield an undetected error. Define ei as the error pattern with i bits in error, thus theprobability of occurrence ofei is =qipn-i. Since theei of interest are those which are the same as thenon-zero codewords, we letAi =number of code words with Hamming weight i, and hence:

=

= n

i

iniiund pqAP

1

)K(

If the minimum weight of the code is d(K) thenA1=A2=... =Ad(K)-1=0 so we have:

Probability of undetected error

=

=n

di

iniiund pqAP

)K(

)K(

Equation 4.13

where Ai =the number of codewords with Hamming weight i

Example 4.26

Consider the (7,4) Hamming code table ofExample 4.25from which we see that:d(K) =3 ! A3 =7, A4 =7, A5 =0, A6 =0, A7 =1

Hence:73443 77)K( qpqpqPund ++=

Ifq=0.01, then:Pund(K) =7 x 10

-6" 7 codewords in a million will have undetected errors


41/58



4.14 Other Error Detecting/Correcting codes

4.14.1 Types of Errors

When examining different codes it is important to consider: Type of Code: Is the code systematic or non-systematic? Information Rate: How many check-bits are used per message bit? Error protection: Number of bit errors that can be detected / corrected. Diagnostic resolution: Range over which bit errors are detected / corrected

(i.e. larger n" lower resolution) Error latency: How much data needs to be received before the error is detected / corrected

(i.e. larger n" longer latencies) Coverage: Which bit/word positions will be protected and in what way. Types of errors: Is the code best suited to random or burst errors?

The discussion so far has assumed that bit errors occur randomly (i.e. channel is memoryless orindependent). In practice, however, the condition that causes the bit error may last long enough to

affect a sequence of bits (i.e. channel has memory). If a sequence of , consecutive bits are suddenlysubject to interference, which causes a large number of them to be in error, then we have a condition

called a burst error of length,. A specific type of burst error is the multiple unidirectional errorwhich forces groups of bits to all go 0 (low) or 1 (high). This can be caused by a common mode offailure in the memory block or bus causing a group of local cells or bus lines to go low (i.e. shorted)or high.

4.14.2 Bit interleaving

Although codes can be designed to specifically handle ,-length burst errors (e.g. the CRC codesdiscussed in Section 4.15), a larger class of codes (e.g. the Hamming codes) exist which areapplicable only to random (or independent channel) errors. One way to convert burst errors torandom errors is to interleavebits entering the channel in a deterministic manner. If a sequence ofbits experience a burst-error the corresponding deinterleavingoperation will spread these errors outand hence randomise them.

Encoder Interleaver Burst-errorChannel Deinterleaver Decoder

Independent-error Channel

InputData

OutputData

Example 4.27

Consider the following 16-bit block of data: b1b2b3b4 b5b6b7b8 b9b10b11b12 b13b14b15b16

1:4 Interleaver: b1b5b9b13 b2b6b10b14 b3b7b11b15 b4b8b12b16

A burst-error of length 4 occurs: b1b5b9b13 b2b6bb1100bb1144bb33bb77b11b15 b4b8b12b16

1:4 Deinterleaver: b1b2bb33b4 b5b6bb77b8 b9bb1100b11b12 b13bb1144b15b16

And the burst error of length 4 has been converted a 1-bit random error in eachn=4 code block.


42/58



4.14.3 m-out-of-n Codes

Them-out-of-n codes are formed as the set ofn-bit vectors, where the vector is a code word if andonly if exactly mbits of the bits are 1. The 2-out-of-5 code is the most common example of such a

code:Table 4.1: 2-out-of-5 code

message number codeword1 000112 001013 010014 100015 001106 010107 10010

8 011009 1010010 11000

Properties Non-systematic code (need to do a table lookup) Single error detection Multiple unidirectional error detection Used in encoding control signals

Since the code is non-systematic its use for encoding and decoding communication messages is

limited. However control signals which do not need to be decoded (just interpreted) can beconfigured to exist asm-out-of-ncodes, and any multiple unidirectional error on the control bus canbe detected. In such a case it is obvious that the number of 1s will change and the error can bedetected.

Example 4.28

Consider a 2-out-of-5 code and assume message 7 (encoded as 10010) is sent.

Suppose a unidirectional error that causes the first 3 bits to go low occurs (most common errorwhen adjacent memory cells or bus lines experience a common failure like a short-circuit), then the

received message will be 00010 and this error is detected (since only one bit is a 1).Suppose that now the first three bits are all tied high, then the received message will be 11110 andthis error is also detected (since four bits are now 1).

4.14.4 Checksum Codes

Checksum codes are generated by appending an n-bit checksum word to a block of sn-bit datawords, formed as the sum of thesn-bit words using modulo 2n addition.Encoding: use ann-bit arithmetic adder to add thesn-bit data words, with any carry beyond thenth

bit being discarded. The sum is then appended to the data block as then-bit checksumDecoding: use the samen-bit arithmetic adder to add thesn-bit data words and XOR the sum withthe appended checksum, if the result is zero then there is no error.


43/58



Properties Simple and inexpensive, hence large number of uses Single error detection at least Multiple error detection dependent on s and columns in error: error coverage is highest for the

least significant bits

Error coverage can be controlled by weighting the bits or using extended checksums Long error latency (must wait for end of data block) Low diagnostic resolution (error can be anywhere in thesn-bit words) Uses: sequential storage devices, block-transfer peripherals (disks, tapes), read-only memory,

serial numbering, etc.

Example 4.29

Let n=8 and s=8, then we append an 8-bit checksum, Ck, after every block of 8x 8-bit data words

(bytes), wi, calculated as follows: =

=8

1

8k 2modwC

i

i . Consider the following data block (in hex):

01 FA 02 32 62 A1 22 E3

The sum (in hex) =337 and Ck =337(hex) mod 256 =37(hex)(i.e., we simply retain the low-order 8 bits of the sum and discard the rest)

The multiple error coverage can be improved by using an extended checksum ofmbits for n-bitdata wherem>nand the sum is modulo 2maddition (i.e. we retain the low-order mbits of the sum).As an examplen=8 andm=16 and a 2-byte checksum is formed from the sum ofsdata bytes in theblock.

Real-world Checksum examples

International Standard Book Number (ISBN)ISBN 0 13 283796 X

Country

USA

Publisher

(Prentice Hall)Book ID

Check symbol

(X is for 10)

Check symbol (a10 =X =10) is derived such that =

1

1011a

i

ii is divisible by 11, where ak is the kth

digit from the left in the ISBN code. For the above example we see that:Sum =(i =10)*(a1 =0) + (i = 9)*(a2 = 1) +8*3 + 7*2 + 6*8 + 5*3 + 4*7 + 3*9 + 2*6 +(i =1)*(a10 =X =10) =187 which is divisible by 11

UWA Student Number

The last digit of the 7 digit student number, s7, is the check digit derived such that:

Sum=(8*s1 +4*s2 +6*s3 +3*s4 +5*s5 +2*s6 +s7) is divisible by 11


44/58



4.15 Cyclic Codes

4.15.1 Introduction

The problem with checksum codes is that simply summing bits to form a checksum (or check bits)does not produce a sufficiently complex result. By complex we mean that the input bits shouldmaximally affect the checksum, so that a wide range of bit errors or group of bit errors will affectthe final checksum and be detected. A more complex checksumming operation can be formed byperforming division rather than summation. Furthermore the remainder not the quotient is used, asthe remainder gets kicked about quite a lot during the calculation, and if more bytes were added tothe message (dividend) its value could change radically again very quickly.

In the previous sections the idea of the parity-check matrix, check-bit generation and syndrome

decoding as a means for efficient design and implementation of linear codes was shown from thepoint of view of binary matrix linear algebra. The cyclic codes are a subclass of linear codes whichsatisfy the following two properties:

Cyclic codes

Linearity property: the sum of any two codewords is also a codewordCyclic property: any cyclic shift of a codeword is also a codeword

Example 4.30

Consider the word: 01001100One cyclic shift to the right: 00100110Another: 00010011Yet another: 10001001

If the code is a cyclic code all of the above would be codewords.

In a systematic cyclic code the check-bits are generated as the remainder of a division operation andas indicated this provides a more powerful form of checksumming. The cyclic codes are analysednot by representing words as binary vectors and using matrix binary linear algebra but byrepresenting words as binary polynomials and using polynomial arithmetic modulo 2. Like linearcodes the cyclic codes also permit efficient implementations of the encoding and decodingoperations, especially fast logic implementations for serial data transmission. The properties ofcyclic codes also allow greater scope for designing codes for specialised uses, especially in handlingburst errors. Not surprisingly all modern channel coding techniques use cyclic codes, especially theimportant class of Cyclic Redundancy Check (CRC) codes for error detection.


45/58



4.15.2 Code polynomials and the cyclic property

Let thek-bit information or message word be represented as:

m0m1 mk-2mk-1

wherem0 is the MSB (most significant bit) andmk-1 is the LSB (least significant bit).

Let then-bit code word be represented as:

c0c1 cn-2cn-1

wherec0 is the MSB andcn-1 is the LSB.

And let the (n-k)-bit check-bits be represented as:

b0b1 bn-k-2bn-k-1

We assume the code is systematic hence:

c0c1 cn-k-2cn-k-1cn-kcn-k+1 cn-2cn-1 b0b1 bn-k-2bn-k-1m0m1 mk-2mk-1

Same reversed notation as used by Communication Systems by S. Haykin

We are adopting the same notation that the textbook by S. Haykin uses. Compared to the previousdefinition inSection4.7, there are two major differences:1. The indexing sense has been reversed, and indexing starts at 0 not 1.2. The codeword representation is now reversed. The bits are now running to the right.

Previously the 7-bit codeword 1101110implied a message of 1101 followed by the check-bit

sequence 110, all running to the left: Now the same 7-bit codeword is bit reversed to

0111011 which implies a message of 1011 followed by the check-bit sequence 011, allrunning to the right: In both cases the message and check-bits are actually identical.

We denote the column vectors using this reversed notation with a , that is:

]0111011[]1101110[ TT == rr

For a codeword of lengthnwe define thecode polynomial, C(x), as:

=

=1

0

)C(n

i

iixcx

We can similarly define themessage polynomial, M(x) is formed as:

=

=1

0

)M(k

i

iixmx

and let:

=

=1-

0

)B(kn

i

iixbx

The for a systematic code we see that:


46/58



C(x) =B(x) +xn-kM(x)

Equation 4.14Let C(j)(x) representj cyclic shifts to the right of C(x), it can be shown that:

xjC(x) =Q(x)(xn +1) +C(j)(x)Equation 4.15

That is:C(j)(x) =xjC(x) mod (xn +1)

Equation 4.16

That is the remainder when dividing xjC(x) by (xn +1) represents the polynomial of the codewordcyclically shifted to rightj times.

NOTE:Remember we are using mod-2 binary arithmetic so: xi xj"xi +xj ; xi +xi =0 and 1 +1 =0

Example 4.31

Consider the codeword: 010 01100 withn=8 andk=5.The left-most bits 010 are the check-bits and hence:

xxxx =++= 2010)(BThe right-most bits 01100 are the message bits and hence:

2432 00110)(M xxxxxxx +=++++=The codeword polynomial is then:

54233

54765432

)()(M)(B)(C

00110010)(C

xxxxxxxxxxx

xxxxxxxxxxx

++=++=+=

++=+++++++=

Say C(x)

01001100 is now cyclically right shifted by 3 bits, then we get C(3)(x)

10001001, andhence:

74)3(54 1)(C)(C xxxxxxx ++=++=Now:

8743 )(C xxxxx ++=

And:

)1(mod)(C 83 +xxx !1

11

1

47

8

4788

+++

+++

xxx

xxxx

=C(3)(x)

Let:

)(R)1(mod)(C 83 xxxx =+where R(x) is the remainder, then:

)(R)(Q)1()(C 83 xxxxx ++=

where Q(x) is the quotient. From the above results we have that:

Q(x) =1 and R(x) =x7 +x4 +1

)1(1)1()(C 4784783 ++++=++= xxxxxxxx


47/58



Now consider the (n-k+1) bit word represented by:

g0g1 gn-k-1gn-k

where, by definition, g0 =1 andgn-k =1 The corresponding polynomial is known as the:

Generator Polynomial

knkn

i

ii xxgx

=++=

1-

11)G(

Equation 4.17

We choose G(x) to be a factor ofxn +1 (i.e. xn +1 =G(x)H(x), where H(x) is known as theparity-check polynomial) and form all words of the form C(x) =A(x)G(x) as code polynomials belongingto the code defined by G(x). The code so defined is obviously linear, but is it cyclic? That is, will acyclic shift of C(x) yield a polynomial which can be defined by, say, A2(x)G(x) (i.e. C

(j)(x) =A2(x)G(x) for some A2(x)) and thus belong to the code? UsingEquation 4.15with the appropriatesubstitutions (C(x) =A(x)G(x) and (xn +1) =G(x)H(x)), we have:

xjA(x)G(x) =Q(x)G(x)H(x) +C(j)(x)

Dividing throughout by G(x) we get:

)(A))H(Q()A()G(

)(C

)G(

)(C))H(Q()A(

2

)(

)(

xxxxxx

x

x

xxxxx

jj

jj

=+=

+=

and hence C(j)(x) =A2(x)G(x). Thus:

Result 4.6

If the degreen-k polynomial G(x) as defined by Equation 4.17is a factor of (xn +1), then all codepolynomials (i.e. codewords) formed as the product A(x)G(x), where A(x) is any degree k-1polynomial, belong to the (n,k) cyclic code defined by G(x).

NOTE

1.The degree of the polynomials encountered so far are: C(x) is a polynomial of degreen-1 M(x) is a polynomial of degreek-1

G(x) is a polynomial of degreen-k B(x) is a polynomial of degreen-k-1

2. The product of a polynomial of degree a with a polynomial of degree b is a polynomial ofdegreea+b. Hence A(x) is a polynomial of degreek-1 sincen-1 =(k-1) +(n-k).

3. Since A(x) is a polynomial of degree k-1, there are 2k possible A(x) and hence 2k possible codepolynomials.

4. G(x) is governed only by the degree (n-k) and the fact that it must be a factor of (xn +1), thus forthe same G(x) we may be able design different codes for different appropriate values ofn.

In the same way that the parity-check matrix, H, was used to define a linear code the generatorpolynomial, G(x), is used to define a cyclic code.


48/58



4.15.3 Encoding/Decoding cyclic codes (algebraic method)

By making A(x) =M(x) we simply derive C(x) =M(x)G(x), but this will not give us a systematic(i.e. separable) code. For a systematic code we must have:

C(x) =A(x)G(x) =B(x) +xn-kM(x)

Hence dividing the above through by G(x):

)G(

)B()A(

)G(

)M(

x

xx

x

xx kn+=

Thus B(x) is the remainder left when dividing xn-kM(x) by G(x).

Encoding Systematic Cyclic Codes

1.Express the message word in polynomial form, M(x)2.Formxn-kM(x).3.Dividexn-kM(x) by G(x) to find the remainder B(x).

4.Form the codeword polynomial, C(x) =B(x) +xn-kM(x)5.Convert C(x) to binary form to form the codeword.

Example 4.32

Consider G(x) =1 +x +x3 with n=7 and k=4 (we will later see that this G(x) produces a (7,4)Hamming code). Thus C(x) =x3M(x) +B(x), where B(x) is the remainder left when dividing x3M(x)by G(x). The following are examples of encoding systematic cyclic codes algebraically for themessages 1000, 1010 and 1011:

Message

m0m1m2m3

M(x) x3M(x) B(x) C(x) Codeword

b0b1b2m0m1m2m31 0 0 0 1 x3 1+x 1+x+x3 1 1 0 1 0 0 01 0 1 0 1+x2 x3+x5 x2 x2+x3+x5 0 0 1 1 0 1 01 0 1 1 1+x2+x3 x3+x5+x6 1 1+x3+x5+x6 1 0 0 1 0 1 1

B(x) for Message 1000)(Q

)(B

)(M)(G 3x

x

xxx

*

!

1

1

1

1

3

33

+

++

++

x

xx

xxx

B(x) for Message 1010

)(Q

)(B

)(M)(G 3x

x

xxx

*

!

2

2

235

353 1

x

x

xxx

xxxx

++

+++

B(x) for Message 1011


49/58



)(Q

)(B

)(M)(G 3x

x

xxx

*

!

1

1

1

1

23

3

3

24

234

235

45

346

3563+++

++

+

++

++

++

+

++

++++xxx

xx

xx

xxx

xxx

xxx

xx

xxx

xxxxx

For decoding consider R(x) as the received word polynomial of degree n-1. Let S(x) denote theremainder of dividing R(x) by G(x). Since C(x) =A(x)G(x) then if there are no errors we expect R(x)=C(x) and hence no remainder (i.e. S(x) =0). If S(x) 0 then we know there is an error. S(x) iscalled thesyndrome polynomial.

How is S(x) related to E(x), theerror polynomial?

We have:

R(x) =C(x) +E(x)Now:Q(x)G(x) +S(x) =A(x)G(x) +E(x)

where Q(x) is the quotient and S(x) is the remainder when R(x) is divided by G(x). Then:

E(x) =(Q(x) +A(x))G(x) +S(x)

Hence the syndrome can be obtained as the remainder of dividing the error polynomial, E(x), byG(x) (i.e. S(x) =E(x) mod G(x)). In the same way the syndrome was dependent only on the errorpattern for linear codes the syndrome polynomial is only dependent on the error polynomial forcyclic codes. Thus the syndrome polynomial can be used to find the most likely error polynomial.

Decoding Cyclic codes

1.Express the received word in polynomial form, R(x).2.Divide R(x) by G(x) to find the remainder, S(x)3. If S(x) =0, there is no error.4. If S(x) 0 then an error has occurred and the coset leader error polynomial for S(x) is used to

correct the error.


50/58



Example 4.33

Consider the cyclic code from Example 4.32 with G(x) = 1 + x + x3. Say the codeword

]0011010[T =c is transmitted, but ]0011110[T =r is received (an error in bit r4). To decode weform:

R(x) =x2 +x3 +x4 +x5

and divide R(x) by G(x) to yield the remainder S(x):

)(Q

)(S

)(R)(G

x

x

xx

*

!

xx

xx

xxx

x

xxx

xxxxxx

+

+

++

++

+++++

2

2

24

4

235

23453 1

Thus S(x) =x2 +x" ]011[T =s , and since S(x) 0 an error is detected and syndrome decodingcan be used to attempt to correct the error.

NOTE

1.The generator polynomial, G(x), can be shown to be related to the generator matrix, G, and theparity-check polynomial, H(x), can be shown to be related to the parity-check matrix, H.

2.The Hamming codes are cyclic codes! Specifically a (7,4) Hamming code is designed with

G(x) =1 +x+x

3

. Since G(x)H(x) =x

7

+1, the corresponding H(x) =1 +x+x

2

+x

4

can be used toconstruct the (7,4) Hamming code parity-check matrix.

3.S(x) is a polynomial of degreen-k-1, E(x) is a polynomial of degreen-1.

Exercise: Verify that the codewords generated with G(x) as defined above indeed produce a (7,4)Hamming code

4.15.4 Deriving the Gen

unencrypted 4

Documents