Download - Unencrypted 4
-
7/29/2019 Unencrypted 4
1/58
Information Theory and Coding 314Dr. Roberto Togneri
Dept. of E&E EngineeringThe University of Western Australia
Section 4Error-Control Codes
(Channel Coding)
DISCLAIMER
Whilst every attempt is made to maintain the accuracy and correctness of these notes,the author, Dr. Roberto Togneri, makes no warranty or guarantee or promise expressor implied concerning the content
-
7/29/2019 Unencrypted 4
2/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-2
4. ERROR-CONTROL CODES (CHANNEL CODING)
4.1 Introduction
In the previous chapter we considered coding of information sources for transmission through anoiseless channel. In that case efficient compact codes was our primary concern. In general,though, the channel is noisy and errors will be introduced at the output. In this chapter we will beconcerned with codes which can detect or correct such errors. We restrict our attention to the classof binary codes and BSC channels. Our model for a noiseless channel was:
SourceEncoder
SourceDecoder
Modulator
Digital ChannelDemodulator
message
no errors
message
In the presence of noise we include an additional channel decoder/encoder. This allows us toseparate source representation (using compact codes) from error protection (using error-controlcodes).
Properties of Channel Coding
1.The channel encoder takes the binary stream output of the source encoder, and maps each fixed-block messageor informationwordof length L bits, to a fixed-block codewordof length N. Thuschannel coding usesblock codeswith N >L. The difference, N-L, represents thecheck-bitsused
for channel coding.
2.We restrict ourselves to binary channels. Hence the source encoding uses a binary code and thedata entering and leaving the channel encoder, channel decoder and channel is binary.
3.With L bits of information we need M =2L codewords, each N-bits in length. Thus we willreceive either one of the M =2L codewords or any one of the remaining 2N-2L words whichrepresent the codewords with detectable bit errors. It is this mapping operation and the extra N-L check bits which can be exploited for error-control.
Our model for a noisy channel is:
SourceEncoder
SourceDecoder
Modulator
Digital ChannelDemodulator
ChannelEncoder
ChannelEncoder
L bitmessage
N bitcodeword
N bitword
L bitmessage
NOISE
bit errors no errors
feedbackchannel error detected?
-
7/29/2019 Unencrypted 4
3/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-3
NOTE
1.With error correction the error is detected and corrected at the receiver. A communication systemwhich implements this form of channel coding is known as a Forward-Error-Correction (FEC)error-control system.
2.With error detection the error is detected and the transmitter is required retransmit the message.This requires a reliable return feedback channel from receiver to transmitter. The simplestimplementation is a stop-and-wait strategy whereby the receiver acknowledges (ACK) eachcorrect reception and negatively acknowledges (NACK) each incorrect reception. The transmitterthen retransmits any message that was incorrectly received A communication system whichimplements this form of channel coding is known as a Automatic-Repeat-Request (ARQ) error-control system.
3. In packet-switched data networks the, possibly, source-coded messages are transmitted in N-bit
codeword blocks called frames, packetsor segments (depending on which network layer handlesthe data) which incorporate not only the message but a control header and/or trailer. ARQ errorcontrol is used based on systematicerror-detecting channel codes with the check-bits included inthe appropriate checksum field of the header or trailer.
4.2 Information Rate (fixed-rate source coding)
It should be obvious that since N >L the rate of data entering the channel is greater than the rate ofdata entering the channel encoder. Let:
(bps)ratebitchannel1
(bps)ratebitcodingsourcefixed1
c
s
=
=
c
s
n
n
It should be evident that: [ ] [ ]sccs nn /LNor/LN == Equation 4.1
NOTE: If the source coder and channel bit rates are such that sc nn /L is not on integer, then a
dummy bit will occasionally have to be transmitted over the channel.
Channel Coding Information Rate
c
s
s
c
n
nR ===
N
L
Equation 4.2
Channel Code Redundancy:L
N1==
RCred
Equation 4.3
Excess Redundancy: L
L-N
1
1
== RE
red
Equation 4.4
-
7/29/2019 Unencrypted 4
4/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-4
Example 4.1
(a)_The source coder rate is 2 bps and the channel bit rate is 4 bps. Say L = 3, then
6]2/4[3]/[LN === sc nn and the channel coder will generate N-L =3 check bits for every L =3-bit message block and hence produce a N =6-bit codeword. The check bits can be used for error-
control (e.g. even parity-check)
(b)_
The source coder rate is 3 bps and the channel bit rate is 4 bps. Now L3
4N
= .
If L =3 ! N =4 (1 check bit for every 3-bit message block to produce a 4-bit codeword)
If L =5 ! 63
26N = (1 check bit for every 5-bit message block to produce a 6-bit codeword)
BUT there is a mismatch between the source code and channel data rates. Express the fraction as the
ratio of the two small integers dandn:nd=
32 thusn=3 andd=2. To synchronise the data rates
we need to transmitd=2 dummy bits for everyn=3 codewords / message blocks:
L =5, n=3" X X X X X X X X X X X X X X X =15 bitsN =6, d=2" + + + + + + + + + + + + + + + + + + ** =20 bits
Thus the average rate will be the correct (20/15) =(4/3). Both the check-bits +and dummy bits *can be used for error-control.
4.3 Decis ion Rules and Channel Decoding
With a noiseless channel we know with probability 1 which input was transmitted upon reception ofthe output. However, when the channel is noisy there is usually more than one possible inputsymbol that may have been transmitted given the observed output symbol. In the presence of noisewe must use a decision rule.
Decision Rule
Consider a channel with anr-symbol input alphabet:
A ={ai}, i =1,2 ..., r
and ans-symbol output alphabet:
B ={bj}, j =1,2 ..., s.
The decision ruled(bj) is any function specifying a unique input symbol for each output symbol, bj.
The decision ruled(b) is any function specifying a unique input word for each output word, b.
-
7/29/2019 Unencrypted 4
5/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-5
Example 4.2
Consider the channel matrix:
3
2
1
INPUT
a
a
a
4.03.03.0
5.03.02.0
2.03.05.0
OUTPUT
321
!! "!! #$bbb
Possible decision rules are:
Rule 1:"
33
22
11
)(
)(
)(
abd
abd
abd
===
and Rule 2"
23
22
11
)(
)(
)(
abd
abd
abd
===
With the second decision rule if 2311 bbbbb= then 22112311 )()( aaaabbbbdbd ==
The aim of a good decision rule is to minimise the error probability of making a wrong decision.
4.3.1 Minimum-Error Probability Decoding
We choose*)( abd j =
wherea* is given by:
P a b P a b ij i j( * / ) ( / ) Equation 4.1
Now From Bayes' law we have:
)(
)()/(
)(
*)(*)/(
j
iij
j
j
bP
aPabP
bP
aPabP
Equation 4.2SinceP(bj) is independent of ai inEquation 4.2we have:
)()/(*)(*)/( iijj aPabPaPabP
Equation 4.3
Thus the minimum-error probability decoding requires not only the channel matrix probabilities butalso the a priori probabilitiesP(ai). In some cases we do not have knowledge ofP(ai).
-
7/29/2019 Unencrypted 4
6/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-6
4.3.2 Maximum-Likelihood Decoding
We chose*)( abd j =
wherea* is given byP b a P b a ij j i( / *) ( / )
Equation 4.4
Thus maximum-likelihood decoding depends only on the channel. This is an advantage in situationswhere theP(ai) are not known. Although not minimum-error, maximum - likelihood decoding does
result in minimum-error decoding when the inputs are equiprobable.
4.3.3 Error probability
If we choose the input according to the decision ruled(bj) then the probability of getting it right is:
P a d b bj j( * ( ) / )=
and the probability of error:
P E b P a bj j( / ) ( * / )= 1
P(E/bj) is the conditional error probability given the outputbj.
MaximisingP(a*/bj) will minimise the error, hence the name minimum-error probability decoding!
The overall probabilityPE is the average ofP(E/bj) over all thebj.
=B
jjE bPbEPP )()/(
We can equivalently define:
=A
iiE aPaEPP )()/(
Equation 4.5and
= cBb ii abPaEP )/()/(Equation 4.6
whereb Bc is the summation over members of theB alphabet for whichd(bj) ai.
-
7/29/2019 Unencrypted 4
7/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-7
Example 4.3
Refer to same channel ofExample 4.2. Also assume we are given thatP(a1) =0.3, P(a2) =0.3 andP(a3) =0.4.
Maximum Likelihood DecodingThis requires finding the maximumP(bj/a*) for the given bj, that is the maximum value along the
column of the channel matrix )/( ij abPP corresponding tobj:
)/( ij abP
3
2
1INPUT
a
a
a
4.03.03.0
3.02.0
2.0
OUTPUT
321
0.5
0.30.5
!! "!! #$bbb
!
23
12
11
)(
)(
)(
abd
abd
abd
===
Note thatd(b2) =a2 =a3 are also correct. We calculate the error probability using:
==cBb
i
A
i
A
iiE abPaPaPaEPP
)/()()()/(
To derive
=CBb
ii abPaEP )/()/( we sum across theai row excluding thebj column ifd(bj) =ai:
3
2
1INPUT
a
a
a
% % &'(
!! "!! #$
61.0)()/(
40.04.00.1
15.03.05.0
06.03.02.0
4.03.03.0
3.02.0
2.0
OUTPUT
321
=
===
EP
iaPiaEP
0.5
0.30.5
bbb
ThusPE =0.61. Note thatP(E/a3) =1.0 is an obvious result since the decision rule never decides a3.
Minimum Error DecodingThis requires finding the maximumP(bj/a*)P(a*) for the givenbj, that is we need to know the inputprobabilities. We form the augmented channel matrix which is pre-multiplied by P(ai) and then find
the maximum value along the column of the augmented channel matrix )()/( iij aPabPP corresponding tobj:
)()/( iij aPabP
3
2
1INPUT
aa
a
0.160.12
0.15
12.015.009.006.0
06.009.0
OUTPUT
321
!!! "!!! #$bbb
!
33
32
11
)(
)(
)(
abd
abd
abd
=
==
And we calculate the error probability as before:
3
2
1INPUT
a
a
a
% % &'(
!! "!! #$
57.0)()/(
12.04.03.0
30.03.00.1
15.03.05.0
3.0
5.03.02.0
2.03.0
OUTPUT
321
=
===
E
P
i
aP
i
aEP
0.40.3
0.5
bbb
ThusPE =0.57 which is lower than maximum likelihood decoding.
-
7/29/2019 Unencrypted 4
8/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-8
Decision Rule for modern BSCs and typical source coders
The following will typically be the case:1.q
0 whenever ab
2. d(a,b) =d(b,a)
3. d(a,b) +d(b,c) d(a,c) (triangle inequality)
Example 4.4
Letn=8 and consider:
11001010
0100100010001011
=
==
c
ba
!
2),(
2),(
4),(
=
==
cad
cbd
bad
, and ),(),(),( cadcbdbad +
-
7/29/2019 Unencrypted 4
9/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-9
4.4.1 Decoding Rule using the Hamming Distance
As stated, for modern communications systems the maximum likelihood decoding rule will yieldthe same result as the minimum error decoding rule. When considering the decision rule for
deciding which N-bit codeword a was sent given the received N-bit word b, then a is chosen tomaximiseP(b/a). Say we transmit the N-bit codeword a and observe the N-bit word bat the output(of the BSC). Assume d(a, b) =D, then this means that a and b differ in exactly D places, thus Dbits are in error (with probabilityq) and N-D bits are OK (with probabilityp=1 - q). Thus:
( ) ( ) ( ) DND/ = pqabP
Sinceq t
Property 4.1
ProofLet a* be the codeword that is sent and b the received word. Assume there are t bit errors intransmission thend(a*,b) =t. To detect thatb is in error then it is sufficient to ensure that bdoes notcorrespond to any of theai codewords, i.e. d(b, ai) >0 for all i. Using the triangle inequality we have
for any other codewordai: )*,()*,(),()*,(),()*,( badaadabdaadabdbad iiii +To ensure d(b, ai) > 0 we must have: )*,()*,(0)*,()*,( badaadbadaad ii >> and sinced(a*,b) =t we get the final result that:
taad i >)*,( !d(K) >tand this is the condition for detecting up to all t-bit errors
Error CorrectionA block code K is said to correct t errors provided that for each code word a and each word bobtained by corrupting t bits in a, the maximum likelihood decoding leads uniquely to a. Thisproperty is important in situations where the communication system uses FEC error-control coding.
Error Correcting Property
A code corrects all terrors if and only if its minimum distance is greater than 2t:d(K) > 2t
Property 4.2
-
7/29/2019 Unencrypted 4
11/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-11
ProofLet a* be the codeword that is sent and b the received word. Assume there are t bit errors intransmission thend(a*,b) =t. To detect that b is in error and ensure maximum likelihood decodinguniquely yieldsa* (error can be corrected) then it is sufficient that d(a*,b) d(a*,b) we must have:
)*,(2)*,()*,()*,()*,( badaadbadbadaad ii >>and sinced(a*,b) =t we get the final result that:
taad i 2)*,( > !d(K) >2tand this is the condition for correcting up to all t-bit errors
Example 4.6
Even-parity check Code
The even-parity check code has N =L +1 by adding a parity-check bit to the L-bit message. For thecase L =2 we have:Message Codeword
00 00001 01110 10111 110
By comparing the Hamming distance between different codewords we see thatd(K) =2 (i.e. no twocodewords are less than 2 distance from each other) and by Property 4.1d(K) =2 >t =1 and as
expected this code will be able to detect all single-bit errors.
Repetition CodeThe repetition code is defined only for L =1 and performs error correction by repeating the messagebit (N-1) times (for N odd) and then using a majority vote decision rule (which is the same as theHamming distance decoding rule). Consider the N=3 repetition code:
Message Codeword0 0001 111
We see thatd(K) =3. FromProperty 4.1d(K) =3 >t=2 and this code can detect all 2-bit errors.Furthermore fromProperty 4.2d(K) =3 >2(t=1) and this code can correct all 1-bit errors. TheHamming decoding rule will perform 1-bit error correction. To see this and the majority votingoperation:
Non codeword Closestcodeword
Majoritybit
Action
001 000 0 message 0010 000 0 message 0011 111 1 message 1100 000 0 message 0101 111 1 message 1
110 111 1 message 1Note: Using majority voting of the received bits is a much simpler implementation for decoding thefinal message than using the Hamming distance decoding rule.
-
7/29/2019 Unencrypted 4
12/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-12
Code fromExample 4.5Message Codeword
00 00001 00110 01111 111
We see that d(K) =1 (e.g. distance between codewords 000 and 001) which would imply this codecannot not even do single-bit error detection let alone error correction. In Example 4.5 theHamming distance decoding rule was either performing 1-bit error detection or 1-bit errorcorrection. The explanation is that this code can detect most but not all 1-bit errors, specifically ifcodeword 000 is sent and there is a bit error in b3 we receive 001 and decode this to codeword 001(this is known as an undetected error as there is no way to detect such errors).
Code K6Code K6 will be used in a later example and is defined as follows
Message
(L =3)
Codeword
(N =6)000 000000001 001110010 010101011 011011100 100011101 101101110 110110111 111000
By comparing codewords we have thatd(K) =3 and hence this code can correct all 1-bit errors. Forexample, ifa=010101 is sent and b=010111 is received than the closest codeword is 010101 andthe error in b5 is corrected. This will always be the case no matter which codeword was sent andwhich bit is in error. It should be noted that there are only M =2L =23 =8 valid codewordscompared to 2N =26 =64 possible received words.
4.4.3 Upper boundB(N,d(K)) on number of codewords
Thed(K) for a particular block code K specifies the codes error correction/detection properties. Say
we want to design a code of length N and minimum distance d(K). What is the bound,B(N,d(K)) onM (and hence L), for any given (N, d(K)), i.e. findB(N,d(K) such that M B(N,d(K))?
This is sometimes known as the main coding theory problem (of finding good codes).Some results for B(N,d(K)) are:
B(N, 1) =2N
B(N, 2) =2N-1
B(N, 3) =2
N +1
N
B(N, 4) =2
N
N-1
-
7/29/2019 Unencrypted 4
13/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-13
In general, we have:
B(N, 2t+1) =B(N+1, 2t+2) for t=0,1,2,
Equation 4.7
thus if the we know the bound for evend(K) we can calculate the bound for oddd(K).
For evend(K) one bound (of the apparently many) is the Plotkin bound for evend(K):
N-(K)2
(K)2=(K))(N,
d
ddB providedd(K) is even and 2d(K) >N d(K)
Equation 4.8
Example 4.7
Our channel coder encodes messages of L =2 bits (M =4). What is the minimum block length, N,and possible code for the following requirements:
1-bit error detection
We needd(K) >t=1!d(K) =2, so M =4B(N,2) =2N-1! N =3 (smallest N)We need to design a code with L =2, N =3 and ensured(K) =2, so here it is:
Info Code00 00001 01110 10111 110
The even-parity check code!
1-bit error correction
We needd(K) >2(t=1) ! d(K) =3, so M =41N
2)3,N(N
+=B ! N =5 (smallest N)
We need to design a code with L =2, N =5 and ensured(K) =3, so here it is:Info Code00 0000001 0101110 1011111 11100
2-bit error detection
We need d(K) >(t =2) ! d(K) =3 ! same code as 1-bit error correction except that thedecoding rule is simply to indicate an error ifbai for all i.
2-bit error correction
We needd(K) >2(t=2) =4 ! d(K) =5, soB(N,5) =B(N+1,6) =N11
12
)1N()K(2
)K(2
=
+dd
using the Plotkin bound, and M =4N11
12
! N =8. We need to design a code with L =2, N =
8 and ensured(K) =5. Fun!
-
7/29/2019 Unencrypted 4
14/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-14
4.4.4 Issues
The problem with using the Hamming distance or encoding and decoding is that with L bitinformation words we generate 2L code words which we must store, and perform message word to
code word comparisons, which is an expensive operation. This represents an exponential increase inmemory and computational power with L. Only in the specific case of the repetition code is anefficient implementation (using majority voting) possible. Later we will develop a way ofsystematically encoding and decoding codes in an efficient manner by considering the importantclass ofbinary linear codes.
4.5 Block Error Probability and Information Rate
Let: Kn = block code of lengthn(code withn N-bit codewords)Pe(Kn) = block error probability for code KnPeb(Kn) = bit error probability for code KnR(Kn) = information rate for code Kn
The bit error probability Peb(Kn) indicates the probability of bit errors between the transmittedmessage and decoded message. This measures the true errors in using Kn. The block errorprobability Pe(Kn) indicates the probability of decoding to the incorrect codeword block. It shouldbe noted that Pe(Kn) Peb(Kn) since even with an incorrect codeword block not all of the bits willbe in error.
Example 4.8
Consider a channel coding with L=3 and N=6. The 3-bit information sequence 010 is encoded as010111 and sent through the channel. However due to a bit errors the received word is 011101 andthe channel decoder decodes this as codeword 011111 and message 011. Although the block errordecoding is 100% in error (i.e. Pe(Kn) =1.0) the first 2 bits are OK so the bit error decoding is only33% incorrect (i.e. Peb(Kn) =0.33).
When comparing different codes it is normal to usePe(Kn) rather thanPeb(Kn), sincePe(Kn) is easierto calculate thanPeb(Kn) andPe(Kn) represents the worst-case performance.
What is the relationship betweenPe(Kn) andR(Kn)?
Example 4.9
Consider a BSC withq=0.001 thenPeb(Kn) =0.001. How do we improve this and what is the pricethat we pay?
(n =3) repetition codeInfo Code
0 0001 111
We note that since L =1 thenPeb(Kn) =Pe(Kn). How do we calculatePe(Kn)? The majority vote andHamming distance decoding rule will fail to yield the correct codeword if: all n=3 bits are in error q3
-
7/29/2019 Unencrypted 4
15/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-15
2 out ofn=3 bits are in error pqpq 22 323 =
Hence:Pe(Kn) =q
3 +3q2p=3 x 10-6much better thanq=1 x 10-3, but there is a cost sinceR(Kn) =1/3
(n =5) repetition code
Info Code0 000001 11111
The majority vote and Hamming distance decoding rule will fail to yield the correct codeword if: all n=5 bits are in error q5
4 out ofn=5 bits are in error pqpq 44 545 =
3 out ofn=5 bits are in error 2323 1035
pqpq =
Hence: Pe(Kn) =q5 +5q4p+10q3p2 =10-8, but nowR(Kn) =1/5 !
With repetition codesn
R n1
)K( = and there is an exchange of message rate for message reliability.
But is there a way of reducingPe(Kn) without a corresponding reduction inR(Kn)? Yes!
Example 4.10
Consider a BSC withq=0.001 and the restriction thatR(Kn) =0.5. We design two codes K4 and K6in an attempt to lower Pe(Kn) and still keepR(Kn) =0.5
Code K4 (N =4, L =2)We transmita1 once and transmita2 three times to produce the following code:
I nfo Code00 000001 011110 100011 1111
We perform conditional error correction by assumingb1 is correct (a1 =b1) and then do a majorityvote on the remainingb2b3b4 bits to obtaina2. This decoding will be correct when: all n=4 bits are correct p4
b1 is correct and there is one error in the remaining 3 bits qpqp 33 313 =
Hence: 1 - Pe(Kn) =p4 +3p3q!Pe(Kn) =1 - p
4 - 3p3q=0.001
But this is indeed better if we consider Peb(Kn). In the following, let c mean correct and i meanincorrect. We can derive :Pci =Pr(a1 =c, a2 =i) q(3p
2q+p3) =! first bit only is incorrect (only 1 of the 2-bits is wrong)
Pic =Pr(a1 =i , a2 =c) =p(q3 +3pq2)! second bit only is incorrect (only 1 of the 2-bits is wrong)
Pii =Pr(a1 =i , a2 =i) =q(q3 +3pq2)! both bits are incorrect
Pcc =Pr(a1 =c , a2 =c) =p(3p2q+p3)! both bits are correct (decoding is correct)
-
7/29/2019 Unencrypted 4
16/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-16
And over a message with L =2 bits we can show that:
( ) ( )3223 32
322
)(0)(1)(2)K( pqp
qpqqq
pPPPPP ccicciiineb +++
=
+++=
Hence for q=0.001: Peb(Kn) =0.0005 1 bit errors (Show This!).Hence: Pe(Kn) =1 - p
6 - 6p5q=0.000015
-
7/29/2019 Unencrypted 4
17/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-17
channel by introducing redundancies. Shannon's Fundamental Theorem will tell us that for largeenough N, ifRC we can encode for error-free transmission:Shannons's Fundamental Theorem
Every binary symmetric channel of capacity C >0 can be encoded with an arbitrary reliability andwith information rate, R(Kn) C, arbitrarily close to C. That is, there exist codes K1, K2, K3 .such thatPe(Kn) tends to zero andR(Kn) tends toC with increasingn:
CRPn
en
==
)(Klim0,)(Klim nn 0limand0,)(Ki.e. 111 =>= nn CR
That is, IfR(Kn) C, thenPe(Kn) 0 for large enoughn, and as a bonusR(Kn) C!
4.6.1 Engineers Proof
The formal proof of Shannon's Theorem is quite lengthy and involved. We only present the salientfeatures of the proof here (which is why this is an engineers proof# ).
Assume an arbitrarily small number >1 0and that we have to design a code Knof lengthnsuchthat ( ) .K 1= CR n To do this we need an information length of )(L 1= Cnand hence:
11)()(K =
= C
n
CnR n
This means we need)1(2M
=
Cncodewords.
Shannon's proof makes use of random codes. Given any number nwe can pick M out of the 2n
binary words in a random way and we obtain a random code Kn. If)1(2M
=
Cnthen we know
that ( ) 1K = CR n butPe(Kn) is a random variable.
Denote:
))(KE(~
nee PP =as the expected value ofPe(Kn) for a fixed value ofn, but a completely random choice of Mcodewords. The main, and difficult part, of the proof (which we conveniently omit) is to show that:
as0~ nPeThus given arbitrarily small numbers 1>0 and 2>0 we can findnsuch that 2
~
-
7/29/2019 Unencrypted 4
18/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-18
Converse of Shannon's Theorem
In every binary symmetric channel of capacity C whenever codes Kn of lengths nhave information
rates at leastC +1 (1 >0) then the codes tend to be totally unreliable:( ) ( ) 1KK lim1 =+ nenn PCR
That is, IfR(Kn) >C, thenPe(Kn) 1 for large enoughn.
Example 4.11
Consider the BSC fromExample 4.10. With q =0.001 we know that:9886.0loglog1 =++= qqppC
We can now answer the4.5.1Issuesquestions:1.SinceR=0.5
-
7/29/2019 Unencrypted 4
19/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-19
4.7.2 Binary Linear Codes (Take 1)
Change in notation
n=N length of block code (entering the channel)k=L number of information / message bits in each block (entering channel encoder).
An important class of binary codes are the binary linear codes which can be described by systems oflinear equations as follows:
Binary linear codes (one definition)
Denote by xi the ith bit of the code word. Assume we have a message or information word ofkbits
which is encoded into a codeword ofn bits. In a binary linear code all the 2k codewords satisfy n-klinear equations inxi for i =1,2, n (i.e. n-kequations withnunknowns).
Furthermore, we can rearrange the equations to be homogenous (i.e. the RHS is zero)
Binary linear codes (another definition)
Every homogenous system of linear equations defines a linear code. We can also show that everylinear code can be described by a homogenous system of linear equations.
NOTE
Since there aren-kequations innunknowns then we haven - (n-k) =k independent variables andn-k dependent variables. Since the variables are binary this means we have 2k solutions which is asexpected.
Example 4.13
n-length repetition code (n odd)
Info Code:x1x2x3xn
0 0000
1 1111
Withk=1 then we needn-k=n-1 equations in thenunknowns (x1x2 x3xn) which fully describesthe repetition code:
1
13
12
xx
xx
xx
n =
==
)!
0
0
0
1
31
21
=+
=+=+
nxx
xx
xx
)
-
7/29/2019 Unencrypted 4
20/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-20
n-length even-parity check codeConsider the case ofn=3 (i.e. k=2):
Info Code:x1x2x3
00 00001 011
10 10111 110
In general n =k -1 and we expect n-k =1 equation which will fully specify the even-parity checkcode, and of course its the equation used to set the check-bit, x3, itself:
121 +++= nn xxxx * ! 021 =+++ nxxx *
4.7.3 Parity Check Matrix
Given a linear code K described by a system of (n-k) homogenous equations in (n) unknowns weconstruct the (n-k) x (n) matrix, H, with the co-efficients of the equations. That is, the ith row ofHexpresses theith equation. H is known as the parity-check matrix.
Parity Check Matrix and Parity Check equations
The binary matrix H is called a parity check matrix of a binary linear code K of length (n) providedthat the code words K are precisely those binary wordsxT =[x1x2 xn] which fulfil:
0
0
0
2
1
or
x
x
x
n
0HxH =
=
))andH has (n-k) rows and (n) columns
The system of (n-k) homogenous equations are known as the parity-check equations
NOTE: The systemHx =0will have 2k solutions.
Example 4.14
H for Repetition Code
0
0
0
1
31
21
=+
=+=+
nxx
xx
xx
) ! 0Hx =
=
n
n
n
x
x
x
x
x
n
1
3
2
1
1000001
0100001
0001001
0000101
0000011
1)
!!!!! "!!!!! #$
*
*
))+)))))
*
*
*
Hence, H is a (n-1) x (n) matrix
-
7/29/2019 Unencrypted 4
21/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-21
H for Even-parity check code
021 =+++ nxxx * ! [ ] 0Hx =
=
n
n
n
xx
x
x
1
2
1
11111 )!!! "!!! #$
*
Hence, H is a (1) x (n) matrix
4.7.4 Binary Linear Codes (Take 2)
If the wordx is a codeword thenHx =0, and ify is also a codeword thenHy =0. But ifz=x+ythenHz =Hx +Hy =0, z is also a codeword. This gives us the proper definition of a binarylinear code.
Binary Linear Code (proper definition)
A binary block code is said to be linear provided that the sum of any arbitrary two code words isalso a code word.
Example 4.15
Repetition CodeInfo Code:
x1x2x3xn
0 00001 1111
There are only two codewords and 0000 +1111 =1111 which is of course a codeword, sothe repetition code is a binary linear code.
Even-parity check codeThe even-parity check code is a binary linear code.
Code K4 fromExample 4.10 Info Code00 000001 011110 100011 1111
By adding any two codewords in can be shown that K4 is a binary linear code:0111 +1000 =11111111 +1000 =0111
Code K6 fromExample 4.10Showthat K6 is a binary linear code
-
7/29/2019 Unencrypted 4
22/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-22
Code fromExample 4.5Info Code00 00001 00110 01111 111
Is this code linear? Consider:001 +011 =010! this is not a binary linear code
4.7.5 Rectangular Code
As we shall see describing binary linear codes by the parity-check matrix will facilitate both theanalysis and design of codes. However in some cases special codes can be constructed which can beanalysed in a different and more intuitive way. One such class of code is the r x s rectangular codewhich is used when transmitting row by row or two dimensional data blocks. This is a binary linear
code of lengthn=rs, whose codewords are considered asr x smatrices. Each row and column hasr-1 ands-1 information bits respectively.
3 x 4 rectangular coder =3 ands =4"n =3 x 4 =12 andk =(3-1) x (2-1) =6. To analyse this code we represent thecodeword (x1, x2, , x12) in the following matrix form:
r =3
s=4
1211109
8765
4321
xxxx
xxxx
xxxx
The information bits are:x1 x2 x3 x5 x6 x7
The check bits are:x4 x8 x9 x10 x11 x12
Each check-bit is an even-parity check over the corresponding row or column. If one includes thecheck-bits themselves then there are r +s rows and columns to check over and hence r +s =7parity-check equations:
checks-parityrow
0
0
0
1211109
8765
4321
=+++
=+++=+++
xxxx
xxxx
xxxx
checks-paritycolumn
0
0
0
0
1284
1173
1062
951
=++=++=++
=++
xxx
xxx
xxx
xxx
We expectn-k=6 parity-check equations so one of the equations is in fact redundant. However all 7equations are used to derive the parity-check matrix description:
-
7/29/2019 Unencrypted 4
23/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-23
0Hx =
=
12
11
6
2
1
100010001000
010001000100
001000100010
000100010001
111100000000
000011110000
000000001111
x
x
x
x
x
)
)
rowparity-checks
column
parity-checks
4.7.6 Systematic Binary Linear Codes
Systematic (Separable) Binary Linear Codes
Block codes in which the message or information bits are transmitted in unaltered form are called
systematic codes. Specifically for binary linear codes, consider the (k) length messagemT =[m1, m2,, mk] (mi is thei
th bit if the message). The (n) length codeword is represented by:
xT =[mT |bT] =m1, m2, , mk, b1, b2, , bn-k wherebi is theithcheck bit
That is, the codeword is formed by appending the (n-k) check bits to the (k) information bits
NOTE: Unless otherwise stated, all codes we develop will be systematic codes
Communication Systems by S. Haykin uses a reversed vector representation
The text by S. Haykin adopts the notation: b0, b1, , bn-k-1, m0, m1, , mk-1which should compared to our notation: m1, m2, , mk, b1, b2, , bn-k
The notation adopted by S. Haykin assumes bits run to the right with mk-1 being the LSB (leastsignificant bit) of the message. The notation adopted here assumes bits run to the left with m1being the LSB of the message. Thus the bit sense is reversed and the subscript notation is different.
Comparing our notation to that of S. Haykin:(LSB) m1 =mk-1, m2 =mk-2,, bn-k-1 =b1, bn-k =b0 (MSB)
Example 4.16
The repetition code is obviously systematic
n-length even-parity check codeConsider the case ofn=3 (i.e. k=2 andn-k=1):
Infom1m2
Codex1x2x3
00 00 001 01 110 10 111 11 0
same
This is a systematic code sincex1x2 =m1m2 andx3 =b1 is the check-bit
-
7/29/2019 Unencrypted 4
24/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-24
Code K4 fromExample 4.10We have thatn=4, k=2 andn-k=2:
Infom1m2
Codex1x2x3x4
00 00 0001 01 11
10 10 0011 11 11
same
This is a systematic code sincex1x2 =m1m2 andx2x4 =b1b2 are the check-bits
Code K6 fromExample 4.10Showthat K6 is a systematic code
By makingm1, m2, , mk the independent variables and theb1, b2, , bn-k the dependent variablestheb1, b2, , bn-k can be expressed as explicit functions of them1, m2, , mk variables only. Hence:
Result 4.1
In encoding, the (n-k) check-bits are appended to the (k) length message to form the resultant (n)-length codeword.In decoding, after the correct codeword has been chosen (i.e. errors corrected), the (n-k) bitsappended to the (n)-length codeword are stripped to form the (k)-length message.
4.7.7 Systematic form for H
Define:b as the (n-k) column vector of check bitsmas the (k) column vector of information bits
cas the (n) column codeword vector =
b
mx
P as the (n-k) x (k) coefficient matrixG as the (n) x (k) generator matrixHas the (n-k) x (n) parity-check matrix
Since the check-bits are generated from the information bits we represent this operation by usingP :
b =Pm
Equation 4.9
and
=
P
IG k , whereIk is the (k) x (k) identity matrix, generates the codeword:
=
=
==
b
m
Pm
mm
P
IGmc k
Equation 4.10
-
7/29/2019 Unencrypted 4
25/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-25
and ifH =[P | In-k ], whereIn-k is the (n-k) x (n-k) identity matrix, then:
0bbbPmb
mI|[PHc =+=+=
= ]n-k
Exercise: Show that HG =0
Systematic Form for H
H =[ P | In-k ]Equation 4.11
NOTE
1.The systematic form for H makes it a trivial operation to extract P and hence generate the checkbits from the information bits.
2. IfH is in non-systematic form, the resulting parity-check equations will need to be manipulated,or H can be directly manipulated by standard row operations into systematic form.
Communication Systems by S. Haykin uses a reversed matrix representation
The parity-check matrix systematic structure adopted by S. Haykin is different than that describedhere and both the rows and columns of H are reversed. Thus a parity-check matrix, H, from S.Haykin:
=
mnmm
n
n
aaa
aaa
aaa
*)+))
*
*
21
22221
11211
H is equivalent to our:
=
11121
21222
12
aaaaaa
aaa
n
n
mmmn
**
))+)
*
H
Example 4.17
Repetition code
0Hc =
=
1
2
2
1
1
1000001
0100001
0001001
0000101
0000011
n
n
b
b
b
b
m
)
*
*
))+)))))*
*
*
n-1
P In-1
m
b
-
7/29/2019 Unencrypted 4
26/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-26
[ ] Pmb =
=
=
1
1
2
2
1
1
1
1
1
m
b
b
b
b
n
n
))
[ ] Gmc =
=
=
1
1
2
2
1
1
1
1
1
1
1
m
b
b
b
b
m
n
n
))
I1
P
Even-parity check code
[ ] 0Hc =
=
1
1
2
2
1
11111
b
m
m
m
m
n
n
)*
P I1
n-1
[ ] [ ] Pmb =
==
1
2
2
1
1 1111
n
n
m
m
m
m
b )*
Gmc =
=
=
1
2
3
2
1
1
1
2
2
1
11111
10000
00100
00010
00001
n
nn
n
m
m
m
m
m
b
m
m
m
m
)
*
*
))+)))*
*
*
)
P
In-1
-
7/29/2019 Unencrypted 4
27/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-27
4.7.8 Issues
Given the 2k solutions to the systemHx =0which we use as the codewords, and the corresponding2k information words or messages, we can generate a table of codewords with 2k entries. Encodingwill then involve a table lookup, and decoding will involve searching the same table for the closest
matching codeword (Hamming distance decoding rule). Both operations involve an exponentialincrease of complexity withk. However there is a better way: systematic binary linear codesmake encoding a simple logic operation on thek-bit message. syndrome decoding involves a simple logic operation on the n-bit received word for error
detection. Error correction will involve a lookup operation of a 2n-k table which becomes asimple logic operation on then-bit received word for the case of single-bit error correction.
It should be noted that lookup of a 2n-k table is much less expensive than finding the closest matchin a 2k table. Not only is a table lookup operation more efficient to implement (e.g. as an indexingoperation) but (n-k)
-
7/29/2019 Unencrypted 4
28/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-28
4.8.2 Syndrome
Let rbe the received word:
r =ci +e
whereci is the ith codeword that was transmitted ande is the error pattern of the transmission error.
Error correction involves identifying the error pattern, e, given only r. Once we know ewe obtainthe correct codeword by: ci =r +e. But how do we getefromr? Consider:
Hr =Hci +He=0+He=s
The (n-k) column vector, s, is called thesyndrome.
Now let:]|||||[ 1321 nn hhhhhH = *
wherehi corresponds to theith column ofH. And let:
=
ne
ee
)
2
1
e
whereei is theith bit ofe. Consider:
shhhhhhhhHe =+++=
= ][][][]|||||[ 22112
1
1321 nn
n
nn eee
e
e
e
*)
*
Consider a bit error in bitsj, kand l of the codeword thenej =ek =el =1 and all other ei =0. Hence itfollows that:
][][][ lkj hhhs ++=
That is, the syndrome is the sum of the columns inH which correspond to the bits in error. Thus weget the following result:
Result 4.2
The syndrome, s, only depends on the error pattern, e, and the syndrome is the sum of the columns
ofH which correspond to the bits in error, that is == 1s.t.for][ ii eihs .Notice that although s is calculated fromr it is related to e. But does knowing s allow us to get e?Unfortunately there are many error combinations which can yield the same syndrome (i.e. we cantsimply doe=H-1ssinceH is non-invertible). We make the following observations:
1.Consider the two error patterns, ei and ej, which yield the same syndrome, thenHei +Hej =s+s =0, henceci =(ei +ej) is by definition a codeword. Equivalently, ifei yields sthenei +calso yields the sames for any and all codewordsc.
Result 4.3
All error patterns that have the same syndrome differ by a codeword and a non-zero codewordadded to an error pattern will result in a different error pattern with the same syndrome.
-
7/29/2019 Unencrypted 4
29/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-29
2.The collection of all the error patterns that give the same syndrome is called acoset of the code.Since there are 2n-k different syndromes then there will be 2n-k different cosets for the code. FromResult 4.3since there are 2k codewords there will be 2k distinct error patterns in each coset.[Note that with 2n-k cosets and 2k error patterns per coset we will have 2n-k x 2n-k =2n errorpatterns and hence all possible error pattern combinations will have been accounted for]. Thusfor a given syndrome there will be 2k distinct error patterns in the coset. Which error pattern is
the right one?
3.Based on the Hamming distance decoding rule in the particular coset associated with thesyndrome calculated, it is reasonable to select the error pattern with the minimum number of biterrors (i.e. minimum number of bits that are 1), as this is the most likely error pattern to occur.The error pattern with the minimum number of bits being 1 in a particular coset is known as thecoset leader. Hence we choose the coset leader and add that to r to yield the most likelycodeword that could have been sent.
4. If there is more than one candidate coset leader this implies that for that particular syndrome the
error cannot be corrected since there is more than one error pattern which is equally likely.
4.8.3 Encoding of Systematic Codes
IfH is in systematic form (seeEquation 4.11), then the co-effiicient matrix, P, is trivially extracted.IfH is not in systematic form, it will need to be converted to systematic form by the appropriate rowoperations. Alternatively, the system of equationsHx =0can be converted directly to the formb =Pmby the appropriate algebraic operations.
Encoding the (k) bit information word, m, to the (n)-bit codeword, c, proceeds as follows:
1.Generate the check-bit vector, b =Pm
2.Form
=b
mc
Example 4.19
Consider the encoding process for the code fromExample 4.18with parity-check matrix:
=
01110
10010
01001
H
SinceH is not in systematic form we perform the following row operations in the order stated swap rows 2 and 3 swap rows 1 and 2 add row 2 to row 1
and hence:
=
=
10
01
11
10010
01001
00111
PHsys
Thus we get the check-bit generation equations:
-
7/29/2019 Unencrypted 4
30/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-30
=
=
2
1
3
2
1
10
01
11
m
m
b
b
b
Pmb
23
12
211
mb
mb
mmb
==
+=
25
14
213
xx
xx
xxx
==
+=
The code table is now derived by generating the check-bits from the message and appending thecheck-bits to the message to form the codeword:
Infox1x2
Codex1x2x3x4x5
0 0 0 0 0 0 00 1 0 1 1 0 11 0 1 0 1 1 01 1 1 1 0 1 1
4.8.4 Syndrome Decoding
Decoding the (n)-bit received word, r, to retrieve the (k)-bit information word, m, proceeds asfollows:
1. Calculate the syndromes=Hr2. For error detection:
ifs0 then there is an error and an indication has to be sent to retransmit the message ifs=0 then there is no error andc=r, hence go to step 4.
3. For error correction: Determine the unique coset leader, ec, of the coset corresponding to s and then correct the
most likely error pattern by c=r +ec If there is more than one candidate coset leader then only error detection is possible and anindication has to be sent to retransmit the message.
4. Extract the message, m, by stripping off the (n-k) check bits appended toc.
But how do we determine the coset leader? FromResult 4.2andResult 4.3at least two approachessuggest themselves:1. Find the minimum number of columns in H that when added together yield the syndrome. The
locations of the columns specify the bit positions of the coset leader, ec.2. Find one combination, any combination, of columns in H that when added together yields the
syndrome. Define the error pattern, e, such that ei =1 if column i was used in the combinationand then form the coset by adding each 2k codeword to eand then locate the coset leader as theerror pattern with the minimum number of bits being 1.
Since the above operations are expensive to perform for each received word, the usual practice is toprime the decoder with the pre-determined coset leader for each of the possible 2n-k syndromes. Thecoset leader is then found by usingsto index (lookup) the 2n-k table of coset leaders.
In the case of single-bit errors, fromResult 4.2 ifsmatches the ith column ofH then bit i is in error(the coset leader hasei =1, and all other bits are 0) and the codeword, c, is obtained by inverting theith bit ofr. Thus there is no need to explicitly determine the coset leader.
-
7/29/2019 Unencrypted 4
31/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-31
Example 4.20
Consider the decoding process for the code fromExample 4.18.SayrT =[01111] is received. The syndromes is calculated:
=
==01
0
1
11
1
0
1001001001
00111
Hrs
=
0
1
0
s matches only the 4th column ofH, " bit 4 is in error (i.e. ecT =[00010]) and hence
cT =[01101] which is a valid codeword. The message sent is thenmT =[01].
Say rT = [00111] is received. The syndrome
==
1
1
1
Hrs does not match any column of H.
Looking for the minimum number of columns which when added yieldswe see that:
shh =
=
+
=+
1
1
1
0
1
0
1
0
1
42 and shh =
=
+
=+
1
1
1
1
0
0
0
1
1
51
Hence there are two candidate coset leaders representing double-bit errors:ec1
T =[01010] andec2T =[10001] and thus the error cannot be corrected. Indeed we see that
c1T
=(ec1 +r)
T
=[01101] andc2T
=(ec2 +r)
T
=[10110] are equally distant fromr
T
=[00111].
4.8.5 Encoding and Decoding Implementations
EncodingGeneration of the (n-k) check-bits from themmessage bits, b =Pm, can easily be implemented inhardware using XOR logic gates or making use of XOR assembly language instructions forimplementation in software or embedded systems.
XORlogic gate
array
km
m
m
)
2
1
km
m
m
)2
1
knb
b
b
)
2
1
+
n
n
k
k
c
c
c
c
c
c
1
1
2
1
)
)
-
7/29/2019 Unencrypted 4
32/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-32
DecodingAssuming an FEC system, for each syndrome, s, the corresponding unique coset leader, ec, isderived and stored. Thus syndrome decoding becomes a table lookup operation using s as the indexinto a table of size 2n-k. The generation of the syndrome can be implemented using XOR logic gatesor special XOR assembly instructions. The table lookup operation can be implemented using a (n-k)-to-2n-k decoder and inverters can be used to correct the required bits in error.
XORlogic gate
array
nr
r
r
)
2
1
kns
s
s
)
2
1
n-kto2n-k
decoder
inverterlogicenable
inverterlogicenable
inverterlogicenable
inverterlogicenable
2n-k
inputORgateor
buffer
nc
c
c
)
2
1
)
4.9 Hamming Weight
Hamming weight / Minimum weight
The Hamming weight of a word, w(x), is defined as the number of bits distinct from 0. For eachnon-trivial code K, the smallest Hamming weight of a code word distinct from 0T =[000...0] iscalled the minimum weight of K, w(K).
Example 4.21
Hamming weight examples:w(xT =[111]) w(111) =3
w(101000) =2w(110110) =4
The minimum weight of the codes considered so far are:w(K) =n for then-length repetition codew(K) =2 for then-length even-parity check codew(K4) =1 for code K4 fromExample 4.10w(K6) =3 for code K6 fromExample 4.10
For the r x s rectangular code, K, it is not necessary to list the codewords in order to find theminimum weight. All that is needed is to use the row and column parity-check equations andattempt to find the non-zero codeword solution which uses the least number of bits equal to 1. It can
thus be shown thatw(K) =4 for any r x srectangular code.
-
7/29/2019 Unencrypted 4
33/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-33
Relationship between Hamming distance and Hamming weight for binary linear codes
For each non-trivial binary linear code the minimum distance, d(K), is equal to the minimumweight, w(K). That isd(K) =w(K)
Property 4.3
FromProperty 4.1andProperty 4.2we can now say:
Error Detection / Correction for binary linear codes
A binary linear code corrects (or detects) all terrors if and only if its minimum weight is larger than2t (or larger than t, respectively)
Property 4.4
Example 4.22
w(K) =n for then-length repetition code hence:" detect all (n-1) bit errors and correct all (n-1)/2 errors
w(K) =2 for then-length even-parity check code" detect all 1 bit errors
w(K4) =1 for code K4 fromExample 4.10" no error protection at all (code K4 corrects single-bit errors conditional on x1 being correct, but
ifx1 is corrupted this is not detected, hence code K4 has no unconditional error protection)
w(K6) =3 for code K6 fromExample 4.10
" corrects all 1 bit errors and detects all 2-bit errors
w(K) =4 for r x srectangular code" corrects all 1-bit errors and detects all 3-bit errors
4.10 Designing the Parity-Check matrix
Let x be the codeword with a Hamming weight equal to the minimum weight, w(K), of the code K.
IfH is a parity-check matrix for K, then we knowHx=0. Now by definitionxhas exactlyw(K) bitswhich are 1, the remaining bits being 0. Thus the operation Hx =0 effectively sums w(K) columnsofH to 0. This gives us the following results:
Result 4.4
For a linear code K with parity-check matrix, H, if the minimum number of columns ofH that sumto zero isn, thenw(K) =n.
Result 4.5
For a linear code K with parity-check matrix, H, if no combination ofnor less columns ofH sum tozero, thenw(K) >n.
-
7/29/2019 Unencrypted 4
34/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-34
4.10.1 Structure ofHfor single-error detection
For single-bit error detectionw(K) =2 or w(K) >1. FromResult 4.5this means no single column ofH must sum to zero, i.e. no column ofH must be a zero column. This means w(K) is at least 2 andcan detect all single-bit errors.
Property of H for single-error detection
A binary linear code K detects single-bit errors if and only if every parity-check matrix of K hasnon-zero columns (i.e. no column is the all-zeros column).
4.10.2 Structure ofHfor single error correction
For single-bit error correction w(K) =3 or w(K) >2. FromResult 4.5this means no single column(i.e. n=1) ofH must sum to zero and no two columns (i.e. n=2) ofH must sum to zero. The firstcondition implies non-zero columns and the second condition implies no two columns of H areidentical (if and only if two columns are identical will they sum to zero), that is, the columns arepairwise distinct.
Property of H for single-error correction
A binary linear code K corrects single-bit errors if and only if every parity-check matrix of K hasnon-zero, pairwise distinct columns (i.e. no column is the all-zeros column and no two columns arethe same).
If a binary matrix withr
rows has to have nonzero, pairwise distinct columns this means that thenumber of columns, c
-
7/29/2019 Unencrypted 4
35/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-35
Since the columns are non-zero, pairwise distinctw(K) =3
Infox1x2
Codex1x2x3x4x5
0 0 0 0 0 0 00 1 0 1 0 1 1
1 0 1 0 1 1 01 1 1 1 1 0 1
The minimum weight of the code is, as expected, w(K) = 3. Hence this is a single-bit errorcorrecting code. Note that the code fromExample 4.18is also another solution.
(b)It is now required to design the most efficient code for 2-bit error correction with k =2. Thisrequires a code with at least w(K) = 5. To determine the smallest n that permits this, the upperbound expressionB(n,5) that was derived inExample 4.7 for the same case of M =4, d(K) =5 andL =2, was N n=8, andn-k=6 check bits, thusH is a 6x8 matrix.
To design the matrix we note that to be in systematic form, predefines the 6x6 identity matrix, I6.How do we choose the remaining 2 columns (which defineP)? Sincew(K) =5 we must ensure thatno combination of 4 or less columns will add to zero. Here is one solution:
=
10000001
01000001
00100010
00010011
00001010
00000111
H
P I6
18
17
26
215
24
213
xx
xx
xx
xxx
xx
xxx
=
==
+==
+=
Infox1x2
Codex1x2x3x4x5x6x7x8
0 0 0 0 0 0 0 0 0 00 1 0 1 1 1 1 1 0 01 0 1 0 1 0 1 0 1 11 1 1 1 0 1 0 1 1 1
From which we see that the minimum weight of the code is indeed w(K) =5. Also 4182 ==R which
is not that good. We may need to increasekto achieve better efficiency.
-
7/29/2019 Unencrypted 4
36/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-36
4.11 Perfect Codes for Error Correction
A binary linear (n,k)-code requires 2k codewords of length n. Thus there are 2n possible words oflengthn and 2k of these are codewords and 2n - 2k of these are non codewords. If the code correctsup to terrors, the non codewords are either:
Type 1: non codewords of distancet or less from precisely one of the codewords, that is, the t orless bits in error can be corrected.
Type 2: non codewords of distance greater than t from two or more codewords, that is, the errorscan only be detected.
If all the non codewords are ofType 1 then the code is a perfect code.
Properties of Perfect Codes1. Perfect codes will always be able to correct the most likely errors for any word received, there is
no need to handle detected errors (there will always be a unique coset leader).2. Perfect codes guarantee maximum information rate for t-bit error correction (this is intuitively
obvious since the unnecessaryType 2 non-codewords dont exist and fewer words meansmallern).3. A perfect code correcting up to t errors will also have the smallest allowable minimum distance
of 2t+1.
Conceptual definition of a perfect code
A linear code of length n is called perfect for t errors provided that for every word r of length n,there exists precisely one code word of Hamming distancetor less fromr.
Operational definition of a perfect code
With onlyType 1non-codewords, r, thens=Hr, must be able to correct up to t-bit errors. Thus the2n-k possible values of s must be the same as the total number of different combinations of0, 1, 2, , t-1, tbit errors of ann-bit word. Thus a linear (n,k) code is perfect for terrors if:
=
=
t
i
kn
i
n
0
2
Equation 4.12
Example 4.24Consider the code fromExample 4.23(a):
Info Code0 0 0 0 0 0 00 1 0 1 0 1 11 0 1 0 1 1 01 1 1 1 1 0 1
Consider the 5-bit word 00111 It is not of distance (t=1) or less from precisely one codeword It is, in fact, of distance 2 from two codewords: 01011 and 10110! NOT a perfect code
Furthermore:
615
058222
1
0
325 =
+
=
===
=
=
t
i
knin
-
7/29/2019 Unencrypted 4
37/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-37
4.12 Hamming Codes - Perfect Codes for Single Error Correction
The Hamming codes we now consider are an important class of binary linear codes. Hammingcodes represent the family of perfect codes for single-error correction.
For single-error correctiont=1, and since the code is perfect fromEquation 4.12:
121
0
+=
=
=
=
ni
nt
i
kn = ! n=2m-1 wherem=n-k and k=n-m=2m-m-1
Properties of a (n,k) Hamming Code
Block length: n=2m-1 m 3
Information length : k=n- m=2m-m-1
Min. distance: d(K) =3
Error: single bit correction.
NOTE
1. For single-error correction, H must have non-zero, pairwise distinct columns. For mrows wecan have, at most, 2m-1 non-zero and distinct columns for H (i.e n 2m- 1 ).
2. Since
n
mn
n
kR
== , the more columns we have (larger n) per fixed m the better our
information rate. Thus the condition for perfect codes, n=2m-1, implies maximum informationrate.
Hamming Code (one possible definition)
A binary linear code is called a Hamming code provided that it has, for some number m, a paritycheck matrix H ofmrows and 2m-1 columns such that each non-zero binary word of length mis acolumn ofH.
4.12.1 Syndrome Decoding for Hamming Codes
Since the Hamming codes are perfect codes for single-error correction, all 2n possible receivedwords, r, are either a codeword or a codeword corrupted by 1 bit. Thus for each possible syndrome,s, there must be only one coset leader. From the syndrome decoding condition for single-errorcorrecting codes we have the following decoding procedure for Hamming codes:
1.Calculate the syndrome: s=Hr2. Ifs is zero there is no error.3. Ifs is non-zero, let i be the column of H which matches s, then correct the ith bit of the received
word to form the codeword, c.4.Decode to the information word, m, by stripping off the (n-k) check bits appended toc.
-
7/29/2019 Unencrypted 4
38/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-38
Example 4.25
Consider the following (7,4) Hamming code parity-check matrix:
=
1010101
1100110
1111000
H "3
4and7
====
knm
kn
Design the encoding logic, list the codewords and perform syndrome decoding on:0110010 0110101 1101100 0001111
Code TableWe need to convert H to systematic form:
=
1001011
0101101
0011110
sysH !
4217
4316
4325
xxxx
xxxx
xxxx
++=++=++=
messagex1x2x3x4codewordx1x2x3x4x5x6x7
messagex1x2x3x4codewordx1x2x3x4x5x6x7
0000 0000000 1000 10000110001 0001111 1001 10011000010 0010110 1010 10101010011 0011001 1011 10110100100 0100101 1100 11001100101 0101010 1101 11010010110 0110011 1110 11100000111 0111100 1111 1111111
Encoding Logic
x4 x3 x2 x1
messagex4x3x2x1
x7 x6 x5
codewordx7x6x5x4x3x2x1
UP for first 4bits
DOWN forlast 3 bits
N.B. Bits are serially shifted to the right with the LSB first (right-hand side) and MSB last (left-hand side), thus the message and codeword bits are reversed.
-
7/29/2019 Unencrypted 4
39/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-39
Syndrome DecodingWe calculates=Hr (using the original non-systematic form ofH):
rT sT i cT mT
0110010 111 7 0110011 01100110101 011 3 0100101 0100
1101100 010 2 1001100 10010001111 000 - 0001111 0001
NOTE
1.The first three Hamming codes are: m n k R
3 7 4 4/7 =0.57 4 15 11 11/15 =0.73 5 31 26 26/31 =0.84
2.The information rate for the Hamming codes go as12
1
=
=m
m
n
mnR but althoughR tends
to 1 rapidly for larger mwe correct only single errors for increasingly large blocks less errorprotection.
Block Error Probability for Hamming Codes
22121
120
121)(
=
mm
qppmPmm
e
4.13 Types of Errors - Probabil ity of Undetected Error
Assume the codeword ci was transmitted and r is received. The following types of errors can bepresent when decoding a word:1. No error (s=0andr =ci) " ideal
2. Detected error (s0andranyc):a) correctly correct (coset leader r =ci" non-fatal)b) cannot correct (two or more coset leaders " non-fatal)c) incorrectly correct (coset leader r =cjci" fatal)
3. Undetected error (s=0andr =cjci) " fatal
In FEC error-control both 2(c) and 3 are fatal (and also 2(b) if there is no ARQ capability). ForARQ error-control only 3 is fatal since the case of detected errors in 2 are dealt with in the sameway (by re-transmission). Since most communication systems use ARQ, it is the undetected errorsthat are a universal problem.
-
7/29/2019 Unencrypted 4
40/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-40
4.13.1 Probability of undetected error
An undetected error occurs if the received word, r, is a different codeword, (r =cj), then wastransmitted, ci. Thus:
e=ci+r =ci +cj (eis the sum of two codewords!eis a codeword itself)
Condition for an undetected error
An undetected error occurs if and only ifeis a nonzero code word.
The probability of undetected error, Pund(K), is the sum of the probability of occurrence of the errorpatterns which yield an undetected error. Define ei as the error pattern with i bits in error, thus theprobability of occurrence ofei is =qipn-i. Since theei of interest are those which are the same as thenon-zero codewords, we letAi =number of code words with Hamming weight i, and hence:
=
= n
i
iniiund pqAP
1
)K(
If the minimum weight of the code is d(K) thenA1=A2=... =Ad(K)-1=0 so we have:
Probability of undetected error
=
=n
di
iniiund pqAP
)K(
)K(
Equation 4.13
where Ai =the number of codewords with Hamming weight i
Example 4.26
Consider the (7,4) Hamming code table ofExample 4.25from which we see that:d(K) =3 ! A3 =7, A4 =7, A5 =0, A6 =0, A7 =1
Hence:73443 77)K( qpqpqPund ++=
Ifq=0.01, then:Pund(K) =7 x 10
-6" 7 codewords in a million will have undetected errors
-
7/29/2019 Unencrypted 4
41/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-41
4.14 Other Error Detecting/Correcting codes
4.14.1 Types of Errors
When examining different codes it is important to consider: Type of Code: Is the code systematic or non-systematic? Information Rate: How many check-bits are used per message bit? Error protection: Number of bit errors that can be detected / corrected. Diagnostic resolution: Range over which bit errors are detected / corrected
(i.e. larger n" lower resolution) Error latency: How much data needs to be received before the error is detected / corrected
(i.e. larger n" longer latencies) Coverage: Which bit/word positions will be protected and in what way. Types of errors: Is the code best suited to random or burst errors?
The discussion so far has assumed that bit errors occur randomly (i.e. channel is memoryless orindependent). In practice, however, the condition that causes the bit error may last long enough to
affect a sequence of bits (i.e. channel has memory). If a sequence of , consecutive bits are suddenlysubject to interference, which causes a large number of them to be in error, then we have a condition
called a burst error of length,. A specific type of burst error is the multiple unidirectional errorwhich forces groups of bits to all go 0 (low) or 1 (high). This can be caused by a common mode offailure in the memory block or bus causing a group of local cells or bus lines to go low (i.e. shorted)or high.
4.14.2 Bit interleaving
Although codes can be designed to specifically handle ,-length burst errors (e.g. the CRC codesdiscussed in Section 4.15), a larger class of codes (e.g. the Hamming codes) exist which areapplicable only to random (or independent channel) errors. One way to convert burst errors torandom errors is to interleavebits entering the channel in a deterministic manner. If a sequence ofbits experience a burst-error the corresponding deinterleavingoperation will spread these errors outand hence randomise them.
Encoder Interleaver Burst-errorChannel Deinterleaver Decoder
Independent-error Channel
InputData
OutputData
Example 4.27
Consider the following 16-bit block of data: b1b2b3b4 b5b6b7b8 b9b10b11b12 b13b14b15b16
1:4 Interleaver: b1b5b9b13 b2b6b10b14 b3b7b11b15 b4b8b12b16
A burst-error of length 4 occurs: b1b5b9b13 b2b6bb1100bb1144bb33bb77b11b15 b4b8b12b16
1:4 Deinterleaver: b1b2bb33b4 b5b6bb77b8 b9bb1100b11b12 b13bb1144b15b16
And the burst error of length 4 has been converted a 1-bit random error in eachn=4 code block.
-
7/29/2019 Unencrypted 4
42/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-42
4.14.3 m-out-of-n Codes
Them-out-of-n codes are formed as the set ofn-bit vectors, where the vector is a code word if andonly if exactly mbits of the bits are 1. The 2-out-of-5 code is the most common example of such a
code:Table 4.1: 2-out-of-5 code
message number codeword1 000112 001013 010014 100015 001106 010107 10010
8 011009 1010010 11000
Properties Non-systematic code (need to do a table lookup) Single error detection Multiple unidirectional error detection Used in encoding control signals
Since the code is non-systematic its use for encoding and decoding communication messages is
limited. However control signals which do not need to be decoded (just interpreted) can beconfigured to exist asm-out-of-ncodes, and any multiple unidirectional error on the control bus canbe detected. In such a case it is obvious that the number of 1s will change and the error can bedetected.
Example 4.28
Consider a 2-out-of-5 code and assume message 7 (encoded as 10010) is sent.
Suppose a unidirectional error that causes the first 3 bits to go low occurs (most common errorwhen adjacent memory cells or bus lines experience a common failure like a short-circuit), then the
received message will be 00010 and this error is detected (since only one bit is a 1).Suppose that now the first three bits are all tied high, then the received message will be 11110 andthis error is also detected (since four bits are now 1).
4.14.4 Checksum Codes
Checksum codes are generated by appending an n-bit checksum word to a block of sn-bit datawords, formed as the sum of thesn-bit words using modulo 2n addition.Encoding: use ann-bit arithmetic adder to add thesn-bit data words, with any carry beyond thenth
bit being discarded. The sum is then appended to the data block as then-bit checksumDecoding: use the samen-bit arithmetic adder to add thesn-bit data words and XOR the sum withthe appended checksum, if the result is zero then there is no error.
-
7/29/2019 Unencrypted 4
43/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-43
Properties Simple and inexpensive, hence large number of uses Single error detection at least Multiple error detection dependent on s and columns in error: error coverage is highest for the
least significant bits
Error coverage can be controlled by weighting the bits or using extended checksums Long error latency (must wait for end of data block) Low diagnostic resolution (error can be anywhere in thesn-bit words) Uses: sequential storage devices, block-transfer peripherals (disks, tapes), read-only memory,
serial numbering, etc.
Example 4.29
Let n=8 and s=8, then we append an 8-bit checksum, Ck, after every block of 8x 8-bit data words
(bytes), wi, calculated as follows: =
=8
1
8k 2modwC
i
i . Consider the following data block (in hex):
01 FA 02 32 62 A1 22 E3
The sum (in hex) =337 and Ck =337(hex) mod 256 =37(hex)(i.e., we simply retain the low-order 8 bits of the sum and discard the rest)
The multiple error coverage can be improved by using an extended checksum ofmbits for n-bitdata wherem>nand the sum is modulo 2maddition (i.e. we retain the low-order mbits of the sum).As an examplen=8 andm=16 and a 2-byte checksum is formed from the sum ofsdata bytes in theblock.
Real-world Checksum examples
International Standard Book Number (ISBN)ISBN 0 13 283796 X
Country
USA
Publisher
(Prentice Hall)Book ID
Check symbol
(X is for 10)
Check symbol (a10 =X =10) is derived such that =
1
1011a
i
ii is divisible by 11, where ak is the kth
digit from the left in the ISBN code. For the above example we see that:Sum =(i =10)*(a1 =0) + (i = 9)*(a2 = 1) +8*3 + 7*2 + 6*8 + 5*3 + 4*7 + 3*9 + 2*6 +(i =1)*(a10 =X =10) =187 which is divisible by 11
UWA Student Number
The last digit of the 7 digit student number, s7, is the check digit derived such that:
Sum=(8*s1 +4*s2 +6*s3 +3*s4 +5*s5 +2*s6 +s7) is divisible by 11
-
7/29/2019 Unencrypted 4
44/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-44
4.15 Cyclic Codes
4.15.1 Introduction
The problem with checksum codes is that simply summing bits to form a checksum (or check bits)does not produce a sufficiently complex result. By complex we mean that the input bits shouldmaximally affect the checksum, so that a wide range of bit errors or group of bit errors will affectthe final checksum and be detected. A more complex checksumming operation can be formed byperforming division rather than summation. Furthermore the remainder not the quotient is used, asthe remainder gets kicked about quite a lot during the calculation, and if more bytes were added tothe message (dividend) its value could change radically again very quickly.
In the previous sections the idea of the parity-check matrix, check-bit generation and syndrome
decoding as a means for efficient design and implementation of linear codes was shown from thepoint of view of binary matrix linear algebra. The cyclic codes are a subclass of linear codes whichsatisfy the following two properties:
Cyclic codes
Linearity property: the sum of any two codewords is also a codewordCyclic property: any cyclic shift of a codeword is also a codeword
Example 4.30
Consider the word: 01001100One cyclic shift to the right: 00100110Another: 00010011Yet another: 10001001
If the code is a cyclic code all of the above would be codewords.
In a systematic cyclic code the check-bits are generated as the remainder of a division operation andas indicated this provides a more powerful form of checksumming. The cyclic codes are analysednot by representing words as binary vectors and using matrix binary linear algebra but byrepresenting words as binary polynomials and using polynomial arithmetic modulo 2. Like linearcodes the cyclic codes also permit efficient implementations of the encoding and decodingoperations, especially fast logic implementations for serial data transmission. The properties ofcyclic codes also allow greater scope for designing codes for specialised uses, especially in handlingburst errors. Not surprisingly all modern channel coding techniques use cyclic codes, especially theimportant class of Cyclic Redundancy Check (CRC) codes for error detection.
-
7/29/2019 Unencrypted 4
45/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-45
4.15.2 Code polynomials and the cyclic property
Let thek-bit information or message word be represented as:
m0m1 mk-2mk-1
wherem0 is the MSB (most significant bit) andmk-1 is the LSB (least significant bit).
Let then-bit code word be represented as:
c0c1 cn-2cn-1
wherec0 is the MSB andcn-1 is the LSB.
And let the (n-k)-bit check-bits be represented as:
b0b1 bn-k-2bn-k-1
We assume the code is systematic hence:
c0c1 cn-k-2cn-k-1cn-kcn-k+1 cn-2cn-1 b0b1 bn-k-2bn-k-1m0m1 mk-2mk-1
Same reversed notation as used by Communication Systems by S. Haykin
We are adopting the same notation that the textbook by S. Haykin uses. Compared to the previousdefinition inSection4.7, there are two major differences:1. The indexing sense has been reversed, and indexing starts at 0 not 1.2. The codeword representation is now reversed. The bits are now running to the right.
Previously the 7-bit codeword 1101110implied a message of 1101 followed by the check-bit
sequence 110, all running to the left: Now the same 7-bit codeword is bit reversed to
0111011 which implies a message of 1011 followed by the check-bit sequence 011, allrunning to the right: In both cases the message and check-bits are actually identical.
We denote the column vectors using this reversed notation with a , that is:
]0111011[]1101110[ TT == rr
For a codeword of lengthnwe define thecode polynomial, C(x), as:
=
=1
0
)C(n
i
iixcx
We can similarly define themessage polynomial, M(x) is formed as:
=
=1
0
)M(k
i
iixmx
and let:
=
=1-
0
)B(kn
i
iixbx
The for a systematic code we see that:
-
7/29/2019 Unencrypted 4
46/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-46
C(x) =B(x) +xn-kM(x)
Equation 4.14Let C(j)(x) representj cyclic shifts to the right of C(x), it can be shown that:
xjC(x) =Q(x)(xn +1) +C(j)(x)Equation 4.15
That is:C(j)(x) =xjC(x) mod (xn +1)
Equation 4.16
That is the remainder when dividing xjC(x) by (xn +1) represents the polynomial of the codewordcyclically shifted to rightj times.
NOTE:Remember we are using mod-2 binary arithmetic so: xi xj"xi +xj ; xi +xi =0 and 1 +1 =0
Example 4.31
Consider the codeword: 010 01100 withn=8 andk=5.The left-most bits 010 are the check-bits and hence:
xxxx =++= 2010)(BThe right-most bits 01100 are the message bits and hence:
2432 00110)(M xxxxxxx +=++++=The codeword polynomial is then:
54233
54765432
)()(M)(B)(C
00110010)(C
xxxxxxxxxxx
xxxxxxxxxxx
++=++=+=
++=+++++++=
Say C(x)
01001100 is now cyclically right shifted by 3 bits, then we get C(3)(x)
10001001, andhence:
74)3(54 1)(C)(C xxxxxxx ++=++=Now:
8743 )(C xxxxx ++=
And:
)1(mod)(C 83 +xxx !1
11
1
47
8
4788
+++
+++
xxx
xxxx
=C(3)(x)
Let:
)(R)1(mod)(C 83 xxxx =+where R(x) is the remainder, then:
)(R)(Q)1()(C 83 xxxxx ++=
where Q(x) is the quotient. From the above results we have that:
Q(x) =1 and R(x) =x7 +x4 +1
)1(1)1()(C 4784783 ++++=++= xxxxxxxx
-
7/29/2019 Unencrypted 4
47/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-47
Now consider the (n-k+1) bit word represented by:
g0g1 gn-k-1gn-k
where, by definition, g0 =1 andgn-k =1 The corresponding polynomial is known as the:
Generator Polynomial
knkn
i
ii xxgx
=++=
1-
11)G(
Equation 4.17
We choose G(x) to be a factor ofxn +1 (i.e. xn +1 =G(x)H(x), where H(x) is known as theparity-check polynomial) and form all words of the form C(x) =A(x)G(x) as code polynomials belongingto the code defined by G(x). The code so defined is obviously linear, but is it cyclic? That is, will acyclic shift of C(x) yield a polynomial which can be defined by, say, A2(x)G(x) (i.e. C
(j)(x) =A2(x)G(x) for some A2(x)) and thus belong to the code? UsingEquation 4.15with the appropriatesubstitutions (C(x) =A(x)G(x) and (xn +1) =G(x)H(x)), we have:
xjA(x)G(x) =Q(x)G(x)H(x) +C(j)(x)
Dividing throughout by G(x) we get:
)(A))H(Q()A()G(
)(C
)G(
)(C))H(Q()A(
2
)(
)(
xxxxxx
x
x
xxxxx
jj
jj
=+=
+=
and hence C(j)(x) =A2(x)G(x). Thus:
Result 4.6
If the degreen-k polynomial G(x) as defined by Equation 4.17is a factor of (xn +1), then all codepolynomials (i.e. codewords) formed as the product A(x)G(x), where A(x) is any degree k-1polynomial, belong to the (n,k) cyclic code defined by G(x).
NOTE
1.The degree of the polynomials encountered so far are: C(x) is a polynomial of degreen-1 M(x) is a polynomial of degreek-1
G(x) is a polynomial of degreen-k B(x) is a polynomial of degreen-k-1
2. The product of a polynomial of degree a with a polynomial of degree b is a polynomial ofdegreea+b. Hence A(x) is a polynomial of degreek-1 sincen-1 =(k-1) +(n-k).
3. Since A(x) is a polynomial of degree k-1, there are 2k possible A(x) and hence 2k possible codepolynomials.
4. G(x) is governed only by the degree (n-k) and the fact that it must be a factor of (xn +1), thus forthe same G(x) we may be able design different codes for different appropriate values ofn.
In the same way that the parity-check matrix, H, was used to define a linear code the generatorpolynomial, G(x), is used to define a cyclic code.
-
7/29/2019 Unencrypted 4
48/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-48
4.15.3 Encoding/Decoding cyclic codes (algebraic method)
By making A(x) =M(x) we simply derive C(x) =M(x)G(x), but this will not give us a systematic(i.e. separable) code. For a systematic code we must have:
C(x) =A(x)G(x) =B(x) +xn-kM(x)
Hence dividing the above through by G(x):
)G(
)B()A(
)G(
)M(
x
xx
x
xx kn+=
Thus B(x) is the remainder left when dividing xn-kM(x) by G(x).
Encoding Systematic Cyclic Codes
1.Express the message word in polynomial form, M(x)2.Formxn-kM(x).3.Dividexn-kM(x) by G(x) to find the remainder B(x).
4.Form the codeword polynomial, C(x) =B(x) +xn-kM(x)5.Convert C(x) to binary form to form the codeword.
Example 4.32
Consider G(x) =1 +x +x3 with n=7 and k=4 (we will later see that this G(x) produces a (7,4)Hamming code). Thus C(x) =x3M(x) +B(x), where B(x) is the remainder left when dividing x3M(x)by G(x). The following are examples of encoding systematic cyclic codes algebraically for themessages 1000, 1010 and 1011:
Message
m0m1m2m3
M(x) x3M(x) B(x) C(x) Codeword
b0b1b2m0m1m2m31 0 0 0 1 x3 1+x 1+x+x3 1 1 0 1 0 0 01 0 1 0 1+x2 x3+x5 x2 x2+x3+x5 0 0 1 1 0 1 01 0 1 1 1+x2+x3 x3+x5+x6 1 1+x3+x5+x6 1 0 0 1 0 1 1
B(x) for Message 1000)(Q
)(B
)(M)(G 3x
x
xxx
*
!
1
1
1
1
3
33
+
++
++
x
xx
xxx
B(x) for Message 1010
)(Q
)(B
)(M)(G 3x
x
xxx
*
!
2
2
235
353 1
x
x
xxx
xxxx
++
+++
B(x) for Message 1011
-
7/29/2019 Unencrypted 4
49/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-49
)(Q
)(B
)(M)(G 3x
x
xxx
*
!
1
1
1
1
23
3
3
24
234
235
45
346
3563+++
++
+
++
++
++
+
++
++++xxx
xx
xx
xxx
xxx
xxx
xx
xxx
xxxxx
For decoding consider R(x) as the received word polynomial of degree n-1. Let S(x) denote theremainder of dividing R(x) by G(x). Since C(x) =A(x)G(x) then if there are no errors we expect R(x)=C(x) and hence no remainder (i.e. S(x) =0). If S(x) 0 then we know there is an error. S(x) iscalled thesyndrome polynomial.
How is S(x) related to E(x), theerror polynomial?
We have:
R(x) =C(x) +E(x)Now:Q(x)G(x) +S(x) =A(x)G(x) +E(x)
where Q(x) is the quotient and S(x) is the remainder when R(x) is divided by G(x). Then:
E(x) =(Q(x) +A(x))G(x) +S(x)
Hence the syndrome can be obtained as the remainder of dividing the error polynomial, E(x), byG(x) (i.e. S(x) =E(x) mod G(x)). In the same way the syndrome was dependent only on the errorpattern for linear codes the syndrome polynomial is only dependent on the error polynomial forcyclic codes. Thus the syndrome polynomial can be used to find the most likely error polynomial.
Decoding Cyclic codes
1.Express the received word in polynomial form, R(x).2.Divide R(x) by G(x) to find the remainder, S(x)3. If S(x) =0, there is no error.4. If S(x) 0 then an error has occurred and the coset leader error polynomial for S(x) is used to
correct the error.
-
7/29/2019 Unencrypted 4
50/58
ITC314 Information Theory and Coding 314 623.314
22/01/01 Dr. Roberto Togneri, E&E Eng, UWA Page 4-50
Example 4.33
Consider the cyclic code from Example 4.32 with G(x) = 1 + x + x3. Say the codeword
]0011010[T =c is transmitted, but ]0011110[T =r is received (an error in bit r4). To decode weform:
R(x) =x2 +x3 +x4 +x5
and divide R(x) by G(x) to yield the remainder S(x):
)(Q
)(S
)(R)(G
x
x
xx
*
!
xx
xx
xxx
x
xxx
xxxxxx
+
+
++
++
+++++
2
2
24
4
235
23453 1
Thus S(x) =x2 +x" ]011[T =s , and since S(x) 0 an error is detected and syndrome decodingcan be used to attempt to correct the error.
NOTE
1.The generator polynomial, G(x), can be shown to be related to the generator matrix, G, and theparity-check polynomial, H(x), can be shown to be related to the parity-check matrix, H.
2.The Hamming codes are cyclic codes! Specifically a (7,4) Hamming code is designed with
G(x) =1 +x+x
3
. Since G(x)H(x) =x
7
+1, the corresponding H(x) =1 +x+x
2
+x
4
can be used toconstruct the (7,4) Hamming code parity-check matrix.
3.S(x) is a polynomial of degreen-k-1, E(x) is a polynomial of degreen-1.
Exercise: Verify that the codewords generated with G(x) as defined above indeed produce a (7,4)Hamming code
4.15.4 Deriving the Gen