EE465: Introduction to Digital Image Processing 1
Lossy Image Compression:a Quick Tour of JPEG Coding StandardWhy lossy for grayscale images?
Tradeoff between Rate and DistortionTransform basics
Unitary transform and properties Quantization basics
Uniform Quantization and 6dB/bit RuleJPEG=T+Q+C
T: DCT, Q: Uniform Quantization, C: Run-length and Huffman coding
EE465: Introduction to Digital Image Processing 2
Why Lossy?In most applications related to consumer
electronics, lossless compression is not necessary What we care is the subjective quality of the
decoded image, not those intensity valuesWith the relaxation, it is possible to
achieve a higher compression ratio (CR) For photographic images, CR is usually below 2
for lossless, but can reach over 10 for lossy
EE465: Introduction to Digital Image Processing 3
A Simple ExperimentBit-plane representation
A=a0+a12+a222+ … … +a727
Least Significant Bit (LSB)
Most Significant Bit (MSB)
Example A=129 a0a1a2 …a7=10000001
a0a1a2 …a7=00110011 A=4+8+64+128=204
EE465: Introduction to Digital Image Processing 4
A Simple Experiment (Con’t)How will the reduction of gray-level
resolution affect the image quality? Test 1: make all pixels even numbers (i.e.,
knock down a0 to be zero) Test 2: make all pixels multiples of 4 (i.e., knock
down a0,a1 to be zeros) Test 3: make all pixels multiples of 4 (i.e., knock
down a0,a1,a2 to be zeros)
EE465: Introduction to Digital Image Processing 5
Experiment Results
original Test 1
Test 2 Test 3
EE465: Introduction to Digital Image Processing 6
How to Measure Image Quality?Subjective
Evaluated by human observers Do not require the original copy as a reference Reliable, accurate yet impractical
Objective Easy to operate (automatic) Often requires the original copy as the reference
(measures fidelity rather than quality) Works better if taking HVS model into account
EE465: Introduction to Digital Image Processing 7
Mean Square Error (MSE)
Peak Signal-to-Noise-Ratio (PSNR)
H
i
W
j
jiYjiXHW
MSE1 1
2)],(),([1
)(255log102
10 dBMSE
PSNR
Objective Quality Measures
original decoded
Question: Can you think of a counter-example to prove objective measure is not consistent with subjective evaluation?
EE465: Introduction to Digital Image Processing 8
Answer
Shifted (MSE=337.8) Original cameraman image
By shifting the last row of the image to become the first row, we affectlittle on subjective quality but the measured MSE is large
EE465: Introduction to Digital Image Processing 9
Overview of JPEG Lossy Compression
Flow-chart diagram of DCT-based coding algorithm specified by Joint Photographic Expert Group (JPEG)
T Q C
EE465: Introduction to Digital Image Processing 10
Lossy Image Compression:a Quick Tour of JPEG Coding Standard
Why lossy for grayscale images? Tradeoff between Rate and Distortion
Transform basics Unitary transform and properties
Quantization basics Uniform Quantization and 6dB/bit Rule
JPEG=T+Q+C T: DCT, Q: Uniform Quantization, C: Run-length
and Huffman coding
EE465: Introduction to Digital Image Processing 11
An Example of 1D Transform with Two Variables
x1
x2
y1y2
1111
21,
1111
21
2
1
2
1 AAxyxx
yy
Transform matrix
(1,1)(1.414,0)
EE465: Introduction to Digital Image Processing 12
11 NNNN xy Aforward transform
NNNN
N
N x
xx
aa
aa
y
yy
2
1
1
111
2
1
Generalization into N Variables
N
i
TiNiiii aabbxy
11 ],...,[,
basis vectors (column vectors of transform matrix)
EE465: Introduction to Digital Image Processing 13
Decorrelating Property of Transform
x1
x2 y1y2
x1 and x2 are highly correlated
p(x1x2) p(x1)p(x2)
y1 and y2 are less correlated
p(y1y2) p(y1)p(y2)
EE465: Introduction to Digital Image Processing 14
Transform=Change of CoordinatesIntuitively speaking, transform plays the
role of facilitating the source modeling Due to the decorrelating property of transform,
it is easier to model transform coefficients (Y) instead of pixel values (X)
An appropriate choice of transform (transform matrix A) depends on the source statistics P(X) We will only consider the class of transforms
corresponding to unitary matrices
EE465: Introduction to Digital Image Processing 15
A matrix A is called unitary if A-1=A*T
Unitary Matrix and 1D Unitary Transform
Definition conjugatetranspose
When the transform matrix A is unitary, the defined transform is called unitary transform
Example
TAAA
1111
21,
1111
21 1
For a real matrix A, it is unitary if A-1=AT
xAy
EE465: Introduction to Digital Image Processing 16
Inverse of Unitary TransformFor a unitary transform, its inverse is defined by
yyx T *1 AA
NNNN
N
N y
yy
aa
aa
x
xx
2
1
**1
*1
*11
2
1
Inverse Transform
N
i
Tiii Nii
aabbyx1
** ],...,[,1
basis vectors corresponding to inverse transform
EE465: Introduction to Digital Image Processing 17
Properties of Unitary TransformEnergy compaction: only few transform
coefficients have large magnitude Such property is related to the decorrelating
role of unitary transformEnergy conservation: unitary transform
preserves the 2-norm of input vectors Such property essentially comes from the fact
that rotating coordinates does not affect Euclidean distance
EE465: Introduction to Digital Image Processing 18
Energy Compaction Example
1009898
100
,
111111111111
1111
21 xAHadamard matrix
200
198
1009898
100
111111111111
1111
21xy
A
insignificant
significant
EE465: Introduction to Digital Image Processing 19
Energy Conservation*xy A A is unitary 22 |||||||| xy
Proof
2
1
2***
**
1
22
||||||)(
)()(||||||
xxxxxx
xxyyyy
N
ii
TTT
TTN
ii
AA
AA
EE465: Introduction to Digital Image Processing 20
Numerical Example
43
,1111
21 x
A
17
21
43
1111
21xy A
Check:
252
17||||,2543||||22
2222
yx
EE465: Introduction to Digital Image Processing 21
Implication of Energy Conservation
xy A
A is unitary
Q TNyyy ]ˆ,...,ˆ[ˆ
1T
Nyyy ],...,[ 1
TNxxx ],...,[ 1
T T-1
TNxxx ]ˆ,...,ˆ[ˆ
1
xy ˆˆ A)ˆ(ˆ xxyy
A
22 ||ˆ||||ˆ|| xxyy
EE465: Introduction to Digital Image Processing 22
Summary of 1D Unitary TransformUnitary matrix: A-1=A*T
Unitary transform: A unitaryProperties of 1D unitary transform
Energy compaction: most of transform coefficients yi are small
Energy conservation: quantization can be directly performed to transform coefficients
xy A
22 |||||||| xy 22 ||ˆ||||ˆ|| xxyy
EE465: Introduction to Digital Image Processing 23
NNNNNN XAYDo N 1D transforms in parallel
NNN
N
NNN
N
NNN
N
xx
xx
aa
aa
yy
yy
1
111
1
111
1
111
From 1D to 2D
Nixy ii 1, AT
iNiiT
iNii yyyxxx ],...,[,],...,[ 11
Ni yyy |...||...|1 Ni xxx
|...||...|1
EE465: Introduction to Digital Image Processing 24
TNNNNNNNN AXAY2D forward transform
NNN
N
NNN
N
NNN
N
NNN
N
aa
aa
xx
xx
aa
aa
yy
yy
1
111
1
111
1
111
1
111
1D column transform 1D row transform
Definition of 2D Transform
EE465: Introduction to Digital Image Processing 25
2D Transform=Two Sequential 1D Transforms
Conclusion: 2D separable transform can be decomposed into two sequential The ordering of 1D transforms does not matter
TAXAY
AXY1 TTT )(
11 AYAYY row transform
column transform
TTT )(2 AXXAY
2AYY
row transform
column transform
(left matrix multiplication first)
(right matrix multiplication first)
EE465: Introduction to Digital Image Processing 26
Energy Compaction Property of 2D Unitary Transform
Example
94979910010096979894941001009998100100
X
5.105.125.025.01
25.425.215.505.391
Y
TAXAY
111111111111
1111
21A
insignificant coefficients (th=64)
A coefficient is called significant if its magnitudeis above a pre-selected threshold th
EE465: Introduction to Digital Image Processing 27
Energy Conservation Property of 2D Unitary Transform
TAXAY 22 XY
Example:
2/12/12/12/1A
4321
X
0215
YTAXAY
2222222222 ||||0125304321 YX
2-norm of a matrix X
N
i
N
jijx
1 1
22 ||X
A unitary
You are asked to prove such property in your homework
EE465: Introduction to Digital Image Processing 28
TAXAY *ˆˆ AYAX *T
T)AXA(XYY ˆˆ
22 ||ˆ||||ˆ|| XXYY
Implication of Energy Conservation
Q YYX T T-1 X
Similar to 1D case, quantization noise in the transform domainhas the same energy as that in the spatial domain
EE465: Introduction to Digital Image Processing 29
Important 2D Unitary TransformsDiscrete Fourier Transform
Widely used in non-coding applications (frequency-domain approaches)
Discrete Cosine Transform Used in JPEG standard
Hadamard Transform All entries are 1 N=2: Haar Transform (simplest wavelet
transform for multi-resolution analysis)
EE465: Introduction to Digital Image Processing 30
Discrete Cosine Transform
CA
NNN
N
NN
aa
aa
..............................
......
1
111
NlNkNkl
N
NlkNakl
1,2,2
)1)(12(cos2
1,1,1
EE465: Introduction to Digital Image Processing 31
11 NN xy C 11
1
NN yx Cforward transform inverse transform
1D DCT
Real and orthogonalTCCCC 1*
fast implementation available: O(Nlog2N)
excellent energy compaction property
DCT is NOT the real part of DFT
Fact The real and imaginary parts of DFT are generally not orthogonal matrices
Properties
EE465: Introduction to Digital Image Processing 32
2D DCTCXCCXCY T
Notice the excellent energy compaction property of DCT
Original cameraman image X Its DCT coefficients Y(2451 significant coefficients, th=64)
EE465: Introduction to Digital Image Processing 33
Summary of 2D Unitary TransformUnitary matrix: A-1=A*T
Unitary transform: A unitaryProperties of 2D unitary transform
Energy compaction: most of transform coefficients yi are small
Energy conservation: quantization can be directly performed to transform coefficients
TAXAY
22 |||||||| XY 22 ||ˆ||||ˆ|| XXYY
EE465: Introduction to Digital Image Processing 34
Lossy Image Compression:a Quick Tour of JPEG Coding Standard
Why lossy for grayscale images? Tradeoff between Rate and Distortion
Transform basics Unitary transform and properties
Quantization basics Uniform Quantization and 6dB/bit Rule
JPEG=T+Q+C T: DCT, Q: Uniform Quantization, C: Run-length
and Huffman coding
EE465: Introduction to Digital Image Processing 35
What is Quantization?In Physics
To limit the possible values of a magnitude or quantity to a discrete set of values by quantum mechanical rules
In image compression To limit the possible values of a pixel value or a
transform coefficient to a discrete set of values by information theoretic rules
EE465: Introduction to Digital Image Processing 36
ExamplesUnlike entropy, we encounter it everyday
(so it is not a monster)Continuous to discrete
a quarter of milk, two gallons of gas, normal temperature is 98.6F, my height is 5 foot 9 inches
Discrete to discrete Round your tax return to integers The mileage of my car is about 55K.
EE465: Introduction to Digital Image Processing 37
A Game Played with BitsPrecision is finite: the more precise, the
more bits you need (to resolve the uncertainty) Keep a card in secret and ask your partner to
guess. He/she can only ask Yes/No questions: is it bigger than 7? Is it less than 4? ...
However, not every bit has the same impact How much did you pay for your car? (two
thousands vs. $2016.78)
EE465: Introduction to Digital Image Processing 38
Scalar vs. Vector QuantizationWe only consider the scalar quantization
(SQ) in this course Even for a sequence of values, we will process
(quantize) each sample independentlyVector quantization (VQ) is the extension
of SQ into high-dimensional space Widely used in speech compression, but not for
image compression
EE465: Introduction to Digital Image Processing 39
Definition of (Scalar) Quantization
x f f -1 xsS
original value
quantizationindex
quantized value
Quantizer
x Q x
f finds the closest (in terms of Euclidean distance) approximation of x from a codebook C (a collection of codewords) and assign its index to s; f-1 operates like a table look-up to return the corresponding codeword
EE465: Introduction to Digital Image Processing 40
Numerical Example (I)
0 1-1
C={-1,0,1}
Three codewords
elsexx
xQ0
5.015.01
)(
S={1,2,3}
1 2 3
red: codewordsgreen: index in C
x
x
0.5-0.5
1
-1
xR
EE465: Introduction to Digital Image Processing 41
Numerical Example (II)
24 408
C={8,24,40,56,72,…,248}
sixteen codewords
]255,0[,1616
8)(
xxxQ
S={1,2,3,…,16}
1 2 3
red: codewordsgreen: index in C
x
x
...
...
248
16
x{0,1,…,255}
16 32
24
40
(note that the scales are different)
8floor operation
EE465: Introduction to Digital Image Processing 42
Numerical Example (III)
225 222 235 228 220 206 209 44 49 56 64 42 128 106 94 27
Q
xx
]255,0[,1616
8)(
xxxQ
232 216 232 232 216 200 216 40 56 56 72 40 136 104 88 24
For scalar quantization, each sample is quantized independently
Notes
Quantization is irreversible
EE465: Introduction to Digital Image Processing 43
Uniform Quantization for Uniform Distribution
Uniform Quantization
A scalar quantization is called uniform quantization (UQ) if all its codewords are uniformly distributed (equally-distanced)
24 408 ... 248
Example (quantization stepsize =16)
Uniform Distribution
elseAAxAxf
0],[2
1)(
-A A
1/2A
x
f(x)denoted by U[-A,A]
EE465: Introduction to Digital Image Processing 44
Quantization Noise of UQ
- /2 /2
1/
e
f(e)
Quantization noise of UQ on uniform distribution is also uniformly distributed
Recall 22
121Variance of U[- /2, /2] is
-A A
EE465: Introduction to Digital Image Processing 45
6dB/bit Rule222
31)2(
121 AAs Signal: X ~ U[-A,A]
22
121eNoise: e ~ U[- /2, /2]
NA2
Choose N=2n (n-bit)codewords for X
)(02.6log20
12
3log10log10 102
2
102
2
10 dBnNA
SNRe
s
N=2n
(quantization stepsize)
EE465: Introduction to Digital Image Processing 46
Lossy Image Compression:a Quick Tour of JPEG Coding Standard
Why lossy for grayscale images? Tradeoff between Rate and Distortion
Transform basics Unitary transform and properties
Quantization basics Uniform Quantization and 6dB/bit Rule
JPEG=T+Q+C T: DCT, Q: Uniform Quantization, C: Run-length
and Huffman coding
EE465: Introduction to Digital Image Processing 47
JPEG Coding Algorithm Overview
EE465: Introduction to Digital Image Processing 48
Transform Coding of ImagesWhy not transform the whole image
together? Require a large memory to store transform
matrix It is not a good idea for compression due to
spatially varying statistics within an imageIdea of partitioning an image into blocks
Each block is viewed as a smaller-image and processed independently
It is not a magic, but a compromise
EE465: Introduction to Digital Image Processing 49
Block-based DCT Example
JI
note that white lines are artificially added to the border of each 8-by-8 block to denote that each block is processed independently
EE465: Introduction to Digital Image Processing 50
Boundary Padding
padded regions
Example
1213141516171819
1213141516171819
1213141516171819
1213141516171819
When the width/height of an image is not the multiple of 8, the boundary isartificially padded with repeated columns/rows to make them multiple of 8
EE465: Introduction to Digital Image Processing 51
Work with a Toy Example
169130173129
170181170183
179181182180
179180179179
169132171130
169183164182
179180176179
180179178178
167131167131
165179170179
177179182171
177177168179
169130165132
166187163194
17611615394
153183160183
Any 8-by-8 block in an image is processed in a similar fashion
EE465: Introduction to Digital Image Processing 52
Encoding Stage I: Transform• Step 1: DC level shifting
169130173129
170181170183
179181182180
179180179179
169132171130
169183164182
179180176179
180179178178
167131167131
165179170179
177179182171
177177168179
169130165132
166187163194
17611615394
153183160183
412451
42534255
51535452
51525151
414432
41553654
51524851
52515050
393393
37514251
49515443
49494051
412374
38593566
48122534
25553655
128 (DC level)
_
EE465: Introduction to Digital Image Processing 53
• Step 2: 8-by-8 DCT
412451
42534255
51535452
51525151
414432
41553654
51524851
52515050
393393
37514251
49515443
49494051
412374
38593566
48122534
25553655
1342
1209
4021
1344
3055
4773
3046
3216
113916
109621
1793310
8101720
10242727
1326078
44131827
273856313
Encoding Step 1: Transform (Con’t)
88DCT
EE465: Introduction to Digital Image Processing 54
Encoding Stage II: Quantization
99103101120
100112121103
98958778
92726449
9211377103
1048110968
64555637
35242218
62805669
87515740
29222416
17141314
55606151
58264024
19141610
12121116
Q-table
8,1
ˆ:
:
1
ji
Qsxf
Qx
sf
ijijij
ij
ijij
: specifies quantization stepsize (see slide #28)
Notes: Q-table can be specified by customerQ-table is scaled up/down by a chosen quality factor Quantization stepsize Qij is dependent on the coordinates (i,j) within the 8-by-8 block Quantization stepsize Qij increases from top-left to bottom-right
EE465: Introduction to Digital Image Processing 55
Encoding Stage II: Quantization (Con’t)
1342
1209
4021
1344
3055
4773
3046
3216
113916
109621
1793310
8101720
10242727
1326078
44131827
273856313
0000
0000
0000
0000
0000
0000
0000
0000
0000
0001
1011
0111
0001
0123
2113
23520
Example
f
xijsij
EE465: Introduction to Digital Image Processing 56
Encoding Stage III: Entropy Coding
Zigzag Scan
0000
0000
0000
0000
0000
0000
0000
0000
0000
0001
1011
0111
0001
0123
2113
23520
(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB)
zigzag scan
End Of the Block:All following coefficients are zero
EE465: Introduction to Digital Image Processing 57
Run-length Coding(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB)
DCcoefficient
ACcoefficient
- DC coefficient : DPCM coding- AC coefficient : run-length coding (run, level)
(5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB)
(0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1),(0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB
Huffman codingencoded bit stream
encoded bit stream
EE465: Introduction to Digital Image Processing 58
JPEG Decoding Stage I: Entropy Decoding
(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB)
Huffman decodingencoded bit stream
AC coefficients
DC coefficient
DPCM decoding
(0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-1),(0,-1),(2,0),(0,1),(0,2),(0,3),(0,-2),(0,1),(0,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB
encoded bit stream
EE465: Introduction to Digital Image Processing 59
JPEG Decoding Stage II: Inverse Quantization
(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,0,0,1,1,0,1,EOB)
0000
0000
0000
0000
0000
0000
0000
0000
0000
0001
1011
0111
0001
0123
2113
23520
zigzag
0000
0000
0000
0000
0000
0000
0000
0000
0000
00040
2902416
0141314
00051
0268072
38141630
243655320
f-1
EE465: Introduction to Digital Image Processing 60
JPEG Decoding Stage III: Inverse Transform
0000
0000
0000
0000
0000
0000
0000
0000
0000
00040
2902416
0141314
00051
0268072
38141630
243655320
169130185127
170181162192
179181181184
179180178183
173128161129
169191175186
174181168186
178181182172
170131177128
170185166187
171187168172
180169169174
175124170120
168193171197
158143148119
153186140195
412571
42533464
51535356
51525055
450331
41634758
46534058
50535444
423490
42573859
43594044
52414146
474438
40654369
3015209
25581267
88IDCT
128 (DC level)
+
EE465: Introduction to Digital Image Processing 61
Quantization Noise
169130173129
170181170183
179181182180
179180179179
169132171130
169183164182
179180176179
180179178178
167131167131
165179170179
177179182171
177177168179
169130165132
166187163194
17611615394
153183160183
169130185127
170181162192
179181181184
179180178183
173128161129
169191175186
174181168186
178181182172
170131177128
170185166187
171187168172
180169169174
175124170120
168193171197
158143148119
153186140195
X X^
MSE=||X-X||2^Distortion calculation:
Rate calculation: Rate=length of encoded bit stream/number of pixels (bps)
EE465: Introduction to Digital Image Processing 62
JPEG Examples
1000
90 (58k bytes)50 (21k bytes)10 (8k bytes)
best quality,
lowest compression
worst quality,
highest compression
EE465: Introduction to Digital Image Processing 63
Wavelet vs. DCT
JPEG (CR=64) JPEG2000 (CR=64)discrete cosine transform based wavelet transform based
EE465: Introduction to Digital Image Processing 64
Image Compression SummaryJPEG=T+Q+C (so is JPEG2000 except T
becomes wavelet transform instead of DCT)
Key idea: sparsity of transform coefficients (most transform coefficients are quantized to zero)
Main weakness: blocky artifacts at low bit rates (many deblocking algorithms have been proposed in the literature)
EE465: Introduction to Digital Image Processing 65
One-Minute SurveyHow much do you think you have
understood today’s lecture? A) >90%, B)70-90%, C)50-70%, D) <50%
How long have you spent on final project so far?
What is the best score you have reached so far? (please indicate your status: G/UG)