![Page 1: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/1.jpg)
Video Compression Concepts
Nimrod PelegUpdate: March. 2009
1
![Page 2: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/2.jpg)
The need for compression• Typical Movie ≈ 150 GBytes
– 100 minutes * 60 sec/min * 30 frames/sec * 525 rows * 525 columns/row * 24 bits/pixel ≈ 1200 Gbits ≈ 150*109 Bytes
• DVD can hold only 4.7 GBytes– Compression is a must!
• Minor noticeable difference
• More than 1:10 for image and up to 1:100 for video
• Compression in usually lossy2
![Page 3: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/3.jpg)
Sample Applications• Most images and video sequences are stored in
compressed form
• Many DSP algorithms are more efficient when implemented in the compressed domain
For example• Face recognition & tracking• Motion detection• Image & video retrieval• Logo insertion• Transcoding
3
![Page 4: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/4.jpg)
Redundancy ⇔ Irrelevancy
• Redundancy
loss
N N+1 N+2
Luminance Histogram
Spatial: pixel-to-pixel or spectral correlation
Temporal: frame-to-frame similarity
Statistical: non-uniformdistribution of data
• Irrelevancy relates to an observer viewing an image
Redundancy + Irrelevancy high compression ratio
4
![Page 5: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/5.jpg)
5
Spatial Redundancy & Irrelevancy
• What is the value of the missing pixel? It is 39.• How critical is it to correctly reproduce it?
![Page 6: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/6.jpg)
6
Contrast Sensitivity Function
The CSF illustrates the limited perceptual sensitivity to high spatial frequencies
![Page 7: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/7.jpg)
7
Visual Masking
![Page 8: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/8.jpg)
8
Visual Masking
![Page 9: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/9.jpg)
9
Video Compression enablers
Video clips are:• Spatially redundant• Statistically redundant• Temporally redundant
Human eyes are:• Less sensitive to high spatial frequencies• Less sensitive to chromatic resolution• Less sensitive to distortions in “busy” areas
Chromatic Modulation Transfer Function
Temporal redundancy
![Page 10: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/10.jpg)
Interframe Compression• The simple way is a frame-by-frame
compression (JPEG-like): Limited compression ratio
• Interframe compression exploits temporal redundancies : similarity between successive frames– In addition to spatial, spectral and psychovisual
redundancies in still image compression)• No Random Access at all frames !
10
![Page 11: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/11.jpg)
Possible Strategies• 3-D Waveform coding
(TC / Sub-Band)• Object/Knowledge based coding• Motion Compensation based coding
11
![Page 12: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/12.jpg)
Motion Compensation Approach
• Segmenting each frame to “changed” and “unchanged” regions (Vs. previous frame)
• Pixels in changed region are encoded after motion compensation process
MotionDetection Estimation Compensation
12
![Page 13: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/13.jpg)
13
Motion Estimation• Estimates the displacement of image structures from one
frame to another, not necessarily true motion• Coding: Difference and Motion Vector (MV)
Reference frame Current Frame
Matchingblock
MotionVector
AB
C
SearchWindow
(dx,dy)
![Page 14: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/14.jpg)
The Technique: Block Matching
14
![Page 15: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/15.jpg)
Motion Detection
• Amount of changed data varies from frame to frame (VLC): a buffer is needed
• A motion detection algorithm should be used
• MC-DPCM is efficient if displacement vectors estimation is accurate !
15
![Page 16: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/16.jpg)
Motion Compensation (Cont’d)• Ideally: motion info for each pixel
– too expensive
• Semantically: motion data for each region or object– second generation coding techniques
• Simplified: motion info • for each 16x16 macroblock
16
![Page 17: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/17.jpg)
Motion Model• Affine motion:
– translation, rotation, scale– 6 parameters in 2D– complex
• Simplified translational model:– motion vectors (MV): 2 parameters
17
![Page 18: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/18.jpg)
ME Options• optimal motion vector?
– investigate all positions within a search window– keep the one with minimum Mean Square Error– MV = corresponding translation
18
![Page 19: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/19.jpg)
Motion Estimation Techniques
The choice of sending zero MV, is always available, with no cost…and worst results.
• Backward prediction– Predict where the pixels in a current frame were
in a past frame• Forward prediction
– Predict where the pixels in a current frame will go to in a future frame
19
![Page 20: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/20.jpg)
Motion Estimation
Block Matching Method• Motion vector is estimated by pixel domain
search procedure• Most popular due to less h/w complexity• Basic idea:
Frame KFrame K+1
Search Window
MotionVector
20
![Page 21: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/21.jpg)
Matching Criteria• Maximum cross-correlation• Maximum pel matching count (MPC)• Minimum mean squared error (MSE)
Not popular in VLSI due to square operation• Sum of absolute difference (SAD)
Very popular but problematic due to several possible local minima
1 1
0 0_ ( , ) _ ( , )
N N
i jSAD curr block i j ref region i j
− −
= =
= −∑∑
21
![Page 22: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/22.jpg)
Motion Estimation - Example
Current blockSAD map
Reference region Best match
22
![Page 23: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/23.jpg)
Other Parameters
• Complexity and implementation
• Visual results ...
23
![Page 24: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/24.jpg)
Full Search & Zero MV results
24
![Page 25: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/25.jpg)
Search Procedure• Full search (inside the search window)• Three step• Conjugate direction
12 pixels
3-Step Conjugate directionSearch order
Best Match
25
![Page 26: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/26.jpg)
3 or 4 Step Search Results
Full search
No search
26
![Page 27: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/27.jpg)
Conjugate Direction Search
27
![Page 28: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/28.jpg)
Search Algorithm results
Algorithm Ave. SNR(dB) CaculationsFull search 28.46 100% (~7M)Zero Search 27.7 0Conjugate 27.9 2%Orthogonal 28.1 3.6%Decimation 28.2 33%Pyramid and more...
28
![Page 29: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/29.jpg)
Motion Vectors Map - Example
And now….
29
![Page 30: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/30.jpg)
MC Transform Coding• Temporal prediction error
is 2D transform coded by segmenting the displaced frame difference into blocks, and encoding the DCT coeff. of each block
• Intra and Inter modes
30
![Page 31: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/31.jpg)
Field and frame modes
31
![Page 32: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/32.jpg)
Video DeInterlacing
32
![Page 33: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/33.jpg)
Other MC Compression Techniques • Vector Quantization:
Prediction error signal encoded by VQ
• Sub-Band Coding:Frame prediction error (residual) is decomposed into 2D subbands
33
![Page 34: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/34.jpg)
Video Compression (Cont’d)
• Complete process:– Encode every N frame “JPEG style”– Between the “JPEG style” frames, predict the
motion from frame to frame– Subtract the predicted frame from the original oneand encode the difference
• Compression ratio: about 100:1 and more
34
![Page 35: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/35.jpg)
Motion Compensation Example
•frame #0 •frame #1
•difference framemotion compensated
difference frame35
![Page 36: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/36.jpg)
Prediction Technique
CurrentFrame
Past Frame FutureFrame
Bidirectional Interpolation36
![Page 37: Video Compression Concepts - University of Haifacs.haifa.ac.il/~nimrod/Compression/Video/V2conspt2009.pdf · 2009. 6. 6. · • More than 1:10 for image and up to 1:100 for video](https://reader035.vdocuments.mx/reader035/viewer/2022062611/612721705315b770be6448e6/html5/thumbnails/37.jpg)
Why Bi-directional reference ?
37