resolution enhancement of low-quality videos using a high-resolution frame
Post on 08-May-2015
794 Views
Preview:
DESCRIPTION
TRANSCRIPT
Resolution enhancement of
low-quality videos using a
high-resolution frame
Tuan Pham 1
Lucas van Vliet 1
Klamer Schutte 2
1. Quantitative Imaging Group, Delft University of Technology, The Netherlands
2. TNO Physics and Electronics Laboratory, The Netherlands
Visual Communications and Image Processing 2006
SPIE paper 6077-8
© Tuan Pham (pham@tnw.tudelft.nl) 2
Contents
1. Influence of compression on Super-Resolution
2. Example-based SR in the DCT domain
3. Application in video compression
© Tuan Pham (pham@tnw.tudelft.nl) 3
Super-Resolution reconstruction
•Problems with compressed inputs:
– Registration: less accurate motion vectors due to compression error
– Fusion: outlier intensities due to ringing and blocking artifacts
– Deblur: distorted and truncated frequency spectrum due to
quantization
© Tuan Pham (pham@tnw.tudelft.nl) 4
•Each 8x8 image block is projected onto 64 orthogonal bases
→ 64 DCT coefficients per block
Discrete Cosine Transform
Input image 64 DCT basis functions DCT coefficients
•The coefficients are then quantized (lossy) & compressed (lossless)
© Tuan Pham (pham@tnw.tudelft.nl) 5
Quantization noise and blur
Quantization noiseJPEG quality 80
24
68
24
684
5
6
7
8
xy
MS
E o
f all
8x8
bloc
ks
Average error variance of 8x8 blocks
Average DCT attenuation factor
© Tuan Pham (pham@tnw.tudelft.nl) 6
SR by texture synthesis
Hi-Res texture source
Interpolation
Multi-frame fusion
or
Texture transfer
Hi-Res blurred intermediate
video
Super-Resolved output video
Lo-Res compressed input video
•Example-based SR1 aims to fill-in the missing frequencies:
1. Freeman et al., “Example-based Super-Resolution”, IEEE Comp.Graph. & App., 2001
•We propose a DCT-domain SR synthesis algorithm which
can be applied directly on the compressed video stream.
© Tuan Pham (pham@tnw.tudelft.nl) 7
Resize versus SR in DCT domain
•Super-resolution in DCT domain:
– fill-in plausible high-frequency coefficients instead of zero-ing
1. Mukherjee and Mitra, “Image resizing in the compressed domain using subband DCT”, CSVT, 2002
•Image resize1 in DCT domain:
– zero filling of high-frequency content followed by transcoding
transcodeenlarge
0 fill-in
8x8DCT
LR DCT
16x16 DCT 2x HR DCT
0
0 0
8x8DCT
8x8DCT
8x8DCT
8x8DCT
8x8DCT
© Tuan Pham (pham@tnw.tudelft.nl) 8
Example-based SR in DCT domain
MeanAbs + ε
8x8 quantized DCT
15 LR DCTs 33 overlapping pixels
DC
Best Match
Training data
SR decoded image
+
AC
÷
×α
×
•Block-wise SR synthesis in a raster-scan order:
– Correspondence constraint: low-freq. coefficients of LR - HR blocks should match
– Spatial continuity constraint: boundary pixels of adjacent HR blocks should match
© Tuan Pham (pham@tnw.tudelft.nl) 9
DCT-domain SR
Texture source: CIF frame 0
DCT-domain SR synthesis
+
Input QCIF frame 20 (JPEG quality 80)
Example-based SR (Freeman)
vs spatial domain SR
© Tuan Pham (pham@tnw.tudelft.nl) 10
DCT-domain SR vs spatial domain SR
Texture source: CIF frame 0
DCT-domain SR
+
Input QCIF frame 50 (JPEG quality 80)
Local Linear Embedding SR (CVPR’04)
© Tuan Pham (pham@tnw.tudelft.nl) 11
Importance of a good texture source
input (JPEG quality 50) source from 30-frame away
wrong source
QCIF input good but not striking poor SR result
+
© Tuan Pham (pham@tnw.tudelft.nl) 12
•LR compressed video + a regularly updated HR source
SR using updated HR frames
QCIF input at JPEG quality 80 CIF output using 1 HR frame / sec
•Application: video coding for devices with limited resources
© Tuan Pham (pham@tnw.tudelft.nl) 13
Conclusions
•SR reconstruction of compressed image is difficult because of:
– truncated frequency spectrum due to heavy quantization
– space-variant quantization noise
•DCT-domain SR is better than spatial domain SR because it:
– produces better results
– is more efficient
•Application of SR synthesis using a HR frame:
– video compression for systems with limited resources (phones, cameras)
© Tuan Pham (pham@tnw.tudelft.nl) 14
Tuan Pham
pham@tnw.tudelft.nl
Thank you !
top related