video coding with spatio -temporal texture synthesis and edge-based inpainting
DESCRIPTION
Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting. Chunbo Zhu , Xiaoyan Sun , Feng Wu , and Houqiang Li. ICME 2008. Outline. Introduction Framework Previous work Encoder Decoder Experiment Result Conclusion. Introduction. Texture synthesis - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/1.jpg)
Video Coding with Spatio-temporal Texture Synthesis and Edge-based inpainting
Chunbo Zhu , Xiaoyan Sun , Feng Wu , and Houqiang Li
ICME 2008
![Page 2: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/2.jpg)
Outline
•Introduction•Framework
▫Previous work▫Encoder▫Decoder
•Experiment Result•Conclusion
![Page 3: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/3.jpg)
Introduction
•Texture synthesis•Image/video inpainting
![Page 4: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/4.jpg)
Introduction
Encoder
Decoder
[1] D. Liu, X. Sun, F. Wu, S. Li, and Y.-Q. Zhang, “Image compression with edge-based inpainting,” IEEE Transactions on Circuits and Systems for Video Technology, Oct. 2007.
Original image
Edge map Removed blocks
Structure propagation
Texture synthesis
JPEG
![Page 5: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/5.jpg)
Introduction
•Texture regions can be well synthesized.▫Selectively removed during encoding.▫Restored by texture synthesis and edge-
based inpainting in the decoder.•Assistant information can be extracted to
guide restoration.▫Region types, edge information, motion
parameters. Keep spatio-temporal consistency
![Page 6: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/6.jpg)
Framework—previous work
[2] C. Zhu, X. Sun, F. Wu, and H. Li, “Video coding with spatio-temporal texture synthesis,” IEEE International Conference on Multimedia and Expo, 2007.
I B PBB …
![Page 7: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/7.jpg)
Framework—encoder
I B PBB …
Local Motion Region
Global Motion Region
Same as previous work
![Page 8: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/8.jpg)
Framework—encoder
![Page 9: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/9.jpg)
Structure blocks
Textureblocks
Block Categorization
Edge detection
Block categorizati
on
Exemplar selection
examplars
Check connective degreeremoved blocks do not constitute a large region
![Page 10: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/10.jpg)
Framework—encoder
![Page 11: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/11.jpg)
Motion Threading
[3] L. Luo, F. Wu, S. Li, Z. Xiong, and Z.Q. Zhuang, “Advanced motion threading for 3D wavelet video coding,” Signal Processing: Image Communication, Vol. 19, Issue 7, pp. 601-616, Aug. 2004.
•Spatial and temporal consistency▫Block-based backward motion estimation▫Motion threads in texture regions are pruned
B0 B2 B3 B4 B5B1
![Page 12: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/12.jpg)
Exemplar Selection• All the motion threads are 3-D exemplar
candidates• Spatio-temporal variation(SVT)
N : length of a thread which consists of N blocks Bi(i=1…N) : spatio-temporal 6-neighboring blocks of each block : variance of a blockE() : mean pixel value of a block
Choose the threads with higher variation
Output a sequence of binary masks indicating which blocks are removed
![Page 13: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/13.jpg)
Framework—encoder
![Page 14: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/14.jpg)
Sprite Generation•GMR
▫pixel motion can be described by a parametric geometrical model
•Sprite (background mosaic)▫A sprite for a GOP
[4] Y. Lu, F. Wu, S. Li, and Y.-Q. Zhang, “Efficient background video coding with static sprite generation and arbitrary-shape spatial prediction techniques,” IEEE Transactions on Circuits and Systems for Video Technology, May 2003.
(x, y) and (x’,y’) are coordinates of pixels in original frame and reference image
![Page 15: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/15.jpg)
Framework—encoder
![Page 16: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/16.jpg)
removable blocksunremovabl
e blocks
Region removal in GMR
• Mapping into original frame inverse warping
• if one 8x8 block in an original frame contains more than half of the removable pixels, it is selected as a removable block and then skipped when encoding. Otherwise, it is selected as an exemplar and coded with H.264 encoder.
Edge Detectio
n
Region Classificatio
n
Mapping
Exemplar SelectionR
SRF
SPT
RS: Removable blocks in spriteRF: Removable blocks in original frame
![Page 17: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/17.jpg)
Framework—decoderMask1: region mapMask2: removing map
![Page 18: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/18.jpg)
Decoder—Spatio-temporal texture synthesis in LMR•Texture synthesis module
▫Arbitrary-shaped regions▫Perform frame by frame▫Square patch▫Confidence map
Similarity of candidate patch and target patch𝑆=𝑆𝑆𝐷 (𝑊 𝑡 ,𝑊𝑐 )+𝛼 ∙𝑆𝑆𝐷 (𝑊 𝑡 ′ ,𝑊 𝑐)
Squared difference of known pixels
![Page 19: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/19.jpg)
Framework—decoder
![Page 20: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/20.jpg)
Decoder—Edge-based inpainting in GMR
Warping
Blending
Edge inpaintin
g
Texture Synthesis
GMR
GMR parameter
SPT edges
SPTH
SPT
![Page 21: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/21.jpg)
Experimental Results• Implemented into JM 10.2 of H.264• YUV 4:2:0 sequence format with CIF
resolution(352x288)• rate distortion optimization (RDO) and CABAC were
turned on
![Page 22: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/22.jpg)
Experimental Results
![Page 23: Video Coding with Spatio -temporal Texture Synthesis and Edge-based inpainting](https://reader030.vdocuments.mx/reader030/viewer/2022032805/56813399550346895d9aa5e8/html5/thumbnails/23.jpg)
Conclusion• In this paper, we present a video coding
scheme in which some regions are removed in the encoder and restored in the decoder by spatio-temporal texture synthesis and edge-based image inpainting.
• To keep temporal consistency, different types of motion have been considered in region removal and restoration for both textural and structural regions.