multiview video
Post on 21-Jan-2016
85 Views
Preview:
DESCRIPTION
TRANSCRIPT
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 1
Multiview Video
Kai-Chao Yang
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 2
Outline
History of Video Coding Standards Definition of Multiview Video Applications of Multiview Video Concept of Stereo Video Multiview TV/Video System
Multiview Content Capture Correction Coding Display
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 3
History of Video Coding Standards
H.264/MPEG4 AVCH.262/MPEG-2
1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010
H.261 H.263 H.263+ H.263++
MPEG-1 MPEG-4 v2/visual
SVC MVC
ISOMPEG
JVT
ITU-TVCEG
FRExt
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 4
Definition of Multiview Video
Multiview video Multiple cameras are used to simultaneously acquire va
rious viewpoints of a scene.
Multiview video coding Encoding of sequences captured simultaneously from m
ultiple cameras using a single video stream.
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 5
Applications of Multiview Videos
Free viewpoint TV The viewers can experience the free viewpoint navigatio
n within the range covered by the cameras http://www.youtube.com/watch?v=0yP_J6M4fiU
Three-dimensional TV
Immersive teleconference
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 6
Examples (1/4)
Entertainments 3D game, FVV, …
Medicine 3D microscope, 3D endoscope, …
Education 3D model, 3D classroom, …
Surveillance Nurse, object recognition and tracking, …
Simulation Flight simulation, …
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 7
Examples (2/4)
3D microscope and 3D endoscope
http://www.njneurosurgeons.com/news/2010/04/19/new-jerseys-first-3d-endoscopic-surgery-performed-by-ninj-faculty.html
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 8
Examples (3/4)
Pedestrian Tracking
Kyungnam Kim and Larry S. Davis, "Multi-Camera Tracking and Segmentation of Occluded People on Ground Plane using Search-Guided Particle Filtering", European Conference on Computer Vision (ECCV), LNCS, 2006.
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 9
Examples (4/4)
Flight Simulation
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 10
Concept of Stereo Image/Video
3D information: Determination of relative depths in a perceived scene Stereopsis Accommodation of the eyeball Occlusion
Linear perspective
Vertical position
…
2D knowledge
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 11
Concept of Stereo Video (3D Video)
Stereopsis visual perception leading to the sensation of de
pth from the two slightly different projections of the world onto the retinas of the two eyes
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 12
Concept of Stereo Video
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 13
Multiview TV System
Content Delivery DisplayEncoding Decoding
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 14
Multiview Content
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 15
Multiview Content
Camera capture Stereo cameras Camera array
Computer generated Objects is defined Depth is inherent
Conversion from 2D video Detecting objects Assigning depth Filling occluded parts
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 16
Camera Capture
Stanford multi-camera array (128 video cameras)
Google street view camera (Image only)
Panasonic Full-HD 3D camera
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 18
Correction
Rectification of misalignment
Normalization of colors
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 19
Multiview Video Coding
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 20
Representation Formats (Full-Resolution)
Full-Resolution Stereo and Multiview Double data rate for stereo videos N-fold data rate for N-view videos
Efficient compression is the key issue MVC extension of H.264/AVC
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 21
Representation Formats (Stereo Interleaving) Stereo Interleaving
A multiplex of the two views into a single sequence Spatial multiplexing
The left and right views are sub-sampled and interleaved into a single frame
Temporal multiplexing The left and right views are interleaved as alternating frames
Advantages Compatible with existing codecs and delivery infrastructure
Drawbacks Loss of spatial or temporal resolution
RLRL
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 22
Representation Formats (Depth-based) Depth-based Format
2D video + depth map
Advantages Backward compatibility with the older coding standards Supporting both stereo and multiview displays Depth is adjustable
Drawbacks Limited depth range Occlusions Other views have to be synthesized
2D viewSynthesized
view
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 23
Multiview Video Coding
Single view
Multi-view Independent encoding
Low coding complexity but also low coding efficiency Inter-view prediction
Exploiting both spatial and temporal redundancy
I PB B B B B B P
I PB B B B B B P
I PB B B B B B P
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 24
Multiview Video Coding
Multiview camera
PB
BB
PB
BB
I
PB
BB
PB
BB
B
PB
BB
PB
BB
P
PB
BB
PB
BB
I
PB
BB
PB
BB
B
PB
BB
PB
BB
P
PB
BB
PB
BB
I
PB
BB
PB
BB
B
PB
BB
PB
BB
P
B
I B P
P
PB
P
P
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 25
View Generation
View 1 View 2synthesis
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 2626
View Generation
View synthesis Disparity-based
e.g. view1view3/2 view2 object(0,0) object(3,0) object(6,0)
Depth-based 2D + depth map
http://www.imec.be/ScientificReport/SR2007/html/1384302.html
DEPTH MAPS EXTRACTION FROM MULTI-VIEW VIDEOShttp://www.youtube.com/watch?v=KtRSbey1sKM
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 27
View Generation
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 28
Display (3D space)
Volumetric video
Holographic video
http://www.youtube.com/watch?v=kIDgC2no1uo
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 29
Display
Free viewpoint video
http://www.youtube.com/watch?v=vyhz8KgW49E http://www.youtube.com/watch?v=GSumx0Zs2XA
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 30
Display with 3D Glasses
Stereo video with glasses Left view and right view
Passive and active
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 31
Display with 3D Glasses
Anaglyph, polarized, and shutter glasses
http://www2.ciw.com.cn/h/2562/357866-17902.html
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 32
Display
Lenticular panel and Parallax barrier
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 33
Display
Stereo view display without glasses
A1A1A1A1A1A1A1A1A1A1
A2A2A2A2A2A2A2A2A2A2
Pixel 2
Pixel 1
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 34
Display
Multiview display without glasses
A2A2A2A2
A1A1A1A1B1B1B1B1C1C1C1C1
B2B2B2B2C2C2C2C2
Pixel 2
Pixel 1
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 35
Conclusions
An overview of multiview video from capture to display.
Issues Display
Reduction of brightness Reduction of refresh rate Reduction of temporal of spatial resolution
Coding efficiency Decoding delay 2D 3D View synthesis …
2010/10/13 VCLAB, National Tsing Hua University, Taiwan 36
Reference
A. Vetro, “Representation and Coding Formats for Stereo and Multiview Video,” Tech. Rep. TR2010-011, April, 2010.
S. Gaël, “Depth Map Estimation and Use for 3DTV,” Tech. Rep. n0379, February, 2010.
Y.-S. Ho and K.-J. Oh, “Overview of Multi-view Video Coding,” IWSSIP, 2007.
許精益 and 黃乙白 “ 3D 立體顯示技術之發展與研究 ,” 光學工程第 98 期 , 96 年 6 月 .
top related