slides: real-time spherical videos from a fast rotating camera
DESCRIPTION
Real-Time Spherical Videos from a Fast Rotating Camera http://link.springer.com/chapter/10.1007%2F978-3-540-69812-8_32TRANSCRIPT
![Page 1: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/1.jpg)
Real-Time Spherical Videos from a Fast Rotating Camera
Frank Nielsen, Alexis Andre and Shigeru Tajima
Sony Computer Science Laboratories Inc, Japan
Ecole Polytechnique, France
June 26th , 2008
![Page 2: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/2.jpg)
Outline• Problem• Related Work• Two Prototypes:
–Line Sensor•Workflow•Results
–Area Sensor•Workflow•Results
![Page 3: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/3.jpg)
Three core problems
• We want to generate (1) full spherical (2) videos at (3) high framerates (>25fps).
![Page 4: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/4.jpg)
Main Contribution
• Framework for acquisition of full spherical videos with a special tailored fast-rotating camera
• High quality smooth image (no seams)
![Page 5: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/5.jpg)
Problem 1: Full-Spherical Images
• Two traditional ways:–Stitching of a small number of images
taken with a fish eye lens (Quicktime VR® )
–Succession of 1-pixel wide vertical strips (Spheron™ )
![Page 6: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/6.jpg)
Inherent problems
• Use of step motors for automatic acquisition– Slow– Need to rewind (avoid cable twist)
• Rough calibration is ok: a slight seam is acceptable (for discrete stitch)
![Page 7: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/7.jpg)
Problem 2: Full-Spherical Videos
• Use of multiple cameras (Sony Fourth View)– Parallax problems
• Virtual panoramic videos (Agarwala et al.)– Wide field of view– Very low frame rate (wrt. the rotation speed)
![Page 8: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/8.jpg)
Our approach
• Unique camera, rotating very fast
(1800rpm – 30fps)– No parallax problems (MCOP image)– No calibration between various cameras
• Connection– Slip rings -> no need for a step motor– Gigabit Ethernet connection with remote control
![Page 9: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/9.jpg)
Challenge
• 1D rotation angles unknown:– Register the data: we need to find the
rotation angle:
– Then, extract frames every revolution
θ(t) ?
θ(t) = t mod(2π)
![Page 10: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/10.jpg)
System Setup
![Page 11: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/11.jpg)
Slip rings for Ethernet Connection
![Page 12: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/12.jpg)
Camera Platform
![Page 13: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/13.jpg)
Movie No parallax barrier
![Page 14: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/14.jpg)
Line Area Sensor
• Black and white CCD • very fast sensor (67000 lines per second)• Smooth spherical image
(not unique center of projection, MCOP)
-> calibration not required
• Only 1/3 of the vertical axis (lens/sensor)• Choosing appropriate lens (tailored lens)
![Page 15: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/15.jpg)
Raw Data: Plain unrolling
Acquisition speed may not be constant
![Page 16: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/16.jpg)
Raw Data
![Page 17: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/17.jpg)
Raw Data
One round One Round
![Page 18: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/18.jpg)
Frame Registration
• Based on current data and past data
• We look for similar bands of pixels
(Dynamic programming-like search)– Only works if most of the scene is static
![Page 19: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/19.jpg)
Registration
SSD
Score
ReferencePoint
CurrentPoint
![Page 20: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/20.jpg)
Registration
SSD
Score
ReferencePoint
CurrentPoint
![Page 21: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/21.jpg)
Registration
SSD
Score
ReferencePoint
CurrentPoint
![Page 22: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/22.jpg)
Registration
SSD, or normalized mutual information, etc.
![Page 23: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/23.jpg)
Registration
Minimum: One round!
![Page 24: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/24.jpg)
Registration
Best Match
ReferencePoint
CurrentPoint
![Page 25: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/25.jpg)
Registration
NextPoint Search Zone (expand)
![Page 26: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/26.jpg)
Registration: Frame Extraction
Yields potential for super-resolution (non-uniform sampling)
![Page 27: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/27.jpg)
Result movie (cropped vertically)
![Page 28: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/28.jpg)
Results
• Rotation speed might not be uniform across one revolution:– Slight jittering/tripod
• Changes in speed affect resolution– In a production setting, speed must be controlled
carefully.
![Page 29: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/29.jpg)
Issues
• Not robust to dynamic scenes
• Fails on homogenous scenes (ambiguity)
• Use of a small visual hint help registration– Bright LED at a specific position (visual pollution)
![Page 30: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/30.jpg)
Area Sensor
• Align the nodal point with the rotation axis to avoid parallax (similar to spherical images problems)
• 24-bit color VGA frames• horizontal resolution fixed• need to control shutter time and/or rotation
speed to get enough overlap
Key difference: Camera can be calibrated exactly on its nodal point(multi-camera cannot)
![Page 31: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/31.jpg)
“Traditional Approach”
• Stitch frames: need to register each frame with the previous ones
• Horizontal shift depends on the rotation speed and on the shutter time
• But resolution is fixed (programmable ROI)
![Page 32: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/32.jpg)
Workflow for 2D stitching
![Page 33: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/33.jpg)
Registration for two frames
![Page 34: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/34.jpg)
Defish (=map to env. map.)
![Page 35: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/35.jpg)
Registration: crude w/o blending
Actually perform only this in coordinate systems (no intermediate images)
Obvious seam
![Page 36: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/36.jpg)
Smooth blending
Better methods: Laplacian multi-spectral, Poisson blending, etc.(but we are concerned with real-time systems…)
![Page 37: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/37.jpg)
Panoramic roll: never-ending sequence
Artefact comes from the model of fish-eye projection. Calibrate rayel!
![Page 38: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/38.jpg)
Results (movie)
![Page 39: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/39.jpg)
Inherent Problems
• Lens imperfections / resolution decrease near the edges of the lens
• Perfect fisheye model or calibration is needed• Motion Blur
![Page 40: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/40.jpg)
Imperfect Fisheye Model
• Slight seam– On static images: can be ignored
(error propagated to the edges of the image)– On movies: disturbing moving seam
![Page 41: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/41.jpg)
Motion Blur
• Lighting conditions force us to use long exposure time (wrt. the hardware):– Motion blur appears– Ideally horizontal motion blur– At the desired speeds, unrecoverable blur
![Page 42: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/42.jpg)
(horizontal) Motion blur
0rpm 150rpm 250rpm
![Page 43: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/43.jpg)
Summary
• Path to high-frame rate full-spherical videos is promising– Better hardware is needed– H/W technologies is already here
• Classical framework– Established techniques– Optimization available
![Page 44: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/44.jpg)
Future Work
• Horzontal deblur
• Registration in more dynamic scenes
• ROI selection
• Applications for AI applications
![Page 45: Slides: Real-Time Spherical Videos from a Fast Rotating Camera](https://reader033.vdocuments.mx/reader033/viewer/2022052823/5553ffcfb4c90577468b4d69/html5/thumbnails/45.jpg)
ETVC08Emerging Trends in
Visual Computing
Information geometry
Computer vision
Computer graphics
Machine learning
Computational Geometry
Registration is free