![Page 1: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/1.jpg)
Stereoscopic Analyzer
On-Set Assistance Systemfor 3D Capturing
Frederik Zilly
![Page 2: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/2.jpg)
The ProductF
red
erik
Zil
ly
With Touchscreen to see at NAB Show 09
![Page 3: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/3.jpg)
The ProductF
red
erik
Zil
ly
With Touchscreen to see at NAB Show 09
![Page 4: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/4.jpg)
Fre
der
ik Z
illy
Outline
• Conventional Workflow in Stereo Production
• The Vision: Automization of Stereo Workflow
• Concept of Stereoscopic Analyzer (STAN)
• Implementation Details
• Applications Beyond Standard Stereo
• Conclusions and Outlook
![Page 5: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/5.jpg)
Mirror Rig
Fre
der
ik Z
illy
Current Workflow of Stereo Production
Manual StereoPost-
Production
Camera 1
Camera 2Vertical disparitiesUnmatched ColoursLens Distortions
Eye strainVisual fatigueHeadache
Viewfinder
Rectified Stereo-PairsMatched Colors
Manual Adaptation of baseline to scene geometry:Measure Distance of Near/Far Clipping Plane
Bad Stereo Good Stereo
![Page 6: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/6.jpg)
Mirror Rig
Fre
der
ik Z
illy
The Vision: Automization of Stereo Workflow
• For Post-Production: supply Meta-Data for offline processing
• For 3DTV: Apply corrections in Real-Time
• Short Feedback Loop for manual and/or automatic reconfiguration
Realtime Optimization
Of Stereo Parameters
Camera 1
Camera 2Bad Stereo Good Stereo
Feedback
![Page 7: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/7.jpg)
3DTV Post-ProductionF
red
erik
Zil
ly
Analyze
Off
line
Co
rrec
tio
n
Meta Data
Stereo Input
3D Viewfinder
Monitor
Control Update
Zconv
PlayoutProxy
![Page 8: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/8.jpg)
Live Stereo for 3DTVF
red
erik
Zil
ly
Analyze
RT Correction
Meta Data
Stereo Input
3D Viewfinder
Monitor
Control Update
Zconv
![Page 9: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/9.jpg)
Analyze Stereo Pair
Control
• Baseline• Color: rough• Focus• Iris• Zoom
Correction
• Rectification• Color: fine• Lens Distortion• Convergence Plane
– Shift / Crop / Scale
Fre
der
ik Z
illy
Monitor
• Stereo Quality• Framing Alert• Vectorscope• Disparity Histogram
![Page 10: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/10.jpg)
Analyze: Fast GPU Feature DetectorF
red
erik
Zil
ly
• Detect Feature Points
• Estimate Fundamental Matrix F
– Apply constraints induced by epipolar geometry to feature points
• Point correspondences:
![Page 11: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/11.jpg)
Monitor: 3D ViewfinderF
red
erik
Zil
ly
Scene Points
Disparity Histogram
1/30 Screen Width
![Page 12: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/12.jpg)
Monitor: Disparity AnalysisF
red
erik
Zil
ly
Near clipping plane warningPossible framing warning
Convergence Plane
Zconv
![Page 13: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/13.jpg)
Correction of Geometrical DistortionsF
red
erik
Zil
ly
• Correct Lens Distortions & Keystones in Real-Time
• For RT-Preview or Live Transmission
• Handle Meta-Data in Machine & Human Readable Format
• Supply Proxy Driver for Post-Production and Playout
![Page 14: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/14.jpg)
3D Viewfinder in DetailF
red
erik
Zil
ly
Actual Baseline
Calc. Baseline
Left shifted Image Right shifted Image
Disparity Range
Zconv
GUI Design by KUK Film Production
![Page 15: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/15.jpg)
3D Viewfinder continuedF
red
erik
Zil
ly
Different View Modes
Color CalibrationWaveformVectorscope
Disparity Histogram
![Page 16: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/16.jpg)
Features under DevelopmentF
red
erik
Zil
ly
• Preview of Depth Information and Scene Depth Structure for Post-Production Purposes
• Real-Time Depth Calculation for Live 3DTV Using Video-Plus-Depth Formats
• Rectified Stereo enables efficient scanline search
• Color Corrected Stereo helps SAD-based algorithms, improves coding efficiency
• Known and/or controlled disparity range makes Depth Estimation faster and more robust
• Enable Zoom• Needed for Live 3DTV, however introducing new challenges
![Page 17: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/17.jpg)
Beyond Stereo: Scenario for Multi Video + DepthF
red
erik
Zil
ly
Mirror Rig
Liv
e R
egis
trat
ion
&C
olo
r C
orr
ecti
on
Camera 1
Camera 2
Sat Cam 1
Sat Cam 2
Live Multi-Video+Depth
BroadcastR
ealt
ime
Dep
th-E
stim
atio
n
Depth estimation more efficient for registered and color corrected data
![Page 18: Stereoscopic Analyzer On-Set Assistance System for 3D Capturing Frederik Zilly](https://reader038.vdocuments.mx/reader038/viewer/2022103023/56649e405503460f94b32017/html5/thumbnails/18.jpg)
Fre
der
ik Z
illy
Conclusions and Outlook
• STAN: Camera Assistance System for 3D Production
• Proof of Concept and Algorithmic Prototype
• NAB‘09: First Presentation to Public
• First Hardware Prototype Planned for NAB‘10
• Extensions under Development
• RT Capture of Video-Plus-Depth
• Application to Multi-View Stereo
• Live Broadcast (incl. Zooming)