1. Color-plus-Depth Level-of-Detail in 3D Tele-immersive Video: A Psychophysical ApproachWanmin Wu, Ahsan Arefin, Gregorij Kurillo, Pooja Agarwal, Klara Nahrstedt, Ruzena BajcsyUniversity of Illinois at Urbana-ChampaignUniversity of California, BerkeleyACM Multimedia 20111

2. Outline Background (3D Tele-immersion) Motivation Color-plus-Depth Level-of-Detail (CZLoD) Our Psychophysical Study on CZLoD Perception-based reduction of CZLoD Experimental Results Conclusion2 3. Background: 3D Video in Tele-immersion Scottsdale Urbana 3D CapturingInternet3D Capturing Video Streaming3D Visualization 3D Visualization3 4. Challenge Huge ComputationPoor 3D Video Resource DemandPerformanceEach 3D video frame taking low frame rate,too long to processflickering/freezing effects 4 5. MotivationPOOR PERFORMANCE EXAMPLE 5 6. How can we improve 3D video performance?Past approaches: system-centric (algorithmic optimization) 3D Capturing: Depth reconstruction [Wurmlin03][Vasudevan10] Video Streaming: Coordinated data transport protocol [Ott02] 3D Visualization: rendering [Towles02]Our (orthogonal) approach: human-centric (psychophysics)Reduces per-frame computation resource usage by up to 60%AND significantly improves overall video quality6 7. InspirationRaw Image: 108.5 KB JPEG: 9.4 KBHuman vision is limited.Free data reduction in tele-immersive videomay be possible.7 8. Is Free Data Reduction Possible in Tele-immersive Video?HOW TELE-IMMERSIVE VIDEO IS GENERATEDMeshingDepth Mapping Left Left FgndRightMeshBackground + =ColorSubtraction 2D Capture Right+ = 3D Frame Texture Mapping 8 9. Is Free Data Reduction Possible in Tele-immersive Video?HOW TELE-IMMERSIVE VIDEO IS GENERATED Depth TextureMappingMapping Number of: A Critical AccurateAccurateVertex Pixels Metric Mapping MappingNon-vertexLinear InterpolationPixels 9 10. Is Free Data Reduction Possible in Tele-immersive Video? METRIC: Color-plus-Depth Level-of-Detail (CZLoD) = Number of vertices in mesh Accuracy and density of color and depth maps Computation resource usage 22K1K vertices vertices Our idea: keep CZLoD at a minimally necessary level 10 11. Is Free Data Reduction Possible in Tele-immersive Video?HOW MUCH IS MINIMALLY NECESSARYNumber of vertices (in current video frame) CZLoD Degradation Ratio = 1 - Number of vertices (in baseline/best-quality frame) 0%90%10% 80% 20%70%30%60%40% 50% 1 When degradation becomes noticeable 2 When degradation becomes unacceptable11 12. Is Free Data Reduction Possible in Tele-immersive Video? EXPERIMENTAL METHODAscending Method of Limits (Psychophysics) Baseline~10%Baseline~20% Baseline~90%(best) degraded (best) degraded(best) degraded1 Do you notice any difference in quality between the clips? Voting:2 Do you feel any clip has an unacceptable quality? 12 13. Is Free Data Reduction Possible in Tele-immersive Video? RESULTS CZLoD Degradation Ratio 100% Just Unacceptable Degradation Ratio: 80%90%Just Noticeable Degradation Ratio: 60%70% 40% 20%0% Free data reduction in tele-immersive video is indeed feasible. 13 14. Overview of Our Approach UnderstandingApplying thePerceptual Limits Perception Understanding on 3D Tele-Thresholds to Systemimmersive VideoDevelopment 14 15. Applying to System DevelopmentMAIN IDEA Frame RateJust Unacceptable12 fpsJust Noticeable7 fps3 fps0% 60% 70% 80% 90%CZLoD Degradation Ratio15 16. Applying to System DevelopmentARCHITECTURE2D Capture, CZLoD Degradation Ratio Controller Bkgnd Subtraction Control Parameter DecisionFrame Rate MeshingEngine CZLoD Degradation Ratio{unnoticeable, acceptable} 3D Depth QoS Monitors for abnormal Mapping Monitorframe rate, etc. 16 17. Applying to System DevelopmentSYSTEM EVALUATION Average Frame Rate Improvement160% 150%140%120%100% 80% Free Improvement 60% 40% 20%0%imperceptibleperceptible 10% 20% 30%40%50% 60% Average CZLoD Degradation Ratio17 18. Quality AdaptorUSER EVALUATIONAnonymous CrowdsourcingComparison Test(Unimpaired) (Adapted)78 Users TheSlightly Same, 3.8 Better, 12.0%80%MuchBetter, 32. 10%Better, 51. 30% 18 19. Demo 19 20. Conclusion Main Contribution We introduce a psychophysical approach for 3D tele-immersive video that reduces per-frame computation resource usage by up to 60% for free and significantly improves overall perceived video quality.Perception-based degradation ofColor-plus-Depth Level-of-Detail 20 21. Thank YouWanmin Wu:wanmin.wu@gmail.comTEEVE Project:http://cairo.cs.uiuc.edu/projects/teleimmersion21