1

3D Tele-Collaboration3D Tele-Collaboration

over Internet2over Internet2

Herman Towles, UNC-CH Herman Towles, UNC-CH

representing members of therepresenting members of the

National Tele-Immersion Initiative (NTII)National Tele-Immersion Initiative (NTII)

ITP 2002ITP 2002

Juan-les-Pins, FranceJuan-les-Pins, France

06 December 200206 December 2002

3D Tele-Collaboration3D Tele-Collaboration

over Internet2over Internet2

Herman Towles, UNC-CH Herman Towles, UNC-CH

representing members of therepresenting members of the

National Tele-Immersion Initiative (NTII)National Tele-Immersion Initiative (NTII)

ITP 2002ITP 2002

Juan-les-Pins, FranceJuan-les-Pins, France

06 December 200206 December 2002

2

NTII Collaborators & Co-NTII Collaborators & Co-authorsauthors

• University of North Carolina at Chapel University of North Carolina at Chapel HillHill– Wei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and Henry FuchsWei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and Henry Fuchs

• University of PennsylvaniaUniversity of Pennsylvania– Nikhil Kelshikar, Jane Mulligan, and Kostas DaniilidisNikhil Kelshikar, Jane Mulligan, and Kostas Daniilidis

• Brown UniversityBrown University– Loring Holden, Bob Zeleznik, and Andy Van DamLoring Holden, Bob Zeleznik, and Andy Van Dam

• Advanced Network & ServicesAdvanced Network & Services– Amela Sadagic and Jaron LanierAmela Sadagic and Jaron Lanier

• University of North Carolina at Chapel University of North Carolina at Chapel HillHill– Wei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and Henry FuchsWei-Chao Chen, Ruigang Yang, Sang-Uok Kum, and Henry Fuchs

• University of PennsylvaniaUniversity of Pennsylvania– Nikhil Kelshikar, Jane Mulligan, and Kostas DaniilidisNikhil Kelshikar, Jane Mulligan, and Kostas Daniilidis

• Brown UniversityBrown University– Loring Holden, Bob Zeleznik, and Andy Van DamLoring Holden, Bob Zeleznik, and Andy Van Dam

• Advanced Network & ServicesAdvanced Network & Services– Amela Sadagic and Jaron LanierAmela Sadagic and Jaron Lanier

3

Clear Motivation to Provide Clear Motivation to Provide

• Higher ResolutionHigher Resolution

• Larger, more immersive Field-of-ViewLarger, more immersive Field-of-View

• Participants at Accurate Geometric ScaleParticipants at Accurate Geometric Scale

• Eye Contact Eye Contact

• Spatialized Audio (Group settings)Spatialized Audio (Group settings)

• More Natural Human-Computer InterfacesMore Natural Human-Computer Interfaces

• Higher ResolutionHigher Resolution

• Larger, more immersive Field-of-ViewLarger, more immersive Field-of-View

• Participants at Accurate Geometric ScaleParticipants at Accurate Geometric Scale

• Eye Contact Eye Contact

• Spatialized Audio (Group settings)Spatialized Audio (Group settings)

• More Natural Human-Computer InterfacesMore Natural Human-Computer Interfaces

4

Related Work Related Work

• Improved Resolution & FOVImproved Resolution & FOV– Access Grid – Childers et al., 2000Access Grid – Childers et al., 2000– Commerical, multi-channel extensions of ‘Commerical, multi-channel extensions of ‘1-camera to 1-display1-camera to 1-display’’

• Gaze-AwarenessGaze-Awareness– MONJUnoCHIE System – Aoki et al., 1998MONJUnoCHIE System – Aoki et al., 1998– Blue-C Project - Kunz and Spagno, 2001-2002Blue-C Project - Kunz and Spagno, 2001-2002– VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002

• 3D Reconstruction/New Novel Views3D Reconstruction/New Novel Views– CMU’s Virtualized Reality Project – Narayanan, Kanade, 1998CMU’s Virtualized Reality Project – Narayanan, Kanade, 1998– Visual Hull Methods – Matusik, McMillan et al, 2000Visual Hull Methods – Matusik, McMillan et al, 2000– VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002

• Human Computer InterfacesHuman Computer Interfaces– T-I Data Exploration (TIDE) – Leigh, DeFanti et al., 1999T-I Data Exploration (TIDE) – Leigh, DeFanti et al., 1999– VisualGlove Project - Constanzo, Iannizzotto, 2002VisualGlove Project - Constanzo, Iannizzotto, 2002

• Improved Resolution & FOVImproved Resolution & FOV– Access Grid – Childers et al., 2000Access Grid – Childers et al., 2000– Commerical, multi-channel extensions of ‘Commerical, multi-channel extensions of ‘1-camera to 1-display1-camera to 1-display’’

• Gaze-AwarenessGaze-Awareness– MONJUnoCHIE System – Aoki et al., 1998MONJUnoCHIE System – Aoki et al., 1998– Blue-C Project - Kunz and Spagno, 2001-2002Blue-C Project - Kunz and Spagno, 2001-2002– VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002

• 3D Reconstruction/New Novel Views3D Reconstruction/New Novel Views– CMU’s Virtualized Reality Project – Narayanan, Kanade, 1998CMU’s Virtualized Reality Project – Narayanan, Kanade, 1998– Visual Hull Methods – Matusik, McMillan et al, 2000Visual Hull Methods – Matusik, McMillan et al, 2000– VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002VIRTUE Project – Cooke, Kauff, Schreer et al., 2000-2002

• Human Computer InterfacesHuman Computer Interfaces– T-I Data Exploration (TIDE) – Leigh, DeFanti et al., 1999T-I Data Exploration (TIDE) – Leigh, DeFanti et al., 1999– VisualGlove Project - Constanzo, Iannizzotto, 2002VisualGlove Project - Constanzo, Iannizzotto, 2002

5

XTP: ‘Xtreme Tele-PresenceXTP: ‘Xtreme Tele-Presence

UNC ‘Office of the Future’

Andrei State 1998

6

Research SnapshotsResearch Snapshots

7

Presentation Outline Presentation Outline

• Motivation and Related WorkMotivation and Related Work

• NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed– Acquisition and 3D ReconstructionAcquisition and 3D Reconstruction

– Collaborative Graphics & User InterfacesCollaborative Graphics & User Interfaces

– Rendering & DisplayRendering & Display

– Network Network

• ResultsResults

• Future ChallengesFuture Challenges

• Motivation and Related WorkMotivation and Related Work

• NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed– Acquisition and 3D ReconstructionAcquisition and 3D Reconstruction

– Collaborative Graphics & User InterfacesCollaborative Graphics & User Interfaces

– Rendering & DisplayRendering & Display

– Network Network

• ResultsResults

• Future ChallengesFuture Challenges

8

Scene Acquisition & Scene Acquisition & Reconstruction Reconstruction

• Foreground: Real-Time Stereo Algorithm Foreground: Real-Time Stereo Algorithm – Frame Rate: 2-3 fps (550MHz Quad-CPU) - Frame Rate: 2-3 fps (550MHz Quad-CPU) - REAL-TIME!REAL-TIME!

– Volume: 1 cubic meterVolume: 1 cubic meter

– Resolution: 320x240 (15K-25K foreground points)Resolution: 320x240 (15K-25K foreground points)

• Background: Scanning Laser Background: Scanning Laser Rangefinder Rangefinder – Frame Rate: 1 frame in 20-30 minutes - Frame Rate: 1 frame in 20-30 minutes - OFFLINE!OFFLINE!

– Volume: Room-sizeVolume: Room-size

– Resolution: More data than you can handle!Resolution: More data than you can handle!

Composite Composite

Live Foreground & Static BackgroundLive Foreground & Static Background

• Foreground: Real-Time Stereo Algorithm Foreground: Real-Time Stereo Algorithm – Frame Rate: 2-3 fps (550MHz Quad-CPU) - Frame Rate: 2-3 fps (550MHz Quad-CPU) - REAL-TIME!REAL-TIME!

– Volume: 1 cubic meterVolume: 1 cubic meter

– Resolution: 320x240 (15K-25K foreground points)Resolution: 320x240 (15K-25K foreground points)

• Background: Scanning Laser Background: Scanning Laser Rangefinder Rangefinder – Frame Rate: 1 frame in 20-30 minutes - Frame Rate: 1 frame in 20-30 minutes - OFFLINE!OFFLINE!

– Volume: Room-sizeVolume: Room-size

– Resolution: More data than you can handle!Resolution: More data than you can handle!

Composite Composite

Live Foreground & Static BackgroundLive Foreground & Static Background

9

Real-Time Foreground Real-Time Foreground AcquisitionAcquisition

• Trinocular Stereo Reconstruction Trinocular Stereo Reconstruction AlgorithmAlgorithm– After background segmentation, find corresponding After background segmentation, find corresponding

pixels in each image using MNCC methodpixels in each image using MNCC method

– 3D ray intersection yields pixel depth3D ray intersection yields pixel depth

– Median filter the disparity map to reduce outliersMedian filter the disparity map to reduce outliers

• Produce 320x240 Depth Maps (1/z, Produce 320x240 Depth Maps (1/z, R,G,B) R,G,B)

• Trinocular Stereo Reconstruction Trinocular Stereo Reconstruction AlgorithmAlgorithm– After background segmentation, find corresponding After background segmentation, find corresponding

pixels in each image using MNCC methodpixels in each image using MNCC method

– 3D ray intersection yields pixel depth3D ray intersection yields pixel depth

– Median filter the disparity map to reduce outliersMedian filter the disparity map to reduce outliers

• Produce 320x240 Depth Maps (1/z, Produce 320x240 Depth Maps (1/z, R,G,B) R,G,B)

=

Images courtesy of UPenn GRASP Lab

+

10

UNC Acquisition ArrayUNC Acquisition Array

Five Dell 6350 Quad-Processor

ServersSeven Sony Digital 1394 Cameras – Five Trinocular

Views

11

Stereo Processing Stereo Processing SequenceSequence

Camera Views

Disparity Maps

3 Views ofCombined

Point CloudsImages courtesy of UPenn GRASP Lab

12

Collaborative Graphics & User Collaborative Graphics & User I/F I/F

13

Shared 3D ObjectsShared 3D Objects

• Scene Graph SharingScene Graph Sharing– Distributed, Common Scene Distributed, Common Scene

Graph DatasetGraph Dataset

– Local Changes, Shared Local Changes, Shared Automatically with Remote Automatically with Remote NodesNodes

• Object Manipulation Object Manipulation with 2D & 3D with 2D & 3D PointersPointers

– 3D Virtual Laser Pointing 3D Virtual Laser Pointing DeviceDevice

– Embedded magnetic trackerEmbedded magnetic tracker

– Laser beam rendered as part Laser beam rendered as part of Scene Graphof Scene Graph

– One event/behavior buttonOne event/behavior button

• Scene Graph SharingScene Graph Sharing– Distributed, Common Scene Distributed, Common Scene

Graph DatasetGraph Dataset

– Local Changes, Shared Local Changes, Shared Automatically with Remote Automatically with Remote NodesNodes

• Object Manipulation Object Manipulation with 2D & 3D with 2D & 3D PointersPointers

– 3D Virtual Laser Pointing 3D Virtual Laser Pointing DeviceDevice

– Embedded magnetic trackerEmbedded magnetic tracker

– Laser beam rendered as part Laser beam rendered as part of Scene Graphof Scene Graph

– One event/behavior buttonOne event/behavior button

14

Rendering System Rendering System Overview Overview

15

3D Stereo Display3D Stereo Display

• Passive Stereo & Circular PolarizationPassive Stereo & Circular Polarization– Custom Filters on ProjectorsCustom Filters on Projectors

– Lightweight GlassesLightweight Glasses

– SilveredSilvered Display Surface Display Surface

• Front ProjectionFront Projection– Usable in any office/roomUsable in any office/room

– Ceiling-mounted ConfigurationsCeiling-mounted Configurations

• Two Projector StereoTwo Projector Stereo– 100% Duty Cycle100% Duty Cycle

– Brighter & No flickerBrighter & No flicker

– Permits multi-PC RenderingPermits multi-PC Rendering

• Passive Stereo & Circular PolarizationPassive Stereo & Circular Polarization– Custom Filters on ProjectorsCustom Filters on Projectors

– Lightweight GlassesLightweight Glasses

– SilveredSilvered Display Surface Display Surface

• Front ProjectionFront Projection– Usable in any office/roomUsable in any office/room

– Ceiling-mounted ConfigurationsCeiling-mounted Configurations

• Two Projector StereoTwo Projector Stereo– 100% Duty Cycle100% Duty Cycle

– Brighter & No flickerBrighter & No flicker

– Permits multi-PC RenderingPermits multi-PC Rendering

16

View-Dependent RenderingView-Dependent Rendering

• HiBallHiBall 6DOF 6DOF TrackerTracker– 3D Position & Orientation3D Position & Orientation

– Accurate, Low latency & Accurate, Low latency & noisenoise

– Headband-mounted SensorHeadband-mounted Sensor

– HiBall to Eyeball HiBall to Eyeball CalibrationCalibration

• PC Network Server PC Network Server

• HiBallHiBall 6DOF 6DOF TrackerTracker– 3D Position & Orientation3D Position & Orientation

– Accurate, Low latency & Accurate, Low latency & noisenoise

– Headband-mounted SensorHeadband-mounted Sensor

– HiBall to Eyeball HiBall to Eyeball CalibrationCalibration

• PC Network Server PC Network Server

17

Rendering Configurations Rendering Configurations

• One PC Configuration (Linux)One PC Configuration (Linux)– Dual-channel NVIDIA graphicsDual-channel NVIDIA graphics

• Three PC Configuration (Linux)Three PC Configuration (Linux)– Separate left & right-eye rendering PCs w/NVIDIA graphicsSeparate left & right-eye rendering PCs w/NVIDIA graphics

– One PC used as network interface, multicasts depth map One PC used as network interface, multicasts depth map stream to rendering PCsstream to rendering PCs

• Performance – 933MHz PCs & GeForce2Performance – 933MHz PCs & GeForce2– Interactive Display Rates of 25-100fpsInteractive Display Rates of 25-100fps

– Asynchronous updates of 3D Reconstruction (2-3Hz) & Scene Asynchronous updates of 3D Reconstruction (2-3Hz) & Scene Graph (20Hz)Graph (20Hz)

• Newest Rendering Configuration 10-20XNewest Rendering Configuration 10-20X– 2.4GHz, GeForce4, Multi-Threaded, VAR Arrays2.4GHz, GeForce4, Multi-Threaded, VAR Arrays

• One PC Configuration (Linux)One PC Configuration (Linux)– Dual-channel NVIDIA graphicsDual-channel NVIDIA graphics

• Three PC Configuration (Linux)Three PC Configuration (Linux)– Separate left & right-eye rendering PCs w/NVIDIA graphicsSeparate left & right-eye rendering PCs w/NVIDIA graphics

– One PC used as network interface, multicasts depth map One PC used as network interface, multicasts depth map stream to rendering PCsstream to rendering PCs

• Performance – 933MHz PCs & GeForce2Performance – 933MHz PCs & GeForce2– Interactive Display Rates of 25-100fpsInteractive Display Rates of 25-100fps

– Asynchronous updates of 3D Reconstruction (2-3Hz) & Scene Asynchronous updates of 3D Reconstruction (2-3Hz) & Scene Graph (20Hz)Graph (20Hz)

• Newest Rendering Configuration 10-20XNewest Rendering Configuration 10-20X– 2.4GHz, GeForce4, Multi-Threaded, VAR Arrays2.4GHz, GeForce4, Multi-Threaded, VAR Arrays

18

Network Considerations Network Considerations

• All Tests over Internet2All Tests over Internet2

• Data Rates of ~20-75 Mbps from Data Rates of ~20-75 Mbps from Armonk, NY and Philadelphia into Armonk, NY and Philadelphia into Chapel HillChapel Hill

– 320 x 240 Resolution320 x 240 Resolution

– Up to 5 Reconstruction Views per siteUp to 5 Reconstruction Views per site

– Frame Rates 2-3 fpsFrame Rates 2-3 fps

• TCP/IP TCP/IP

• Latency of 2-3 seconds typicalLatency of 2-3 seconds typical

• All Tests over Internet2All Tests over Internet2

• Data Rates of ~20-75 Mbps from Data Rates of ~20-75 Mbps from Armonk, NY and Philadelphia into Armonk, NY and Philadelphia into Chapel HillChapel Hill

– 320 x 240 Resolution320 x 240 Resolution

– Up to 5 Reconstruction Views per siteUp to 5 Reconstruction Views per site

– Frame Rates 2-3 fpsFrame Rates 2-3 fps

• TCP/IP TCP/IP

• Latency of 2-3 seconds typicalLatency of 2-3 seconds typical

19

Presentation Outline Presentation Outline

• Motivation and Related WorkMotivation and Related Work

• NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed– Acquisition and 3D ReconstructionAcquisition and 3D Reconstruction

– Collaborative Graphics & User InterfacesCollaborative Graphics & User Interfaces

– Rendering & DisplayRendering & Display

– Network Network

• ResultsResults

• Future ChallengesFuture Challenges

• Motivation and Related WorkMotivation and Related Work

• NTII Tele-Collaboration TestbedNTII Tele-Collaboration Testbed– Acquisition and 3D ReconstructionAcquisition and 3D Reconstruction

– Collaborative Graphics & User InterfacesCollaborative Graphics & User Interfaces

– Rendering & DisplayRendering & Display

– Network Network

• ResultsResults

• Future ChallengesFuture Challenges

20

Results Results

‘‘Roll the Tape’Roll the Tape’‘‘Roll the Tape’Roll the Tape’

21

SummarySummary

• ‘‘One-on-One’ 3D Tele-Immersion One-on-One’ 3D Tele-Immersion TestbedTestbed

• Life-size, view-dependent, passive Life-size, view-dependent, passive stereo displaystereo display

• Interact with shared 3D Objects using Interact with shared 3D Objects using a virtual laser pointera virtual laser pointer

• Half-Duplex Operation todayHalf-Duplex Operation today

• Operation over Internet2 between Operation over Internet2 between Chapel Hill, Philadelphia and ArmonkChapel Hill, Philadelphia and Armonk

• Audio over H.323 or POTSAudio over H.323 or POTS

• ‘‘One-on-One’ 3D Tele-Immersion One-on-One’ 3D Tele-Immersion TestbedTestbed

• Life-size, view-dependent, passive Life-size, view-dependent, passive stereo displaystereo display

• Interact with shared 3D Objects using Interact with shared 3D Objects using a virtual laser pointera virtual laser pointer

• Half-Duplex Operation todayHalf-Duplex Operation today

• Operation over Internet2 between Operation over Internet2 between Chapel Hill, Philadelphia and ArmonkChapel Hill, Philadelphia and Armonk

• Audio over H.323 or POTSAudio over H.323 or POTS

22

Future ChallengesFuture Challenges

• Improved 3D Reconstruction QualityImproved 3D Reconstruction Quality– Larger Working Volume, Faster Frame Rates – 60 camerasLarger Working Volume, Faster Frame Rates – 60 cameras

– Fewer Reconstruction Errors (using structured light and Fewer Reconstruction Errors (using structured light and adaptive correlation kernels)adaptive correlation kernels)

• Reduce System Latency and Reduce System Latency and Susceptibility to Network CongestionSusceptibility to Network Congestion– Pipelined architecturePipelined architecture

– Shunt Protocol (between TCP/UDP and IP layers) that Shunt Protocol (between TCP/UDP and IP layers) that allows multiple flows to do coordinated congestion controlallows multiple flows to do coordinated congestion control

• Full Duplex OperationFull Duplex Operation

• Unobtrusive OperationUnobtrusive Operation– No headmounts, No eyeglasses!No headmounts, No eyeglasses!

• Improved 3D Reconstruction QualityImproved 3D Reconstruction Quality– Larger Working Volume, Faster Frame Rates – 60 camerasLarger Working Volume, Faster Frame Rates – 60 cameras

– Fewer Reconstruction Errors (using structured light and Fewer Reconstruction Errors (using structured light and adaptive correlation kernels)adaptive correlation kernels)

• Reduce System Latency and Reduce System Latency and Susceptibility to Network CongestionSusceptibility to Network Congestion– Pipelined architecturePipelined architecture

– Shunt Protocol (between TCP/UDP and IP layers) that Shunt Protocol (between TCP/UDP and IP layers) that allows multiple flows to do coordinated congestion controlallows multiple flows to do coordinated congestion control

• Full Duplex OperationFull Duplex Operation

• Unobtrusive OperationUnobtrusive Operation– No headmounts, No eyeglasses!No headmounts, No eyeglasses!

23

Thank You Thank You

Research funded byResearch funded byAdvanced Network and Services, Inc. and Advanced Network and Services, Inc. and

National Science Foundation (USA)National Science Foundation (USA)

Research funded byResearch funded byAdvanced Network and Services, Inc. and Advanced Network and Services, Inc. and

National Science Foundation (USA)National Science Foundation (USA)

24

UPenn Acquisition ArrayUPenn Acquisition Array

Fifteen Sony Digital 1394 Cameras – Five Trinocular Views

25

System Overview System Overview

26

Past ExperimentsPast Experiments

UPennPhiladelphia

AdvancedArmonk, NY

UNCChapel Hill

3D Data + 2D Images

3D Data + 2D Images

2D Video + Audio

UNCChapel Hill

3D Data + 2D Images

2D Video + Audio

AdvancedArmonk, NY

Scene Bus Data

With Collaboration

w/o Collaboration