many-camera systems - cmu panoptic...
TRANSCRIPT
![Page 1: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/1.jpg)
Many-Camera Systems:
How They Started at CMU up to EyeVision at 2001 Superbowl
CVPR 2017 in Honolulu
Takeo Kanade Robotics Institute
Carnegie Mellon University
![Page 2: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/2.jpg)
Video-Rate Multi-Camera Stereo Machine Project (1991-1996) for Real-time Range Mapping
Okutomi-Kanade
Multi-baseline stereo
theory 1990
Multi-camera Stereo
v.1
v.2 Outputs: Color Image 256 x 256 x 24 bits
Depth Image 200 x 200 x 8 bits at 30 fps
![Page 3: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/3.jpg)
Soft Camera (aka Virtual Camera):
Occlusion Problem
Range
Intensity/Color
View Synthesis
Soft Camera Output
![Page 4: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/4.jpg)
Whole-Scene Modeling
We use many, many
cameras!!
![Page 5: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/5.jpg)
3D Dome Analog system
with 10 BW Cameras - circa 1994 with 51 Color Cameras - circa 1995
![Page 6: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/6.jpg)
Old and fun stories
Grad student vs. Robot
51 Analog
Video tape recorders
![Page 7: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/7.jpg)
Old and fun stories
Grad student vs. Robot
Synchronization – nontrivial problem
![Page 8: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/8.jpg)
Old and fun stories
Grad student vs. Robot
Synchronization – nontrivial problem
Our own digitizers
![Page 9: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/9.jpg)
Old and fun stories
Grad student vs. Robot
Synchronization – nontrivial problem
Our own digitizers
Calibration pattern on special non-
stretchable paper
![Page 10: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/10.jpg)
4D Full Body Surface Modeling
*
. . .
Stereo
Stereo
Stereo Merging
Texture
Map
![Page 11: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/11.jpg)
Voxel Merging
Depth Map 1 Depth Map 2
5m x 5m x 3m
1 cm3 voxel
500 x 500 x 300
Implicit function:
> 0 outside F(x,y,z) = 0 on the surface < 0 inside {
[Levoy, etal]
[Ikeuchi, etal]
[Kanade, etal]
![Page 12: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/12.jpg)
4D Modeling One on One – circa 1995
![Page 13: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/13.jpg)
Example:
3-Man Basketball
4D Digitization (1995)
Input
sequence
4D Model
Synthetic court
![Page 14: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/14.jpg)
Fully Digital “3D Room” ~2000
SynchronizationSignal Generator
Time Code Generator
TCT: Time Code Translator
TCT
TCT
TCT
S-Video
Main Memory
CPU
HDD
Digitizing PC 1
TCT
TCT
TCT
S-Video
Main Memory
CPU
HDD
Digitizing PC 2
TCT
TCT
TCT
S-Video
Main Memory
CPU
HDD
Digitizing PC n
ControlPC
New 3D Room ~ 2003
![Page 15: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/15.jpg)
Quality of Life Technology Center
Input
sequence
(These are movies.)
Fly-Through View
4D Digitization (~2000): Man-Sofa-Ball
Digital 3D Room: 39 High Quality Cameras
![Page 16: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/16.jpg)
Real Time 4D Digitization
• 64 x 64 x 64
• 10 frames/sec (with 5 PCs)
• Avatar creation
(These are movies.)
![Page 17: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/17.jpg)
Example 5:
Dance (Aug 2000)
4D Digitization
Click to play
![Page 18: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/18.jpg)
Markerless Motion Transfer
Input
video
martial arts master
kung-fu fighting
ballet expert
dancing
Motion
Tracking
Motion
Rendering Processing
Kinematics
Modeling
Output
video
ballet expert
kung-fu fighting
martial arts master
dancing
![Page 19: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/19.jpg)
SubjectE performs SubjectS’s THROW motion
![Page 20: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/20.jpg)
EyeVision
at
Super Bowl
Movie "Matrix"-like replay anywhere in the field
![Page 21: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/21.jpg)
EyeVision at Super Bowl XXXV 2001 33 robot cameras
Control Room Stadium
33 Cameras
![Page 22: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/22.jpg)
Trailer and wires
![Page 23: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/23.jpg)
EyeVision “Best of”
![Page 24: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/24.jpg)
Bigger ideas that didn’t happen?
![Page 25: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/25.jpg)
Interview Room of the Future
Multi-modal
Interactive
Real-time feedback
![Page 26: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/26.jpg)
Scale ?
Scale
(Normalized)
”Complexity”
Room Country City Campus Stadium
10 3
10 6
10 9
10 13
“1”
(1000km)2 x 1km (10m)2 x 1m (100m)2 x 10m (1km)2 x 100m (10km)2 x 100m
![Page 27: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/27.jpg)
MEMS-based
Steerable
Balloon
Cameras,
GPS,
Radio links
3D Country
![Page 28: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/28.jpg)
with Yuichi Ohta (Tsukuba)
Hideo Saito (Keio)
Akimichi (Takenaka)
Seiki Inoue (NHK)
O-ita Stadium “Free View Point” Project
![Page 29: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/29.jpg)
Stadium
Computer
Virtual Wall
Individuals
TV
3D Displays
Network
3D TV
![Page 30: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/30.jpg)
Central
Computing
Modeling
• How much computation at the center or at the TV computer ?
• Where/how the storage at the user end?
• What the most efficient intermediate representation ?
• What interactions by users ?
3D Dome Sports Arena
Network, or Broadcast
Settop Box
User’s
TV computer
Intermediate Data Representation
Offline
distributor
Computation vs. Bandwidth
![Page 31: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/31.jpg)
Computation?
Estimate:
500 x 500 x 500, 50 NTSC Cam, 30 f/s 100 Gflop
Progress:
Jan 1998: 10,000 x Realtime with a few SGIs (14 days elapse time)
Feb 1999: 1,000 x Realtime with 20 PCs (4 days elapse time)
Jan 2002: Modeling: < 50 x Realtime Display: Realtime
![Page 32: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/32.jpg)
Computation?
Estimate:
500 x 500 x 500, 50 NTSC Cam, 30 f/s 100 Gflop
0 0 0 1000 HDTV 60 20 Pflop
Progress:
Jan 1998: 10,000 x Realtime with a few SGIs (14 days elapse time)
Feb 1999: 1,000 x Realtime with 20 PCs (4 days elapse time)
Jan 2002: Modeling: < 50 x Realtime Display: Realtime
![Page 33: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/33.jpg)
camera
Visual Communication Lab
camera display
display
Cloned face
PC Cluster
![Page 34: Many-Camera Systems - CMU Panoptic Datasetdomedb.perception.cs.cmu.edu/tutorials/.../slides/invited1_Kanade.pdf · Many-Camera Systems: How They Started at CMU up to EyeVision at](https://reader031.vdocuments.mx/reader031/viewer/2022022612/5b9a32ed09d3f2c41b8cedb4/html5/thumbnails/34.jpg)
Many-Camera Systems
“Numerosity is power.”
Early efforts:
Primitives and Pretty good
Fun and useful
Challenges in devices, algorithms,
computation, communication
AND application scenarios