improving the speed of virtual rear projection: a gpu-centric architecture
DESCRIPTION
Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture. Matthew Flagg, Jay Summet, James M. Rehg GVU Center College of Computing Georgia Institute of Technology. Ubiquitous Interactive Displays. Every flat surface can be an interactive display. VRP: Shadow Elimination. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/1.jpg)
Improving the Speed ofVirtual Rear Projection:
A GPU-Centric Architecture
Matthew Flagg, Jay Summet,James M. Rehg
GVU CenterCollege of Computing
Georgia Institute of Technology
![Page 2: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/2.jpg)
Matthew Flagg © 2005 22
Ubiquitous Interactive Displays
Every flat surface can be an interactive display
![Page 3: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/3.jpg)
Matthew Flagg © 2005 33
VRP: Shadow Elimination
Single Projector Case
![Page 4: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/4.jpg)
Matthew Flagg © 2005 44
Half power shadows
Shadow Elimination
Double Projector Case
Passive VRP
![Page 5: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/5.jpg)
Matthew Flagg © 2005 55
Shadow Elimination
Boosting projector outputs
Proportional feedback law
![Page 6: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/6.jpg)
Matthew Flagg © 2005 66
Occluder Light Suppression
Detecting occluded pixels
![Page 7: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/7.jpg)
Matthew Flagg © 2005 77
Detecting occluded pixels
Occluder Light Suppression
![Page 8: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/8.jpg)
Matthew Flagg © 2005 88
Detecting occluded pixels
Occluder Light Suppression
![Page 9: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/9.jpg)
Matthew Flagg © 2005 99
Detecting occluded pixels
Occluder Light Suppression
Nonlinear feedback law
![Page 10: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/10.jpg)
Matthew Flagg © 2005 1010
Virtual Rear Projection
Show ICCV’03 demo video
![Page 11: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/11.jpg)
Matthew Flagg © 2005 1111
2 Challenges for VRP
High image qualitySeams between display regions projected by
different projectorsPhotometric Uniformity
Fast CompensationAvoid perception of shadows caused by
system lagImage processing required to ensure high
image quality
![Page 12: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/12.jpg)
Matthew Flagg © 2005 1212
Camera view of screen must be unobstructedRequires reference image capture before
occlusionCannot be co-located with projector
Shadows still perceptibleShadow detection image processing
performed on CPU
Limitations With Previous Work
![Page 13: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/13.jpg)
Matthew Flagg © 2005 1313
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and Pausch
CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with DirectX9.0
New Approach
![Page 14: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/14.jpg)
Matthew Flagg © 2005 1414
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0IR backlit camera image
![Page 15: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/15.jpg)
Matthew Flagg © 2005 1515
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and Pausch
CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with DirectX9.0
New Approach
Turn off occluded pixels
![Page 16: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/16.jpg)
Matthew Flagg © 2005 1616
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0Occluder Light Suppression
![Page 17: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/17.jpg)
Matthew Flagg © 2005 1717
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and Pausch
CHI’02 Projector Roles:
Shadow Eliminator Blinding Light Suppressor
Image Processing on GPU Pixel Shaders Render-To-Texture with DirectX9.0
New Approach
Turn on occluded pixels with second projector
![Page 18: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/18.jpg)
Matthew Flagg © 2005 1818
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0Shadow Elimination
![Page 19: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/19.jpg)
Matthew Flagg © 2005 1919
New Approach
Detect Occluders, Not Shadows Co-locate projector with camera Active IR imaging Based on work by Tan and
Pausch CHI’02 Projector Roles:
Blinding Light Suppressor Shadow Eliminator
Image Processing on GPU Pixel Shaders Render-To-Texture with
DirectX9.0Shadow Elimination and Occluder Light Suppression
![Page 20: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/20.jpg)
Matthew Flagg © 2005 2020
Fast Compensation: GPU-Centric Approach
![Page 21: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/21.jpg)
Matthew Flagg © 2005 2121
Fast Compensation: GPU-Centric Approach
1. Warping, background subtraction
2. Median filtering anddilation for inter-frame tolerance
3. Gaussian blur for blending
4. Compositing and warping
![Page 22: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/22.jpg)
Matthew Flagg © 2005 2222
Pixel Shader Pipeline
camera
texture
background
texture
render
texture 1
render
texture 2
back
buffer
display
image
(A) (B) (C)
![Page 23: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/23.jpg)
Matthew Flagg © 2005 2323
Addressing Image Quality
LAM for left projector
LAM for right projector
Luminance Attenuation Maps (LAMs) Simple feedback-based
approach to accommodate non-linearities of projector-camera
Seam Blending
seam – no blending
seam – with blending
![Page 24: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/24.jpg)
Matthew Flagg © 2005 2424
Virtual Rear Projection Results
Play Video
![Page 25: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/25.jpg)
Matthew Flagg © 2005 2525
Virtual Rear Projection Results
System Component LatencyCamera Capture to PC-Memory 9.09ms
PC-Memory to GPU-Memory 1.70ms
Pixel Shaders 2.14ms
Projector 40.27ms
Total Latency 53.20ms
Image processing speed increased from 15Hz to 110Hz (camera capture rate), placing limit on the projector (85Hz refresh rate)
Projector latency accounts for 76% of total system latency! With occluder movement tolerance of 5cm, shadows are imperceptible up to 94 cm/sec (fast walking)
![Page 26: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/26.jpg)
Matthew Flagg © 2005 2626
Conclusion Presented new approach to VRP
Occluder Detection in IR spectrum All processing moved to GPU
2 System Challenges Met Display Image Quality Shadow Perception Avoidance
Shadows eliminated fast enough to accommodate walking
![Page 27: Improving the Speed of Virtual Rear Projection: A GPU-Centric Architecture](https://reader035.vdocuments.mx/reader035/viewer/2022062811/56815f6d550346895dce7568/html5/thumbnails/27.jpg)
Matthew Flagg © 2005 2727
Future Work Explore hardware solutions
Recent results show an LCD projector having ½ the latency of a DLP and LCOS
User Study VRP currently used in Collaborative Design Lab in School of
Aerospace Engineering Replicate laboratory evaluation of passive VRP with new active
VRP system Improve Image Quality
Better seam blending