3DDI Visualization MURI

UC Berkeley and MIT

UC-MIT

3DDI: Overview

Project pipeline:

3D capture:Modeling,simulation

Rendering 3D Display

Applications:Tele-surgery

TrainingCollaboration

UC-MIT

3DDI: Goals• Direct Interaction: no

gloves or glasses.

• Animated content: interaction in real time.

• Content is real-world: 3D models from live capture and modeling.

Laser scanner

3D display

Virtual object

UC-MIT

Task: 3D capture using range scanner• To build a solid-state, high-accuracy electronic

range-finding scanner.

• The system should serve as a replacement for mechanical scanners and motion-capture devices and be usable indoors and outdoors.

• Desired performance: Outdoors, sub-meter accuracy at 100s of meters,

scans in less than a second. Indoors, millimeter accuracy at several meters, scans

at 20-60 frames/sec.

UC-MIT

Subtasks

• Fabrication, testing and improvement of high-power, flip-bonded VCSEL arrays.

• Integration of scanner components. Design of custom elements (modulator, amplifier and power supplies).

• Purchase and integration of coupling optics.• Illumination demo with VCSEL source,

photomultiplier and CCD.• Wrote code for image sequence processing

and calibration.• Static scan example, integration with dynamic

authoring tool (Steve Chenney).

UC-MIT

3D Imaging System, U.C.B.

Fuji Lens

MCPImaging Optics

CCDVCSEL Array

Power Supply HF Signal

Portable Platform!

IR Light

UC-MIT

• Bottle image: depth range ~ 1.2m. Accuracy ~ 0.3cm

First Scanned image

UC-MIT

Task: Model Capture Using Pose Cameras

• Urban geometry

• Textures/BRDFsfor reillumination

… How can we import 3D scene data quickly and automatically?… Starting point for visualization, design, simulation, teaching.

… Develop effectivesensors, automatedand semi-automatedsoftware tools for rapid environment capture

• Synergistic efforts at UCB: 3D scanner Illumination capture

UC-MIT

Goals of integrated effort• Acquire geo-referenced digital imagery

of MIT campus from ground, air

• Extract building exteriors from imagery,using fully automatic techniques

• Model building interiors semi-automaticallyfrom existing 2D building floorplans

• Attach dense interior phototexturesto geometry, semi-automatically

• Integrate photometrics, interaction,and dynamic simulation from UCB

UC-MIT

Acquisition of geo-referenced imagery• Argus platform performs sensor fusion

of imagery, navigation information

UC-MIT

Geo-referencing of multiple nodes• Currently semi-automated process requir-

ing less than one person-second per image

UC-MIT

Texture extraction• Estimation based on weighted medians

UC-MIT

Progress year 3• Acquire hemispherical interior imagery

• Merge ground, aerial geo-ref’d imagery

• Extension to temporal modeling Continuous site modeling of changing site Test: Building 20 demolition, construction

UC-MIT

Task: Capturing Geometry and Reflectance from Photographs

• Input from Cameras, Pose Cameras, Laser Scanners

• Output to Conventional and 3D Displays

UC-MIT

Progress year 1

• Extend Facade to Parametrized Curved Objects

• Visibility Processing and Real-time Rendering

• Campanile Movie

• High Dynamic Range Photography

UC-MIT

Research Highlights Year 1

• Façade: extended to circularly symmetric objects.

• Façade: Accelerated using α-blending.

Campanile Movie shown at SIGGRAPH’97

UC-MIT

Progress year 2

• Photometric Properties of Architectural Scenes

• Capturing and Using Complex Natural Illumination

• Video Motion Capture

UC-MIT

Research Highlights Year 2• Calculation of radiance with known

(outdoor) illumination:

• Re-rendering under novel lighting:

UC-MIT

Research Highlights Year 2• Rendering synthetic objects into real scenes using

HDR photography. Real+Synthetic objects:

UC-MIT

Research Highlights Year 2

• Acquisition of motion data from video using kinematic models:

UC-MIT

Progress year 3

• Reflectance Recovery from MIT Pose Camera Data

• Inverse Global Illumination

UC-MIT

A Synthetic Sunrise Sequence

5:00am 5:30am 6:00am 6:30am

7:00am 8:00am 9:00am 10:00am

One Day at the End of March

UC-MIT

Inverse Global Illumination Algorithm Developed

Reflectance Properties

Radiance Maps

Geometry Light Sources

UC-MIT

Real vs. Synthetic for Original Lighting

UC-MIT

Real vs. Synthetic for Novel Lighting

UC-MIT

Progress Year 4

• Input Multiple range scans

of a scene Multiple photographs

of the same scene

• Output Geometric meshes of

each object in the scene

Registered texture maps for objects

UC-MIT

Overview

RangeImages

RadianceImages

PointCloud

PointGroups

Meshes SimplifiedMeshes

CalibratedImages

TextureMaps Objects

Registration Segmentation Reconstruction

PoseEstimation

Texture MapSynthesis

UC-MIT

Segmentation Results

UC-MIT

Camera Pose Results

• Accuracy: consistently within 2 pixels• Correctness: correct pose for 58 out of 62 images

UC-MIT

Texture-Mapping and Object Manipulation

UC-MIT

Image-based Modeling and Rendering• 3rd Generation--Vary spatial

configurations in addition to viewpoint and lighting

Novel Viewpoint Novel Viewpoint & Configuration

UC-MIT

Texture-Mapping and Object Manipulation

UC-MIT

Task: Authoring Huge, Dynamic Visual Simulations

• Efficiency Too much time is spent computing needless dynamic

state, and dynamic authoring is not integrated with geometric design.

• Control Physics doesn’t do what an author wants

• Success is measured through speedups and the control of example scenarios.

UC-MIT

3D Capture

Modeling,Simulation

Rendering

3D Display

• Take models from measured data. Eg: architecture

• Author scenarios and simulate the dynamics. Eg: a traffic accident

• Provide dynamic models for efficient rendering.

• Integration example: Simulating with a scanned bottle.

How it relates to MURI

UC-MIT

Year 1: Culling with consistency

• Exploit viewer uncertainty to achieve efficient dynamics culling

• Significant speedups demonstrated: Around 5x for test environments. Arbitrary depending on the world.

• Tools released for VRML authoring.

• Papers in I3D, VRML98 and CGA.

UC-MIT

Year 2 and 3: Directing Scenarios

• Use physical sources of randomness (eg. rough surfaces, variable initial conditions) to direct physical simulations

• Year 2: Directing a single body

• Year 3: Directing multiple interacting bodies

• Along the way: Fast multi-body simulation techniques

UC-MIT

Integration Example: Details

• Captured data and 3d rendering must be linked by an authoring phase.

• Extract radius information from 3D bottle scan, plus estimate of variance.

• Simulate using MCMC to achieve a goal - balls are deflected by bottles to land in the right place.

• Render on autostereoscopic display.

UC-MIT

Task: Integration of Modeling and Simulation• Incorporate data from multiple sources:

Geodetic capture (MIT); floorplan extrusion, instancing (UCB)

• Geometry compilation for responsiveness: Scaleable, persistent proximity/visibility database (UCB,

MIT)

• Natural, extensible constraint-based interaction Object associations framework (UCB)

• Physically-based kinematics: Fire simulation (UCB; shown in ‘98) Impulse-response simulation (UCB)

UC-MIT

Several generations of system components:• 1990-93: WalkThrough system (UCB)

Rapid visualization of complex models

• 1993-94: Radiosity integration (Princeton) Diffuse illumination throughout model

• 1994-95: Object associations (UCB) Natural object instancing & placement

• 1994-97: FireWalk, Impulse (UCB) Physically-based fire, kinematic simulations

• 1996-99: Façade, Skymaps (UCB) High-fidelity photo-assisted modeling

• 1996-99: City Scanning (MIT) Acquisition of extended urban models

UC-MIT

Dataset Integration: Geo-referencing• Argus data is geodetically registered

UC-MIT

Dataset Integration: Exterior structure• Exteriors in UCB FireWalk framework

UC-MIT

Integration of UCB object associations• Infrastructure supports editing at any scale

UC-MIT

Exterior to interior transition

• Seamless transition to Tech Square interior

UC-MIT

Transition: building approach

• Gravity association keeps us to local ground

UC-MIT

Visibility modifications: exterior, interior

• Cell-portal visibility applies throughout

UC-MIT

Door passages using object assocations

• Opening doors to allow passage

UC-MIT

Integration of UCB floorsketch, firewalk

• Tech Square interiors modeled by procedural floorplan extrusion, furniture instancing

UC-MIT

Integration of UCB Impulse-Response• Automated generation of RBL objects

Requires specification as union of convex parts

• Initial integration: population, visualization

UC-MIT

Extension to Impulse: sleeping objects• Added “sleep state” for objects coming to

rest

UC-MIT

Extension to Impulse: interaction• Added interactive application of forces

UC-MIT

Task: Novel 3D Displays

• Re-design the MIT holographic-video display for heightened utility.

• Design a new autostereoscopic video display for multiple viewers.

UC-MIT

Relationship to the rest of the field:

• The holographic video display is the first of its kind, and is unique in its size (75mm x 125 mm) and its capability for rapid interaction.

• The autostereoscopic display is unique in its ability to provide binocular stereo video to multiple viewers in arbitrary locations, without the use of viewing aids such as spectacles.

UC-MIT

Interactive Holographic Video

UC-MIT

Autostereoscopic Display

Multiple viewers(three, so far)Micropolarizer-based spatial multiplexing

UC-MIT

Viewer Tracking in Progress

recognizer finds left eye(s).

video signal to viewer-tracking LCD

UC-MIT

Task: Telesurgery

To integrate elements of the MURI pipeline for visualization in the performance and training of surgery:

• Capture of anatomical data

• Modeling of deformable objects

• Haptic interaction with models

• 3D display of models

UC-MIT

Progress year 1

Developed virtual environment for surgical training:

• Organ models from Visible Human data

• Simple deformable modeling, using 2D meshes of masses-springs-dampers

• Basic instrument interactions, without force feedback: grasping, cutting, stapling, electrocautery

• Commercial laparoscopic interface without force feedback (Immersion Corp.)

UC-MIT

Progress years 2 & 3

• Added haptic capability to surgical simulation:

Custom 4 degree of freedom laparoscopic interface, based on commerical 3 DOF device (Sensable Tech Phantom)

• Non-linear, graded finite-element modeling for real-time performance and good accuracy & scalability.

• Tested environment in surgical training course at UCSF

UC-MIT

Current Simulation:Gallbladder removal

Removal of soft tissue using electrocautery tool

UC-MIT

3DDI: Overview

Project pipeline:

3D capture:Modeling,simulation

Rendering 3D Display

Applications:Tele-surgery

TrainingCollaboration

UC-MIT

Programmatic Evaluation I

• Research on components and system integration successful One example of complete pipeline--from

scanning to display shown (bottle). Multiple examples of integration of two or

more modules -- walkthru, outdoor reflectance modeling, simulation

UC-MIT

Programmatic Evaluation II

• Research on components has not connected well with original applications. The virtual surgery work does not make much use of the technologies developed in the project.

• Propose shift of primary motivating application to urban model capture, visualization and simulation

UC-MIT

Budget Adjustments

• MIT Phase out research on holographic display Continue autostereoscopic development Increase funding of urban modeling

• UCSF Virtual surgery: phase out completely

• UC Berkeley Increase funding for modeling from laser

scanner data

UC-MIT

3DDI: Overview

Project pipeline:

3D capture:Modeling,simulation

Rendering 3D Display

Application:Exterior and InteriorUrban Environments

UC-MIT

Scenario: Rapid Capture of, andTraining in, Urban Environments

• Acquire high-fidelity geometric and photometric models of real environments

• Provide ability to simulate, visualize and physically interact with this environment

• Enhance photorealism with 3D displays

UC-MIT

An Example Sequence of Interactions

• The images in the following sequence obviously appear synthetic; we want to achieve this functionality while maintaining photorealism.

UC-MIT

Flyby of Model of Real Urban Environment• Can be modified by adding virtual buildings

UC-MIT

Seamless Exterior to Interior Transition

• Incorporate geometric, photometric detailto increase photorealism, immersion

UC-MIT

Compiled Proximity, Visibility Information• Increases interactivity, decreases network

traffic among multiple users of model

UC-MIT

Interact with Physical Objects• Increased ability for natural interaction with

running physical simulation

UC-MIT

Directing Behaviors• Construct problem solving contexts for

training

UC-MIT

Research and Engineering Aspects

• Instrumentation

• Exterior capture

• Interior capture

• Real-time Interaction

• Directable Dynamics

• 3D Display

• System Integration

• Representation!

UC-MIT

Progress over the last 12 months• VCSEL array scanner

• Modeling from range and image data: Berkeley

• Modeling from pose cameras: MIT

• Authoring with dynamics

• Real-time Simulation of Physically Realistic Global Deformations

• System integration in Walkthru framework

UC-MIT

Progress on VCSEL array scanner• Design of row-addressable VCSEL array to

provide a scanning source.

• Fabrication of chip prototypes with bonding to silicon.

• Testing and characterization.

UC-MIT

Progress on Modeling: MIT

• Acquisition sensor improvements

• Faster, more accurate spherical imagery

• Improved sub-pixel edge detection

• Automated rotational alignment

• Improved texture, occlusion estimation

• Off-planar relief estimation (Fua, Leclerc)

• Symbolic window extraction (Wang)

• Framework for indoor/outdoor visibility

UC-MIT

Progress on Authoring with Dynamics

• Combine the visibility structure of a model with a model of object dynamics Objects guarantee where they will not be Cull dynamics safely

• Objective: Frame rate depends on number of objects

in view

• Demonstration: A complex world where frame rate (largely)

depends on number of objects in view

UC-MIT

Progress on Real-time Simulation of Physically Realistic Global Deformations

•Combines the best features of several models FEM accuracy (theory of elasticity) No distortion (due to the nonlinear strain) Diagonalized mass matrix (similar to particle

system) Graded mesh size of O(n^2) (comparable to

BEM)

UC-MIT

System Integration in Walkthru Framework

• Create an object-oriented, extensible databasein which various types of models can be stored.

• Develop rendering paradigms by which this DB can be explored by many users simultaneously.

• Create the hooks for the attachment of simulators which may allow “work-on-demand” control.

• Allow all interactions to happen over the Internetbetween different types of computers and OS.

UC-MIT

Walkthru Framework

• Environment to model complex dynamic worlds with user interaction: Cell-based visibility culling: Pre-loading of scene parts -- based on

expected demand, derived from user motion.

• Generic simulation interface; integrated: CFAST: NIST’s Fire Simulator IMPULSE: Rigid Body Dynamics

UC-MIT

Progress Summary

• Built a shared, object-oriented data base.

• Extensions beyond just geometry (SYLIF).

• Tools for model generation from floorplans.

• Used in joint model developments with MIT.

• Integration of scene data from Malik’s group.