stereoscopic video overlay with deformable registration

Stereoscopic Video Overlay with Deformable Registration

Balazs VagvolgyiProf. Gregory Hager

CISST ERC

Dr. David Yuh, M.D.Department of Surgery

Johns Hopkins University

The CASA Project

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Today’s Surgical Assistant: A Simple Information Channel

The CASA Project

Stereo surface tracking

Stereo tool tracking

Virtual fixtures with

da Vinci Robot

Task graph execution system

HMM-based Intent Recognition

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Information Fusion with

da Vinci Display

Ultrasound

Capabilities of a Context-Aware Surgical Assistant (CASA)

Tissue Classification

PreoperativeImagery

The CASA Project

Stereo surface tracking

Stereo tool tracking

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Information Fusion with

da Vinci Display

Developing a Context-Aware Surgical Assistant (CASA)

PreoperativeImagery

Information Overlay

• Problem setting:– Given pre-operative scan

data from a suitable imagingmodality

– Video sequence from a stereo endoscope

• Add value– Overlay underlying anatomy on the stereo video

stream (x-ray vision)

– Include annotations or other information tied to imagery

Key Problem: Nonrigid registration of organ surface to data

Inputs: What Do We Know?

1. Pre-operative 3D model- most probably volumetric- only a portion of it will be visible on the endoscope- anatomy will be deformed during the surgical procedure

2. Camera system properties can be measured- optical & stereo calibration- local brightness/contrast/color response

3. Stereo image stream- 3D surface can be reconstructed- texture information

4. A guesstimate of model–endoscope 3D relationship- We can guess where to start searching [i.e. patient position]

Outputs: What Do We Generate?

1. Position of 3D model registered to stereo image

2. Model deformed to the current shape of anatomy

3. Rendering a synthetic 3D view on the stereo stream

4. Everything done real-time

Original Image Stereo Data Deformed Mesh

2D 3D

All this in a flow chart

Stereo imagepre-processing

Building andoptimizing

disparity map

DeformableRegistration to

3D surface

3D texturetracking

Recognizingdeformations

optical parameters

stereo video stream

Imageoverlay

disparity

3D data

image data

parameters3D model

Classical Stereo Vision: The Problem

• Blocks of each image are compared using SAD

• Optimization for each block independently on entire depth range

+ Very fast implementation (GPU)

¬ Lousy results

Small Vision Systemfrom Videre Design

(w/o structured light):

• Input images downsized to several scale levels (½, ¼, …)• Each scale processed with the same algorithm

– Propagate coarse search results to the finer scale

+ Quality of disparity map is better + Even faster than single scale computation¬ Requires

structured light

Solution #1: Lighting and Multi-Scale

SVL implementation(using structured light):

• Solve a (spatially) global optimization with regularization

– O(D) = min SAD(D) + Smooth(D)

• GLOBAL optimum found in polynomial time

Solution #2: Dynamic Programming

1. Defining the recursive cost function

2. Memoization

3. Finding lowest cost path, which is the disparity map (DM in red)

SmoothnessError

Solution #2: Dynamic Programming

Dynamic Programming on Images

• Minor issue: previous approach applies to scanline

• Approximate DP applied to entire image

- 3D disparity space (D):

- Cost function (C):

- Memoization (P):

Dynamic Programming: Results

Dynamic Programming: In Vivo Results

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Stereo recordings from the da Vinci robot Focal length of ~ 700 pixels ~5mm baseline Distance to surface of 55mm to 154mm.

Raw Disparity Map Textured 3D Model

Surface to 3D Model Registration

• Inputs:– point cloud from the stereo surface modeler– point cloud generated from a model or volume image

• Outputs:- transformation to register the 3D model to the 3D surface

QuickTime™ and a decompressor

are needed to see this picture.

Results: Rigid Registration

Complete system (stereoplus registration) operatesat 5 frames/second

QuickTime™ and a decompressor

are needed to see this picture.

Current algorithm usesIPC with modificationsto account for occlusionsdue to viewpoint (z-buffer)

From Rigid to Deformable

• Calculate residual errors in z direction

• Define a spring-mass system

• Perform local gradient descent

Deformable Registration Results

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Final registration error of < 1mm exceptfor the area where the tool enters the image

Coming in CASA

The Language of Surgery

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Tool Tracking

Tissue Surface Classification

Thank you!

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.