Augmented Augmented Reality in Spine Reality in Spine

SurgerySurgery

By : Esam Elkhatib MDBy : Esam Elkhatib MDAssist. Prof. of NeurosurgeryAssist. Prof. of NeurosurgerySuez Canal Medical SchoolSuez Canal Medical School

EgyptEgypt. ealkhatib@yahoo.com. ealkhatib@yahoo.com

Definition of Augmented Definition of Augmented RealityReality

• Virtual Environments (VE): Completely Virtual Environments (VE): Completely replaces the real worldreplaces the real world

• Augmented Reality (AR): User sees Augmented Reality (AR): User sees real environment; combines virtual real environment; combines virtual with realwith real

• Supplements reality, instead of Supplements reality, instead of completely replacing itcompletely replacing it

• Photorealism not necessarily a goalPhotorealism not necessarily a goal

Definition of AR Definition of AR continuedcontinued

• Blends real and virtual, in real Blends real and virtual, in real environment.environment.

• Real-time interactive.Real-time interactive.

• Registered in 3-D.Registered in 3-D.

• Includes idea of removing part of real Includes idea of removing part of real environment (a.k.a. mediated or environment (a.k.a. mediated or diminished reality)diminished reality)

Why.. Using ARWhy.. Using AR

1- Enhance perception of and interaction 1- Enhance perception of and interaction with the real worldwith the real world

2- Potential for productivity improvements in 2- Potential for productivity improvements in real-world tasksreal-world tasks

3- Relatively new field with many problems, 3- Relatively new field with many problems, but much progress has occurred recentlybut much progress has occurred recently

History of ARHistory of AR

• 1960’s Sutherlands / sproulls first Head 1960’s Sutherlands / sproulls first Head mounted display “HMD” system.mounted display “HMD” system.

• Early 1990's: Boeing coined the term "AR" Early 1990's: Boeing coined the term "AR" • Early to mid 1990's: UNC ultrasound Early to mid 1990's: UNC ultrasound

visualization projectvisualization project• 1994: Motion stabilized display [Azuma]1994: Motion stabilized display [Azuma]• 1994: Fiducial tracking in video see-1994: Fiducial tracking in video see-

through [Bajura / Neumann]through [Bajura / Neumann]• 1996: UNC hybrid magnetic-vision tracker1996: UNC hybrid magnetic-vision tracker

Growth of field: projectsGrowth of field: projects

• 2000: Custom see-through HMDs2000: Custom see-through HMDs

• Mixed Reality Systems Laboratory Mixed Reality Systems Laboratory (Japan)(Japan)

– http://www.mr-system.co.jphttp://www.mr-system.co.jp

• - (Germany) - (Germany) http://www.arvika.de/– 2003: Project ARVIKA2003: Project ARVIKA

Applications: medicalApplications: medical

• "X-ray vision" for surgeons"X-ray vision" for surgeons

• Aid visualization, minimally-invasive Aid visualization, minimally-invasive operations, UNC Chapel Hill.operations, UNC Chapel Hill.

Optical see through HMDOptical see through HMD

HMDHMD

Video see through HMDVideo see through HMD

Video see through HMDVideo see through HMD

Video monitor ARVideo monitor AR

Optical StrengthsOptical Strengths

• Simpler (cheaper)Simpler (cheaper)

• Direct view of real worldDirect view of real world

• Full resolution, no time delay (for real Full resolution, no time delay (for real world)world)

• SafetySafety

• Lower distortionLower distortion

• No eye displacement.No eye displacement.

Video StrengthsVideo Strengths

•Digitized image of real Digitized image of real worldworld

•Matchable time delaysMatchable time delays

•More registration, More registration, calibration strategiescalibration strategies

•Medical: video for Medical: video for calibration strategies is calibration strategies is superiorsuperior

Registration and Registration and TrackingTracking

• Tracking is the basic technology for AR.Tracking is the basic technology for AR.– Without accurate tracking you can't generate Without accurate tracking you can't generate

the merged real-virtual environmentthe merged real-virtual environment

• Tracking is Tracking is significantlysignificantly more difficult in AR more difficult in AR than in Virtual Environmentsthan in Virtual Environments

• "Tracking is the stepchild that nobody "Tracking is the stepchild that nobody talks about." - Henry Sowizral, Dec 1994 talks about." - Henry Sowizral, Dec 1994 Scientific AmericanScientific American

The Registration The Registration ProblemProblem

•Virtual & Real must stay properly alignedVirtual & Real must stay properly aligned

• If not:If not:– Compromises illusion that the two coexistCompromises illusion that the two coexist

•Prevents acceptance of many Prevents acceptance of many applicationsapplications

•Silly Q?Silly Q? Do you want a surgeon cutting Do you want a surgeon cutting into you if the virtual cut-marks are into you if the virtual cut-marks are misaligned?misaligned?

Vision-basedVision-based ApproachesApproaches• Fiducials in environment (LEDs, colored Fiducials in environment (LEDs, colored

dots)dots)

• Template matchingTemplate matching

• Restricted environment with known Restricted environment with known objectsobjects

• More sensors (e.g. laser rangefinder)More sensors (e.g. laser rangefinder)

• Keep user in the loop (manual Keep user in the loop (manual identification)identification)

Research Directions in Research Directions in tracking & Registrationtracking & Registration

• Hybrid tracking systemsHybrid tracking systems– Combine approaches, cover weaknessesCombine approaches, cover weaknesses– Systems built for greater input variety and Systems built for greater input variety and

bandwidthbandwidth

• Hybrid systems and techniques -e.g. Hybrid systems and techniques -e.g. use multiple registration techniquesuse multiple registration techniques

Real Time....Does it Real Time....Does it exist?exist?• True real-time systemsTrue real-time systems

• Must synchronize with the real worldMust synchronize with the real world

• Time becomes a first class citizenTime becomes a first class citizen-Time critical rendering-Time critical rendering

• Goal: Accurate tracking at long ranges, in Goal: Accurate tracking at long ranges, in unstructured environmentsunstructured environments

Tracking TechnologiesTracking Technologies

• Active sourcesActive sources

• Optical, magnetic, ultrasonicOptical, magnetic, ultrasonic

• Requires structured, controlled environmentRequires structured, controlled environment

• Restricted rangeRestricted range

• Magnetic vulnerable to distortionsMagnetic vulnerable to distortions

• Ultrasonic: ambient temperature variationsUltrasonic: ambient temperature variations

• Optical is often expensiveOptical is often expensive

Markerless TrackingMarkerless Tracking

• Tracking without adding special Tracking without adding special markers markers

• Provides: Real-time operationProvides: Real-time operation

• Enables: Outdoor operationEnables: Outdoor operation

Face Augmentation Face Augmentation Markerless TrackingMarkerless Tracking

• Lepetit, et al: Fully Automated and Stable Lepetit, et al: Fully Automated and Stable Registration for Augmented Reality Registration for Augmented Reality applications 2003 (Tokyo).applications 2003 (Tokyo).

• Uses one generic 3D face model, grabs Uses one generic 3D face model, grabs image texture of user's.image texture of user's.

Head orientation changes and Head orientation changes and backgroundsbackgrounds

Intraoperative ARIntraoperative AR

• AR for intra-operative visualization and AR for intra-operative visualization and navigation has been a subject of navigation has been a subject of intensive research and development intensive research and development during the last decade.during the last decade.

• Besides accuracy and speed of the Besides accuracy and speed of the system we have to improve the 3D system we have to improve the 3D visualization systemsvisualization systems

• In-situ visualization offers a higher degree In-situ visualization offers a higher degree of freedom for the programmer than of freedom for the programmer than classical on screen visualization classical on screen visualization

Intraoperative ARIntraoperative AR continuedcontinued

• Before the surgery: a model of the patient's Before the surgery: a model of the patient's anatomy from a CT scan has to be derivedanatomy from a CT scan has to be derived

• During surgery tracking the location of the During surgery tracking the location of the surgical instruments in relation to patient surgical instruments in relation to patient and anatomy modeland anatomy model

• Helping the surgeon in both the global and Helping the surgeon in both the global and local navigation, providing a global map and local navigation, providing a global map and 3D information beyond the local 2D view3D information beyond the local 2D view

Pedicle Screw Placement under Pedicle Screw Placement under Video-Augmented Flouroscopic Video-Augmented Flouroscopic ControlControl • Set up a camera of augmented mobile c-Set up a camera of augmented mobile c-

arm (CAMC) and used it to perform pedicle arm (CAMC) and used it to perform pedicle screw placement under video controlscrew placement under video control..

• Using a prototypic Iso-C-3D fluoroscope Using a prototypic Iso-C-3D fluoroscope (Siemens Medical Solutions, Erlangen) with (Siemens Medical Solutions, Erlangen) with an additionally attached CCD-camera.an additionally attached CCD-camera.

StepsSteps

• During the workflow, two During the workflow, two markers for position markers for position control are attached.control are attached.

• After taking a single x-After taking a single x-ray-shot, the image is ray-shot, the image is merged with the video merged with the video imageimage

• The entry point and placement of the tool-tip The entry point and placement of the tool-tip accordingly are carried out under video-control.accordingly are carried out under video-control.

• After alignment of the tool’s axis with the optical After alignment of the tool’s axis with the optical axis, drilling “down the beam” can be started.axis, drilling “down the beam” can be started.

• If markers do not coincide in video and x-ray If markers do not coincide in video and x-ray images, the patient has moved and therefore a images, the patient has moved and therefore a new x-ray image has to be taken.new x-ray image has to be taken.

AdvantagesAdvantages

• In pedicle approach a considerable reduction in In pedicle approach a considerable reduction in flouroscopy time and thus radiation dose.flouroscopy time and thus radiation dose.

• A single-shot x-ray image is sufficient as soon A single-shot x-ray image is sufficient as soon as the pedicle axis is recognized.as the pedicle axis is recognized.

• A new x-ray can be taken at any time for A new x-ray can be taken at any time for updating the intervention (movement, updating the intervention (movement, implants).implants).

• Pedicle identification and needle insertion is Pedicle identification and needle insertion is possible and tracking of the instrument enables possible and tracking of the instrument enables the visualization of the tip of the instrument the visualization of the tip of the instrument even after entering the surface (skin/draping).even after entering the surface (skin/draping).