knowledge systems lab jn 9/13/2015 an advanced user interface for pattern recognition in medical...

JN 04/19/23

Knowledge Systems Lab

An Advanced User Interface for Pattern Recognition in Medical Imagery:

Interactive Learning, Contextual Zooming, and Gesture Recognition

Joshua R. NewKnowledge Systems Laboratory

Jacksonville State University

JN 04/19/23


Outline

• Introduction• Techniques: Segmentation, Magnification,

Exploration

• Solutions: – Interactive Learning– Contextual Zooming– Gesture Recognition

• Conclusions

JN 04/19/23


Introduction

Medical imagery… • Consists of millions of images produced

annually which doctors must gather and analyze

• Entails several modalities for each patient, such as MRI, CT, and PET

Refine techniques for facilitating comprehension of this data

JN 04/19/23


Outline

• Introduction• Techniques: Segmentation,

Magnification, Exploration


• Conclusions

JN 04/19/23


Techniques

• Common techniques for facilitating data comprehension:– Segmentation – Labeling of images– Magnification – Precision viewing– Exploration – Interacting intuitively with

complex, 3D data

JN 04/19/23


Why Segmentation?

• Doctors and radiologists:– Spend several hours daily analyzing

patient images (ie. MRI scans of the brain)– Search for patterns in images that are

standard and well-known to doctors

• Why not have the doctor teach the computer to find these patterns in the images?

JN 04/19/23


Why Magnification?

• Doctors and radiologists:– Must be able to precisely view and select

regions/pixels of the image to train the computer

– Can easily lose where they are looking in the image when using magnification

• Why not use visualization techniques to preserve context while allowing precise selections?

JN 04/19/23


Why Exploration?

• Doctors and radiologists:– Need to intuitively interact with the system

to maximize task performance– Need to perform this interaction while

being unencumbered

• Why not use vision-based recognition to allow interaction with the data?

JN 04/19/23


Outline


Exploration


• Conclusions

JN 04/19/23


Problems & Solutions

• Problem #1: Segmentation• Solution #1: Interactive Learning

• Problem #2: Magnification• Solution #2: Contextual Zoom

• Problem #3: Exploration• Solution #3: Gesture Recognition

JN 04/19/23


Platform

• Med-LIFE:– “L”earning of MRI image patterns– “I”mage “F”usion of multiple MRI images– “E”xploration of the fusion and learning

results in an intuitive 3D environment

• Images used from “The Whole Brain Atlas”– http://www.med.harvard.edu/AANLIB/home.html

JN 04/19/23


Outline


Exploration


• Conclusions

JN 04/19/23


Simplified Fuzzy ARTMAP

• Simplified Fuzzy ARTMAP (SFAM)– An AI neural network

(NN) system– Capable of online,

incremental learning– Takes seconds for

tasks that take backpropagation NNs days or weeks to perform

JN 04/19/23


Vector-based Learning

• Two “vectors” are sent to this system for learning:– Input feature vector provides the data from

which SFAM can learn– ‘Teacher’ signal indicates whether that

vector is an example or counterexample

JN 04/19/23


Feature Vector

• Pixel values from images (16 for each slice)

JN 04/19/23


Learning Visualization

• Vector-based graphic visualization of learning

Array of Pixel Values

x

y

Category 1 - 2 members

Category 2 - 1 member


0.30 0.45

JN 04/19/23


T2

Learning Associations

Full Results Detailed Results

JN 04/19/23


Varying Vigilance

• Only one tunable parameter – vigilance– Vigilance can be set from 0 to 1 and corresponds to the

generality by which things are classified(ie. vig=0.3=>human, vig=0.6=>male, 0.9=>Joshua New)

0.675 0.75 0.825

JN 04/19/23


Input Order Dependence

• SFAM is sensitive to the order of the inputs

x

y


Category 2 - 1 member


Vector 3

Vector 1

Vector 2

JN 04/19/23


Heterogeneous Network

• Voting scheme of 5 Heterogeneous SFAM networks to overcome vigilance and input order dependence– 3 networks: random input order, set vigilance

– 2 networks: 3rd network order, vigilance ± 10%

JN 04/19/23


Network Segmentation Results

JN 04/19/23


Segmentation Results

Threshold results

Overlay results

Trans-slice results

JN 04/19/23


Segmentation Screenshot

JN 04/19/23


System Demonstration

Interactive Learning

JN 04/19/23


Segmentation Solution

• Doctors and radiologists:– Spend several hours daily analyzing patient

images (ie. MRI scans of the brain)– Search for patterns in images that are standard

and well-known to doctors

• Solution:– Doctors and radiologists can teach the

computer to recognize abnormal brain tissue– They can refine the learning systems results

interactively

JN 04/19/23


Outline


Exploration


• Conclusions

JN 04/19/23


Zooming Approaches

Inset Overlay

Chip Window

JN 04/19/23


Research & Business

• Carpendale PhD Thesis– Elastic Presentation Space – rubber sheet

images via mathematical constructs

• IDELIX (www.idelix.com)

– Pliable Display Technology – software development kit (SDK) product

– Boeing: 20% increase in productivity

JN 04/19/23


Zoom Visualization

Wireframe View

Contextual Zoom

JN 04/19/23


System Demonstration

Contextual Zoom

JN 04/19/23


System Comparison

Previous System Zoom Overlay Contextual Zoom

JN 04/19/23


Magnification Solution

• Doctors and radiologists:– Must be able to precisely view and select

regions/pixels of the image to train the computer– Can easily lose where they are looking in the

image when using magnification• Solution

– They can precisely select targets/non-targets– They can zoom for precision while maintaining

context of the entire image– The interface facilitates task performance

through interactive display of segmentation results

JN 04/19/23


Outline


Exploration


• Conclusions

JN 04/19/23


Motivation

• Gesturing is a natural form of communication:– Gesture naturally while talking– Babies gesture before they can talk

• Interaction problems with the mouse:– Have to locate cursor– Hard for some to control (Parkinsons or

people on a train)– Limited forms of input from the mouse

JN 04/19/23


Motivation

• Problems with the Virtual Reality Glove as a gesture recognition device:– Reliability– Always connected – Encumbrance

JN 04/19/23


System Diagram

StandardWeb Camera

Rendering

User InterfaceDisplay

Han

d

Movem

ent

User

Gesture Recognition

System

ImageCapture

Update Object

Image Input

JN 04/19/23


System Performance

• System:• OpenCV and IPL libraries (from Intel)

• Input:• 640x480 video image• Hand calibration measure

• Output:• Rough estimate of centroid• Refined estimate of centroid• Number of fingers being held up• Manipulation of 3D skull in QT interface in

response to gesturing

JN 04/19/23


Calibration Measure

• Max hand size in x and y orientation (number of pixels in 640x480 image)

JN 04/19/23


Saturation Extraction

Saturation Channel Extraction (HSL space):

Original ImageOriginal Image Hue

Lightness

Saturation

JN 04/19/23


Gesture Recognition Pipeline

JN 04/19/23



a) 0th moment of an image:

b) 1st moment for x and y of an image, respectively:

c) 2nd moment for x and y of an image, respectively:

d) Orientation ofimage major axis:

2

arctan2

00

022

00

20

00

112

cc

cc

yM

Mx

M

M

yxM

M

),(220 yxIxM

),(00 yxIM

),(10 yxIxM ),(01 yxIyM

),(202 yxIyM

JN 04/19/23



)(*19.0 HandSizeYHandSizeXRadius

• The finger-finding function sweeps out a circle around the rCoM, counting the number of white and black pixels as it progresses

• A finger is defined to be any 10+ white pixels separated by 17+ black pixels (salt/pepper tolerance)

• Total fingers is number of fingers minus 1 for the hand itself

JN 04/19/23


System Setup

SystemConfiguration

SystemGUI Layout

JN 04/19/23


Interaction Mapping

Gesture to Interaction Mapping

Number of Fingers:

2 – Roll Left3 – Roll Right

4 – Zoom In5 – Zoom Out

JN 04/19/23


Gesture Recognition Demo

JN 04/19/23


Exploration Solution

• Doctors and radiologists:– Need to intuitively interact with the system to

maximize task performance– Need to perform this interaction while being

unencumbered

• Solution– Can use intuitive gesturing to interact with

complex, 3D data– Can interact by simply moving their hand in

front of a camera, requiring no physical device manipulation

JN 04/19/23


Outline


Exploration


• Conclusions and Future Work

JN 04/19/23


Interactive Learning

• Users can teach the computer to recognize abnormal brain tissue

• They can refine the learning systems results interactively

• They can save/load agents for background diagnosis on a database of medical images or to allow expert analysis in the absence of a well-paid expert

JN 04/19/23


Contextual Zoom

• They can zoom for precisely viewing and selecting targets/non-targets while maintaining context of the entire image

• The interface facilitates task performance through interactive and customizable display of segmentation results

• This system can be used with any 2D images and even with 3D datasets with some minor alterations

JN 04/19/23


Gesture Recognition

• Can use intuitive gesturing to interact with complex, 3D data

• Can interact by simply moving their hand in front of a camera, requiring no physical device manipulation

• Easily replicated and distributable

• Mapping gestures to interaction is an independent stage

JN 04/19/23


Gesture Recognition

• Dynamic Gesture Recognition

• Other interface applications include: graspable interfaces, 3D avatar / MoCap, multi-object manipulation in virtual environments, and augmented reality

JN 04/19/23


Platform

• Med-LIFE integration effort– Gesture Recognition has already been

integrated into Med-LIFE’s Exploration tab– Contextual Zoom and Interactive learning

have been combined, but not yet integrated into Med-LIFE’s learning tab

• Med-LIFE will function as a single application for medical image analysis

knowledge systems lab jn 9/13/2015 an advanced user interface for pattern recognition in medical...

Documents