CADIP Information Visualization Work: SFA

David S. EbertDavid S. Ebert

Computer Science & Electrical Computer Science & Electrical Engineering Department Engineering Department

University of Maryland Baltimore University of Maryland Baltimore CountyCounty

ebert@.umbc.eduebert@.umbc.edu

David S. EbertDavid S. Ebert

Computer Science & Electrical Computer Science & Electrical Engineering Department Engineering Department

University of Maryland Baltimore University of Maryland Baltimore CountyCounty

ebert@.umbc.eduebert@.umbc.edu

Christopher D. Shaw Christopher D. Shaw

Computer Science Department Computer Science Department

University of ReginaUniversity of Regina

cdshaw@acm.orgcdshaw@acm.org

Christopher D. Shaw Christopher D. Shaw

Computer Science Department Computer Science Department

University of ReginaUniversity of Regina

cdshaw@acm.orgcdshaw@acm.org

Talk Outline

BackgroundBackgroundVisualization Goals and ChallengesVisualization Goals and ChallengesSFA OverviewSFA OverviewProcedural Shape VisualizationProcedural Shape Visualization• ResultsResults

Recent Work with SFARecent Work with SFA• New FeaturesNew Features• Interactive Lens VisualizationInteractive Lens Visualization

Future DirectionsFuture Directions

BackgroundBackgroundVisualization Goals and ChallengesVisualization Goals and ChallengesSFA OverviewSFA OverviewProcedural Shape VisualizationProcedural Shape Visualization• ResultsResults

Recent Work with SFARecent Work with SFA• New FeaturesNew Features• Interactive Lens VisualizationInteractive Lens Visualization

Future DirectionsFuture Directions

Background & Introduction

Visualization Visualization • Transforms the abstract and symbolic Transforms the abstract and symbolic

into the geometricinto the geometric• Harnesses the human perception Harnesses the human perception

system (visual?)system (visual?)

Glyph-based Volume RenderingGlyph-based Volume Rendering• Advantages of volume renderingAdvantages of volume rendering

• Encodes multidimensional and Encodes multidimensional and multivariate informationmultivariate information

Visualization Visualization • Transforms the abstract and symbolic Transforms the abstract and symbolic

into the geometricinto the geometric• Harnesses the human perception Harnesses the human perception

system (visual?)system (visual?)

Glyph-based Volume RenderingGlyph-based Volume Rendering• Advantages of volume renderingAdvantages of volume rendering

• Encodes multidimensional and Encodes multidimensional and multivariate informationmultivariate information

Visualization Goal & Strategy

Goal:Goal:• Effectively convey information to the userEffectively convey information to the user

• Increase the quantity and clarity of the informationIncrease the quantity and clarity of the information

• Display only as much information as is perceptually Display only as much information as is perceptually understandableunderstandable

Strategy: Strategy:

• Use perceptual cues to aid understanding of Use perceptual cues to aid understanding of multidimensional and multivariate datamultidimensional and multivariate data

Goal:Goal:• Effectively convey information to the userEffectively convey information to the user

• Increase the quantity and clarity of the informationIncrease the quantity and clarity of the information

• Display only as much information as is perceptually Display only as much information as is perceptually understandableunderstandable

Strategy: Strategy:

• Use perceptual cues to aid understanding of Use perceptual cues to aid understanding of multidimensional and multivariate datamultidimensional and multivariate data

Interface Goal & Strategy

Want 3D User Interfaces that are as easy to use as Want 3D User Interfaces that are as easy to use as the WIMP stylethe WIMP style

• Use 3D Input devicesUse 3D Input devices

• Exploit 3D perception with animation & interactionExploit 3D perception with animation & interaction

• Enable fine manipulationEnable fine manipulation

• Avoid user pain and fatigueAvoid user pain and fatigue

Want 3D User Interfaces that are as easy to use as Want 3D User Interfaces that are as easy to use as the WIMP stylethe WIMP style

• Use 3D Input devicesUse 3D Input devices

• Exploit 3D perception with animation & interactionExploit 3D perception with animation & interaction

• Enable fine manipulationEnable fine manipulation

• Avoid user pain and fatigueAvoid user pain and fatigue

Challenge

Rate of Information Increase Greater than Screen Rate of Information Increase Greater than Screen Resolution IncreaseResolution Increase

• Rapid increase in number of information sourcesRapid increase in number of information sources

• Bandwidth of sources increasingBandwidth of sources increasing

• Dimensionality increasing Dimensionality increasing

Rate of Information Increase Greater than Screen Rate of Information Increase Greater than Screen Resolution IncreaseResolution Increase

• Rapid increase in number of information sourcesRapid increase in number of information sources

• Bandwidth of sources increasingBandwidth of sources increasing

• Dimensionality increasing Dimensionality increasing

Solutions

More Effective Visualization TechniquesMore Effective Visualization Techniques

Effective Use of Human PerceptionEffective Use of Human Perception• Utilize visual perception characteristics Utilize visual perception characteristics

• Add shading cues & stereopsis to increase pre-attentive 3D Add shading cues & stereopsis to increase pre-attentive 3D perceptionperception

• Utilize proprioception - body’s innate sense of its position in Utilize proprioception - body’s innate sense of its position in spacespace

More Effective Visualization TechniquesMore Effective Visualization Techniques

Effective Use of Human PerceptionEffective Use of Human Perception• Utilize visual perception characteristics Utilize visual perception characteristics

• Add shading cues & stereopsis to increase pre-attentive 3D Add shading cues & stereopsis to increase pre-attentive 3D perceptionperception

• Utilize proprioception - body’s innate sense of its position in Utilize proprioception - body’s innate sense of its position in spacespace

Perceptive Senses Available

Utilized in Near TermUtilized in Near Term

• VisualVisual

– 3D Spatialization3D Spatialization

– Volume / SizeVolume / Size

– ColorColor

– Shape (curvature)Shape (curvature)

– OpacityOpacity

– TextureTexture

• HapticHaptic

Utilized in Near TermUtilized in Near Term

• VisualVisual

– 3D Spatialization3D Spatialization

– Volume / SizeVolume / Size

– ColorColor

– Shape (curvature)Shape (curvature)

– OpacityOpacity

– TextureTexture

• HapticHaptic

ExperimentalExperimental

• ProprioceptionProprioception

• AuditoryAuditory

• Olfactory Olfactory

Ergonomics IssuesErgonomics Issues

• Pain & FatiguePain & Fatigue

ExperimentalExperimental

• ProprioceptionProprioception

• AuditoryAuditory

• Olfactory Olfactory

Ergonomics IssuesErgonomics Issues

• Pain & FatiguePain & Fatigue

Our Glyph-based Visualization System: SFA

Minimally-Immersive VR InterfaceMinimally-Immersive VR Interface

Multidimensional & Multivariate Data Multidimensional & Multivariate Data VisualizationVisualization

Utilizes Many Perceptual CuesUtilizes Many Perceptual Cues

Minimally-Immersive VR InterfaceMinimally-Immersive VR Interface

Multidimensional & Multivariate Data Multidimensional & Multivariate Data VisualizationVisualization

Utilizes Many Perceptual CuesUtilizes Many Perceptual Cues

SFA System Features

Glyph RenderingGlyph Rendering• Data dimensions mapped to glyph attributesData dimensions mapped to glyph attributes

Two-handed InteractionTwo-handed Interaction

Stereo ViewingStereo Viewing

Multivariate, Multidimensional Time-varying DataMultivariate, Multidimensional Time-varying Data

Regular and Irregular GridsRegular and Irregular Grids

Glyph RenderingGlyph Rendering• Data dimensions mapped to glyph attributesData dimensions mapped to glyph attributes

Two-handed InteractionTwo-handed Interaction

Stereo ViewingStereo Viewing

Multivariate, Multidimensional Time-varying DataMultivariate, Multidimensional Time-varying Data

Regular and Irregular GridsRegular and Irregular Grids

Rendering Within SFA

Visualizable Parameters:Visualizable Parameters:

• Location (3)Location (3)

• Color (1-3?)Color (1-3?)

• Transparency (1)Transparency (1)

• Size (1-3)Size (1-3)

• Shape (1-14)Shape (1-14)

• Surface Detail (1)Surface Detail (1)

Visualizable Parameters:Visualizable Parameters:

• Location (3)Location (3)

• Color (1-3?)Color (1-3?)

• Transparency (1)Transparency (1)

• Size (1-3)Size (1-3)

• Shape (1-14)Shape (1-14)

• Surface Detail (1)Surface Detail (1)

Perceptual Cues Used

Shape / TextureShape / Texture

SpatializationSpatialization

ColorColor

Volume / SizeVolume / Size

ProprioceptionProprioception

StereopsisStereopsis

Shape / TextureShape / Texture

SpatializationSpatialization

ColorColor

Volume / SizeVolume / Size

ProprioceptionProprioception

StereopsisStereopsis

Minimally-ImmersiveUser Interface - Fishtank VR

• Access to environmentAccess to environment

• Collaboration possibleCollaboration possible

• Low cost (< $10K)Low cost (< $10K)

• Stereo viewingStereo viewing

• Two-handed interaction: Two-handed interaction:

– 3-space trackers with buttons3-space trackers with buttons

• Each hand has a distinct roleEach hand has a distinct role

– Left hand sets up contextLeft hand sets up context

– Right hand performs fine manipulationRight hand performs fine manipulation

• Access to environmentAccess to environment

• Collaboration possibleCollaboration possible

• Low cost (< $10K)Low cost (< $10K)

• Stereo viewingStereo viewing

• Two-handed interaction: Two-handed interaction:

– 3-space trackers with buttons3-space trackers with buttons

• Each hand has a distinct roleEach hand has a distinct role

– Left hand sets up contextLeft hand sets up context

– Right hand performs fine manipulationRight hand performs fine manipulation

Two-Handed Minimal Immersion

Takes Advantage of Takes Advantage of Proprioceptive Sense Proprioceptive Sense

““Hold” Volume in 1 Hand, Hold” Volume in 1 Hand, Operate on it with Other Operate on it with Other

Hands Form a Kinematic Hands Form a Kinematic Chain (Guiard):Chain (Guiard):• Left hand is base link, right Left hand is base link, right

hand operates relative to it hand operates relative to it Left is low frequency, Right is Left is low frequency, Right is high frequencyhigh frequency

Takes Advantage of Takes Advantage of Proprioceptive Sense Proprioceptive Sense

““Hold” Volume in 1 Hand, Hold” Volume in 1 Hand, Operate on it with Other Operate on it with Other

Hands Form a Kinematic Hands Form a Kinematic Chain (Guiard):Chain (Guiard):• Left hand is base link, right Left hand is base link, right

hand operates relative to it hand operates relative to it Left is low frequency, Right is Left is low frequency, Right is high frequencyhigh frequency

Left HandLeft Hand• Position and orientation of Position and orientation of

volumevolume

• Selection of drawing context Selection of drawing context from 3D hierarchical menufrom 3D hierarchical menu

Right Hand Right Hand • Volume subsettingVolume subsetting

• Probe data volumeProbe data volume

• Interactive lens visualizationInteractive lens visualization

Left HandLeft Hand• Position and orientation of Position and orientation of

volumevolume

• Selection of drawing context Selection of drawing context from 3D hierarchical menufrom 3D hierarchical menu

Right Hand Right Hand • Volume subsettingVolume subsetting

• Probe data volumeProbe data volume

• Interactive lens visualizationInteractive lens visualization

Scientific Visualization Results

Application: Application:

• Solar Solar Magnetohydrodynamics Magnetohydrodynamics SimulationsSimulations

Performance: Performance:

• 2000+ glyphs at interactive 2000+ glyphs at interactive rates on an Indigo2 High rates on an Indigo2 High ImpactImpact

Application: Application:

• Solar Solar Magnetohydrodynamics Magnetohydrodynamics SimulationsSimulations

Performance: Performance:

• 2000+ glyphs at interactive 2000+ glyphs at interactive rates on an Indigo2 High rates on an Indigo2 High ImpactImpact

Information Visualization

Visualize High Dimensional Abstract Data SpacesVisualize High Dimensional Abstract Data Spaces

• Examples:Examples:

– Document similarities (50,000+ dimensions)Document similarities (50,000+ dimensions)

– Database query routing, retrieval of meta-dataDatabase query routing, retrieval of meta-data

– Financial dataFinancial data

Document Corpus ManagementDocument Corpus Management

• Information analysis, not just retrievalInformation analysis, not just retrieval

• Goals: identify trends; find anomalies, themesGoals: identify trends; find anomalies, themes

Visualize High Dimensional Abstract Data SpacesVisualize High Dimensional Abstract Data Spaces

• Examples:Examples:

– Document similarities (50,000+ dimensions)Document similarities (50,000+ dimensions)

– Database query routing, retrieval of meta-dataDatabase query routing, retrieval of meta-data

– Financial dataFinancial data

Document Corpus ManagementDocument Corpus Management

• Information analysis, not just retrievalInformation analysis, not just retrieval

• Goals: identify trends; find anomalies, themesGoals: identify trends; find anomalies, themes

Results

Visualization of Wall Street Journal CorpusVisualization of Wall Street Journal Corpus SFA SFA

ResultsResults IVEE Results IVEE Results

Visualization of Wall Street Journal CorpusVisualization of Wall Street Journal Corpus SFA SFA

ResultsResults IVEE Results IVEE Results

Shape Visualization

Utilizing Pre-attentive Ability to Utilizing Pre-attentive Ability to Understand Shape (Parker et al.)Understand Shape (Parker et al.)

Shapes Shouldn’t Detract from Spatialization Shapes Shouldn’t Detract from Spatialization

Intuitive Shape Mappings (curvature)Intuitive Shape Mappings (curvature)

Utilizing Pre-attentive Ability to Utilizing Pre-attentive Ability to Understand Shape (Parker et al.)Understand Shape (Parker et al.)

Shapes Shouldn’t Detract from Spatialization Shapes Shouldn’t Detract from Spatialization

Intuitive Shape Mappings (curvature)Intuitive Shape Mappings (curvature)

Parameterized Procedural Shape Visualization

Automatic Generation of Shapes for Data Automatic Generation of Shapes for Data VisualizationVisualization

Map Data Range to the Parameter RangeMap Data Range to the Parameter Range

+ EasyEasy

+ Supports continuous data rangesSupports continuous data ranges

+ Parameterization generates a visual orderParameterization generates a visual order

– May be very slow to generate shapesMay be very slow to generate shapes

Automatic Generation of Shapes for Data Automatic Generation of Shapes for Data VisualizationVisualization

Map Data Range to the Parameter RangeMap Data Range to the Parameter Range

+ EasyEasy

+ Supports continuous data rangesSupports continuous data ranges

+ Parameterization generates a visual orderParameterization generates a visual order

– May be very slow to generate shapesMay be very slow to generate shapes

Parameterized Procedural Shape Visualization

Exploring Three Approaches:Exploring Three Approaches:

•Fractal DetailFractal Detail

• Vary surface roughness from Vary surface roughness from smooth to mountainoussmooth to mountainous

– Data value determines Data value determines amount of fractal amount of fractal perturbation to the surfaceperturbation to the surface

•SuperellipsoidsSuperellipsoids

•Implicit SurfacesImplicit Surfaces

Exploring Three Approaches:Exploring Three Approaches:

•Fractal DetailFractal Detail

• Vary surface roughness from Vary surface roughness from smooth to mountainoussmooth to mountainous

– Data value determines Data value determines amount of fractal amount of fractal perturbation to the surfaceperturbation to the surface

•SuperellipsoidsSuperellipsoids

•Implicit SurfacesImplicit Surfaces

Superquadrics - Superellipsoids

Introduced to Graphics by Barr in 1981Introduced to Graphics by Barr in 1981Extends Quadric Surfaces Extends Quadric Surfaces • With two exponents that control the overall With two exponents that control the overall

““roundness”roundness” or “ or “pointiness”pointiness” of the shape. of the shape.

Introduced to Graphics by Barr in 1981Introduced to Graphics by Barr in 1981Extends Quadric Surfaces Extends Quadric Surfaces • With two exponents that control the overall With two exponents that control the overall

““roundness”roundness” or “ or “pointiness”pointiness” of the shape. of the shape.

infinity2,1,

(1sin

)(2sin(1cos

)(2cos(1cos

< < 0 ,Π ≤ ≤Π−2

Π<Θ<

2

Π−

)Θ=

)Θ=

)Θ=

nn

nzrz

nnyry

nnxrx

β

β

β

Superellipsoids

QuickTime™ and aCompact Video decompressor

are needed to see this picture.

Perception of Superellipsoid Shapes

Just Noticeable Difference Experiment Just Noticeable Difference Experiment Completed Completed (Chris Shaw)(Chris Shaw)

Both Exponents Changed TogetherBoth Exponents Changed Together

Result: 21 Superellipsoid Shapes Result: 21 Superellipsoid Shapes Perceivable Perceivable

Just Noticeable Difference Experiment Just Noticeable Difference Experiment Completed Completed (Chris Shaw)(Chris Shaw)

Both Exponents Changed TogetherBoth Exponents Changed Together

Result: 21 Superellipsoid Shapes Result: 21 Superellipsoid Shapes Perceivable Perceivable

Scientific Visualization -Solar Wind MHD Simulation

Opacity = j vorticity Opacity = j vorticity

Shape = j velocity Shape = j velocity -1-1

• convex = positive velocity convex = positive velocity

• diamonds = zero j velocitydiamonds = zero j velocity

• concave = negative velocityconcave = negative velocity

Opacity = j vorticity Opacity = j vorticity

Shape = j velocity Shape = j velocity -1-1

• convex = positive velocity convex = positive velocity

• diamonds = zero j velocitydiamonds = zero j velocity

• concave = negative velocityconcave = negative velocity

Scientific Visualization II

Shape, color, opacity mapped to j vorticityShape, color, opacity mapped to j vorticity Shape, color, opacity mapped to j vorticityShape, color, opacity mapped to j vorticity

Shape, color, opacity mapped to j vorticity

Information Visualization

X =GoldX =Gold

Y =US $Y =US $

Z =Federal ReserveZ =Federal Reserve

Shape = Stock PricesShape = Stock Prices

Color =NoriegaColor =Noriega

X =GoldX =Gold

Y =US $Y =US $

Z =Federal ReserveZ =Federal Reserve

Shape = Stock PricesShape = Stock Prices

Color =NoriegaColor =Noriega

Implicit Shape Visualization (Rohrer, Ebert, Sibert)

Implicit Surface Modeling (ISM)Implicit Surface Modeling (ISM)• Blobby models, metaballs, soft objectsBlobby models, metaballs, soft objects• Arbitrarily shaped functionsArbitrarily shaped functions• Automatic blending, CSGAutomatic blending, CSG• Procedurally-generatedProcedurally-generated

Global Density Field (F) Global Density Field (F) • Produced from sum of potential field Produced from sum of potential field

source functionssource functions

Implicit Surface Modeling (ISM)Implicit Surface Modeling (ISM)• Blobby models, metaballs, soft objectsBlobby models, metaballs, soft objects• Arbitrarily shaped functionsArbitrarily shaped functions• Automatic blending, CSGAutomatic blending, CSG• Procedurally-generatedProcedurally-generated

Global Density Field (F) Global Density Field (F) • Produced from sum of potential field Produced from sum of potential field

source functionssource functions

F(p) = fi dist(p)

Implicit Shape Visualization Techniques

1. Document Content1. Document Content• Shape (blobby text) Shape (blobby text)

2. Corpus Relationships2. Corpus Relationships• Proximity clusteringProximity clustering

3. Combine3. Combine• Content (shape) Content (shape)

• Relationships (cluster shells)Relationships (cluster shells)

• Focus+contextFocus+context

1. Document Content1. Document Content• Shape (blobby text) Shape (blobby text)

2. Corpus Relationships2. Corpus Relationships• Proximity clusteringProximity clustering

3. Combine3. Combine• Content (shape) Content (shape)

• Relationships (cluster shells)Relationships (cluster shells)

• Focus+contextFocus+context

Document Visualization: Blobby Text

Map n Dimensions to Equally-Map n Dimensions to Equally-spaced Spherical Directions in spaced Spherical Directions in 3-space3-space

• nn directions emanating from directions emanating from origin of sphereorigin of sphere

• Magnitude of directional vectors Magnitude of directional vectors proportional to corresponding proportional to corresponding data valuedata value

• Point source field function at end Point source field function at end of vectorsof vectors

Map n Dimensions to Equally-Map n Dimensions to Equally-spaced Spherical Directions in spaced Spherical Directions in 3-space3-space

• nn directions emanating from directions emanating from origin of sphereorigin of sphere

• Magnitude of directional vectors Magnitude of directional vectors proportional to corresponding proportional to corresponding data valuedata value

• Point source field function at end Point source field function at end of vectorsof vectors

n

New Examples

Transparency Textured Implicits

Corpus Visualization

Compute Document Compute Document SimilaritiesSimilarities

• Relationship graphRelationship graph

3D Proximity Clustering3D Proximity Clustering

• Mass-spring simulationMass-spring simulation

VisualizationVisualization

• Fit implicit surface model to Fit implicit surface model to cluster spacecluster space

Compute Document Compute Document SimilaritiesSimilarities

• Relationship graphRelationship graph

3D Proximity Clustering3D Proximity Clustering

• Mass-spring simulationMass-spring simulation

VisualizationVisualization

• Fit implicit surface model to Fit implicit surface model to cluster spacecluster space

Combined Visualization

3D Proximity Clustering3D Proximity Clustering• Semi-transparent cluster shellsSemi-transparent cluster shells

Individual DocumentsIndividual Documents• Content-based shapeContent-based shape

Focus+ContextFocus+Context

Examples:Examples:

3D Proximity Clustering3D Proximity Clustering• Semi-transparent cluster shellsSemi-transparent cluster shells

Individual DocumentsIndividual Documents• Content-based shapeContent-based shape

Focus+ContextFocus+Context

Examples:Examples:

InfoViz and Shakespeare’s Richard II and Richard III

Recent Work with SFA

New FeaturesNew Features

• Multi-vector / glyph display Multi-vector / glyph display added (3 data dimensions added (3 data dimensions mapped to each vector direction)mapped to each vector direction)

• Multiple dataset display addedMultiple dataset display added

• Data filtering for each data Data filtering for each data dimensiondimension

• Communication with Fanatic Communication with Fanatic TelltaleTelltale

• Communication with Jackal Communication with Jackal agent completedagent completed

New FeaturesNew Features

• Multi-vector / glyph display Multi-vector / glyph display added (3 data dimensions added (3 data dimensions mapped to each vector direction)mapped to each vector direction)

• Multiple dataset display addedMultiple dataset display added

• Data filtering for each data Data filtering for each data dimensiondimension

• Communication with Fanatic Communication with Fanatic TelltaleTelltale

• Communication with Jackal Communication with Jackal agent completedagent completed

• Standard 2HI techniques now Standard 2HI techniques now overloaded on mouse interface overloaded on mouse interface (subsetting, scale, rotation, (subsetting, scale, rotation, zoom) zoom)

– Only 2HI feature not available in Only 2HI feature not available in mouse interface is the Slice Viewmouse interface is the Slice View

• Substantial documentationSubstantial documentation

• Interactive lens visualizationInteractive lens visualization

• Multi-threaded interactionMulti-threaded interaction

• Standard 2HI techniques now Standard 2HI techniques now overloaded on mouse interface overloaded on mouse interface (subsetting, scale, rotation, (subsetting, scale, rotation, zoom) zoom)

– Only 2HI feature not available in Only 2HI feature not available in mouse interface is the Slice Viewmouse interface is the Slice View

• Substantial documentationSubstantial documentation

• Interactive lens visualizationInteractive lens visualization

• Multi-threaded interactionMulti-threaded interaction

Interactive Lens Visualization Techniques

Three Types:Three Types:• Contour plateContour plate

• Texture-based volume renderingTexture-based volume rendering

• X-ray projection (thin slab)X-ray projection (thin slab)

Allows Display of More Data Allows Display of More Data Variables / DimensionsVariables / Dimensions

To Appear at IEEE To Appear at IEEE Visualization ‘99Visualization ‘99

Three Types:Three Types:• Contour plateContour plate

• Texture-based volume renderingTexture-based volume rendering

• X-ray projection (thin slab)X-ray projection (thin slab)

Allows Display of More Data Allows Display of More Data Variables / DimensionsVariables / Dimensions

To Appear at IEEE To Appear at IEEE Visualization ‘99Visualization ‘99

Contour Plate of Wall Street Journal

“X-Ray” Plate of WSJ Data

SFA Initial Perceptual Studies

Perceptual Study Experiment Completed this Perceptual Study Experiment Completed this SummerSummer

Initial Results:Initial Results:

• 10 steps in transparency10 steps in transparency

• 21 in superquadric shape21 in superquadric shape

Preattentive test -- “Where’s Waldo?”Preattentive test -- “Where’s Waldo?”

• Experiment Completed, analysis under wayExperiment Completed, analysis under way

Perceptual Study Experiment Completed this Perceptual Study Experiment Completed this SummerSummer

Initial Results:Initial Results:

• 10 steps in transparency10 steps in transparency

• 21 in superquadric shape21 in superquadric shape

Preattentive test -- “Where’s Waldo?”Preattentive test -- “Where’s Waldo?”

• Experiment Completed, analysis under wayExperiment Completed, analysis under way

Multi-Threaded Interaction:Software Architectures

The usual drawing process is:The usual drawing process is:

• Draw on BackRGB while FrontRGB refreshes CRT Draw on BackRGB while FrontRGB refreshes CRT

– Clear BackRGB frame buffer and Z-bufferClear BackRGB frame buffer and Z-buffer

– Draw all objects in BackRGB+ZDraw all objects in BackRGB+Z

– Swap Back and Front RGB buffersSwap Back and Front RGB buffers

– Front buffer is used to refresh CRT Front buffer is used to refresh CRT

• All graphical objects updated simultaneouslyAll graphical objects updated simultaneously

• Many polygons to draw implies a slow update rateMany polygons to draw implies a slow update rate

The usual drawing process is:The usual drawing process is:

• Draw on BackRGB while FrontRGB refreshes CRT Draw on BackRGB while FrontRGB refreshes CRT

– Clear BackRGB frame buffer and Z-bufferClear BackRGB frame buffer and Z-buffer

– Draw all objects in BackRGB+ZDraw all objects in BackRGB+Z

– Swap Back and Front RGB buffersSwap Back and Front RGB buffers

– Front buffer is used to refresh CRT Front buffer is used to refresh CRT

• All graphical objects updated simultaneouslyAll graphical objects updated simultaneously

• Many polygons to draw implies a slow update rateMany polygons to draw implies a slow update rate

Drawing with 3D Trackers

Usual Process:Usual Process:• Get 3D Tracker position & orientationGet 3D Tracker position & orientation

• Process button hits and interaction with 3D sceneProcess button hits and interaction with 3D scene

• Draw 3D sceneDraw 3D scene

• Draw 3D cursorsDraw 3D cursors

• Swap BuffersSwap Buffers

MR ToolkitMR Toolkit• Written by Shaw & Green at U of AlbertaWritten by Shaw & Green at U of Alberta

• Licensed by 600 academic & research sites worldwideLicensed by 600 academic & research sites worldwide

Usual Process:Usual Process:• Get 3D Tracker position & orientationGet 3D Tracker position & orientation

• Process button hits and interaction with 3D sceneProcess button hits and interaction with 3D scene

• Draw 3D sceneDraw 3D scene

• Draw 3D cursorsDraw 3D cursors

• Swap BuffersSwap Buffers

MR ToolkitMR Toolkit• Written by Shaw & Green at U of AlbertaWritten by Shaw & Green at U of Alberta

• Licensed by 600 academic & research sites worldwideLicensed by 600 academic & research sites worldwide

3D Interaction

For True Interaction, NeedFor True Interaction, Need

• >10 updates/second>10 updates/second

• Lags < 100msLags < 100ms

Lag -- Still the Most Significant ProblemLag -- Still the Most Significant Problem

• Lags > 50ms are noticeableLags > 50ms are noticeable

• Lags >100ms start to affect performanceLags >100ms start to affect performance

• Lags >500ms destroy interactionLags >500ms destroy interaction

– Users adopt Users adopt move-and-waitmove-and-wait strategy strategy

For True Interaction, NeedFor True Interaction, Need

• >10 updates/second>10 updates/second

• Lags < 100msLags < 100ms

Lag -- Still the Most Significant ProblemLag -- Still the Most Significant Problem

• Lags > 50ms are noticeableLags > 50ms are noticeable

• Lags >100ms start to affect performanceLags >100ms start to affect performance

• Lags >500ms destroy interactionLags >500ms destroy interaction

– Users adopt Users adopt move-and-waitmove-and-wait strategy strategy

Fast Update

How to Get Update Rates > 10 Updates/sec?How to Get Update Rates > 10 Updates/sec?• Buy a fast graphics computerBuy a fast graphics computer• Separate application from graphicsSeparate application from graphics

– SGI Performer uses a 3-stage pipelineSGI Performer uses a 3-stage pipeline– MR Toolkit “Computation” processMR Toolkit “Computation” process– Decoupled simulationDecoupled simulation

• Draw less stuffDraw less stuff– Decimate polygonal modelDecimate polygonal model

• Draw cheap approximationsDraw cheap approximations– Textures approximate object, perhapsTextures approximate object, perhaps

How to Get Update Rates > 10 Updates/sec?How to Get Update Rates > 10 Updates/sec?• Buy a fast graphics computerBuy a fast graphics computer• Separate application from graphicsSeparate application from graphics

– SGI Performer uses a 3-stage pipelineSGI Performer uses a 3-stage pipeline– MR Toolkit “Computation” processMR Toolkit “Computation” process– Decoupled simulationDecoupled simulation

• Draw less stuffDraw less stuff– Decimate polygonal modelDecimate polygonal model

• Draw cheap approximationsDraw cheap approximations– Textures approximate object, perhapsTextures approximate object, perhaps

Lag

Lag Is the Time From User Input to Graphics Lag Is the Time From User Input to Graphics ResponseResponseSum of Lags in SystemSum of Lags in System+ Tracker lagTracker lag+ Data transmission lagData transmission lag+ Drawing lagDrawing lag

How to Get Low Lags?How to Get Low Lags?• Predictive filtersPredictive filters• Tuning tracker data timingTuning tracker data timing• Fast update rateFast update rate

Lag Is the Time From User Input to Graphics Lag Is the Time From User Input to Graphics ResponseResponseSum of Lags in SystemSum of Lags in System+ Tracker lagTracker lag+ Data transmission lagData transmission lag+ Drawing lagDrawing lag

How to Get Low Lags?How to Get Low Lags?• Predictive filtersPredictive filters• Tuning tracker data timingTuning tracker data timing• Fast update rateFast update rate

Multi-Threaded Drawing

Another Solution Class:Another Solution Class:

• Draw in multiple frame buffer segments simultaneouslyDraw in multiple frame buffer segments simultaneously

Draw 3D Cursors on Overlay PlanesDraw 3D Cursors on Overlay Planes

• Perform 3D interaction in real timePerform 3D interaction in real time

Draw “Full 3D” Scene in RGB + Z PlanesDraw “Full 3D” Scene in RGB + Z Planes

• Allow different update rates on the screenAllow different update rates on the screen

• Same advantage as mouse cursor + menus on a 2D frame Same advantage as mouse cursor + menus on a 2D frame bufferbuffer

Another Solution Class:Another Solution Class:

• Draw in multiple frame buffer segments simultaneouslyDraw in multiple frame buffer segments simultaneously

Draw 3D Cursors on Overlay PlanesDraw 3D Cursors on Overlay Planes

• Perform 3D interaction in real timePerform 3D interaction in real time

Draw “Full 3D” Scene in RGB + Z PlanesDraw “Full 3D” Scene in RGB + Z Planes

• Allow different update rates on the screenAllow different update rates on the screen

• Same advantage as mouse cursor + menus on a 2D frame Same advantage as mouse cursor + menus on a 2D frame bufferbuffer

Video

Video Shows:Video Shows:

• SFA with multi-threaded 3D trackersSFA with multi-threaded 3D trackers

Video Shows:Video Shows:

• SFA with multi-threaded 3D trackersSFA with multi-threaded 3D trackers

Multi-Thread Advantages

Allows Real-time Update of 3D Tracker CursorsAllows Real-time Update of 3D Tracker Cursors

• Overlay update can do menu interactionOverlay update can do menu interaction

– Interaction with syntactic elementsInteraction with syntactic elements

• RGB update can take as long as it likesRGB update can take as long as it likes

• Limited semantic update is possibleLimited semantic update is possible

– picking on the front-buffer data setpicking on the front-buffer data set

• Both RGB and Overlay threads are separate from any Both RGB and Overlay threads are separate from any “Computation” process or thread“Computation” process or thread

Allows Real-time Update of 3D Tracker CursorsAllows Real-time Update of 3D Tracker Cursors

• Overlay update can do menu interactionOverlay update can do menu interaction

– Interaction with syntactic elementsInteraction with syntactic elements

• RGB update can take as long as it likesRGB update can take as long as it likes

• Limited semantic update is possibleLimited semantic update is possible

– picking on the front-buffer data setpicking on the front-buffer data set

• Both RGB and Overlay threads are separate from any Both RGB and Overlay threads are separate from any “Computation” process or thread“Computation” process or thread

Plans

• Create More Java-based Create More Java-based Tools (e.g., Alpha, Colormap Tools (e.g., Alpha, Colormap Editors)Editors)

• Continue to Develop Version Continue to Develop Version That Communicates With That Communicates With Current Agency ToolsCurrent Agency Tools

• Finish Time-sequence Finish Time-sequence SupportSupport

• Experiment with Performance Experiment with Performance of Java-based Glyph of Java-based Glyph VisualizationVisualization

• Create More Java-based Create More Java-based Tools (e.g., Alpha, Colormap Tools (e.g., Alpha, Colormap Editors)Editors)

• Continue to Develop Version Continue to Develop Version That Communicates With That Communicates With Current Agency ToolsCurrent Agency Tools

• Finish Time-sequence Finish Time-sequence SupportSupport

• Experiment with Performance Experiment with Performance of Java-based Glyph of Java-based Glyph VisualizationVisualization

• Add Isosurface Rendering for Add Isosurface Rendering for Cluster and Metadata DisplayCluster and Metadata Display

• Explore Metadata Explore Metadata Visualization Techniques and Visualization Techniques and Visualization of LSI ResultsVisualization of LSI Results

• Extend Perceptual Studies to Extend Perceptual Studies to User Studies of SFAUser Studies of SFA

• Add Isosurface Rendering for Add Isosurface Rendering for Cluster and Metadata DisplayCluster and Metadata Display

• Explore Metadata Explore Metadata Visualization Techniques and Visualization Techniques and Visualization of LSI ResultsVisualization of LSI Results

• Extend Perceptual Studies to Extend Perceptual Studies to User Studies of SFAUser Studies of SFA

Conclusion

Glyph Rendering AllowsGlyph Rendering Allows • Comprehensible display of multiple variablesComprehensible display of multiple variables• Spatialization of complex relationshipsSpatialization of complex relationships

Minimally-immersive Visualization AidsMinimally-immersive Visualization Aids • 3D perception and feature detection 3D perception and feature detection

Procedural Shape Generation Procedural Shape Generation • Useful for encoding multiple data dimensions (1-14)Useful for encoding multiple data dimensions (1-14)• Allows visual exploration of information spacesAllows visual exploration of information spaces

More Effective Use of the Human Perceptual More Effective Use of the Human Perceptual System is Essential for VisualizationSystem is Essential for Visualization

Glyph Rendering AllowsGlyph Rendering Allows • Comprehensible display of multiple variablesComprehensible display of multiple variables• Spatialization of complex relationshipsSpatialization of complex relationships

Minimally-immersive Visualization AidsMinimally-immersive Visualization Aids • 3D perception and feature detection 3D perception and feature detection

Procedural Shape Generation Procedural Shape Generation • Useful for encoding multiple data dimensions (1-14)Useful for encoding multiple data dimensions (1-14)• Allows visual exploration of information spacesAllows visual exploration of information spaces

More Effective Use of the Human Perceptual More Effective Use of the Human Perceptual System is Essential for VisualizationSystem is Essential for Visualization

Acknowledgments - Collaborators

UMBC / Baltimore / D.C.UMBC / Baltimore / D.C.

• Randy Rohrer (DoD)Randy Rohrer (DoD)

• Aaron Roberts (NASA)Aaron Roberts (NASA)

• Jim Kukla, Pradyut Panda, Ted Jim Kukla, Pradyut Panda, Ted Bedwell, Amen ZwaBedwell, Amen Zwa

• Chris Morris, Alex Eller, Joe Chris Morris, Alex Eller, Joe Romano, Ian SoboroffRomano, Ian Soboroff

Funding: DoD, NASA, NSF, Funding: DoD, NASA, NSF, NSERCNSERC

UMBC / Baltimore / D.C.UMBC / Baltimore / D.C.

• Randy Rohrer (DoD)Randy Rohrer (DoD)

• Aaron Roberts (NASA)Aaron Roberts (NASA)

• Jim Kukla, Pradyut Panda, Ted Jim Kukla, Pradyut Panda, Ted Bedwell, Amen ZwaBedwell, Amen Zwa

• Chris Morris, Alex Eller, Joe Chris Morris, Alex Eller, Joe Romano, Ian SoboroffRomano, Ian Soboroff

Funding: DoD, NASA, NSF, Funding: DoD, NASA, NSF, NSERCNSERC

U. Regina / U. WashingtonU. Regina / U. Washington

• James Hall, Dee Jay Randall, James Hall, Dee Jay Randall, Brook Bakay, Christine Blahut, Brook Bakay, Christine Blahut, Ben Korvemaker, Aaron JonesBen Korvemaker, Aaron Jones

U. Regina / U. WashingtonU. Regina / U. Washington

• James Hall, Dee Jay Randall, James Hall, Dee Jay Randall, Brook Bakay, Christine Blahut, Brook Bakay, Christine Blahut, Ben Korvemaker, Aaron JonesBen Korvemaker, Aaron Jones

MR Toolkit Structure

Distributed ProcessesDistributed ProcessesDistributed ProcessesDistributed Processes

Tracker ServersComputation

Master Process

3D Model

Display

Inter-Process Communication

Shared memory

Many Update Loops

The overlay scheme introduces 2 loopsThe overlay scheme introduces 2 loopsThe overlay scheme introduces 2 loopsThe overlay scheme introduces 2 loops

Tracker Servers

OverlayThread

Computation

RGB Thread

3D Model Tracker Thread

Shared memoryShared memory

Display

Many Update Loops

Overlays may access 3D ModelOverlays may access 3D ModelOverlays may access 3D ModelOverlays may access 3D Model

Tracker Servers

OverlayThread

Computation

RGB Thread

3D Model Tracker Thread

Shared memoryShared memory

Display