Next Generation 4-D Next Generation 4-D Distributed Modeling and Distributed Modeling and Visualization of BattlefieldVisualization of Battlefield

Avideh ZakhorUC Berkeley

MURI on Mobile Augmented Battlespace Visualization

Participants: Avideh Zakhor, (UC Berkeley) Bill Ribarsky, (Georgia Tech) Ulrich Neumann (USC) Pramod Varshney (Syracuse) Suresh Lodha (UC Santa Cruz)

Battlefield VisualizationDetailed, timely and accurate picture of the modern battlefield vital to militaryMany sources of info:

eye witness, aerial photographs, sonar, Synthetic Aperture Radar (SAR), Multi-Spectral Imaging (MSI), Hyper-Spectral Imaging (HSI). Foliage PENetration (FOPEN) radar, Electro-Optic (EO), Infra-Red (IR), Moving Target Imaging (MTI)

Major Challenges

Disparate/conflicting sources of info must be combined.

Impossible for ONE individual to collect and comprehend

Specially trained technicians for each info source Effectiveness of information combining and

fusion determined by its usability. Must avoid information overload in presenting

the data.

Historical Perspective

Sand box:

box filled with sand shaped to replicate battlespace terrain.

Commanders moved around small physical replicas of battlefield objects to direct situation.

Historical Perspective

Paper maps and acetate: As intelligence arrives, technicians use

grease pencils to mark new info on acetate.

Commanders draw on the acetate to plan battlefield situations.

Time consuming: several hours to print, distribute and update.

Many opportunities for introducing errors.

Historical Perspective

Joint Maritime Command Information System (JMCIS): computerized battle space visualization two fundamental limitations:

clutter when displaying too much info.Two dimensional display: 3D info is lost.

Historical Perspective

Responsive workbench with Dragon: Inherently 3D. Workstation renders 3D, back-projected

on a horizontal screen. Users in the viewing area interact with

the bench through 3D mouse, pinch gloves, speech recognition.

Stereographic display with LCS shutter glass.

Improvements over WRB/Dragon

Workbench not suitable for mobile soldier with PDA. Augmented reality can enhance Dragon/WRB:

sensors distributed among the soldiers, can be used both to navigate and to update the visualization database.

Need to deal with uncertainty: represent, compute, and visualize uncertainty information without cluttering.

Time should become the 4th dimension. 4D model construction. Play back/visualize the last 24 hours.

Agile, Mobile, Collaborative Testbed

A networked, collaborative whole Earth terrain database linking workstations, large projected displays, and mobile handheld systems.

Mobile users carry systems with handheld or augmented displays providing 3D terrain visualizations.

Mobile users receive, record, and transmit information about the world.

Users of stationary 3D displays collect and evaluate information provided by mobile users and route data and intelligence back to mobile users.

Collaboration is through annotation of the virtual geo-spatial database.

Central Terrain Database

NAVE Virtual Workbench

Mobile Displays

Collaboration Channels

Wireless Terrain Datalink

Terrain Datalink

Position, Terrain Markup, Route Finding, Weather,

Friendly/Foe advisories, etc.

w/GPS, bearing, tilt sensors and wireless data.

Local databases download and cache data according to bandwidth, movement, and

rendering speed of each platform.

Wireless

Wireless

Architecture: Agile, Mobile, Collaborative Testbed

Research Agenda

Model construction: initially constructed by registering sensor imagery to

reference imagery, maps, elevation data, etc. Four dimensional: time treated on the same footing as space.

Model update: Distributed mobile or stationary users contribute to updating the database via their sensors.

Mobile, real time visualization, interaction and navigation within the database; augmented reality sensors tracking and registration required;

Uncertainty processing for model construction and update, as well as uncertainty visualization.

Tracking & Registration

Database Visualization

Model Update & Construction

Uncertainty processing

Uncertainty visualization

Technical Challenges

3-D and 4D Model Construction

Develop a framework for fast, automatic and accurate 3D model construction for objects, scenes, rooms, buildings (interior and exterior), urban areas, and cities. Incorporate time element (4-D).

Models must be easy to compute, compact to represent, suitable for insertion in large hierarchical visualization databases, to facilitate high quality view synthesis and visualization from views that were not necessarily captured during data collection process.

Strategy: Fusion of multiple data sources: intensity, range, GPS, panoramic

cameras. Incorporate apriori models, e.g. CAD, DEM, DTED, elevation data,

maps

Geo-registration and Tracking

Develop techniques for unemcumbered, wide area and real time tracking for (a) augmentation and (b) visualization.

Strategy: Estimate real time 6-DOF tracking by fusion of multiple data streams with variable uncertainty: GPS, vision, inertial gyros, accelerometers, compass, laser range finders

Real time prototype will use the resulting tracking algorithms on an augmented reality PDA to geo-register and navigate the user within the geo-spatial database.

Mobile Visualization

VGIS: general framework for global geo-spatial data and visualization.

Hierarchical, scalable data structures with fast access that include (a) time; (b) uncertainty; c) all varied products.

Dynamic data structures with fast update: Real time data is put in dynamic cache until system finds time to integrate it with the online data base.

Automated detail management for (a) uncertainty; (b) all visual products;

Intelligent retrieval and visual data mining. Multi-modal interaction in multiple environments.

Technical Challenges: Uncertainty Computation and Visualization

Physicalphenomenon

Informationacquisition

-sensors -humans -databases

Uncertainty computation

Uncertaintyvisualization

-target deformation -glyphs -animation

Info. processing& fusion

-computational algorithms

Analysis

-interaction -decision

Transformation

-sampling -quantization -compression

Uncertainty Processing

Representation and computation issues: many formalisms: probability, possibility, evidence theory;

Transformations; Fusion: data, feature and decision levels

uncertainty aware fusion algorithms for dynamic distributed networks; various topologies for fusion networks: serial, parallel, tree network, non-tree feedback network.

decentralized statistical inference algorithms Time Critical computation and quality of service issues:

Data continually arriving, requiring re-computation. tradeoff between precision and speed of uncertainty

computation; fast, yet imprecise answers.

Uncertainty Visualization and Validation

Present uncertainty in an intuitive uncluttered way; Display devices: screen space, mobility Modality: vision, audio, haptics Data types: scalar/vector/tensor, discrete/continuous,

static/dynamic Uncertainty visualization techniques: glyphs,

deformation, transparency, texture, superimposing/backgrounding, augmented reality.

Validation with novice and expert users: Tasks for mobile battlefield; measure accuracy

and speed performance of user; conduct stat. analysis.

Proof of Concept : Multi-modal Interactions for PDA

Extend interaction methods for workstations to PDA;

Wired (electromagnetic) and unwired (vision) interaction modes, hand gestures

Visible and infrared sensors added to PDA Demonstrate a hybrid vision/inertial tracking PDA

demonstrate VGIS visualization on portable display context

Multiple mobile users collaborate on identifying and locating features or targets.

Facilities and Equipment

G. Tech: GVU Future Computing Lab (FCL); Mobile augmented test-bed.

USC: IMSC lab; CGIT lab.UCSC: VizLabSyracuse: Sensor Fusion Laboratory

(SFL)Berkeley: Video and Image Processing

(VIP) Lab.

Collaboration and Management issues

Dr. Hassan Foroosh currently with University of Maryland, will be the technical liason amongst the five universities.

Telephone conference amongst team members periodically to discuss status of the project.

Live seminars broadcast via video conferencing amongst five campuses, once every two months.

Annual workshops and retreats for PI and graduate students and all other researchers to exchange ideas.

Transitions

Air Force Research Lab (Graniero)Army Research Lab (Emmerman, Tocarcik)Navy Research Lab (Rosenblum)Lawernce Livermore Lab (Ucelton)Sun Microsystems (Sowizraj)Intel (Liang)Planet 9 (Colleen)Geometrix (Zwern)Hughes Research Lab (Azuma)

Cross Collaboration

UCB USC G.T. SYR UCSC

Model const. & update

X x x x

Tracking & reg. x X x xMobile visual. database

x x X x x

Uncertain.processing

x x X x

Uncertain.Visualization.

x x X

Outline of Talks

3D model construction for visualization (UC Berkeley)

Geo-registration and tracking for augmentation and visualization(USC)

Mobile visualization in dynamic, augmented battlespace (Georgia Tech)

Uncertainty processing and information fusion (Syracuse)

Uncertainty visualization and validation (UC Santa Cruz)