army research office multidisciplinary university research

Army Research Office Multidisciplinary University Research Initiative

Unified Research on Network-based Hard/Soft Information Fusion

Third Annual Review Period August 1, 2011 thru July 31, 2012

Drs. Rakesh Nagi (PI), Moises Sudit (co-PI), James Llinas (Emeritus PI)

Center for Multisource Information Fusion

State University of New York at Buffalo

Buffalo, New York, USA

{nagi, sudit, llinas}@buffalo.edu

Dr. John Lavery, ARO PM

Army Research Office

Research Triangle Park, NC 27709 [email protected]

http://en.wikipedia.org/wiki/File:ARL_logo.png


mailto:[email protected]

mailto:[email protected]

MURI Overview Agenda

1. Why is this an important area of research? 2. What is MURI Project trying to accomplish (scientific

objective and technical approach)? 3. Who are the MURI team members? 4. What are the Challenges in Hard/Soft Information

Fusion? 5. Project Scientific Issues and Progress to Date:

Underlying scientific principles, prior state-of-the art advances that favored a multidisciplinary approach, scientific barriers, potential scientific advances.

6. Summary Architecture 7. Scientific breakthroughs in this research

2


1. Importance of the Problem In a few words…

3

“The attacks of September 11th 2001 killed 2,996 people. Despite the subsequent declaration of a war on terror, over the past ten years thousands more have been killed by terrorists of all hues.”

http://www.economist.com/blogs/dailychart/2011/09/global-terrorism-deaths


2. Scientific Objective and Technical Approach

4

NEW

INPUTS

NEW

DECISION

PARADIGMS

LIMITED

DEDUCTIVE

KNOWLEDGE

CONTEXTUALLY

RICH


2. The Objectives of the MURI

“In a nutshell”

5


Some Distinctions in Hard and Soft Observational Data

Totally distinct from Hard Sensors Philosophy: Relations not directly

observable—require reasoning over properties of entities*

* Brower, J., (2001) "Relations without Polyadic Properties: Albert the Great on the Nature and Ontological Status of Relations." Archiv für Geschichte der Philosophie 83: 225–57.

Humans can also judge intangibles --emotional state

DDAATTAA CCHHAARRAACCTTEERRIISSTTIICC HHAARRDD DDAATTAA SSOOFFTT DDAATTAA IIMMPPLLIICCAATTIIOONNSS FFOORR FFUUSSIIOONN

PPRROOCCEESSSSIINNGG

OBSERVATIONAL SAMPLING RATE

HIGH LOW --“Out-of-sequence” processing --agile Temporal Reasoning

SEMANTIC CONTENT ATTRIBUTES FOR SINGULAR ENTITIES

HIGHLY VARIABLE; INCLUDES: -JUDGED RELATIONS -JUDGED INTANGIBLES

--Auto semantic labeling for Hard Data --Rich domain Ontology --Data Association at varying levels of abstraction

ACCURACY, PRECISION RELATIVELY HIGH, GOOD REPEATABILITY (PRECISION)

COGNITIVE AND LINGUISTIC CONSTRAINTS YIELD LOWER ACCURACY AND PRECISION

Requires new strategies and techniques for Common Referencing and Data Association

--Calibrated, --Physics-based, --Known errors,

--Known parametric

Performance --Attribute-

Centric

-Weakly Calibrated, -Neural-based, -Weakly Known

errors, -Weakly Known

parametric Performance

-Attribute-and Relation Centric

6



3. Project Team

7

Penn State University Dr. David Hall, Principal Investigator

US Army Research Office Dr. John Lavery, Program Manager

State University of New York at Buffalo Drs. Rakesh Nagi , James Llinas, Moises Sudit, Principal Investigators

Tennessee State University Dr. Amir Shirkhodaie, Principal Investigator

Iona College Dr. Ronald Yager, Principal Investigator


3. The Team in the Large

• University at Buffalo – 10 Graduate students; 1 PhD awarded, 2 Master’s awarded; 2

Undergraduate students

• Penn State University – 7 Graduate students; 2 Staff; 4 Master’s awarded; 1 PhD

expected 2012

• Tennessee State University – 4 Graduate students; 1 Post-doc; 3 Undergraduate students

• IONA College – Faculty only

• All-Hands Count: About 40 people

8


4. Fundamental Hard-Soft Fusion Challenges

• Hard-Soft Fusion Architecture (what/when to fuse?) – Data-level Fusion (Raw Fusion) – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)

• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty

• Uncertainty representation and transformation

– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty

• Contextual Information – Observational gap-filling, interpretation

• Association of Hard and Soft Information – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) – Asynchronous (out of sequence) information arrival

• Processing of Hard or Soft Information – How does the one influence the parameters, biases or algorithms for the processing of the other

• Lack of Datasets for Technology Development and Verification • Traditional Challenges in Hard Fusion • High-level Fusion (Situation Understanding) • Concepts of Employment (and Transition) 9

FUSION

DISCIPLINES

Computational Linguistics

Probability-Possibility Theory

Human Factors Engineering

Computer Science

Operations Research/Opt.

Information Fusion Systems Engineering

Electrical Engineering

Design of Experiments V& V

Technology Management


Fundamental Hard-Soft Fusion Challenges

• Hard-Soft Fusion Architecture (what/when to fuse?) [T1] – Feature-level Fusion (Measurement or Early Fusion) – Estimate-level Fusion (Late Fusion)

• Nature of Soft Information – Symbolically expressed (symbolic ambiguity) – Human observations have bias and (seldom characterized) uncertainty [T5]

• Uncertainty representation and transformation [T4]

– Results of cognitive fusion are unavailable (feature to symbol) – Natural language processing difficulty [T2]

• Contextual Information – Observational gap-filling, interpretation [T3]

• Association of Hard and Soft Information [T8] – Symbolic versus Numerical (Baghdad vs. 33°20’ N, 44°26’ E) [T7] – Asynchronous (out of sequence) information arrival [T6]

• Processing of Hard or Soft Information [T10] – How does the one influence the parameters, biases or algorithms for the processing of the

other

• Lack of Datasets for Technology Development and Verification [T11] • Traditional Challenges in Hard Fusion [T12] • High-level Fusion (Situation Understanding) [T9] • Concepts of Employment (and Transition) [T13]

10


Synthetic Dataset for Hard+Soft SYNCOIN (PSU)

9/22/2012 11

Soft Messages • Developed SYNCOIN, including interlaced scenarios,

600 text messages and synthetic hard data

Hard Sensor Data Collect

Combined Hard and Soft • Realistic COIN situations • Associable (entities, locations, etc.) • Complementary • Insightful and non-trivial •Non-linear (intertwined, out-of-sequence data) • Confounding elements (background noise) • Linguistic and human observational uncertainty

Ground Truthed for T&E










other


12


Tractor: Soft Data Understanding (UB)

9/22/2012 13

Tractor Processing Pipeline •GATE -> Conversion to syntactic propositional

graph -> CBIR -> Syntactic/Semantic mapping

Soft Message Understanding • Tractor converts Soft Data into Semantic

descriptions (of entities, events, relationships) called semantic propositional graphs

• Improves co-referencing in GATE • 65 Rules for Syntactic to Semantic

mapping • Context-Based Information Retrieval Adds relevant geographical information; ontology; mereology


14

Tractor: Soft Data Understanding

Transitions

- Jim Hendler (for Army Network Science CTA project)

- Mark A. Thomas, USA CIV (US)

- Gabor (Gabe) Schmera, SPAWAR

409,414 Wikipedia documents ~264,000,000 words ~1,700,000 unique entities Tractor processed @~ 0.03 secs/word










other


15


16

Objectives: • Support development of hard/soft information fusion • Develop methods for the aggregation of uncertain information • Provide formalisms for the representation and modeling of soft information

DoD Benefit: • Better use of available information

Scientific/Technical Approach

• Fuzzy Set Theory

• Monotonic Set Measure

• Dempster Shafer Theory

• Mathematical theory of aggregation

• Computing with Words

Accomplishments • Set measure model of fusion

• Role of expert instructions

• Nature of Soft information

• Modeling different uncertainty modes

Challenges • Mixed uncertainty mode fusion • Complexity of Soft information

Computing with WordsComputing with Words

Representation

(Translation)

Fusion

Inference Reasoning

Retranslation

Soft Information

Hard

Information

Fusion

Instructio

Uncertainty Representation and Transformation

(IONA College)


Uncertainty Representation and Transformation

(IONA College)

• The Fuzzy measure has the capability of modeling in a unified framework many different types of knowledge about the value of a variable

• Fuzzy measures closed under aggregation operations are needed for Multi-Source Information Fusion

• The measures of assurance and opportunity generalize some fundamental concepts used in uncertainty modeling

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

30 35 40 45

Age Uncertainty Function

Uncertainty Function for Observed Age of 42 Years by

Observer Trained in Age Estimation

• Observation: person, age 42 years old

– Observer Characteristic : trained in age estimation

– Uncertainty Function: from MURI Human Source Characterization efforts

• Bias: overestimate age by 4.07 years

• Variance: error is normally distributed with standard deviation of 1.65 years










other


18


Human Source Characterization (UB)

19

Objectives •Humans make observations (e.g. age

estimation of a target, number of targets) •How accurate are humans? •What is the influence or training? •How do environmental conditions (e.g.,

lighting) influence their estimate? •How to characterize uncertainty and eliminate

bias?

Scientific/Technical Approach • Literature based review of existing

research in psychology, decision-making, perception, psychophysics

• Focus on qualifying variables impact on observational error

• FM study for COIN relevant observational categories

Context Aware Data Fusion


Transition Target: Uncertainty Alignment Engine

ARL APG Infrastructure Program

20

Objectives:

• Align observations with consideration for observational uncertainties

– Assess observational biases/variances

• Represent observation uncertainties in unified framework – Transform observations where required to common representation

• Automate process for uncertainty alignment

• Enable human observation database maintenance

DOD Program Benefit:

• Uncertainty alignment enables accurate uncertainty propagation through situation assessment process

Initial Prototype: Technology Readiness Levels (TRL): • Uncertainty alignment (UA) database format

developed

• Database implementation tested for prototype observation categories

• Prototype models relied on empirical studies

• UA logic implemented within existing dirty graph matching code

• UA database editor developed

Architecture










other


21


Sensor Selection Based on Physical and Human Targets

Figure 6: Summary of Selected PSU Hard Sensors

Selection Criteria • Sensors that are representative of

tactically deployed sensors • Sensors that provide informational

“value added” to the inference process on our selected targets •A suite that can be utilized in real

demonstrations and campus-based experiments •At least one sensor that allows

innovations for the hard sensor processing flow

Selected Sensors • LIDAR • Short-Wavelength Infra-red (SWIR) • Long Wavelength Infra-red (LWIR) •Visual Video •Acoustic Sensors

22


Hard Sensor Fusion (PSU)

23

2D/3D Video

3D Flash Lidar

Genlock Multiple Hypothesis Particle Filter Color Tracker

Multiple Hypothesis Lidar Range Segmentation Tracker

Fusi

on

En

gin

e

3D

2D/3D

• The objective of the implementation is to incorporate range data from the lidar which will be used to guide the color tracker in cases where track is lost due to occlusion.


3-D

Po

int

Clo

ud

N

ois

e P

roce

ssin

g Ortho-

rectify 3-D image to 2D image Flash LIDAR

Visual (2-D)

Video

Pixel Level

Fusion • Color Intensity Mapping

Range-Intensity Fusion

PCA Shape Decomposition • Point Cloud Classification • 3D Descriptors

X I/D Att.

2-D

Imag

e P

roce

ssin

g

Feature Extraction • Ransac • HOG • SIFT • Hough -

Visual (2-D)

•Pattern recognition • Jones & Rehg’s Gaussian mixture models D

ata

Ass

oci

atio

n v

ia m

ult

i-ca

me

ra h

om

ogr

aphy

&

geo

met

ry-b

ased

met

ho

ds

Target Tracking • K. Filter • P. Filter • LOB

X

Target ID • Neural

nets • Bayes

I/D

Rep

ort

(st

ate

vect

or)

leve

l A

sso

ciat

ion

/Co

rrel

atio

n

Rep

ort

(st

ate

vect

or)

leve

l tar

get

trac

kin

g

TML

Tran

sfo

rm

Hard Sensor Processing Flow

• SWIR •VNIR

•LWIR

Pixel-Level Fusion Point-Cloud range estimates mapped to 2-D camera components

Year 1 Year 2 Year 3+

24


An Integrated Approach Hard (PSU+TSU) and Soft (UB)

Fusion

25


Acoustic Signature Analysis Based on STFT Spectrogram

(PSU)

Short-Time Fourier Transform (STFT)

Discrete Wavelet Transform (DWT)

Kernel-Based Gaussian Mixture Model (K-GMM)

Alkilani, A., Shirkhodaie, A., “Vibro-Acoustic Analysis of Contented Containers,” in review for publication in SPIE 2012 Defense and Security Conference, April 2012, Orlando, FL.


Classification of Handled

Objects By Their Sound (PSU)










other


28


Conceptual Spaces Framework: Symbolic vs. Numerical

29

R-Tree for Indexing Regions

Sample of Regions in Iraq

0.2-0.5 ms per query

Bombing

Bombing










other


30


Data Association (UB)

31

Objectives •Associating or co-referencing entities

from multiple soft, hard, or soft and hard sources

Novelty •Principled Mathematical optimization

approach considering full graph relationships •Cloud Computing Solution

Taxonomy of Association Problems


Highlights and Technology Transition

32

Transitions

- I2WD A2SF program

- AFOSR (Intel Client)

!

"

#!

#"

$!

$"

# % & #$ #'

!"#$%&'

($)(*$+,(*-,&.#$/'&#",(01$

2$3*. 4( 00. *0$

( )*+,

( )*+$

( )*+#










other


33


High-Level Fusion: Situation Understanding, Analytics (UB)

34

Objectives • New Fusion Models for

Complex Relational Data

• Uncertainty representation in graphs and graph matching

• Incremental real-time SA

• Simultaneous analysis of multiple situations of interest (AND-OR graphs)

• User display, provenance/ pedigree reach back.

Similarity Scoring and Uncertainty


Stochastic Graph Matching Results Display

35

Transitions

- I2WD A2SF program

- AFOSR (Intel Client) - ARL


Supporting Sensmaking: Network Visualization (UB)

36

Objectives •Humans make observations (e.g. age

estimation of a target, number of targets) •How accurate are humans? •What is the influence or training? •How do environmental conditions (e.g.,

lighting) influence their estimate? •How to characterize uncertainty and eliminate

bias?

Enhanced Network Visualization


Agenda

37







The MURI Architecture

“In brief”

38


Networked Services:

ESB/SOA/OGC/AMQP


Agenda

40







Outcomes & Breakthroughs

1. Created the first known Hard and Soft "SYNCOIN" dataset for fusion technology and analytics development.

This counter insurgency dataset has about 600 natural language messages with interleaved vignettes for the development and testing of natural language processing methods, and hard sensor associable counterparts for developing semantic extraction and association technologies. This unique data sets is inspired by a Counter-Insurgency (COIN) scenario in Bagdad and contains synthetic soft (human report) data, synthetic hard (physical sensor) data, and real hard data collected using human-in-the-loop vignettes collected at a special facility in central Pennsylvania. The data are augmented by extensive “ground truth” information including, scene setter descriptions, identification and location of all events and activities, social network information, database schema, reference maps, and word cloud diagrams.

2. First ever comprehensive characterization of human observations that are context sensitive. Results published in a submitted paper for the Information Fusion journal, and transition to ARL pending through the ARL infrastructure initiative. Characterizing the human observer is an essential “human” source characterization and common referencing step for hard-soft fusion.

3. An novel approach to Natural Language Processing through "Tractor" and context overlay so that no information is lost and all semantically meaningful information is extracted. Relevant contextual information (as a human would have when reading text) is added so that machine reasoning is feasible. This is a goal, but it is an important discriminator from NLP engines that are simply interested in "information retrieval" of entities and relationships.

41


Outcomes & Breakthroughs

4. Best in class graph association engine. First Map-Reduce implementation for distributed computing. The event association engine, which includes location normalization has been compared to others in the literature. Paper under revision for ACM trans on Info Sys. Association results provide better Precision, Recall and F-measure than other approaches in the literature. Results to be published [Note here that since there is no agreed corpus, a direct comparison with other authors is sometime not accurate.] Journal paper revised for Naval Research Logistics establishes a Lagrangian approach to Graph Association. Results within 5% of the optimal are produced.

5. Graph Matching: Most comprehensive suite of graph matching engines to include uncertainty, AND-OR template graphs (which are suitable to model PIRs), and incremental graph information.

6. 2D to 3D image fusion. We have developed advanced algorithms for the fusion of hard sensor data that include Lidar, SWIR, LWIR, visual video and acoustic sensors. The algorithms perform fusion at the data level – mapping 2-D image data to 3-D Lidar data, creating a “smart pixel” 3-D point cloud that allows multi-spectral feature extraction, hierarchical object classification (based on object size, shape and spectral characteristics), and very accurate target location. New algorithms for multiple hypothesis tracking provide the capability to track dynamic objects in complex environments including tracking of multiple people through crowds including obscurations.

7. Creation and demonstration of a distributed cyber infrastructure for fusion. We have explored the design and implementation of an infrastructure to support distributed information fusion. The toolbox of techniques include; communication methods and protocols, extensions of Service Oriented Architecture (SOA) and Message Oriented Middleware (MOM) paradigms, optimized information flow and tasking, complex event processing, and utilization of community standard data representations. Demonstrations of this infrastructure have been created using tools such as GeoSuite.

42


army research office multidisciplinary university research

Documents