closing the loop for isp using performance prediction dec-05

Closing the Loop for ISPusing Performance Prediction

Dec-05

Greg Arnold, [email protected]

Sensors Directorate

Air Force Research LaboratoryAFRL/SNAT, Bldg 620; 2241 Avionics Circle

WPAFB OH 45433-7321; (937) 255-1115x4388

2

Trilogy of Thoughts / Goals

• Playground- Urban SASO (Security & Stability Ops)• ISP- context is UAV swarms & S-S fusion

– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem

• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,

• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness

Uninhabited Air Vehicle

3

Target RecognitionLevels of Discrimination

• Detection: the level at which targets are distinguished from non-targets, i.e., clutter objects such as trees, rocks, or image processing artifacts.

• Classification: the level at which target class is resolved, e.g., building, vehicle, or aircraft.

• Recognition: the level at which target subclass is determined, e.g., for a tracked vehicle, tank, APC, or ADU.

• Identification: the level at which the model/make of a target is resolved, e.g., for a tank: M60, M1, or T72.

• Fingerprint: the serial number of a particular instance of a target, i.e. Vince’s Caravan vs. Lori’s Caravan.

Coarse

Fine

4

Target RecognitionLevels of Automation

• Interactive decision aid: – Human and machine work interactively

• Automatic decision aid: – Machine is autonomous from input of data to

output to human– Human makes final decision (Human-in-the-loop)

• Autonomous system: – Machine makes the final decision– Human is NOT in the loop

5

What Does ATR Mean?

Community Function Execution Speed

Intelligence Assist a Human Interpret Data*

Non Real-Time (weeks – years) (hours)

Surveillance Assist a Human Recognize Objects

Non Real-Time (hours-days)

Targeting/Fire Control

Assist a Human Recognize Target

Non Real-Time (seconds – minutes)

Autonomous Guided Munition

Autonomously Acquire Target

Real-Time (milliseconds)

Battle Damage Assessment

Assist a Human Evaluating Effects

Sooner the Better (milliseconds-days)

6

Found Something

Go Get It

ISR Goals

ISR Goals• No Sanctuary• Persistent (PISR)• All Weather• Day / Night• All Terrain (city, country,

forest, desert, ocean)

• Moving & Stationary• Safety !!!

Intelligence, Surveillance, and Reconnaissance

ShotKill Chain

Hit It!

7

Automated Target Recognition (ATR) Insights

• Information Limited: Believe current performance is information limited– Human (Data >> Information)

• Pixels/Pupils ratio• Better SNR, resolution, modalities

– Machine• False Alarms (Google Search)• Finer Discrim./Obscuration (>> higher resolution)

• 3-D: Intuitively understand geometric (3-D) information

• UAVs: UAVs transform the CID problem!

8

Sensors Directorate Structure

• SN Directorate– SNA: Sensor ATR Technology– SNJ: Light (EO) Sensors– SNR: Radio (RF) Sensors– SNZ: Applications

• SNA Division: Mike Bryant, Lori Westerkamp (Ed Zelnio)– SNAA: Evaluation– SNAR: Applications– SNAS: Modeling and Signatures– SNAT: Innovative Algorithms

• SNAT Branch: Dale Nelson, Rob Williams– Generation After Next Technologies & Algorithms (Greg

Arnold)– Tracking and Registration (Devert Wicker)– Vigilance (Kevin Priddy)

9

ATR Thrust Scope

GEOLOCATE

FIND

TRACK

ID M60

FU

NC

TIO

NS

SENSORS

MATURATION PROCESSAlgorith

ms

Signatures

Assessment0 100

1

PD

FAR

10

Challenge ProblemsStandard MetricsATR Theory

Characterized PerformanceHigh Performance ComputingOperational Databases

ASSESSMENT& FOUNDATION

Sensor Data Management System (SDMS)

ATR Thrust Approach - Subthrusts

INNOVATIVE ALGORITHMS

FIND FIX TRACK & ID

Phenomenology ExplorationEM ModelingSynthetic Data

Operational Target Models/Databases

SIGNATURES & MODELING

SignatureCenter

5

10

15

20

10

20

30

40

02.5

5

7.5

10

5

10

15

20

12

3-D ATR− 3-D Imaging for RF Floodlight− 3-D for urban context− ATR Theory challenge problem

ATRC

ap

abili

ty /

Dif

fic

ult

y

Near Mid Far

Spiral D

evelopment

We need a generalized pattern recognition capability that will classify things previously unseen, actively manage assets, and predict the intent and actions of combatants.

Adaptive ATR − On-the-fly modeling / reacquisition− Reasoning with uncertainty− Adaptive metrics derived from user

ATR for Anticipatory ISR − Multi-X fusion for PISR/TST− Dynamic GIG Sensor Management− ATR Theory for Anticipation

Goals

13

Assumptions / BeliefsBackground / Framework

• Must Define Problem & EOC’s– Whether or not applying model-based vision– Necessary for testing algorithm capabilities

• Model-Based Vision– More than just CAD models– Characterization of the data

and the system at some level– “If I can’t model it, I don’t understand it”

• Physics-Based Vision– What can we do before appealing to statistics

Real World Dimensionality

DataModel

s

14

Operating Conditions (OCs)

TargetsTargetsSensorsSensors

InteractionsInteractions

EnvironmentEnvironment

OCs: Everything that changes the sensor response. Most OCs have infinite variation

15

Real world variability:Extended Operating Conditions (EOCs)

. . .

20 Target Types

6 DOFPose

Squint & Depression

AngleArticulation

x

y

zObscuration

Variants

Billions20 6524

23

22

61 QQQQQQN Billions20 65

24

23

22

61 QQQQQQN

Configuration

16

Discrimination vs. Robustness

Data Models

Robustness

Disc

rimin

atio

n

Seria

l #

Sam

e Se

nsor

Sens

or T

ype

Synt

hetic

Shape

Reflectivity

Tuni

ng

MSEMSE

Quantized

Quantized

Metrics

Metrics

Binary

Binary

Shape

Shape

Points

Points

17

Using Information More Effectively

More Information

ATR Driven Sensing

Multisensor Approaches

Bio-Inspired Adaptive ATR

Challenge Space

1717

18


• Playground- Urban SASO• ISP- context is UAV swarms & S-S fusion




19

Vertically Integrated Sensor Exploitation for Generalized Recce & Instant Prosecution

(VISEGRIP)

20

TheoryTheory

Goal: Quantify the accuracy, completeness, & relevance of information with demonstrable authority. Challenge problem: support counter-WMD

Objective: Theory & algorithm research to incorporate ATR Theory principles into Sensor Mgmt infrastructure modified to implement confirmatory sensing & interrogation

Payoff: Pattern Recognition discipline that is more expressive to assure users that source is authoritative and information is “actionable”

Confirmatory Sensing & Interrogation

AlgorithmsAlgorithms

BackgroundBackground

ATR Theory•Aims to design and predict performance of sensor data exploitation systems•Includes all forms of sensor data exploitation i.e. target detection, tracking, recognition, and fusion

Information TheoryStudies the collection and manipulation of information

Query Generation What question to ask

Query Processing When, How, & Who to ask

Data Fusion Align redundant informationAssess unique or contradictory informationAssimilate valuable information

Evidence Assessment Quantify accuracy and completeness of assertionsPredict a window of opportunity

22






23

ATR-Driven Sensing

Cueing, Prioritization for the Human

~1’10’ ~1’

Short Circuits

Resonant Cross Slots Cavity

Low Dielectric/Low Loss Face Sheet

24






Uninhabited Air Vehicle

25

What is your Objective Function?

• L-p (L-1, L-2, L-infinity)• Diffusion Distance• Hausdorff• Chamfer• Ali-Silvey• Earth Movers Distance• Chi Squared• Entropy• Kullback-Liebler• Mutual Information• Maximum Likelihood• Renyi

• Pd (Prob. of Detection)• Pcc (correct classification)• Pe (Prob. of Error)• Pfa (False Alarm)• Confusion Matrix• Precision• Recall• ROC Curve

26

‘Clear Box’ View of ATR

0 100

1

PD

FAR

Environment

DetectDetectTrackTrack

GeolocateGeolocateIDID

SensorTargetATR

DecisionsHuman

Decisions

TrainedFeatures Templates Models

Target Knowledge

FeatureExtractor

Discriminator

DecisionRule

Target Knowledge

Target Models &Database

27

ATR/Fusion Processes

Target Models &DatabaseSensor

Model

Environment

DetectDetectTrackTrack

GeolocateGeolocateIDID

Sensor(s)TargetATR

DecisionsHuman

Decisions

Sensor Management

Registration

Environment Model

Performance Model

Adaptation

Behavior Models

Anticipate

28

Performance Model is the Lynchpin

• ATR System is dependent on the Performance Model• Need performance prediction

– Determine where / when to use sensors– Estimate effectiveness of sensors for given task

– Sensor Management– Registration– Learning– …

29

If Somebody Asks…

• Typical DARPA question– “Is it physically possible to do X?”– We’ve invested $K and achieved P% performance, is it

worth investing more?

• Examples– How likely are we to detect a dismount with an HSI

system with 1m spatial resolution? 1ft? 1in?– We spent $40M and achieved 80% of perfection.

Have we reached the knee in the performance curve?– Organization X says it can build a system to do Y.

Does this violate physics?

30

Aspects of ATR Theory Objectives

• Measure the information content of sensor imagery

• Given a set of data and a MOP, determine attainable performance range

• What are the critical design constraints to achieve a desired outcome, using this data?

• Estimate exploitation level of available information

• Establish “feedback loop” between ATR designers and sensor developers

• What are the critical design constraints to achieve a desired outcome at a particular level of confidence?

• Information gain from using models and data adaptively (learning)

• Determine theoretical upper bound on performance of given ATR

• Given an ATR system and a set of data, determine how much information can be exploited

• Determine how close a given system comes to achieving the optimal bound

• What are the critical design constraints to achieve a desired outcome, using this sensor and algorithm?

• What was the benefit of adding ‘this’ (additional data/processing)?

Data Assessment Design System Evaluation

31

Problem Simplification

• Having said all that, let’s examine a problem for which we have some intuition

• 4 or 5 points undergoing rotation, translation, and maybe scale and skew

• 1-D, 2-D, and 3-D• Understand the projection from world to sensor

32

What is Shape?

• Pose and scale invariant, coordinate independent characterization of an arrangement of features.

• Residual geometric relationships that remain between features after “mod-ing out” the transformation group action.

• Captured by a “shape space” where each distinct configuration of features (up to transformation) is represented by a single point.

33

Beyond Invariants

Invariants +

Projection

Object-Image Relations

34

Generalized Weak Perspective

• Projection model applicable to optical images(pinhole camera)

• Approximates full perspective for objects in ‘far field’

• Affine transformations on 3-space, and in the image plane (2-space)

• Denoted GWP

35

Affine Transformations

• In 3D

110003333231

2232221

1131211

z

y

x

taaa

taaa

taaa

(Rotate, Scale, Skew | Translate) (3-D Point)

36

GWP Projection3D to 2D

11...111

...

...

1321

1321

nn

nn

vvvvv

uuuuu

11...111

...

...

...

1000 1321

1321

1321

4

4

3

3

2

2

1

1

nn

nn

nn

zzzzz

yyyyy

xxxxx

b

a

b

a

b

a

b

a

Image Projection Object

37

Object-Image Relation Motivation

Object 1

Object 2Image 1

Image 2

• Image 1 is not equivalent to Image 2 (in 2-D)

• Object 1 is not equivalent to Object 2 (in 3-D)

38

Object - Image Relations Concept

“The relation between objects and imagesexpressed independent of the camera parameters

and transformation group”

(1) Write out the camera equations (geo or photo)

(2) Eliminate the group & camera parameters(3) Recognize the result as a relation between

the object and image invariants.

But pure elimination is VERY difficult even for polynomials.

39

Weak PerspectiveObject - Image Relations

(Generalized)

• Parallel things remain parallel• The object size is 1/10 the distance from the camera• (Standard Position Method)

40

Weak Perspective Camera

3-D Model Pi={xi,yi,zi} N-points (3N DOF)Rotate,Translate,Scale,Shear (12 Constraints)

3N-12 Absolute Invariants

2-D Image qi={ui,vi} N-points (2N DOF)Rotate,Translate,Scale,Shear (6 Constraints)


Camera ModelN-points (2N DOF)Union 2-D & 3-D (8 Constraints)

2N-8 relations

Need 5 corresponded points (minimum)q1 q2

q3

q4 q5

P1 P2

P3

P4 P5

41

||

||,

||

||,

||

||

4321

5432

4321

5431

4321

5321

PPPP

PPPP

PPPP

PPPP

PPPP

PPPP

3-D Invariants

P1 P2

P3

P4 P5 3-D Model Pi={xi,yi,zi,1} 5-pointsGL3+Translation (12 Constraints)


Invariant is a function of the Ratio of Determinants:

A useful standard position is:

11111100001000010

53

52

51

III

42

||

||,

||

||,

||

||,

||

||

321

532

321

432

321

521

321

421

qqq

qqq

qqq

qqq

qqq

qqq

qqq

qqq

11111100010

5242

5141

iiii

2-D Invariants

2-D Image qi={ui,vi,1} 5-pointsGL2+Translation (6 Constraints)


Invariant is a function of the Ratio of Determinants:

A useful standard position is:

q1 q2

q3

q4 q5

43

11111100001000010

1000010001

11111100010

53

52

51

42

41

5242

5141

III

ii

iiii

Object - Image RelationGeneralized Weak Perspective Camera

(2-D Standard Position) = (Camera Transform) (3-D Standard Position)

The camera transforms the first 4 object point to image points,the remaining points satisfy the object - image relation iff:

5342525253415151 ; IiIiIiIi

11111

1000

0100

0010

100011111

100

010

53

52

51

0232221

0131211

5242

5141

I

I

I

vttt

uttt

ii

ii

Eliminate camera transform parameters:

44

Object-Image Relation Abstraction

All objects that could have

produced the image.

All images of the object.

x

uObject-Image

Relations

GX n /

Space ShapeObject 3 GU n ˆ/

Space Shape Image2

45

GWP Shape Spaces

• The shape spaces in the GWP case are Grassmann manifolds

• In 3D– Gr(n-4,H) or dually the Schubert cycle of 4-planes in

Gr(4,n) which contain (1,….,1)– Manifold has dimension 3n-12

• In 2D– Gr(n-3,H) or dually the Schubert cycle of 3-planes in

Gr(3,n) which contain (1,….,1)– Manifold has dimension 2n-6

H is the subspace of n-space orthogonal to the vector (1,…,1)

46

Why

• We associate to our object data, viewed as a linear transformation from n-space to 4-space, its null space K of dimension n-4.

• Likewise to our image data in 2D we associate the null space L of dimension n-3.

11...111

...

...

...

1321

1321

1321

nn

nn

nn

zzzzz

yyyyy

xxxxx

47

Global Shape Coordinates

• Better than local invariants

• Come from an isometric embedding of the shape space in either Euclidean space or projective space.

• Matching expressed in these coordinates will gracefully degrade

48

Example in GWP

• 3D, n = 5 feature points

• Global shape coordinates are the Plucker coordinates (or dual Plucker coordinates) of the 4xn object data matrix or the 3xn image data matrix.

1111

det],,,[,,,lkji

lkji

lkji

lkji zzzz

yyyy

xxxx

lkjiM

111

det],,[,, tsr

tsr

tsr vvv

uuu

tsrN

49

Global Object-Image Relations

• General– If and only If conditions– Overdetermined set of equations

• GWP– To match, K must be contained in L (iff)– This incidence condition can be expressed in terms of the

global shape coordinates– For n=5, 10 (non-independent) relations that look like:

[1234][125]-[1235][124]+[1245][123]

Locally only 2 of the 10 are independent, because the locus V of matching pairs (object shape, image shape) in the 7 dimensional product space XxY has dimension 5, codimension 2.

50

Beyond Object-Image Relations

Object-Image Relations

+

Matching

Object-Image Metrics

51

• We intuitively know that if we want to measure something we need a metric… ATR is no different.

• How far apart are these points?

1. The triangle inequality provides efficient match searching

2. Reliable & predictable Unknowns rejection3. Theoretical performance prediction

Why Metrics?

52

The Triangle Inequality Advantage

u: image, x*: prototype object, xk: object from group

Shape Space

u

Measure the distance from the image to each shape object?

Search the group iff the distance to the prototype is less than the sum of the max intragroup distance and noise threshold.

X*

X*

X*

u

Measure the distance from the image to each shape prototype!

Group 1

Group 2

Group 3

53

Using the Triangle Inequality

*

*

noise*

Offline,

*

tMeasuremen

*

**

**

Threshold],[

],[max],[],[

],[],[],[

],[],[],[

x

d

jKj

k

kk

kk

dxud

xxdxudxud

xxdxudxud

xxdxudxud

x

Equivalent

Grouping

Decision

u: image, x*: prototype object, xk: object from group

Search the group iff the distance to the prototype is less than the sum of

the max intragroup distance and noise threshold.

XkX*

uShape Space

Reject beyond Thresholdnoise

54

What are Shape and Distance?

• Shape: What is left after translation & rotation are removed (more generally, the group)

• This is the (Partial) Procrustes definition of distance– R represents rotation and T represents translation– Procrustes normalizes the size of the objects

2222111

,,,21 )ˆ()ˆ(inf),(

2211

TORTOROOdTRTR

p

55

New Metrics?

Any ol’ metric just won’t do…

1. Invariant to translation & rotation of 3-D object (+ more)

2. Invariant to the camera projection (+ discretization)

• This leads to the concept of Object-Image Relations (O-IR’s)

• Incomplete

– O-IR’s are only surrogate metrics

– =0 iff the object and image features are consistent

• Object-Image Metrics satisfy all the metric properties

• Shape Space is NOT Euclidean!• There is some evidence that human similarity perception is not always

metric

56

Metrics on the Shape Spaces

• How to compare objects to images!

• We want a natural shape matching metric– Invariant to transformations of the 3D or 2D data,– e.g. Rotations, translations, or scale of the object or image

• Generalize Weak Perspective– We use the natural Riemannian metric on the Grassmannian

to measure distances between object shapes and image shapes

– This involves the so called principal angles between subspaces and is easily computed from the original data matrices via QR decomposition and SVD.

57

Object-Image Metrics

Two ways to compute an “object to image” distance

1. Object SpaceCompute the minimum distance in object space from the given object to the set of all objects capable of producing the given image

2. Image SpaceCompute the minimum distance from the given image to the set of all images produced by that object

58

Object-Image Metrics & Duality

],[inf],[Obj xxdxud uEuxu

],[inf],[Img xExu

uudxudx

Duality Theorem: ],[],[ ImgObj xudxud

xu: all objects that could have produced the

image.

ux: all images of the object.

x

uObject-Image

Relations

Matching can (in principle)Matching can (in principle) be performed in either object be performed in either object or image space without loss of performance !or image space without loss of performance !

GX n /3 GU n ˆ/2Object Shape Space Image Shape Space

59

Duality

• Theorem - with suitable normalization

These metrics are the same!

In the GWP case this distance turns out to

be the distance between two subspaces of

different dimension defined again by using

principal angles.

60

Image Geodesics

•2 random images

•Geodesic between them

•Not Linear

•Not the projection of a line

•Not even coplanar•Geodesics on the this cone have the same length as the calculated image distance!

61

Orthographic Shape Space3 Points in 1-D & 2-D

• 3 points modulo translation, rotation, reflection yields…

• 1-D: Surface of a 30o cone w/ axis along {1,1,1}

• 2-D: Interior of the cone

• {0,0,0} object @ origin• {a,a,b} objects partition cone• Scale: lines through origin

• Geodesics on the this cone have the same length as the calculated image distance!

-2

0

2

-2

0

2

-4

-3

-2

-1

0

-2

0

2

-2

0

2

Rotation on the ‘wrong’ side aboutthe centroid rotates the cone (isotropy condition).

62

Object-Image Relations (1)

•Fix an Object

•Set of Images it can produce

•Always circumscribe the cone

•Not conic sections!

•Equilateral triangle produces

a slightly smaller circle

•‘Image’ produces line to origin

63

Object-Image Relations (2)

• Fix an Image• Set of objects that could

produce the given image• “Bent over cone”• Touches along line through

origin and image• Eventually converges to the

cone surface– Large objects must be

nearly collinear to produce the image

64

Epsilon Balls & Noise Analysis

• The set of all shapes of distance 1 from the given shape (image)

• The image + Gaussian IID noise added to each image point location(std. dev. 0.5)

• Still working the analytic model of noise in the shape space

65

Intrinsic Separability

• How many different shapes can I hope to identify?– Shape space as a unit volume– Epsilon balls defined by metric– Noise balls generated by Gaussian noise

• 5 points in 3D (Generalized Weak Perspective)– Epsilon in [0, 0.73] (max radius of ball)– P(Random Shape in Epsilon Ball)=1.37*Epsilon

• Example: Epsilon=0.01– P(Random Shape in Epsilon Ball)=0.014– Requires as least 73 balls to cover shape space– (Could be more or less efficient coverings)– Separability on a gross level

66

Summary

• Integration of Sensing and Processing• Active Vision• ATR Theory

• These are all connected & overlapping areas• Provides a rich field of problems and applications

ATR Theory

Active VisionISP

67

Open Problems in Inf. Exploitation

• Open Problems: ATR Theory– Classification before Recognition before ID– Representation

• Modeling free forms spanning discrete-continuous• Uncertainty in models (i.e. target variability)?• Cross-sensor phenomenology (registration / fusion)• Correspondence

– Unmixing: automated methods for separating foreground / background in various scenarios (ID before segmentation or pose estimation)

– Intrinsic Separability• How objects separate in quotient space as a function of sensor• Confusers• Unknowns: How to separate knowns from unknowns

– (Object-Image) Metrics• Efficient Search of Large Databases • How to choose the metrics based on the expected noise model & the type of

quotient space derived from the choice of metric• Long Poles:

– Probabilities for Fusion / Reasoning with Uncertainty – Recognition By Components (including construction/decomposition) – Non-Gaussian, Nonlinear Analysis (i.e. most models and algorithms assume

these two properties) – Adaptive systems– How to modulate the prior probabilities on-the-fly

closing the loop for isp using performance prediction dec-05

Documents