1

Managing sensors with uncertain performance

characteristics

Mark P. Kolba and Leslie M. CollinsECE Department, Duke University

MURI ReviewJune 2006

Sensor management“[Directing] the right sensor on the right platform to the right target at the right time”1

GOAL: Development of an effective, realistic sensor management framework for the landmine detection problem

Manage an increasingly diverse and complex suite of sensors to achieve rapid detection of landminesKeep operator out of harm’s way

Information-theoretic static target detection (IT-STAD) frameworkAn information-based formulation by Kastella is chosen as the basis for this work2

Computationally tractableSuitable for realistic use

Maximization of a measure of information is reasonableMathematical framework for sensor managementChoice of information measure is flexible

1 R. Mahler, Objective functions for bayesian control-theoretic sensor management, I: multitarget first-moment approximation. Proc. IEEE Aerospace Conf., vol. 4, p. 4/1905-4/1923, 2002.2 Kastella, K., Discrimination gain to optimize detection and classification. IEEE Trans. Systems, Man, and Cybernetics—Part A: Systems and Humans, 1997. vol. 27, no. 1, pp. 112-116.

2

0 2 4 6 8 100

1

2

3

4

5

6

7

8

9

10Cell grid containing five targets

IT-STAD frameworkM sensors search for N targets in a gridBinary cell states and sensor observationsState probabilities calculated as

Sensor takes next observation to maximize expected discrimination gain

xc,k is observation k in cell cXc,k is observations 1, 2, . . ., k in cell c

( ) ( ) ( )( )∑

= ==

==1

0 ,

,, ln,

s Qc

PcPc sSP

sSPsSPQPD

Sc = s denotes the state of cell c being s

( ) ( ) ( )( ) ( )∑

=−

−

==

==== 1

01,,,

1,,,,

jkcccmkc

kcccmkckcc

XjSPjSxP

XsSPsSxPXsSP

( )[ ] ( ) ( )∑=

+++ ==1

0,,1,1,,1, ,,,

jkcmkcckckcckc XjxPQPDmXQPDE

( ) ( )[ ] ( )ckckcckckc QPDmXQPDEmXD ,,,, ,,1,, −=∆ +

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

Additional featuresIncorporates constrained and unconstrained sensor motionAllows sensor platforms with multiple sensing modalities on each platformIncorporates sensor cost of use and greedily maximizes the ratio of expected discrimination gain to observation costAllows non-uniform priors to take advantage of a priori knowledge about the scenario at handIs robust to unknown target number in the initialization of state probabilities

3

Sample results

1 sensor 5 targets

uniform prior “road” prior

multimodal sensing:Simulations compare

discrimination-directed search to direct (blind) search using different SNR values or sensor combinationsUse probability of error as performance metric

Pd Pf cost

S1: 0.90 0.40 1

S2: 0.90 0.20 1

S3: 0.99 0.02 10

Uncertainty analysisConsider a real-world scenario: unknown and irregular ground, unfamiliar obstacles, unknown target and clutter types, unknown propagation characteristics

Uncertainty is present in the problem

Uncertainty in Pd and Pf may be both assumed and/or true, creating four cases:

next

nextfinished

Assumptioncertain uncertain

Truth certainuncertain

PD/PF

Uncertain Pd and Pf will have beta densities (natural conjugate prior) parameters r and k

Smaller k corresponds to more uncertaintyConsider three uncertainty levels: low, medium, and high, for which k = 100, k = 10, and k = 5, respectively

next

4

New mathematics

Probability of making an observation given the cell state:

( ) ( ) ( )( ) ( )∑

=−

−

==

==== 1

01,,,

1,,,,

jkcccmkc

kcccmkckcc

XjSPjSxP

XsSPsSxPXsSP

( )[ ] ( ) ( ) ( )∑∑= =

+++ ====1

0

1

0,,1,1,,1, ,,,

j skcccmkcckckcckc XsSPsSjxPQPDmXQPDE

Maintain densities for Pd and Pf in each cell: Pd,c and Pf,c

( ) ( ){ } ( ) ( ){ }( ) ( ){ } ( ) ( ){ }cfPcccfPcc

cdPcccdPcc

PfESxPPfESxPPfESxPPfESxP

cfcf

cdcd

,,

,,

,,

,,

1000111011

ββ

ββ

−======−======

State probability update and expected discrimination calculation:

Update Pd,c (or Pf,c) after an observation:

( ) ( ) ( )( ) ( )∫ =

===

cdcdccdc

cdccdccccd dPPfSPxP

PfSPxPSxPf

,,,

,,, 1,

1,1,

Pd,c and Pf,c densities are maintained for each of the M sensors

Uncertain Pd and Pf

First consider when uncertainty is truly not present in the problem

1 sensor, 5 targets 3 sensors, 5 targets

5

Uncertain Pd and Pf

certain

k = 10

k = 100

k = 5

1 sensor 5 targets

Uncertainty truly present

Effect of uncertainty modeling

1 sensor 1 target

1 sensor 5 targets

3 sensors 5 targets

6

Uncertainty on real dataApply uncertainty modeling to GA Tech dataPerformance is improved most significantly using k = 10, with nearly a 50% reduction in Pe at time = 1000All uncertainty modeling provides some improvement

Pd Pf cost

S1: 0.850 0.323 1

S2: 0.850 0.085 1

S3: 0.950 0.056 1

Performance analysisGOAL: Development of an effective, realistic sensor management framework for the landmine detection problemOBSERVATION: Uncertainty is present in realistic problems, meaning that information that is assumed by the sensor manager will not correspond precisely with truthAnalyze performance of the sensor manager when information that is utilized within the framework is not correct

Erroneous prior informationMismatched densities for uncertain Pd and Pf

7

Erroneous priorSeveral prior densities are created for the erroneous prior analysis

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

uniform vertical road high-level scan horizontal road offset vert road

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

2 4 6 8 10

1

2

3

4

5

6

7

8

9

10

shift = 0 shift = 1 shift = 2 shift = 3 shift = 4

Robustness to both large and small changes examined

Erroneous prior results

1 sensor, 5 targets

Plot legends show the prior density that is assumed

1 sensor, 5 targets

Performance is robust to small shifts in the prior density, while large shifts cause significant performance degradations

8

Erroneous prior results

Previous results have compared discrimination-directed search performance using erroneous priors with direct search performance using correct priorsAlso important to compare discrimination-directed search using erroneous priors with direct search performance that would be obtained using those same erroneous priors

Case 1: Vertical road (shift = 0) is true Vertical road (shift = 0) is assumed

Case 2: Vertical road (shift = 0) is true Vertical road (shift = 4) is assumed

Mismatched beta densities

In sensor management framework, beta densities are used to describe sensor Pdand Pf when uncertainty is present

Three different uncertainty levels: k = 100, k = 10, and k = 5

Examine performance when the true and assumed beta densities are mismatched

( ) ( )( ) ( ) ( ) 11 1,| −−− −

−ΓΓΓ

= rkr xxrkr

kkrxfβ

Truth: Assumption:

k = 10k = 100, k = 10, or k = 5

OR

Truth: Assumption:

k = 100k = 100, k = 10, or k = 5

For example:

9

Mismatched beta resultsPlot legends show the beta density that is assumed

1 sensor, 5 targets 1 sensor, 5 targetsk = 100 density is true k = 10 density is true

Mismatched beta results

When there is low uncertainty (k = 100), there is little performance difference for any of the assumptionsFor medium uncertainty (k = 10), assuming that high uncertainty (k = 5) is present causes minimal performance loss, and vice versaFor medium and high uncertainty, assuming that low uncertainty is present causes a noticeable performance degradationThese results suggest the following:

Safer to assume higher uncertainty rather than lower if unsurePerformance is reasonably robust to mismatches in beta densities

1 sensor, 5 targetsk = 5 density is true

10

Declaration-based approachGOAL: Development of an effective, realistic sensor management framework for the landmine detection problemOBSERVATION: Current performance metric, Pe, displays a number of inadequacies when considering an applied setting

Does not give direct information about Pd and Pf as is often desired in landmine detection applicationsPe calculation as it has been formulated requires knowledge of the number of targets present in the sceneEstimated cell locations of the targets are selected based on the largest posterior state probabilities of containing a target

Reasonable if target number is known . . . but consider the following example:

P(no target | data):P(target | data):

Cell number: 1 2 3 4 5

0.99 0.99 0.99 0.97 0.990.01 0.01 0.01 0.03 0.01

Would an operator actually wish to say that a target is present in cell 4?

Searching for one target:

Declaration-based approachRather than estimate the target locations based on the largest posterior state probabilities, make declarations about the contents of each cell based on the data that has been observed

Possible declarations: target, no target, undecided (need more info)

Declarations model realistic behavior and also allow Pd and Pf to be calculated and compared to the total number of measurements or to a total cost measure for use as a performance metricBenefits of declaration-based approach

Pd and Pf may be straightforwardly calculatedKnowledge of the number of targets in the scene is not requiredAvoids the problem of choosing low-probability cells as containing targets

To implement the declaration-based approach, use the sequential probability ratio test (SPRT)8

8 Wald, A., Sequential Analysis. New York: John Wiley & Sons, Inc., 1947.

11

SPRT implementationHypothesis is target vs. no target:

Observations are binary:

After m observations have been made in a cell, calculate Zm:

Once a TARGET or NO TARGET declaration has been made, that declaration is final and will not be changed

( )( )

( )( ) f

d

f

d

PHxfPHxf

PHxfPHxf

−==−==

====

1|01|0

|1|1

0

1

0

1

H0: no target presentH1: target present

If B < Zm < A declare UNDECIDEDIf Zm ≥ A declare TARGET PRESENTIf Zm ≤ B declare NO TARGET PRESENT

( ) ( ) ( )( ) ( ) ( )00201

11211

||||||HxfHxfHxfHxfHxfHxfZ

m

mm

L

L=

Thresholds A and B defined as

αβ

αβ

−=

−=

1 and 1 BA

α is Type-I error (choose H1 when H0 is true)β is Type-II error (choose H0 when H1 is true)

Simulation resultsResults are presented for the following search techniques:

Discrimination-directed search: Uses sensor manager. Once a final declaration is made in a cell, that cell is never observed againDirect search (w/o skipping): Blind search that continues to observe all cells on each pass through the grid (no information from sensors incorporated into search pattern)Direct search (w/ skipping): Blind search that sweeps through the grid but skips cells that have a final declaration (primitive sensor management)

α = 0.05β = 0.004

12

Simulation results

42450.94627930.976871.999Direct (w/ skip) M

7250.9027460.964766.9977700.997

Direct (w/ skip) NM

40370.95430950.96913481.000Direct (no skip) M

15980.92916410.9531215.99712830.996

Direct (no skip) NM

24510.97218840.9915171.000Disc M

4600.9125190.9624900.9974670.997

Disc NM

costE[Pd]costE[Pd]costE[Pd]costE[Pd]

k = 5k = 10k = 100certain

Both discrimination-directed and direct search in the table above are at 0dBNM: Uncertainty not modeledM: Uncertainty modeled

Cost is given in arbitrary time units

Now consider searches at 0 dB when sensor Pd and Pf are uncertain

AMDS dataData for 320 cells: 92 mines, 178 clutter objects, and 50 blanksTwo sensors: GPR and EMIFor each sensor, binary observations are generated by processing the sampled portion of raw data and comparing the resulting decision statistic to a threshold

GPR: summed, whitened energyEMI: energy

0 200 400 600 800 1000 1200-100

0

100

200

300

400

500

13

AMDS resultsPerformance is plotted as Pd vs. costPd vs. Pf curve is also givenDiscrimination-directed search achieves the same Pd at lower cost than either of the direct search techniques

Pd Pf

EMI: 0.793 0.531

GPR: 0.801 0.509

Each sensor has cost of use equal to 1

α = 0.05

β = 0.05

AMDS results

Now incorporate uncertainty modeling; sample results presented for k = 10

Uncertainty modeling increases the cost, but allows better Pdperformance to be achieved after a large number of observations

14

AMDS resultsAnother useful performance metric to consider is the expected probability of detection after a large number of observations have been madeUncertainty modeling improves the expected Pd

Discrimination-directed search provides the best expected Pd at the best expected cost (with and without uncertainty modeling)

28,5400.9446,5690.8375,1310.808Direct (w/ skip)

22,4620.84325,4320.82924,4570.812Direct (no skip)

26,1510.9505,5920.8314,4340.804DiscrimE[cost]E[Pd max]E[cost]E[Pd max]E[cost]E[Pd max]

k = 10k = 100certain

Expected costs are given in arbitrary time units (i.e., same as Pd vs. cost plots)

ConclusionsThe IT-STAD framework for sensor management has been presented, based on Kastella’s discrimination gain technique, that incorporates multiple sensors and targets, realistic cost constraints, and uncertainty modelingExtensive simulation has demonstrated that discrimination-directed search performance is superior to the performance of a direct search technique; the performance improvement is typically 3-6dBPerformance of the sensor manager has been shown to be robust to reasonable errors in assumed information and to be computationally superior to an alternative sensor management technique (static-detection JMPD)The sensor manager has been successfully implemented on real landmine data (GA Tech data and AMDS data), and the IT-STAD sensor manager has again outperformed direct search on the real datasets