WatchdogConfident Event Detection in Heterogeneous Sensor Networks

Matthew Keally1, Gang Zhou1, Guoliang Xing2

1College of William and Mary, 2Michigan State University

Overview

Problem Statement

Challenges

Related Work

Contributions

Design

Evaluation

2

Confident Event Detection

Many applications for event detection have stringent accuracy requirements and demand long system lifetimes Vehicular traffic monitoring Falls in elderly patients Military/intrusion detection

Perform confident event detection Meet user-defined false positive and false negative rates in

the presence of in-situ sensing reality Reduce energy usage to extend system lifetime

3

Challenges of Confident Event Detection

How to cluster the right sensors to meet user accuracy requirements? Learn the detection capabilities of individual sensors and

clusters Use part of the detection capability to meet user

requirements and save energy

How to efficiently perform collaboration between heterogeneous sensors to meet user requirements? Difficult for modality-specific models and data fusion Need a generic solution

How to adapt detection capability to runtime observations? Easier observations and harder observations need different

detection capabilities

4

Related Work

Sensing Coverage Do not address user accuracy requirements Do not explore detection capability of deployment

Modality-specific Sensing Models and Data Fusion User requirements not met in reality Difficult to perform heterogeneous sensor fusion Do not cluster the right sensors to meet user requirements

Machine Learning Do not address user accuracy requirements Do not adapt sensing capability to runtime observations

5

Motivation: Related Work Shortfalls

Vehicle Detection: sensing irregularity Same distance, different accuracies Accuracy can increase with distance

Sensing Coverage may overdetect or underdetect events

Theoretical sensing models assume all sensors are identical

6

Motivation: Related Work Shortfalls

Different clusters (C1,C2,C3) have the same accuracy, 100%, better than individual sensors Difficult to capture for existing works: Due to lack of

knowledge of detection capability of different sensors and clusters

7

Watchdog Contributions

A confident and energy efficient event detection framework Choose the right sensors to meet user requirements Generic framework that provides heterogeneous sensor fusion

Adapt detection capability to runtime observations Easy observations: low-power sentinel sensors Hard observations: higher-power reinforcement sensors

Performance evaluation: two scenarios Monitor traffic entering and leaving computer science building Vehicle detection using Wisconsin trace data Compare against sensing coverage and signal attenuation

model

8

Watchdog Design Overview

Efficient heterogeneous collaboration

Explore detection capability of a deployment

Cluster the right sensors to meet user requirements

Adapt detection capability to runtime observations

Node

Local Aggregation

Runtime EventDetection

RequestReinforcement

Data

ClusterGeneration

Sentinel and ReinforcementSelection

Sensor

Aggregator

TrainingResults

Observations

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

9

Cluster Generation

Goal: determine detection capability of Individual sensors and sensor clusters A specific deployment

Method Randomly generate up to M clusters for each cluster size For each generated cluster

Step 1: Train a Hidden Markov Model for the cluster HMM is good for heterogeneous sensor fusion HMM captures time and space correlation of sensor data

Step 2: Determine cluster FP/FN based on the HMM decision and ground truth at each time interval

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

10

Step 2: Determine cluster FP/FN based on the HMM decision and ground truth

At each aggregation interval:

Determine event detection decision with trained HMM

Compare cluster detection decision with ground truth

Get the cluster FP/FN (accuracy)

Determine FP/FN for each possible event probability

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

11

Sentinel and Reinforcement Selection

Choose sentinel cluster: low detection capability

– Meets user's FN requirement

– Makes easy detection decisions

Choose reinforcement cluster: higher detection capability

– Meets both FP and FN requirements

– Used to make more difficult detection decisions

All other sensors go to sleep

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

12

Runtime Event Detection

Goal: adapt detection capability to runtime observations

– Easier observations and harder observations need different detection capabilities

Method:

– Sentinels and reinforcements form local observations at each aggregation interval

– Sentinels report non-default observations to the aggregator to make detection decisions

– Reinforcements requested when sentinel event probability false positive rate exceeds user requirements

– Reinforcements return non-default observation data and aggregator makes a confident decision

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

13

Runtime Event Detection

User requirements: u.FN = u.FP = 0.05

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

14

52

54

56

60

AcousticSeismic

Sentinels

Reinforcements

Aggregator

Time interval 0 1 2 3 4 5

t=1: No Event, s.FN = .01 < u.FNt=2: Event, s.FP = .02 < u.FPt=3: No Event, s.FN = .01 < u.FNt=4 :Undecided, s.FP = .45 > u.FPt=4 :Event, r.FP = 0.3 < u.FPt=5: No Event, s.FP = 0.2 < u.FP

Evaluation

App1: Wisconsin SensIT trace data

– Vehicle detection at a fixed location

– 75 nodes with acoustic, seismic, and infrared sensors

– 100ms aggregation interval

App2: Computer Science Building Traffic Monitor

– Five IRIS motes mounted on main entrance door

– MTS 310: 2-axis accelerometer, 2-axis magnetometer, acoustic, and light sensors

– Define event as when someone opens the door and walks through

– 4s aggregation interval

Compare with a modality-specific sensing model

– Distance-based signal attenuation

– Data fusion for event decisions

Compare with V-SAM, a state of the art protocol for handling sensing irregularity

– Measure data similarity between sensors

– Keep awake only sensors with dissimilar readings

15

Only a limited & discrete number of FP/FN rates supported by the deployment

For a specific FP/FN rate, a large number of clusters may be available

During runtime detection, Watchdog meets FP/FN explored during training

Exploring Detection Capability & Meeting Requirements

16

Compare with V-SAM: Accuracy

V-SAM with k-coverage and similarity coverage

Watchdog outperforms all with near perfect accuracy

17

Compare with Modality-Specific Sensing Model: Accuracy

Vehicle detection with acoustic sensors

– Select clusters with two different ranges to target location: near (<25m) and far (>40m)

Watchdog always meets user requirements

Modality-specific model ignores in-situ sensing reality

18

Compare with Modality-Specific Sensing Model: Energy

Watchdog clusters the right sensors to meet user requirements

– Meets requirements with reduced energy

Watchdog adapts its capability to runtime observations to save energy

Modality-specific sensing model uses all sensors in the cluster19

Adapting Detection Capability to Runtime Observations

Experimental setting

– Vehicle trace data and sensors from <25m

– User requires 0% false positives and false negatives

Watchdog clusters the right sensors to meet user requirements

Neither V-SAM nor the modality-specific sensing model adapts detection capability to runtime observations

Sentinel FP/FN(%)

Reinforcement FP/FN (%)

Reinforcement Requests (%)

9.5/0.0 0.0/0.0 21.0

20

Conclusions and Future Work

Existing works do not provide event detection with confidence, we need to

– Cluster the right sensors to meet user requirements

– Provide a generic approach for heterogeneous deployments

– Adapt detection capability to runtime observations

Watchdog: a confident event detection framework

– Meets user accuracy requirements

– Exceeds accuracy of existing approaches

– Uses knowledge of detection capability to save energy

Future Work

– Online and distributed detection

21

Thanks to NSF grants ECCS-0901437 and CNS-0916994

22

Compare with V-SAM: Training Length

Watchdog achieves maximum performance with a short training

V-SAM requires little training, but is less accurate

23

Local Aggregation

Allows for heterogeneous sensor fusion

Raw data is combined to form a single observation

– Use a common aggregation technique

Discrete, finite number of possible observations

– Same number for each sensor and modality

– Allow for comparison between sensors of all modalities

– We use two discrete observations

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

24

Event Probability Discussion

Differentiate the accuracy between different event probabilities

– Some observations are more reliable than others

– Probabilities near 0.5 are more inaccurate

Determine FP and FN for each of p probability ranges (p=10)

– Probability between .1 and .2 has zero false negatives

– Probability between .9 and 1.0 has 6% false positive rate

– Ranges with no events have 100% false positive or false negative rates

Local AggregationCluster GenerationSentinel & Rein. SelectionRuntime Event Detection

25