human factors aspects of anomaly detection...

NRC COHSI 2/12/20091

Human Factors Aspects of Anomaly Detection Systems

Thomas Sanquist, Thomas Sheridan, John Lee, Nancy Cooke

Committee on Human-System IntegrationNational Research Council

February 12, 2009


Background

• COHSI outreach to government agencies in September, 2008

• An underlying theme among a number of agencies seemed to be anomaly detection related to various applications:

– Safety

– Security

– Traffic management

– Diagnosis

• Consensus on committee to explore general area in greater depth at current meeting

• Many other applications beyond these areas: medicine, industrialprocess monitoring, aircraft predictive maintenance, energy system management, etc.

• Discussion format


Session Overview

• Characterize general features of human-mediated anomaly detection systems

• Review issues and engineering approaches to anomaly detection

• Review selected aspects of application examples

– Radiation detection

– Landmine detection

– Visual scene surveillance

– Shipping traffic patterns/maritime domain awareness

– Operator Impairment Detection (Lee)

– Group Communication Anomalies (Cooke)

• Common human factors issues across types of systems

• Knowledge gaps – how to address them?

• Role of NRC workshops and consensus studies


Anomaly Detection Defined

• Finding patterns in data that do not conform to expected behavior.– Anomalies, outliers, discordant observations, exceptions, aberrations, surprises,

peculiarities, contaminants.– Most of our training emphasized the use of averages, but there is also value in

studying extreme values in a variety of fields

• Applications: – Fraud detection– Insurance & health care– Intrusion detection– Fault detection in safety critical systems– Surveillance for military and security

• The term anomaly detection first came into the literature in the mid-1980s in the realm of computer network intrusion detection systems (IDS)


The Principal Messages

• Very little HFE work in these types of systems (except air passenger screening & landmine detection)

• Basic research dominated by algorithm development and refinement

• Applied research involves demonstration projects

• Demonstrations do not entail systematic data collection from operators– This feedback loop can help to improve technical aspects of

systems

• Systems engineering risk:– Large-scale technical systems deployed which require constant

staffing to compensate for technical performance issues


Selected Resources

• Chandola, Banerjee and Kumar (2009). Anomaly Detection: A Survey. ACM Computing Surveys, in press.

• Axelsson, S. (2000). The Base-Rate Fallacy and the Difficulty of Intrusion Detection. ACM Transactions on Information and System Security. 3(3). 186 – 205.

• Chen, H., et al. (2005) Imaging for Concealed Weapon Detection. IEEE Signal Processing Magazine. 52.

• Dee, H.M and Velastin, S.A. (2008). How close are we to solving the problem of automated visual surveillance? Machine Vision and Applications. 19: 329 – 343.

• Kristin M. Schweitzer; Andrew S. Bodenhamer (2007). Visual Detection of Land Mines. ARL-TR-4073.

• MacDonald, J., et al. (2003). Alternatives for Landmine Detection. RAND Monograph Report 1608.

• Parasuraman, Sheridan, Wickens (2000). A Model for Types and Levels of Human Interaction with Automation. IEEE Transactions on Systems, Man, and Cybernetics Part a: Systems and Humans, Vol. 30, No. 3.

• USCG Acquisition Directorate, NAIS Fact Sheet http://www.uscg.mil/ACQUISITION/programs/pdf/NAIS.pdf

• DHS National Plan to Achieve Maritime Domain Awareness. http://www.dhs.gov/xlibrary/assets/HSPD_MDAPlan.pdf

• Valera and Velastin (2005). Intelligent Distributed Surveillance Systems: A Review. IEE Proceedings on Visual Imagery and Signal Processing. 152(2). 192 – 204.

• very few human performance studies in this area


A Personal Example

• Call from Columbus, OH credit union before 8AM PDT Monday morning (how do I know who they really are....?)

• Unusual transactions flagged by Bank of America neural net software• Have you traveled to Atlantic City recently?• Do you have your credit card? (no....uh-oh....)• Cash transfer transactions on card with $9K limit:

– Saturday evening: $12, Sands Swingers– Sunday: $539.99, Harrah’s– Sunday: $2627.99, Bally’s Park Place– Sunday: $2629.99, Sands Hotel & Casino– Sunday: $2629.99, Harrah’s

• Anomalies were detected, reported to credit union keeping normal business hours.

• System failed to alert in time to prevent loss • Numbers and timing suggest perpetrators knew how system

worked• My loss covered by credit union insurance


Why are anomalies important?

• Associated with significant actionable information:

– Credit card fraud

– Excessive work hours, worker impairment

– Smuggling

– IED location

– Terrorist activity

– Medical problem

• From scientific standpoint, study of outliers enriches our understanding of human capabilities and limitations, such as:

– “short” and “long” sleepers

– “larks” and “night owls”

– Neuropsychological studies, e.g., HM,

AJ


Three general types of human interaction

PhysicalSignal

Amplify

Condition PresentOr Absent?

PhysicalSignal

Amplify, Process, Alarm Criteria

Operator Evaluationand Resolution

PhysicalSignal

Condition PresentOr Absent?

?

NRC COHSI 2/12/200910

Human Interaction and Levels of AutomationLEVELS OF AUTOMATION IN SIGNAL DETECTION

COMPUTER HUMAN

1 NA Senses raw data

Decides S or N

2 Senses raw data Observes display

Displays it in human friendly format Decides S or N

3 Senses raw data Observes display

Performs initial filtering of noise Decides S or N

Displays result to human friendly format

4 Senses raw data Observed display

Decides S or N Considers computer decision

Also displays human friendly filtered data Makes final decision S or N

5 Senses raw data Observed display

Decides S or N Considers computer decision

Also displays human friendly filtered data Also considers confidence etc.

Also displays confidence or other parameters Makes final decision S or N

NRC COHSI 2/12/200911

Human Factors and Anomaly Detection

• Provide a means for human awareness of sensory or non-sensory phenomena, and/or alert human attendants to events of interest on the basis of automated decision criteria

• There are varying levels of human involvement in this process

• Human operators can provide: physical detection, decision makingverification, interpretation and classification, context.

• New jobs/tasks created to handle resolution of imperfect system output

• Influenced by many “traditional” human factors such as workload,signal-to-noise ratio, pace of data flow, length of watch.

• System-specific human factors such as low base rate of events, complexity of masking, transparency of algorithms, high false alarm rates.

NRC COHSI 2/12/200912

NRC COHSI 2/12/200912

The Trust Issue

Calib

rate

d Tru

st

Overtrust:Trust exceedssystem capabilities

Distrust:Trust falls short of system capabilities

Trust

Automation Capability(trustworthiness)

Types of information underlying trust

• Purpose: Intended application

• Process: Sensors and data processing algorithms

• Performance: Precision and consistency

NRC COHSI 2/12/200913

Challenges

• Difficult to define region encompassing all possible normal behaviors.– Boundary between normal and anomalous not precise (e.g., taking

pictures at Penn Station)

• Malicious adversaries adapt to make anomalous observations appear normal

• Across domains the definition of normal evolves

• Exact notion of anomaly varies across domains:– Small deviation in body temp might be anomalous

– Small deviation in stock market normal (not that we have seen any of these lately…..)

• Availability of labeled (ground truth) data for training of models use for anomaly detection a major issue

• Data often contain noise which looks similar to anomalies

NRC COHSI 2/12/200914

Types of anomalies

Collective anomaly – multiple data instances which by themselves are not anomalous, but contiguous occurrence makes them so

Point – individual data instance deviant with respect to the rest

Contextual – anomalous with respect to surrounding context

NRC COHSI 2/12/200915

Many Data Processing Techniques and Variants

• Statistical Profiling

• Neural Networks

– Multi-layered perceptrons, Neural trees, Adaptive Resonance Theory, Radial Basis Function, Hopfield Networks, Oscillatory Networks

• Support Vector Machines

• Rule-based systems

• Bayesian Networks

• Clustering Algorithms

• Spectral Analysis

• Nearest Neighbor Analysis

NRC COHSI 2/12/200916

A typical paragraph from an anomaly detection article….why we need the HF perspective

NRC COHSI 2/12/200917

Comparison of decision architectures

• Traditional 4-stage human performance model

• Typical Signal Processing Algorithm for Land Mine Detection

Sensory Processing

Perception/Working Memory

Decision MakingResponse

Selection & Action

Preprocessing Detection Discrimination Decision

Output

NRC COHSI 2/12/200918

Comparison of decision architectures, continued

• Visual surveillance systems

NRC COHSI 2/12/200919

Taxonomy for Anomaly Detection Systems –a means for comparing across domains

• Main Purpose

• Domain of Application

• Time Frame of Detection (real-time, post-hoc)

• Role of Human Operator (sensory, interpretation, adjudication....)

• Secondary applications

• Nature of Anomaly Data (point, collective, contextual...)

• Data Processing Approach

• Data-to-Construct linkage

• Base rate of occurrence

• False positive rates

• Positive Predictive Value of Alarm

• Ground truth data availability

NRC COHSI 2/12/200920

Some specific anomaly detection systems

NRC COHSI 2/12/200921

Radiation Detection

NRC COHSI 2/12/200922

Radiological Threat

– Radiation Portal Monitoring (RPM) program, Domestic Nuclear Detection Office & Customs and Border Protection

– Nuclear Weapons

• State Weapon

• Improvised Nuclear Weapon

– Nuclear Weapons Material (Special Nuclear Material - SNM)

– Radiation Dispersal Device (RDD)

– Radiological Material for use in Construction of a RDD

– Other Illegal or Illicit Radioactive Material (i.e. ContaminatedSteel, inappropriately manifested/marked material, other radiological contraband)

NRC COHSI 2/12/200923

Elements of RPM system

NRC COHSI 2/12/200924

Basis of human factors problem with radiation detection:poor threat classification

NRC COHSI 2/12/200925

Fundamental human factors issues inradiation portal screening

• Low probability – high consequence events

– Low base rate of illicit nuclear material movement (estimate = 4.5 smuggling events per 11,000,000 commercial truck transits)

– Comparatively higher rate of “nuisance alarms” (multiple times per day) – Naturally Occurring Radioactive Material: ceramics, fertilizer, etc. (1 –2 NORM alarms per 100 vehicles)

– Very low probability of true threat alarm (1 every 2- 4 years)

• Frequent attention to non-informative alarms leads to mistrust in automation, complacency, possibly ignoring/disabling alarms

• Dedicated manpower requirement to resolve nuisance alarms –estimated cost for one major port = $15M

• Potential solution: threat likelihood alarms

– Requires extending the simple point anomaly detection based on standard deviation of measure over background radiation

– Incorporate more sophisticated sensors (spectral), energy windowdiscrimination, and fusion of manifest and notice-of-arrival data.

NRC COHSI 2/12/200926

Radiation Detection Anomaly Detection Summary

• Main Purpose: Threat Detection and Interdiction

• Domain of Application: Radiation Detection at Border Crossings

• Time Frame of Detection: Real-time

• Role of Human Operator: Evaluation and adjudication of system output

• Secondary applications: Prevention of contaminated material entering country

• Nature of Anomaly Data: Point

• Data Processing Approach: Statistical, Spectral

• Data-to-Construct linkage: Moderate (range of threat types considerable)

• Base rate of occurrence: unknown but assumed to be extremely low(estimates from IAEA)

• False positive rates: Very high

• Positive Predictive Value of Alarm: Very low

• Ground truth data availability: yes, based on secondary exam and manifest declaration

NRC COHSI 2/12/200927

Land Mine Detection

NRC COHSI 2/12/200928

The Land Mine Problem

• 15 – 20,000 victims per year in 90 countries

• 40 – 50 million mines remain to be cleared

– 100K per year are cleared

– 1.9 million new mines laid every year

• Extensive contamination of agricultural land in Afghanistan reduces usage by up to 80%

• Devastating human and economic consequence

NRC COHSI 2/12/200929

Fundamental Human Factors Issues inLand Mine Detection

• High consequence event, high probability in certain areas• Demining tools similar to those employed in WWII using electromagnetic

induction (EMI).• Audible signal provided to operator• Limitation is inability to discriminate mines from non-mine metal clutter• False alarm rate = 99.7%, true positive % = 0.3% (500K/200M)• Sensitivity varies by detector, location and soil type• 1 de-miner killed for every 1000 – 2000 mines cleared• Excessive time spent investigating false alarms leads to fatigue and

carelessness. All buried items signaled by detector are investigated manually

• Visual cues are ignored with excessive focus on detector (Davison, ARL)

NRC COHSI 2/12/200930

Solution Approaches for Landmine Problem

• Other technologies being researched: ground penetrating radar, acoustic/seismic, vapor trace detection (very basic level), nuclear quadrupole resonance.

• Dual sensor program to combine EMI and GPR into Handheld Standoff Mine Detection System (HSTAMIDS)– Evolved to AN/PSS-14 (Army-Navy/Portable Special Search)

• Uses dual outputs to operator – this would be an area for further HFE development

• ARL studies show that 33% of simulated mines can be detected on basis of visual cues alone – recommend further training to reinforce this modality, and studies of unique auditory signatures of specific mines.

• Further research with fused multi-sensor data into single output for operator.

NRC COHSI 2/12/200931

Improving Detection of Land Mines Jim Staszewski

Carnegie-Mellon University

• Problem: Poor land mine detection

• Solution: identify expert and bootstrap expert’s detection strategy

• Field Implementation:

– New training adopted by Army

• Evidence of Success:

– Improved detection rates after training

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

PRE POST

P(D

)

CEBES

Training

Traditional

Training

NRC COHSI 2/12/200932

Landmine Detection System Summary

• Main Purpose: Landmine detection and clearing• Domain of Application: Previously mined areas• Time Frame of Detection: Real-time• Role of Human Operator: Evaluation and adjudication of system

output• Secondary applications: None• Nature of Anomaly Data: Point• Data Processing Approach: • Data-to-Construct linkage: Weak - Moderate (range of mines

considerable, much clutter in environment)• Base rate of occurrence: high in identified areas• False positive rates: Very high• Positive Predictive Value of Alarm: low• Ground truth data availability: Yes, but only following physical

investigation

NRC COHSI 2/12/200933

Visual Surveillance Systems

http://www.cernium.com/WMV/Belo_Cernium.asf

NRC COHSI 2/12/200934

Threat Basis

• Crowded public areas

• Unusual or suspicious behavior

• Unauthorized presence, crowd formation

• Left-behind packages/luggage

• Concealed explosives

NRC COHSI 2/12/200935

Watch the Texas Border

• http://www.blueservo.net

• This system links cameras along the Rio Grande to the public via streaming video

• 500 lbs of marijuana seized in December as a result of report from this system

NRC COHSI 2/12/200936

Fundamental Human Factors Issues inVisual Surveillance Systems

• Looking for precursors to low probability, high consequence events (e.g., leaving a bomb behind)

• Data overload: screen to camera ratio = 1:4 to 1:30; ratio of operators to screens up to 1:16

• MABA-MABA – humans are better at detecting unusual circumstances, machines are better at detecting small changes in static or clutter environments. Good for exclusion zone monitoring.

• Boredom – some officers play “hide and seek” with on-the-ground personnel, some train the cameras on their own vehicles, etc.

• There is much contextual knowledge that has yet to be codified as systems are in a relatively primitive state

– PNNL estimates that COTS is 60% accurate at best, with tendency to have higher miss rate than false alarm rate

NRC COHSI 2/12/200937

Visual Surveillance System Summary

• Main Purpose: Unusual event detection in peopled space

• Domain of Application: Public spaces, controlled spaces

• Time Frame of Detection: Real-time, post-hoc

• Role of Human Operator: Evaluation and adjudication of system output

• Secondary applications: None

• Nature of Anomaly Data: Point, contextual, collective

• Data Processing Approach: HMM, Bayesian, Numerical Clustering

• Data-to-Construct linkage: Weak (range of behaviors very high)

• Base rate of occurrence: low

• False positive rates: High (high miss rate too)

• Positive Predictive Value of Alarm: Very low

• Ground truth data availability: Limited

NRC COHSI 2/12/200938

Maritime Domain Awareness

NRC COHSI 2/12/200939

Maritime Domain Awareness (DHS national plan)

• The effective understanding of anything associated with

the global maritime domain that could impact the United States’ security, safety, economy, or environment.

• Achieved by improving our ability to collect, fuse, analyze, display, and disseminate actionable information and intelligence to operational commanders and decision makers.

• Integrate relevant Cold War Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR)legacy systems and operational concepts with current and emerging sensor capabilities and applicable procedures. These capabilities

must be fused in a common operating picture that is

available to maritime operational commanders

NRC COHSI 2/12/200940

Automatic Identification System: An MDA “feed” from a civilian-based system (IMO)

• AIS signal includes:

– Ship ID

– Course

– Ship dimensions

– Cargo

– Destination

– ETA

• VHF system using GPS transponder data and ship navigational instruments (gyrocompass, speed indicator, etc)

NRC COHSI 2/12/200941

Nationwide AIS (from NAIS fact sheet)

• AIS data collected by NAIS will be combined with other government intelligence and surveillance data to form a holistic, overarching view of maritime traffic within or near U.S. and territorial waters.

• How will the Automatic Identification System help to increase security? .... by increasing awareness of vessels in the maritime domain, especially vessels approaching U.S. ports. AIS corroborates and provides identification and position of vessels not always possible through voice radio communication or radar alone.

• Corroboration seems to be a key concept that would be served by anomaly detection.

• NAIS personnel also concerned about using anomaly detection to determine if vessels are “spoofing” the system.

• Traffic management applications of AIS are well-underway; security applications just starting.

• NAIS is a system-of-systems

NRC COHSI 2/12/200942

Nationwide Automated Identification System

NRC COHSI 2/12/200943

Maritime Scenarios

Anomaly detection in the maritime domain

Proc. SPIE, Vol. 6945, 2008; Jean Roy, Defense Canada.

NRC COHSI 2/12/200944

MDA/NAIS operational sequence (adapted from National Plan for Maritime Domain Awareness)

• Intel that IND being carried by cargo vessel

• Electronic Notice of Arrival filed by vessel

• ATS notes anomaly in cargo manifest

• AIS vessel track flagged

• COP shows available security assets which are deployed

• Vessel interdicted and cargo seized

NRC COHSI 2/12/200945

MDA system human factors

• Threat indicators need to be fused from diverse systems which are not yet integrated

• Transparency of this process to human system agents will be more complex than stand-alone systems

• Area charts (maps) used when operational tempo is urgent (wide area, persistent, natural interaction, multiple resource tracking)

• Range of threat scenarios extremely broad

• Lack of operational experience

NRC COHSI 2/12/200946

NAIS/MDAAnomaly Detection Summary

• Main Purpose: Threat Detection and Interdiction• Domain of Application: Maritime vessel, cargo, personnel• Time Frame of Detection: Predictive, Real-time• Role of Human Operator: Detection, evaluation and adjudication of

system output• Secondary applications: Traffic Management (?), • Nature of Anomaly Data: Point, Collective, Contextual• Data Processing Approach: Multiple• Data-to-Construct linkage: Weak (range of threat types

considerable)• Base rate of occurrence: unknown but assumed to be extremely low

despite recent spike in piracy• False positive rates: unknown• Positive Predictive Value of Alarm: unknown• Ground truth data availability: probably if vessel boarded

NRC COHSI 2/12/200947

NRC COHSI 2/12/200947

Operator Impairment Detection

NRC COHSI 2/12/200948

Real-time, behavior-based impairment detection complements alcohol interlocks

§ Sensor limits and low baserate

§ Pharmacodynamics—BAC levels can increase while driving

§ Drinking while driving

§ Impairment under .08 BAC

NRC COHSI 2/12/200949

Behavior-based sensors of alcohol impairmentBehavior-based sensors of alcohol impairment

Sensor technology largely available for production vehicles

NRC COHSI 2/12/200950

Decoupling of eyes and steering

Cross-correlogram—steering and eyes

40ml Vodka 100ml Vodka

Marple-Horvat etl al (2008)

Corr

ela

tion

NRC COHSI 2/12/200951

NRC COHSI 2/12/200951

Behavior-based Impairment Detection Summary

• Main Purpose: Real-time behavior-based impairment• Domain of Application: Driving and other safety-critical situations• Time Frame of Detection: Real-time, post-hoc• Role of Human Operator: Interpretation and behavior adjustment• Secondary applications: Fatigue, distraction, and prescription drug

impairment countermeasures• Nature of Anomaly Data: Point, contextual, collective• Data Processing Approach: SVM, Bayesian• Data-to-Construct linkage: Moderate (neurological basis of alcohol

impairment well understood)• Base rate of occurrence: low• False positive rates: High (potentially high miss rate too)• Positive Predictive Value of Alarm: Moderate• Ground truth data availability: Limited unless there is secondary

investigation.

NRC COHSI 2/12/200952

Detection of Teamwork Failures through Communications Monitoring

and Analysis

NRC COHSI 2/12/200953

Team Failures in Collaboration, Communications, Coordination, Command-and-Control

• USS Vincennes shoots down Iranian airbus (1988)

• Challenger/Columbia accidents tied to poor organizational decision making (1986/2003)

• Response to 9/11 reveals communication breakdowns (2001)

• Katrina response lacked coordination (2005)

• Sago Mine disaster report cites poor command-and-control (2006)

• VA Tech communications substandard (2007)

• Friendly fire incidents• Various health care mishaps attributed to

poor teamwork• Unmanned Aerial Systems

• Real-time detection of teamwork breakdowns needed for just-in-time intervention and prevention

• Team communications (voice, text chat, email) provide ongoing data stream for monitoring

• Identify anomalous patterns; detect change

NRC COHSI 2/12/200954

Detecting Meaningful Patterns in Communication Data

Selection of interesting data using cheapest fastest methods

(e.g., number of words, time speaking)

Analysis identifies data in need of further processing

(e.g., communication flow patterns)

Most expensive/detailed analysis on select data

(e.g., content-based analysis)

Communication

timing stability

ChainMaster,

Procedural

Networks

(PRONET),

transition analysis

Semantic

correlations,

Latent Semantic

Analysis Lag

Coherence

DYNAMIC

Avg. time of

following behavior

Following behavior

(Dominance)

Avg. # of words,

Latent Semantic

Analysis,

Communication

Density

STATIC

TIMINGFLOWCONTENT

Abeg

AendPend

Dbeg

Dend

Pbeg

Dbeg

Dend

Pbeg

Dbeg

Pend Aend

Abeg

P-D fight

PRONET: Communication Flow Analysis

Tie patterns to team performance

Tiered application of analytic methods

NRC COHSI 2/12/200955

ChainMaster Deviations from Expected Enron Email Patterns Map onto Organizational Change

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

1 2 3 4

Time Period

Co

ntr

ol-

Exp

eri

men

tal

Sim

ilari

ty

Enron Files for Bankruptcy

Skilling

Resigns

More

Change

Less

Change

Connect Detected Patterns to Team State

NRC COHSI 2/12/200956

1 sec.

(0,1,1) (0,1,

0)

(0,0,1)

(0,0,0)

(0,0,1)

(1,0,0)

Represent Data to Human

0 500 1000 1500 2000 2500 3000 3500 4000-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5x 10

6

Time (s ec)

Va

r(Y

)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50

0.5

1

1.5

2

2.5

3

3.5x 10

13 Spectral Slope = -1.0576

Frequency

Po

we

r

Sonification of change in flow patterns

NRC COHSI 2/12/200957

NRC COHSI 2/12/200957

Teamwork Failure Detection Summary

• Main Purpose: Real-time detection of teamwork failures• Domain of Application: Team communications• Time Frame of Detection: Real-time• Role of Human Operator: Initial analysis, monitoring, detection,

interpretation, and intervention • Secondary applications: Team training, threat assessment• Nature of Anomaly Data: Point, contextual, collective• Data Processing Approach: Sequential data analysis, dynamical systems

modeling, latent semantic analysis, spectral analysis• Data-to-Construct linkage: Moderate (better for structured tasks; can detect

change and anomalies; diagnosis more difficult)• Base rate of occurrence: low• False positive rates: ?• Positive Predictive Value of Alarm: Moderate• Ground truth data availability: Dependent on other existing measures of

team performance and outcome

NRC COHSI 2/12/200958

Summary

• Threat-oriented anomaly detection systems try to address areas where humans underperform machines– Physical occurrences undetectable by human senses

– Making many repetitive observations

– Continuous operations

• Nuisance alarms and low rates of occurrence for events of interest limit utility

• Across a range of phenomena, from basic atomic quanta to individual and collective behavior, the data-to-construct linkage can be strengthened

• Noise in the system, i.e., normal variation, requires human agents to provide context and interpretation

NRC COHSI 2/12/200959

Research Needs

• Catalogue the range of anomaly systems, beyond threat detection,to identify systems with greater and lesser utility

• Identify performance improvements that can be facilitated by human factors analysis

• Foster cross-domain information exchange

• Promote research approaches that incorporate systematic HF studies into technology demonstrations

• Evaluate cost-benefit implications of low utility system staffing versus developmental or no-action alternatives

• Develop systems on basis of positive predictive value of alarms provided

• Approach through workshops and/or consensus studies via NRC

human factors aspects of anomaly detection...

Documents