1

Trace Detector Evaluation Criteria

Dr. Susan F. Hallowell

Introduction The trace detection industry is offering numerous products to an increasing customer base for the detection and interdiction of explosives, chemical warfare agents, drugs and toxic industrial chemicals for field applications. For the topic of this discussion, only explosive trace detection (ETD) systems will be discussed. These ETD systems detect explosive residues at extremely low levels, and can be highly specific in their identification of each explosive. In order to make objective comparisons of one detector to another, various aspects of each system must be examined on an equal basis. Generally, ETD systems that arrive in airports for testing and evaluation have already passed laboratory qualification. This simply means that systems are able to detect and identify threat substances at a specified level in a simulated environment. What works in a simulated environment does not always work well in the real world. Airport operational tests are critical to reveal how well individual ETD systems meet the challenges of the demanding environment in the airport ecosystem.

How can airports measure performance? An airport may take into account laboratory results indicating systems’ ability to detect threat substances, it is more important that airport decision-makers conduct their own real world operational tests to ensure that these systems work and become part of the seamless flow of passengers through airports. This is measured by:

False Alarm Rate

Through-put/Clear-down

Ease of Operation

False Alarms: Cost Money When a system alarms, it must be resolved. This means checkpoint personnel and alarm resolution protocol must be followed. That process slows throughput, requires additional personnel and may even require law enforcement to respond to a false alarm. Even a 1 percent false alarm rate has a major impact on airport operations. For example, at an airport the size of Heathrow Airport, with 201,000 passenger arrivals and departures per day, that means nearly 2000 false alarms per day. This costs into the millions of dollars per year per airport. (only a few percent are actually ETD) Through-put: Trace Detection should not impact the flow of passengers at checkpoint Legacy systems have traditionally been limited by the number of samples taken per hour and by the ability to come back on-line to analyze the next samples.

2

Systems utilizing ion mobility spectrometry (IMS) have traditionally not been able to screen at a very high frequency. These systems struggle to recover from alarms quickly and can take up to several hours to come back on-line for use after an alarm. Ease of Operation ETD systems originated as an adaption of laboratory instruments. Today, systems are available that deliver laboratory performance to airport screeners and are easy to use. Ease of use means that the systems are plugged in, warmed up, and are ready to screen quickly. Quick to learn, self calibrating, internal diagnostics, easy to operate,

How to evaluate systems in your airports

When conducting tests in your airport, several factors will impact operation of systems. ETD evaluation is based on three major criteria: operational performance, human factors and total cost of operation/ownership. Operational Performance and Human Factors

Interaction of operator with instrument - is it easy for screeners to use? Does the system warm up and become operational quickly at the beginning of a screening

operation? Is it simple to prepare for use? Is the calibration and verification a simple process? Is it simple to use for screening? Are the results easy to interpret? Do the results provide information needed for any follow up questions the screener may need

to ask (e.g. Is it important to know the explosive identity in "clearing” an alarm? Does the screener have any problem collecting the sample or inserting it into the machine? Is the monitor easy to see? Does the system seem to be intuitive and easy-to-use for the screener? Can the screener "do it wrong" or is the detector engineered such that the operation is fairly

fool proof? Is there a good training package associated with the device? Is there a high degree of confidence that alarm is real?

Total Cost of Operation/Ownership Make sure to gain detailed information and specific costs of:

Dopants and calibrants

Consumables

Maintenance

False alarm rates

Radiation-related costs (if the unit has a radioactive source or utilizes ionizing radiation) Other Considerations

Reliability, maintenance, and service offerings o MTBF (Mean Time Between Failure) o What happens when the system needs maintenance?

3

o Does the company have a remote diagnostic capability? o How long does it take to get service?

Is the system is collecting data in a format that meets your needs? o Do you need a record of the number of samples/false alarms per day? o Use by operators o Other factors?

Be aware that there are environmental factors, such as ambient temperature, dust, humidity,

environmental pollutants, and other chemicals that may be present in the environment (deicers,

fertilizers, etc.) which might affect the ETD systems’ performance.

ETD Trial Recommendations Operational Testing: Allow time for testing -- run a minimum of two weeks to two months of airport operational testing. This will yield critical data for purchasing decisions:

How well manufacturers’ claims align with user experience How well systems respond to environmental conditions Complete performance data collection

Test 2-3 units of a particular manufacturer’s system:

Controls for any performance variations between systems Gathers more experimental data Delivers more operator feedback

4

Operational performance

Factors Definition Results/Score

Operational false alarms Operational false alarms are critical measures for

an ETD because the rate impacts throughput and

the availability of the system. ETD systems with

high operational false alarms can also increase the

total cost of operation.

Operational false alarms in the field in %

Number of false alarms per year

Cost of rescanning caused by false alarms

Cost per trap

………………………

………………………

………………………

………………………

Clear-down time The clear-down time represents the time the

system requires to clear out the previous sample

and be ready for the next. Clear-down algorithms

are different from one manufacture to another.

The major key parameters for clear-down are:

Average clear-down time after an alarm

Average clear-down time for non-alarm

samples

………………………..

………………………..

Pd (Probability of detection) Probability of detection (Pd) represents the

probability that the ETD system will detect an alarm

when presented with a threat. The ETD system

includes the instrument, the sampling mechanism,

and the sampling wand. Pd is typically evaluated in

the laboratory environments with real threats

employing fairly sophisticated designed

experiments. These results may be obtained

directly from the test center.

Mean time between failures

(MTBF)

Mean time between failures (MTBF) is the elapsed

time between inherent failures of any given system

in operation. This MTBF time can be supplied by

the manufacturer and can be determined in the

field during the operational tests. It is typically

measured in hours.

………………………..

5

Mean Time to Repair (MTTR)

This is the mean operational downtime required to diagnose and repair a unit due to all critical and non-critical failures during operating hours.

System availability The system availability is defined as:

Availability = (MTBF / (MTBF+ MTTR))

Where MTBF is the mean time between failure and

MTTR, mean time to repair. Availability is a

measure of % of time. High performance ETD

systems must meet an availability of 97%.

…………………………

…………………………

………………………..

Remote diagnostics Factory service personnel can evaluate system

performance and identify faults without site visits

and can make on-line adjustments.

Remote Diagnostic capability

Remote system adjustment

Remote connections with a network would

need to be enabled

To be cleared with airport facility security

plan

………………………….

………………………….

Automated maintenance Automated maintenance consists of performance

of a maintenance task on a regular basis without

operator action.

Automated maintenance capability

Scheduled maintenance capability

…………………………..

…………………………..

Human Factors

6

Factors Definition Results/Score

Ease of use The interaction time between the operator and the

instrument should be minimal and very easy to perform.

For an operator, the number of daily operations for the

system should be limited to:

Startup/Warm time of 30 min or less

Calibration (automatic or manual)

Verification

………………………

………………………

………………………

Sampling wand

Ease of installing/removing the trap from the wand

Robustness of the wand

Comfort and ease of use

……………………….

……………………….

………………………

System calibrations ETD performance can be impacted by environmental

conditions and calibrations are performed to optimize the

system and ensure the highest level of detection.

How many calibrations per day

Does calibration require operator intervention

……………………….

………………………

System verifications The verification procedure involves sampling a known

substance from the sample trap to ensure the system is

properly calibrated and that the system’s hardware

(desorber, sampling flow, traps) are functioning properly.

Systems can be evaluated by measuring the number of

Daily verifications

Daily use of traps

………………………..

………………………..

Navigation through

the GUI (software

interface)

The GUI should be intuitive and easy for the user. Complex

interaction slows the operator and distracts them from

their primary task – screening. Automated features such

as internal calibration and system prompt messages greatly

reduce and simplify system operation.

Number of interactions per day from calibration

and verification

Ease of use/intuitiveness

Localization (local language)

…………………………..

………………………….

…………………………..

Total cost of ownership

7

Factors Definition Results/Score

Unit cost The costs for ETD equipment vary between different

manufacturers and can range between $20k and

$60K.

Acquisition cost per unit

Total cost of acquisition

………………………..

……………………….

Operational expense The Cost of operating and maintaining any ETD

system may exceed the cost of acquisition in a short

period of time. Estimating the operating expense is

extremely important. Airports must determine how

the system will be used, what materials and labor

are necessary to run ETD systems.

Determine cost of Consumables

Determine cost of Maintenance

Determine cost of labor

…………………………

………………………...

…………………………

Sample traps cost Manufacturers select sampling traps with excellent

efficiency for particle collection during swipe-

sampling step. These traps vary greatly from one

manufacturer to another and can have different

useful lifespans

Airports should consider the following to evaluate

the cost of traps:

Are traps certified by any governmental

agency?

Are traps multiuse purpose?

Number of samples per trap?

Cost per trap?

…………………………..

………………………….

………………………….

………………………….

False Alarm cost Every time an alarm is detected, the operator must

resolve all alarms by following their internal protocol

to resolve the alarm. This operation requires time

8

and its cost can be estimated as follows:

FA cost = Number of FA per year ×cost for each

individual trap × labor cost associated with

resolving the alarm

…………………………

Radioactivity and ionization

radiation

Radiation safety plan

Radioactive material license

National and international transport

Disposal cost

………………………..

……………………….

……………………….

……………………….

Conclusion Operational testing is absolutely essential to determine which ETD system meets the needs of the

airport environment. No airport should buy any systems without testing systems in their unique

environment. While ETD manufacturers may provide performance data, is its critical that airports collect

their own data to verify manufacturers’ claims. Utilizing results of robust airport tests and consideration

of the factors enumerated above will yield the critical data operators need to make an informed

decision. Choosing the right system will add to a positive passenger experience, will save resources that

are consumed by false alarms and other operational problems, and will in fact increase security at

airports.