EC

Representation and Exploitation of Contextual

Knowledge in Maritime Surveillance

Galina RogovaEncompass Consulting

Jesús GarcíaUniversity Carlos III of Madrid

EC

Outline1. Introduction2. Piracy Problem3. Maritime surveillance4. Fusion in Maritime Surveillance

– Level 1– HLF

5. Context in Information Fusion6. Reasoning and uncertainty7. Ontological representation

– Ontology categorization– Knowledge representation

• Conclusions

EC

Piracy Threat

Immanent Threat(Tri-partite Whole)

Opportunity* :• limitless range of vulnerable targets• enormous volume of commercial ships• the need for ships to pass through congested (and ambush-prone) maritime choke points

•vast territorial waters• skeleton crews• limited resources for monitoring territorial waters and ports

•lack of international laws

Intent: Goals (profit) Objectives Directives

*Rand Corporation testimony, 2009, A. Dali, Piracy attacks in the Malaaca strait.

Opportunity

Inte

nt

Cap

ability

Potential Threat(two-part whole)

Inte

nt

Cap

ability

Opportunity

EC

Piracy Threat* (cont’d)Opportunity (cont’d):•lax naval and coastal security•corruption and easily compromised judicial structures

• situation in Somalia• ready willingness of ship owners to pay ransom• limited inter-government cooperationCapability:

• supply of automatic weapons available thanks to the proliferation of small arms•well organized crime/terrorist rings possesseing:

ability to manufacture false identity papers for the crew and vessel and fake cargo documentsa broker network to sell the stolen goods general poverty

*Rand Corporation testimony, 2009, A. Dali, Piracy attacks in the Malaaca strait.

Immanent Threat(Tri-partite Whole)

Opportunity

Inte

nt

Cap

ability

Potential Threat(two-part whole)

Inte

nt

Cap

ability

Opportunity

EC

Piracy and Terrorism: Differences

•Different goals

–Piracy is an economically driven phenomenon

–Terrorism aims at undermining the oceanic environment to secure political, ideological or religious imperatives

• Piracy specific consequences: undermining and weakening government legitimacy by encouraging corruption among elected officials and bureaucrats.

• Pirates are not martyrs, which affects the attack implementation and means, e.g. no suicide attacks, no bomb building capabilities

• May require specific surveillance means

EC

Piracy and Terrorism: Commonality

• Possible common consequences:– a direct threat to the lives and welfare– direct economic impact in terms of fraud, stolen cargos and

delayed trips, which could undermine a maritime state’s trading ability

– major environmental catastrophe• Similar opportunities due to similar limitless range of vulnerable

targets:– enormous volume of commercial ships– necessity to pass through congested (and ambush-prone)

maritime choke points

*A Dali

EC

Piracy and Terrorism: Commonality (cont’d)

• Common tactics are possible, e.g. terrorists could hijack ships carrying huge loads of highly flammable natural gas to undertake a suicide mission or to ram a hijacked vessels against the cruise center, the container terminal or an oil refinery

• Piracy and terrorist attacks are carefully planed and orchestrated, both types require significant resources and well organized crime/terrorist rings.

• Willingness to risk their lives although often with different goals

EC

Piracy and Maritime security

Piracy problem is to a great degree similar to the terrorism problem and represents a part of the general problem of maritime security.

Surveillance technology developed in the maritime security domain can be adopted to fight piracy

EC

Maritime Surveillance: Why and How

“In a world where small water craft can be turned into weapons against navy destroyers and pirates can hold ships for ransom, surveillance of the sea is of increasing importance.”Potential capability monitoring requirements: Small arm trafficking (capability monitoring)Pirate organization network (intent monitoring)Immanent threat monitoring:Monitoring of high vulnerability areas (ports, navigation routes)Realized threatSecondary underlying threat

*http://www.nurc.nato.int/research/msa.htm

EC

Maritime Surveillance

“Surveillance resources are inadequate to monitor the world’s shipping channels tools that help maritime surveillance analysts identify suspicious activity are extremely valuable.” *

*http://www.nurc.nato.int/research/msa.htm

Overall goal:

Alert decision makers on vessel behaviors of interest with minimal false alarm

EC

Data & Information sources(examples)

– Vessel Traffic Systems (VTS) – Port Traffic Management System (PTMS)

• Special sensors and systems for oil and gas ports increase the safety of vessel transit and berthing.

– AIS– Civilian and military sensors– Coastal patrols– Unmanned aerial vehicles (UAVs), – Unmanned surface vehicles (USVs)– Spot reports– Visual sightings– General communication reports from coastal patrols.

• Ex: shore-based emergency communications, search and rescue service, etc– Documents, procedure– Open source information (web-based, radio,…)

– Ref.

EC

Data and Information Categories

• Observables: obtained by persistent surveillance, i.e. continuous tracking and tracing of vessels with observations systems Ex.: statements about the size of vessels and ships (large,

small), speed (slow, high), and track behavior (loitering, stopped, and continuously ahead).

• A priori knowledge including vessel characteristics (size, speed…), current practices and trade activities about information about recent activities of groups and persons and events .

• Learned knowledge• Contextual information.

12

EC

Maritime Surveillance (Challenges)

•Large and heterogeneous areas•Enormous volume of commercial ships •Diverse operations and Decision makers •Huge amount of Information of variable quality• Dynamic and unpredictable environment•Heterogeneous sources of information (sensors, human reports)•Delay in data transmission

13

EC

Fusion and Surviellance

L2,3: Situation & threat assessment:Where, what, who

Context

L1: continuous detection, ID, tracking and tracing of vessels with observation systems

Observables by persistent surveillance•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports

Knowledge base: RulesA priori beliefs…

EC

Situation and Threat assessment

• Higher Level Fusion (situation and threat assessment) produces knowledge about the state of the environment by evaluating relations between entities, their aggregations, characteristics and behavior within a specific context

• HLF provides multiple decision makers with answers to the questions such as: • What is going on• Is anything unexpected or suspicious going on?• Where?• What are the possible explanations (what does it mean)?• What is the impact (what can be expected in the future)?

EC

Challenges: Situation and Threat Assessment

How to transform data and information into knowledge and deal with:– complex distributed system modeling

– scalability problem

– time constraints and time delays

– communication within and between systems, decision makers, ,various agencies, countries,…

– secure information exchange

– heterogeneous data, e.g. structured (databases) and unstructured data (text messages)

– designing a mechanism for inferencing from a given state of knowledge to a possible explanation for a hypothesized situation

– Taking into account information quality (reliability, trust, relevance)

– Uncertain and unknown context.

EC

Context

The problem“Given an entity of interest (a physical object, a situational item, and an event) what context or a sequence of contexts can be formed , such that a task about this entity can be accomplished.”*

The Merriam-Webster dictionary: “The events or circumstances that form, or influence, the environment, within which something exists or takes place.”

L. Gong, 2005

EC

Context

18

Context*

“Context-of X”a situation of interest containing a set of entities and relations between them and providing constraints and expectations about X

“Context-for X”items externally related to X and selected to help better understand and manage a given situation.

X (“reference items”) represents any physical or conceptual entity and event to be considered, e.g. a boat moving towards a cruse ship.

* L. Gong, 2005 Steinberg & Rogova (2008)

How to choose context-of X?

How to represent context-for X?

EC

“Context Of” (CO)

•Guides response to a situation (“what can be done given a certain context” ).• Provides information for detection of possible underlying threat.• Offers dynamics of the situation (in the context of he willingness to pay ransom)• Facilitates effective communications between actors by constraining the meanings of messages.•Constrains goals, objective, functions, actions of the situation responders Example: the geophysical and geopolitical world situation such as situation in Somalia, relations between different countries in Malacca Strait.

CO is characterized by “problem variables” (entities, relationships, and activities).

CO is either given, estimated, or discovered

EC

“Context For” (CF)• CF is used for:

– Action optimization– Better situation understanding

• CF is characterized by “contextual variables” (auxiliary)

• May be static e.g. maps, or dynamic e.g. weather or geopolitical situation.

• CF is either selected or obtained by direct observations

EC

Context in Fusion and Situation managements

•Re-planning•Response•Prevention•Resource management

Situation Management

FUSION PROCESSES

•Objects•Relations•Vulnerabilities•Situations•Threats

L2

L1

L3

L4

CONTEXTS

•Contextual variables•Constrains for:

–Objects–Processes,–Preferences–Hypothesis

•Actions•Decisions•Situational

items •Predicted potential threat•Immanent threat•Realized Threat

Context:•Discovered •Estimated•Predicted

Operational knowledge•Goals, •Objectives•Functions•Plans, Actions•Operational requirements

Domain ontology

Observations•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports

Expected critical situations• Characteristics• Planned actions• Required resources

Knowledge base: •Rules•A priori beliefs…

Constrains for:•Goals•objectives•Functions•Actions•Message Meanings•Plans

Constrains for:•Ontology•Observations•Expected situational items, characteristics

Rogova, 2009

EC

Harbor Context

• Multiple areas and operations• Congestion

EC

Malacca Strait Context

http://www.?????? 23

* A. Dali

Piracy and international issues*

•dense trafficking, between neighboring countries with possible tensions between countries and conflicts in the past:

• smuggling of weapons,

•presence of international peace forces,…

•Lack of international piracy laws

•Numerous vessels with ‘normal’ missions

EC

Harbor traffic rules as context for maritime

surveillance• Harbor general rules

– Identification: ships entering/leaving the harbor must have a permission of Harbor Authority: destination, arrival/departure times, passengers/cargo, ...

– Speed limit: The speed limit usually is defined for areas, lower in inner parts and higher outer. Typical values may be 5-10 knots within the harbor areas

– Navigation: there are predefined limited areas for different categories of vessels. Crossing generally forbidden

• Procedures for special vessels

– Some ships are obliged to use pilotage service: overloaded, carrying cargo potentially dangerous or pollution (oil derivatives, chemicals, etc)

– Depending on maneuvering capability, there may be specifications of towage: entering the port, mooring positions, docking places, etc., with a number of tug boats depending on length and draught of vessel.

24

EC

Approach

Framework for contextual

ontology-based reasoning

Anomaly? ORInsufficient Quality of: • Contextual knowledge?• Observations• Fusion processes?

ObjectsRelations

L2L1

L4 L2

Fusion Processes

Domain ontology

Current Context

(CO)

Environment monitoring•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports

Is the belief in anomaly justified?

Expected:• objects• situational items• characteristics• behavior

Assessed:• objects•relations• situational items• characteristics• behavior

no

Constraints for:•Objects•Processes•relations

Consistent?

StatisticsRulesArguments

Constrains for:•Objects•Processes,•Beliefs•Preferences•Hypothesis•Arguments

CF

EC

Dealing with Inconsistency

How and whether to change the state of the knowledge (e.g.

threat/no threat) due to the arrival of new information if new

information contradicts to prior knowledge in the context

under consideration.

Involves:

• methods for detecting inconsistency

• abduction: a means to determine the sources of the contradiction (e.g., whether this contradiction is because of insufficient quality of contextual knowledge, observations, fusion processes)

How and whether to change the state of the knowledge (e.g.

threat/no threat) due to the arrival of new information if new

information contradicts to prior knowledge in the context

under consideration.

Involves:

• methods for detecting inconsistency

• abduction: a means to determine the sources of the contradiction (e.g., whether this contradiction is because of insufficient quality of contextual knowledge, observations, fusion processes)

Anomaly? ORInsufficient Quality of: • Contextual knowledge• Observations• Fusion processes?

Requires reasoning under uncertainty

EC

Inconsistency Detection• May be based on explicit “normalcy” or “anomaly” models by using:

– Values of characteristics or behavior of situational items obtained from databases.

– Rules, e.g. presence or absence of certain characteristics.– Statistical information obtained from databases (e.g. hypothesis

testing)

Problems• Scalability: There are too many patterns of normal behavior • There are less abnormal patterns than normal but the problem with the

“black swan.”

What is normal depends on context (geophysical, vessel type,…)

EC

Inconsistency Detection (cont’d)

Incremental learning operating on evolving data (“on-line stream classification problem”)

–Accumulated statistics

–Neural networks

– Case-based reasoning

Advantages:

Can identify unseen earlier patterns of behavior or characteristics

Drawbacks

A pattern that might be considered as an anomaly, could become normal with time when more information is available.

Anomaly detection triggers the process of believe update to find the source of inconsistency, which requires reasoning under uncertainty

EC

Reasoning Under Uncertainty

Qualitative methods(Uncertainty is handled by manipulation of symbols)

Quantitative methods(Uncertainty encoded by numbers)

• Probability theory–Classical probability (chance)–Bayesian (subjective belief)

• Possibility theory (incompleteness)• Fuzzy theory (vague information)• Evidence theory (ignorance, ambiguity)

–Dempster-Shafer–Transferable belief• …

• Default logic(incompleteness) • Argumentation (inconsistency)• …

Hybrid Methods

Selection of a single formalism of a hybrid system depends on a particular problem

EC

Belief Based Argumentation

Belief Based Argumentation* is an approach to non-monotonic reasoning

under uncertainty combining symbolic logic with belief theories for judging

hypotheses about the unknown world by utilizing given knowledge.

Quantitative partComputes and combines beliefs

that hypotheses and the

arguments, which bear on them,

are valid.

Quantitative partComputes and combines beliefs

that hypotheses and the

arguments, which bear on them,

are valid.

Belief Based Argumentation

Qualitative partFinds arguments in favor and

against a hypothesis about a

possible cause consistent with

observations.

Qualitative partFinds arguments in favor and

against a hypothesis about a

possible cause consistent with

observations.

* Rogova et al (2004,2005) based on R. Haenni, et al, (2000)

* Rogova et al (2004,2005) ,R. Haenni, et al, (2000)

EC

Why Belief Based Argumentation

Allows for:

• contextual reasoning with harbour rules and regulations

• dealing with uncertainty

• reasonng under the open world assumption (incomplet information and non-exhaustive hypotheses)

• incorporating subjective knowlegdeinclude both numeric and symbolic information

EC

Example

Discovery of possible threat from a boat based on:• Boat features (speed, direction, type, flag, etc.)

• Spatio-temporal relations between the suspicious boat and others, or relations between the boat and harbor zones

• Beliefs assigned to assumptions are based on the observed spatio-temporal relations and the correspondence of the boat behaviour to the rules and regulation

32

An argument pro hypothesis “treat” for a vessel under a flag of convenience built as a conjunction of the following uncertain assumptions:

A1: the suspicious boat is too close to a vessel sailing in the opposite direction

A3: The vessel following in the opposite direction is a big cruise ship

A4: The suspicious boat is increasing its speed

EC

Fusion and Surveillance

L2,3: Situation & threat assessment:Where, what, who

Context

L1: continuous detection, ID, tracking and tracing of vessels with observation systems

Observables by persistent surveillance•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reportsKnowledge base: RulesA priori beliefs…

EC

Fusion Challenges: Level l

• Tracking: accurate detection and estimation of targets dynamics • Design aspects: Analysis of appropriate fusion architectures and algorithms in

this environment• Experimental analysis: demonstrate through an operative prototype

– Focus in efficiency, point to the highest computation load to allow acceptable performance in real time.

– Use of contextual information for adjustment to adapt the algorithms to operating conditions

34

EC

Fusion Challenges: Level l

35

SENSOR INPUTS

MULTI-ALGORITHM TRACKING SYSTEM TRACKS

EC

Sensor Fusion in MD• Example: AIS-radar Fusion system

– Distributed architecture with specific logic for each source– Output: global tracks: ID, reference time, WGS84 location,

speed/course over ground

EC

Sample Scenario

• Simulated scenarios and real data analysis with Cape Verde Islands, 7 available sesors (4 RADAR, 3 AIS Stations)

EC

Sensor Fusion in MD• Specific sensor processing

– Radar with clutter, noise, detection– AIS: ID, speed processing, aperiodic

EC

L1 Algorithm design• Local Tracks Management

Cycle plots acquisition

Temporal Spatial Association

Unassociated plots

Local track / plot

associations

Deletion checks

Local tracks creation

Local tracks update

Gating

Plott type ID Match

radar

AIS

EC

L1 Algorithm Design

• Global Tracks Management

Local processing

Inconsistency check

Redundancy check

Empty global tracks check

Splitting process

Global tracks fusing process

Global track deletion

Updated local tracks acquisition

Previously associated local

tracks

New local tracksGating

AssociationGlobal track creation

Global track update

Local to Global track fusion

EC

Geometric context• Parameter tuning

– Initializtaion areas– Speed threshold– Maeuvering index– Management cycles

Plane tessellation

Nº tile: 1 2 3 4 5 ...

Dock area

Low -speed initialization area

Ground areas 2

(xp, yp)

2

EC

Context-aided tracking

• Context-aided tracking: adaptive estimation of target dynamics accordingly to configuration and uncertainty– Ex.: use of known geometrical conditions and sensor performance– Use of expected behaviours

• Examples: radar data processing in airport surface traffic monitorization

42

EC

Context-aided tracking

• Rule-based tracking: rule-based video processing system: split/merge effect– Examples: video in airport surface traffic monitorization, sports, parking

surveillance, …

43

EC

Fusion Challenges: Level l

• Adaptable tracking processes– Data association/correlation (video)– Track initialization / management logic– Appropriate tracking filters (ex. EKF/IMM tuning)– Object dynamics characterization (ex. manueverability)– Cycle management (efficiency)

• Scenario Context – Maps– Regulation (speed, size, …)– Restricted areas– Trafficability maps

44

EC

Knowledge representation: ontologies

Ontology is the study of the kinds of things that exist.Provides formally structured context-dependent information about dynamic reality capable of capturing situational entities and the wealth of relations between them.

Formal upper level ontological analysis to represent Real World in general

Formal upper level ontological analysis to represent Real World in general

Formal Ontology of a

Specific Domain

Domain-specific ontological analysis to represent domain specific characteristics of Real World

Informs Constrains

SituationOntology

Dynamic Environment

Uncertainty

EC

Types of Domain specific Ontologies

• Domain Specific ontologies: elements in the application domain, such as known types of vessels and ports– To acquire data from information sources in a meaningful and consistent

manner

• Situation ontologies. capture a situation or series of states in the application space using concepts from the content ontologies. – Situation ontologies are input to search patterns of interest, e.g. the

specific intents that we want the system to recognize. – Ex.: a search pattern for smuggle might: rendez-vous with another

vessel mid-sea or the vessel type does fit the current location and time.

[] Ref: A. van den Broek FUSION 2011

46

EC

Use of Ontologies for Knowledge Representation in Fusion

– capture domain knowledge to represent objects, objet attributes, and relations between them

– capture domain knowledge not only about about entities and status, but also domain semantics (concepts, relationships, etc).

– Explicit representation of formal semantics such as Description Logics enables automated reasoning

Ontology-based systems usually have their logical basis in any type of classical logic (extensions of rule systems)

47

EC

Representation

• A priori specialized knowledge (terminological axioms)– Geometrical elements of harbor: moving lines, buoys, restricted areas,

etc.– Vessel classification: vessel types and properties– Rules of operation: margin speeds, procedures (crossing only

orthogonal), rules of priority

• Instantiation (extensional axioms)– Properties of actual entities of the current scenario

• Static (e.g., delimitation of harbor zones)• Dynamic (e.g., vessel a position)

48

EC

Ex.: Harbor traffic rules• Harbor general rules

– Identification: ships entering/leaving the harbor must have a permission of Harbor Authority: destiny, arrival/departure times, passengers/cargo, ...

– Speed limit: The speed limit usually is defined for areas, lower in inner parts and higher outer. Typical values may be 5-10 knots within the harbor areas

– Navigation: there are predefined limited areas for different categories of vessels. Crossing generally forbidden

• Procedures for especial vessels

– Some ships are obliged to use pilotage service: overloaded, carrying cargo potentially dangerous or pollution (oil derivatives, chemicals, etc)

– There may be specifications of towage: entering the port, mooring positions, docking places, etc., with a number of tug boats depending on length and draught of vessel

• External procedures– Outer channels: Merchant ships have to proceed following a “safe speed”

avoiding waves to small boats/vessels– Zones considered dangerous for maritime traffic due to military exercises

49

EC

divided_into

extends_from

extends_to

isA

isA isA isA

isA isA isA

delimited_byadjacent_to

partially_overlaps

Representing areas (static)RCC: Region Connection

Calculus

50

EC

Represent vessels (dynamic)

has_track_snapshotis_related_to_track

has_property

isA isA isA

position

isA

isA

has_value

51

EC

Reasoning: Approach

Framework for ontology-based

reasoning Anomaly? ORInsufficient Quality of: • Contextual knowledge?• Observations• Fusion processes?

ObjectsRelations

L2L1

L4 L2

Fusion Processes

Domain ontology

Current Context

(CO)

Environment monitoring•Sensors •Open source information •Intelligence reports•Observers’ reports•Essential facility reports

Is the belief in anomaly justified?

Expected:• objects• situational items• characteristics• behavior

Assessed:• objects•relations• situational items• characteristics• behavior

no

Constraints for:•Objects•Processes•relations

Consistent?

StatisticsRulesArgumentsContext

exploitation in specific domain

Constrains for:•Objects•Processes,•Beliefs•Preferences•Hypothesis•Arguments

EC

RACER as reasoning support

• RACER inference engine– Commercial reasoner to reason with Description

Logics ontologies

• Features– Supports standard OWL DL

– RCC-based spatial reasoning (semantics for topological reasoning: inclusion, adjacency, overlapping, etc.)

– nRQL extension• Instance querying• Rule definition

– Deductive rules– Abductive rules

Ontology model

53

ReasoningSupport

EC

Rule Manipulation

• Abductive rules– Include additional individuals in the consequent, which are created as

new instances of the ontology Scene recognition• Interpret what is happening from the basic tracking data• Create upper-level instances from lower-level data: objects from tracks

(correspondence), activities from objects (recognition), impacts from activities (evaluation)

– Tracking feedback• From the interpretation of the scene (instances of the CL model), feedback

rules create instances of recommendations to the tracker

EC55

Information Flow with Ontology Reasoning

EC

Example

• Video scene representation (PETS 2002)– Static and dynamic entities to remove reflecting elements

56

(170, 180)

(165, 105)

(415, 60) (485, 55)

(490, 65)

(485, 140)

(475, 145)

(410, 130)

(445, 100)mirror

mirrored area

mirror1 is a mirrormirrored_area1 is the area of influence of a mirror1mirror1 is in (445, 100)mirror1 area is delimited by points: (415, 60), (490, 65), (475, 145), (410, 143)...position & area of mirror1, mirroredArea1, etc.

EC

Example

• Video scene representation (PETS 2002)– Static and dynamic entities to remove reflecting elements

57

(360, 105)

(25x45) (25x40)

(425, 100)

001 002

track

track

EC

Summary

• General requirement: Modern fusion systems must be adaptive and context-sensitive, customized to application knowledge

• Highlighted some open issues: A general approach to consider context exploitation and uncertainty-based reasoning, related with the specific needs in maritime analysis

• Explained an overall process based on an ontological representation, with threat recognition and abductive reasoning– Support for reasoning with rule manipulation engine

58

EC

Piracy Domain Specific Challenges

• Context challenges: – characterization of variables: completeness vs complexity, plausibility,

relevance, etc– Context discovery– Formal representation– Context dynamics

• Piracy domain specific ontology– incorporating specific rule and regulations– wide areas: interoperability among different countries– specific methods for ontology management (scale)

• Anomaly (normalcy) model: semantic problem

59

EC

Specific Technology Challenges

• Completely integrate the Context Layer and the Input Sensor Layer (Tracking)

• Performance issues• Multi-modal multi-sensor context extension• Incorporating uncertain/imprecise knowledge (both in the input and

the outputs)• Algorithms and architectures, new sensors, knowledge extraction,

user interaction, etc.• Scalability

60

EC

Representation and Exploitation of Contextual Knowledge in

Maritime Surveillance

Galina RogovaEncompass Consulting

Jesús GarcíaUniversity Carlos III of Madrid

MANY THANKS FOR YOUR ATTENTION!!