Download - Multimedia Surveillance Systems
University of Genova
Department of Biophysical and Electronic Engineering
Multimedia Surveillance Systems
Prof. C.Regazzoni
Data Fusion and Bayesian Interaction
Modeling for Cognitive Ambient Intelligence
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia Surveillance Systems (MSS)
Multimedia Surveillance Systems (MSS) are
designed for assisting the human operator(s) in the
monitoring of various kinds of environments, e.g.:
Transport environments (railway stations, level crossings,
car parks, urban roads, highways, etc.);
Commercial and financial environments (supermarkets,
banks, jewel shops etc.);
Tourist environments (naturalistic parks, tourist villages,
historical town centres, etc.).
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Characterization
A Multimedia Surveillance System is characterized by: Processing of heterogeneous data from different types of
sensors;
Transmission over heterogeneous communication channels;
Multiple signal representation and event detection abstraction levels for presentation to an improved human interface.
A MSS is a system able to process, transmit and present in different ways heterogeneous data transmitted over various communication channels.
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Examples of functionalities (1)
Detection of thefts and vandalism;
Detection of unauthorized accesses (both of people
and vehicles) to restricted areas;
Prevention of dangerous situation related to the
presence of hidden bombs;
Prevention of dangerous situations related to the
presence of unauthorized persons in areas only
allowed to vehicles (e.g. railway tracks, highway
tollgates, airport strips etc.);
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Examples of functionalities (2)
People counting and crowding estimations;
Detection of cars stopping within level crossing
areas;
Traffic monitoring and control.
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Overall video surveillance market
Estimated marked size based on manufacturer total
revenue (not end user)
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Overall surveillance market
2009 Global video surveillance market: $8.26 Billion
Asia: $3.28 Billion
Americas: $2.87 Billion
EMEA: $2.05 Billion
Projected 2014 market size and implicit Compound
annual growth rate (CAGR):
Americas: $4.48 Billion with CAGR of 8.9%
Asia: $6.6 Billion with CAGR of 15.1%
EMEA: $3.3 Billion with CAGR of 10%
vIdeo and SIgnal Processing for Telecommunications – ISIP40
IP Video adoption by segments
vIdeo and SIgnal Processing for Telecommunications – ISIP40
IP Video adoption by segments
The graph shows the percentage adoption of network
vs analog video by segment.
Residential: ~15%
Healthcare and Education: ~45%
Retail and Banking: ~12%
Transport and Government: ~35%
High correlation between network video adoption and
surveillance over larger physical area
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Grow projections by segment
The table forecasts growth rates by segment
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Grow projections by segment
Key claims to note:
Transportation market size projected to more than double to
$3.6 Billion (or 25% of the global market)
Government market size projected to double to $2.3 Billion
Banking/finance and retail projected for lowest growth rates
vIdeo and SIgnal Processing for Telecommunications – ISIP40
IP Video Surveillance growth
IP video surveillance product sales will increase by 200% total between 2010 and 2012 overtaking analog CCTV sales
Mainstream 720p/1.3MP cameras cut into the cost advantage of analog cameras
Mainstream hybrid DVRs remove the barriers of legacy deployments to migrate to IP
Emerging managed/hosted video undermines analog's advantages in small deployments
Emerging panoramic cameras further cut the cost advantage of analog cameras
Maturing IP camera standards will cut costs and reduce complexity
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Video analytics
Only "moderate growth" projected this year and next
30% annual growth rate by 2012-2013
2008 sales volume: 38,000 channels server based,
40,000 channels edge based (cameras, encoders)
Government and Transportation about 50% of total
video analytics market
IMS states, "The market is still awaiting that “killer
application,” and no one seems certain yet what that
will be"
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Scientific projects
US Projects (Defense Advanced Research Projects Agency-DARPA)
[Active] Mind’s Eye (ME - develop in machines a capability that exists only in animals: visual
intelligence)
[Active] Combat Zones That See (CTS - track everything that moves in a city by linking up a massive
network of surveillance cameras to a centralized computer system)
[Active] Heterogeneous Aerial Reconnaissance Team (HART – Aerial surveillance)
[1997-2000] VSAM Video Surveillance and Monitoring
European Projects
[2010-2013] SUPPORT Project - Autonomous Monitoring for Securing European Ports
[2010-2012] VANAHEIM Project - Autonomous Monitoring Of Underground Transportation
Environment
[2008-2011] COFRIEND Project - Cognitive Vision System able to adapt itself to unexpected
situations
[2005-2008] SERKET Project - Video Surveillance Architectures
[2004-2006] AVITRACK Project - Airport Apron Monitoring
[2002-2004] INMOVE Intelligent Mobile Video Environments
[2002-2004] REOST Railway Electro Optical System for Safe Transportation
[2000-2003] ADVISOR - Subway Monitoring
[1994-1997] PASSWORDS, AVS-PV - Video Surveillance
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Scientific projects
Italian national interest projects
[1999-2000] Sistemi intelligenti per l’elaborazione e la trasmissione di segnali multidimensionali per applicazioni di video-sorveglianza in tempo reale (Ministero dell’Università e della Ricerca Scientifica);
[2002-2006] VICOM-FIRB Virtual Immersive Communications (Fondo Italiano per la Ricerca di Base - Ministero dell’Istruzione, dell’Università e della Ricerca);
[2002-2004] Industrial grant Elsag PSA (sistema di video-sorveglianza con l’uso di sensori video omnidirezionali);
[2002-2005] Architetture distribuite ed eterogenee per sistemi di sorveglianza multi-sensoriali (Prot. N. 7280/297 MIUR);
[2008-2010] SINTESIS (Integrated Security System with Distributed Intelligence)
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Reference model for a generic MSS
Sensor
layer
Peripheral
intelligence
Network
access
HUB modem
Local processing
layer Network layer Operator layer
PC cluster
Network
interface
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
First generation (1960-1980)
Switching
devices
Local processing
layer Network layer (coax
75 ohm cable
network)
monitor cluster,
control panel and
analog transceiver
Sensor
layer
Analog
transceiver
Storage and
retrieval
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Second generation (1980-2000)
Network
access
HUB WAN
and/or LAN
modem
Local
processing
layer
(channel
coding and
TX) Network layer Operator layer
PC cluster
(attention
driving)
Network
interface
VDSL
IEEE
802.11
Cable modem
Sensor
layer
Peripheral
intelligence:
source coding
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Multimedia video-surveillance systems
Third generation (2000-?)
Peripheral
intelligence
(embedded image
processing and
understanding)
Network
access
HUB modem
Local
processing
layer Network layer
Operator layer
Network
interface
Sensor
layer
UMTS
Cable modem
ADSL
ISDN
Decision planning
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Historical and technological evolution of
video-surveillance systems
Sensor type
Information processing
Information transmission
Information presentation
Relevant processing tools
First Generation (1960-1980)
Analog None Analog/Digital CCTV
Cyclic multiplexing of analog/digital video signals
Analog TV cameras, box for digital video encoding
Second Generation (1980-2000)
Analog/ Digital
Centralised at operator level
Analog CCTV, Analog over coax networks, wireless digital, wired digital
Automatic signalling of “interesting situations” (e.g. alert signalling)
Analog/digital cameras, frame grabbers, PCs, digital wireless standards (e.g, 802.11), digital wired standards (cable modems, xDSL, etc.)
Third Generation (2000 - ?)
Digital Centralised/ distributed (configurable)
Wireless digital, wired digital (configurable)
Configurable signalling at semantic level
Digital intelligent sensors, communtication interface language (e.g. JAVA), software radio technologies for radio link reconfigurability
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Beyond 3rd Generation: cognitive
surveillance
Improved perception and reasoning over monitored events will allow efficient dynamic context representation within complex environments
Extended communications new research trends go in the direction of extended interaction capabilities: User will be able to interact with occurring events
• Directly
• Through virtual “agents”
Towards a new paradigm…. Cognitive Surveillance
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Cognitive surveillance
Sensing Analysis &
Representation
Action &
Comms Decision
Physical/Virtual
World
Beyond 3rd generation
AVS systems
Data Fusion
--------
Learning
Extended comms
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Beyond 3rd Generation
Research lines:
Extended Perception/reasoning
Fusion coordination problems (physical synchronization and
action timing)
Learning stage needed (increasing difficulty with increasing
complex interactions)
Extended communications management
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Extended perception & reasoning
The described scene understanding basic methods
cooperate to achieve
extended
perception
extended
reasoning
Mainly based on low level
scene understanding
techniques
Increasing abstraction level
information processing
vIdeo and SIgnal Processing for Telecommunications – ISIP40
What can we do now
Intelligent sensors
Video Analytics
• Limited set of robust 24/7/12 functionalities
– E.g. Intrusion detection, People counting, Traffic analysis
Networking
Architecture
• Thousands of connected heterogeneous (e.g. analog, IP, etc.)
cameras
• High speed wireless connections
Coverage type
• On line metadata and video streaming
• First examples of networked embedded video analytics
vIdeo and SIgnal Processing for Telecommunications – ISIP40
What can we do now
Control Center
Architecture
• Coexistence of multiple vendors Video Management
Systems
• Audio-Video centered multimedia event
presentation
Functionalities
• A posteriori Event/Situation interactive retrieval
• Off-line Data fusion and Decision support
vIdeo and SIgnal Processing for Telecommunications – ISIP40
What would/could we have now/tomorrow
Cognitive sensors
Video analytics
• 24/7 extended set of robust functionalities (e.g. crowd analysis,
action recognition, etc.)
Cognitive surveillance/Operator in the loop
• Integrated on line anomaly detection and decision support
Networking
Architecture improvements
• Heterogeneous networks interoperability - Ad hoc fault
resistent networks - Scalable multilevel data transmission
Extended coverage with software/cognitive nodes networks
• Mature networked embedded nodes - On line understanding
and prediction of complex multicamera events
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Control center
Architecture
• Interoperable standards for VMS
• Heterogeneous data source fusion and presentation
• Tools for operator reactions monitoring
Functionalities
• Improved performance retrieval techniques (time
and space)
• Cooperative distributed on line analysis and
prediction for safety mainteinance
• Cooperative and distributed automated learning
from experience
What would/could we have now/tomorrow
vIdeo and SIgnal Processing for Telecommunications – ISIP40
SIGNALS
FEATURES
OBJECTS
SITUATIONS
PREDICTIONS
DECISION
Info
rmation p
er
tim
e a
nd
space
unit
1995 2000 2005 2010 2015 2020
CHASING
?
?
?
?
Technological trends in Cognitive sensors
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Control Centre
Intelligent Sensing
Node(s)
Mobile Terminal
Network
Cognitive Node
Cognitive surveillance system
architecture
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Networking: architecture and extended
coverage
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Info
rmation p
er
tim
e a
nd s
pace
unit
2000 2005 2010 2015 2020
RETRIEVAL
PREDICTION
DECISION
SUPPORT
?
?
?
?
SINGLE SENSOR
Technological trends in Control centers
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Examples of existing video-surveillance
systems (Communications)
Wireless networks: CCTV(R) products from Wavelength (UK) (first generation commercial system which transmits JPEG compress images with IEEE 802.11 standard). About 11.000 USD;
Wireless networks: system for recognition of stolen car number plate with intelligent sensors processing in-situ the information acquired among highways traffic (prototypic system operating in USA) (S. S. Shyne, “Distributed surveillance network utilises neural networks for stolen vehicle detection”, Proceedings of the SPIE, Vol. 2938, 1997, pp. 186-190).
Wired Network: System for video surveillance over Closed Circuit Twisted Pair (CCTP) for the digital age; a security system for airports, offices, schools, healthcare facilities, and government & military.
Wired network: system for turistic villages monitoring, developed in the CEE project AVS-RIO, with information transmission over a coaxial cable CATV network (second generation prototypic system operating in Riomaggiore municipality).
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Examples of existing video-surveillance
systems (Video management systems)
Genetec
OnSSI
Milestone
Avigilon
Mirasys
March Networks
Axis
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Examples of existing video-surveillance
systems (Computer Vision)
Scene understanding (Multiprocessor and Multisensor): AxonX (http://www.axonx.com/): SigniFire (fire and smoke detection)
Acuity (http://acuitysystem.com/): AcuVI (waterside protection)
AgentVI (http://agentvi.com/): VI-System (Real-Time Detection & Alert Software)
VideoIQ (http://www.videoiq.com/): iCVR object tracking embedded/server based
VCATechnology (http://www.vcatechnology.com/): embedded and pc based traffic analysis
Bosch (http://www.boschsecurity.it): Intelligent Video Analysis (IVA)
IoImage (http://www.ioimage.com): embedded and server based object tracking
ObjectVideo (http://www.objectvideo.com/): embedded and server based intrusion detection, traffic analysis
TechnoAware (http://www.technoaware.com): server based or embedded object tracking, traffic management, people counting, intrusion detection
vIdeo and SIgnal Processing for Telecommunications – ISIP40
MSS main physical features
System architecture;
Environmental coverage: # and type of sensors;
System complexity: # and processing power of
processors;
Bandwidth and communication channels features.
Network characteristics: open or dedicated network
Processing capabilities: # frames/sec;
vIdeo and SIgnal Processing for Telecommunications – ISIP40
MSS main logical features
Multiple access characteristics: #of end users
simultaneously supported
System functionalities # and type of supported
functionalities;
System performances: misdetection and false alarms
probabilities;
vIdeo and SIgnal Processing for Telecommunications – ISIP40
MSS Methodologies
- Data alignment
- Attribute estimation
- Object identity estimation
Multi-sensor integration Data fusion
-Agent based systems
-Cooperative systems
-Understanding
-Intelligence distribution
Artificial intelligence
-Channel coding
-Multiple access techniques
- Reconfigurable devices and networks
Signal transmission Communications
- Compression algorithms
- Change detection
-Information detection and hiding
- Source Coding and transmission
- Object recognition
Signal processing
- Neural networks
-PCA Analysis
-Model learning
-Feature extraction and selection for event
classification
Pattern recognition
- Shape, motion or texture analysis
- Tracking
-Sensor calibration
–Object and behavior characterization
–Event classification
Computer Vision
Examples of methodologies Purpose Research field
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Different methodologies in MSS
Computer Vision
Tracking example
1st detected blob
2nd detected blob
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Different methodologies in MSS
Data Fusion
Shape extraction without
using multisensor
information
Shape extraction using
multisensorial
information
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Different methodologies in MSS
Pattern Recognition
Retrieval and classification with SHOSLIF trees
Probed samples Retrieved samples
Voronoi space tessellation
Quasi-Voronoi space
tessellation
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Different methodologies in MSS
Artificial Intelligence
An agent framework
No matchingMatching
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Different methodologies in MSS
Communication
ADSL phone line
HFC net
Wireless network
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Signal and Video Processing for MSS
Signal Processing for MSS must focus the system
attention on the most important information
(depending on the functionality) for:
Recognition
Communication
There are several constraints regarding:
Processing time / real-time requirements (for recognition);
Transmission time / available transmission band (for
communication).
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Signal Processing for MSS
Signal Processing techniques vary on the basis of the
processed data nature (audio, video, tactile, etc.)
Video sensors will be considered (passive sensors)
Sensors layerData processingfor alarm generation
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Signal Processing for MSS
Steps that follows Signal Processing techniques are mainly:
Transmission of data over heterogeneous channels to a remote site. Data can be transmitted over open networks (such as Internet) or dedicated channels. Main problems in these cases are related with protection of information against noise and unauthorized access and data integrity;
System intelligence: signal processing methods for MSS are able to automatically extract an high level symbolic representation of the processed data suitable for pattern recognition and computer vision steps in order to wake up the attention of the human operator.
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Intrusion detection
vIdeo and SIgnal Processing for Telecommunications – ISIP40
People counter
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Behavior analysis
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Abandoned objects
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Traffic management
vIdeo and SIgnal Processing for Telecommunications – ISIP40
PTZ Tracking
vIdeo and SIgnal Processing for Telecommunications – ISIP40
PTZ tracking
vIdeo and SIgnal Processing for Telecommunications – ISIP40
Other functions