research and future perspectives on intelligent video surveillance systems
DESCRIPTION
Research and Future Perspectives on Intelligent Video Surveillance Systems. Monique THONNAT Senior Scientist Head of Orion research team INRIA Sophia Antipolis FRANCE. Introduction 1/4. Which Security Problems? Safety and security of goods and human beings - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/1.jpg)
Research and Future Perspectives on Intelligent Video Surveillance Systems
Monique THONNAT
Senior Scientist Head of Orion research team
INRIA Sophia AntipolisFRANCE
![Page 2: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/2.jpg)
2
Introduction 1/4
Which Security Problems? Safety and security of goods and human beings
Safety: protection in case of accident or incident (e.g.: fall)
Security: protection against a malicious act (ex.: bomb) Evolution
1980: guarding, human surveillance 1990: start of video cameras setting up, remote
surveillance Highways, parking lots, subways, malls …
2000: explosion of video-surveillance Set of new laws Better understanding of the general public Threats of tragic events, terrorism acts
![Page 3: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/3.jpg)
3
Introduction: control room 2/4
![Page 4: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/4.jpg)
4
Introduction: issue 3/4
Definition: intelligent video surveillance automatic analysis of the video streams
Why ? Cost: 1 human operator /6-10 video streams Current paradox : the more there are video
cameras, the less these video camera are observed E.g. 600 000 video cameras in London need for the policemen to look at stored videos tapes
Effectiveness: duration of vigilance for an operator: > 1 or 2hours: % of the attention is lost
![Page 5: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/5.jpg)
5
Introduction 4/4
Huge information flow Few pertinent information
Video streams
Video cameras Surveillance control rooms
Selection of the information Increase of the detection rate
Intelligent video
surveillance software
Intelligent video
surveillance software
Alarms
Video cameras
Video streams
Surveillance control rooms
![Page 6: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/6.jpg)
6
Intelligent Video Surveillance 1/3demonstration
From object detection to complex event recognition(e.g.: violence)
![Page 7: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/7.jpg)
7
Intelligent Video Surveillance Definition: Data captured by video surveillance cameras Real time and automated analysis of video sequences Video understanding= from people detection and
tracking to behavior recognition
Recognition of complex behaviors: of individuals (e.g. fraud, graffiti, vandalism, bank
attack) of small groups (e.g. fighting) of crowds (e.g. overcrowding) interactions of people and vehicles (e.g. aircraft
refueling)
Intelligent Video Surveillance 2/3
![Page 8: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/8.jpg)
8
Intelligent Video Surveillance 3/3Typical problems
Metro station surveillance Surveillance inside trains
Building access control Airport monitoring
![Page 9: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/9.jpg)
9
Video Understanding for Intelligent Video Surveillance 1/13
Definition:Cognitive Vision is a new research field mixing: Computer vision techniquesfor object detection, description, categorization and tracking Artificial intelligence techniquesfor knowledge acquisition, reasoning (e.g. spatial and temporal
reasoning,…), learning (e.g. categories, structures, parameters…)
Software engineering techniquesfor vision software design, integration, reusability, evaluation
Reference: http:www.eucognition.org/ecvision site and roadmap
![Page 10: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/10.jpg)
10
Intelligent Videosurveillance:
How? A Cognitive vision approach for video understanding mixing:
computer vision: 4D analysis (3D + temporal analysis)
artificial intelligence: a priori knowledge (scenario,
environment)
software engineering: reusable software platform (VSIP)
Video Understanding for Intelligent Video Surveillance
2/13
![Page 11: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/11.jpg)
11
Video Understanding for Intelligent Video Surveillance 3/13
4 D analysis:multi-cameras
tracking
Video understanding
People detection
and tracking
Scenario recognition
A PRIORI KNOWLEDGE:• 3d models of the environment • Camera calibration• Scenario Models
Alarms
People detection
and tracking
Interpretation of the videos from pixels to alarms
![Page 12: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/12.jpg)
12
Video Understanding for Intelligent Video Surveillance
4/13
SegmentationSegmentation ClassificationClassification TrackingTracking Scenario RecognitionScenario Recognition
Alarms
access to forbidden
area
3D scene modelScenario models A priori Knowledge
Objective: Interpretation of videos from pixels to alarms
![Page 13: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/13.jpg)
13
Behavior recognition: approach based on a priori knowledge
model of the empty scene (3D geometry and semantics)
models of predefined scenarios
a language for representing scenarios based on
combination of states and events more than 20 states and 20 events can be used
a reasoning mechanism for real time detection of states,
events and scenarios (e.g. temporal reasoning,
constraints solving techniques)
Video Understanding for Intelligent Video Surveillance
5/13
![Page 14: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/14.jpg)
14
3D Scene Model: Barcelona Metro Station Sagrada Famiglia mezzanine (European project ADVISOR)
Video Understanding for Intelligent Video Surveillance
6/13
![Page 15: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/15.jpg)
15
States, Events and Scenarios : State: a spatio-temporal property involving one or several actors on a time interval
Ex : « close», « walking», « seated»
Event: a significant change of states
Ex : « enters», « stands up», « leaves »
Scenario: a long term symbolic application dependent activity
Ex : « fighting», « vandalism»
Video Understanding for Intelligent Video Surveillance
7/13
![Page 16: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/16.jpg)
16
Vandalism scenario description :
Scenario(vandalism_against_ticket_machine, Physical_objects((p : Person), (eq : Equipment, Name = “Ticket_Machine”) )
Components ((event s1: p moves_close_to eq) (state s2: p stays_at eq)
(event s3: p moves_away_from eq) (event s4: p moves_close_to eq)
(state s5: p stays_at eq) ) Constraints ((s1 != s4) (s2 != s5)
(s1 before s2) (s2 before s3) (s3 before s4) (s4 before s5) ) ) )
Video Understanding for Intelligent Video Surveillance
8/13
Scenario Recognition : Temporal constraints
![Page 17: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/17.jpg)
17
Video Understanding for Intelligent Video Surveillance 9/13Vandalism in metro (Nuremberg, Germany)
![Page 18: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/18.jpg)
18
Video Understanding for Intelligent Video Surveillance 10/13
3d Scene Model of 2 bank agencies
objet du contexte
mur et portezone d’accès
salle du coffrerue
rue
salle automates
zone d’entrée de l’agence
zone des distributeurs
zone de jour/nuit
zone devant le guichet
zone derrière le guichet
zone d’accès au bureau du
directeur
zone de jour
ported’entrée
porte salleautomates
armoire
guichet
commode
![Page 19: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/19.jpg)
19
Video Understanding for Intelligent Video Surveillance
11/13Bank Monitoring: Bank attack scenario description :
scenario Bank_attack_one_robber_one_employeephysical_objects: ((employee : Person), (robber : Person), z1: Back_Counter, z2: Entrance_Zone, z3: Front_Counter, z4: Safe, d: Safe_door)
components: (State c1 : Inside_zone(employee, z1)) (Event c2 : Changes_zone(robber, z2,z3))
(State c3 : Inside_zone(employee, z4)) (State c4 : Inside_zone(robber, z4))) constraints : ((c2 during c1) (c2 before c3) (c1 before c3) (c2 before c4) (c4 during c3) (d is open))
![Page 20: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/20.jpg)
20
Video Understanding for Intelligent Video Surveillance
12/13
bank monitoring Recognition of a bank attack scenario: an employee is e behind a counter, an aggressor enters, goes behind the counter, then he goes with the emplyee towards theSTR (secured technical room), they enter in the STR then they leave ethe STR and they go toward the exit of the agency.
![Page 21: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/21.jpg)
Examples : Brussels and Barcelona Metro Surveillance
Exit zone
Jumping over barrier
Blocking
Overcrowding
Fighting
Group
behavior
Crowd
behavior
Individual
behavior
Groupbehavior
Video Understanding for Intelligent Video Surveillance
13/13
21
![Page 22: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/22.jpg)
22
Impact: Visual surveillance of metro stations, bank agencies, trains,
buildings and airports
5 European projects (PASSWORDS, AVS-PV, AVS-RTPW,
ADVISOR, AVITRACK)
4 contracts with End-users companies (metro, bank, trains)
2 transfer activities with Bull (Paris) and Vigitec (Brussels)
Cooperation over more than 11 years with partners
Creation in 2005 of a start-up Keeneo www.keeneo.com
Conclusion 1/5
![Page 23: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/23.jpg)
23
Conclusion 2/5
Hypotheses: fixed cameras 3D model of the empty scene predefined behavior models
Results:
+ Behavior understanding for Individuals, Groups of people, Crowd or Vehicles
+ an operational language for video understanding (more than 20 states and events)
+ a real-time platform (10 to 25 frames/s)
![Page 24: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/24.jpg)
24
Conclusion 3/5 Current issues
Systems have poor performances over time, can be hardly modified and do not use enough a priori knowledge
shadowsstrong perspectivetiny objects
close view
clutterlightingconditions
![Page 25: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/25.jpg)
25
Knowledge Acquisition Design of learning techniques to complement a
priori knowledge: Frequent events, scenario model learning European project CARETAKER
Object description Fine human shape description:3D posture models Crowd description: European project SERKET
Reusability is still an issue for vision programs Video analysis robustness Dynamic configuration of programs and parameters
Conclusion: Where we go 4/5
![Page 26: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/26.jpg)
26
Conclusion 5/5 Posture Recognition
Current image and binary image
Instantaneous posture
Postures recognised along the time
![Page 27: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/27.jpg)
27
Conclusion 5/5 Crowd Behavior
Motion direction detection
Abnormal direction of people in a crowd
![Page 28: Research and Future Perspectives on Intelligent Video Surveillance Systems](https://reader035.vdocuments.mx/reader035/viewer/2022070414/56814d99550346895dbaf0d0/html5/thumbnails/28.jpg)
28
Video Understanding demo?
Airport Apron Monitoring “Unloading Operation” European AVITRACK project