VIDEO ACTIVITY RECOGNITION VIA RGB-D

Francois BREMOND Carlos fernando Crispim junior Nazli TEMUR

1

1) General Project Description

2) Contribution 3) Conclusion 4) Proposal 5) Bibliography

CONTENT

& Project & Motivations & Challenges & Goals

2

PROJECTGeneral Project Description

3

PROJECT

➤ Domain of the project is real-time, automatic video sequences understanding.

➤ In this work, RGBD sensor is used to acquire 3D images, detect people and recognize interesting activities.

➤ The SUP library, which is developed by STARS Team is used for detection, tracking and recognition of the people.

➤ For privacy purposes we have focused ONLY on the depth information of RGBD cameras

4

INPUT / DATASET

➤ Dataset : Home Care [Nursing Home] recording on depth data, videoclips.

➤ 1 day[24 hours] of video is chosen out of months of recording of unconstraint recording.

➤ Already annotated video is used after verification.

➤ Total Annotated Events 197 with the numbers of

Enter_Restroom: 18

Exit_Restroom : 19

Leave_Area_In_Bedroom : 75

Enter_Area_In_Bedroom : 76

Sitting : 9

5

MOTIVATIONGeneral Project Description

6

MOTIVATION

➤ Population aging is a motivation to develop intelligent technologies to support the life of older people , especially via monitoring systems.

➤ Activity recognition is a key way for better assessment of monitoring.

➤ So that my motivation is to improve activity recognition for providing comfort to the elderly people’s daily life.

http://www-sop.inria.fr/members/Carlos-Fernando.Crispim_Junior/demos.html 7

CHALLENGESGeneral Project Description

8

CHALLENGES

➤ People Detection • Occlusions[Perspective challenge]

12

3

9

CHALLENGES

➤ People Detection • Noises

1

2

10

CHALLENGES

➤ People Tracking • Re-identification

• Overcome Occlusions

• Overcome Noise

1

2 3

11

CHALLENGES

➤ Event Recognition • How to model activities

• How to recognition human activities in real-time settings, past-present approach for video analysis

12

3

12

CHALLENGES

➤ Event Recognition • How to model activities

• How to recognition human activities in real-time settings, past-present approach for video analysis

34

5

13

CHALLENGES

➤ Event Recognition • How to estimate target destination

• How to maintain different actors’ speed to not be in trouble with duration

• How to infer actors interaction with ambiguous, non-observed parts of the scene

A

B C

14

GOALEvaluate Current Event Recognition Pipeline for Activity Recognition in

Unconstrained Environment

Study the event modeling language to implement new models. Study the limitations of existing solutions.

Propose more suitable event models.

TO DO

15

CONTRIBUTIONS

16

CONTRIBUTIONS

➤ Proposition of a new approach to reason about actors that leave/enter the scene in the midst of noisy people observations.

➤ - Previously: Empty scene (hard constraint)

➤ - Proposition: Increase/Decrease in the number of actors

Note: Empty Scene restricts the events to a single actor.

Thanks to newly proposed scenarios the event recognition can be done when multiple actor/noise present on the scene.

17

CONTRIBUTIONS

➤ Proposition of a new layout for the spatial zones.

18

EXAMPLE SCENARIOS OF THINTH LANGUAGECompositeEvent(EXIT_RESTROOM, PhysicalObjects((p1 : Person), (z1 : Zone), (sc : Scene)) Components((c1: CompositeEvent Disappear(p1,sc)) (c2: PrimitiveEvent IN_AREA_IN_RESTROOM(p1,z1))) Constraints((c1 justBefore c2) (duration(c2) > 0.5)) Alarm ((Level : URGENT)) )

CompositeEvent(Disappear, PhysicalObjects((p1 : Person), (sc : Scene)) Components((c1: PrimitiveState Person_exists(p1)) (c2: PrimitiveState Negative_Change_Actor_Number(sc))) Constraints( (c1 justBefore c2) (!(Exist(p1)))) Alarm ((Level : URGENT)) )

PrimitiveState(Negative_Change_Actor_Number, PhysicalObjects((sc : Scene)) Constraints ((AttributeChange(sc->NumberOfActors) = 2)) Alarm ((Level : URGENT)) )

ATTRIBUTES

➤ Composite Event

➤ Primitive Event

➤ Primitive State

19

CONCLUSION

20

CONCLUSION

In this project, description based video activity recognition and evaluation is studied.

Improvement proposals are suggested on Event Modeling and Trajectory Analysis Topics.

Thanks to the newly proposed event models, it is possible to recognize events in a scene where multiple actors and noises are present.

For the evaluation process, we benefited from several tool such as SUP Event Recognition Platform, Thinth Event Modeling Language ,Viseval ,Viper, KreateTool.

While evaluating , we have used True Positive,False Positive and False Negative measurements; F1 score, precision, recall metrics.[TP,FP,FN]

Evaluation on previously defined event models are completed.

Currently, latest assessment on newly defined zones and event models is ongoing because of Viseval tool related error.

21

PROPOSALS

22

PROPOSALS

➤ Use of Trajectory is investigated. Trajectory usage is promising, but to be able to implement a precise solution, state of art for the trajectory need to be reviewed wrt occlusion and noise reduction.

23

BIBLIOGRAPHY

24

[1] C. Crispim-Junior, K. Avgerinakis, V. Buso, G. Meditskos, A. Briassouli, J. Benois-Pineau, Y. Kompatsiaris and F. Bremond. Semantic Event Fusion of Different Visual Modality Concepts for Activity Recognition, Transactions on Pattern Analysis and Machine Intelligence - PAMI to appear, 2016.

[2] C. Crispim-Junior, V. Bathrinarayanan, B. Fosty, A. Konig, R. Romdhane, M. Thonnat and F. Bremond. Evaluation of a Monitoring System for Event Recognition of Older People. In the 10th IEEE International Conference on Advanced Video and Signal-Based Surveillance 2013, AVSS 2013, Krakow, Poland on August 27-30, 2013.

[3] C. Crispim-Junior, B. Fosty, R. Romdhane, F. Bremond and M. Thonnat. Combining Multiple Sensors for Event Recognition of Older People. In the 1st ACM International Workshop on Multimedia Indexing and Information Retrieval for Healthcare, MIIRH 2013, Copyright 2013 ACM 978-1-4503-2398-7/13/10, http://dx.doi.org/10.1145/2505323.2505329, Barcelona, October 22, 2013.

[4] Alberto Avanzi, Francois Bremond, Christophe Tornieri and Monique Thonnat, Design and Assessment of an Intelligent Activity Monitoring Platform, in EURASIP Journal on Applied Signal Processing, special issue in "Advances in Intelligent Vision Systems: Methods and Applications", 2005.

[5] E. Corvee and F. Bremond. Haar like and LBP based features for face, head and people detection in video sequences. In the International Workshop on Behaviour Analysis, Behave 2011, Sophia Antipolis, France on the 23rd of September 2011.

[6] C. Crispim-Junior and F. Bremond. Uncertainty Modeling Framework for Constraint-based Elementary Scenario Detection in Vision System. In the First International Workshop on Computer vision + ONTology Applied Cross-disciplinary Technologies in conjunction with ECCV 2014, CONTACT-2014, Zurich, Switzerland, September 7th, 2014.

[7] A. König, C. Crispim, A. Covella, F. Bremond, A. Derreumaux, G. Bensadoum, R. David, F. Verhey, P. Aalten and P.H. Robert. Ecological Assessment of Autonomy in Instrumental Activities of Daily Living in Dementia Patients by the means of an Automatic Video Monitoring System, Frontiers in Aging Neuroscience - open access publication - http://dx.doi.org/10.3389/fnagi.2015.00098, 02 June 2015

25

THANKS!

26

SHOWCASE

27