cultural event recognition with visual convnets and temporal models
TRANSCRIPT
CULTURAL EVENT RECOGNITION WITH VISUAL CONVNETS AND TEMPORAL MODELS
Amaia Salvador Matthias Zeppelzauer Daniel Manchón-Vizuete Andrea Calafell Xavier Giró-i-Nieto
Slides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
ChaLearn Worshop
ACKNOWLEDGMENTS
Albert Gil Josep Pujal
Slides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
2
OUTLINESlides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
3
MOTIVATION: Cultural Heritage
4Chinese New year
MOTIVATION: Cultural Heritage
5Carnival Rio
MOTIVATION: Cultural Heritage
Boston Marathon 6
Classic onsite explorers
7
Onsite social media is big data...
8
...and online explorers need our help
9
CHALEARN: Looking at People
10
OUTLINESlides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
11
RELATED WORK: A multimodal problem
Source: Boston Globe
12
RELATED WORK: A multimodal problem
MediaEval Social Event
Detection
13
OUTLINESlides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
14
TEMPORAL PATTERNS IN CULTURAL EVENTS
DIADA DE SANT JORDI (Saint George’s Day)
15
TEMPORAL PATTERNS IN CULTURAL EVENTS
16
TEMPORAL PATTERNS IN CULTURAL EVENTS
17
TEMPORAL PATTERNS IN CULTURAL EVENTS
18
CASTELLERS (Human Towers)
TEMPORAL PATTERNS IN CULTURAL EVENTS
19
TEMPORAL PATTERNS IN CULTURAL EVENTS
20
days of the year from January 1 to December 31
Event class(from 1 to 50)
TEMPORAL PATTERNS IN CULTURAL EVENTS
21
Carnivals
Heiva
Viking Festival
OUTLINESlides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
22
SVM Linear classifier
IMAGE CLASSIFICATION: FC Layers
24
FC6 FC7 FC8
Resize
IMAGE CLASSIFICATION: Data augmentation
25
Center Crop
256
256
227
227
IMAGE CLASSIFICATION: Data augmentation
26
5x crop
256
256
256
5x crop
Mirror
IMAGE CLASSIFICATION
27
Small gain with temporal refinement.
IMAGE CLASSIFICATION
28
IMAGE CLASSIFICATION
29
IMAGE CLASSIFICATION
30
IMAGE CLASSIFICATION
31
IMAGE CLASSIFICATION
32
IMAGE CLASSIFICATION
33
Small gain of layer fusion on fine-tuned features.
IMAGE CLASSIFICATION
34
IMAGE CLASSIFICATION
35
IMAGE CLASSIFICATION
36
IMAGE CLASSIFICATION
37
OUTLINESlides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
38
DATA AUGMENTATION
39
DATA AUGMENTATION
40
DATA AUGMENTATION
41
DATA AUGMENTATION
42
DATA AUGMENTATION
43
OUTLINESlides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
44
CONCLUSIONS
45
CONCLUSIONS
46
...so next steps should focus on a unified solution of deep learning for all modalities.
CONCLUSIONS
47
Thank you ChaLearn ! Thank you Boston !
48
Follow us Slides online@DocXavi
http://bit.ly/chalearn-2015-upc-stp
49