epitome – a social game for photo albuma social game for ... · epitome – a social game for...
TRANSCRIPT
![Page 1: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/1.jpg)
1
Epitome A social game for photo albumEpitome – A social game for photo album summarization
Ivan Ivanov, Peter Vajda, Jong-Seok Lee, Touradj Ebrahimi
Multimedia Signal Processing GroupEcole Polytechnique Federale de Lausanne, Switzerland
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 2: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/2.jpg)
2
Number of photos taken in our daily life is increased
Motivation
Number of photos taken in our daily life is increased Social networks – friends, family and colleagues can
access photosU h ll f i i h lif Use photos to tell story of some important events in the life
Summarization – get a quick overview of a set of photos Gaming provides a new way of motivatingg p y g
people to make the subjective dataacquisition interesting and enjoyable
From the game you can expect your friendsg y p yto do summarization
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 3: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/3.jpg)
3
Humans are best at solving some types of problems
Why game?
Humans are best at solving some types of problems Humans like to play games Build a game that is
Fun to play Solves a hard problem – photo album summarization Collects research data
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
Von Ahn & Dabbish, CHI 2004
![Page 4: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/4.jpg)
4Why on a mobile phone?
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 5: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/5.jpg)
5
Social game “Epitome”
Goal
Social game Epitome Album summarization – selecting a set of photos from a
larger collection which best preserve the visual information of the entire collectionthe entire collection
Implementation on a mobile phone Show only a limited number of images which can be fit into
one mobile screen Game – enjoyable, pleasant, funny The goal is to see whether showing a reduced set of images g g g
is enough to summarize a large number of images in one album
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 6: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/6.jpg)
6System overview
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 7: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/7.jpg)
7How players play?
Pl 1Player 1“Select the Best!”
game
Entire Facebook album
0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0
BestCount
BestFreq
0 0 0 0 1 0
1 1 1 1 1 1
1
BestCouImporta ntBest
eq
ncFre
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 8: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/8.jpg)
8How players play?
Pl 2Player 2“Select the Best!”
game
Entire Facebook album
0 0 0 0 0 0 1 0 0 0 0
0 0 1 1 1 1 1 1 0 0 0
BestCount
BestFreq
0 0 1 0 1 0
2 2 2 2 1 1
1
BestCounImportanc tBest
eFreq
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 9: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/9.jpg)
9How players play?
PlPlayer n“Select the Best!”
game
Entire Facebook album
1 0 1 2 0 1 3 2 1 0 1
4 5 5 8 8 8 9 8 7 5 5
BestCount
BestFreq
1 0 1 2 0 0
5 6 6 9 9 9
2
BestCounImportanc tBest
eFreq
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 10: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/10.jpg)
10How scores are calculated?
PlPlayer“Select the Best!”
game
Entire Facebook album
1 0 1 2 0 2 3 2 1 0 1
5 6 6 9 9 9 9 8 7 5 5
BestCount
BestFreq
2 0 2 1 0 1
10 10 9 8 6 6
4
BestCounImportanc tBest
eFreq
0 2 0 0 17 0 22 0 0 2 0 4 0 22 0 12 0 0 17Importance
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
0.2 0 0.17 0.22 0 0.2 0.4 0.22 0.12 0 0.17Importance
![Page 11: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/11.jpg)
11How players play?
Pl 1Player 1“Split it!”
game
Entire Facebook album
0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0
SegmCount
SegmFreq
0 0 0 0 0 0
1 1 1 1 1 1
1
SegmCSegme ountS
ntategm q
ionFre
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 12: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/12.jpg)
12How players play?
Pl 2Player 2“Split it!”
game
Entire Facebook album
0 0 0 0 0 1 0 0 0 0 0
0 0 1 1 1 1 1 1 0 0 0
SegmCount
SegmFreq
0 1 0 0 0 0
2 2 2 2 1 1
1
SegmCSegme ountS
ntategm q
ionFre
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 13: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/13.jpg)
13How players play?
PlPlayer n“Split it!”
game
Entire Facebook album
0 1 0 0 0 3 3 2 1 2 0
4 5 5 8 8 8 9 8 7 5 5
SegmCount
SegmFreq
0 1 0 1 0 3
5 6 6 9 9 9
1
SegmCSegme ountS
ntategm q
ionFre
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 14: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/14.jpg)
14How scores are calculated?
PlPlayer“Split it!”
game
Entire Facebook album
0 1 0 1 0 3 3 2 1 2 0
5 6 6 9 9 9 9 8 7 5 5
SegmCount
SegmFreq
3 3 2 1 2 0
10 10 9 8 6 6
4
SegmCSegme ountS
ntategm q
ionFre
0 0 17 0 0 11 0 0 4 0 3 0 22 0 12 0 33 0Segmentation
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
0 0.17 0 0.11 0 0.4 0.3 0.22 0.12 0.33 0Segmentation
![Page 15: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/15.jpg)
15Photo album summarization
Entire FacebookEntire Facebook album
SegmentationSegmentationvalue
Image index
Importance value
Image index
The most representative photos
Image index
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
representative photos
![Page 16: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/16.jpg)
16Play the game!
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 17: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/17.jpg)
17
Dataset
Evaluation
Dataset “HP Challenge 2010: High Impact Visual Communication” 6 datasets 120 photos dataset 1 120 photos dataset 1
dataset 2
dataset 3
dataset 4
dataset 5
d t t 6
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
dataset 6
![Page 18: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/18.jpg)
18
63 participants (61% males and 39% females aged 18−65
Experiments
63 participants (61% males and 39% females, aged 18−65, with different backgrounds and cultural differences)
Collection of the ground truth dataS l t 5 th t t ti h t f th h l lb (20 Select 5 the most representative photos of the whole album (20 photos), while looking at all photos of that album
Experiment 1 – “Split it!” Perform a segmentation of the reduced image collection by splitting it Perform a segmentation of the reduced image collection by splitting it
into two parts which have distinct semantic meanings
Experiment 2 – “Select the Best!” Select only one the most representative photo of the reduced Select only one, the most representative photo of the reduced
collection
Experiments 1 and 2 simulate playing with our game
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 19: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/19.jpg)
19Evaluation of user performance
OneOneFacebook album
Best1
B t
Pl
0 0 0 1 0 0 1 1 0 0 0
0 1 0 1 0 0 1 0 0 0 0
0 0 1 1 0 0 0 1 0 0 0
Best2
Best3
ayer
0 1 0 1 0 0 1 0 0 0 0
Player 1 Score = 4 Player 3 Score 2
s 0 0 0 0 1 1 0 1 0 0 0Best4
Player 1 Score1 = 4
Player 2 Score2 = 3
Player 3 Score3 = 2
Player 4 Score4 = 2
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 20: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/20.jpg)
20
The distribution of the participants’ performance
Results
The distribution of the participants performance
Max performance = 315
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 21: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/21.jpg)
21
The most representative photos selected by the proposed
Results
The most representative photos selected by the proposed method
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 22: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/22.jpg)
22
Two social games for an album summarization on mobile
Conclusion
Two social games for an album summarization on mobile phone: “Select the Best!” and “Split it!”
These games concentrates on small tasks: selection of the most representative photo of a reduced set of images andmost representative photo of a reduced set of images and partitioning of the reduced set into two distinct parts
In average, our summarization game achieves 80% of the b t f f diff t ti i tbest performance of different participants
The summarization sequence of photoscan be used to create a collage of onelb f lb t balbum, a cover for an album, or to be
included in a photo book
Multimedia Signal Processing GroupSwiss Federal Institute of Technology
![Page 23: Epitome – A social game for photo albumA social game for ... · Epitome – A social game for photo albumA social game for photo album summarization Ivan Ivanov, Peter Vajda, Jong-Seok](https://reader033.vdocuments.mx/reader033/viewer/2022061003/60b1b60b8963d44aa91dff13/html5/thumbnails/23.jpg)
23
Thank you for your attention!
Ivan Ivanov, PhD student, [email protected]@epfl.ch
Multimedia Signal Processing GroupSwiss Federal Institute of Technology