human computation & esp game
DESCRIPTION
Presenter: Lin, Sin-Yan. Human Computation & ESP Game. 2008/12/19. References. Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game , paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/1.jpg)
Human Computation & ESP Game
2008/12/19
Presenter: Lin, Sin-Yan
1
![Page 2: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/2.jpg)
References
• Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.
• Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems.
• Ahn, L. V. (2007) Human computation, paper presented at the Proceedings of the 4th international conference on Knowledge capture.
• Ahn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67.
• Chen, L.-J., Wang, B.-C., Chen, C.-Y., King, I. & Lee, J. (2008) An analytical study of puzzle selection strategies for the esp game. Report for Institute of Information Science, Academia Sinica (Taipei, Taiwan, ROC).
• Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the esp game. Report for Microsoft Research, Microsoft Corporation.
2
![Page 3: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/3.jpg)
Outline
• Human Computation
• Human Computation System and Application
• Human Computation in E-Learning
• The ESP Game
• Mathematical Model in ESP Game
• System Gain
• Image selection
• Automatic playing the ESP Game
• Problems of ESP Game
• Conclusion and Comments
3
![Page 4: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/4.jpg)
Human Computation
• Concepts are started at 2001: The Open Mind Initiative
• Some task are trivial for humans, but continue to challenge computer algorithms
• Treat human brains as processors in a distributed system, each can perform a small part of a massive computation
• But humans require some incentive to become part of a collective computation
• By Games (Ahn and Dabbish, 2004) (Ahn and Dabbish, 2008)
Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systemsAhn, L. V. & Dabbish, L. (2008) Designing games with a purpose. Communications of the ACM, 51(8), 58-67.
4
![Page 5: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/5.jpg)
Human Computation System- Peekaboom
5Ahn, L. V., Liu, R. & Blum, M. (2006) Peekaboom: A game for locating objects in images, paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems.
![Page 6: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/6.jpg)
Human Computation System- PhotoShoot (1/2)
6Cheng, K.-Y. (2006). PhotoShoot: A Web-Game for User Assisted ROI Labeling. Information Management. Taipei, Taiwan, National Taiwan University. Master.
![Page 7: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/7.jpg)
Human Computation System- PhotoShoot (2/2)
7
![Page 8: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/8.jpg)
Human Computation System- ImageHunter
8Chen, L.-H. (2008). Accuracy Analysis in Productivity Games: A Case Study on Landmark Annotation. Department of Computer Science and information Engineering. Taipei, Taiwan, National Taiwan University. Master: 62.
![Page 9: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/9.jpg)
Human Computation System- PhotoSlap
9Chang, T.-H. (2007). Productivity Game Design and Gameplay Analysis: Play and Annotate with PhotoSlap. Department of Computer Science and Information Engineering. Taipei, Taiwan, National Taiwan University. Master: 52.
![Page 10: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/10.jpg)
Human Computation System- reCAPTCHA
10
• Proposed by Ahn (2008)
• Gathering the characters that optical character recognition (OCR) software has been unable to read
![Page 11: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/11.jpg)
Human Computation in E-Learning
• 人力計算具有利用人類直覺的特性• 動機
• 傳統英文閱讀學習及評量無法達成的地方• 目的
• 用 TAGs表達英文閱讀學習過程中最直覺的想法• 方法
• Human Computation and Game Design
11
![Page 12: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/12.jpg)
Human Computation in E-Learning- why tag
• Free, uncontrolled vocabulary
• Social relations of TAGs
12
photophoto
artart
desigdesignn
imagimagee
colorcolor
taggintaggingg
researcresearchh
![Page 13: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/13.jpg)
Agenda
• Human Computation
• Human Computation System and Application
• Human Computation in E-Learning
• The ESP Game
• Mathematical Model in ESP Game
• System Gain
• Image selection
• Automatic playing the ESP Game
• Problems of ESP Game
• Conclusion and Comments
13
![Page 14: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/14.jpg)
The ESP Game
• The first human computation system to take advantage of people’s desire to be entertained and provide useful metadata
Ahn, L. V. & Dabbish, L. (2004) Labeling images with a computer game, paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems
Randomly selected image
Input Keywords Input Keywords
Keyword Match
14
![Page 15: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/15.jpg)
Mathematic Model in ESP Game
• There are many parameters in ESP Game
• We can use those parameters to modeling many things that we want to know
• Ex.
• Number of players
• Number of labels per game
• Number of guesses to reach a consensus
• Good and bad word assumption
• (Human Computation) System Gain
• Image selection
• User scoring model
15
![Page 16: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/16.jpg)
Mathematic Model in ESP Game- System Gain Modeling (1/2)
• The performance of human computation system
• The purpose of ESP Game tries to collect images’ tags
• Two aspect of thinks
• Take as many tags as possible for each image
• Playing the maximum number of image
• A metric to evaluate the system gain
16
N
SlnNlnG
![Page 17: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/17.jpg)
Mathematic Model in ESP Game- System Gain Modeling (2/2)
17
C2
k)ln(Vln(T)ln(r)
k)ln(Vln(T)ln(r)k))ln(V(ln(T)(ln(r))
ln(r))k)(ln(Vln(r))(ln(T)
r)kln(Vln(T/r)G
2
2
N
SlnNlnG
2
k)ln(Vln(T)
er
![Page 18: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/18.jpg)
Mathematic Model in ESP Game- Image Selection (1/3)
• Different image selection method
• Random
• Fresh-first
• Maximize ln(N)
• Optimal (OPSA)
• Largest possible system gain
• P0: all images that have not been played
• P1: all images that have been played at least once, but less than r times
• P2: other images
18
![Page 19: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/19.jpg)
Mathematic Model in ESP Game- Image Selection (2/3)
19
T / N
![Page 20: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/20.jpg)
Mathematic Model in ESP Game- Image Selection (3/3)
20
M: Total number of images in the system
Wang, B.-C., C.-W. Lin, et al. (2008). An Analytical Study and Modeling of the ESP game. Taipei, Taiwan, Institute of Information Science, Academia Sinica.
![Page 21: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/21.jpg)
Mathematic Model in ESP Game- Automatic Playing the ESP Game (1/4)
• Given enough instances of labeled images as training data
• Similar with Language model
• Assign probability to the next label to be added
• Plays the ESP Game without looking at the image
• Limitation: at least one “off-limits” in the image
21
Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.
![Page 22: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/22.jpg)
Mathematic Model in ESP Game- Language Modeling (1/2)
• Model the sequence of word usage of a certain language
• Given a Language Model: M
• P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this)
22
![Page 23: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/23.jpg)
Mathematic Model in ESP Game- Language Modeling (2/2)
• P(I love this game) = P(I) P(love|I) P(this|I love) P(game|I love this)
• Markov assumption:
• To give reasonable predictions, only the prior local context (the last few words) affects the next word
• n-gram word model
• Bi-gram:
• P(I love this game) = P(I) P(love|I) P(this|love) P(game|this)
23
![Page 24: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/24.jpg)
Mathematic Model in ESP Game- Automatic Playing the ESP Game (2/4)
• Given enough instances of labeled images as training data
• At least one “off-limits” in the image
• t: next label
• T: off-limits set
• P(“t is next label” | “set T already present”)
24
Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.
)(
)()|()|(
TP
tPtTPTtP
)()|( tPtTP
![Page 25: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/25.jpg)
Mathematic Model in ESP Game- Automatic Playing the ESP Game (3/4)
• P(“t is next label” | “set T already present”)
• Conditional independence assumption
• The problem of zero probability
25
Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.
)()|()|( tPtTPTtP
Tt
i
i
ttPtTP )|()|(
![Page 26: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/26.jpg)
Mathematic Model in ESP Game- Automatic Playing the ESP Game (4/4)
• The problem of zero probability
26
Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation.
)()|()1()|(~
iii tPttPttP
Tt
i
i
ttPtTP )|()|(
C
tPttP
TtP Tti
i
)()|(
)|(
~
![Page 27: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/27.jpg)
Mathematic Model in ESP Game- Others
• User scoring model
• More informative tags
27
![Page 28: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/28.jpg)
Problems of ESP Game
• There is a lot of redundancy in the tag sets
• Even when tags are not exactly synonyms, they are often “to be expected” given the other tags
• There is a tendency to match on colors
• People tend to add more generic labels
Weber, I., Robertson, S. & Vojnovi´C, M. (2008) Rethinking the ESP game. Report for Microsoft Research, Microsoft Corporation. 28
![Page 29: Human Computation & ESP Game](https://reader033.vdocuments.mx/reader033/viewer/2022051315/56813426550346895d9b1279/html5/thumbnails/29.jpg)
Conclusion and Comments
• The first thing of Human Computing research is finding a clear PURPOSE
• How your game or method to achieve the purpose
29