crowd-based evaluation methods

Social Computing @ Know-Center

1

. Christoph Trattner 15.12.2015 – Graz, Austria

Crowd-Based Evaluations

Christoph TrattnerKnow-Center

[email protected]

@Graz University of Technology, Austria


2


Before I will start in this class, I will talk a bit about

myself (background), what I am doing, etc.


3


Where do I come from (Austria)?


4


Graz


5


Trondheim


6


Academic Back-Ground?

Studies Computer Science at Graz University of

Technology & University of Pittsburgh

Worked since 2009 as scientific researcher at the KMI &

IICM (BSc 2008, MSc 2009)

My PhD thesis was on the Search & Navigation in Social

Tagging Systems (defended 2012)

Since Feb. 2013 @ Know-Center Leading the SC Area @ KC

@TUG

WebScience

Semantic Technologies

@NTNU

Datamining


7


What is my group doing?

… we research on novel methods and tools that exploit

social data to generate a greater value for the

individual, communities, companies and the society as

whole.

Our competences:• Network & Web Science

• Science 2.0

• Crowdsourcing

• Predictive Modeling

• Social Network Analysis

• Information Quality Assessment

• User Modeling

• Machine Learning and Data Mining

• Collaborative Systems

Our Services:• Social Analytics: Hub-, Expert -, Community -

, Influencer -, Information Flow-, Trend

(Event) Detection, etc.

• Information Quality Assessment

• Social & Location-based Recommander

Systems

• Customer Segmentation

• Social Systems Design


8


Ok, lets start….


9



Definition

Crowd1. A large number of people gathering together.

2. A group of people.

3. A large number of things positioned or considered together.

Source: http://www.thefreedictionary.com/crowd


10



Definition

Crowdsourcing

is the process of obtaining needed services, ideas, or

content by soliciting contributions from a large group

of people, and especially from an online community,

rather than from traditional employees or suppliers.

Source: http://en.wikipedia.org/wiki/Crowdsourcing


11


Crowd-Based Evaluation

And what is crowd-based evaluation?

• Evaluation with a bunch (a lot) of people

• In the literature also often referred as:• „crowdsourcing“

• Evaluation with crowdsourcing platforms


12


Crowd-Based Studies vs. Lab Studies

Crowdsourcing

• Study-type: Controlled

• Payment: 0.1 USD

• Design: Between

subject

• Duration: Short

(seconds)

• Users: > 100

• Duration: minutes

Lab Studies

• Study-type: Controlled

• Payment: > 30USD

• Design: Within subject

• Duration: Long (hours)

• Users: < 100

• Duration: days/weeks


13


Is crowdsourcing something new?

1714 – The Longitude Prize: When the British government was

trying to find a way to measure a ship’s longitude, they offered

the public a monetary prize to whomever came up with the best

solution. (2014 http://longitudeprize.org/)

1884 – Oxford English Dictionary (OED): 800 volunteers

catalogued words to create the first fascicle of the OED

2001 – Launch of Wikipedia

2004 – Toyota’s first “Dream car art” contest: Children were asked

globally to draw their ‘dream car of the future.’

2006 – Jeff Howe coined the term crowdsourcing in Wired

(magazine).


14


Crowdsourcing Plattforms


15



16


Crowdsourcing Platforms for Evaluation

Amazon Mechanical Turk

URL: https://www.mturk.com/mturk/welcome

• Founded March 2007- 100,000 workers in over 100

countries.

• 2011 - over 500,000 workers from over 190 countries

in January 2011.

• Tasks = Hits

• Workers = Turkers

https://www.mturk.com/mturk/welcome


17


Mturk - workflow


18


What do I have to do

...as a turker?


19


Mtruk.com


20



21



22



23


What do I have to do

...as a hit submitter?


24



25



26



27


Source: http://dev.globis.ethz.ch/crowdstudy/res/eics2013.pdf


28



29



30



31



32


Who are the Turkers?


33


Mturk Study

Ross, J., Irani, I., Silberman, M. Six, Zaldivar, A., and Tomlinson, B.

(2010). "Who are the Crowdworkers?: Shifting Demographics in Amazon

Mechanical Turk". In: CHI EA 2010. (2863-2872)

Survey of 573 people


34


Mturk Study


35


Mturk Study


36


Mturk Stats

Source: http://mturk-tracker.com/general/general-data/


37


Mturk Alternatives

Biggest Limitation:

„Officially“ only available (as Hit submitter) in the US!

Alternatives:

• CrowdFlower• Founded in 2007 in SF

• Over 153 other countries

• Jobs = Hits

• except from that more or less identical to MTurk


38


CrowdFlower.com


39


CrowdFlower - Stats

Source: http://venturebeat.com/2013/11/14/how-enterprises-use-crowdsourcing-infographic/


40


CrowdFlower - Stats


41


CrowdFlower - Stats


42


CrowdFlower - Stats


43


CrowdFlower - Stats


44


Mturk: Alternatives

Clickworker:

• Qualification: Little Knowledge – 2 pre-assessment

needed (language check)

• Registration Fee: free

• Min. Payment: 1USD

Example Tasks:

• Tagging

• Uploading videos

(on other sites)

Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/

http://clickworker.com

http://clickworker.com


45


Mturk: Alternatives

Microworkers:

Qualification: Little Knowledge

Registration Fee: free

Min. Payment: 0.10 USD

Example Tasks:

• Twitter follower tasks

• Tagging

• Uploading videos

(on other sites)Source: http://easyworkfromhomejobsonline.com/top-5-amazon-mechanical-turk-alternatives-sites-like-mturk/

http://microworkers.com

http://microworkers.com


46


...is crowdsourcing a reasonable alternative for „real“

user studies?


47


Crowdsourcing User Studies

• In 2008 Kittur et al. conducted 2 experiments to test

the utility of Mechanical Turk as a user study platform

• In both experiments workers had to rate quality of

Wikipedia articles

• Comparative study – expert raters vs. turkers

Kittur, A., Chi, E. H., & Suh, B. (2008, April). Crowdsourcing user studies with Mechanical Turk.

In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 453-456).

ACM.


48


Experiment 1

Study Design

• 14 radomly chosen articles

• 7-point likert scale

• writing, factually accuracy, neutrality, structure...

• Questions - Wikipedia “Featured article criteria“

Results

• 58 users provided 210 ratings

• 93 ratings within 24hrs (very fast)

• Correlation was only marginally significant

r = 0.50, p = .07


49


„The results from Experiment 1 provided only weak

support for the utility of Mechanical Turk as a user

measurement tool.“


50


Experiment 2

Study Design

• More or less indendical to Experiment 1

• Contrary to Exp. 1, users had to input how many

references, images, and sections the article had.

• After that they were asked the overall article quality

question

Results

• 124 users provided 277 ratings

• Correlation was higher AND statistically significant

r = 0.66, p = 0.01


51


Stats Mturk Experiment


52


Other Applications

CrowdSearch

Yan, T., Kumar, V., & Ganesan, D. (2010, June). Crowdsearch: exploiting crowds for accurate

real-time image search on mobile phones. In Proceedings of the 8th international conference on

Mobile systems, applications, and services (pp. 77-90). ACM.


53


Eval search index


54


Eval Search index


55


Eval Search Index

Lin, Y., Trattner, C., Brusilovsky, P. and He, D.: The Impact of Image Descriptions on User Tagging

Behavior: A Study of the Nature and Functionality of Crowdsourced Tags, Journal of the Association for

Information Science and Technology, 2015. (in press)


56


...ok that‘s basically it


57


Thank you!

Christoph Trattner

Email: [email protected]

Web: christophtrattner.info

Twitter: @ctrattner

Sponsors:


58


Any questions?

crowd-based evaluation methods

Science

austria graz social

austria social computing

christoph trattner15

austria crowdbased studies

austria academic

social data

daysweekssocial computing

social analytics