an explorative approach for crowdsourcing tasks design

AN EXPLORATIVE APPROACH FOR CROWDSOURCING TASKS DESIGN

Marco Brambilla

Stefano Ceri

Andrea Mauri

Riccardo Volonterio

Introduction• OBJECTIVE: selecting the best execution strategy for the

specific human computation task

• ISSUE 1: Dealing with crowds introduces many concurring objectives and constraints

• ISSUE 2: Very large datasets, high costs of selecting the wrong strategy

• Performers• Selection• Rewarding

• Cost• Object specific or global

• Time• Quality

• Convergence criteria

SOCM'15, Monday, May 18 2An explorative approach for Crowdsourcing tasks design

Current approaches• Tool to simplify the configuration

• Do not provide support on PROs and

CONs of alternatives in settings definition

• Define a mathematical formulation of the problem • small set of decisions • NP-hard classes


Our Approach to strategy selection

• We propose a domain-independent, explorative design method

• Rapid prototyping and execution in the small in order to select the design parameters to be used for big datasets


Define a representative

set of execution strategies

Execute them on a small dataset

Collect quality measures

Decide the strategy to be used with the

complete dataset

Conceptual Model


Conceptual Model (2)• Platform: where the task will be executed • Cardinality: the number of object shown to the performer• Reward: e.g., the cost of a HIT on Amazon Mechanical

Turk, or game rewards• Agreement: e.g., majority based decision for each object

This list can be extended in order to satisfy specific user needs


Candidate Strategy• Each candidate strategies is thus represented by a set of parameters

describing the model instance considered

S = {s1, s2, . . . , sn} where n is the number of considered parameters

• Example: • an execution on Amazon Mechanical Turk • 3 objects per HIT, • “2 workers over 3” agreement • 0.01$ per answer

Sexample = [“AMT”, 3, 2/3,0.01]


Quality measures

Strategies need to be evaluated by using a set of quality measures • Cohen’s kappa coefficient: a statistical measure of inter-

annotator agreement for categorical annotation tasks• Precision of responses: percent of correct responses• Execution time: the elapsed time needed to complete the

whole task. • Cost: the total amount of money spent or impact on the

social network cause by our activity.


Evaluation of the strategies

Split the dataset in 2 (small and

large)

Run all the strategies on

the small dataset

Collect the quality

measure(s)

Select the “best”

strategy


With |small| << |large|

Experiment

Two main assumptions

1. The execution of a strategy on the small and large datasets are correlated

2. The cost of performing all experiments in the small followed by one (the best) experiment in the large is affordable


Experiment (2)• We designed an image labeling crowdsourcing task in

which we ask the crowd to classify pictures related to actor.

• Design dimensions• Number of images shown in

each microtask• Agreement level for each picture• Cost of each AMT HIT

• Dataset• 900 images related to actors retrieved from Google Images• Subselection of 90 random images as small dataset


Experiment (3)• Then we selected 8 different strategies and we ran them

on both the small and large dataset (to validate correlation hyp.)


Experiment (4)• We calculated all quality measures of the strategies

• Selection of best strategy depends on weight given to the measures• E.g., in the example we compared the strategies wrt the trade-off

between precision and cost


Results• First assumption:

• we calculated the Pearson correlation coefficient, for each design dimension


Cost Precision Agreement Duration

Pearson 0.999 0.619 0.707 0.915

Results (2)• Second assumption:

• Cost for executing all the 8 strategies on the small dataset: $22.49• Cost for executing the selected strategy: $16.86• Total: 39.95$

• The difference between the cost of experiments in the small and in the large increases a lot with big input data• Hint: in real scenarios (tens of K of objects), numerosity of small vs.

big >= 2 orders of magnitude

• If you selected a random strategy, you may have found worst quality and higher cost


Conclusion• Our method is applicable and can lead to quantifiable

advantages of cost and quality• Trade-off between the additional cost and the added value

is affordable

Future Works

• Formalizing the process for selecting candidate strategies and the “best” one (currently empirical selection)

• Iterative tuning: multi-level or separate dimensions• Testing on bigger datasets and with more design

dimensions


Thanks for your attention

Any Questions?

Stefano Ceri [email protected] Brambilla [email protected] Mauri [email protected] Volonterio [email protected]


an explorative approach for crowdsourcing tasks design

Software