millionaire: a hint-guided approach for crowdsourcing · millionaire: a hint-guided approach for...
TRANSCRIPT
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Millionaire: A Hint-guided Approach for Crowdsourcing
Bo Han
Joint work with Quanming Yao, Yuangang Pan,Ivor W. Tsang, Xiaokui Xiao, Qiang Yang, and Masashi Sugiyama
Centre for Artificial Intelligence, University of Technology Sydney, AustraliaCenter for Advanced Intelligence Project, RIKEN, Japan
November 15, 2018
Bo (UTS & RIKEN) Millionaire November 15, 2018 1 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Outline
1 Background: What are crowdsourcing and existing approaches?
2 Motivation: What are our unique novelties and contributions?
3 Hint-guided setup: Modelling the physical interface mathematically.
4 Hint-guided mechanism: Designing a mechanism from game theory.
5 Numerical experiments: Towards the real-world deployment.
Bo (UTS & RIKEN) Millionaire November 15, 2018 2 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Crowdsourcing
Figure : Amazon Mechanical Turk.
Bo (UTS & RIKEN) Millionaire November 15, 2018 3 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Baseline approach
Which one is the Sydney Harbour Bridge ?
A B
Figure : Baseline approach.
Bo (UTS & RIKEN) Millionaire November 15, 2018 4 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Modeling baseline approach
Single-stage setting:
select
{“A” PA,i ∈ [ 1
2 , 1),
“B” PA,i ∈ (0, 12 ].
Additive mechanism: assume 0 ≤ d− ≤ d+, fa : {D+,D−} → R+1,
where fa(D+) = d+, fa(D−) = d−. The additive mechanism f is:
f ([a1, . . . , aG ]) =G∑i=1
fa(ai ),
where the state evaluations of a worker’s responses to G questions aredenoted by a1, . . . , aG ∈ {D+,D−}.
1The states “D+” and “D−” denote correct and incorrect answers.Bo (UTS & RIKEN) Millionaire November 15, 2018 5 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Additive and multiplicative mechanisms
0 Gu
min
umax
AdditiveMultiplicative
Figure : Comparisons between additive and multiplicative mechanisms.
Bo (UTS & RIKEN) Millionaire November 15, 2018 6 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Motivation
Figure : Who wants to be a millionaire with extra help.
Bo (UTS & RIKEN) Millionaire November 15, 2018 7 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Guess with hints
Which one is the Sydney Harbour Bridge ?
A B
(a) Main stage.
The Sydney Harbour Bridge is fixed with a pair of
concrete pylons at each end of the arch.
Which one is the Sydney Harbour Bridge ?
A B
(b) Hint stage.
Figure : Hybrid-stage setting in hint-guided approach.
Bo (UTS & RIKEN) Millionaire November 15, 2018 8 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Hybrid-stage setting
Main stage: “? & Hints” → “H” option:
select
“A” PA,i ∈ [ 1
2 + ε, 1),
“B” PA,i ∈ (0, 12 − ε],
“H” PA,i ∈ ( 12 − ε,
12 + ε).
Hint stage: workers’ belief given hints > threshold T :
select
{“A” PA|H,i ∈ [T , 1),
“B” PB,|H,i ∈ [T , 1).
Bo (UTS & RIKEN) Millionaire November 15, 2018 9 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
From setting to mechanism
Each response of G questions gets evaluated to one of four states:
D+: main and correct;D−: main and incorrect;H+: hint and correct;H−: hint and incorrect.
We formulate any payment mechanism as a scoring function
f : {D+,D−,H+,H−}G → [µmin, µmax].
Our goal is to design f such that its expected payment for each worker isstrictly maximized under the hybrid-stage setting.
Bo (UTS & RIKEN) Millionaire November 15, 2018 10 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Designing goals
Definition (Incentive Compatibility)
f is incentive-compatible if (1) f incentives the worker to choose answersby her belief and (2) The expected payment is strictly maximized in bothstages.
Definition (Mild No-free-lunch Axiom)
If all answers in G questions are either wrong or based on hints, then thepayment should be zero, unless all attempted answers are correct. Moreformally, f (a) = 0, ∀a ∈ {D−,H+,H−}G\{H+}G .
Bo (UTS & RIKEN) Millionaire November 15, 2018 11 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Two propositions
Proposition
Let d+ = f (D+), d− = f (D−), h+ = f (H+) and h− = f (H−). WhenN = G = 1, f satisfies Definition 1 if it meets:
d+ > d−, h+ > h−, d+ > h+ → assist our setting in detectinghigh-quality workers;d+−d−
1−2ε ≥h+−h−
2ε → directly answer if confident;
d+ − d− ≤ 2T−11/2−ε(h+ − h−)→ leverage hints if unsure.
Proposition
Given 1 ≤ G ≤ N, f satisfies both Definitions 1 and 2 if ε ∈ [εmin, 1/2) forεmin = T −
√T 2 − 1/4.
Bo (UTS & RIKEN) Millionaire November 15, 2018 12 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Hint-guided payment mechanism
Inputs:Step 1: fm: fm(+D) = 1;fm(−D) = 0;fm(+H) = 1/2−εmin
2T−1 ;fm(−H) = 0.Step 2: a1, · · · , aG ∈ {+D,−D,+H,−H} are evaluations to G gold.Step 3: Set µmin and µmax properly.
The payment is:Step 4: f ([a1, . . . , aG ]) = (µmax − µmin)
∏Gi=1 fm(ai ) + µmin.
Algorithm 1: Hint-guided Payment Mechanism.
Remark
The multiplicative form not only incentivizes workers to use hints properly,but also prevents spammers.
Bo (UTS & RIKEN) Millionaire November 15, 2018 13 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Uniqueness
Theorem
∀T ∈ (5/8, 1) and 1 ≤ G ≤ N, f in Algorithm 1 satisfy both Definitions 1and 2 if and only if ε = εmin.
Definition (Harsh No-free-lunch Axiom)
If all answers in G questions are either wrong or based on hints, then thepayment for the worker should be zero. More formally, f (a) = 0,a ∈ {D−,H+,H−}G .
Theorem
∀T ∈ (5/8, 1) and ε ∈ [0, 1/2), when 1 ≤ G ≤ N, there is no mechanismsatisfies both Definitions 1 and 3.
Bo (UTS & RIKEN) Millionaire November 15, 2018 14 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Merits of hint-guided approach
Table : Comparison of related approaches and our hint-guided approach.
Perspective Metric Baseline Skip-based Self-corrected Hint-guided
requester large label quantity X 7 X Xhigh label quality 7 X - X
worker worker quality detection 7 7 7 Xspammer prevention 7 X X X
platform low money cost 7 X - Xrealization X X 7 X
Bo (UTS & RIKEN) Millionaire November 15, 2018 15 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Label quantity and quality
Table : % of the completion of three tasks.
Data set BaselineSkip-based
Hint-guided
Sydney Bridge 100.00 74.00 99.11
Stanford Dogs 99.72 58.18 99.91
Speech Clips 58.33 30.00 75.00
0%
20%
40%
60%
80%
100%
Baseline Skip-based Hint-guided
Correct Incorrect
Sydney Bridge
Per
cen
tag
e (i
n %
)
(a) Sydney Bridge.
0%
20%
40%
60%
80%
100%
Baseline Skip-based Hint-guided
Correct Incorrect
Stanford DogsP
erce
nta
ge
(in
%)
(b) Stanford Dogs.
0%
20%
40%
60%
80%
100%
Baseline Skip-based Hint-guided
Correct Incorrect
Speech Clips
Per
cen
tag
e (i
n %
)
(c) Speech Clips.
Figure : % of correct answers and incorrect answers.
Bo (UTS & RIKEN) Millionaire November 15, 2018 16 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Prediction of aggregated labels
0.00
0.10
0.20
0.30
0.40
0.50
5 6 7 8 9 10
Baseline Skip-based Hint-guided
Err
or
of
Ag
gre
gate
d L
ab
els
Number of workers
Sydney Bridge
(a) Sydney Bridge.
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
5 6 7 8 9 10
Baseline Skip-based Hint-guided
Err
or
of
Ag
gre
gate
d L
ab
els
Number of workers
Stanford Dogs
(b) Stanford Dogs.
Figure : Error of aggregated labels.
Bo (UTS & RIKEN) Millionaire November 15, 2018 17 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Worker quality detection
Table : Error rate (in %) for aggregating two crowdsourced labels.
Number of Workers 5 10
Sydney Bridge origin 38.33 16.67rescale 30 11.67
Stanford Dogs origin 12.5 4.5rescale 12 4
Bo (UTS & RIKEN) Millionaire November 15, 2018 18 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Spammer prevention and money cost
0
10
20
30
40
50
60
Sydney Bridge Stanford Dogs Speech Clips
Single(+) Single(×) Hybrid(×)
Pay
men
t (C
ents
)
(a) Spammer prevention.
0
10
20
30
40
50
60
Sydney Bridge Stanford Dogs Speech Clips
Baseline Skip-based Hint-guided
Pay
men
t (C
ents
)
(b) Money cost.
Figure : Average payment. (a) explores interaction between settings and “$”.
Bo (UTS & RIKEN) Millionaire November 15, 2018 19 / 20
Background Motivation Hint-guided setup Hint-guided mechanism Numerical experiments
Conclusions
Hint-guided approach = hybrid-stage setting + hint-guided paymentmechanism.
Extending hybrid-stage setting from binary choice to multiple choice.
Multiple-level hints (coarse to fine) + unsure option.
Bo (UTS & RIKEN) Millionaire November 15, 2018 20 / 20