programming with the wisdom of the crowd with the wisdom of the crowd 1 daniel w. barowy daniel g....
TRANSCRIPT
Programming with the Wisdom of the Crowd
1
DANIEL W. BAROWY
DANIEL G. GOLDSTEIN SIDDHARTH SURIEMERY D. BERGER
2
3
5
7
iCalorieCounter
Photograph Your Food
OK
8
iCalorieCounter
Photograph Your Food
OK
1. take a photo
8
iCalorieCounter
Photograph Your Food
OK
1. take a photo
230.3 kcal
3. return estimate
8
iCalorieCounter
Photograph Your Food
OK
1. take a photo
230.3 kcal
3. return estimate2. algorithms (???)
10
iCalorieCounter
Photograph Your Food
OK
1. take a photo 2. algorithms (???)
230.3 kcal
3. return estimate
10
iCalorieCounter
Photograph Your Food
OK
1. take a photo 2. machine learning
230.3 kcal
3. return estimate
12
iCalorieCounter
Photograph Your Food
OK
1. take a photo 2. machine learning
230.3 kcal
3. return estimate
12
iCalorieCounter
Photograph Your Food
OK
1. take a photo 2. humans
230.3 kcal
3. return estimate
13
VoxPLA programming language
utilize workers
functions
for estimates that lets you
as if they are just ordinary
in
Automatically handles scheduling, payment, and quality.
and .
(like cloud functions!)
14
15
16
16
becomes
2. crowdsourcing
17
iCalorieCounter
Photograph Your Food
OK
1. take a photo
230.3 kcal
3. return estimate
2. crowdsourcing
17
iCalorieCounter
Photograph Your Food
OK
1. take a photo
230.3 kcal
3. return estimate
VoxPL
18
Challenge: work quality
Why VoxPL?
18
Challenge: work quality
Why VoxPL?
18
Challenge: work quality
Why VoxPL?
18
Challenge: work quality
Why VoxPL?
19
i.e., do workers agree?
Challenge: work quality
Why VoxPL?
Is this a giraffe?
yesno
Is this a giraffe?
yesno
yes no yes
Is this a giraffe?
yesno
yes no yes
Take the majority opinion
Is this a giraffe?
YesNo
21
Which ones don’t belong? What does this plate say?
XXXXXX
≈ answer choices drawn from a finite, low-cardinality set.
Labeling tasks
Is this a giraffe?
YesNo
21
Which ones don’t belong? What does this plate say?
XXXXXX
≈ answer choices drawn from a finite, low-cardinality set.
Labeling tasks
“majority opinion” is inadequate (AutoMan: A Platform for Integrating Digital and Human Computation,
Barowy, Curtsinger, Berger, McGregor; OOPSLA ’12)
Is this a giraffe?
yesno
yes no yes
Is this a giraffe?
yesno
Is this a giraffe?
yesno
P(majority) = 1 !!!
How many calories?
331 10 352
How many calories?
331 10 352
“majority opinion” is meaningless
24
≈ answer choices drawn from an infinite, (or practically infinite) high-cardinality set.
How many calories? Where is this person’s nose? How hot is it in this photo?
Estimation tasks
How many calories?
331 10 352
mean = 231
How many calories?
331 10 352
Challenge: outliers One large outlier can skew the mean!
mean = 231
How many calories?
331 10 352
Challenge: outliers One large outlier can skew the mean!median: must corrupt more than 1/2 of sample to skew estimate.
mean = 231median = 331
VoxPL default estimator: L1 median
(x: 253, y: 134)
L1 median: point that minimizes distance to all other points
28
Challenge: what is a “good” estimate?
29
A few extra sprinkles don’t matter for calorie counting!
“Good” is domain-specific.
What would it taketo trust that the median value of 244 is a good estimate?
30
What would it taketo trust that the median value of 244 is a good estimate?
30
95 out of 100 times we ask the crowd, the estimate is between 194 and 294 kcal.
What would it taketo trust that the median value of 244 is a good estimate?
30
300200 400244
95 out of 100 times we ask the crowd, the estimate is between 194 and 294 kcal.
What would it taketo trust that the median value of 244 is a good estimate?
30
300200 400244
I.e., a confidence interval.
What would it take
The set of estimates that are indistinguishable due to sampling error.
to trust that the median value of 244 is a good estimate?
30
300200 400244
What would it taketo trust that the median value of 244 is a good estimate?
30
300200 400244
The set of opinions that are not likely to belong to the Homers or the Benders.
What would it taketo trust that the median value of 244 is a good estimate?
30
300200 400244
“Donut contains 244±50 kcal.”
What would it taketo trust that the median value of 244 is a good estimate?
31
300200 400244
“Donut contains 244± kcal.”25
How to get tighter estimates?
32
300200 400244
“Donut contains 244± kcal.”25
How to get tighter estimates?
32
300200 400244
“Donut contains 244± kcal.”25
Ask more people.
How to get tighter estimates?
32
300200 400244
“Donut contains 244± kcal.”25
Error decreases as sample size increases.
How to get tighter estimates?
32
300200 400244
“Donut contains 244± kcal.”25
Error decreases as sample size increases.
±25 calories is a great estimate!
iCalorieCounter
Photograph Your Food
OK
VoxPL
±25 kcal L1 median$5.00
iCalorieCounter
Photograph Your Food
OK
VoxPL 230.3± 111
±25 kcal L1 median$5.00
iCalorieCounter
Photograph Your Food
OK
VoxPL 230.3± 111>±25
±25 kcal L1 median$5.00
iCalorieCounter
Photograph Your Food
OK
VoxPL 230.3±49>±25
±25 kcal L1 median$5.00
iCalorieCounter
Photograph Your Food
OK
VoxPL 230.3±22good!
±25 kcal L1 median$5.00
34
VoxPLdef numCalories(url: String) = estimate ( budget = 5.00, confidenceInterval = SymmetricCI(25), text = "How many calories are in the food pictured?", imageUrl = url )
35
numCalories(breakfast)
VoxPL also lets you compose estimates
35
numCalories(breakfast)
numCalories(lunch)
+
VoxPL also lets you compose estimates
35
numCalories(breakfast)
numCalories(lunch)
+
numCalories(dinner)
+
VoxPL also lets you compose estimates
35
numCalories(breakfast)
numCalories(lunch)
+
numCalories(dinner)
+
VoxPL also lets you compose estimates
(how? bootstrap)
Calorie Counter
36
• VoxPL: desired ±50 kcal actual mean error: ±51.5 kcal
• 208 images of school lunches w/ground truth kcal
How good are VoxPL's estimates?
• Competitive with professional nutritionists!
37
±50 cost: $1.28
Calorie CounterHow cheap are the estimates?
VoxPL:
±50 cost: $100-200/hrNutritionist:
38
http://automan-lang.org
Harnessing the crowd to do estimates (+ labeling) with automatic budgeting, scheduling & quality control
Programming with the Wisdom of the Crowd