combining statistics and human judgement
TRANSCRIPT
Combining Statistics and Expert Human Judgment
for Better Recommendations
Brad Klingenberg, Stitch [email protected] MLconf San Francisco 2015
Three lessons
Lessons from having humans in the loop
Humans in the loop
Lessons from having humans in the loop
Humans in the loop
It works really well, but it’s complicated
Lessons from having humans in the loop
Humans in the loop:
It works really well, but it’s complicated
Lesson 1: There’s more than one way to measure success
Lessons from having humans in the loop
Humans in the loop:
It works really well, but it’s complicated
Lesson 1: There’s more than one way to measure success
Lesson 2: You have to think carefully about what you’re predicting
Lessons from having humans in the loop
Humans in the loop:
It works really well, but it’s complicated
Lesson 1: There’s more than one way to measure success
Lesson 2: You have to think carefully about what you’re predicting
Lesson 3: Humans can say “no”, and this complicates experiments
Humans in the loop at Stitch Fix
Stitch Fix
Stitch Fix
Stitch Fix
Stitch Fix
Styling at Stitch Fix
Personal styling
Inventory
Styling at Stitch Fix: personalized recommendations
Inventory Algorithmic recommendations
Statistics & ML
Styling at Stitch Fix: expert human curation
Human curation
Algorithmic recommendations
Lesson 1: There’s more than one way to measure success
Traditional recommenders
Learning through feedback
Humans in the loop
Learning through feedback
Measuring success
In the end, you are usually interested in optimizing
and this may make sense for the combined system.
But when optimizing an algorithm, it is important to consider selection
Optimizing interaction
For a set of algorithms with the same marginal performance,
We generally prefer the algorithms that
Optimizing interaction
For a set of algorithms with the same marginal performance,
We generally prefer the algorithms that
● increase agreement and reduce needed searching (credible and useful recommendations)
Optimizing interaction
For a set of algorithms with the same marginal performance,
We generally prefer the algorithms that
● increase agreement and reduce needed searching (credible and useful recommendations)
● make the humans more efficient (effortless curation)
Optimizing interaction
For a set of algorithms with the same marginal performance,
We generally prefer the algorithms that
● increase agreement and reduce needed searching (credible and useful recommendations)
● make the humans more efficient (effortless curation)● have a better user experience (fewer bad or annoying recommendations)
Logging selection
This means logging and analyzing selection data
Lesson 2: You have to think carefully about what you’re predicting
Training a model
What should you predict?
Naive approach: ignore selection and train on success data
Advantages
● “traditional” supervised problem● simple historical data
Censoring through selection
Problem: selection can censor your data
Censoring through selection
Problem: selection can censor your data
Censoring through selection
Problem: selection can censor your data
Arms flaunted
SuccessYes
No
Yes No
?
?
p
1-p
Predicting selection
What about predicting selection?
Predicting selection
● Simple, but selection is not really success
● There is a much more direct feedback loop
Training a model
You should probably consider both.
It is most interesting when they disagree
Selection model Success model
vs
Good disagreement
Ignoring an inappropriate recommendation
Client request: “I need an outfit for a glamorous night out!”
Good disagreement
Ignoring an inappropriate recommendation
Client request: “I need an outfit for a glamorous night out!”
Bad disagreement
Stylist not choosing something that would be successful
Predicted probability of success = 85%
?
Bad disagreement
Stylist not choosing something that would be successful
Could lack trust in the recommendation: importance of transparency
Predicted probability of success = 85%
?Based on her
recent purchase
Lesson 3: Humans can say “no”, and this complicates experiments
-or-
“the downside of free will”
Testing with humans in the loop
Toy example: Suppose we want to test a (bad) new policy
Testing with humans in the loop
New rule: all fixes must contain polka dots!
Toy example: Suppose we want to test a (bad) new policy
An experiment
Control Test (Polka Dots Rule)
Selective non-compliance
Humans may not comply. Or, they may comply only selectively
Hmm, no“Please don’t send me
any polka dots” - client X
Test (Polka Dots Rule)
Selective non-compliance
Control Test (Polka Dots Rule)
Selective non-compliance
Control Test (Polka Dots Rule)
Selective non-compliance
Humans help avoid bad choices - this is great for the client!
But, this can obscure the effect you are trying to measure.
Selective non-compliance
Humans help avoid bad choices - this is great for the client!
But, this can obscure the effect you are trying to measure. Helpful analogy: non-compliance in clinical trials. This has been intensively studied
Lessons from having humans in the loop
Humans in the loop
It works really well, but it’s complicated
Lesson 1: There’s more than one way to measure success
Lesson 2: You have to think carefully about what you’re predicting
Lesson 3: Humans can say “no”, and this complicates experiments
Thanks!
Questions?(we’re hiring!)