data-driven off a cliff: anti-patterns in evidence-based decision making
TRANSCRIPT
Data-Driven Off a CliffAnti-patterns in evidence-based decision making
Ketan Gangatirkar & Tom Wilbur
Data-Driven Off a CliffAnti-patterns in evidence-based decision making
Ketan Gangatirkar & Tom Wilbur
I helppeopleget jobs.
Indeed is the #1 job site worldwide
Headquartered in Austin, Texas
We have tons of ideas
We have tons of bad ideas
Occasionally, we have good ideas
It’s hard to tell the difference
What helps people get jobs?
The only reliable way is to see what works
We set up experiments
We collect results
We use the data to decide what to do
We’ve used data to make good decisions
But having data is not a silver bullet
We’ve also used data to make bad decisions
Science is hard
ProblemRunning an experiment can ruin the experiment
Change Effect on productivity
Brighter light UP
Dimmer light UP
Warmer UP
Cooler UP
Shorter breaks UP
Longer breaks UP
Change Effect on productivity
Brighter light UP (temporarily)
Dimmer light UP (temporarily)
Warmer UP (temporarily)
Cooler UP (temporarily)
Shorter breaks UP (temporarily)
Longer breaks UP (temporarily)
Change Effect on productivity
Brighter light UP (temporarily)
Dimmer light UP (temporarily)
Warmer UP (temporarily)
Cooler UP (temporarily)
Shorter breaks UP (temporarily)
Longer breaks UP (temporarily)
ProblemStatistics are hard
Simpson’s Paradox
Using data is more than just statistics
+ + + +
=
Good math. Bad idea.
Bad practices can undermine good math
You don’t need me to teach youto be bad at math
I’ll teach you to be bad at everything else
Anti-Lesson 01
Be impatient
p-value is the standard measure ofstatistical significance
p-value is by measurement, not experiment
If you check results on Monday,that’s one measurement
If you check results on Tuesday,that’s another measurement
Got the result you want?
Declare victory!
Move quickly! Becauseresults and p-values can shift fast
of “winning” A/B tests stopped earlyare false-positives
80%
http://bit.ly/1LtaLkV
Anti-Lesson 02
Sampling is easy
Beware the IEdes of MarchStory
Building Used Cars Search
Shoppers specifying price, mileageor year do better
Nudge shoppers to specify price,mileage or year
+3% conversion
After rollout, conversion > +3%
Why?
We’d taken a shortcut in ourtest assignment code
X
Users on oldest browsers got ignored
Distorted sample Distorted results
Anti-Lesson 03
Look only at one metric
If a little bit is good, a lot is great
Indeed has a heartStory
❤ > ★ ?
+16% Saves on search results page
Everyone ❤s ❤s!
❤s everywhere!
Hearted
Not so fast
Did ❤ help people get jobs?
❤ jobs: +16%Clicks: no change
Applies: no changeHires: no change
I helppeople❤ jobs.
Upsell teamStory
We formed an “upsell team”and measured their results
+ =
Success measure
It’s working!Upsells
So why isn’t revenue moving?Overall Revenue
+ 0 -
= ⅓+⅓
-⅓
What you measure is what you motivate
Redefine success to include all outcomes
Upsell Team revenue +200%
Anti-Lesson 03: Reloaded
Look at all the metrics
It's better for them. Is it better for us?
Job applications: UpJob clicks: Down
Recommended Jobs traffic: UpJob views: Sideways
New resumes: UpReturn visits: Down
Logins: UpRevenue: Down
(and it goes on…)
We didn’t really know what we wanted
Too much noise from too many metrics
I helppeopleget jobs.
Anti-Lesson 04
Be sloppy with your analysis
We engineer features rigorously
SpecificationSource control Code review
Automated testsManual QA
MetricsMonitors
...
But analysis…
Bad analysis won’t take down Indeed.com
200 million job seekers don’t careabout our sales projections
So we don’t try as hard with analysis code
SpecificationSource control Code review
Automated testsManual QA
MetricsMonitors
...
DublinersStory
Indeed reports on economic trends
South Carolinians wanted to move to Dublin
Dublin?
No, the other one
Incorrect IP location mapping
IP blocks for South Carolinagot reallocated to London, England
Worse things can happen
Growth and DebtStory
“Growth in a Time of Debt”Carmen Reinhart and Kenneth Rogoff
2010
Public debt > 90% GDPleads to slower economic growth
Governments made policy based on this
Genetic MutationStory
SEPT2 to a geneticist is Septin 2
SEPT2 to Excel is 42615
Does your company use spreadsheets?
How do you know they’re correct?
Under-spending Advertisers
Story
Employer budgets ran outbefore the end of the day
So no evening job seekers saw the jobs
How big was this missed opportunity?
Clicks received1260Out of budget time20:00
% of day w/o budget0.1667
Potential clicks1260 / (1 - 0.1667) = 1512Missed clicks1512 * 0.1667 = 260
Missed Clicks Report
Dear Customer,
You got 1,260 clicks yesterday.Your daily budget ran out at 8:00pm.
If you funded your budget through the whole day, you’d get another260 clicks - a +20% improvement!Get More Clicks
Assumption
100
75
50
25
00:00 2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00
Missed = 260 clicks (+20%)
0:00
Reality
100
75
50
25
0
Missed = 100 clicks (+8%)
2:00 4:00 6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 0:000:00
Naive analysis bad recommendation
Anti-Lesson 05
Only look for expected outcomes
Zero results pages from misspelled locations
Goals: fewer ZRPs, more job clicks
Zero-results pages
-2.7%
Job clicks
+8%
+1,410%
Ad revenue
+1,410%
Ad revenue
ads
Ad revenue after fix
Treatment onhomepage
Effect onsearch page
Anti-Lesson 06
Metrics, not stories
I helppeopleget jobs.
How do I know if people got jobs?
I need employers to tell me
One employer hired 4500 people in 45 minutes!
Nope
Accurate recording of outcomes helps us
It doesn’t help employers
They don't care about usingthe product “right”
Go away!
There is no “user story”
Right metrics + wrong story = wrong conclusion
Anti-Lesson 06: Parte Deux
Story over metrics
Stories are seductive
Even incorrect stories are seductive
Taste BudsStory
Taste map
Totally wrong
Every bite you eat proves it’s wrong
People still believe it
Job AlertsStory
Success for emails is well understood
New subscriptions: GoodEmail opens: Good
Clicking on stuff: GoodUnsubscribing: Bad
I helppeopleget emails.
I helppeopleget jobs.
What does job seeker success look like?
01Search for jobs
02Sign up for alerts
03Click on some jobs
04Apply to some jobs
05Get a job!
06Unsubscribe from emails
People with new jobs don't need job alerts
The standard story for email fails here
Light and Dark ReduxStory
It’s a persuasive story
But the original study was flawed
Hawthorne Revisited
… the variance in productivity could be fully accounted for by the fact that the lighting changes were made on Sundays and therefore followed by Mondays when workers’ productivity was refreshed by a day off.”
https://en.wikipedia.org/wiki/Hawthorne_effect
We con people with stories
We con ourselves with stories
Anti-Lesson 07
Believe in yourself
Believing in yourself can be good
“My startup will succeed.”
Often it’s bad
“I’d never fall for a scam like that.”
“I knew it all along.”
“I’m too smart to make that mistake.”
Every story of mistakes is deceptive
We tell stories with 20/20 hindsight
When we live the story, we live in the fog
You won’t think you’re making a mistake
Search your past for mistakes
Painful, embarrassing mistakes
If you didn’t find any, you’re exceptional
Either you’re making mistakes you find
Or you’re making mistakes you don’t find
How do you defend against mistakes?
The first step is admitting you have a problem
There are 174 cognitive biases[citation needed]
Data can help you make better decisions
Or more confidently make bad decisions
Data can’t make you a better decision-maker
Good data + bad decision-maker = bad decision
Our anti-lessons teach youhow to use data badly
Do the opposite to do better
Lesson 01Lesson 02Lesson 03Lesson 04Lesson 05Lesson 06Lesson 07
Be patientSampling is hardFocus on a few, carefully chosen metricsBe rigorous with your analysisWatch out for side effectsUse metrics and storiesPlan for fallibility
Learn from our mistakes
Be prepared for your own
Learn MoreEngineering blog & talks http://indeed.tech
Open Source http://opensource.indeedeng.io
Careers http://indeed.jobs
Twitter @IndeedEng
