2014 Startup Digest SF/SV July user survey analysis
David Kim and Peter Shin
Summary
• Learn to ask better questions
– Win: we have a high net promoter score (8)!
– Loss: we do not have clear drivers of success
• Of 96 unique responses:
– 30% were Founders, VCs, or C-level readers
– 5% were interns/students
• We should focus on success metrics that matter
Net promotion score distribution
-
5
10
15
20
25
30
35
3 4 5 6 7 8 9 10
# re
spo
nse
s
score
distribution of net promotion scores
Cross-section of responses
But What Factors Drive NPS?
• And are those factors statistically significant?
• Hypothesis (very limited given the data set we collected – see appendix for the survey): – Organizers promote SD more highly
– Years lived in the area is also positively correlated to score
• Or mathematically: – NPS = a*(Organizer) + b*(Yrs lived) + intercept
– Where Organizer is a dummy variable (i.e. 0 or 1)
Regression results
• Surprisingly, longer you live here, less likely you are to recommend the digest to friends. Call it cynicism
• Sadly, neither variables are statistically significant
– specifically, t-stat not above 2 (or p-value not low enough)
• Moreover, adjusted R^2 suggests that these factors basically have no explanatory power of the NPS
Conclusion & Afterthoughts
• In conclusion, we know we have a very high net promoter score (8)
– But, we don’t know why. We did not design our survey with the intent of discovering key metric drivers, but that would have been nice
• 100 responses among tens of thousands of readers is rather small
– Key takeaway is we did not have a baseline starting out
– Now we have one, and we can improve upon it
Afterthoughts, cont.
• We suspect that NPS is a vanity metric
• The real measures that matter are:
– Reach (equivalent to revenues for a business, and reflects sharing)
– Click-through rates (reflects usefulness to users, and therefore reflects quality perception)
• In future surveys or partnership initiatives, we have the above baseline, and can optimize efforts that increase those outcomes
Appendix: data clean-up
• Original survey link: – https://docs.google.com/forms/d/19k2gXjkDmplPo3n7BsK
v1VgrmlP-UvSADLnABYTr5Cc/viewform
• Data clean-up notes – Organizer column: 0 = no, 1 = yes
– Yrs lived: reduced <1 yr and n/a to 0
– Job position: created new categories to reflect responses as closely as possible
Appendix: data tables