data-driven product innovation
TRANSCRIPT
![Page 1: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/1.jpg)
Data-‐Driven Product Innova1on
Xin Fu LinkedIn
Hernán Asorey Salesforce
![Page 2: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/2.jpg)
WHY?
2
![Page 3: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/3.jpg)
Data Science Fuels Product Innova1on
3
![Page 4: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/4.jpg)
Example of Data Guiding Product Dev
Queries
4
Unique Searchers Unique
Searchers
Sessions Queries
Session * *
=
![Page 5: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/5.jpg)
How Did Data Science Help?
§ Metric to define product “true north”
§ Tools to help understanding
§ A/B experiments to learn and iterate
5
![Page 6: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/6.jpg)
Data Science’s Role in a Product Org
§ Model 1: Data Science as an Owner
§ Model 2: Data Science as a Service
§ Model 3: Data Science as a Partner
6
![Page 7: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/7.jpg)
Data Science as an Owner
§ Operates in a “hacky way” (early stage companies)
§ Key steps in business rely heavily on data – Recommender – Relevance – Matching – Scoring
§ Mostly back-‐end, rela1vely more stand-‐alone features
7
![Page 8: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/8.jpg)
Data Science as a Service
§ Engagement is “on-‐demand”, project-‐based § Examples:
– some of the BI roles – strategy roles (consultancy) – data API to product – modeling for specific purposes: propensity to {x}, where x: {buy, ahrite, convert, etc.}
8
![Page 9: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/9.jpg)
Data Science as a Partner
§ Plays ac1ve role in every stage of Product Life Cycle
§ Shares the ul1mate goal of product success § Oken requires an embedded engagement model
9
![Page 10: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/10.jpg)
Product Life Cycle
1. Idea1on
2. Design & Specs
3. Development
4. Test & Iterate
5. Release
10
![Page 11: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/11.jpg)
1. Idea1on
2. Design & Specs
3. Development
4. Test & Iterate
5. Release
Data-‐Driven Product Innova1on Framework
1. Hypothesis & Ques1ons
2. Holis1c Evalua1on Criteria
3. Tracking & Valida1on
4. Experimenta1on & Analysis
5. Monitoring & Learning
11
![Page 12: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/12.jpg)
5. Monitoring & Learning
§ Start from good reliable repor1ng of key product metrics
§ Par1cularly important for new partnership (“credibility projects”)
12
![Page 13: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/13.jpg)
Partner with Data Infrastructure Team to Build Great Tools
13
![Page 14: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/14.jpg)
4. Experimenta1on & Analysis
Ronny Kohavi’s key note “Online Controlled Experiments: Lessons from Running A/B/n Tests for 12 Years” Ya Xu et al.’s talk “From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks”
§ A company-‐wide plaqorm for A/B tes1ng, ramping, and advanced targe1ng needs
§ Automated repor1ng and analysis capabili1es
14
![Page 15: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/15.jpg)
Streams of Innova1on around A/B Tes1ng
15
![Page 16: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/16.jpg)
3. Tracking and Valida1on
§ Joint ownership between data scien1st, engineer, QA and product manager
§ Quarterly sign-‐off process from product owners
§ Created a tool for ongoing monitoring
16
![Page 17: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/17.jpg)
2. Holis1c Evalua1on Criteria Opportunity for strong thought leadership from data scientists Past: focus on product-specific metrics
– Relevance change ⇒ CTR – Profile redesign ⇒ # Profile Views
Now: standardized, tiered metric system
– Site-wide Tier 1 metrics – Product-specific Tier 2 / Tier 3
metrics 17
Tier 1 Tier 2 Tier 3
![Page 18: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/18.jpg)
1. Hypotheses & Ques1ons
Components of a good analysis
What we would like to see
18
INSIGHTS RECOMMENDATIONS DATA
![Page 19: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/19.jpg)
Effec1ve Ways to Drive Ac1ons
Secure a sponsor and build buy in
Consider the scale of target audience and technology
Verify by tes1ng with low-‐cost prototypes 19
Outline expected benefits
![Page 20: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/20.jpg)
Data Science in Ac1on!
20
![Page 21: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/21.jpg)
OrgDNA Abstract: Develop prescrip1ve tool to help admins configure orgs. Use machine learning to iden1fy the rela1onships between perms, prefs, user behavior and adop1on metrics.
Value Crea;on: § Increased product adop1on § Increased customer engagement
Accomplishments: § Decoding of user permissions and preferences into flat binary table for analysis § Joined in ini1al test target metrics (red accounts, tenure) and behavioral variables § Data QA and sanity checks complete (explore distribu1ons and ini1al correla1ons)
Sponsor: SVP of Mobile Benefi;ng Audience: Salesforce Administrators at Customer Loca1ons 21
![Page 22: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/22.jpg)
Just the Beginning…
22
Permission Frequency By Edi1on Preference Frequency By Edi1on
![Page 23: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/23.jpg)
Just the Beginning…
23
Super High Reten;on Group (cluster 2) -‐-‐Days_S1 > 16 -‐-‐CHURN 1%
High Reten;on Group (cluster 1) -‐-‐Days_S1 between 8 and 16 -‐-‐CHURN 5%
Lower Reten;on Group (cluster 4) -‐-‐Days_S1 < 8 -‐-‐CHURN 34%
High Ac;vity, Low Reten;on Group (cluster 3) -‐-‐Days_S1 < 4 -‐-‐Ac1vi1es Per Day > 6 -‐-‐CHURN 46% Cluster Averages
NMI Score = 0.082
![Page 24: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/24.jpg)
Data Science as a Partner
Quality & Speed of Product
Decisions
24
![Page 25: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/25.jpg)
Data Science as a Partner
§ Focus on Product and balance between – Product innova1on and opera1on – Velocity and scale – Theore1cal research and prac1cal impact
§ Leverage technology for reliability and sustainability – Reliability is key – Reduce “one-‐offs” – Speed mahers
25
![Page 26: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/26.jpg)
Case Study
26
![Page 27: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/27.jpg)
Data-‐Driven Product In Ac1on
In order to increase the connec1on density of new users, we will lookup the exis1ng members whose address books contain these new users as contacts The exis1ng members will be no1fied that their contacts just joined LinkedIn and will be prompted to connect with them
27
![Page 28: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/28.jpg)
Anatomy of a Data-‐Driven Product
1. Problem Statement 2. Opportunity Analysis
– Iden1fy target audience – Verify feasibility
3. Holis1c Evalua1on Criteria – Including impact es1mate
4. Low-‐cost Tes1ng with Tracking 5. Analysis of Test Results 6. Hand off for Scalable Implementa1on
28
![Page 29: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/29.jpg)
Step 1. Problem Statement
Goal: Increase the connec1on density of new LinkedIn members. Why? § X percent of new users have Y or few
connec1ons at 30 days aker registra1on § High connec1on density => Beher engagement
29
![Page 30: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/30.jpg)
Step 2. Opportunity Analysis
30
Target Audience: 5 (50%) Of the 10 new members, 5 are eligible Feasibility Analysis: Poten1al Upside: X new connec1ons for each new member For the 5 eligible New Members, median number of poten1al ‘welcomer’ is 2 X = Median (1,2,2,1,3)
No1fica1on System Load: Of the 10 exis1ng members, 4 are poten1al ‘welcomers’ Member Overload: The volume of no1fica1ons to receive by poten1al welcomers has a distribu1on with Median=2 and Max=3
E1
N7
E10
E2
E9
E8 N8
N4
N1
N6
E5
N3
E6
E4
E3
E7
N2
N9
N5
N10
AàB : A appears in B’s address book N: New Member E: Exis1ng Member
1
2
2
3
1
![Page 31: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/31.jpg)
Step 3. Holis1c Evalua1on Criteria
§ Primary Metrics (New Members) – connec1on density – reten1on rate
§ Secondary Metrics (Exis;ng Members) – Connec1on invita1ons sent – Connec1on density – Engagement
• Impact Es;mate – Need to make assump1ons about conversion rate
31
![Page 32: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/32.jpg)
Step 4. Low-‐Cost Tes1ng
§ Email Test § Daily offline Hadoop job to generate the list of members eligible to receive the email (with eng. review)
§ Randomize for the recipients § Partner Involvement
§ Work with Engineering and QA to set up tracking § Work with Ops, SRE and Customer Service teams to set up monitoring
32
![Page 33: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/33.jpg)
Step 5. Analysis of Test Results
§ Email open rate, invite rate, invite acceptance rate
§ Es1mate impact on exis1ng members
§ Es1mate impact on new members
33
![Page 34: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/34.jpg)
Step 6. Scalable Implementa1on
Handed off to Engineering to scale up the impact:
§ Expand target audience, e.g. to dormant members
§ Offline email to online and mobile no1fica1ons
§ Introduce relevance rules
34
![Page 35: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/35.jpg)
Exercise
35
![Page 36: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/36.jpg)
Design Your Data-‐Driven Product
Please refer to the hand-‐out
§ 1. Pick an idea for a data-‐driven product either for your ins1tu1on, or for your favorite Web product
§ 2. Outline your plan to evaluate the idea – Problem statement – Opportunity analysis (target audience, feasibility) – Holis1c Evalua1on Criteria and poten1al impact – Low-‐cost tes1ng plan
§ 3. Iden1fy who you can partner with
36
![Page 37: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/37.jpg)
Data Science’s Role in a Product Org Data
Science Team
Structure Importance of Priori@za@on
Velocity Sustainability Proac@ve Approach
As an Owner Single
As a Service
Center of Excellence
As a Partner
Hub & Spoke
37 Low High
![Page 38: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/38.jpg)
Data Science as Product Partner
PARTNERSHIP
TECHNOLOGY
TALENTS
§ Start w/ credibility projects § Context and ownership § Scalable path to innova1on
§ Leverage technology to improve quality and speed
§ Reliability is key § Automate, Automate!
? 38
![Page 39: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/39.jpg)
Data Science as Product Partner
§ We put great emphasis on understanding and solving real business problems – Data sense: what is possible – Product sense: what is valuable
§ We value people who think holis1cally and in scale – Cross-‐product impact – Viral effect
39
Talents
![Page 40: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/40.jpg)
Tutorial Takeaway
PARTNERSHIP
TECHNOLOGY
TALENTS
§ Start w/ credibility projects § Context and ownership § Scalable path to innova1on
§ Leverage technology to improve quality and speed
§ Reliability is key § Automate, Automate!
§ Passion for product innova1on § Proac;ve in partnership § Impact oriented
Data Science as a Partner
40
![Page 41: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/41.jpg)
41
THANK YOU
![Page 42: Data-Driven Product Innovation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5878f8d31a28ab49608b6039/html5/thumbnails/42.jpg)
Data-‐Driven Product Innova1on
Xin Fu LinkedIn
Hernán Asorey Salesforce