looker's ben porterfield - asking the right questions
TRANSCRIPT
Ben Porterfield Founder, VP Engineering
Business Analytics: Asking the Right
Questions
B U S I N E S S I N T E L L I G E N C E
Operational Control How many sales did I do today?
Understand & Improve Experience Are users engaging? Do they like the new features?
Make business decisions Should we start delivering in a new city?
—ANDREW LEONARD Salon
“Data indicated that the same subscribers who loved the original BBC production also gobbled down movies starring Kevin Spacey or directed by David Fincher”
1 1 Tracking Data
2 Storing Data
3 Merging Data (ETL)
4 Retrieving Data
5 Analysis & Decision Making
The Analytical Process
Tracking Data
What To Track?
Views Clicks
In-app actions
E v e n t
Users Orders
Inventory
T r a n s a c t i o n a l
Embed in product process
Server-side too
Taxonomy matters
Tracking - Event Data
Every new feature should come with events
Lots of non-transactional events happen on server
Big flat event space becomes unwieldy
Storing Data
Go with SQL
Store all states
Keep it clean
Storing - Transactional Data
NoSQL could be a burden long-term
Even offline processes Messy schema = complicated analytics
—MICHAEL ERASMUS Back-end Engineer, Buffer
“We were relying on MongoDB…while it was easier for developers to play with the data, it became a hurdle for other team members.”
Own it Use eco-system too
Store all the IDs
Storing - Event Data
Or, at a minimum, be able to get it
Lots of great SaaS event platforms
Need to be able to correlate events to transactions
Merging Data
O t h e r D a ta?
Tra n s a c t i o n a l D a ta E ve n t D a ta
Ra w Q u e r i e s B i z- Us e r To o l s
You should combine transaction and event data, +more
Use an analytical database
Redshift is current leader
Difficult - data is heavy
A p p l i ca t i o n
WITH user_order_activity AS ( SELECT user_id, age FROM ORDERS GROUP BY user_id) SELECT AVG(users.age) as average_age_of_purchaser FROM user_order_activity LEFT JOIN users ON user_order_activity.user_id = users.user_id
SUMMARY
Traditional Approach
OLAP / Data Summaries S I L O E D
Restricted Q&A L I M I T E D
I
G
M N
L
D
Q
B A
P
R
S
Q
D I F F I C U LT & C O M B E R S O M E ETL - Heavy Transformation
E N D US E R B I T E A M E T L T E A M E D W T E A M
W A N T T O A S K N E W Q U E S T I O N S ?
A B
? C F
X
E B
A EVENT DATA
TRANSACTIONAL DATA
Modern Approach
3 R D PA RTY A P P
A P I
A N Y D E V I C E
Transformation at Query F L E X I B L E
Anywhere for Anyone A C C E S S I B L E C O N S O L I D A T E D
Simple Extract & Load
I
G
M N
D
Q
A
P
R
S
Q
T U W
X
G Q
U
S
A
Z
Data Modeling Layer A G I L E
D A T A T E A M E N D U S E R S
Data Model
- name: first_purchasers type: single_value base_view: orders measures: [orders.first_purchase_count] listen: - name: orders_by_day_and_category title: "Orders by Day and Category" type: looker_area base_view: order_items
I N N O V A T I O N
TRANSACTIONAL DATA
EVENT DATA
Z
B
Q A A Z
M P P | R E D S H I F T | I M P A L A
Asana’s Data Infrastructure
Retrieving Data
—TODD LEHR SVP Engineering, Dollar Shave Club
“We have a developer name Juan and any reports we needed would flow through him.”
—TODD LEHR SVP Engineering, Dollar Shave Club
“When he got backlogged, our team didn’t have access to the data immediately.”
—ANNIE CORBETT Business Intelligence Analyst, Venmo
“Initially whenever we were asked for data, we would write a custom script…”
—ANNIE CORBETT Business Intelligence Analyst, Venmo
“..and then repeat this process whenever the product team wanted to extend the timeframe.”
What’s selling? What colors and sizes is it selling in?
What’s getting returned? Is there a particular size/color?
Is there a product people buy first that increases their likelihood of becoming a repeat customer?
Questions from a retail buyer at e-commerce store:
Get them the tool
Decisions vs. data science
Game-changing insights
Self-Service is Key
People with questions are running the businesses.
“Should we open a new market in Maine?”
Don’t only come from analyst group
Analysis and Decision Making
1 1 Clearly define success metrics
2 Look for low-hanging fruit
3 Go one level deeper
Analysis and Decision Making
Analysis and Decision Making: Success Metrics
Focus on desired outcome What do you want users to experience?
Measure Engagement In most cases this is first-line business analytics
Measure Retention Are people coming back?
S U C C E S S M E T R I C S
H O W T O T R A C K E N G A G E M E N T ?
Not with page views Usually not even with time on page
Upworthy’s attention minutes Lots of indicators (mouse, video, etc)
Looker’s approximate usage Any event in 2 minute window
Deriving Approximate Usage
SELECT event.created_at AS created_date, event.user_id as user_id, COUNT(*) AS count, COUNT(DISTINCT CONCAT( CONCAT(event.user_id,'|',event.user_browser_id), FLOOR(UNIX_TIMESTAMP(event.created_at)/(60*2)) ) )*2 AS approximate_usage_in_minutes FROM event GROUP BY created_date, user_id
created_date user_id count approximate_usage
1/10 1 123 100 minutes
1/10 2 228 50 minutes
1/10 3 45 80 minutes
Derived Tables
SELECT orders.user_id as user_id COUNT(*) as lifetime_orders MIN(orders.created_at) as first_order MAX(orders.created_at) as latest_order COUNT(DISTINCT DATE_TRUNC('month’)) as
distinct_months_with_orders FROM orders GROUP BY user_id
Transactional
Event
Analytical
Derived Table
Insights
Start simple
Most useful at row level
Great for cohorts and sessionization
Derived Tables
Subselects until slow, SQL on cron works surprisingly well
Don’t roll up data, pre-compute facts
Tiered derived dimension vs. some other metric
Derived Table - User Order Facts
SELECT orders.user_id as user_id COUNT(*) as lifetime_orders MIN(orders.created_at) as first_order MAX(orders.created_at) as latest_order COUNT(DISTINCT DATE_TRUNC('month’)) as
distinct_months_with_orders FROM orders GROUP BY user_id
user_id lifetime_orders first_order latest_order distinct_months_with_orders
1 10 1/10/15 2/14/15 2
Derived Table + Sourcing
Derived Table + Sourcing
Churn Users that will likely never do X again
Usage How likely to purchase if they do X
Time to transaction How long till first X
Retention Are users coming back
??? Invent a metric
Repeat buyers What’s different about them
Pay/Charge Mistake.
It was clear some users were
accidentally paying instead of charging, but it wasn't clear
how widespread the problem was and
whether it was worth prioritizing a fix
Inventing Metrics
Identify behavior
Measure % of population
Experiment
Inventing Metrics
Can be good or bad – just something possibly significant
Who is doing this thing? Ability to play with numbers is crucial
Analysis and Decision Making: Low-hanging Fruit
This is the kind of very visual, very
data‑driven piece of analysis that
helps us think, "Is opening the sale at
noon the right decision?”
???
Low-hanging Fruit
Out of stocks are huge detractors from
the customer experience - it sucks ordering something and then not getting
it - as well as revenue we failed to
capture
Low-hanging Fruit
Analysis and Decision Making: One Level Deeper
While this immediate insight might have led us to focus on small groups, this didn’t match our expectations of
people planning an outing on a Friday
night, prompting us to look further.
One Level Deeper
2 3 4
Time To Book
2 3 4
Group Size
We analyze all the platform data
available - When someone attempts to sign, completes the signup, pushes an app, has spend,
etc
One Level Deeper
Even though it looks like we were
having nice incremental
growth, looking into the details we see
some things to look into further
One Level Deeper
Don’t confuse an increase in a metric with success.
Put data in analytical database
Give business users tool
Define success metrics
Takeaways
Make sure it’s fast and speaks SQL
Empower them to answer their own questions
Focus on engagement and retention
