user analytics testing - seleniumcamp 2015
TRANSCRIPT
What We’re Talking about
Overview of Web/User Analytics
Explanation of A/B Testing
Why this matters to you
Examples
How we test this stuff
Web Analytics - Advanced
Conduct Experiments
Tell stories from disparate points of data
Incremental learning
“If you are not paying for it… you are the
product being sold”
--Andrew Lewis (blue_beetle)
A Newer Example
Our 404 page had nothing on it
People landed there a lot by mis-typing store names
Should we put coupons on it?
A/B Test:
A: Control – No coupons on 404 page
B: Test – Coupons on 404 page
Keep it simple: top 15 coupons site-wide
Slice in 10% of traffic
User Analytics – Telling a Story
OK, so people did some clicking How many?
How many resulted in a transaction?
The big question: The amount of money we expect to make from
coupons on the 404 page: Is it worth the bandwidth? The load on the
servers/database?
Is it worth the potential future maintenance of this page?
Drawing a Conclusion Results after 2 weeks of testing tell us that the
B variation won!
Enough people used coupons, so it justified the relatively low expense
Ergo: Continue to put coupons on the page
Promote “B” test to “A”
New A/B Test: should we indicate coupon popularity on the 404 page coupons?
A: Control – No popularity indication
B: Test – Indicate coupon popularity
Real-world Examples
Shopping cart—shipping & tax calculation
Suggesting products and content based
on cookie, not login
Why You Should Care
What if the beacons they’re sending contain the wrong information?
But furthermore…
This is everywhere
It is only growing
Companies are becoming smart
(Really really smart)
You do not want to miss this opportunity to provide value
Why You Should Really Care
As a tester: There is a team of people working on this
It gets worked into features as they are developed
It is rarely called out separately in a scheduled task
It rarely receives QA outside of the PM and BI people who really care about it
*This is anecdotal, but I have yet to be told I’m wrong
Fortunately, It’s Easy
Usually one extra HTTP request, made
during a navigation event
Intercept this request, then verify the data
within it
Examples
Wells Fargo (s.gif)
Amazon (a.gif)
Netflix ([….]?trkId=xxx, beacon?s=xxx)
The New York Times (pixel.gif, dcs.gif)
OpenTable (/b/ss/otcom)
(and RetailMeNot – can you find it?)
Classic Approach
Marketing asks the BI team to figure out our ROI on TV ads during a period of time
BI requests PM to create a series of analytics
PM gives Dev the particulars
Dev assigns the code task to the newest person on the team
If anyone tests it, it’s also the newest person on the team
Classic Approach
Manual testing of web analytics is about
as exciting as reconciling a large column
of data with another large column of
data
…what if it’s wrong?
…what if it changes?
…why not let the software do it?
Test Cycle
@Test
Launch Browser
Navigate to
Position
Start Proxy
Perform Main
Action
Stop Proxy
Clean-up Test
Conclusion
User Analytics are your CEO’s favorite
subject!
Deliver real value—million-dollar decisions
are made with this data
Can be implemented with just as many
bugs as any other kind of software