Download - Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve
![Page 1: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/1.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Steve CollinsCTO / Cofounder Swrve
![Page 2: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/2.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
“One accurate measurement is worth 1,000 expert opinions.”
Admiral Grace Murray Hopper
![Page 3: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/3.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
A/B Testing
Split Test Measure Choose Deploy
![Page 4: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/4.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Start simple
0101
![Page 5: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/5.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
■Identify “low hanging fruit”■Choose minimal viable
implementation■Get a win as early as possible
Time to 1st interstitial? Tutorial guide
character?
vs
Force Login?
![Page 6: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/6.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
![Page 7: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/7.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
{variant = A}
UID = ABCD
![Page 8: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/8.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#1: Start Simple
Event(Tut_A)
A: 1,356 completedB: 1,793 completed
![Page 9: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/9.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Build a data driven culture
0202
![Page 10: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/10.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
■Data driven != lots of metrics / reports
#2: Data Driven Culture
![Page 11: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/11.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#2: Data Driven Culture
■Beware the HiPPOs
![Page 12: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/12.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#2: Data Driven Culture
■Link experiments to bottom line
Time to 1st interstitial?
CTR In-app Revenue
![Page 13: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/13.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Remove latency
0303
![Page 14: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/14.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#3: Remove Latency
■Close the loop
Measure
Take actionIterate, fail-fast
TEST
![Page 15: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/15.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#3: Remove Latency
Measure / Adapt
Measure / Adapt
DevelopDevelopARTENG
PM
00111011100011100001000001010101
Test FrameworkTest Framework
![Page 16: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/16.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Kill switch
0404
![Page 17: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/17.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#4: Kill Switch
■3 Schools of thought:1. Fail early / fast
![Page 18: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/18.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#4: Kill Switch
■3 Schools of thought:1. Fail early / fast2. Thorough QA
DraftDraft
StageStage
PublishPublish
QA Users
All Users
![Page 19: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/19.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
![Page 20: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/20.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#4: Kill Switch■3 Schools of thought:
1. Fail early / fast2. Thorough QA3. Ramp-up
![Page 21: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/21.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Minimize variation
0505
![Page 22: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/22.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#5: Minimize Variation
A B
+102%
www.abtests.com and diythemes.com
![Page 23: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/23.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Lookout for longitudinal impacts
0606
![Page 24: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/24.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#6: Longitudinal Impacts
■Clear winning variant:
Conversions up
![Page 25: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/25.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#6: Longitudinal Impacts
■But long-term KPIs are negative■Segment your metrics by variant
Test Start Time
KPI
![Page 26: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/26.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#6: Longitudinal Impacts
■Beware temporal effects
24 hrs
![Page 27: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/27.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Treat new and repeat users differently
0707
![Page 28: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/28.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#7: Treat new and repeat users differently
A
B
![Page 29: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/29.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#7: Treat new and repeat users differently
■“Primacy Effect”– Test new users only– Test users on a new app-version only– Ensure consistent user experience
![Page 30: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/30.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Test offers not prices
0808
![Page 31: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/31.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
15 15
#8: Test offers not prices
■Sometimes better to avoid direct testing
20 20
![Page 32: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/32.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
2525
#8: Test offers not prices
■Sometimes better to avoid direct testing
25 25
-20%-40%
![Page 33: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/33.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Run the math
0909
![Page 34: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/34.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#9: Run the math
■Null hypothesis testing
![Page 35: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/35.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#9: Run the math
■Null hypothesis testing
![Page 36: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/36.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#9: Run the math
■How long to run test?
![Page 37: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/37.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Save your results
1010
![Page 38: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/38.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
#10: Save your results
■ Name■ When it ran■ Owner■ Hypothesis■ Variants■ Result■ KPIs
![Page 39: Top 10 Tips for Mobile App Optimization using A/B Testing Steve Collins CTO / Cofounder Swrve](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649ead5503460f94bb5323/html5/thumbnails/39.jpg)
Top 10 Tips for Mobile App Optimization using A/B Testing
Thanks
"Planet Cute" art by Daniel Cook (Lostgarden.com)48 px Web Icons by emey87.deviantart.com
■Some handy links:– http://www.swrve.com/blog/ – http://www.kaushik.net – http://20bits.com/article/statistical-analysis-and-ab-testing
– http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/