getting it booking right
TRANSCRIPT
![Page 1: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/1.jpg)
Ge#ng it Booking right – Mobile apps
Iaroslav, Khramov
Team Leader & Android developer
![Page 2: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/2.jpg)
Iaroslav Khramov TL & Android developer
About me
![Page 3: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/3.jpg)
Making Booking right apps
ExperimenAng • Why • How • What do you achieve
Developing TesAng Releasing
![Page 4: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/4.jpg)
Booking.com
• Founded and based in Amsterdam • Biggest accommodaAon plaKorm in the world • > 8,000 dedicated employees
Mobile apps: • iOS & Android • 7 app development teams • ~35 apps developers
![Page 5: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/5.jpg)
Booking.com apps
Android • > 10M downloads * • raAng 4.6 out of 5 *
iOS • raAng “4+” out of 5 **
* according to Google Play Store on 23/03/2015 ** according to Apple Store on 23/03/2015
Facebook Vkontakte Instagram YouTube Gmail Inbox WhatsApp
4.0 / 5 * 4.2 / 5 * 4.5 / 5 * 4.1 / 5 * 4.3 / 5 * 4.2 / 5 * 4.4 / 5 *
![Page 6: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/6.jpg)
Our challenges
• Users all around the world -‐ Millions of users -‐ Our user doesn’t have a unique profile -‐ High user expectaAons
• 40+ languages • > 600k properAes • expert opinion vs. users opinion
![Page 7: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/7.jpg)
What do we want to achieve?
Product that is: • User friendly • TrusKul • Reliable • …
![Page 8: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/8.jpg)
What do we want to achieve?
Product that is: • User friendly • TrusKul • Reliable • …
Provide our users the best experience before, during and after making a
booking
![Page 9: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/9.jpg)
Are you sure?
• Are you sure that new design is good? • Are you sure this change will not break the app? • Are you sure customers will like our new feature? • Are you sure …. ?
![Page 10: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/10.jpg)
Variant A Lek hand up
Variant B Right hand up
Can you please stand up
Lets play!
![Page 11: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/11.jpg)
Which got more money on Kickstarter?
Smartphone Controlled Paper Airplane
Jot: CapaciAve Touch Stylus
A B
![Page 12: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/12.jpg)
Winner: Version B
A B
Which got more money on Kickstarter?
![Page 13: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/13.jpg)
Winner: Version B
A B
Which got more money on Kickstarter?
People with hand up - please take a seat
![Page 14: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/14.jpg)
Which variation performed better?
A B
![Page 15: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/15.jpg)
Which variation performed better?
Winner: Version A
“The truth is, someAmes it’s bener to have no proof than low proof.”
![Page 16: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/16.jpg)
Which variation performed better?
Winner: Version A
“The truth is, someAmes it’s bener to have no proof than low proof.”
People with hand up - please take a seat
![Page 17: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/17.jpg)
Which got more requests? A B
![Page 18: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/18.jpg)
Winner: Version A
“Like most people, if you guessed that Version B won, you are wrong. Version A surprisingly had a 24% increase in submissions without using an image on the page.”
Which got more requests?
![Page 19: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/19.jpg)
Winner: Version A
“Like most people, if you guessed that Version B won, you are wrong. Version A surprisingly had a 24% increase in submissions without using an image on the page.”
Which got more requests?
People with hand up - please take a seat
![Page 20: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/20.jpg)
What is better for conversion? A B
![Page 21: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/21.jpg)
What is better for conversion?
Winner: Version A
In this example removing the social bunons resulted in an increased conversion to the basket of 11.9%.
![Page 22: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/22.jpg)
What is better for conversion?
Winner: Version A
In this example removing the social bunons resulted in an increased conversion to the basket of 11.9%.
People with hand up - please take a seat
![Page 23: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/23.jpg)
More completions?
A B
![Page 24: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/24.jpg)
More completions?
Winner: Version B
“…surely adding an eTrust image would improve form compleAons, it makes everyone feel safe and secure, doesn’t it? The answer is no. Although it’s only a slight rise of 12.6%, Version B, without the image, proved to be more effecAve.”
![Page 25: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/25.jpg)
More completions?
Winner: Version B
“…surely adding an eTrust image would improve form compleAons, it makes everyone feel safe and secure, doesn’t it? The answer is no. Although it’s only a slight rise of 12.6%, Version B, without the image, proved to be more effecAve.”
People with hand up - please take a seat
![Page 26: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/26.jpg)
Which Copy Increased Trial Sign-Ups? A B
![Page 27: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/27.jpg)
Which Copy Increased Trial Sign-Ups?
Winner: Version A
“In this test, Version A increased sign-‐ups by 38% – a big rise. However, your gut feeling might iniAally be that version B is the bener design.”
![Page 28: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/28.jpg)
Which Copy Increased Trial Sign-Ups?
Winner: Version A
“In this test, Version A increased sign-‐ups by 38% – a big rise. However, your gut feeling might iniAally be that version B is the bener design.”
People with hand up - please take a seat
![Page 29: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/29.jpg)
Decision-making using data: A/B testing
• When should you should run an experiment (use A/B tesAng)? Need to decide which opAon is bener
• What are the alternaAves? Expert opinion / intuiAon Historical data
![Page 30: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/30.jpg)
Why is experimentation better?
• Our intuiAon is not very good (people are complex)
• Historical data
(becomes outdated fast)
![Page 31: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/31.jpg)
What do we get if using A/B testing?
• Reduced design discussions • Less dependent on expert opinions • Make sure all changes are actually (commercial)
improvements • AddiAonal safety net for new changes in the app
![Page 32: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/32.jpg)
How complex is it?
if ( newFeatureEnabled == true ) { // new feature code
} else { // old reliable solu2on
}
![Page 33: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/33.jpg)
+ add control from server-side
Today: {
Experiment A: ON Experiment B: ON
}
Server Control experiments
At any moment of Ame: {
Experiment A: ON Experiment B: OFF
}
![Page 34: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/34.jpg)
It’s easy! Can I start experimenting now?
Yes, but first: • Pick a metric • Find users to include in an experiment
![Page 35: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/35.jpg)
Metrics
Metrics are the main determinant of success!
1. Pick your own metric! 2. Diversify metrics!
![Page 36: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/36.jpg)
Metrics
Metrics are the main determinant of success!
1. Pick your own metric! 2. Diversify metrics!
Metrics for Booking.com?
![Page 37: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/37.jpg)
What is my metric?
Type of business Possible key metrics
Ecommerce: Booking.com, Amazon
items sold, conversion, loyalty
Content: Euronews, CNN visitors, loyalty, engagement, ad clicks
Social Networking visitors, engagement, ad clicks
Gaming: Angry birds purchases, subscripAons, ad clicks
![Page 38: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/38.jpg)
Come up with new idea to improve your
app Run an
experiment to see which
variant is the most successful
Select the winner based on
metric
Remove non-‐winner from the
code
Experiment lifecycle
![Page 39: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/39.jpg)
Example of an experiment A B
![Page 40: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/40.jpg)
Tools for mobile that will allow you to start experimenting today?
Yes, just Google it! For example: • Airlock • Arise.io • AppAmize • Leanplum …
![Page 41: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/41.jpg)
A/B testing in apps provides
• Final changes are real improvements • All changes are server-‐side controlled and can be
enabled \ disabled at any point of Ame
![Page 42: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/42.jpg)
A/B testing in apps provides
• Final changes are real improvements • All changes are server-‐side controlled and can be
enabled\disabled at any point of Ame
Is it enough for our apps?
![Page 43: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/43.jpg)
Booking apps development
• Use Experiments • Develop fast • Be stable • Release fast • Monitor how you are doing
![Page 44: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/44.jpg)
Source: XKCD 303 http://imgs.xkcd.com/comics/compiling.png
![Page 45: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/45.jpg)
Fast, iterative development
Android – slow, > 3 min to build on big projects
• Buck + exopackage • Booking DCL (internal soluAon) • others
iOS – usually not a problem
![Page 46: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/46.jpg)
Trunk development
What & how: • Everything goes into master • Local branches up to a developer • Tests keep master stable
Why: • As a result saves Ame on merging
![Page 47: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/47.jpg)
Making sure project builds and stable
• Hooks • CI • TesAng
![Page 48: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/48.jpg)
Hooks
Master can always build!
git push
Developer
pass
fail
git repository
Hooks:
build & run test
![Page 49: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/49.jpg)
CI
git push git repository
CI system Great job
Developer
pass
fail
Email && jabber developer about the problem
• all dev build • special builds
• TranslaAons • UI tests
![Page 50: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/50.jpg)
Testing
• Dedicated tesAng team • Easy maintainable tests on core funcAonality + UI tests • Bugbash sessions • Internal releases • 3rd party tesAng
![Page 51: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/51.jpg)
Release model
Train release model • Regular • Pre-‐planned schedule
![Page 52: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/52.jpg)
Release model
Train release model • Regular • Pre-‐planned schedule
Nobody waits: you take this train or the next one
Author: Maurits Vink http://commons.wikimedia.org/wiki/File:VIRM6.jpg
![Page 53: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/53.jpg)
Why train model?
• Everyone know when the next release will take place • Team can plan the scope to be done • Complexity of each release more or less the same
![Page 54: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/54.jpg)
App roll out
Rolling out apps not as fast as web: Android: • ~1 day to all users
iOS : • Depends on Apple • Apple review process 1-‐5 days
![Page 55: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/55.jpg)
App roll out
Rolling out apps not as fast as web: Android: • ~1 day to all users
iOS : • Depends on Apple • Apple review process 1-‐5 days
Hot fixes can’t be fast…
( Experiments can help! )
![Page 56: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/56.jpg)
Safe app release
Android: • staged rollouts • alpha & beta tesAng iOS: • Enterprise signing program • TestFlight – beta tesAng for iOS 8
![Page 57: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/57.jpg)
Staged rollout
Get thing done right for Android users • Make sure that everything goes according to plan • NEW! 'Halt' A Staged Rollout if found a problem • When sure -‐ release to 100%
iOS: • No support for staged rollouts
![Page 58: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/58.jpg)
Crash & data monitoring
Monitor your crashes, errors and main metrics via: • CrashlyAcs, • Bugsense, • Google AnalyAcs, • Flurry AnalyAcs, • ….
![Page 59: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/59.jpg)
Booking apps
ExperimenAng • Why to try • How • What do you achieve
Developing TesAng Releasing
![Page 60: Getting it Booking right](https://reader033.vdocuments.mx/reader033/viewer/2022042817/55a78c751a28ab266e8b47cf/html5/thumbnails/60.jpg)
References
1. Smartphone Controlled Paper Airplanehnps://www.kickstarter.com/projects/393053146/powerup-‐30-‐smartphone-‐controlled-‐paper-‐airplane
2. Jot: CapaciAve Touch Stylus by Adonit!hnps://www.kickstarter.com/projects/531383637/jot-‐capaciAve-‐touch-‐stylus/descripAon
3. 12 Surprising A/B Test Results to Stop You Making AssumpAonshnp://unbounce.com/a-‐b-‐tesAng/shocking-‐results/
4. 5 Landing Page A/B Tests (And Their Surprising Results) hnp://blog.hubspot.com/markeAng/5-‐landing-‐page-‐a/b-‐tests-‐and-‐their-‐surprising-‐results
5. Removing Social Sharing Bunons Increases Conversions. Yes, You Heard That Right!hnps://vwo.com/blog/removing-‐social-‐sharing-‐bunons-‐from-‐ecommerce-‐product-‐page-‐increase-‐conversions/
6. Source: XKCD 303 hnp://imgs.xkcd.com/comics/compiling.png 7. Foto van twee VIRM VI stellen te Amsterdam Bijlmer ArenA. Author: Maurits Vink
hnp://commons.wikimedia.org/wiki/File:VIRM6.jpg