a blueprint for a/b testing and optimization on...
TRANSCRIPT
How Best-in-Class App Developers Use Data to Develop Amazing Mobile Apps
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
2
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
EVALUATING MOBILE STRATEGY
QUANTIFYING GOALS ON MOBILE
HOW CONVERSION RATE OPTIMIZATION (CRO) FITS IN
TESTING
ANALYTICS
SEGMENTATION
PERSONALIZATION
COMPARING UX & OPTIMIZATION: WEB VS. MOBILE
UX MEETS CRO ON THE WEB
UX MEETS CRO ON MOBILE
CASE STUDY: TASK RABBIT - WEB VS. MOBILE
TECHNICAL DYNAMICS ON THE WEB VS. MOBILE
CRO ON MOBILE: ACQUISITION
OPTIMIZING FOR APP STORES
OPTIMIZING FOR REGISTRATION/KEY ACTIONS
CRO ON MOBILE: RETENTION
TECHNICAL CONSIDERATIONS
PRODUCT CONSIDERATIONS
CONCLUSION
0 1
03
02
04 05
06
CONTENTS00
3
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
A majority of Americans now own a smartphone of some kind –
positioning mobile as one of the most significant opportunities for
consumer and B2B technology companies.
It’s no surprise that the app market is growing, with more than one
million apps both in the Apple App Store and Google Play Store.
How do you make your app stand out from the crowd? The first
step is to craft a best-in-class user experience.
Optimizely has already built a market-leading platform for
creating best-in-class user experiences on the web. With users
spending an increasing amount of time on mobile devices, it’s
become critically important for businesses to be able to deliver
amazing user experiences that span the web and mobile.
Delivering on this, however, is easier said than done. Many budding
app developers will soon learn that while their web and mobile
users are often the same people, those people have very different
goals and behaviors when engaging on mobile vs. when engaging
on the web.
That’s why we wrote this guide – to provide you with the most
comprehensive and in depth resource to learn:
(1) What makes an awesome mobile user experience
(2) How to evolve your mobile strategy to create highly engaging
products
(3) How to optimize that user experience for conversions
WHILE WEB AND MOBILE USERS ARE
OFTEN THE SAME PEOPLE, THOSE
PEOPLE HAVE VERY DIFFERENT GOALS
AND BEHAVIORS WHEN ENGAGING ON
MOBILE VS. WHEN
ENGAGING ON THE WEB.
1. HTTP://PEWINTERNET.ORG/REPORTS/2013/SMARTPHONE-OWNERSHIP-2013.ASPX
1
4
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
01 EVALUATING MOBILE STRATEGY
With smartphone penetration rates as high as 74% , your com-
pany’s mobile strategy is more important than ever. Mobile-only
companies like Instagram, WhatsApp, and Snapchat have forged a
path to billion-plus dollar valuations. Not to mention, today’s most
innovative firms like Eventbrite, an event management platform,
and Percolate, software for content marketers, are investing heav-
ily in developing mobile-focused solutions.
What’s your mobile strategy? How can your brand make the
strongest impact possible?
2. HTTP://MASHABLE.COM/2013/08/27/GLOBAL-SMARTPHONE-PENETRATION/
2
5
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
JUST-FOR-FUN APPS ARE
EVERYWHERE — WHICH MAKES IT
TOUGH FOR APP DEVELOPERS
TO STAND OUT.
Years ago, a successful mobile strategy meant ‘having an app.’ But
the landscape has quickly changed. Hollywood success stories are
becoming less and less common. Just-for-fun apps are everywhere
— which makes it tough for app developers to stand out.
The secret to a successful mobile app is no secret at all. Plain and
simple, top mobile apps delight users by 1: making their lives easier
and/or 2: providing entertainment.
Consumers crave elegant user experiences, workflow tools, and
highly engaging games. They’re looking for resources to be more
productive, opportunities for brain breaks, and touch points to
stay connected with friends and family.
That’s why analytics, A/B testing, and conversion rate optimization
are extremely important to the app design process. You need
to be learning about your users, testing your assumptions, and
exploring areas for improvement.
QUANTIFYING GOALS ON MOBILE
Before building an app, product owners will need to clearly define
their revenue objectives. This process starts with identifying key
metrics that align with an engaging – and profitable – mobile user
experience.
Start by thinking about the metrics that define success on a
desktop. Some of the user experience objectives (i.e. user engage-
ment and monetization) will be similar. You’ll want your customers
to be engaging with the product in a way that drives revenue. At
the same time, you need to approach your products goals with a
mobile-first mindset, understanding that you’re reimagining your
products for a completely new device.
Pay attention to the following metrics when designing for mobile:
1. DAILY ACTIVE USERS (DAUS): DAU measures your app’s
engagement levels – how many unique users visit the app daily?
A high DAU count is a sign that your app is something that people
need (or really love). For instance, users might love your game
and decide to play it, every day, on their morning train commutes.
Users might rely on your app to organize their notes (like
Evernote), book vacation rentals (like HotelTonight), and organize
their content marketing programs (like Percolate).
6
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
2. AVERAGE RATING & REVIEWS: This metric quantifies what
users think of your app and directly impacts performance in app
marketplaces.
App reviews are also a great place for developers to research user
concerns -- which can directly influence your brand positioning
and future product releases.
So how do you get users to review your app?
Well, you ask them.
But here’s the thing. Simply telling users to “leave a great review”
won’t be enough. In addition to prompting users to review your
app, you need to think about how to position these requests within
your user experience.
Most apps will generally prompt a user to leave a rating or review
upon launching the app - interrupting the general user flow. A bet-
ter way to prompt the user is after they have finished doing what
they need to do within your app -- or after they’ve accomplished
something that gives them a sense of achievement (like reaching a
high score within a game). For example, FIFA 2012 prompts you to
leave a review for the app after you have won a major tournament.
Apple’s App Store uses app ratings as a factor in calculating Top Chart rankings. It’s tough for apps to bounce back from a low ranking -- even if they’re downloaded fairly often.
7
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
4. RETENTION: Retention is calculated as DAUs divided by
number of downloads. It’s a key driver for apps with in-session
monetization opportunities. Are your DAUs increasing as down-
loads increase? If not, it is worth digging deeper into why users are
abandoning your app.
Apps using the freemium model as their main source of revenues
must be sure that they have the proper mechanics in place to
maximize retention and upselling.
As an example, Clash of
Clans (a popular mobile
game) offers virtual
currency and cleverly
timed push notifications.
Some app companies
have even determined
the length of time it takes
to monetize a user and
will even serve ads to
competing games when
a user has past that time
frame and still has yet to
monetize.
3. DOWNLOADS: Downloads of an app are the equivalent of an
account sign-up on the web and play a crucial role in the app’s
ranking. It’s important to monitor downloads with DAU to ensure
that both are trending positively. If you see a spike in downloads,
do you see a similar spike in DAU? If not, you may have to revisit
your retention features (and usage levels) or test different push
notification messages.
As an example, the Carrot To-Do app does a great job of using
humor in its push notifications to pull users back into the app.
Virtual currency and push notifications in Clash of Clans
Carrot To-Do in the App Store
8
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
5. APP CRASHES / FAILURES: While difficult to track,
it is important to have crash analytics built into your app.
Unfortunately, the default Google Play and Apple analytics do not
capture all crashes, so it’s important to use an additional crash
analytics tool.
To circumvent a crash
or feature request from
turning into a negative
review, app developers are
implementing feedback
functionality so users can
provide performance issues
or new feature suggestions
directly within the app.
For example, Tweetbot, a popular Twitter client, implements a
feedback mechanism within their settings page.
Tweetbot’s in-app feedback mechanism
Negative bug-related app review left by a Tweetbot user
9
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
HOW CONVERSION RATE OPTIMIZATION (CRO) FITS IN
02 Conversion rate optimization describes the structured and sys-
tematic approach to improving the performance of your website
or mobile app through analytics and user feedback. Simply put,
it’s the practice of running experiments to 1: find out why users
aren’t doing what you want them to do, 2: test your assumptions,
3: iterate, and 4: test again.
CRO is a long-established process that began in the early days
of the Internet -- long before smartphones entered the picture.
The process starts with understanding your company’s unique
objectives and key performance indicators (KPIs). Techniques such
as A/B testing and qualitative research allow business owners to
use data to make informed user-driven decisions.
To understand how CRO fits in with mobile, you need to
understand its application on the web. Before we jump back into
mobile-specific workflows, let’s start with a quick refresher by
reviewing the following nuts and bolts of CRO.
10
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
TESTING
A/B tests allow companies to compare page versions (or ele-
ments) against one another. Each page variation is then measured
by its effectiveness in comparison to the other. For example, a
marketer may compare a page with a green call-to-action (CTA)
to a page with a red CTA. A/B tests focus on only one element at
a time.
ANALYTICS
Analytics empower you to assess your current conversion rate,
identify trouble areas, and design an execution action.
Pay attention to the following metrics:
• CONVERSION RATE: This metric bridges the connection between
website traffic and revenue.
Are users taking the action that you would like them to? If not,
analyze your conversion flow and start testing the page(s) with
the biggest drop off. This metric will help you make sure that
users are completing desired goals within your app (making
purchases, returning quickly, encouraging friends to download
the app, etc.)
• BOUNCE RATE: This metric captures the percentage of visitors
who leave your website after viewing a single page.
If your web pages have a high bounce rate, it could be that
users are expecting something different than what you are
presenting. A high bounce rate could be a symptom of a con-
fusing, cruddy, or less-than-intuitive app design.
Bounce rates will tell you what’s happening, but you need to dig
deeper to understand why. Qualitative research -- with a service
like UserTesting -- can help you understand what users are
expecting from your website or mobile product.
• EXIT RATE: This metric captures the percentage of visitors who
leave your website after viewing a specific page.
Each page on your website will have its own exit rate. Pay close
attention to the exit rate on your crucial pages such as the
homepage, pricing page, and checkout flow. If you see a high
exit rate on a page during the checkout flow, it may be worth
testing that page to see if you can improve user retention.
• AVERAGE TIME ON SITE: This engagement metric gives you an idea
of how long people are staying on your website.
Are visitors who spend more time on your site more likely to
convert? Or are users spending a lot of time on your site, but
your conversion rate is very low? Optimizing these metrics will
reveal whether you need to make changes or test different call
to actions. Focus your attention to variables that are likely to
improve user engagement.
• AVERAGE PAGE VIEWS: This engagement metric tells you how
many pages the average visitor goes through before leaving
your website.
Average page views is another metric to measure in your con-
version optimization. Are visitors who see more pages on your
11
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
site more likely to convert? Are visitors who read 3 or more blog
posts likely to convert into trial users? If so, it is worth testing a
call to action at the end of each blog post.
Know your KPIs. These metrics will help you develop laser-focused
optimization experiments. On the web, these goals will typically
include leads, user account sign-ups, e-commerce checkouts, and
direct sales.
Although a completely different platform, on mobile, you will find
similar goals that relate to user engagement, retention, conversion,
and revenue. Example metrics to optimize include downloads,
conversions from free users into paid customers, and retention
rates.
KNOW YOUR KPIS. THESE METRICS WILL
HELP YOU DEVELOP LASER-FOCUSED
OPTIMIZATION EXPERIMENTS.
12
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
SEGMENTATION
No two visitors are alike. One visitor may be new to your brand
while the other may be a long-time customer. A well-optimized
website ensures that your brand aligns with both user experiences.
Businesses will typically design segments based on the following
characteristics:
• Where is the user in the sales funnel
• New versus repeat visitor
• The type of customer (small business vs. enterprise)
• Decision maker or influencer in purchase decision
• Demographic variables like gender and age
• Levels of technical sophistication
PERSONALIZATION
Personalization can be thought of both from a technical per-
spective and also from a marketing or promotional perspective.
From a technical point of view, personalization involves displaying
information about the user that has been saved from prior visits.
App developers can then create algorithms to behaviorally target
users with app features (or offers) that they’ll find relevant.
Amazon has done a great job of personalizing each user’s experi-
ence. Simple things such as displaying the visitor’s name, showing
the products previously viewed by the user, and present products
a visitor may be interested in based on past data.
Personalization can also be incorporated in your static marketing
or promotional copy. You can use surveys to find common
objections about your service or features that customers find most
valuable. You can use these findings to craft a more personalized,
human message. Speak in a language that resonates with your
audience. Your audience will see that your brand fully understand
their needs and that you have the right product to solve their
problems.
The bottom line of personalization is to show the right marketing
message to the right audience at exactly the right time -- in a
human, engaging, and personally relevant way. Present each user
with a unique experience on your website.
13
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
COMPARING UX & OPTIMIZATION: WEB VS. MOBILE
03User experience (UX) involves more than just aesthetics. This
concept encompasses the totality of how your user engages with
your website, product, or app. What’s important to keep in mind
is that user experiences span multiple touch points, especially for
customers who engage with your brand across multiple platforms.
You need to pay attention to the following:
• How easy your site or app is to use
• How fast your product is
• How much friction exists when users complete key actions
UX and CRO go hand-in-hand. Across platforms, you need to
guide your audiences through the conversion funnel.
There are a few major differences when it comes to designing
conversion optimized user experiences on the web versus on
mobile. In this section, you will discover the key differences and
how to best optimize for each screen.
14
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
UX MEETS CRO ON THE WEB
On the web, we are primarily stationary with our keyboard and
mouse. Key distractions include other browser tabs, IMs, email
notifications, and other software programs.
On the web, you’ll be focusing primarily on two aspects of UX:
1. Reducing friction in the
form of wasted clicks, excess
pages, slow page loads, and
other friction points that
cause users to give up.
2. Reducing doubt and
indecision from the user’s
mind that may cause them to
not convert.
DESIGN FLOWS NOT PAGES
Too often, we prematurely turn our focus to page design and
information architecture, when we should focus on the user flows
that need to be supported by our designs.
When starting your next web project, consider starting with a
detailed look at the objectives of the user and the business. This
practice will allow you to scheme the various flows that address
both parties.
User objectives could range from finding case studies to learning
more about a product to making a purchase. While business
objectives include email sign-ups to filling out a form to a phone
call.
Visitors to your website come from different sources with varying
levels of brand engagement. Each visitor has her own goals in
mind and it’s up to you to map
each user flow to the appropriate
conversion funnel.
It’s important to prioritize quality
over quantity of users.
In other words, it’s better to have
100 engaged, high-converting
users than 1,000 who use your
app once and then disappear.
High-quality users are especially important when using paid
acquisition channels. You want to make sure you are attracting
users that you can ultimately monetize.
Focus your efforts on the user flows that will have the biggest
impact on your visitors and result in the biggest gain for your
business.
IT’S BETTER TO HAVE 100 ENGAGED,
HIGH-CONVERTING USERS THAN
1,000 WHO USE YOUR APP ONCE
AND THEN DISAPPEAR.
LANDING PAGE
EMAILSIGNUP
SOCIAL MEDIA SHARE
15
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
BEST PRACTICES FOR UX DESIGN While there is no exact formula for the perfect landing page, there are common elements that you should monitor to maximize your conversions.
• MAIN HEADLINE – The landing page headline should comple-ment the source of the traffic. This is especially true when you are driving paid traffic to your landing page.
• SECONDARY HEADLINE – Try to address a specific point that is related to the content of the landing page rather than having a vague or uninteresting headline.
• COMPLIMENTARY VIDEO OR PHOTO – Customer testimonial videos, product demos, or strong visuals can have a positive impact on visitors.
• SOCIAL PROOF – Build trust with your visitor by incorporating trust symbols such as testimonials, press mentions, number of customers, and guarantee banners.
• STRONG CALL TO ACTIONS – The primary call to action button should stand out from the rest of the page and contain words that your visitors might be searching for such as “free,” “buy,” “download,” etc.
• STREAMLINED LINKS – Simplicity is key to an optimized landing page. Keep links to other parts of your website to a minimum, as they distract your visitors and can have a negative impact on conversions.
• CONTINUOUS TESTING – Continue to test different headlines, call to actions, and videos to see what resonates with your visitors. In addition, consider testing two completely different site designs. This comparison can have a big impact to your long-term success.
UX MEETS CRO ON MOBILE
Although you are working with a much smaller screen size, the
same basic web UX principles apply to mobile.
It is still important to start with the objectives of the user. Starting
with a goal-first approach, focus on designing your user flows
before thinking about what you’re crafting on each screen. You
also want to reduce friction in the form of wasted taps, excess ads
or screens, big app sizes, and complicated user flows.
DESIGN IS ESSENTIAL ON MOBILE
Apple’s culture revolves around design excellence. It’s no coinci-
dence that Apple’s apps have surged in popularity -- leading some
to argue that design is the most critical component in building a
successful app. Here are some tips to get you started:
• 1. POSTPONE SIGN-UP TO AVOID ABANDONMENT: App developers
frequently struggle to create interfaces with continuous user
engagement. That’s because consumers are non-committed.
In fact, the average iOS and Android user only downloads three
to five apps per month.
On top of that, 26% of the apps are only opened once and never
used again, and another 48% are opened 10 times or less.
Moreover, mobile users have become weary of apps that auto-
matically post activity to social media. To earn consumer trust,
app developers are moving towards a new trend, requiring users
to sign up after they’ve become dedicated users. Forcing a user
to sign up too early may result in one-star reviews and complete
abandonment.
16
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
• 2. CONSIDER GESTURES: The Apple iOS Human Interface Guidelines
emphasize that an app should allow users to directly manipulate
on-screen objects. By simply allowing users to tap, pinch,
or swipe the screen you will find that your users are more
engaged. It will be easier for users to experience the results of
their actions.
In an iOS app, people experience direct manipulation when
they:
• Rotate or otherwise move the device to affect on-screen
objects
• Use gestures to manipulate on-screen objects
• Can see that their actions have immediate, visible results
• 3. ENFORCE CONSTRAINTS: One of the biggest challenges in mobile
design is undoing the years of designing on the desktop and
the web with its large screen sizes. On the web, it makes total
sense to create long pieces of content or forms with many
fields -- users are sitting at a desk and have the patience (and
computing power) to absorb more information.
On mobile, however, every new page or field can add more
complexity to the app and make for a poor user experience.
Product owners need to approach app design with a ‘mobile-
first’ mindset. In other words, stop thinking like you’re designing
for the web. Mobile presents a completely new user experience.
You need self-control when designing mobile apps. Here’s
what we mean by ‘enforcing constraints’:
Developers can avoid the ‘content and information overload’
problem by stripping the app – its screen and elements – to
the bare essentials. Great mobile design empowers users to
do more with less. For example, your app can allow users to
perform all the necessary features within the app with just the
use of one thumb. Or remove all navigation within your app and
allow the user to use gestures to navigate through your app.
Key Gestures on mobile Devices
17
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
Let’s take the two to-do list apps in the screenshot as examples.
The one of the left appears cluttered and hard to navigate. And
it is not completely clear how a user can mark a task as com-
plete. In addition, there is wasted screen space at the bottom
where the “Home” and “Add” buttons reside.
But the one on the right, Clear, shows all the tasks in a neat
manner and uses the iPhone’s built-in gestures to manipulate
your to-do. Users can pull down on the app to add a new to-do
list or swipe to the left to mark a task as complete.
• 4. RESPECT THE TAP: Every swipe, tap, and pinch that a user takes
will have a lasting impact on UX. Each small gesture should
help a user make progress. Think: lots of return for very minimal
effort.
Sam Shank, CEO of Hotel Tonight, illustrates that booking a hotel
on the app requires only three taps and a swipe – totaling about
eight seconds. Compare those stats to the apps of Priceline (52
taps, 102 seconds) and Hotel.com (40 taps, 109 seconds). It is
no wonder that Hotel Tonight has exploded as a mobile only
company.
A cluttered to do list app (left) vs. a simplified, gestures-first to do list app (right)
18
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
CASE STUDY: TASKRABBIT - WEB VS MOBILE
Let’s take a look at TaskRabbit and
the key differences between their
UX on the web and on mobile.
HOMEPAGE: On the web,
TaskRabbit has a lot more
screen real estate, so they have
decided to promote becoming a
TaskRabbit, posting a task, and
“TaskRabbit for Business.” On
mobile the company focuses on
having users sign up or log in
immediately upon launch.
TASKS SCREEN: In this screen, you
will notice that the web version
has a list of all the different
categories of tasks. On the web,
TaskRabbit can also use more
visual content along with a clear
call to action.
DESKTOP HOMEPAGE
DESKTOP TASKS SCREEN
MOBILE HOMEPAGE
MOBILE TASKS SCREEN
19
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
POST A TASK SCREEN: The web version of this screen is the basic
form that most audiences are accustomed to seeing.
In the mobile version, each section of the form – Location,
Supplies, Deadline, etc. – has its own screen and users have to nav-
igate back to complete the entire form. While this UX adds more
taps and screens to the overall flow, the app varies the gestures of
each screen so users are engaged with each section.
DESKTOP POST A TASK SCREEN MOBILE POST A TASK SCREEN
TECHNICAL DYNAMICS ON THE WEB VS.
MOBILE
Eric Ries in his book The Lean Startup pop-
ularized the term minimum viable product
(MVP) - a development technique in which a
new product or website is created with the few
features necessary to satisfy early adopters.
The final, complete set of features is only
designed and developed after considering
feedback from the product’s initial users.
While the concept of a minimum viable
product has exploded on the web, it is still very
difficult to build minimum viable products on
mobile.
On the web, changes can be implemented and
pushed to production in a matter of minutes.
On mobile, however, developers must account
for each app store’s review process, time for
users to upgrade their apps, and the nuances
of each mobile platform.
To further explain the technical dynamics of
developing and designing for the web versus
mobile, we asked someone who has familiarity
building products in both spaces.
20
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
Steve P. Young led Product Marketing for SmartShoot - an online
marketplace of professional photographers and filmmakers. He
was the lead product manager of SmartShoot’s bid platform
that allowed anyone to submit a photo or video brief and receive
quotes from the network.
He also is an app developer with more than 10 apps in the App
Store and Google Play Store. Furthermore, he hosts a podcast,
Mobile App Chat, where he’s interviewed over 100 app developers
including Tweetbot, Shazam, Facebook Messenger, and many
more.
Q&A WITH STEVE P. YOUNG
OPTIMIZELY: What are the key differences in building on the web versus
mobile?
STEVE: The main difference is speed of development. On the web, there
are many tools that already exist that allow us to quickly build prototypes
and get it in front of users. Some tools are so easy to use that marketers
these days can test their business hypotheses without the help of
engineering.
However, on mobile those tools do not exist. Development requires time to
code, publish on the app stores, and analyze the results.
In addition, on mobile you have to test on different devices with varying
screen sizes and computing power.
And with the recent release of iOS 7, many app developers had to
overhaul their entire design. Can you imagine Chrome or Firefox dictating
how a website should look and operate?
OPTIMIZELY: What about the key differences in user experience?
STEVE: On the web, users are a little more forgiving. When you reach a
404 page, you can just hit the back button and find your way around a
website. Moreover, many websites provide an online chat feature, so users
can immediately get answers in case they are lost.
This is certainly not the case on mobile. If an app isn’t intuitive and its
design confuses the user, there’s no way to get an immediate response
from the developer. Also if an app crashes, users are generally quick to
leave a negative review or simply delete the app.
OPTIMIZELY: From your experience what have you learned about the
mobile space?
STEVE: It’s definitely an exciting space to be in. Things are constantly
changing and you are hearing more success stories from indie developers.
Through my 100+ interviews, here are three key lessons about building
apps:
• Be a feature: Some of the best apps in the world solve one specific
need. On the web, we like our tools to have a lot of features. However,
on mobile we are happy when our apps to just do one thing really
well.
• Don’t distill the web onto mobile: With such a small screen size,
don’t try to distill what you have on the web onto a mobile screen.
The experience should be completely different and you should take
advantage of the touch screens and incorporate taps and swipes into
your interface.
• Build a minimum viable experience/value: It is incredibly hard to
rapidly prototype on mobile, so think about creative ways you could
prototype the experience or value of your app. Don’t start with
interface of your app, start with the experience you’d like the user to
have with your app.
21
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
CRO ON MOBILE: ACQUISITION
04
Thanks to Optimizely, A/B testing and optimization on the web
requires little technical assistance and presents few barriers.
Mobile is a different story.
Marketers on the web have easier access to data and with the help
of tools can make changes on the fly without engineering help.
On mobile, marketers and product managers are more reliant on
engineering teams and do not have the freedom to make changes
like they do on the web.
22
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
Fortunately, there are some non-technical “hacks” that marketers
and product managers can do to help optimize adoption and
engagement with their apps.
Before we do that let’s take a look at a typical conversion funnel
for an app:
OPTIMIZING FOR APP STORES
App Store Optimization (ASO) is the equivalent of search engine
optimization (SEO) for mobile apps. The goal is to make it easy
for users to find your app in the App Store through careful selec-
tion of keywords that specific user intents.
The nice thing about ASO is that it requires very little technical
knowledge and often times can be optimized without the need of
an engineering team. However, it’s important to note that ASO is
not a real-time A/B test. You need to compare data from before
and after the change.
In fact, when performing ASO, you are analyzing past results with
current results to see if the change you made had an impact.
App Store Optimization tests can involve:
• APP ICON: It’s important to test your icon even before releasing
the app. You can buy some ads on mobile advertising networks
such as Admob or online advertising through Google Adwords
and Facebook. What you want to find out is which app icon is
getting the highest click through rate.
• APP NAME: According to Nielsen, 63% of Android and iOS users
have utilized search to discover new apps. That’s why it’s
important to optimize your app name with highly sough after
keyword phrases. ASO tools such as Straply and Sensor Tower
are great places to start. In the online marketing world, having
keywords in the title of your web page helps with SEO. In the
app store, the same principle applies with your app name.
Awareness
App Store Search
Download
Open App
Registration/Key Action
Usage &Retention
3. HTTP://WWW.SLIDESHARE.NET/VICTORI98PT/STATE-OF-MOBILE-Q32011-BY-NIELSEN%20
3
23
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
• KEYWORDS: On the web, keyword meta-data no longer influences
search rankings. An app’s keyword data, however, actually has an
impact on its search rank. Along with your app’s name, having
relevant, high-traffic words in your keyword meta-data can help
improve your apps search ranking.
• SCREENSHOTS: While the term may imply a screen capture of
your app, you should think of your screenshots as a banner
advertisement. Both the Google Play and Apple App Store
prominently display screenshots rather than an app’s description.
Besides the app icon, your app’s screenshot is the strongest
visual representation of your app, so you want to draw the user
to download it. Again, just like the app icon, you can test using
traditional banner ads to see which screenshot yields the highest
click throughs.
OPTIMIZING FOR REGISTRATION/KEY ACTIONS
A recent study from Compuware shows that 80-90% of apps are
deleted after only one use. Mobile customers are unforgiving. If you
do not make an immediate first impression, it’s likely that your app
will be deleted.
Without an optimized onboarding experience, you are not only
losing users, but also an opportunity to monetize your app.
Let’s unpack this a little more. Unlike the web where we have mul-
tiple tabs open and going to a website takes a matter of seconds,
on mobile we have to go through the trouble of finding your app,
downloading it, and then remembering to actually open it.
The user has gone through three steps just to see what your app
has to offer, so retaining as much of that 90% as possible is critical.
How do you ensure you make the best first impression possible?
Make sure your app does not make some of these common mis-
takes we see from app developers during the onboarding process:
REGISTER FIRST, TRY LATER: Apps that require registration before
using it, lose up to 56 p ercent of its users. Mobile users want to
see and engage with your app before deciding to join.
LENGTHY REGISTRATIONS: The mobile screen is small enough, so
requiring a lot of information from your users will have a negative
effect on their usage. You may want to use multiple screens to
allow the user to enter in bite-size information. For example,
having three screens with 2-3 fields each is much better than
having one screen with 9 fields.
LONG LOAD TIME: Mobile customers have a low tolerance for unsta-
ble and buggy apps. Apple has emphasized design and UX within
its default apps, so users have come to expect elegant designs and
intuitive user flows.
WITHOUT AN OPTIMIZED ONBOARDING EXPERIENCE, YOU ARE NOT ONLY LOSING USERS, BUT ALSO AN OPPORTUNITY TO MONETIZE YOUR APP.
24
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
CRO ON MOBILE: RETENTION
05
As outlined in this paper, mobile app optimization can yield huge
benefits for app developers. At the same time the reality of today’s
marketplace is that users are interacting with brands and products
across desktop web and native mobile apps.
However given the technical constraints of executing an A/B test or
a personalization campaign within a native mobile app, many app
developers will simply not test at all. This leaves valuable valuable
opportunities on the table for better conversion rates, higher
engagement, and increased revenue.
25
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
That’s why Optimizely is investing in product enhancements that
enable customers to optimize end-to-end experiences across all
devices, websites, and apps their visitors are using. Our goal is to
reduce the friction required to run in-app experiments that lead to
optimized user experiences that achieve the goals app developers
have set out to accomplish.
TECHNICAL CONSIDERATIONS
Before deciding on a mobile app optimization strategy, it’s import-
ant to consider the technical challenges mentioned above. There is
no mobile app equivalent to the DOM (Document Object Model),
which can be manipulated by JavaScript. This introduces the
complexity of requiring engineering support to test changes that
would be much more straightforward to implement on the web.
Moreover companies are becoming increasingly reliant upon third
party SDKs for analytics, monetization, and optimization, all of
which require technical knowledge.
Here are some key technical considerations. While not specific to
A/B testing and optimization, they are important to keep in mind:
APP STORE REVIEW: Each app store has its own review cycle. On
Google Play it could be a matter of hours, where on the Apple App
Store, it could be anywhere from 5-7 days. This delay makes it hard
for developers to push changes. Control remains squarely in the
hands of the platforms. While tools like Parse can make it possible
to release updates quicker, it is still difficult for a non-technical
marketer or product owner to have control over an A/B test or
feature change.
USERS MUST UPGRADE APP: The landscape changed a bit when Apple
introduced automatic updates with iOS7, but it is still no guarantee
that users have this featured turned on. Some power users have
actually turned this feature off to save battery life. Even with the
feature turned on, it is hard to predict when the update will reach
every user.
USERS MAY NOT BE ONLINE: We have all had connection problems
here and there. While this may not be a huge concern, it is still
something to consider when releasing an update. You can’t always
guarantee that users will have Internet connection, so you need to
ensure that your app has the proper messaging in place to address
offline issues. Users can always upgrade when they’re back online,
but in the interim, they may feel frustrated. This one moment of
frustration has the potential to wreak havoc on user engagement.
As a marketer, you may think -- ‘well, they’ll hop online later.’
Keep in mind, however, that consumers want instant results and
gratification.
PRODUCT CONSIDERATIONS
Technical considerations aside, we don’t see much of a difference
between A/B testing and optimization on the web vs. mobile. As
mentioned above, Optimizely is expanding its already market-lead-
ing web optimization platform to support mobile app A/B testing
and optimization.
We built Optimizely for mobile apps using the same philosophy as
we did when building our web product. Of course, it’s also import-
ant to recognize and account for the fact that there are tactical
differences when optimizing across devices.
26
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
When choosing your native mobile app optimization strategy, it’s
important to keep these tenets in mind:
CROSS-PLATFORM. Users are the same across web and mobile, but
their behaviors and motivations are different. That’s why it is
important to choose a native app optimization framework that
works across both the web and mobile devices. This framework
should allow you to craft different experiences to cater to
device-specific needs, while still carrying over information between
web and mobile to make sure the experiences is consistent no
matter where the user is engaging.
FAST. The app development space is one of the fastest-paced that
exists today. It’s never been more important to build apps that
react to the needs of your users and to the market in general.
Any app optimization platform you choose should take this into
account. It should be easy to roll out changes without a code push.
It should minimize dependence on developer time as much as
possible.
ANALYTICS AND INSIGHTS. Experimentation and optimization is no
good without concrete data to take action on. When choosing an
optimization platform, it’s critical to pick one that will give you the
insights an analysis you need to make decisions using the data
that’s being generated.
SPEED, STABILITY, AND SECURITY. Mobile apps are different than
websites in that they’re ultimately end-to-end programs that users
are installing on their devices. Once an app is installed, it needs to
be able to function and scale independently. It’s important that any
solution you choose does not compromise the fidelity of your app,
because that will ultimately reflect on your app’s rating and, by
extension, its future success.
INTEGRATES YOUR ECOSYSTEM. If your app is like most apps, it’s
already leveraging third party SDKs for things like analytics and
crash reporting. For something as fundamental as A/B testing
and optimization, it’s critical to choose a platform that integrates
with the most commonly used mobile app SDKs out there. Your
team shouldn’t have to relearn a new set of tools or define a new
workflow just to optimize your app. Barriers like that can prevent
you from being able to implement a culture of experimentation
within your organization.
USERS ARE THE SAME ACROSS WEB AND MOBILE, BUT THEIR BEHAVIORS AND MOTIVATIONS ARE DIFFERENT.
27
A BLUEPRINT FOR A/B TESTING AND OPTIMIZATION ON NATIVE MOBILE APPS
CONCLUSION
It’s never been more important to deliver compelling experiences
that delight and engage users on mobile devices. For years,
marketers and product managers have embraced a culture of
experimentation and testing, and as a result, web experiences are
becoming increasingly effective at improving users’ experiences
while also helping companies accomplish their goals.
However this culture of experimentation has not yet made it’s way
into the realm of mobile app development. This is largely due to
the reality that implementing a mobile app optimization strategy
typically requires a prohibitively high level of involvement from
engineering teams who could otherwise be working on feature
development.
In developing its mobile app optimization framework, Optimizely’s
goal is to reduce the necessity to involve engineering teams to
the extent that it can be adopted in the mobile app community as
heavily as it has been adopted in the web community.
Learn more about Optimizely’s mobile app optimization platform
at http://optimizely.com/mobile.
ABOUT OPTIMIZELY
Optimizely is the world’s leading experience optimization platform,
providing A/B testing, multivariate testing, and personalization for
individuals and organizations with and without technical expertise.
The platform’s ease of use empowers organizations to conceive
of and run experiments that help them make better data-driven
decisions. With targeting and segmentation using powerful real-
time data, Optimizely meets the diverse needs of any business
looking to deliver unique experiences to their visitors.
OPTIMIZELY.COM/MOBILE
Email: [email protected]
Phone: 1-800-252-9480
Facebook: facebook.com/optimizely
Twitter: twitter.com/optimizely
Amsterdam OfficeNes 761012 KE Amsterdam
San Francisco Office631 Howard Street, Suite 100San Francisco, CA 94105
To learn more about Optimizely for Mobile Apps, visit
ABOUT THIS GUIDE A Blueprint for A/B Testing and Optimization on Native Mobile Apps
Written By:Sean OliverSenior Product Marketing Manager, Optimizely@SeanOliver
Designed By: Jon Saquing@jsaq
Thank you to Ural Cebeci, Steve Young, and Ritika Puri.