optimizely workshop: mobile walkthrough

Mobile Walkthrough: From 0 to Experiment

Pamela Ongchin Mobile Solutions Architect

An Optimizely Online Workshop

A few tips:This webinar will be recorded.

We strongly encourage you to take notes!

Feel free to ask questions at any time.

There will be a live Q&A at the end of this session.

1. What are the biggest challenges in mobile app development? How can I address these with experiments?

2. How do I set up an Optimizely mobile experiment?

3. How does A/B testing integrate with my development process?

4. What are best practices when designing experiments?

After today’s workshop, you should be able to answer the following questions

But first, tell us a little more about you!

What department do you work in?

MarketingEngineering

Product

poll

What is the most challenging issue that you face in your mobile app?

question

On average, apps lose 80% of their new usersSource: Andrew Chen’s blog, http://andrewchen.co/

Some of the top apps in 2014

The top apps follow the same patternSource: Andrew Chen’s blog, http://andrewchen.co/

It’s difficult to keep users engaged

80% of new users drop off validate new features

It’s difficult to be agile …

high technical demands slow release process

one-star reviews

Common pain points:

• Measure the effectiveness of new features.

• Make data-driven decisions on what to build next based on test results.

• Develop experiences that retain users and extend lifetime value.

A/B testing can help you become more agile

The stylist and customer were 43% more likely to connect and ship a trunk of clothing, a key step in Trunk Club’s customer conversion funnel

Due to higher quality of leads from the app, stylist productivity improved 90%

No credit card required Credit card required

Okay, so A/B testing can help me be more agile.

What should I test?

What is the one key business metric that matters most for your app?

RevenueSignups

Engagement

poll

Key Metrics for Mobile Apps

Key Metrics for Mobile Apps

Opportunities to A/B test

1. Onboarding

2. Product detail screens

3. Promotions and upsell

4. Homescreen

5. Registration flow and checkout flow

Top 5 areas Optimizely mobile customers are testing

1. Onboarding (show value, engagement)



4. Homescreen



Secret Escapes Optimizes Acquisition

• Goal: Looking to optimize LTV, sign up rates, and justify mobile ad spend

• Experiment: Should the app require users to sign up in order to view travel deals?

Experiment Setup

Version 1 Version 2

Sign up gate with mandatory sign in on first user experience

Sign up not required, added a “skip”

button to close the sign up form

Results

• Defying all expectations, a mandatory signup gate was by far the optimal experience

• Sign up rates increased 200%, LTV average increased, and LTV:CAC ratio was positive




4. Homescreen



Add to cart (or bag)




4. Homescreen



In-app message: Upsell




4. Homescreen



Facebook layout test




4. Homescreen



Checkout Flow

1. Onboarding (first value, engagement)



4. Homescreen



Now, let’s create a mobile experiment.

Make the Variations What to test?

Set Audiences Who sees the experiment?

Set Traffic Allocation How many visitors are included? How are they split?

1

2

3

Set Goals Why are you testing? How will you measure success?

4

Preview & Test Does it work the way you want to?

4

5 Key Steps

5




1

2

3


4


5

5 Key Steps




1

2

3


4


4

5 Key Steps

5

A/B Testing Timeline

Brainstorm


Brainstorm Prioritize & Design


Brainstorm Prioritize & Design


Implement

Brainstorm Prioritize & Design QA & Approve


Implement

Brainstorm Prioritize & Design QA & Approve Run


Implement

Brainstorm Prioritize & Design QA & Approve Run Analyze &

Review


Implement

• Know what tests you want to run at the beginning of your sprint.

• Have a developer involved in the process.

• Use phased rollouts for new features.

• Use live variables and code blocks.

• Integrate Optimizely with your analytics platforms.

Best Practices

Pop Quiz!

If the Optimizely SDK is installed, you can make visual changes to your app in between

releases without additional code.

TrueFalse

poll

If the Optimizely SDK is installed, you can make visual changes to your app in between

releases without additional code.

True

poll

If I want to test an entirely new feature in my app, I need to get my developer involved.

TrueFalse

poll

If I want to test an entirely new feature in my app, I need to get my developer involved.

True

poll

Use A/B to validate new features.

A/B testing is a huge opportunity to improve metrics in your app.

Once your SDK is installed (by your developer), you’ll be able to make visual changes without additional code. Testing new features requires developer involvement.

Final Takeaways

Q&A

Have a question? Type it into the chat box!