A/B Testing in Android

Nir Hartmann

Drippler

Droidcon Tel-Aviv 2014

Why do we need A/B Testing?

• Tests takes the guesswork out

• Enables data-backed decisions

• Enhances engagement and retention

Road map

• What is an A/B test ?

• Segmentation

• Multiple Experiments

What is an A/B Test ?

• Case study – onboarding screen

Define the test

• Hypothesis – The layout with the Google+ button at the left will increase the number of total registered users.

• Goal – A registered user (the user can skip registration).

• View event

- Login fragment onCreate().

- Login activity onCreate().

• Variables – Facebook button position (left or right).

• Participants – New users.

Amazon A/B Testing SDK

• Very customizable, you can do just about anything as long as you know what it is you want to do

• It’s free

• Drippler created an open source library that simplify the process

Setup the A/B test

• Setup identifier

– https://developer.amazon.com/al/index.html

Setup the A/B test

• Create a project

Setup the A/B test

• Create the test

Dive into the code

• https://github.com/Drippler/ABTester

public class MyApplication extends Application {

@Overridepublic void onCreate() {

super.onCreate();ABTester.init(getApplicationContext(),

"my_public_key", "my_private_key");

}}

private void initLoginActivityTest() {

try {

ABTester.syncPreFetch(

TimeUnit.SECONDS.toMillis(15),

new ABTest("Login page test", false, "Facebook is

first”) );

} catch (TimeoutException e) {

// Couldn't reach amazon servers

}

}

Fetch the test

Login Fragment

@Override

public void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

setRetainInstance(true);

ABTester.recordEvent(

"Login fragment shown",

false);

}

Login Fragment@Override

public View onCreateView(LayoutInflater inflater,

ViewGroup container, Bundle savedInstanceState) {

boolean shouldShowFacebookFirst =

ABTester.getBoolean("Login page test", "Facebook is

first", false);

if (shouldShowFacebookFirst)

return inflater.inflate(R.layout.facebook_first, null);

else

return inflater.inflate(R.layout.google_first, null);

}

Report goal event

public void onUserLoggedIn() {ABTester.recordEvent("Sign in", false);

}

@Overrideprotected void onPause() {super.onPause();/* Submit the events that were previously stored locally. * asynchronously * call it in the onPause() method of an activity */ABTester.submitEvents();

}

Analyze the results

Variation Views Conversions Conversion rate

Change

Google+first

1064 320 30.08%

Facebook first

1043 250 23.97% -20.30%

Dice experiment

Goal: maximize the amount of 3’s we get in a 200 dice roll

Dice experiment

Hypothesis: wearing a hat will increase the chance to roll a 3

Analyze the results

Variation Views Conversions Conversion rate

Change

Hat off 200 31 15.50%

Hat on 200 38 19.00% +22.58%

Conversion is never a single number.

Confidence level

• Measure the reliability of an estimate

– The confidence levels help us understand if the results are different merely by chance or by reason

• 95% confidence level is considered good

Analyze the results

Variation Views Conversions Conversion rate

Change

Google First

1064 320 30.08% ±2.32%

Facebook First

1043 250 23.97% ±2.18%

-20.30%

• Confidence level of 99%

Analyze the results

https://developer.amazon.com/public/apis/manage/ab-testing/doc/math-behind-ab-testing

Choose the best variation

• Launch

– Choose the winning variation

– Control the percentage of customers that receive a new feature

Road map

• What is an A/B test ?

• Segmentation

• Multiple Experiments

http://visualwebsiteoptimizer.com/split-testing-blog/wp-content/uploads/2010/10/2010.09.10.ab_.png

Segmentation

Define the test

• Hypothesis – Coloring the “Rate” button, will increase the button’s click rate

• Goal – Click event on the “Rate” button

• View event – RateUsDialogFragment show();

• Variables – “Rate” button color

• Participants – All users

Create the test

Rate us DialogFragment

public class RateUsDialog extends DialogFragment {

public static void show(FragmentManager fm,

int color) {

RateUsDialog rateUs = new RateUsDialog();

Bundle extras = new Bundle();

extras.putInt(“color”, color);

rateUs.setArguments(extras);

rateUs.show(fm, “my tag”);

ABTester.recordEvent("Rate us dialog shown", false);

}

Rate us DialogFragment

@Override

public Dialog onCreateDialog(Bundle

savedInstanceState) {

int color = getArguments().getInt("color");

return createColoredDialog(color);

}

Rate us DialogFragment

private Dialog createColoredDialog(int color) {

...

.setPositiveButton("Rate", new OnClickListener() {

@Override

public void onClick(DialogInterface dialog, intwhich) {

ABTester.recordEvent("Rate button click", false);

}

});

return myDialog;

}

Rate us test1) Asynchronously prefetching SplashActivity

Default timeout is 60 seconds, and can be overriden by

using preFetch(long timeout, ABTest... Test)

ABTester.preFetch(

new ABTest("Rate us test", false, "Rate button color")

);

Rate us test2) Show the dialog

String fetchedColor = ABTester.getString(

"Rate us test", "Rate button color", "#F5F5F5");

int color = Color.parseColor(fetchedColor);

RateUsDialog.show(getFragmentManager(), color);

3) Submitting the results onPause()

ABTester.submitEvents();

Analyze the results

Variation Views Conversions Conversion rate Change Confidence

Control (white)

865 234 27.05%± 1.51%

Variation A (green)

904 250 27.65%± 1.49%

-0.2%

Variation B (red) 830 230

27.71%± 1.55% +2.4% 51%

What can I do with these results?

Segmentation

Variation Views Conversions Conversion rate Change Confidence

Control (white)

432 92 21.30% ± 1.97%

Variation A (green)

464 165 35.56% ± 2.22% +66.9% 98.7%

Variation B (red)

420 120 28.57% ± 2.20% +34.1%

Variation Views Conversions Conversion rate Change Confidence

Control (white)

433 142 32.79% ± 2.26% +22.2% 97.1%

Variation A (green)

440 85 19.32% ± 1.88% -27.9%

Variation B (red)

410 110 26.83% ± 2.19% -18.8%

Under 40

Over 40

Define the test

• Hypothesis – Coloring the “Rate” button, will increase the button click rate

• Goal – Click event on the “Rate” button

• View event – RateUsDialogFragment show();

• Variables – “Rate” button color

• Participants – All users

Define the test

• Hypothesis – Coloring the “Rate” button, will increase the button click rate

• Goal – Click event on the “Rate” button

• View event – RateUsDialogFragment show();

• Variables – “Rate” button color

• Participants – Age specific tests

Setup the A/B test

• Create a segment

Create the segment

Assign the segment

• In your code before the fetch

– ABTester.addDimension(”age", myAge);

• Dimension will be remembered forever

• ABTester library will automatically add a “percentile” dimension

– ABTester.addDimension(“percentile”,

new Random().nextInt(100));

Road map

• What is an A/B test ?

• Segmentation

• Multiple Experiments

http://unbounce.com/a-b-testing/shocking-results/

Multiple Experiments

Multiple Experiments

Multiple Experiments

• Serial tests– Run the tests one after the other, without the

need to redistribute your app

– More accurate but takes more time

ABTester.preFetch(

new ABTest("Rate us test", false, "Rate button

color", "Rate actionbar icon")

);

Multiple Experiments

• Parallel tests– Run the tests together, increasing the ‘noise’ for

dependent tests – Faster

ABTester.preFetch(new ABTest("Rate us button", false,

"Rate button color”)new ABTest("Rate us actionbar", false,

"Rate button icon”) );

Not a replacement for common sense

Thanks, any questions ?

Don’t overthink it, use Drippler’s A/B test library

https://github.com/Drippler/ABTester

Nir Hartmann, nhartmann@drippler.comDrippler

Droidcon Tel-Aviv 2014