Test ButlerReliable Android Testing, at Your Service

Drew Hannay (@drewhannay)

Android Testing @ LinkedIn● ~5000 instrumentation tests

● 2 runs per commit (pre-commit + post-commit)

● ~50 commits per day

● ~100 nightly runs to detect flaky tests

● 5000 * ((2 * 50) + 100) =

~1 million tests run per day

● Emulator flakiness

○ Animations can cause Espresso flakiness

○ CPU can randomly go to sleep

○ WiFi can randomly disconnect

○ Lock screen can appear and block test runs

○ Other apps can crash/ANR in the background and

block test runs

● Most of these happen < 1% of the time

○ But with 1 million tests run per day, that’s not

insignificant

● Tests shouldn’t fail due to issues with the

testing infrastructure

Testing Environment

Controlling the environment from tests● Turn WiFi on/off

● Change locales

● Set screen orientation

● Turn GPS on/off

Extra permissions● Many of these problems are fixable via reflection hacks

● Often requires special Android permissions

○ e.g. SET_ANIMATION_SCALE

● Painful at best, dangerous at worst

○ Add permissions and remember to remove before releasing (bad!)

○ Add permissions only to the debug manifest (better)

○ Either way gives your app different behavior in dev than in production (bad!)

● Some permissions are “development” permissions

○ These need to be granted with adb before they can be used

Introducing Test Butler● Two-part project: App & Library

● Library communicates with Test Butler app via RPC

● Test Butler app has permission to change

device settings

App apk Test apk

Test Butler Library

Test Butler apk Emulator

Test Butler: Permissions● Test Butler requests lots of permissions so your app doesn’t need to

● Signed with the system keystore for the AOSP emulators

● All signature-level permissions are granted automatically

○ No need to grant permissions with adb!

● This also means it can access permissions that can’t normally be used by

non-system apps

Test Butler: Custom IActivityController● Holds signature permission SET_ACTIVITY_WATCHER● Allows it to set a custom implementation of IActivityController

Test Butler: Custom IActivityController● No more system crash or ANR dialogs breaking tests!

● Caveat:

○ isUserAMonkey will always return true while using Test Butler

● Workaround:

○ Write a custom isUserAMonkey method that returns false

while running instrumentation tests

How do I use it?

How do I use it?

What’s next?● Add sample scripts to download & install the Test Butler app on emulators

● Add additional emulator controls (e.g. toggle airplane mode)

● Stabilize the rest of the Android test pipeline (stay tuned!)

Questions

References● Test Butler Github Repo

● Test Butler Announcement Blog

● Going Green: Cleaning up the Toxic Mobile Environment