To ∞ (~65K) and beyond!Sebastiano Gottardo, Android Engineer +SebastianoGottardo - @rotxed

1

Context

2

Context

• OSS, community, libraries, frameworksThe Android open-source community is enormous, and brings to the table thousands of libraries and frameworks.

• ✓ Faster development, contributions and improvements These tools allows you to build better apps quicker. In addition, contributions by other users lead to improvements.

• ✗ Using too many can lead to heavy APK, long loading times, bugs There are tradeoffs.

3

4

65536

Dalvik Executable (DEX)

• Container of the compiled source code .java files are compiled into .class files, then packed into a .dex file

5

Where is the limit

• Not runtime, not DEX, but Dalvik’s instruction set!

6

https://goo.gl/ajlxwX

Crossing the limit

• Unable to build an APK

• Panic

• Console output

7

Older build system Newer build system

Is this going to change?

8

Context

• No(t likely)Lose retro-compatibility!

• ART (Android Runtime)The new runtime used starting with Lollipop provides a way around.

• What to do then?There are different solutions/approaches.

9

Approaches

• MultiDexGoogle’s own solution.

• Secondary DEX Get your hands dirty.

• Shrinking / Reasoning ProGuard is your friend, and so is your brain.

10

MultiDex

11

MultiDex

• Google’s way around this limitation aka what’s in the docs

• Still not the “recommended” approach More on that later on.

• Different working mechanisms depending on platformDalvik ≠ ART.

12

MultiDex: Dalvik VS ART

13

ARTDalvik

MultiDex with ART

• App’s source code is compiled and packed into multiple DEX files Ultimately packed in the APK

• At install time, the OS compiles the code of the DEX files into a single OAT file (ELF)AOT compilation instead of JIT compilation

14

MultiDex: Dalvik VS ART

15

ARTDalvik

MultiDex with Dalvik

• App’s source code is compiled and packed into a single DEX fileUltimately packed in the APK

• Dalvik is unaware of DEX files other than the main oneclasses.dex

• Multiple DEX files must be loaded at runtimeCan be troublesome (a.k.a. sucks)

16

MultiDex - Setup

17

Setup 1/3

1. Gradle-wise setup

18

android { compileSdkVersion 23 buildToolsVersion "23.0.2" defaultConfig { ... minSdkVersion 15 targetSdkVersion 23 ... // Enabling multidex support. multiDexEnabled true } ...}

Setup 2/3

2. Include the MultiDex support library

19

dependencies { compile 'com.android.support:multidex:1.0.0'}

Setup 3/3

3. Enable MultiDex in your code

20

<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.android.multidex.myapplication"> <application ... android:name="android.support.multidex.MultiDexApplication"> ... </application></manifest>

a

public class YourAwesomeApplication extends MultiDexApplication { ...

}b

@Overrideprotected void attachBaseContext(Context base) { super.attachBaseContext(base); MultiDex.install(this); }

c

Advantages

• Well integrated with the build chainNo magic involved.

• Seamless support across platform versionsBeing a support library. One code, one love!

• Extremely easy to enableLiterally takes 3 minutes.

21

Disadvantages

• Limited in choosing what goes whereEsp. true for libraries that sport native components.

• MultiDex on Dalvik is unreliableFor 14 < API < 21 crashes may occur.

• Secondary DEX files are loaded at runtimeStartup time is increased! (unless you defer installation)

• Build times increaseVariants may help (dev — min 21, speeds things up)

22

Secondary DEX

23

Secondary DEX

• Historically, the first workaroundA blog post by Fred Chung, a Googler at the time (2007)

• Relied on customizing the build chain Some Ant magic

• Used an interface-based approach

24

How does it work

• Put one or more libraries into a separate DEX file Limited to .jar files, .aar not supported

• At runtime, load the secondary DEX on a custom ClassLoaderDifferent from the default ClassLoader!

• Invoke the libraries’ methodsInterface, Reflection

25

Quick example w/ Reflection

26

twitter = new TwitterFactory().getInstance(); twitter.setOAuthConsumer(TWITTER_APP_KEY, TWITTER_APP_SECRET); twitter.setOAuthAccessToken(new AccessToken(token, tokenSecret));

Quick example w/ Reflection

27

...twitter = twitterManager.initializeTwitter(TWITTER_APP_KEY, TWITTER_APP_SECRET, token, tokenSecret); ...

public Object initializeTwitter(String TWITTER_APP_KEY, String TWITTER_APP_SECRET, String token, String tokenSecret) { try { Object twitterFactoryInstance = getClassCached(twitterFactoryFQN).newInstance(); Method getInstanceMethod = getMethodCached(twitterFactoryFQN, "getInstance", null); Object twitterInstance = getInstanceMethod.invoke(twitterFactoryInstance, null); Method setOAuthConsumerMethod = getMethodCached(twitterFQN, "setOAuthConsumer", String.class, String.class); setOAuthConsumerMethod.invoke(twitterInstance, TWITTER_APP_KEY, TWITTER_APP_SECRET); Class atClass = getClassCached(accessTokenFQN); Object atInstance = atClass.getDeclaredConstructor(new Class[] {String.class, String.class}).newInstance(token, tokenSecret); Method setOAuthAccessTokenMethod = getMethodCached(twitterFQN, "setOAuthAccessToken", atClass); setOAuthAccessTokenMethod.invoke(twitterInstance, atInstance); return twitterInstance; } catch (Exception e) { e.printStackTrace(); } return null;}

Advantages

• Choose exactly what goes whereTake that, MultiDex

• Choose exactly when to load secondary DEXSo not to impact cold start times

28

Disadvantages

• Quite hard to setupCopy DEX to the appropriate dir, create ClassLoader instance

• Quite hard to work withEsp. if you rely on Reflection

• Versioning! How to handle updates?

29

Improvements

• Same approach as Google’s MultiDexPatches the Class Loader so to look for other DEX files

• Specify the dependencies as ‘provided’So to be resolved at compile/build time, stripped in the final APK

• Best of two worlds (kind of)Removes the need for ifaces, Reflection, still a bit cumbersome

30

Shrinking / Reasoning

31

— Google MultiDex online documentation

Before configuring your app to enable use of 65K or more method references, you should take steps to reduce the total number of references called by your app code, including methods defined by your app code or included libraries.

32

ProGuard in pills

• Shrink, optimize, obfuscate

• Works both for your code and for third-party libraries

• Has an impact on APK size, methods number, performance and security

33

ProGuard ProTips

• Get familiar with writing appropriate rulesThe stricter, the better

34

-keep class !android.support.v7.internal.view.menu.**,android.support.** {*;}

• What’s wrong with the following keep rule?

-keep class com.google.** { *; }

-keepnames class com.google.** { *; }

-keepclassmembernames class * { @com.google.android.gms.common.annotation.KeepName *; }

• Adopts an exclusion-based approach

ProGuard ProTips

35

-keep class !android.support.v7.internal.view.menu.**,android.support.** {*;}

-keep class com.google.** { *; }

-keepnames class com.google.** { *; }

-keepclassmembernames class * { @com.google.android.gms.common.annotation.KeepName *; }

• Adopts an exclusion-based approach

• Get familiar with writing appropriate rulesThe stricter, the better

• What’s wrong with the following keep rule?

ProGuard: default rules

36

android { ... buildTypes { release { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' } } }

• ProGuard already includes common rules for common cases~/sdk/tools/proguard/lib/proguard-android.txt

-keepclasseswithmembernames class * { native <methods>; }

DexGuard: beefier, for beefier companies

37

• ProGuard’s bigger (and pricier) brother

• Advanced features: res/strings/.so encryption, stronger obfuscation,tamper detection, its own multidex! and more

• Pluggable encryption! I mean, seriously, that exclamation mark has a reason

• Default rules are much more extensive

ProGuard and third-party libs

38

• Many provide the ProGuard statement (a.k.a RTFM)e.g., Square, Facebook, …

• Pay attention to ‘keep-all’ rules!especially for libraries with many methods

… wait, how do I count libraries methods?

39

… how do I count methods at all?!

inloop’s “APK method count”

40

inloop’s “APK method count”

41

inloop’s “APK method count”

• Simple website with drag-and-drop interface

• Drop the APK and see the (locally) computed countNo APKs uploaded :+1:

• http://inloop.github.io/apk-method-count/

42

✓ Easy, immediate, private✗ Build every time

ClassyShark

• Useful Java tool to inspect APKsWritten by Boris Farber (Googler)

• Inspect number of classes, members, methods, dependencies, …Acts as a simple decompiler, too

43

ClassyShark

44

ClassyShark

45

ClassyShark

• Useful Java tool to inspect APKsWritten by Boris Farber (Googler)

• Inspect number of classes, members, methods, dependencies, …Acts as a simple decompiler, too

46

✓ Easy, plenty of info, multi-purpose✗ Build every time, more oriented to agnostic debugging

dexcount-gradle-plugin

• Gradle plugin, developed by KeepSafe

• See the current methods count every time you buildJust open the Gradle console

• Outputs per-package information in a ${variant}.txt file

47

dexcount-gradle-plugin

48

dexcount-gradle-plugin

49

dexcount-gradle-plugin

• Gradle plugin, developed by KeepSafe

• See the current methods count every time you buildJust open the Gradle console

• Outputs per-package information in a ${variant}.txt file

50

✓ Keep the count controlled at a glance✗ Cumbersome to see single deps count

MethodsCount

• Developed by Sebastiano Gottardo, Nicola Miotto, Dario Marcato

• Allows to see methods and dependencies for libraries beforehand

• Compare different libraries in terms of “lightweight-ness”

51

MethodsCount

52

• More than 20000 libraries and dependenciesWhat’s missing is calculated upon the first request

• For each entry

1. dex/jar size

2. direct count + dependencies count

3. multiple versions (monitor changes over time) + charts

MethodsCount

53

MethodsCount

54

MethodsCount

55

MethodsCount - AS plugin

56

• Parses `build.gradle` and directly shows the count

• Convenient when you drop a dependency right away

MethodsCount - AS plugin

57

• Parses `build.gradle` and directly shows the count

• Convenient when you drop a dependency right away

MethodsCount

58

✓ See the count before having to hit the limit✓ Compare different libraries Worth having 4000 methods instead of 900?✓ Analyze dependencies No, Guava is not a good excuse to have 15000+ methods✓ Integrated in AS, thanks to the plugin

✗ The count is before ProGuard → Upper bound!

Bottom line

59

Bottom line

• The limit is here, at least for some timeLearn to deal with it

• The recommended way is “Do NOT cross the limit!”And use ProGuard. Your nerves will thank you later!

• If you do need to cross, there are solutions (with drawbacks)MultiDex, secondary DEX

60

Routine

1. PM shows interest in a new library:eyes_roll:

2. With dex-gradle-plugin, check the current count

3. Check MethodsCount to see how many methods, size and deps

4. Evaluate whether fits a secondary DEX

5. Consider alternatives (again, MethodsCount)

61

Useful resources

• Timothy Mellor: https://github.com/tmelz/multidex_notes

• Jeroen Mols: http://jeroenmols.com/blog/2016/05/06/methodcount/

62

@rotxed+SebastianoGottardo

Follow me