[ieee 2012 ieee fifth international conference on software testing, verification and validation...

Testing Conformance of Life Cycle Dependent Properties of Mobile Applications

Dominik Franke∗ and Stefan Kowalewski†Embedded Software Laboratory

Ahornstraße 5552074 Aachen, Germany

∗[email protected]†[email protected]

Carsten Weise

IVU Traffic Technologies AGBorchersstraße 20

52072 Aachen, Germanyhttp://www.ivu.com

[email protected]

Nath Prakobkosol

King Mongkut’s Universityof Technology

10800 Bangkok, [email protected]

Abstract—Operating systems of modern mobile devices, likee.g. iOS and Android, require the applications to conform toa life cycle model, to ensure the functional correctness of theapplication and its data integrity over exceptional behavioras e.g. out-swapping of the application. The applicationslife cycle events are triggered asynchronously by the systemand depend on the environment. In order to test life cycledependent properties of the applications, we define a unittesting based approach that uses life cycle callback-methods.The method identifies life cycle dependent properties in theapplication specification, and derives assertion-based test casesfor validating the conformance of the properties. Life cycletriggers are used in the test case execution. The paper describesto which application features the approach can be applied, andthe limitations of the approach. A case study demonstrates howto apply our approach to state-of-the-art mobile platforms,using Android 2.2 as an example.

Keywords-testing; unit test; mobile; Android; Android Acti-vity; application life cycle; process life cycle;

I. INTRODUCTION

This paper deals with testing the conformance of a mobile

application to the application life cycle. The application lifecycle refers to the process-related states of an application

– i.e. running, paused, etc. – and the allowed transitions

between these states.

In traditional desktop operating system, the application

life cycle is completely transparent to the application. The

operating system takes care of the states of the life cycle,

and ensures the correct behaviour of the application under all

circumstances. This is different in modern mobile operating

system like e.g. iOS or Android, or mobile frameworks like

e.g. J2ME. For sake of efficiency, and due to the fact that

resources are scarce, these operating systems cannot save the

complete state of an application whenever state changes in

the life cycle occur. Instead, the application itself must take

care that no data of the application’s state is lost when the

application is swapped out or even killed by the underlying

operating system. Thus it is a requirement on the application

to take care of its own housekeeping and ensure that no

data loss can occur due to events in the operating system. If

an application ensures its own correctness through reacting

appropriately to state changes in the application’s life cycle,

we say that the application conforms to the application lifecycle.

It should be noted that this kind of conformance does not

rely on the definition of the life cycle by the underlying

operating system alone. Instead, only the application can

decide which data needs to be preserved over life cycle

events and which data can be safely discarded. Thus the

conformance to the life cycle does not depend on the

underlying operating system alone, but also on the nature

of the application itself.

Testing the conformance to the application life cycle

means that the application must react reasonably to events in

the operating system. Typically events that can lead to data

loss in mobile operating systems are e.g. low memory or low

battery situations. In order to spare resources, the operating

system might decide to swap out or kill an application,

without saving its current state completely. Such events can

also be triggered by other applications, e.g. handling of an

incoming call can force the operating system to reduce the

number of active tasks in the system. To conform to the

application life cycle, the application must react to such

events in the operating system in a way that ensures that

no important data is lost. If the application implements the

appropriate reactions to the events, then it conforms to the

application life cycle, and ensures its correct behavior even

under resource constraints of the operating system.

The fundamentals of our work are presented in Section II

where we introduce application life cycles in detail, explain

their role in the mobile field and show how our approach

is related to unit testing. Section III presents the method of

our approach. Our method is then applied to a case study

using an actual version of the mobile platform Android. Our

results together with an evaluation of our approach are given

in Section IV. Section V concludes this work.

II. BACKGROUND

This section first explains the concept of application life

cycles from the design and developer point of view. The

reasons, why this work focuses on application life cycles

of mobile applications, and how such a life cycle testing

approach might, especially in the mobile area, improve

2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

978-0-7695-4670-4/12 $26.00 © 2012 IEEE

DOI 10.1109/ICST.2012.36

242

2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

978-0-7695-4670-4/12 $26.00 © 2012 IEEE

DOI 10.1109/ICST.2012.36

241

software quality, are given in the second part of this section.

The third part explains why we chose a unit-based testing

approach and where the differences to traditional unit testing

are.

A. Application Life Cycles

In this paper, the term application life cycle refers to the

process-related states of an application during runtime and

transitions between these states. This concept is neither new

nor introduced by mobile platforms. For instance, it plays

a role in each system handling processes [1], often referred

to as process life cycle. To get a better idea of what a life

cycle looks like, Figure 1 sketches the life cycle of a JavaMobile Edition (Java ME) application [2]. Java ME is a

��

��

��

��

��

Figure 1. Sketch of Java ME Application Life Cycle

widespread mobile application platform, whose applications

can be executed, e.g., on many different Symbian, Nokia

and Ericsson devices. For reasons of clarity the life cycle

model in Figure 1 is not complete1. Life cycles are defined

by the underlying platform for a group of processes, like

service or application. All instances of a group follow the

same life cycle. This means, that all MIDlets have the same

application life cycle, given in Figure 1. A MIDlet is the

name of application-instances on the Java ME platform. It

can be either active, paused or destroyed at one moment in

time. Further, during runtime it can change its state, as the

arrows between the states indicate. For instance, a MIDlet

can change its state from active or paused immediately to

destroyed. But it cannot change its state from destroyed to

active without being paused in between.

Some systems, like most current mobile platforms, allow

the application to react on certain state changes. In the

mobile platforms Java ME, iOS, Android and Windows

Phone 7 this is done by a kind of callback concept. We

explain this concept with the help of the example presented

in Figure 2. The details may vary from platform to platform,

but the concept remains the same. For instance, when the

system receives an incoming call, it has to decide what to

do with the currently active application. If the system wants

to display the phone application, it has usually to dismiss

1The complete model can be found in the official MIDP 2.0 specification,see http://jcp.org/aboutJava/communityprocess/final/jsr118/index.html.

��

� ��

��

��

��

��

��

��

Figure 2. Example of a Callback Notification

the currently active application. Since mobile devices have

limited amounts of resources, the system does not only

hide the currently active application, but also transfers it

to a state, where it does not occupy certain resources,

like CPU or RAM, any more. The detailed reaction of

a system to a certain event is platform-dependent. In our

example, presented in Figure 2, the system would send

asynchronously a pause event, which has impact on the

life cycle of the active application. In case of Java ME

the Application Management System [3] would receive the

event and call the corresponding method pauseApp() of

the active MIDlet. In Android, the Activity Manager [4]

receives and processes such asynchronous life cycle events.

The details differ from platform to platform. By overwriting

the pauseApp() callback-method in our example, the

developer would be able to react on the upcoming state

change. After this method is executed, control returns back

to the system and the recently active application is paused.

There are various reasonable actions that can be done

during life cycle state changes, and most developers take

advantage of this opportunities. Here are a few examples:

• In an application with text input (e.g. messenger or

text application), the inserted text is stored, when an

unpredicted event occurs (e.g. incoming call or SMS),

so that after resume the text is still available.

• An active Voice over Internet Protocol (VoIP) appli-

cation shall not affect the current call, if receiving

text messages (e.g. SMS or e-mail) or declining an

incoming call. But it shall hold the current VoIP call,

when accepting an incoming phone call.

• A game application shall be paused in each case of

interference.

• For reasons of power consumption, the GPS- and

Bluetooth-using application shall free both resources,

if the application is not visible to the end-user.

All these actions can be performed by corresponding code in

the callback-methods. With our approach the four scenarios

can be tested, amongst others.

B. The Role of Life Cycles in Mobile Platforms

Application life cycles play a crucial role in mobile

application development. Due to restricted input and out-

put capabilities of mobile devices, mobile platforms often

243242

display only one application at a time to the user. Next

to the one visible application, there might additionally be

some services active in the background. Mobile platforms

have two possibilities to share the limited amount of re-

sources between active processes. Not visible and inactive

applications might get stopped, where they still may remain

in RAM to be loaded quicker, in case of restart. The second

possibility is to destroy these applications, where they do

not occupy any resources at all (e.g. CPU, RAM, GPS, ...).

This scheduling strategy puts a high load on life cycles

of mobile applications, as each time the user opens a new

application, or wants to switch to a different application,

multiple life cycle callback-methods are called. Figure 3

depicts a real sequence of callback-methods and states of two

applications running on an Android 2.2 system. Application

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��!��

��

��

��

��

��

��

"��

Figure 3. Callback Sequence Example from Nested Applications

A is currently running and from this application another

application B is started. A corresponding real-world scenario

could look as follows: Application A is an e-mail applica-

tion, currently presenting the content of an e-mail. The user

clicks on a URL in this e-mail, which triggers the Internet

browser, application B, to be started. The resulting sequence

of callback-methods and state changes is very interesting and

highlights once more the importance of testing application

life cycles. Figure 3 shows that application A is paused,

after calling onPause(). Second, application B, which

is in the state shut down, is paused after calling the life

cycle callback-methods onCreate() and onStart().

After calling onResume(), application B becomes run-

ning, while the state of application A is still paused. After the

state change of application B to running, finally application

A is stopped after invoking callback-method onStop().

This sequence is neither given in the official documentation,

nor in any Android programming guideline, but it is very

important for the development of mobile applications and

thus needs to be tested. Imagine the developer expects appli-

cation A to first shut down completely before application B

is started. This means, he could shut down connections and

free resources in the onStop()-method, before application

A is stopped. As the developer expects B to be started after

A has been finished, he can request necessary resources

for application B in the onCreate()-, onStart()- or

onResume()-method of application B. But as we can see

in Figure 3, this would lead to various problems. First of all,

application B requests resources that have not yet been freed

by application A, as they will be freed in the onStop()-

method. Second, after executing onStop() of application

A, all connections are shut down and resources are freed by

application A. The result is that application B cannot make

use of all these components. Such errors are hard to track and

difficult to debug. Our testing approach and corresponding

tools, implementing this approach, are capable of handling

these kinds of errors, too.

Another difficulty arises from sometimes incorrect and

incomplete application life cycle models, given by platform

vendors. In a previous work [5], we found, next to errors

and incompletenesses in models and documentations, also

inconsistencies between the given life cycle model and corre-

sponding documentation. For instance, we found transitions

described in the documentation, which were excluded by the

model. For this reason, we reverse-engineered life cycles of

the three mobile platforms iOS, Android and Java ME, to

retrieve the real application life cycle models. This work

on testing mobile application life cycles, uses the reverse-

engineered life cycle models from [5]. Our experience on

working with these models, shows that they have less errors

than the officially given models. Since testing application

life cycles requires a good understanding of the application

life cycle, we strongly advise to use correct life cycle models

and documentations, if available.

C. Unit-based Testing

Taking a close look at the different components of mobile

platforms having a life cycle, like applications and services,

it seems that they are somehow independent of each other

concerning their life cycle. For instance, one application

component has usually one view and one specific func-

tion (e.g. show list of contacts). Further, one application

component has exactly one application life cycle. If the

developer wants to provide a second function to the user (e.g.

dialog to add a new contact), he creates a new application.

The second application is then started from within the first

application, like a sub-application. This sub-application has

its own functionality and life cycle, just like the application

it was started from. The two life cycles are independent of

each other in the sense that two applications never interact

synchronously. Passing data between two applications is

done either via shared resources or asynchronous message

passing with corresponding handlers. Triggering life cycle

methods is also done via asynchronous event handling.

This does not happen between two applications directly, but

between each application and the system. Figure 4 sketches,

how life cycle triggering is done on mobile platforms, like

Android and iOS. Dedicated system components, like the

244243

��

� ��

��

��

��

��

��

��

��

��

��

#�

Figure 4. Asynchronous Triggering of Life Cycle Methods

Activity-Manager in Android, are responsible for trigger-

ing corresponding callback-methods in various applications.

They also take care of different policies, like allowing only

one visible application running at a time. Regarding Figure

4, application A gets stopped by the system and application

B gets started. So the system component is the only one

component that leads to a kind of dependency between the

life cycles of two different applications, since it is also

responsible for triggering the correct sequence of life cycle

methods, as shown in Figure 3. But from the perspective

of each single application, this dependency is not visible

and has no effect on the sequence of its triggered life

cycle methods. Even if an application spends too much

time executing one of the callback-methods, the mobile

platforms are able to stop the method execution and to

proceed with the next method [6]–[8]. This procedure is

usually time-triggered. So time is the only factor, with which

one application could influence the triggering of life cycle

methods of another application.

Our test approach does not take time into account. It does

not distinguish, if the time between the execution of two

different callback-methods is one second or one minute. As

we do not consider time, we regard mobile components with

a life cycle, like applications or services, as units. Every

unit, in this sense, has one life cycle. Our approach allows

testing the life cycle of each unit, independent of the life

cycles of other units. This is no unit testing in the original

sense [9], in which a unit is the smallest testable part of an

application or where a module is tested in isolation. Software

components with life cycles are not the smallest testable part

of an application, as they are no method or interface, but a

complete executable component. Such components cannot

be isolated during testing life cycle components, since life

cycle methods need to be triggered by the underlying system,

to check their functionality in the real environment. For this

reason we do not call our approach unit testing, but unit-

based.

III. TESTING APPLICATION LIFE CYCLES

This section describes our approach in three steps. In the

first step it explains when life cycle related properties shall

be tested. The second step introduces how such properties

can be tested for conformance, using the application specifi-

cation. The third step gives an overview, of what properties

and functionality in the mobile device area can be tested

with this approach.

A. When to Test Life Cycle Properties

Requirements on life cycle properties of an application

always imply state changes of the application. State changes

of an application can be recognized on the implementation

side by corresponding life cycle callback-methods, as ex-

plained in Section II-A. Thus our approach uses life cycles

callback-methods of the application under test, to check life

cycle properties. Integrating the tests into the application

under test, goes along with our unit-based approach. The

test-instrumentation shall not influence any other unit than

the tested one, if it is not avoidable at all. The probe effect

shall be minimized.

We want to explain our approach with the help of a small

example. Imagine a small mobile application consisting of

a text input field and two buttons Save and Dismiss. The

following requirement is from the application’s specification:

If the application is paused/stopped/shut downwithout interaction of the user (he did not clickone of the two buttons), the actual content of thetext field shall be available in the same text fieldafter resume of the application.

This functional requirement shall prevent data loss, if the

application is influenced by the system. It also gives hints to

some life cycle actions, like paused, stopped, shut down and

resume. Corresponding life cycle methods, which usually

can be found in the documentation of a life cycle model of

a platform, can be used to check this requirement. The next

paragraph presents how this can be done.

B. How to Test Life Cycle Properties

For checking the behavior of an application during life

cycle actions there is no correct or incorrect, in general.

Each life cycle dependent behavior has to be tested against

the corresponding part of the specification for conformance.

Thus we decided to use an assertion based testing approach,

e.g. as used in JUnit [10]. For all life cycle dependent

functional requirements in the specification assertions have

to be specified and reassessed during runtime.

Applying this idea to the application given in the previous

paragraph (see Section III-A), an assertion might look as

follows:

assertThat(oldTFContent,equalTo(currentTFContent))

It asserts that the content of text field oldTFContent is

equal to the actual content, currentTFContent, of text

field. So this assertion obviously has to be checked, when

the content of the text field after resume, is known. The

245244

question is, when shall assertions be defined and when shall

they be checked?

First of all, this depends on the part of the application

specification that shall be checked. From each part the

developer has to derive the corresponding life cycle method,

in which the old value or status of interest is known, and in

which callback-method the new value or status is available.

Assuming the application life cycle given in Figure 1 for our

current example, the upper assertion had to be defined in the

pauseApp()- and destroyApp()-method, and checked

in startApp(). pauseApp() and destroyApp() are

called, when the application is about to leave the active state.

Current values of all components, at that moment in time, are

known. startApp() is called whenever, the application is

about to resume and present its content to the user. Since

in the startApp()-method various initializing steps are

done, e.g. loading the old content of the text field from the

database and assigning it to the text field, checking assertions

should always be executed last in a callback-method.

Taking a close look at the life cycle, given in Figure

1, reveals that the application might be destroyed, either

from the state active or paused. As described in Section

II-B, some mobile platforms release all memory, after an

application has been destroyed or shut down. For this reason,

it is important that assertions are stored persistently, after

being defined. Since in our unit-based approach assertions

are bound to an application, on some platforms it makes

sense to store the assertions in the application storage. Even

on mobile platforms keeping their applications in sandboxes

(e.g. Android, J2ME), in which applications have restricted

access to resources like memory, each application can access

its own memory area (e.g. databases). So using this memory

for storing assertions is possible. After an application has

been released from RAM, it can restore assertions from

persistent memory and check them.

But how does one complete test case look like? If the

specification is very detailed, like ”After resuming from anincoming call...”, it is often sufficient to execute exactly

this action and to see what happens to the assertions. If

the specification is not very detailed, like in the example

above (”If the application is paused/stopped/shut down ...”),one can execute a certain amount of different test cases to

check all possibilities. As the number of triggers, that cause

different life cycle actions, is limited (e.g. for Android 2.2

there are 26 different triggers, see [5]), such imprecise parts

of a specification can be tested, too. First, a corresponding

catalog of triggers for the life cycle of the target platform has

to be derived. Afterwards all actions, that are of interest for

the test case, have to be executed. Detailed steps to derive

such a catalog, and corresponding catalogs for Android 2.2

and iOS 4, are given in [5].

The overall proceeding is, first to derive assertions from

the specification and to inject them into corresponding

callback-methods, as explained above. Afterwards the spec-

ified action is performed (e.g. receive an incoming call),

either manually or automatically, to execute the required

triggers. The last step is to inform the developer, if assertions

were satisfied or not. Usually each mobile platform provides

a possibility to send messages to a developer console (e.g.

on Android it would be Logcat, an integrated logger tool).

This can be used, to present the test results to the devel-

oper. Section IV presents an example, how results can be

presented to the developer on the mobile platform Android.

Now the final question left is, what can be tested with

this approach?

C. What can be tested?

In general, components and properties can be tested,

whose state or value might change between two life cycle

callback-methods. It can be a GPS component, which is,

after resuming the application, not accessible any more, or

a text field, whose content might have changed. If it is

possible to test such components and features on a specific

platform, depends on how much access the developer has on

the platform. For instance, some platforms allow to check for

the current battery status, others don’t. On some platforms

the developer is able to check for active socket connections,

other platforms limit socket access.

In the following, we provide three different areas of test

cases, which are relevant for current mobile platforms. Our

test approach is not limited to these areas, or the provided

examples. But as noted above, the applicability is strongly

dependent on the possibilities provided by the underlying

platform.

1) Data Persistence: Data persistence is an important

topic in an ever-changing environment, like on mobile

devices. There are three areas regarding data persistence, that

are important to take care of. The first is volatile data, which

includes all information that are not stored persistently by

the system itself, like contents of user interface components,

text fields, text labels, status of radio buttons, colors of

components and so on. After resuming to an application,

the developer might want to test, if the actual color of

an icon corresponds to the color of this icon, when the

application was closed. Another important area is persistentdata. This includes all possibilities to persistently store data,

like database, file system, shared memory area and so on.

Persistent data storages might often be accessed through

different applications. After resuming to an application, the

developer might want to test, if the values of a database entry

did change, since the application has been closed. Changes

might even sometimes be desired, if a kind of background

service is updating any data in the background, like stock

data. The third important area are content providers. They

are often specified as services, to which all applications

have access to. Sometimes applications need a corresponding

permission. For instance, a content provider might provide

access to the address book, to media files, stored on a

246245

device, or a remote data storage, like cloud data. Content

providers are also used to share data between different

applications. This is often done on platforms, which restrict

the communication between applications by, e.g., a sandbox-

concept. A possible content provider scenario is to check

after resume of an application, if some contact information

was edited since the last application shut down.

2) Connection Status: Many mobile applications use dif-

ferent connections, like Internet or Bluetooth. If an appli-

cation is restarted, the status of a connection might have

changed, e.g. it might be unavailable or closed. On resume

of this application, the developer might want to test, if the

status of a connection changed since the last shut down of

the application. Some scenarios could be: Testing if a socket

connection is open, if a download connection is active or if

a Bluetooth-device is still connected to the mobile phone.

3) Hardware Status: Each current mobile device has

multiple different hardware components, like GPS-module,

microphone, speaker, Bluetooth-module, Wi-Fi or display.

The availability and the states of these components might

change after shutting down an application. For instance, the

Bluetooth-module might be bound to a different service,

the microphone might be muted or the display brightness

changed.

IV. CASE STUDY: TESTING ANDROID APPLICATION

LIFE CYCLES

In this case study, we apply our approach to the mobile

platform Android 2.2. First, we execute each single step of

our approach, as explained in Section III. Then we evaluate

the resulting Android life cycle test library, by applying

it to Google’s NotePad application. Finally, we present a

possibility to integrate this life cycle testing approach into

existing mobile platform toolchains, by integrating it into a

life cycle tool that is currently being developed at our chair.

A. When to Test Life Cycle Properties on Android

Following the first step of our approach (see Section

III-A), we first have to identify life cycle callback-methods

of the Android platform. This can either be done by ana-

lyzing the given life cycle model, platform documentation

and guidelines or the source code, if available. How this is

done in detail, we present in [5]. The resulting life cycle

callback-methods for the Android platform are:

• onCreate()• onStart()• onResume()• onPause()• onStop()• onRestart()• onDestroy()

After identifying the life cycle callback-methods, the

developer has to determine where assertions can be defined

and where they can be checked. Following our approach, a

life cycle model is helpful at that point. As we revealed some

problems with the original Android application life cycle

(see [5]), given by the official Android 2.2 documentation,

we use the reverse-engineered application life cycle from [5].

The life cycle model we use, is given in Figure 5. It is the

��

��

��!��

��

��

Process of Activity iskilled

��

��$��

Figure 5. Android Activity Life Cycle [5]

life cycle of an Android Activity, which is the base class for

Android applications (amongst e.g. services). An Android

Activity has four main states: shut down, paused, running

and stopped. The fifth state, entered before onStart() is

called, is no main state, since the application only enters it

for a very short period of time, after calling onCreate()or onRestart(). The application never remains in this

state, thus we do not refer to it as a state, in the following.

Various information, which are relevant for life cycle

related properties, can be extracted from this life cycle

model. For instance, an application in the state running

cannot be killed, without calling any callback-method. At

least onPause() is called. An implication for application

developers is that important data, that must not get lost,

should be stored persistently in the onPause()-method.

Another interesting example is that an application in the state

shut down can be started and stopped, without being running

in between. This implies, the developer cannot assume that

any actions in the methods onResume() or onPause()have been executed prior to onStop().

B. How to Test Life Cycle Properties on Android

First of all, we identify Activities as units in terms of

Section II-C. An Activity has an own and independent life

cycle. Following Section III-B, we use an assertion-based

approach. Assertions can be defined at one point in time

and checked at another point in time, asynchronously. To

store assertions persistently, even after the Activity has been

killed, we use the storage possibilities of the application

under test. For instance, on Android an application has

access to Shared Preferences to store private primitive data,

Internal and External Storage to store data on internal

or external memory or SQLite Database [11]. Depending

on which kind of information has to be stored, we use

247246

the corresponding storage, like using Internal and External

Storage for storing images, or Shared Preferences for simple

string or integer values.

One important design decision is to implement our ap-

proach for Android as a library with a corresponding ap-

plication programming interface (API). This has various

benefits. For developers it is integrable in their applications

with low effort and easy to use, as libraries are a common

way of providing functionality. Another goal is to keep

the library as independent of the Android platform version

and Android Developer Tools versions, as possible. Our

experience with the Android platform and developer tools

is that their development cycles are rather short. This leads

to many version-related problems in some other projects,

where we, e.g., extend the Android Developer Tools. By

having a very loose binding between the library and the

Android platform or development tools, we try to avoid this

dependency and the resulting problems.

The first step consists of a few initializing actions. In

the first method which is called when the application under

test is started, a LCAssertions-object needs to be defined.

This object stores and handles all pending assertions and

checks them at the right time. The initialization in the

onCreate()-method looks as follows:

public void onCreate(...) {...lcassertions = new LCAssertions(

this.getClass().getName());lcassertions.onCreate();

}

Where lcassertions is a global variable of type LCAs-

sertions. Defining LCAssertions requires to pass the package

name to the constructor. This information is used to assign

test case results unambiguously to one Activity. When the

LCAssertions-object has a reference to the current class, it

is ready to store and execute assertions.

But how does the LCAssertions-object know, which

callback-method is currently being called? We solved this

problem by actively notifying LCAssertions each time a

callback-method is called. This keeps the library usable to

the developer and independent of the underlying platform

and application. But these steps need to be manually exe-

cuted by the developer. He needs to add one line of code

to each callback-method of the application under test. This

line consists of a method-call to the LCAssertions-object.

In the callback-method onXYZ() of the application under

test, the developer adds as a last line of code lcasser-tions.onXYZ(), where lcassertions is the name of

the LCAssertions-object in the application under test. In

the code-example above, this is done in the line lcas-sertions.onCreate(), in the onCreate() callback-

method.

An assertion is defined in callback-methods as follows:

AssertionData ad =new AssertionStringInTextView(

currentText, this.mTextView);

An assertion-object is created and stored locally in a vari-

able. Creating an assertion with our library always re-

quires to pass immediately all necessary information, needed

for checking the assertion afterwards. In this example,

an AssertionStringInTextView-assertion is created,

which is a subclass of AssertionData. This assertion is

used to check, if a given string is presented by a TextView-

element, which is a user interface element. This kind of

assertion requires two parameters. The first parameter is the

string value that is expected to be in the text view, when the

assertion is checked. In our example, the string is encap-

sulated in the variable currentText of type string. The

second parameter is a reference to the TextView-element,

that the given string value is expected to appear in. In our

example it is a TextView-element called mTextView, a

global variable of the current Activity.

Next, the assertion needs to be passed to the

LCAssertions-object of the Activity under test, which stores

the assertion persistently and checks it on time. This is done

by the following line:

lcassertions.assertThat(LCAssertions.ON_RESUME, ad);

This line specifies, that the assertion ad is checked in the

onResume()-method. lcassertions stores the asser-

tion ad persistently in a list of assertions, which shall be

checked in the onResume()-method. Since the developer

added lcassertions.onResume() as the last line of

code to the onResume()-method of the application under

test, lcassertions is notified, when it has to check all

assertions bound to onResume(). The lists and checking

of the assertions is encapsulated in the LCAssertions-object

of the library. The developer does not need to take care about

that.

In the next step the test case needs to be executed, as

described in Section III-B. On Android this can be done

either by using a real device or the emulator. We strongly

recommend to use the real device, where possible. But this

is not always possible. For instance, if a life cycle test cases

shall be executed automatically - what we are currently in

another project working on, it is sometimes an advantage

to use the Android emulator, which is part of the Android

Developer Tools. There are tools available to simulate an

incoming phone call or SMS on the emulator and to simulate

user interactions, like touch-gestures, automatically. On a

real device, this is not always possible. Following our

approach, the developer needs to determine next, which

trigger is of interest for the current test case and execute it. A

full list of triggers for the Android 2.2 platform can be found

in [5]. After execution of a test case, the results are printed

248247

to the developer using Android’s Logcat-tool. This tool is

a common way in the Android development process to log

information. Logging is done within the LCAssertions-class.

Using the integrated Logcat-tool, which is another part of the

Android Development Tools, does not require any other tools

or additional libraries to be installed or learned. This sticks

to the requirements of usability and version-independence

of our library. A screenshot of test results, printed with the

Logcat-tool, is presented in Section IV.

C. What can be Tested on Android

The Android platform provides many possibilities to test

life cycle properties. Figure 6 sketches the package structure

of the resulting life cycle test library, based on our approach.

For reasons of clarity, this figure is reduced to some essential

%�&��!��

'��

��

$��

��$��

�� (��

��)�"�*�+��!�

��

��'��

��

��

��

��

��

,��!��

��,��!��

��

��$��

��(��

��

��

��-�.��

#� #�

#�#�

#�#�

%��

Figure 6. Structure of Android Life Cycle Test Library

elements. The main package is called LCFramework. It

provides some classes with the main functionality to the

user, like the LCAssertions-class. LCFramework contains

two other packages, Util and Assertions. The Util-package

includes functionality, which is shared between different

components of this library. The content is not important

for the user of the library. Storing data persistently and

handling assertion lists are parts of the Util-package. The

Assertions-package provides different assertions and their

specific functionality. For the developer, the three packages

Data, Connection and Hardware are important. They stick

to the structure given in our approach (see Section III-C).

Assertions related to data persistence are part of the Data-package. AssertionData is an abstract class, from which all

other data-assertions inherit, like AssertionStringIn-TextView, which was already presented above. Other as-

sertions that can be checked with Android are, for instance:

Checking the state of radio-buttons, content providers or

databases. Connection-related assertions are defined in the

Connection-package. They include checking status of Wifi or

Bluetooth connections, their availability or port-allocations.

The third assertion-package contains assertions related to

hardware. Android allows to check for different properties

of hardware, like status of the Bluetooth-module, kind of

current data-connection (e.g. 2G or 3G) or speaker volume.

We plan to provide the library, with a corresponding API,

guideline and detailed documentation, open source to the

Android community. In the next section, we evaluate this

framework by using it for testing a sample application from

Google’s Android guidelines.

D. Case Study

In this case study we use the library from the previous

section, to test life cycle related properties of the Notepad-application, a sample application given in the official An-

droid guidelines2. The application consists of three different

Activities. One Activity shows a simple list of notes. If the

user clicks on one note or creates a new note, the second

Activity is started from within the first one. The second

Activity displays the note (see left and right Figures in 7)

and allows to edit it. We refer to this Activity as the Editnote Activity. When the second Activity is active and the

user clicks on Edit title in the menu, the third Activity is

started from within the second one. The third Activity is

a small dialog, which only overlays a part of the second

Activity, not the whole screen (see center Figure in 7). It

allows the user to edit the title of the current note.

Figure 7. Views on Notepad Application

To stick with page limitations of this paper, we present

three of the test cases, which we executed. We chose the

second Activity, in which notes can be edited, as the unit

to test. Data loss in this Activity is immediately recognized

by the user and can be frustrating, especially if large or

important paragraphs of a note are lost. Since we do not

have the original specification of the application, we specify

the following requirement for this Activity:

User input to the Edit note Activity shall notget lost, neither by user interaction, like pushingbuttons or switching to other activities, nor byexternal events, like incoming calls or low battery.

2See http://developer.android.com/resources/tutorials/notepad/index.html.

249248

We present, in detail, the following three test cases, which

can be derived from this requirement:

1) The user opens a note, edits it and then presses the

home-button to close the application. After restarting

the application, the inserted text shall still be available.

2) The user opens a note, edits it, then opens the menu

and clicks on button Edit title, to edit the title of the

note. After returning back to the Edit note Activity,

the inserted text shall still be available.

3) The user opens a note, edits it and then receives a call.

After declining the call and returning back to the Editnote Activity, the inserted text shall still be available.

The necessary triggers and expected reactions are taken

from [5]. Test case 1 corresponds to the 11th scenario in

[5], test case 2 to the 23rd, where the small alarm clock

dialog is very similar to the edit title dialog, and test case

3 to the 4th scenario. Based on the expected callback-

method sequences, we place the definition of the Asser-tionStringInTextView-assertion in the onPause()-

method and it will be checked in the onResume()-method,

as explained in IV-B. Figure 5 shows that we are focusing

on the state change between paused and running, which

is always triggered when user interaction is enabled or

disabled.

All three test cases have been executed on a real device,

which was connected to the PC. The device is a Motorola

Milestone 2 with Android 2.2 on it. The setup before

executing each test case was a clean and fresh installation

of the Notepad application and no other active applications

or services running, than the default Android system ser-

vices, which remain started after booting. Since the Notepad

application does not access any type of connection, we

set the mobile device to flight mode, which disables all

connections, like Bluetooth or GSM, to reduce potential

sources of interference. The application was started in each

test case by clicking the application icon on the home screen.

��

��

��

Figure 8. Logcat-Output of all three Test Cases

The Logcat-output of all three test cases is presented in

Figure 8. Test case 1 was executed successfully, as output a)

shows. The inserted text Hello, world! was still in the text

field, after resuming the application from the home-screen.

The output of the second test case, labeled b), presents

that the assertion in the second test case was not affirmed.

When opening a note, which contains the word Hello (see

right Figure in 7), extending it to Hello, world!, pushing

the Menu-button and then clicking on Edit title, the edit-

title dialog opens (see center Figure in 7). If the user then

pushes the Back-button of the device, to return to the Editnote Activity, the inserted text Hello, world!, which can be

seen in the background of the center image in Figure 7,

disappears and becomes the old text, presented in the right

screenshot of Figure 7. This behavior does not correspond to

the second requirement of our specification. But it may be

the desired behavior of Notepad’s developers. We venture to

doubt that, since obviously data is lost. This is represented

by our framework as an orange warning message in b) of

the Logcat-output in Figure 8, stating that the assertion has

failed. Test case 3 was executed successfully, as output c)

of Figure 8 shows. After declining an incoming call and

resuming to the application, the inserted text is still available.

These three test cases show that our approach is applicable

to test mobile platforms and that our implementation of

this approach for the Android 2.2 platform, works well. We

even found some unexpected behavior of Google’s Notepad

application, which we cannot declare as error, since we do

not know the original specification of the application. Next to

the case study, we also use this Android library successfully

in many other projects to test the conformance of life cycle

related properties and it is still being improved and extended.

But an extensive use of the library also highlights the limits.

For instance, the library is not complete in the sense, that

not all possible life cycle properties can be checked with the

current library version. E.g., the current version cannot check

properties of a near-field communication module, which is

available in upcoming Android devices. But regarding our

architecture (see Figure 6), such extensions can be easily

added by inheriting from the correct assertion-class and

implementing the necessary functionality. Due to the loose

coupling of the library to any Android development tool or

Android version, it remains independent and portable, but re-

quires some interaction of the developer. He has to integrate

the library manually into a test application, like calling the

corresponding callback-method of the LCAssertions-object

in each callback-method of the application under test. This

can be reduced by integrating the library in a corresponding

development tool.

As we are currently developing a tool to support the

correct implementation of Android application life cycles

and automatic testing of life cycle related properties, we

integrated the library-functionality as a feature in our tool.

The tool executes the necessary initializing steps to test an

Activity for life cycle properties at the push of a button,

and supports the developer to define assertions. Figure 9

shows the life cycle related test functionality of the tool. By

right-clicking the method-name in the presented life cycle

model, the developer can specify, which kind of assertion

he wants to define in this method. In the upcoming dialog,

250249

Figure 9. Specifying Assertions interactively in a Life Cycle Model

presented in Figure 10, the developer can specify, which type

of assertion he wants to define, corresponding parameters

and in which callback-method the assertion shall be checked.

Figure 10. Specify Details in a separate Dialog

V. CONCLUSION

Checking for conformance of life cycle dependent prop-

erties can be a challenging task and requires methodical

approach. In this paper we present a unit-based test approach

for testing conformance of life cycle dependent properties of

mobile applications, using assertions. Assertions are defined

in one callback-method of the application life cycle and

checked asynchronously in another callback-method. Further

we described, how test cases look like and how they can be

executed using application life cycle triggers.

In a case study we implement this approach as a usable,

easily integrable and, to a large part, version-independent

library for the Android 2.2 platform. The case study shows

how this library is applied to Google’s Notepad applica-

tion. One interesting result after testing the conformance

of Notepad’s life cycle dependent properties to our derived

specification, is that we even found some unexpected be-

havior in this official sample application. In a last step we

present how an integration of this library into development

tools can ease the use and reduce the effort of testing life

cycle dependent properties.

This library is still being improved and extended to cover

more life cycle related features of the Android platform.

We are also extending the presented tool to provide more

functionality to the developer and to speed up and automate

testing of life cycle dependent properties of the Android

platform. As this approach is not limited to the Android

platform, and not even to mobile platforms at all, we are

in the process of defining corresponding implementations of

this approach for iOS and Windows Phone 7.

ACKNOWLEDGMENT

This work was supported by the UMIC Research Centre,

RWTH Aachen University Germany, and the Sirindhorn

International Thai-German Graduate School of Engineering

(TGGS).

REFERENCES

[1] A. Tanenbaum, Operating Systems - Design and Implemen-tation. Englewood Cliffs, NJ, USA: Prentice Hall, 2006.

[2] J. Knudsen and S. Li, Beginning J2ME Platform: FromNovice to Professional, 3rd ed. Berkeley, CA, USA: Apress,2005.

[3] G. P. Perucci and A. Haber, Java 2 Micro Edition. Dordrecht,Netherlands: Springer Netherlands, 2007, pp. 62–94.

[4] R. Meier, Professional Android 2 Application Development.Indianapolis, IN, USA: John Wiley & Sons, 2010.

[5] D. Franke, C. Elsemann, C. Weise, and S. Kowalewski,“Reverse Engineering of Mobile Application Lifecycles,” inProceedings of the 18th Working Conference on ReverseEngineering, 2011, pp. 283–292.

[6] W.-M. Lee, Beginning IOS 4 Application Development. In-dianapolis, IN, USA: John Wiley & Sons, 2010.

[7] M. L. Murphy, Beginning Android 2. Berkeley, CA, USA:Apress, 2010.

[8] R. Rischpater, Beginning Java ME Platform. Berkeley, CA,USA: Apress, 2008.

[9] P. Hamill, Unit Test Frameworks. Sebastopol, CA, USA:O’Reilly Media, 2004.

[10] A. Hunt and D. Thomas, Pragmatic Unit Testing in Java withJUnit. Lewisville, TX, USA: The Pragmatic Programmers,2003.

[11] E. Burnette, Hello, Android: Introducing Google’s MobileDevelopment Platform, 1st ed. Raleigh, NC, USA: PragmaticBookshelf, 2008.

251250

[ieee 2012 ieee fifth international conference on software testing, verification and validation...

Documents