mobile test automation getting started€¦ · 4 xbosoft white paper – mobile test automation...

Mobile Test Automation Getting Started

2

XBOSoft White Paper – Mobile Test Automation www.xbosoft.com

Test automation can help you handle mobile device and platform proliferation with increased coverage and strategic regression

Mobile Test Automation – When, Where, and How Mobile applications continue to multiply as every week thousands of mobile applications are uploaded onto the Internet for end users who download to various types of mobile devices and operating systems. To ensure that these mobile applications work as expected has become quite a challenge and has led to a whole new QA discipline focused on mobile application software. Some of the most prominent challenges in testing mobile applications include: Device and platforms diversity

Android for example; manufacturers like HTC, Moto, Samsung, Sony, etc. embed the Android OS into their devices. Up

to now, Android has around ten different versions. iOS although not as severe, have 3GS, 4, 4S combined with 4.xx

and 5.xx operating system versions.

Application Complexity

Applications are becoming more and more complex and to test them requires more than simple rendering. In addition, mobile websites and browser types add other dimensions to the platform matrix.

Performance How to test performance and what part of the performance do you test? Server side? Network? Device? How to

record time consumed? How to design the right scenarios and more importantly how to find the source of the

bottleneck if there is one?

3


Because of these challenges and more, mobile test automation has become a hot topic in the testing community as a means to tackle platform and application complexity issues while increasing or at least maintaining coverage as the number of platforms continues to increase.

Automation Strategy As with any test automation, the first consideration is what should and what should not be automated, or can/cannot be automated. What parts should remain for manual testing? Automation should be used judiciously in the right places. Given this strategy, the remainder of this paper is organized as follows:

Where Mobile Automation is not a good fit in your mobile testing efforts and manual testing makes more sense.

Where Mobile Automation is a good fit; then you need to figure out what tools to use. We’ve provided a short list to get started, but the list continues to expand.

How to automate mobile testing? We’ve provided a brief example using Selenium WebDriver, a common automation test tool used with Android.

Where to Use Manual Testing (and not use Automated Testing) Negative testing

Negative testing scenarios are quite difficult to design. Even if designed, most often it is not worth it to spend much

time on developing scripts because it may not be executed frequently.

Testing device-specific features used in an application i.e. Camera, accelerometer, GPS & Bluetooth Specific device features are sometimes not accessible by automation tools because APIs are packaged and scenarios

are quite difficult to simulate by codes. Manual testing is more applicable here.

Compatibility between applications like Call & SMS interrupts Automation tools have difficulty simulating these interrupts. It also is difficult to run or execute two different

applications at the same time using automation tools.

Voice-mail or IVR based scenarios

Automation tools are not good at predictive intelligence. None of these tools can send nor verify the voice, like identifying is someone is talking or checking if there is any break. The conversation between the testing scripts and the phones is difficult to implement unless you know exact scenarios and convert them to binary codes. In this case, it’s better to use manual tests.

Where to Use Automated Testing Reproduce bugs

A bug sometimes needs to be reproduced several times to address the root cause, especially for those complex

ones. It is also necessary to execute the same steps to validate if it has been fixed. Manual execution will take a lot of

time for defect verification, especially for complex defects.

Regression tests Many test cases need to be executed for each release to ensure basic functionality has been maintained and not

affected. Although with most you never find a defect, it is best to buy insurance and use automation to reduce

execution duration and effort.

4


Same cases with different environments As listed before, an OS system will be embedded into different devices with different OS versions. That means

one test case can usually be executed on these multiple environments in the same OS. Automating them is

obviously a better option.

Automation Tools

Since using mobile devices to access and surf web sites and mobile applications continue to gain complexity and features, mobile test automation has come to the forefront as a means to reduce regression and test cycle time. There are numerous automation tools available on the market, some free, and some available as plug-ins for existing commercial based tools such as QTP. The table below summarizes a few of the automation tools on the market but is by no means exhaustive as new tools are released frequently.

OS Tool Usage

Android Robotium Unit, UI, Functionality

Scirocco UI

Autoandroid Unit

SeeTest System

Calculon Unit

Selenium Android Driver Web Sites

iPhone TestiPhone Web Applications

Cucumber

DeviceAnywhere Unit, Functionality

Selenium iPhone Driver Web Sites

Blackberry DeviceAnywhere Unit, Functionality

TestQuest Unit, Functionality

Using Selenium WebDriver to Automate Android tests (Selenium Android Driver) For Android, traditional mobile tools like Robotium have limitations in testing browser-based applications. Selenium, which concentrates in testing web sites, has updated its framework using WebDriver to support testing on Android.

Using WebDriver, the browser can be launched on a real device or an emulator. To launch an emulator, it is necessary to install Android SDK with at least one Android Platform, like Android 2.2. Here, we demonstrate using 4.0.3.

5


After the emulator is launched or the device is connected to a PC, there are two ways to use WebDriver. Although the main scripts for both methods are the same, if the test scripts will be executed for platforms other than Android, it is better to use the first method as we show below. The latter method with the Android test framework is only architected for Android.

1. Use remote WebDriver server. Step 1. The WebDriver server will be installed on the device or emulator to receive requests. After starting WebDriver, a browser-like application is launched as shown in Figure 1.

Figure 1: WebDriver start up screen.

Step 2. Create the testing script. Test scripts can be manually coded or exported from Selenium IDE as Java WebDriver code. For example, you can use the Selenium IDE to record the operations of:

1) Open www.google.com 2) Search ‘Cheese!’

Then export the test case as a Java (WebDriver) test case as shown below:

http://www.google.com/

6


Edit the script substituting the Android driver for the Firefox driver. Step 3. Run the script as a JUnit test. It depends on whether or not you use IDE to run the script, but all of the selenium-java jars should be imported to your classpath. Figure 2 shows the results when WebDriver receives the request.

7


Figure 2: Results when Webdriver receives the request.

2. Using Android test framework

In this solution, WebDriver should not be installed. Instead ‘google WebDriver’ should be downloaded as well as the

Android SDK. When executing the cases with this test framework, the Android browser will be launched. In the

downloaded ‘google WebDriver’, there is a folder of examples. Those should be imported as Android JUnit test projects

into your IDE.

For both of these methods, you may encounter a problem that Selenium WebDriver cannot find specific element(s). That is because mobile devices visit different pages which can be geographically specified. For instance, accessing the Google USA home page for mobiles when it is in the US will render different page styles, element ids or element names. As in the example to search ‘CHEESE!’ on Google.com provided by Selenium, you may need to add a clicking operation after opening the Google mobile home page to relocate the page. If not, Selenium may not be able to find the element named ‘q’, the text field of searching box.

8


Conclusion: Despite many of the difficulties and limitations of mobile automation, it can still help you on Unit, UI, functionality, webapp, and websites tests on different devices and systems in many situations. It can save you a lot of time, especially with quick scrum sprints. But you cannot rely solely on automation due to its limitation in simulating scenarios that may include interruption and embedded device functions (i.e. camera). As with platforms other than mobile, it’s best to judiciously decide what should be automated and not, and develop an overall game plan as to where and when to use of exploratory and manual testing because tools can never replace a real set of eyes.

XBOSoft Inc. 640 Rocca Ave.

South San Francisco, CA 94080

+1 408-350-0508 www.xbosoft.com

mobile test automation getting started€¦ · 4 xbosoft white paper – mobile test automation...

Documents