mobile application testing challenges and solutions · emulated devices for mobile application...

Mobile Application Testing Challenges and Solutions

Mobile Device Proliferation

XBOSoft White Paper – Mobile Device Proliferation www.xbosoft.com 2

Introduction

In recent years the mobile application market has witnessed rapid growth. This is having a direct effect on the way that people communicate, bank, play, learn and organize their daily lives. Mobile applications give users access to their applications, especially web-based, at any time and any place. A retail store manager can quickly check product inventory on her smartphone, a busy executive can review his appointment schedule from the coffee shop, a nurse can consult a patient’s history from the bedside and a sales representative can access customer records and report on deal status while walking through an airport.

The explosive increase in the popularity, choice and availability of mobile applications shows no sign of slowing down. And because mobile applications are so easy to download and use, they reach a broad audience. This makes quality of your mobile application even more critical. But mobile platforms have some peculiarities that require special attention with regards to quality. The biggest challenge for testing mobile applications comes from the plethora of devices used to access the application. Devices can differ in many ways:

Screen size Display Input method Operating system Location of end user File formats Device memory Processing power Network Operator

The explosive increase in the popularity, choice and availability of mobile applications shows no sign of slowing down


So are you absolutely sure that your application will work as planned on all intended devices? This question is at the heart of the success of mobile applications. This paper starts by focusing on the device related challenges for mobile application testing, before exploring the options available to counter these challenges. The paper concludes by giving recommendations on how to ensure that you have a solid strategy for testing applications on multiple mobile devices. The number of mobile devices used for both enterprise and leisure creates an obvious challenge to mobile application testing. There are tens of thousands of different client devices that could be used with your mobile application, and they must therefore all be considered when testing. This number can be reduced to an extent based on your intended user group, but each time you reduce the number of device types that you test against you’re taking a chance that your application might not work on a device, locking out a number of potential customers. Below is an overview of some of the differences between handsets that pose a challenge to mobile application testing. Screen sizes – Different mobile devices have

different screen sizes. For example the screen on the iPhone4(S) is 3.5-inches (diagonal) and the screen size on the Nokia N70 is 2.1-inches (diagonal). This will clearly have an impact on how your mobile application will appear. If your application appears on a smaller screen are the images and text still clearly visible? And if on a larger screen does your application look stretched and skewed?

Display – Some devices will have color displays and some (nowadays very few) will still have black and white displays. In addition, the pixels that each device can display may also have an impact on the appearance of your mobile application.

Are you absolutely sure that your application will work as planned on all intended devices?


Input method – The way data is input into a mobile device can vary. The iPhone4(S) utilizes a touch screen, whilst the Nokia N70 uses a conventional keypad. Some PDA devices also use a stylus to input text and make commands. Can your mobile application handle both touch screen and conventional methods of input?

Operating systems – The amount of operating systems are numerous such as Apple iOS, Android, web OS, Windows Mobile, Blackberry OS, Symbian, Maemo, MeeGo and Bada. It is important that the uniformity and consistency of the mobile application remains in place no matter the operating system.

End user location – Depending on the location, different devices are available, and the mother language of the end user will differ. Different countries also have varying laws and regulations regarding mobile devices that effect what the devices are able to do. For example, a person who accesses Google Maps from their mobile device in the UK will be viewing a slightly different version to that which a person in Iran will see. Does your mobile application perform the same on a mobile device that is being used in English, as it does in Chinese Mandarin? (Do you need it to?)

File formats – Mobile devices not only need to support WML or CHTML. Nowadays they need to support a lot more scripting languages such as XTHML, as well as multiple image formats. So determining if you need to support these different formats needs consideration.

Device memory and processing power – Different mobile devices have different memory sizes and varying levels of processing power. This has a direct impact on the download capability of the device and the speed at which your mobile application will work on the device. Is your application actually

capable of being downloaded onto its intended devices? If it can be, what are the minimum requirements for it to execute at a speed that is perceived as satisfactory?

Network operators – A network operator can have a huge impact on the capabilities of a mobile device. The same device using the Vodafone network may have different capabilities than if the network operator was AT&T or Verizon. Today there are well over 450 mobile networks operating around the world: some are CDMA, some are GSM, while others use less common local network standards. The infrastructure of each network is unique, and each network has implemented systems that may behave slightly differently. In addition the majority of network operators have inserted web proxies which are able to dictate how, when, and if you are able to connect to a particular site, thus having a significant impact on the flow of information between your server and the consumer.

Options for Overcoming the Device Challenge There are a number of options available to overcome the device specific challenges of mobile application testing. Each option has their own pros and cons, and different options might be suitable depending on the testing needs of each individual mobile application. The most common options include: Which devices to test? To test using real or emulated devices? And to test in different user locations or test remotely? Which Devices to Test? In an ideal world it would be best to test your application against every possible mobile device that your customers could use. However in reality this is simply not possible as the number of devices available continues to proliferate, with new devices released every day.


Therefore application developers must carefully prioritize and select which devices their application will run on. Here are some ideas to help narrow down the number of devices: Define Target Audience – Users of your mobile application may belong to a specific targeted user group. Certain audiences tend to prefer different types of devices. Audiences can be divided in a number of ways such as geographic location, age, gender, income, hobbies and occupation. For example if you were targeting the UK student market then according to The Mobile Youth Consultancy the market is dominated by two brands (Nokia, 30%, and Sony Ericsson, 27%). Blackberry phones are far behind with only 2%. With these statistics in mind it is clear to see that priority should be given to testing using Nokia and Sony Ericsson devices, before Blackberry is considered. Device Families – There are “device families” that offer a very similar user experience (and therefore may not need to be tested individually). Identify device families between which the user experience is the same or similar. For instance, you may choose to assume that all iOS devices are largely the same given the same physical device (same iOS version), and all Android devices are largely the same while holding one parameter constant and letting other options vary. You can also identify families of phones that have the same characteristics, i.e. Nokia phones with Symbian S40, with a set screen size. Emulated or Real Devices? When testing the compatibility of mobile applications on different devices you have two options: real devices or emulated devices. Testing with real devices involves manually testing your application against a selection, or all of the possible devices that you expect will be used to operate your mobile application. Real devices have the advantage of having all the limitations present in the actual client hardware and firmware combination in the hands of your target users. However testing with real devices is expensive for several reasons including:

Application developers must

carefully prioritize and select which

devices their application will run

on.


The cost associated with purchasing devices – Without subscribing to a contract, the price for an individual device can range anywhere from $50-$1000+. If you plan to test the whole suite of possible devices, then the total cost could spiral upwards quickly.

The cost associated with labor – Testing each device manually is extremely labor intensive.

Another aspect of testing with real devices that needs consideration is that real handsets are not designed with testing in mind. The limited processing power and memory of the handsets does not allow on-board diagnostic software to be loaded, so they lack instrumentation. With real devices you will not be able to record the protocols going back and forth between the device and your application, therefore limiting your ability to isolate problems and make corrections.

On the other hand, an emulated device duplicates the functions of one device, so that the emulated device behaves like (and appears to be) the real device. Emulated devices for mobile application testing have a number of advantages: Ease of transfer between devices – By using

an emulator you can switch device types by simply loading a new device profile, and instantly you have a new device that presents itself to your mobile application in the same way that the real device would.

Quicker Access to Devices – Using an emulator is much faster than switching sim cards and rebooting devices.


Additional Power – Because emulators run on more powerful PC’s and servers, and were designed with testing in mind they are typically fully instrumented to capture detailed diagnostics about the protocols that go back and forth between client and application.

Improved Analysis – When you capture an application defect, you will have the information to isolate and thus correct the problem.

Cost Effective – Emulated devices is cost effective because a single platform with frequent updates of device profiles can be used to test every device on the market, both today and tomorrow. In addition, no data charges are incurred as all browsing is performed via your standard web connection.

Emulators are a useful tool, but you should not assume that just because your

application works perfectly on an emulator it will function in the same way on

the real device. Emulators can often differ in subtle ways from the devices they

simulate. This is especially the case if the developer of the emulator is not the

original creator of the device. Use emulators to cover a wide breadth of devices

that you wouldn’t have access to, and fix any bugs you find, but make sure you

test the application on some real devices as well, just to be sure.

Mobile device emulators fall into three main categories:

Device Emulators –These are generally provided by device manufacturers and

simulate the actual device. Device emulators are excellent for testing your site

or application on a particular device or set of devices. For instance, some

companies offer emulators for some or all of their mobile devices:

Research in Motion (BlackBerry)

Apple (iPhone)

Samsung

Palm

LG

Motorola

Browser Emulators - These simulate mobile browser environments. Although

useful for determining the functionality available in a particular mobile

browser, they are useless for device-specific testing. Example browser

emulators include:

Opera Mini

Openwave


Operating System Emulators - These run within a simulated mobile device environment and provide access to applications running within the operating system, e.g. a Web browser. Operating system emulators are available from: Microsoft (Windows Mobile) Google (Android) Nokia (Series 40 and Series 60)

There are also some generic emulators – mobiForge offers an emulator which can be skinned as a Nokia N70 or a Sony K750. This is a useful starting point for basic testing. Yospace sells a smartphone emulator that supports a number of devices. However, many of the devices supported are now out of date (the Nokia 6310i, for example). Emulators can generally be found in one of three places:

On a manufacturer’s development portal Bundled as part of an SDK As part of a Web site

In general, many device manufacturers offer free emulators in order to encourage development on their platforms.

Test on Location or use Remote Devices? There are well over 450 network operators in the world, some privately owned and some Government owned. The location of the device may significantly impact the access your mobile application. For example in some parts of the world many geographic applications go through a process of censorship. Obviously this has an

impact on the overall quality of your mobile application. How can you be sure that a mobile application accessible from devices in the UK will also be accessible from devices in the USA? Each network operator has implemented systems that behave slightly differently from other vendors, and most mobile operators have inserted mobile web proxies to dictate how, when, and if you are able to connect to a particular site. Some proxies strip vital HTTP headers that your mobile application might depend on to provide functionality or to provide device adaption. Naturally it is possible to test against real networks. One method is to use real devices in their target location to assess the impact on your mobile application, though this is a very costly method. Alternatively many device emulators support modems that allow you to use your emulated devices on the local network – but again there is the cost of travelling into range of the network. The next option is to use a remote real device. This solution consists of a physical handset mounted in a remote box with a remote control unit and a remote control antenna. The remote control unit is physically connected to the device’s screen and keypad control circuits and is capable of pressing keys and collecting screen images. Exposed to the internet, these solutions allow a user on a local PC or a web client to control the device and see what is happening on the remote device. Remote real devices provide an elegant solution to testing on location that can be connected to either live networks or simulated networks.


Remote real device solutions often have the ability to record a test for subsequent replay, a capability that can be useful for regression testing. Remote real device solutions reduce the cost of travel to foreign networks, but can still be expensive, because the cost of the device is now amplified by the cost of the remote control hardware, remote control software and local software. Because there are so many different makes and models of devices it is often too expensive to buy a remote real device solution for all the devices that you need to test against. Fortunately, most of the companies that make this type of equipment offer the ability to rent testing time on a resource that is shared and is managed for you. You simply need to open up an account and you then buy testing time with a given make and model of device, when and where you need it But these solutions are not without disadvantages. First, the testing will take time to set up and has the same labor cost as testing with a real device. Second, like a real device, the solution lacks the diagnostic tools to capture the information needed to isolate and find corrections to problems with your application. And finally you will need to make sure that the solution is located where and when you need it.

Your Mobile Application Testing Strategy – Recommendations With an understanding of the challenges in testing mobile applications, how to develop your strategy for mobile application testing? It is not a matter of choosing one tool or technique: there are simply too many compromises that must be made. The time, budget, and skills available will impact your strategy and most suitable techniques to be

applied. Most likely you will need a combination of testing tools and techniques to meet your quality requirements, but generally you can narrow your choices down to the following best practice recommendations: Invest in a device emulator – This will allow you to perform the bulk of your testing in a well instrumented test environment that is cost effective. Your device emulator should have options such as bypassing the network, using the live network via modems and a good scripting language, so you should look for those features during your selection process. Finally, when you look at device emulators for testing, make sure that they have the instrumentation and network options that provide you with the flexibility that you will need. Take advantage of remote real devices – Having an account with a vendor that allows you to access actual real devices at any time is very useful. You never know when you might need to test on a remote live network with a device that you don’t have. Since these are often pay as you go solutions, they are great options to have up your sleeve. Don’t ignore real devices on real networks – There is no way around it; you will need to perform some manual testing with real devices on real networks. But your strategy should be to use this option only for the top user profiles in your market space.


Summary The rapid rise in mobile applications shows no sign of slowing down. Mobile devices allow us to do many things while on the move and the functionality of mobile applications will continue to grow. When it comes to testing mobile applications there are a number of unique challenges. These challenges are real, can be demanding, and can have disastrous consequences if ignored. Increasingly varied specifications and capabilities mean that arguably the most immediate of these challenges is presented by the actual mobile devices that are being used to access the mobile application. This leads us to ask the following questions. Which devices to test? Use emulated or real devices? Test on location or remotely? The answers to these questions depend heavily on the nature of the application, the budget available for testing and the time available. Deciding which devices to test must be prioritized. Which devices are most popular with users of your application? Can you save time and money by testing one device from each mobile family? Emulated devices have the advantage of being more test compatible, and have the ability to store a large number of different devices and test runs (data analysis). However emulated devices do not have the same quirks that a real application user may experience. Testing devices in all the locations where users might be is probably logistically impossible, and financially not viable, but priority locations should be tested with the devices most prevalent in that market. Testing remotely can save a significant amount of time and man hours. These testing options all have pros and cons, and you will probably find that no one option will satisfy all your objectives. Rather, you will need to consider a testing strategy that combines different testing methods that meet your needs while balancing the tradeoffs between cost, quality and time to market.

XBOSoft Inc. 640 Rocca Ave.

South San Francisco, CA 94080

+1 408-350-0508 www.xbosoft.com

Cover page photo by: Tomo Yun http://www.yunphoto.net/en/

mobile application testing challenges and solutions · emulated devices for mobile application...

Documents