testing mobile applications using perfecto mobile mobilecloud v2

ALM Rangers - Testing mobile applications with Perfecto MobileCloud

Page 2 of 49

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as

of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be

a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the

date of publication.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY,

AS TO THE INFORMATION IN THIS DOCUMENT.

Microsoft grants you a license to this document under the terms of the Creative Commons Attribution 3.0 License. All other

rights are reserved.

2013 Microsoft Corporation.

Microsoft, Active Directory, Excel, Internet Explorer, SQL Server, Visual Studio, and Windows are trademarks of the Microsoft

group of companies.

All other trademarks are property of their respective owners.


Page 3 of 49

Table of Contents Introduction ...................................................................................................................................................................................................................................... 4

Overview ....................................................................................................................................................................................................................................... 4

Visual Studio ALM Rangers ................................................................................................................................................................................................... 4

Contributors ................................................................................................................................................................................................................................ 4

Personas ........................................................................................................................................................................................................................................ 4

Related Resources..................................................................................................................................................................................................................... 4

Exercises ........................................................................................................................................................................................................................................ 5

Device testing approach for IT Applications ....................................................................................................................................................................... 6

Need for Device Testing ......................................................................................................................................................................................................... 6

Challenges in Device Testing ..................................................................................................................................................................................................... 7

Device Testing Lifecycle for IT Applications (DTL-IT) ................................................................................................................................................. 7

Envision Phase ............................................................................................................................................................................................................................ 7

Plan Phase .................................................................................................................................................................................................................................... 8

Build Phase................................................................................................................................................................................................................................ 11

Stabilize Phase ......................................................................................................................................................................................................................... 14

Native App Best Practices ................................................................................................................................................................................................... 14

Environmental preparation ...................................................................................................................................................................................................... 16

Installing the Perfecto Mobile Team Foundation Server (TFS) add-in ............................................................................................................. 16

Accessing the Perfecto Mobile Cloud Platform ........................................................................................................................................................ 17

Hands-On Lab ............................................................................................................................................................................................................................... 20

Exercise 1 Accessing the Perfecto Mobile Cloud and running an automated test (10 min) .............................................................. 20

Exercise 2 Run an automated test in the Perfecto Mobile Cloud within Visual Studio (15 min) ...................................................... 26

Exercise 3 Run an automated test in the Perfecto Mobile Cloud within Microsoft Test Manager (10 min) ................................ 31

Exercise 4 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on TFS (10

min) .............................................................................................................................................................................................................................................. 38

Exercise 5 Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on Team

Foundation Service in the cloud (10 min) .................................................................................................................................................................... 41

Conclusion ...................................................................................................................................................................................................................................... 45

Appendix ......................................................................................................................................................................................................................................... 46

Automation Tool Comparison .......................................................................................................................................................................................... 46

FAQ .............................................................................................................................................................................................................................................. 48


Page 4 of 49

Introduction

Overview How do you develop and test apps on iOS, Android, and Windows Phone devices? You leverage the Visual

Studio Industry Partner (VSIP) solutions and our Team Foundation Server (TFS) integrations. For example, for IOS

and Android development, use solutions from ITRMobility and Xamarin to develop within the Visual Studio IDE

using .NET. You can use Xcode, Eclipse, or IntelliJ and integrate with Team Foundation. Then you test your

mobile application using Perfectos Mobile Cloud solution.

Mobile applications can require complex testing on specific devices for which the acquisition and maintenance

can be expensive. Especially for devices based on Android as there are so many variations. Emulators are useful

but can be expensive and inaccurate. The best solution is to test your mobile applications on the actual device

using Visual Studio Application Lifecycle Management (VS ALM) and TFS tooling enabling an end-to-end

Application Lifecycle Management (ALM) experience. Deploying your application on the actual device, testing

from anywhere globally, and integrating with Visual Studio and TFS provides incredible business value.

The following guidance will describe how to install and setup Visual Studio and TFS to use the Perfecto Mobile

MobileCloud for TFS solution. The document also describes how to use the Perfecto Mobile solution within the

Visual Studio 2012 RTM ALM Virtual Machine or your own VM or local desktop. In addition, we will discuss the

architecture, the mobile cloud, testing of real devices, ScriptOnce, the integration with Visual Studio and

Microsoft Test Manager, and the end-to-end scenario.

Visual Studio ALM Rangers The Visual Studio ALM Rangers are a special group with members from the Visual Studio Product group,

Microsoft Services, Microsoft Most Valuable Professionals (MVP) and Visual Studio Community Leads. Their

mission is to provide out-of-band solutions to missing features and guidance. A growing Rangers Index is

available online1.

Contributors Anil Chandra Lingam, Anisha Pindoria, Brian Blackman, Dan Marzolini, Esteban Garcia, Hassan Fadili, Mathew Aniyan, Oliver

Hilgers, Bipin Puthiyaveetil, and Ranjit Gupta.

Refer to Visual Studio ALM Rangers Personas and Scenarios 2 for more information about the personas and

customers profiles referenced in this guide.

Related Resources Understanding the ALM Rangers http://aka.ms/vsarunderstand

Visual Studio ALM Rangers Home Page http://aka.ms/vsarmsdn

Visual Studio ALM Ranger Solutions http://aka.ms/vsarsolutions

Visual Studio ALM Ranger Solution vsarDeviceTest Project on Codeplex - http://aka.ms/vsardevicetest

1 http://aka.ms/vsarindex

2 http://go.microsoft.com/fwlink/?LinkID=230942


Page 5 of 49

Exercises The Hands-on lab sections will show you how to use the Perfecto Mobile platform and Visual Studio to test

different mobile devices through automated scripts as a manual or general tester. The lab will take approximately

55 minutes to complete. The hands-on lab is structured for you to learn how to:

1. Run an automated test script on the perfecto mobile cloud and review the test results.

2. Create your own Visual Studio Perfecto Mobile Cloud project and run tests against real mobile devices in the

cloud.

3. Run an automated test script against real mobile devices in the cloud within Microsoft Test Manager 2012.

4. Run an automated test script within a Continuous Integration process on TFS Team Build.

5. Run an automated test script within a Continuous Integration process on Team Foundation Service Build in

the cloud.

This guide targets the Microsoft 200-300 level users of Visual Studio 2012. The target group is considered as

intermediate to advanced users of Visual Studio 2012 and has in-depth understanding of the product features in

a real-world environment. Parts of this guide may be useful to the Visual Studio novices and experts but users at

these skill levels are not the focus of this content.

WA

RN

ING

The mobile device environment might change.

Please read the next paragraph carefully.

The exercises run on real devices in the Perfecto Mobile Cloud. As real mobile devices are subjected to software

updates and other environmental changes over time, it is possible that tests could on occasion fail. Therefore, if a

test execution in the exercises fails the root cause can be different. Here are some examples:

Possible error scenario Possible root cause Solution

The URL

https://demo.perfectomo

bile.com is not available

anymore.

The demo environment is not

available anymore and the

production environment can be

used in a trial mode.

Use another URL in the exercises and Visual Studio

Code:

Use the production environment URL:

https://www.perfectomobile.com.

The test fails with an

error due to a device

error.

The referenced mobile device in

the exercise

- doesnt exist anymore.

- is disconnected.

- has a new device ID.

- is locked or blocked by

another user.

Validate the mobile device status:

Please check the portal https://demo.perfectomobile.com

in the area Interactive / Select Device and validate the

device status.

If the needed device has an issue use another available

device. For the exercise the device ID (DUT), device name

and OS name (os) are needed and must be changed. Then

retry the exercise / test execution.


error due to an app error.

The ATM app is not installed on

the selected device.

Validate the app status on the mobile device:

If the ATM app doesnt exist on the mobile device please

contact the Perfecto Mobile support.


error due to an OS

update pop up on the

device start screen.

A pop up message on the device

shows that an OS update is

available. So, the tests script

cannot select the ATM app icon

anymore.

Validate the mobile device screen:

Close the pop up message without executing the OS

device update and retry the exercise / test execution.

Table 1 Possible test execution error scenario.

For additional help please contact the Perfecto Mobile support: http://www.perfectomobile.com.


Page 6 of 49

Device testing approach for IT

Applications

Need for Device Testing The emergence of smartphones and tablets has changed the dynamics of internet usage. As per the latest market

research, over 40% of the users are accessing internet applications using devices other than PC. Success of the web

applications is now heavily dependent on the quality of service (QoS) on both PC and devices. We are now

observing similar patterns with regard to enterprise applications. The culture of BYOD (bring your own device) is

being welcomed by several major enterprises worldwide. This has led to interesting challenges and opportunities,

one of them is huge demand for Device testing, for ensuring consistent experience across devices.

Given the dynamics of the IT applications usage, testing approach must also undergo surgical changes. In this

paper, we highlight the changes needed to testing approach and present the approach for device testing.

Figure 1 - PC verses Devices Spread across verticals


Page 7 of 49

Challenges in Device Testing There are now over 2000+ types of devices in the market and it is not feasible from both cost and time perspective

to test IT applications on every device type. This presents a significant challenge of identifying a minimum set of

devices which would represent the majority of the devices. Also, depending on the application usage, locale and

end-user persona, the minimum set for testing of devices can vary. For example, if a company in Australia wants

to adopt BYOD culture, then minimum set of devices on which an enterprise IT application needs to be device

tested would depend on the devices widely used by the employees in Australia. Hence, identifying the minimum

set of devices is key for successful device testing.

There will be a single testing strategy for PC, mobile and tablets, which optimizes the testing cycle from cost,

quality and time perspective. There is another school of thought that considers strategy for traditional testing on

PC and testing on devices should be separate. While there are some advantages of such approach, we have

observed that, a single testing strategy for both traditional testing on PC and testing on multiple devices is cost

efficacious.

Specially, we have observed following advantages:

Reducing the duplication of test cases

Reuse of test data

Effective sharing of test resources and test artefacts

Device Testing Lifecycle for IT Applications (DTL-IT)

Figure 2 - Device testing life cycle for it applications

Test Spread: Define the % effort in emulators, manual and cloud-based testing.

Envision Phase

How to select the minimum set of devices to be device tested?

Selecting the minimum set of devices that needs to be device tested is the most challenging aspect of device

testing. This would depend on factors such as platform to be supported, business usage of the application and

market data consideration for device market. For a typical web application that will be consumed on multiple

devices would have the following device priority matrix:

Envison

Obtain Customer requirements specific to usage of the application on different devices

Obtain customer requirements on the device spread

Obtain NFR

Plan

Define scope

Conduct Automation Assesement/ROI

Define test spread based on schedule

Create Device Testing Strategy

Define Test Sceanrios

Build

Create device automation framework

Create Test Cases

Stabilize

Execution of test cases on emulators/real devices based on the test spread

Run the Tests

Publish the Test Report


Page 8 of 49

Priority

High Priority Platform (OS)

Browser (IE, Firefox, Chrome, Safari, etc.)

Form Factor (7'', 10'', 4'', etc.)

Medium Priority Input Mechanism (Stylus, Swipe, Camera, GPS, etc.)

Network Connection (GSM,CDMA,BES* -BB, Edge,2G,3G,4G,Wi-FI)

Low Priority Memory

CPU Type (ARM, Intel, etc.)

Table 2 Factors to be considered for device test mix

Depending on the usage of your application, define the device priority matrix and arrive at a set of devices on

which testing needs to be performed. Make sure that, you are choosing only those devices that are critical to your

business. This activity needs to be done during the planning phase. Also, given the dynamic nature of device

market, if time period between planning phase and SIT phase is significant, then re-evaluate the device priority

matrix on every iteration, so that you are including the new devices that have just arrived to the market in your

test mix.

Another strategy for testing the application on multiple devices is to do Beta Release of the application and do

dog-fooding.

Plan Phase

Figure 3 - Testing types across various layers for optimizing the efforts

Following are the key activities that needs to be performed during Plan phase:

Define User Scenarios for the IT application


Page 9 of 49

Gather the cross platform / browser mix requirements of the scenarios and personas accessing those

scenarios which will mimic the real life usage profile mix of the application.

o Ex: 20 % of the total scenarios are accessed through mobile, 10 % through tablets and rest 70%

over desktop/laptop

o Mobile Spread is further split across Safari on IPhone 5 (60%) ,Android 4.0 /Chrome (35%),

Widows 8/ IE10 (15%)

o Desktop spread across Windows 8/Chrome, Windows 8 /Safari, Windows 8 /IE 10

Identify the scenarios that are critical to run on real devices like must /should /nice to run which will help,

propose the right mix to the customer.

Identify the device/browser spread across the customer user base if the customer does not have this

information, get the top N devices that are widely used in the region to help the customer to make the

right choice

Plan for any upcoming device/browser launches and make provision to accommodate these mix to

support agility

Scope the devices/browser mix for the project based on the schedule /availability of devices and budget

of the project.

Scope the test spread mix of devices /emulators/browsers ideal for the engagement based on the above

requirements

Calculate the ROI based on coverage (device spread) versus risk of not testing across all spread

Prepare the test strategy based on the above

Device support strategy

1) How many of the platform and devices should we support?

2) Business Considerations

3) Market data considerations

4)Failing gracefully when a device is not supported

Device Configuration Matrix

Device Model

Input mechanism (Eg:Touch,Key)

Screen Resolution /Orientation

Chipsets/Peripherals (GPS,Camera,Microphone)

Carrier

Locked /Un Locked

Upgrade/Install type (ROM.,OTA, App store, Tethered-kiosk scenario)

Network Type (GSM,CDMA,BES* -BB)

Network Connection (Edge,2G,3G,4G,Wi-FI)

Operating System (Windows/Android/Blackberry)

Browser

Recommendations

1) First Test on the major Desktop browsers to ensure that the system is functionally stable (VS 2012 Coded

UI Selenium support will help for Automation)

2) Run the tests on major platform/devices Emulators (Automation/ Manual based on ROI)

3) If the devices are available then test on real device. Set of Reference devices in-house shared by the team.

Test at least once all the scenarios in the reference devices manually


Page 10 of 49

4) Breadth device testing done by external party where the efforts are more .Needs to identify partners with

this capability.

5) Cloud based mobile testing tools can be used in case the device spread is huge and one wants to

automate once and run the tests across multiple devices over the cloud on a real device. This eliminates the

maintenance overhead of managing the devices internally and optimizes costs by eliminating manual

execution of tests.

Real /Virtual Mobile Testing Decision Table

Table 3 Mobile testing decision table

Based on the combinations identified one can make use of combinatorial algorithms to decide the optimum

combinations of tests to be designed to ensure greater coverage and eliminate risks.

Device Testing Cost Pyramid

Figure 4 - Device testing Cost Pyramid

Emulators/Browser plug in based testing will be the most cost effective solution. The above pyramid helps make

the decision based on the cost factor. This also depends on the number of devices in our test spread and the

Virtual Mobile Testing Tool

Regression Test ,BVT

Sanity

Compatibility

Large set of Functional Testing scope

Wide Range of devices available for testing .Script once run on any device

Devices

UAT

Compatibility

Inter operability

Interrupt Test

Performance

End to end scenario


Page 11 of 49

total number of tests to be executed. In case both variables are low then real device testing will work out to be

more economical compared to cloud based testing.

Test Automation Tools for Device testing

Key requirements for the Device Test Automation Tool:

Should be able to port the automated scripts of one platform to multiple platforms and multiple devices

.Script once run anywhere (Device anywhere/PerfectoMobile)

Speed up test execution

Eliminate manual efforts to reduce total cost

Easy to maintain

Should be extensible to support future platforms and devices with ease.

Should be able to run the tests on emulators as well as devices on demand (Nice to Have on Devices as

Running tests on real device takes lots of time and can be done only in serial fashion)

Resilient and smooth execution of automated tests: This is very important as no one would like to have

an automation that is not stable and needs frequent monitoring for failures. Automation framework

should be able to give consistent and predictable results for every run.

Automation guidelines

Automation on Virtual Device is recommended for the following areas

1) Any scenario that can be run on an emulator can be automated and can be tested without the real

device .Identify the regression suite for the functional test cases and we can automate those scenarios

for all the gestures to ensure that the major risks and coverage issues are addressed as part of early

iteration testing and regression tests are automated as part of iteration scope .Majority of the emulators

comes free.

2) Automation on Real device is recommended in early stages of the application life cycle to detect and fix

the bugs early.

3) Anything that can be run on a emulator for all the top 4 platform/devices

4) Performance and Stress testing on actual devices

5) Some Configuration testing on Emulators

6) Standard out of the box compliance check/design guidelines check

Automation is Not Recommended for the following areas

1) Configuration testing

2) UI look and feel verification testing

3) E2E Testing

4) Native App automation (Assuming not many complex functionality is delivered, Automation wont be

recommended)

5) Scenarios which are already addressed through automation for desktop access

6) Usability testing

Build Phase Following are the key activities that needs to be performed during Build Phase:

Identify the test scenarios based on functional and non-functional requirements


Page 12 of 49

Make sure that, you are identifying User Scenarios that include gestures such as

Touch

Touch Zones

Swipe

Pinch Zoom

Identify the priority scenarios which can be automated based on the return on investment (ROI)

Useful tools for developers to catch bugs early

Tool Name Description Link

Device Test Planning

Helps plan the number of

devices and platforms to be

used in testing.

Planning tool is here

W3C mobileOK Checker

The W3C mobileOK Checker is

one of the best tools available

to you, to help you test the

efficacy of your Website on

mobile devices. This tool

performs several tests on a

Web page before it estimates

the level of compatibility of

your Website with the mobile

Web. W3C has developed a

mobileOK Basic Tests 1.0

specification, which works to

give you a clear idea of the

mobile-friendliness of your

Website.

W3c Mobile OK

Iphoney

A very accurate iPhone tester,

this is also available to you

completely free of cost.

Though iPhoney is not really a

simulator, it enables you to

build 320x480px Websites,

compatible with the iPhone

screen specifications. This

implies that you can test both

your code and your Website

images in the actual Apple

Safari type of environment,

including all the features of the

original iPhone, such as zoom,

plugins, landscape and portrait

modes and so on.

Iphoney


Page 13 of 49

Gomez The Gomez mobile readiness

test analyses your Website on

the basis of over 30 well-

established, fool proof, mobile

Web development techniques.

It then rates your page on a

scale ranging between 1 and 5

points. This tool not only offers

you more-or-less accurate

results, but also offers advice

on how you can further

improve your site in order to

make it even more compatible

with mobile browsers. Make

note of the fact that this tool

first requires you to enter some

personal information, before

you can go ahead and use it.

Gomez

Table 4 Device testing tools

Device Usability Testing

Different elements of the form factor influence the way the web application runs on the device. For example, the

following factors are crucial when it comes to testing web application on modern devices:

a. Screen Orientation

The screen orientation Landscape / Portrait dictates how the application is shown on the browser.

Scenarios must be identified which test the application in different orientations and look out for issues

resulting from page overflowing from the screen, elements not being visible to the user etc. Also, if the

application uses change in orientation as a trigger, that also needs to be exercised.

b. Responsive design

Modern web applications are adopting Responsive Design practices to retain homogenous look and feel

across devices. The main objective of responsive design is to ensure that the same web application when

opened across different devices of multiple form factors, provides similar user experience on all the

devices, regardless of the screen size and resolution. This is accomplished by a combination of CSS and

JavaScript which render the width and layout of elements in such a fashion that all elements are visible

on the screen.

c. Control Replacements

There may be some controls on the web page which may be taking mouse or keyboard exclusive

controls, for example mouse over events, mouse wheel events, pressing escape button etc. When the

web application is run on devices, there should be alternative ways of triggering controls, for example - a

drop down triggered by mouse over could be displayed as a select triggered by touch.

All such controls need to be identified and be tested on devices.

Combinatorial Testing to generate effective number of scenarios

While testing for devices, we could potentially encounter large number of test combinations arising from

multiple device form factors, screen resolutions, screen sizes, OS and browser types etc. It may not be practical to

run all tests on all possible combinations. Hence, it is recommended to come up with a condensed subset of


Page 14 of 49

scenarios which provides maximum test coverage using Combinatorial Testing. Combinatorial test generation

technique is based on the premises that defects occur in a system due to interaction between few numbers of

components (empirically less than 6). If we find all possible combination of components taken 6 at a time, then

we can uncover cover more than 90% of total number of bugs.

There are various tools available which allow users to generate combinatorial scenarios; PICT, ACTS to name a

few.

Security Testing on Devices

The following aspects of security needs to be tested on devices:

1. Authentication:

Apart from the standard authentication methods the web application exposes, there could be device

specific authentication methods like swipe gestures, Google account, Facebook account, device tokens,

device certificates etc.

2. Authorization:

A few aspects of authorization that could be sensitive from devices perspective could be elevation of

privileges, escaping License checks, insecure HTTP methods that could be tampered over network etc.

Depending on the OS, there could be issues with file permissions on the local system

3. Data Storage

Web applications may use local storage for cache management, temporary downloads etc. Appropriate

security testing needs to be done to ensure that this data does not contain any sensitive information and

cannot be accessed by unauthorized parties.

4. Generic Web application security

Security test cases should be executed on the devices which exercise the application vulnerabilities which

exploit the client side, for example:

XSS and HTML Injection

Command Injection (if the application utilizes a shell)

CSRF

SQL Injection

Cookies

HTML5

Stabilize Phase Following are the key activities that needs to be performed during the Stabilize Phase:

Execution of test cases on emulators and real devices as per the Test Spread

Run the Tests

Publish the Test Report

Native App Best Practices Native app testing and automation is no different than the webapp ,but we may have to script for each platform

separately .We have some best practices for native application to ensure that native application that one build

shines in the market place.


Page 15 of 49

1. Application start up Performance App should be responsive Indicate this using some progress bars

2. Application Resumption- Calls, Messages, Back button Click- Recovery handling needs to be good for

example in the middle of browsing if the user gets the call ,User should be shown the same content that

he was browsing before the call.

3. Consistent User Experience Back button- First page back button should exit the app, for subsequent

pages, it should show previous page. Dismissing the dialog should be supported by back button

4. Battery Drain Testing- Overall app should be benchmarked against some existing app in similar category

for Battery drain. E.g.: Social networking app should be benchmarked against a Social networking app.

Subsequent improvement can be done using battery drain tests.

5. Network handling Scenarios: 2G, 3G, Wi-Fi, Wi-Fi with proxies etc.

For example: App should not hang when the network coverage is lost and should provide the right

exception message back to the user.

6. Error Handling should be very meaningful. Not many mechanisms to troubleshoot issues. So, error

messages should be very meaningful.

7. For apps where huge content needs to be added, recommend fetching it after installation. This will help

the build to be smaller in size and help more market penetration as users wont download something

which is bit heavy.

8. Application should not intervene into the platform features functioning. It should not override the

platform capabilities and features. E.g. Location services setting of the app should not override the

location services setting in the device

9. Application should not compromise on the security of data and other security features like XSS scripting

10. Landscape and Portrait mode should be supported

11. Application should work fine in different themes and background colors


Page 16 of 49

Environmental preparation In this section, we will cover the pre-requisites and installation of the MobileCloud for TFS template for Visual

Studio. Your options are to install the MobileCloud for TFS to your desktop where you have Visual Studio

installed or to a Virtual Machine with Visual Studio. MobileCloud for TFS integration with unit testing does not

require TFS. MobileCloud for TFS integration with Microsoft Test Manager does require TFS. The following

guidance will at times seem like a hands on lab, however, some detail is needed as we explain what is needed to

use the MobileCloud for TFS with Visual Studio and TFS.

In order to run Perfecto Mobile scripts from your development environment, you first install the Perfecto Mobile

Cloud for TFS Test Template. The installer creates a Project Template on C:\Program Files (x86)\Microsoft Visual

Studio 11.0\Common7\IDE\ProjectTemplates\CSharp\Test and places Perfecto Mobile assemblies on C:\Program

Files (x86)\PerfectoMobile\Visual Studio. This location is configurable during installation. Test projects created

using the Mobile Cloud Project template use the installed TFS Adapter to make calls to the Perfecto Mobile

service.

Installing the Perfecto Mobile Team Foundation Server (TFS) add-

in

Pre-Requisites

Visual Studio 2012 + Update 2 or higher

Microsoft Test Manager 2012 (required for integration with Test Case)

TFS 2012 Update 2 (if on-premise deployment) or Team Foundation Service3 in the cloud

SQL Server 2008 / 2012 (if on-premise deployment)

Perfecto Mobile Account - https://www.perfectomobile.com/trial/sign_up

NO

TE A user account is required to use the MobileCloud for TFS. This guidance and the HOL exercises assume you have

your own account.

For additional information, you may access the following resources:

- Microsoft Test Manager 2012: http://msdn.microsoft.com/en-us/library/vstudio/dd380763.aspx

- Team Foundation Server 2012: http://msdn.microsoft.com/en-us/vstudio/ff637362.aspx

Performing an Local Installation

Retrieve the installation package perfectomobile-tfs-add-in-installer-1.0.exe from

https://www.perfectomobile.com/support/tfs and run the installer, confirming the User Account Dialog box if it is

displayed. You will have the option to change the installation location folder if you wish.

Once the Perfecto Mobile Installer completes detailed instructions on creating a test class & method to execute

against the Perfecto Mobile cloud can be viewed here: http://aka.ms/perfectomobiletesttemplate

3 http://tfs.visualstudio.com


Page 17 of 49

Installing to a Hyper-V Virtual Machine

Similar, to a desktop deployment, the Perfecto Mobile TFS add-in template solution can be deployed to a virtual

machine running in Hyper-V as long as an active network connection is provided to enable access to the Perfecto

MobileCloud.

Installing to the Brian Keller Virtual Machine

If you are using Brian Kellers Visual Studio ALM Virtual Machine 4 to create and execute test scripts using the

Perfecto Mobile TFS Add-in, additional configuration steps are required to set the current date and time and

configure internet connectivity. All software pre-requisites are installed on the Brian Kellers Visual Studio ALM

Virtual Machine.

Enabling Internet Connectivity

Perfecto Mobile is a cloud-based service so test cases must be executed from an internet-enabled machine.

To configure internet connectivity on an instance of Brian Kellers Visual Studio ALM Virtual Machine, an external

virtual switch with internet connectivity should be present in the Hyper-V manager. The ALM Virtual Machine

should then have a network adapter associated with the external virtual switch added to it. Further information

on creating a virtual network and adding a network adapter can be found here: http://aka.ms/configurevm

Synchronizing the Date and Time

Brian Kellers Visual Studio ALM Virtual Machine is configured with Time-synchronization disabled in order for

the sprint backlogs and reports to work properly. This causes an SSL exception when connecting to the Perfecto

mobile service: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS

secure channel.

In order to fix this issue, the VM should be set to synchronize time with the host. Follow these steps to enable

time-synchronization:

1. On the Virtual Machine settings, select the Integration Services tab

2. Check the time synchronization checkbox.

Once these steps have been completed successfully, the Perfecto Mobile TFS Add-in can be installed following

the instructions detailed in the Local installation section.

Accessing the Perfecto Mobile Cloud Platform This section will address the process and steps to access the Perfecto Mobile Cloud platform for a new or an

existing Perfecto Mobile user. The Perfecto Mobile cloud platform is accessible via the URL

https://demo.perfectomobile.com.

Proper login credentials are required to work with Perfecto Mobile TFS add-in and gain access to the

aforementioned URL. You register for a user account at the Perfecto Mobile at the

https://demo.perfectomobile.com site.

To use the Perfecto Mobile service, you have two options:

As a new user

As an existing user

4 http://aka.ms/almvms


Page 18 of 49

New User Access

New Users must register to gain access to the demo site. Access Registration by entering the URL specified

above in a browser, and then navigate to the Click Here link as shown below.

Figure 5 - New User Registration link.

This link takes you to Perfecto MobileCloud registration page where you supply your registration information.

NO

TE Make note that the web page screen shots of the partners web site is subject to change from what we have shown

you in this guidance.


Page 19 of 49

Figure 6 - Mobile Cloud Platform Registration.

Once the I Accept button has been clicked, the relevant login credentials will be sent to the supplied email

address. User credentials are activated and can be used immediately.

Existing User Access

If you have an existing Perfecto Mobile account then login via the Perfecto Mobile URL as detailed above. After

entering your credentials and clicking Sign In, the landing page will display.


Page 20 of 49

Hands-On Lab If you are intending to follow the Hands-on Lab (HOL) step by step, use these times for each exercise as a

guideline. If, however, you are intending to investigate each step in detail double the times as a bare minimum.

NO

TE

This hands-on lab includes the following exercises:

1. Accessing the Perfecto Mobile Cloud and running an automated test

2. Run an automated test in the Perfecto Mobile Cloud within Visual Studio

3. Run an automated test in the Perfecto Mobile Cloud within Microsoft Test Manager

4. Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Team Build on TFS

5. Continuous Integration - Run an automated test in the Perfecto Mobile Cloud within Build Service on Team

Foundation Service in the cloud

Estimated time to complete this lab: 55 minutes.

Exercise 1 Accessing the Perfecto Mobile Cloud and running an

automated test (10 min)

GO

AL

In this exercise you will use the perfecto mobile cloud, to run a predefined automated test against the Bank of

America app on a specific mobile device. This automated script will run a functional test.

Persona Mapping

This exercise is focused on Christine as a tester.

Exercise

If using the Brian Keller TFS 2012 Demo Virtual Machine5, you can log on as Julia Ilyiana with the standard

password P2ssw0rd.

1. Open Internet Explorer and browse to the URL https://demo.perfectomobile.com.

2. Log on with your username and password.

NO

TE If you dont have an account, please request your own demo account under:

https://www.perfectomobile.com/trial/sign_up.

3. Click on Automation in the menu bar.

NO

TE When an information window New in Automation! appears, click OK.

5 http://aka.ms/BrianKellerAlmVms


Page 21 of 49

4. Click File and Open to open an existing automation script:

Figure 7 - Switch to the automation area.

5. Select folder Public Scripts\Microsoft\ and open the script search_for_atm.

6. The file is a script for test automation and contains different automated test steps:

Figure 8 - List of automated test steps.

7. Click Running View to switch to the test steps execution status view.


Page 22 of 49

8. Click on Run in the menu bar to start the automated test.

9. Select the device Apple iPhone-4 for the field DUT, use the value iOS in the field OS to run the test on

this specific device and click Run:

Figure 9 - Device selection popup window.

10. The test execution starts and the status for each test step :

Figure 10 - Running view with test script status list.

11. Click Watch in the menu bar while the test is running to switch to the Devices Live video view:


Page 23 of 49

Figure 11 - Devices Live video test execution view.

NO

TE The test script runs through the sign in process of the Mobile Banking app and finds the ATM location nearest to a

specific zip code.

12. When the test script completes successfully a test report is available. Click Yes to review the report

results.


Page 24 of 49

13. Click on General to check the test run result summary. There are information available about status, test

duration, tested device, etc.:

Figure 12 - General view of the test run results.

14. Click on Execution Sequence to review the individual test steps and recorded screenshots. Scroll down

to the last Text checkpoint to verify, that the expected text values Expected text (needle): sign in was

found in the loaded app page and that the elapsed test step time didnt reach the set value for Timeout:

40 sec.


Page 25 of 49

Figure 13 - Execution sequence view for all test steps.

15. OPTIONAL: Click also on Errors to see if any errors occurred during test execution (should be empty in

our case).

16. OPTIONAL: Click also on Statistics to get an overview about the passed vs. failed Checkpoints and

Actions.

REV

IEW

This exercise you ran an automated functional test against the mobile banking app provided by the company Bank of

America. For testing you used the perfecto mobile cloud. You have learned how to open an existing test script and

run that automation scripts against a selected mobile device. The generated test run report enabled you to verify the

test run in detail.


Page 26 of 49

Exercise 2 Run an automated test in the Perfecto Mobile Cloud

within Visual Studio (15 min)

GO

AL

In this exercise you will run a predefined automated test against the Bank of America app on a specific mobile device

using Visual Studio. You will code an automated test that will access the perfecto mobile cloud and will run an

automated test script as a functional test.

Persona Mapping


Exercise


password P2ssw0rd.

1. From within Visual Studio, create a new project via Ctrl+Shift+N.

2. Select the Mobile Cloud Project template for Visual C# in the folder Installed -> Templates ->

Visual C# -> Test and click OK:

Figure 14 - Mobile Cloud Project template.

NO

TE If dont see the Mobile Cloud Project template, please install the perfectomobile-tfs-add-in-installer-1.0 extension

for Visual Studio 2012.

3. Open code file MobileCloudTest1.cs and overwrite the string values for mobileCloudServer, user

and password in the public class MobileCloudTest1 with the following values:

// URL of the perfecto mobile cloud static String mobileCloudServer = "https://demo.perfectomobile.com/"; // user name, used for logon static String user = ""; // password, used for logon static String password = "";

Code 1 Parameters to initialize a connection to the perfecto mobile cloud.



Page 27 of 49

These parameters are necessary to connect successfully to the Perfecto Mobile Cloud during the test run.

N

OTE If you dont have an account, please request your own demo account under:

https://www.perfectomobile.com/trial/sign_up.

Overwrite the string values scriptKey and DUT and add a parameter os in the test method ExecuteScript1(): // path to the automate test script in XML format String scriptKey = @"PUBLIC:Microsoft/search_for_atm.xml"; // Device ID of the emulated mobile device String DUT = "C1C30569F52259AD3E081816C30E54457DA55400"; // expected OS of the emulated mobile device String os = "iOS"; Dictionary runTimeParameters = new Dictionary(); // if needed, add additional runtime parameters below the DUT parameter runTimeParameters.Add("DUT", DUT); runTimeParameters.Add("os", os); // execute script with mobile cloud adapter adapter.Execute(scriptKey, TestContext, runTimeParameters, MobileCloudAdapter.REPORT_FORMAT_HTML);

Code 2 Coded start parameters for the test method ExecuteScript1.

NO

TE The value for the parameter DUT means the Device ID. You can find this Device ID in the perfecto mobile cloud

portal in the list of devices as a separate column.

4. Build the solution with Ctrl + Shift + b and select TEST -> Windows -> Test Explorer in the menu

to open the Test Explorer. Now you can see the test method ExecuteScript1 in the Test Explorer.

5. To run the test method right click on ExecuteScript1 and select Run Selected Tests:

Figure 15 - Test Explorer: Run test method ExecuteScript1.


Page 28 of 49

6. OPTIONAL:

While the selected test is running it is possible to view the script execution on the real mobile device

in the Perfecto Mobile Cloud. To see the execution in real time open Internet Explorer and browse to

the URL https://demo.perfectomobile.com/nexperience/dashboard.jsp?autoWatch=true.

Figure 16 See test execution on the real device in the cloud dashboard.

7. After a while you will see the test result. The test method ExecuteScript1 passed. Click on the link

Output to open the test result in a separate view:

Figure 17 - Test Explorer: Passed test method ExecuteScript1.


Page 29 of 49

8. Click on the link in the area Attachments PUBLIC_Microsoft_search_for_atm

Figure 18 - Test Output: Test method ExecuteScript1.

and you will see a html based test execution report:

Figure 19 - Html-based test execution report summary.

9. Scroll down to see the execution time for each test step in the area Execution Sequence:

Figure 20 - Html-based test execution report with execution sequences.


Page 30 of 49

NO

TE If the test fails, the html based test execution report shows which execution sequences failed.

10. Scroll down to see the device information in the area Devices and click on the link Video ():

Figure 21 - Device information.

11. A video starts and shows the recorded test execution:

Figure 22 - Recorded test sequence video window.

REV

IEW

In this exercise you ran an automated functional test within Visual Studio against the mobile banking app provided

by the company Bank of America. For testing you used the perfecto mobile cloud. You have learned how to create a

new Visual Studio Mobile Cloud project. You modified the provided test code to execute an existing test script and

run that automation script against a selected mobile device. The generated html based test sequence report enabled

you to verify the test run in detail. The recorded test execution video was helpful to visually verify the test execution

on the mobile device.


Page 31 of 49

Exercise 3 Run an automated test in the Perfecto Mobile Cloud

within Microsoft Test Manager (10 min)

GO

AL

The Perfecto Mobile Cloud can also be accessed from within a Microsoft Test Manager (MTM) test case. In this

exercise, you will create a test case and enable automation using the unit test you created in exercise 2 so that you

can see the end to end ALM integration for mobile device testing.

NO

TE Microsoft Test Manager requires integration with TFS where unit testing does not. The test project and unit test file

created in Exercise 2 will need to be checked into your TFS repository so that the test plan you create can access the

unit test.

Persona Mapping


Exercise


password P2ssw0rd.

1. From within Microsoft Test Manager, under the Plan tab, create a new test case.

2. Enter the title of the Test Case. There is no need to add any Test Steps.

Figure 23 - Create a new test case.

NO

TE The automation status of the Test Case is set to Not Automated. You will need to use Visual Studio to add the

associated automation as listed in the following guided steps.

3. After saving your test case, open Visual Studio and query the test case previously created.

4. Navigate to the Team menu item in the menu bar within Visual Studio, click on Go to Work Item

and provide the id number for the test case that was created in Microsoft Test Manager.



Page 32 of 49

5. Once the test case has opened, navigate to the ASSOCIATED AUTOMATION tab as shown below:

Figure 24 - Test Case associated automation.

6. Click on the ellipse button. The test name previously created should appear. Highlight the test name

and click OK as shown below.

Figure 25 MTM: Associate the test case with the test script.


Page 33 of 49

7. After saving the test case, the Unit Test will now be associated with the Test Case.

Figure 26 MTM: Test case details.

NO

TE You will now need an Environment to run the test case with automation. If you do not have an Environment, you will

need to create one. If you do not have an Environment configured, the following steps will guide you through the

process of creating a new environment to run the Test Case.

8. Within Microsoft Test Manager, navigate to the Lab Center, select the Lab tab, and click New:

Figure 27 MTM: Create a new environment.


Page 34 of 49

9. Select Standard Environment and provide a suitable name. Then click Next:

Figure 28 MTM: Set environment name.

10. In the Machines tab, select Add Machine.

11. Provide the name of the computer from where the tests will run.

12. Follow the steps and click Verify:

Figure 29 MTM: Add machine

13. Once verification is complete, a new Environment is created and initialized in a running state.

14. The Environment can now be used by associating it in the Test Settings tab in the Lab Center.

15. Within Lab Center, select the Test Settings tab.


Page 35 of 49

16. Click New, provide the Name and select the tests type for Automated as shown below:

Figure 30 - MTM: Configure Environment with Test Settings.

17. Click Next to navigate to the Roles tab and select the Environment created previously.

18. Click Finish to create a new Test Setting. The completed Test Setting is shown below:

Figure 31 MTM: Completed Test Settings.

19. Next, navigate back to the Test Center and click on the properties in the Plan tab.

20. Select the Test Settings and Test Environment created previously.

Figure 32 MTM: Plan properties.


Page 36 of 49

21. Now click on the Modify link under Builds: Build in use section of the properties, click the Assign to

plan button to associate the build triggered in the Visual Studio steps previously.

The completed properties tab should look as follows:

Figure 33 MTM: Completed Properties tab.

22. Navigate to the Test Lab in Test Center and under Run select Run with options. Pick the selection that

was created in the properties section and click Run as shown below.

Figure 34 MTM: Select run options.


Page 37 of 49

23. OPTIONAL:

While the selected test case is running it is possible to view the script execution on the real mobile

device in the Perfecto Mobile Cloud. To see the execution in real time open Internet Explorer and browse

to the URL https://demo.perfectomobile.com/nexperience/dashboard.jsp?autoWatch=true.

Figure 35 See test execution on the real device in the cloud dashboard.

Allow enough time for the test to pass depending on your network connection. A successful test run is shown

below.

Figure 36 - MTM: Successful test run.

The Perfecto Mobile Cloud for TFS add-in will attach the test results to the test case run. Once again, the results

are an html file showing the sequences as they were executed within the Perfecto Mobile Cloud. This is shown

below:


Page 38 of 49

Figure 37 MTM: Html-based test execution report with execution sequences

REV

IEW

In this exercise you created a test case and enabled automation of the functional test within Visual Studio against the

mobile banking app provided by the company Bank of America. For testing you used the perfecto mobile cloud. You

have learned how to integrate a test created in a Visual Studio Mobile Cloud project with Microsoft Test Manager.

The generated html based test sequence report enabled you to verify the test run in detail.

Exercise 4 Continuous Integration - Run an automated test in

the Perfecto Mobile Cloud within Team Build on TFS (10 min)

GO

AL

The Perfecto Mobile Cloud can also be accessed from within a Team Build. In this exercise, you will create a new

build definition and enable testing within Continuous Integration using the unit test you created in exercise 2 so that

you can see the end to end ALM integration for mobile device testing during the build process.

NO

TE

Continuous Integration requires integration with TFS. The test project and unit test file created in Exercise 2 will need

to be checked into your TFS repository so that the Team Build can access the unit test. Follow the steps from the

Exercise 2 on the Hands-on Lab first to create a Mobile Cloud Project in Visual Studio and start with this Exercise.

Persona Mapping


Exercise


password P2ssw0rd.

Before running Perfecto Mobile test cases as part of a Continuous Integration build with Team Foundation

Server, the solution from Exercise 2 must be checked-in to TFS Source Control by following these steps:

1. From Solution Explorer, Right-click on the solution.



Page 39 of 49

2. Select Add Solution to Source Control:

Figure 38 Adding the Solution to Source Control.

3. On the Add Solution to Source Control dialog, select a Source Control Path and click OK:

Figure 39 Selecting the Source Control path.

4. Right-click on the solution again and select Check In.


Page 40 of 49

5. In the Pending Changes tab of Team Explorer, enter comments and click on Check In:

Figure 40 Checking in the solution.

6. Once the solution is checked in, create a new build definition by going to the Builds tab in Team

Explorer and clicking on New Build Definition:

Figure 41 Creating a new Build Definition.

Provide a Name for the new Build Definition and select all the defaults. On the Process tab, verify that your

solution is set as the value for Projects to Build. Expand the Automated Tests section in the build definition,

then Test Source and make sure that the Test Source Spec argument includes the name of your test

assembly. By default, this value is **\*test*.dll. If you selected the default name when creating the Perfecto

solution, your assembly name will be Mobile Coud Project1, which would not be picked up by the build

process. You should update this value to **\**.dll.


Page 41 of 49

Figure 42 Setting Build Definition Arguments.

You may now queue the build and your Perfecto Mobile test case will run during the build.

Figure 43 Build Results, including completed test case.

Exercise 5 Continuous Integration - Run an automated test in

the Perfecto Mobile Cloud within Team Build on Team Foundation

Service in the cloud (10 min)

GO

AL

The Perfecto Mobile Cloud can also be accessed from within a Build based on Team Foundation Service in the cloud.

In this exercise, you will create a new build definition and enable testing within Continuous Integration using the unit

test you created in exercise 2 so that you can see the end to end ALM integration for mobile device testing during

the build process. The difference between this Exercise 5 and the Exercise 4 before is, that Continuous Integration

will now run in the cloud.

NO

TE Continuous Integration requires integration with TFS. The test project and unit test file created in Exercise 2 will need

to be checked into your TFS repository so that the Team Build can access the unit test. Follow the steps from the

Exercise 2 on the Hands-on Lab first to create a Mobile Cloud Project in Visual Studio and start with this Exercise.

Persona Mapping


Exercise


password P2ssw0rd.

When you install Perfecto Mobile on a development machine, a set of assemblies get copied to C:\Program

Files (x86)\PerfectoMobile\Visual Studio. Since Perfecto Mobile cannot be installed on TFS Service, those



Page 42 of 49

assemblies should be included as part of our solution so that they get included with the build and are available

when the test runs during the build.

The assemblies needed are: PerfectoMobile.MobileCloudHttpClient.dll and PerfectoMobile.TFS.Adapter.dll.

Figure 44 Adding Perfecto Mobile assemblies to a solution.

Before running Perfecto Mobile test cases as part of a Continuous Integration build, the solution must be

checked-in to TFS Source Control by following these steps:

1. From Solution Explorer, Right-click on the solution and select Add Solution to Source Control:

Figure 45 Adding the Solution to Source Control.


Page 43 of 49

2. On the Add Solution to Source Control dialog, select a Source Control Path and click OK:

Figure 46 Selecting the Source Control path.

3. Right-click on the solution again and select Check In.

4. In the Pending Changes tab of Team Explorer, enter comments and click on Check In:

Figure 47 Checking in the solution.

5. Once the solution is checked in, create a new build definition by going to the Builds tab in Team Explorer

and clicking on New Build Definition:


Page 44 of 49

Figure 48 Creating a new Build Definition

Provide a Name for the new Build Definition and select all the defaults. On the Process tab, verify that your

solution is set as the value for Projects to Build. Expand the Automated Tests section in the build definition,

then Test Source and make sure that the Test Source Spec argument includes the name of your test

assembly. By default, this value is **\*test*.dll. If you selected the default name when creating the Perfecto

solution, your assembly name will be Mobile Coud Project1, which would not be picked up by the build

process. You should update this value to **\**.dll.

Figure 49 Setting Build Definition Arguments.

You may now queue the build and your Perfecto Mobile test case will run during the build.

Figure 50 Build Results, including completed test case.


Page 45 of 49

Conclusion We have shown you how easy it is to integrate the Perfecto Mobile Cloud for TFS with Visual Studio. Testing of

your mobile application running on real devices in the cloud is possible as strictly unit tests or as automation

tests associated to a test case in Microsoft Test Manager. These integration points allow these tests to be run as

part of a build definition for daily, manual, or continuous integration builds. This integration provides the test

status reporting features, like status of tests (pass/fail) along with the failure reason.

This document does not cover the setup details required to develop for iOS, Android, or Windows Phone. Team

Explorer Everywhere 10 is required to enable developing and building Android and iOS applications. For an

overview of the lifecycle with Mobile Cloud see the overview poster for this ALM Ranger project.

10 http://msdn.microsoft.com/en-us/library/gg413285.aspx


Page 46 of 49

Appendix

Automation Tool Comparison

Feature SeeTest from Experitest PerfectoMobile

Supported Mobile OS Android

iOS

Blackberry

WindowsPhone8

All emulators are supported

Android

iOS

Blackberry

Remote Access See Test does not provide

devices.

Connect all mobile devices to

one central hub/server that

resides in your organization to

form private cloud

Enable remote access from

anywhere in the organization to

any device

Provides devices which are

accessed over cloud.

Test Coverage Object identification by: Native

ID, HTML5-DOM, Image

Recognition and Text

Recognition

Browsers testing

Reboot device

Awake device from sleep mode

Operate device power button

Operate device Back, Call/Hang

Call, Home, Menu buttons

Shell commands for internal

device counters (e.g. CPU,

memory)

Landscape model

Supports Visual Objects,

Native Objects, DOM

Objects

Get Device Info, state, log

Open Device, Reboot device,

rotate device

Keyboard display

Cloud call, SMS, email

Native Object

Identification

Object identification by: id, class,

text, accessibility Label,

accessibility Identifier.

Simple recorder

Object spy for easy identification

of object native properties

Easy instrumentation with no

code change

Supports Native Objects,

DOM Objects

Object Spy


Page 47 of 49

Web & Hybrid Object

Identification

Object identification by: id,

name, class, text, css.

JavaScript execution.

Support for native browser

Operations like click, send text,

select, touch and others.

jQuery support.

Object Spy for easy identification

of object web-DOM / hybrid

properties

Easy instrumentation with no

code change

Supports Native Objects,

Web Objects

Object Spy

Install/Uninstall application

Text Object Identification OCR engine with support for all

languages

Text input

Supports OCR

Text input

Image/Icon Object

Identification

Standard icons/images (e.g.

buttons)

Proprietary icons/images

Auto-sense capabilities for one-

click image extraction

Sensitivity calibration to changes

in image (e.g. size, color,

background, transparency)

Image identification on changing

backgrounds (e.g. transparent,

gradient, moving elements)

Allows simple and intuitive

method for pressing virtual

locations on touch handsets

based on text or image

analysis. The system

automatically finds those

elements, even if their

location changes, without

any change to the script

itself

Gesture Support Multitouch

Swipe

Drag & Drop

Zoom

Scroll inc. conditional scrolling

Swipe

Touch

Trackball

Keyboard Support Device keyboard:

All configurations (alphabet,

punctuation and numbers, and

punctuation and symbols)

All languages

PC keyboard special characters

and capitalization

Regardless of the method

implemented by the

handset (physical or virtual

keyboard, different

languages, capitalization),

simple type text function

hides the complexity of any

string typing


Page 48 of 49

Scripting Capabilities Recorder

Script debugger

Script Multiplier: Script Once, Run

on Multiple Devices

Can convert SeeTest to different

programming languages e.g.

C#,Java etc. and hence

extensibility is high

Recorder

Script Multiplier: Script

Once, Run on Multiple

Devices

Perfecto Script cannot be

converted to any

programming languages

Plug-in to Existing Testing

Frameworks

Plug-in to

QTP

MS Test

Test Complete

Perl

Python

JUnit3, JUnit4, TestNG

Full functionality supported:

Scripting, editing and reporting

(inc. screenshots) capabilities

directly from within test tools

Plug in to

QTP

MS Test

Continuous integration

support

Full and seamless integration to

QC,TFS,JUnit, NUnit, PyUnit,

TestNG, Ant

Full and seamless

integration to QC,TFS etc.

Comprehensive Reporting Capture the screen shots and

gives summary reports

Capture screenshots and

record videos Share test

reports with other

colleagues and partners,

Share devices with others to

demo new apps/features,

reproduce a bug.

Table 5 Leading Automation tools comparison

FAQ Which type of Mobile applications need to be tested on different mobile networks like 2G/3G/4G LTE Conditions?

For specialized applications where on location testing for specific cellular networks is to be considered like Free

usage allowance or two way SMS

Why to test such applications on different mobile networks?

Mobile networks have limited bandwidth and high latency (compared to Wi-Fi and broadband) which effects the

application performance by increasing the page load time on mobile device resulting in testing an application in

real world conditions

How to test an application on different mobile networks?

Network simulation tools helps in reproducing different network conditions by limiting bandwidth and

simulating latency as a result help you in testing whether all mobile users will be getting best user experience in

acceptable response times on mobile devices


Page 49 of 49

Device anywhere and perfecto are also good solutions if it is required to test an application on a particular

networks like US Network, Canada network

Check list for network testing of mobile applications:--

Test behavior of application when there is no network or network problem is there.

Test the application on all supported network connections.

Test application does not hang whenever there is a change in the signal strength.

For a device it should switch to mobile network when there is no Wi-Fi and back to Wi-Fi when there is

both WI-FI and mobile network

Test the robustness of the application in low bandwidth/high latency scenario.

Is the application giving the correct error whenever is some network related issue rather than any

exception or false error.

testing mobile applications using perfecto mobile mobilecloud v2

Documents

microsoft group of companies

mobile applications

visual studio alm rangers

device testing approach

perfecto mobilecloud

date of publication

internet explorer

sql server