mobile test automation: lessons learned in the trenches · book and manage travel on their ios and...

T4 Track

10/4/2012 9:45:00 AM

Mobile Test Automation: Lessons Learned

in the Trenches

Presented by:

Manish Mathuria, InfoStretch

Sha Mohammed, Sabre Airline Solutions

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Manish Mathuria InfoStretch

Manish Mathuria has more than fifteen years of professional and management consulting

services experience. As the CTO and leader of the mobile testing practice at InfoStretch, Manish

leads the team in developing architecture, scalability, performance, and IT strategies for

customers, especially in the mobile domain. InfoStretch provides certification and testing

services to some of the leading national and international app stores. Manish and his team help

optimize major global app stores’ testing and certification processes. They have tested and

certified more than 50,000 mobile applications in the InfoStretch Mobile Lab. Manish speaks

regularly at leading quality and software engineering conferences worldwide.

Sha Mohammed Sabre Airline Solutions

Mr. Sha Mohammed, Director Software Quality Engineering at Sabre Airline Solutions, has over

13 years of experience in Software/IT industry in both product development and consulting

roles. Sha has deep knowledge and experience in software testing and successfully delivering

high volume, high availability eCommerce applications for air travel and finance industries. He

carries an excellent track record in building and sustaining high performing teams and

supporting infrastructure and logistics in large and complex multi-national corporations. Sha

has also had remarkable success in finding the right mix between deploying proven traditional

methods as well as innovative cutting edge technology, tools and processes to enable faster

Time-To-Market with great quality at the right cost. Prior employers include Fidelity

Investments, Ariba, American Express & EMC2.

8/27/2012

1

www.infostretch.com | [email protected] | +408.727.1100 | 2880 Lakeside Drive, Ste 200, Santa Clara, CA

Mobile Test Automation

Lessons from the Trenches

1

1

Manish Mathuria

CTO

InfoStretch Corporation

Mohammad Sha

Director of QA

Sabre Airline Solutions


AGENDA

2

1

Mobile Test

Automation

Tool

Categories

Mobile

Automation –

Best

Practices

Case Study –

Sabre Airline

Solutions

QUESTIONS

2 3 54

Mobile

Landscape

and Testing

Challenges

8/27/2012

2

3(408) 727-1100 | m arket ing@infost re tch.com | 2880 Lakeside Dr i v e Sui te 200, Santa

Clara, CA

www. infostretch.com

MOBILE LANDSCAPEMOBILE TESTING CHALLENGES


Mobile on Fast Track

4

Do you know how many babies are born each day?

• 371,000 DAILY

Impressive, until you realize the number iPhones sold

• 377,900 DAILY

Or the number of Android phones activated.

• 700,000 DAILY

And letʼs not forget the growth of tablets.

• 191,000 DAILY

8/27/2012

3

RiskGap from

requirements

Traditional SW/PC

Mobile Dev.

‘Traditional’

TimeSW Cycles

Mobile

Short Cycle for Mobile

Short Cycle

Mobile dev.

Mobile– Shorter Life Cycle

Product

Version Product Release

timeline

Browsers

Ma

rket

6 years

6 6 yearsyears20052005--20112011

Mobile ALM Process –

Continuous QA

8/27/2012

4

Product

Version Product Release

timeline

Ma

rket

AndroidV2.3 V3

V4V2.3.3 V2.3.4 V2.3.5 V2.3.6V2.3.7V3.1 V3.2

1 year - 2011

1 1 yearyear20112011

Pro

du

ct Ve

rsion

Pro

du

ct Ve

rsion

Pro

du

ct Ve

rsion

Pro

du

ct Ve

rsion

Pro

du

ct Ve

rsion

Pro

du

ct Ve

rsion

Pro

du

ct Ve

rsion

Pro

du

ct Ve

rsion


Clara, CA


MOBILE TESTING IS DIFFERENT

8/27/2012

5


Clara, CA

www. infostretch.comTHE CHANGEFROM TRADITIONAL TESTING:

• Many platforms (Blackberry, Android, iOS, S60, WP7, etc…)

• Many form factors (touch, keyboard, slide, landscape, etc…)

• Many networks (3G, 4G, Wi-Fi, Wi-Max, etc…)

Mobile is fragmented

• More platform changes (8 major

version changes in 2010)

• Handset “life” is short (6-18 months)

• Device Capability?

• How to test location based services?

• Dynamic Carriers/Networks/Devices?

Mobile is dynamic

• Many handsets

• Actual networks

• Accounts, SIM, Access -expensive, time consuming and resource intensive

• Usability, network connectivity, call interruptions, and device providers

Mobile requires logistics


Clara, CA


FROM TRADITIONAL TESTING:

NEW NEEDS + CHALLENGES

•How to identify which devices to test on to support your user base?

•How to identify device limitations and characteristics for feature testing?

•How to report observations during testing on devices to developers?

•How many versions of Mobile OS tests need to be repeated?

•How to find out unique features of each device?

•Offline-Online Content and caching?

•Connectivity to native devices – peripherals

Testing Challenges

• Mobile SDLC is in its nascent stages

• No set quality and development process; Agility required

• Location based, orientation, social media integration

• When to do full test vs partial test?

• Different methods to deploy - making testing more complex

• Ad-hoc testing – beyond what is documented

Software Development

Cycle

•Difficult to test in the field

•No VPN on old devices

•How does the app deal with the backend system changes?

•How to push updates and test them?

•No security standards

•Minimum vulnerability database for apps as well as mobile browsers

•No standard privacy policies and laws set

Enterprise Testing

Challenges

8/27/2012

6

(408) 727-1100 | m arket ing@infost re tch.com | 2880 Lakeside Dr i v e Sui te 200, Santa

Clara, CA


DEFECTS

Business Process

Definitions

Platforms, networks

And UX

requirements

Diagnostics

Test Creation

and

Maintenance

Mobile Specific Web SpecificWeb SpecificMobile + Web

Specific

Test

Requirements

Test

Requirements

Test Strategy

And Plan

Test Strategy

And Plan

Test

Analysis

Capacity

Planning

Pla

tform

s

Manual

Functio

nal

Regre

ssio

n

Test Exec

Capacity

Planning

Capacity

Planning

Tim

ing

Tim

ing

Lo

ad

Lo

ad

Ne

two

rks

Pla

tform

s

Performance

Testing

ISSUES

CollaborateApp DevelopApp Develop

Business

Requirements

Business

Requirements

Define

Use Cases

Define

Use CasesPlatform P

latfo

rms +

UX

Te

stin

g

Optimize Performance + Monitoring

Optimize Performance + Monitoring

Optimize QualityOptimize QualityManage RequirementsManage Requirements

TESTING LIFECYCLEENTERPRISE MOBILE TESTING LIFECYCLE


Clara, CA


MOBILE AUTOMATIONMOBILE AUTOMATION TOOL CATEGORIES

Native Platform

Automation Technologies

Platform Independent

Mobile Automation

Mobile HTML5 Based

Automation

8/27/2012

7

(408) 727-1100 | m arket ing@infost re tch.com | 2880 Lakeside Dr i v e Sui te 200, Santa

Clara, CA


TOOL CATEGORIESMOBILE AUTOMATION TOOL CATEGORIES

Mobile HTML5 Based Automation Native Platform Automation Technologies Platform Independent Mobile Automation

Wh

at

• Drives HTML and JS constructs

• Recognizes Web Controls

• Drives GUI widgets and Controls on

the Native UI Layer

• Leverages screen coordinates, images and

OCR to drive automation

Pro

s

• Cross device platforms

• Cross browsers

• Object aware automation

• Non Intrusive to App

• Ultimate control as good object level

UI layer access

• Cross platform scripting, one script many

devices.

• Has access to the whole device

Co

ns

• Limited to Web and Cross

Platform HTML5 Apps

• Intrusive, often requires code access

• Written specifically for a device

platform

• Limited to the context of the

application

• Limited object awareness, reliance on

image and OCR

• Less effective for complex automation

To

ols

• Selenium Webdriver

• QUnit

• Robotium

• Frank

• UIAutomation

• MonkeyTalk

• PerfectoMobile

• See Test (Experitest)

• Device Anywhere


Clara, CA


MOBILE AUTOMATIONLESSONS FROM THE TRENCHES

8/27/2012

8


Lessons Learnt

15

1

Deal with

Scripting

Challenges

Dealing with

Fragmentatio

n

Test

Execution

Challenges

2 3 4

How to Select

Test Cases

and Devices

for Mobile

Automation?


Clara, CA


TEST CASE/Devices SELECTIONHow do you identify the test cases for Mobile Automation?

•Medium complexity test cases

•Provide coverage for device compatibility tests

ComplexityComplexity

•Jail broken or not?

•Market relevant

•Screen resolution coverage

•OS Coverage

Devices SelectionDevices Selection

•System Interaction

•Peripheral Interaction

•Interaction between multiple apps/ OS-App /Multi Domain

•Location aware, peripheral, battery life, system alerts etc.

Tool and Resource CapabilityTool and Resource Capability

•UI components that will change less.

•Clear understood business processes

•Best Practices for other GUI Automation apply

Stability/ChangeStability/Change

8/27/2012

9


Clara, CA


Scripting Challenges

Widgets will render Different

even on same OS

Android 2.3

Left: Samsung

Right: Sony Ericson

8/27/2012

10

Droid X2 Droid Charge iPhone 4

Form Factor Challenges

Desktop Browser Droid Charge

Test on the actual target devices

8/27/2012

11


Clara, CA

www. infostretch.comDealing with SynchronizationImage Scan & Analysis

Needle

Found?

Exit With Success

Timeout

Exceeded?

Exit With Failure

Yes

Yes

No

No


Clara, CA

www. infostretch.comDealing with Fragmentation

Design your test cases for exceptions!

Virtual FunctionsQTP TestcasesDevice Implementations

Browser

Open

Browser Go

ToSign On

Make

Payment

Confirm

PaymentSign OffText Verify

Make Payment Test caseQTP/Selenium

Address

ATM Search

Browser

Open

Select Acct

View

Schedule

PaymentSync Text Quick Links Nick Name

Virtual Function Library

Browser Go

ToSign On Sign Off

ZIP ATM

Search

Make

PaymentSync Image

Confirm

Payment

Framework

iOSAndroid

HTC Aria

Android

HTC DroidBB Torch LG CU620 LG VX9000

Device Implementations for Sign On (10+)

BB 8520

Devices

8/27/2012

12


Clara, CA


Execution Best Practices


Clara, CA


Invest in Continuous Integration (CI)Automating the Automation

• Automate execution of test cases

• Integrated automated reporting, and notifications

• Prominent tools – Jenkins, Microsoft TFS

What is CI?What is CI?

Invest in a Test Automation lab of devicesInvest in a Test Automation lab of devices

8/27/2012

13


Clara, CA

www. infostretch.comOur CI Integration

Automating the Automation

25

Continuous Integration

Check-inCheck-in

BuildBuild

TestTestReportReport

OptimizeOptimize

e.g. Hudson / Jenkins

Test Framework

Test Assets

Test Reports

e.g. TestNG

e.g. ReportNG

Test Management Tool

e.g. QMetry / HP-QCTest Runner

Setup

Launch

Reporting

Teardown


Clara, CA


Develop Strong Test Recovery Automating the Automation

• Device characteristics

• Test script errors

• Automation tool instability

• Device instability

• Data instability

Automation Tests will break due toAutomation Tests will break due to

• Develop agents that can restart text components on devices

• Enhance test case recovery, capability to restart test case.

Invest in Robust Test recovery systemInvest in Robust Test recovery system

8/27/2012

14


Clara, CA


Sabre Case Study

Business Problem

Automated a mobile web based application that enables the ability for passengers to

book and manage travel on their iOS and Android mobile devices through a simple

and intuitive process flow.

Development and Manual Testers defined 33 user

scenarios test cases for automation to detect regressions.

Test cases are categorized by

a. Book My Flight

b. Manage My Flight

c. Check Flight Status

d. Feature Function Test – integrated testing with the

above 3 categories

Automation starts as soon as Manual tests verify each

functionality.

Application compatibility testing with system notifications

are manually tested.

8/27/2012

15

Test EnvironmentAutomation Tool Selected: Selenium Remote Android and IOS web driver. Automated tests limited

to the Sabre web application only. Selenium based solution does not support automated

application compatibility i.e. system alerts – phone calls while application is running and Blackberry

devices - Perfecto Mobile test automation can be used in this case.

Selenium Test Suites

Webdriver

installed on

each mobile

devices

Jenkins CI


Test Code

Repository

Monitor for

regression

Sabre

Test on Firefox first

– as baseline, then

move to mobile

devices


Some Technical Challenges Faced

• Element ID Discovery –– Test code dependent on locator id changes and UI flow changes especially in agile scrum

development environment. Colocation is critical to promote communication.

– Xpath not supported on all mobile devices.

• Selenium Web driver – connection reset when executing whole test suites on iOS and Android devices on a single run.

– Web driver not responsive at times causing timeouts.

– Still maturing - monthly releases

• On Android– Android Web driver – delay in populating in list box values.

– Problems with launching web applications on Android emulator.

– Compatibility across various devices – i.e. On Android 2.3 OS - Samsung Galaxy and Sony Ericson have different list-box implementations (see next slide)

• On IOS– iOS Web driver does not always reset after test suite run and need device restart.

– Installing Web driver on iOS devices requires “jail broken” device and valid provision profile on device.

8/27/2012

16

Lessons Learned

• Automation team to be co-located with development team especially with agile scrum teams.

• Establish policy to ensure locator ids backward compatibility between sprints.

• Due to the variability of devices and web driver behaviors, architect your automation with this theme “when things change, change it in ONE place”.

• Automation scripts should coordinate the highest level abstraction, (libraries) whenever possible to maintain agility and robustness across devices.

• Agreement on list of devices / vendors to be supported. Not all devices will be have the same way.

• Selenium iOS and Android web drivers are still evolving - expect to do debugging to workaround web driver issues.

Questions?

mobile test automation: lessons learned in the trenches · book and manage travel on their ios and...

Documents