T16

Mobile Testing

5/8/2014 1:30:00 PM

Mobile Testing in the Cloud

Presented by:

Rachel Obstler

Keynote DeviceAnywhere

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Rachel Obstler

Keynote DeviceAnywhere

Rachel Obstler is the senior director for Keynote's DeviceAnywhere platform. Rachel has more than ten years of experience in the wireless industry, working for companies ranging in size from startups to multinational corporations. She most recently managed the wireless data quality products at Telephia Inc. and has experience managing the products of various wireless technology companies, including Metawave Communication (smart antenna technology) and Enuvis (location technology). Rachel spent three years working with Lucent Technologies, where she managed and launched a billion dollar product line of CDMA Base Stations.

4/26/2014

1

Mobile Testing in the Cloud

Helping QA organizations manage the challenges of

a mobile-first world

Rachel Obstler

May 8, 2014

Contents

� The Mobile Imperative

� The Challenge of Mobile Quality

� Solutions

� Optimizing Your Device Testing Strategy

� Automation

1

4/26/2014

2

The Mobile Imperative

DOWNLOADED IN 20131

102BILLIONAPPS

77BILLIONANTICIPATED REVENUE

GENERATED THROUGH

MOBILE APPS BY 20171

#1 PRIORITY: MOBILE

RETAILER TECHNOLOGY PRIORITIES2

2 OF 3 BANKSPREDICT 100% OF THEIR CUSTOMERS

WILL USE MOBILE SERVICES BY 20173

1. Gartner; 2. Forrester, State of Retailing Online, 2014; 3. Metaforic UK

2

The Mobile Imperative

1. Econsultancy; 2. Forbes; 3. ZDNet

ONLY 16%CONSUMERS SURVEYED WILLING TO GIVE

A PROBLEMATIC MOBILE APP

MORE THAN ONE ATTEMPT1

USERS EXPECT A MOBILE APP TO BE RESPONSIVE IN

3 SECONDS3

A MOBILE FRIENDLY

SITE MAKES

2 OF 3USERSMORE LIKELY

TO BUYA COMPANY’S PRODUCT

OR SERVICE2

3

4/26/2014

3

Survey: Quality Expectations of Mobile Websites & Apps

Almost half of the respondents think mobile quality expectations are

the same as desktop, and 35% think they are higher.

10%

49%

36%

5% Quality expectations for mobile applications and

websites are lower than desktop

Quality expectations for mobile applications and

websites are the same as desktop

Quality expectations for mobile applications and

websites are higher than desktop

Not sure

Now when thinking about quality expectations of mobile websites and applications,

which of the following is true? (1,314 respondents)

4

Challenge: Mobile = Devices

� Device platforms, fragmentation, and growth• Multiple OS’s, form factors and screen

resolutions

• Frequent device refresh

� New capabilities to test

• Camera, GPS, direction, orientation, voice, etc.

� Network considerations• Multiple carriers, variable throughput and

latency, disconnected use, network switching

5

4/26/2014

4

Scan & Certify

Instrument

Test

IntegrateObtain Insight

Manage

Deploy

Design & Develop

Challenge: Mobile = Faster, More Iterative, Continuous Activity

6

Challenge: Mobile = Late to the Game

Employee Skillsets

� HP QC/IBM RQM users

� HP UFT/QTP users

� Automation engineers/

programmers

� Manual testers

7

4/26/2014

5

Challenge: Mobile = Spread Across the Organization

Organizations

� Multiple teams

� Geographically diverse

employee base

� IT security requirements

8

Survey: The Mobile Testing Organization is Largely Decentralized

How is mobile website and application testing

organized within your company?(# of respondents in each column)

All

Verticals

(1590)

High

Tech/IT/

Software

(661)

Fin Svcs

& Ins

(234)

Telecom

(151)

Media

(139)

Retail*

(57)

Centralized testing group, mobile-specific 19.7% 20.3% 20.9% 19.9% 22.3% 17.5%

Centralized testing group, not mobile-specific 12.6% 12.0% 12.8% 9.9% 18.7% 10.5%

Individual QA groups within business units/divisions,

mobile-specific35.8% 38.1% 36.3% 44.4% 36.0% 36.8%

Individual QA groups within business units/divisions, not

mobile-specific20.8% 22.5% 20.5% 13.9% 14.4% 17.5%

Testing is predominantly outsourced 3.5% 2.6% 3.4% 2.6% 3.6% 8.8%

Currently not testing any mobile apps or websites 7.5% 4.5% 6.0% 9.3% 5.0% 8.8%

More testing groups are distributed, rather than in one centralized group.

Individual and also centralized groups are more often mobile-specific.

* Retail sample size <100.

9

4/26/2014

6

Survey: Challenges of Testing Mobile Websites and Applications

Regarding challenges encountered

when testing mobile applications

and websites, please rate the

following challenges on a 1 to 10

scale, where 1 is ‘Not at all

challenging’ and 10 is ‘Extremely

challenging’. (1,314 respondents)

1 (Not

Challenging) 2 3 4 5 6 7 8 9

10 (Extremely

Challenging)

Availability of mobile testing experts 4.7% 4.3% 5.8% 7.2% 17.4% 11.3% 15.4% 17.0% 8.7% 8.2%

Implementing the right testing

method/process for mobile3.5% 3.2% 5.3% 7.4% 17.7% 11.5% 18.0% 18.3% 8.4% 6.5%

Availability of proper testing tools 2.7% 3.2% 5.3% 5.4% 16.3% 11.6% 16.4% 16.6% 11.3% 11.3%

Access to mobile devices 6.5% 6.3% 4.7% 4.5% 14.2% 10.0% 13.5% 16.7% 9.9% 13.6%

Having enough time to test 4.2% 3.4% 5.6% 5.9% 16.7% 12.3% 14.9% 14.9% 10.2% 11.9%

10

Survey: Important Functional Testing Features

QAWithin functional testing, what are the main priorities for QA?

Easy access to a variety of device models was rated the most

important functional testing feature across all verticals.

11

4/26/2014

7

Optimize Your Device Testing Strategy

� Determine your priority devices

� Provide easy access to real devices

� Enable local and remote employees

� Provide a secure, enterprise-grade

test environment

12

Use Data to Optimize Test Resources

13

Internal apps

Customer apps

Android Screen Size and Density

Source: Google

Android OS Market Share

Source: Google

4/26/2014

8

Real Devices in the cloudYour computer, connected

to the internet

Your key presses and

mouse clicks sent to

the device

The device’s screen

sent back to your

computer

SaaS-based Real Device Solution

14

Different screen

resolution /

screen size

Unreadable text, blurred images,

misalignment of screen elements,

and items that fall off the screen.

Android

Customizations

On-screen and physical controls

that function differently across devices

Customized handling of inputs and

events (e.g. Samsung Swype vs.

default Android keyboard).

Memory / CPU Low or insufficient memory or

processing power. Impacts of other

services running on the device.

Mobile device characteristics that impact the quality of your application or website, but are not

easily verifiable using an emulated phone or browser, include:

Test on Real Devices?

15

4/26/2014

9

Devices

Shared Devices

� 24X7 access to hundreds of smart

devices

� Use for compatibility testing,

“untrusted device” testing

Private Devices

� 24X7 access to your devices

� Enables geographically diverse team

� Inside corporate firewall or external

Local Devices

� Plug your smart devices on hand

directly into your local computer

� Make use of existing assets, great for

local teams

Use a Combination of Local and Remote Devices

16

Deploy an Enterprise-grade Product

17

Access and Project Management

Multiple levels of user permissions

Control access to all assets, including

devices, scripts and test results

Create different groups to manage access

Security

Password protection features

(enforce change; salted; disable save)

SSL communication

LDAP integration

4/26/2014

10

0

Automate!

18

� Determine target test cases to

automate

� Support agile processes (continuous

integration)

� Find a cross-device testing tool

� Make use of existing (desktop)

automation tools and processes where

possible

� Provide multiple scripting options

because your team is not one size

fits all

Test Automation is Not Yet Widely Used

What percentage of your mobile web

regression tests is currently

automated? (# of respondents in each column)

All Verticals

(1176)

High

Tech/IT/

Software

(520)

Fin Svcs &

Ins (183)

Telecom

(106)

Media

(96)

Retail*

(45)

0% 28.8% 27.1% 26.8% 17.9% 28.1% 35.6%

1-25% 28.7% 31.3% 25.1% 33.0% 29.2% 24.4%

26-50% 17.3% 19.0% 19.7% 15.1% 14.6% 17.8%

51-75% 9.8% 11.0% 8.2% 15.1% 10.4% 2.2%

>75% 4.9% 3.1% 5.5% 5.7% 4.2% 11.1%

Don't know 10.5% 8.5% 14.8% 13.2% 13.5% 8.9%

Majority or organization have less than 25% of mobile web regression

tests automated.

* Retail vertical sample size <100.

19

4/26/2014

11

Mobile Apps – Same Story

What percentage of your mobile native

applications regression tests is

currently automated? (# of respondents in

each column)

All Verticals

(1172)

High

Tech/IT/

Software

(519)

Fin Svcs &

Ins (183)

Telecom

(105)

Media

(95)

Retail

(45)

0% 30.4% 28.3% 27.9% 20.0% 32.6% 46.7%

1-25% 27.2% 29.7% 26.2% 34.3% 22.1% 17.8%

26-50% 15.4% 18.9% 15.8% 11.4% 8.4% 11.1%

51-75% 9.2% 8.7% 10.4% 13.3% 9.5% 6.7%

>75% 4.6% 4.2% 3.8% 5.7% 6.3% 8.9%

Don't know 13.1% 10.2% 15.8% 15.2% 21.1% 8.9%

* Retail vertical sample size <100.

20

Determine Your Targets for Automation

Goals

Test Cases

Frequency

Methodology

Basic acceptance,

build acceptance

Functional testing

across all existing

areas of product

Compatibility

across devices

Very high value

to automate!

High value to

automate

Medium value to

automate

# of Devices

Optimal Test

Strategy

Automated

Testing

Automated

Testing

Manual or

Automated Testing

Smoke

Testing

Regression

Testing

Compatibility

Testing

Deep testing of

new areas of

functionality

Low value to

automate

Manual Testing

New Feature

Testing

Not automated

Exploratory

Testing

Exploration of

functionality from

customer viewpoint

Manual Testing

21

4/26/2014

12

Test

Continuous Integration

Feedback Loop

22

Develop Build

Customer Case Study: Speed Time to Market

� Each build of the native mobile

retail investment app required

a Build Acceptance Test with

300 hundred test cases across

two devices (one Android and

one iOS)

� Run manually, tests were

taking two QA engineers two

weeks (30 test cases per

engineer per day)

� Engineering did not receive

any build feedback until two

weeks had passed

� QA team had QTP skills, and

managed the process using QC

� Automated test cases using

DeviceAnywhere and QTP

� 300 test cases executed

across two devices (in

parallel) now take 24 hours

to run, vastly reducing time

to market

� Manual testers freed up to

test new or more complex

features

� All results saved in QC

BACKGROUND RESULT: Automation on real devices

resulted in 10X Reduction in BAT

time!

CUSTOMER

Director of QA at a major financial institution

23

4/26/2014

13

Device Fragmentation

� Less up front scripting cost

� Reduce ongoing script

maintenance

Object-Level Scripting Important

Cross-Device Testing

24

Customer Case Study: Improve Product Quality

� The current mobile test process

for the consumer mobile banking

app involved outsourcing testing

to a manual test house.

� The available budget afforded

them enough person/hours to

complete their 900 regression test

cases on only one device

� Results and issues were managed

using unwieldy word documents

� As usage picked up, issues in the

field highlighted the need to

ensure quality across a variety of

mobile platforms and devices

BACKGROUND RESULTS

CUSTOMER

VP of Engineering at a major bank

� Now the same 900 test cases are

run across 20 devices, improving

test coverage and quality

� All 20 devices tested in half the time

it took previously to test one device,

improving time to market

� Test results are automatically

available online with screenshots

(no more word docs, improved

process)

25

4/26/2014

14

Survey Your Existing Test Processes

ALM Tools (HP ALM, IBM Rational)

Automation Tools such as HP QTP

Open Source Tools like Selenium,

Robotium

Other tools

26

What desktop test tools is your organization currently using?

Survey: Important Functional Testing Features by Vertical

Integration with ALM (Application Lifecycle

Management) tools were also highly rated for

financial services companies.

Having an ‘integration with open source tools’

(such as Selenium, Robotium, etc.) was rated

very high for retail companies as they

traditionally have been more e-commerce

focused and would often be developing

automation using open source web-based tools.

Were there any other vertical-specific challenges that also rated high?Q

27

4/26/2014

15

Beginning and Advanced Scripting Options Enable All Employees

28

UI-Based Scripting

For complex logic and

large test suite design

and execution

Integration with Leading

Test Tools

Allow employees with

existing skillsets to easily

extend to mobile

Java API

Enables your QA engineers

with programming skills to

write test in Java

Script Recording

Point and click recorder

enables anyone to

immediately create

reusable test scripts

Automation

EngineersUFT/QTP users Programmers New Scripters

Challenges

29

� Mobile = Devices

� Mobile = Faster, More Iterative,

Continuous Activity

� Mobile = Late to the Game

� Mobile = Spread Across the

Organization

4/26/2014

16

Solutions

Challenge Solution(s)

Mobile = Devices � Use data to determine priority devices

� Provide access to real devices in the cloud

� Automate using a cross-device testing tool

Mobile = Faster, More

Iterative, Continuous Activity

� Determine high-value test cases to automate

� Continuous integration

� Automate using a cross–device testing tool

Mobile = Late to the Game � Make use of existing test tools where possible

� Provide a variety of scripting options for differing

skill sets

Mobile = Spread Across the

Organization

� Provide access to local and remote real devices

� Enterprise-grade product to manage multiple

teams

30

Make every digital interaction count

MyKeynote®

TESTANALYZE

Mobile App Testing

Mobile Web Testing

Load Testing

Data Visualization

Competitive Intelligence

Expert Analysis

Mobile App Monitoring

Website Monitoring

Mobile Web Monitoring

MONITOR

31

4/26/2014

17

Keynote: Most Experienced Cloud-based Mobile, App, Web &

Network Performance Company on the Planet

� 1995: Founded

� 500 employees

� 4,000+ customers

� 7,000+ measurement devices in ~300 locations –

most scaled on-demand infrastructure in world

� 700,000,000 performance measurements daily

� 730 Carrier Networks monitored in ~300

locations in 200 countries

� 1,500+ devices in 15 global locations

� Forbes ‘One of the Best 100 Companies in

America’

32

Sao Paolo

Moscow

Beijing

Paris

New

York

Integrating the World’s Largest Global Testing & Monitoring

Network and the World’s Largest Real Device Cloud

33

4/26/2014

18

Enterprise Solutions on a Global Scale

� 3,000 customers in 130

countries

� Business operations in EMEA,

APAC, Americas

� Global infrastructure

� Web performance testing &

monitoring agents in 100

locations

� Mobile testing and monitoring

agents in 250 locations

� 7,000 devices on network

� 270B measurements per year

34

Thank YouQuestions?