A Jacobs

An overview of

Mobile Testing By André Jacobs

A Jacobs

THE RISE AND RISE OF MOBILE

A Jacobs

3

A look at the company that arguably has sparked the explosive growth in smart devices and mobile applications used by billions of people around the world today.

The Apple Story

Apple iPhone is launched January 9, 2007

iPhone was introduced at Macworld on January 9, 2007 in San Francisco.

A Jacobs

4

1,000,000 iPhones Sold September 10, 2007

Just 74 days after launch.

A Jacobs

5

App Store launched July 9, 2008

App downloads July 14, 2008

The App Store opened via an update to iTunes. It allowed Apple to control the quality of apps for the first time.

App Store Downloads Top 10 Million

A Jacobs

6

Apps a big thing September 9, 2008

Apps a really big thing April 24, 2009

iPad launched April 3, 2010

App Store Downloads Top 100 Million Worldwide.

App Store Downloads Top 1 Billion.

Apple introduces the iPad.

September 23, 2008, The First Production Android Device. T-Mobile announces the G1, also known

as the HTC Dream, as the first device on the market to feature the Android 1.0 platform.

A Jacobs

7

Staggering numbers June 2, 2014

7 Years after the first iPhone launch.

75

80

App Downloads In Billions

Both the Apple App store and the Android Play store have roughly 1.2 million apps available for download.

A Jacobs

8

Now 2015

2009-2014

A Jacobs

9 Mobile Adoption

Population

Active Internet Users

Active Social Media Accounts

Unique Mobile Users

7.210 Billion

3.010 Billion

2.078 Billion

3.649 Billion

A Jacobs

10 10

3 Billion videos watched daily

500 Million tweets daily

200 Million hours watched daily

70 Million active users

Twitter There are more than 284 million active Twitter users monthly, 80 percent of which are using mobile devices to tweet (227 Million).

80%

Facebook Videos viewed on a mobile device (2 Billion). 65%

YouTube Videos watched on mobile devices. 40%

Social Media

A Jacobs

11

Can you afford not to have a

mobile testing strategy?

March September March

Software Test Professional Polls

2013

2013

2014

2015 56%

72%

95%

?

Do you currently have a mobile test initiative?

Now

A Jacobs

MOBILE CHALLENGES

A Jacobs

13 13

Mobile Defined

User interface Touch or non-touch display screen and optional mini keyboard.

Handheld Handheld device or handheld computer.

Portable Both compact and lightweight to is made for portability.

Android 5.0 The smarts of Android on screens big and small.

Windows 10 One product, one platform, one store.

Apple iOS 8 Connections between apps and devices

Explosion of wearables Analysts project a boom in the sales of wearables. Some predict sales will climb to 170 million in 2016.

A Jacobs

14 14

Increased complexity

App Functionality

Mobile Specific

Operating Systems

Devices

A Jacobs

15

Radically Changing Software

Large feature rich

12-18 months

6 monthly

Major IT installations

$ Millions

Single purpose app

8-10 weeks

Continuous deployment

Download

$ 50k - $ 100k

… the speed at which this happens; and

… how it is developed and deployed

… the cost point Then Now

Application

Dev Cycle

Upgrades

Distribution

Costs

A Jacobs

APPROACH TO MOBILE

A Jacobs

17

Actual

Accurate performance

Can connect peripherals

Device specific conditions

Procurement

Physical security

Costs

Difficult to share (procure multiple)

Real Devices

A Jacobs

18

Public / private clouds

Web based

Variations

Physical security

Privacy concerns

Availability

Speed

Costs

Remote Devices

A Jacobs

19

Simulates the device

Use more often in early stages of testing

Use existing infrastructure (cost effective)

Can’t connect peripherals

Cannot do carrier testing

Not a true reflection

Availability of specific versions

Emulators

A Jacobs

20

Android iOS

Other

Select an Operating System

Blackberry

Windows

A Jacobs

21 4.2.x

4.4.x

4.3.x

Select relevant OS versions

5.0.x

4.1.x

A Jacobs

22 Specified Devices

Availability: Emulators, Real Devices; Remote Devices

Supported OS versions

Select Devices

Screen Sizes (including tablets e.g.)

Samsung Galaxy Note 3,

Samsung Galaxy S3

A Jacobs

23

Combinations

Testing Types (add Test Automation?)

Build the Pyramid

Costs

System, Compatibility, Automation

A Jacobs

24

IP Protection: Public / Private

Lab tools

Test Automation Solution

Automation Tools

Costs

Real / Emulators / Remote

Private

Real

Security

A Jacobs

25

Costs

Solution

Risk

Present to client

Agree; Make a selection; and Execute

A Jacobs

26

Samsung Galaxy Note 3

An

dro

id 4

.2

An

dro

id 4

.2.1

An

dro

id 4

.2.2

An

dro

id 4

.3

An

dro

id 4

.4

An

dro

id 4

.4.1

An

dro

id 4

.4.2

An

dro

id 4

.4.3

An

dro

id 4

.4.4

An

dro

id 5

.0

An

dro

id 5

.0.1

An

dro

id 5

.0.2

Test case 01 1 2 3 4 5 6

Test case 02 7 8 9 10 11 12

Test case combinations (14+12) x releases

Samsung Galaxy S3

An

dro

id 4

.2

An

dro

id 4

.2.1

An

dro

id 4

.2.2

An

dro

id 4

.3

An

dro

id 4

.4

An

dro

id 4

.4.1

An

dro

id 4

.4.2

An

dro

id 4

.4.3

An

dro

id 4

.4.4

An

dro

id 5

.0

An

dro

id 5

.0.1

An

dro

id 5

.0.2

Test case 01 1 2 3 4 5 6 7

Test case 02 8 9 10 11 12 13 14

A Jacobs

27

Lollipop (5.0)

0%

Kit Kat (4.4)

34%

Jelly Bean (4.2 – 4.3)

49% 4.1.x

21.3% 4.2.x

20.4% 4.3

7.0%

A Jacobs

RECOMMENDATIONS

A Jacobs

29

• Test on emulators at first • Test on selection of real

devices • Top, middle, bottom • Different screen

sizes • Test on remote devices

to fill gaps (might not be possible to procure devices or specific OS versions)

• Focus on backend • Don’t try to test all

devices • Simulate bandwidth

• Build a lab • Select a cloud provider • Select a suitable test

automation tool

Manual Testing

Test Automation

Performance Testing

A Jacobs

30

Mobile is not just another

platform. Have a strategy, plan

your approach.

Think it is easy

OS updates might catch you off guard, make sure you safeguard certain versions. (Research)

OS updates

Don’t rely on emulators only.

Invest in real devices or

remote devices.

Testing on emulators

Know that a continued investment is needed to continually procure devices, pay for licenses, rent remote devices.

Not a once off investment

A Jacobs

31 Take the leap

Look out

Success

Look for emulators early,

Look for operating system versions early. Procure devices early

(consider second hand).

Tips

A Jacobs

CASE STUDY

A Jacobs

33 33 Case Study - Roche Diagnostics

Test Mobile Diabetes Application

Multiple Android Versions

Multiple Mobile Types

Identify issues with different Screen Sizes

Shorten Test Cycles; Decrease Cost

Performance Test Backend

A Jacobs

34 34 Case Study - Roche Diagnostics

SeeTest Cloud (Private); Remote Devices

SeeTest Manual

SeeTest Automation

Mix of real devices and emulators

Microsoft Visual Studio

Neotys NeoLoad

A Jacobs

iLAB, 21e Polo Crescent, Woodmead Office Park, Woodmead, Johannesburg

andre.jacobs@ilabquality.com

@andre_jacobs

So Many

Questions