build your mobile testing expertise

MO PM Tutorial

9/30/2013 1:00:00 PM

"Build Your Mobile Testing

Expertise"

Presented by:

Karen Johnson

Software Test Management, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Karen N. Johnson

Software Test Management, Inc.

Karen N. Johnson is a software test consultant, working on-site and remotely to help her clients

build testing teams and strengthen their testing practice. Karen is considered an expert in

software testing and has been part of the context-driven software testing community for a

number of years. She is a frequent speaker at testing conferences; a contributing author to the

book, Beautiful Testing; and co-founder of WREST, the Workshop on Regulated Software

Testing. Karen has published numerous articles and blogs about her experiences with software

testing.

8/20/2013

1

Build Your Mobile Knowledge section 1

Build Your Mobile Testing Expertise v7 © Karen N. Johnson, 2013 1

agenda

• device selection

• UX

• device settings

• competitive analysis

Build Your Mobile Testing Expertise v7

© Karen N. Johnson, 2013 2

8/20/2013

2

One thumb design

Fat Thumb: A One-Handed Alternative To Pinch-To-Zoom

http://www.fastcodesign.com/1671741/fat-thumb-a-one-handed-alternative-to-pinch-to-zoom#1

Testing One Thumb, One Eyeball Mobile Use

http://www.lukew.com/ff/entry.asp?1664


Touch Gesture Icons

http://www.mobiletuxedo.com/touch-gesture-icons/

Input Guidelines

http://uxdesign.smashingmagazine.com/2011/10/06/not-your-parents-mobile-phone-ux-design-guidelines-smartphones/

Touch Gesture Reference Cards



http://www.fastcodesign.com/1671741/fat-thumb-a-one-handed-alternative-to-pinch-to-zoom





















































8/20/2013

3

Twitter

Content Karen McGrane @karenmcgrane Deane Barker ‏@gadgetopia cleveg ‏ @cleveg SaraWachterBoettcher ‏ @sara_ann_marie UX: Theresa Neil ‏ @theresaneil (watch her website and slideshare) RWD Ethan Marcotte ‏ @beep Responsive Design ‏ @RWD Jordan Moore ‏ @jordanmoore Mobile Brad Frost ‏ @brad_frost Brian Fling ‏ @fling Jason Grigsby, ☁4 ‏ @grigs Jonathan Kohl ‏ @jonathan_kohl steven hoober ‏ @shoobe01 Josh Clark ‏ @globalmoxie


Books

Content Everywhere

by Sara Wachter-Boettcher

Jump Start Responsive Web Design

by Craig Sharkie & Andrew Fisher

Mobile First

by Luke Wrobleski

Mobile Developer's Guide to the Galaxy

see the Enough Software website


https://twitter.com/gadgetopia




https://twitter.com/cleveg





https://twitter.com/sara_ann_marie





https://twitter.com/theresaneil





https://twitter.com/beep





https://twitter.com/RWD





https://twitter.com/jordanmoore





https://twitter.com/brad_frost





https://twitter.com/fling





https://twitter.com/grigs







https://twitter.com/jonathan_kohl





https://twitter.com/shoobe01






https://twitter.com/globalmoxie





8/20/2013

1

device selection

part 1


Device Selection: How will you decide which devices? Possible Factors:

budget

procurement

time

usage statistics

manufacturers & models

operating systems

network carriers

m-commerce

mobile browsers

cookies

installation

responsive web design

input methods

location awareness & map providers

proprietary network

barcodes and scanners

flash



8/20/2013

2

Device Matrix source: Mobile Design and Development, by Brian Fling

*** This is a model for making a decision. Does this model help you?

Class Markup CSS Javascript

Class A

Class B

Class C

Class D

Class F


Budget, Procurement & Time: Budget: Can you afford the devices you want to test? Procurement: Can you find the devices? Time: If you resolve budget and purchase, would your team have the time to test all of the devices?

• Teams often solve the budget issue with using their own devices.

• Teams often solve finding the devices through eBay or Craigslist.


8/20/2013

3

Mobile Statistics: Do market statistics influence which devices to test?

• Country: Usage statistics vary

greatly by country.

• Source: Which source do you trust?

• Analytics: Are you capturing device usage for your own application?


Mobile Manufacturers: & Models: Does manufacturers or specific models influence which devices to test?

• Manufacturers vary by country.

• Devices are updated frequently.

• How often do you update your device coverage?


8/20/2013

4

Operating Systems: How do you decide which OS's to cover ?

• How do you address forced OS updates?

• How do you maintain older versions?


Network Carriers and Settings:

• How do you acquire network simulation for 'other' countries?

• What about different keyboards and languages?

• What about SMS text testing?


8/20/2013

5

Does testing from the m-commerce perspective impact device selection?


Mobile Browsers: If you're testing on mobile browsers, how does that influence device selection?


8/20/2013

6

Cookies: Do cookies influence device selection?

• Does your app or site offer cookies?

What about:

• Auto-login

• Third party login access (i.e. Use Facebook account to login)


Installation Application install & updates


8/20/2013

7

Location Aware: Does location aware influence device selection?

• How does map provider selection influence device selection?


Input Selector & Navigation: Does the input method impact device selection?

• Rotate

• Touch

• Click wheel

• Rollerball

• Pinch & zoom

• Typing/keyboard inputs

• Gestures

Build Your Mobile Testing Knowledge v7 © Karen N. Johnson, 2013 14

8/20/2013

8

Mobile Proprietary Network Access: Are you required to test particular devices?

• Does your mobile testing need to

pass an audit or regulations?

• Are there VPN considerations?


Responsive Web Design: Do you need to check content and layout on different devices?

• People often use both at the same time.

For more on RWD, see:

• http://blog.creativemagma.com/featured/2209/


http://blog.creativemagma.com/featured/2209/



8/20/2013

9

exercise: build a device matrix (or mind map)

• With your team, determine what factors impact your decision.

• Update the checklist of factors to reflect your decisions.

• Build your own device matrix or mindmap to identify what devices to test.



budget

procurement

time

usage statistics


operating systems

network carriers

m-commerce

mobile browsers

cookies

installation


input methods


proprietary network


flash


8/20/2013

10




Class A

Class B

Class C

Class D

Class F


8/20/2013

1

device selection

part 1



budget

procurement

time

usage statistics


operating systems

network carriers

m-commerce

mobile browsers

cookies

installation


input methods


proprietary network


flash



8/20/2013

2




Class A

Class B

Class C

Class D

Class F


Budget, Procurement & Time: Budget: Can you afford the devices you want to test? Procurement: Can you find the devices? Time: If you resolve budget and purchase, would your team have the time to test all of the devices?

• Teams often solve the budget issue with using their own devices.

• Teams often solve finding the devices through eBay or Craigslist.


8/20/2013

3

Mobile Statistics: Do market statistics influence which devices to test?

• Country: Usage statistics vary

greatly by country.

• Source: Which source do you trust?

• Analytics: Are you capturing device usage for your own application?


Mobile Manufacturers: & Models: Does manufacturers or specific models influence which devices to test?

• Manufacturers vary by country.

• Devices are updated frequently.

• How often do you update your device coverage?


8/20/2013

4

Operating Systems: How do you decide which OS's to cover ?

• How do you address forced OS updates?

• How do you maintain older versions?


Network Carriers and Settings:

• How do you acquire network simulation for 'other' countries?

• What about different keyboards and languages?

• What about SMS text testing?


8/20/2013

5

Does testing from the m-commerce perspective impact device selection?


Mobile Browsers: If you're testing on mobile browsers, how does that influence device selection?


8/20/2013

6

Cookies: Do cookies influence device selection?

• Does your app or site offer cookies?

What about:

• Auto-login

• Third party login access (i.e. Use Facebook account to login)


Installation Application install & updates


8/20/2013

7

Location Aware: Does location aware influence device selection?

• How does map provider selection influence device selection?


Input Selector & Navigation: Does the input method impact device selection?

• Rotate

• Touch

• Click wheel

• Rollerball

• Pinch & zoom


• Gestures


8/20/2013

8

Mobile Proprietary Network Access: Are you required to test particular devices?

• Does your mobile testing need to

pass an audit or regulations?

• Are there VPN considerations?


Responsive Web Design: Do you need to check content and layout on different devices?

• People often use both at the same time.

For more on RWD, see:

• http://blog.creativemagma.com/featured/2209/





8/20/2013

9

exercise: build a device matrix (or mind map)

• With your team, determine what factors impact your decision.


• Build your own device matrix or mindmap to identify what devices to test.



budget

procurement

time

usage statistics


operating systems

network carriers

m-commerce

mobile browsers

cookies

installation


input methods


proprietary network


flash


8/20/2013

10




Class A

Class B

Class C

Class D

Class F


8/20/2013

1

Mobile User Interface, Design & User Experience

section 3

© Karen N. Johnson, 2013 1 Build Your Mobile Testing Expertise v7

UX Testing: How will you decide what to test? Possible Factors:

menu styles

navigation

input selection

UI controls

custom controls

links

lists

colors & symbols

currency

images

video

completing forms

login

search, filter & sort

app version

sitemap

RWD

accessibility



8/20/2013

2

Menu Styles

Springboard List


Menu Styles

Tab Gallery


8/20/2013

3

Menu Styles

Carousel Metaphor


Navigating

Giving the user a unique way to move forward & back.


8/20/2013

4

Navigating

• Paging

• Forward & backward

• Cookie crumb trail (websites)


Navigating

• Rotate

• Pinch/zoom

• Accelerometer

• Vertical/horizontal

• Paging forward & backward


8/20/2013

5

Input Selector

• Rotate

• Touchscreen

• Click wheel

• Rollerball

• Pinch & zoom



UI controls

• Entry fields

• Lookup buttons

• Radio buttons

• Drop downs


8/20/2013

6

Custom Controls

• The slider bar on AA’s mobile app is an unusual custom UI control.

• Does your app have any custom controls?


Links

• Do links work?

• Does your site have links to other sites?

• What about the sitemap links?


8/20/2013

7

Lists

• Alignment

• Consistency

• Disruptions from ads or 3rd party feeds


Color-alternating lists

• Grouped rows

• Zebra stripping lists

• Text Alignment

• User Accessibility


8/20/2013

8

Colors & Symbols

• Symbols

• Spacing

• Rotating

• Columns


Currency

User interface considerations with multiple currencies: is the correct symbol for the currency shown?


8/20/2013

9

Images

• Where’s the image?

• You might use your desktop to see if the issue occurs outside of the mobile environment.


Images

• Size of files

• Cache

• Refresh rate

• Full screen

• Wi-Fi drops connection

• Rotate

• Pinch/zoom

• Accelerometer

• Vertical or horizontal


8/20/2013

10

Video & Streaming

• Flash?

• Videos

• Connection speed

• Availability


Completing Forms

At some point, a user needs to enter information. Minimizing input – such as offering smart fill fields on location awareness helps to cut down on the amount of information a user has to enter.


8/20/2013

11

Completing Forms

• Sign-in and search are two of the most common forms on mobile devices.


Login


8/20/2013

12

Passwords

• Case sensitive

• Changing the case

• Seeing the text entry

• Staying logged in (cookies)


Search

• Wildcards

• Text entry

• Case sensitivity

• Alpha and numeric entry

• Dropdowns

• Location based searching


8/20/2013

13

Search & Filter

• Location-based

filtering

• UX considerations

on filters, applying

& clearing

• UX considerations

– scrolling through

many search

results


Search & Sort

When results are pages long on a mobile device, offer the user a means to sort or refine the result set.


8/20/2013

14

Search

• Are there fields that are prefilled based on date, location or time?

• OpenTable is an example of time and location.

• Are there fields that are not offered on mobile but are offered on the desktop version?

• Is there a search button? Is there another method to start or run the search?

• Is there a way to refine the results?


Search: filtering results


8/20/2013

15

A short lesson on responsive design



Phone vs. tablet

People often use both at the same time.

Testing by planning user scenarios is a good way to uncover and identify situations to test.







8/20/2013

16

App Version


• A version page is an opportunity to once again show the app icon and/or company logo.

• What about other maintenance pages?

Sitemaps

“They visually represent the relationship of content to other content and provide a map for how the user will travel through the informational space.” Brian Fling, author Mobile Design and Development


8/20/2013

17

exercise: build a list of ideas for testing based on UX factors

• With your team, determine what UX factor influence testing


• Build your own checklist or mindmap to identify what devices to test.


UX Testing: How will you decide what to test? Possible Factors:

menu styles

navigation

input selection

UI controls

custom controls

links

lists

colors & symbols

currency

images

video

completing forms

login

search, filter & sort

app version

sitemap

RWD

accessibility



8/20/2013

1

Mobile App & Device Settings section 4

© Karen N. Johnson, 2013 1 Build Your Mobile Testing Expertise v7

app, device & browser settings

• Overall app permissions

• Specific app permissions

• App permission-dependencies

• Device settings

• Browser settings

• Cookies

• Wireless & network settings

• Data Roaming

• SD card settings

• Swype

• Display settings

• Battery Settings/Status

• Task Killers


8/20/2013

2

Overall App Permissions

Know the app permissions your mobile app requires.

Consider what the user may or may not have enabled on the device for all applications.


Specific App permissions

What permissions does your application require?


8/20/2013

3

App permission-dependencies

If your app interfaces with 3rd party data such as location finding, what happens when that access is off, down or unavailable?


Device settings

• Consider the environment of the device: what has been enabled or not.

• Consider the ability or restrictions to downloading and installing applications.

• Consider upgrade scenarios and whether permissions have been changed or reset.

• Install testing has returned.


8/20/2013

4

Browser settings


Language settings

• Finding what languages are available on the device

• Choosing a different language

• Accessing and using a site or an app that’s multilingual

• Using different keyboards on devices


8/20/2013

5

Cookies

• Does your site use cookies?

• Check the login process with cookies on/off.


Wireless & Network Settings

Have you ever tested your website when connectivity is down or disrupted? This scenario is more likely with mobile devices.


8/20/2013

6

Network Mode

CDMA is used in the US (also in China and India) but everywhere else uses GSM or other modes of connection. It can be challenging to test like a user when you live in a different country and some options are not available on devices where you are located.


Network search

Users can switch from and to an application or browser session while other activities are taking place (including incoming phone calls) does your app handle that disruption?


8/20/2013

7

Connecting to a network

A user could switch to your app or website when they’re not connected.


Connection errors

If your app uses the SD card for storage – what if the card is already full or another app writes to the card?


8/20/2013

8

Data roaming

Would you enable your testing team to leave the office and experience mobile as they travel around the office?


SD card settings

• Does your app store data on a card?

• What if the card is removed?

• What if a different card is inserted?


8/20/2013

9

SIM cards

• A SIM card or Subscriber Identity Module is a chip.

• SIM holds personal identity information, cell number, phone book, text messages and other data.

• Again, what happens if the card is full? Changed? Removed?


Swype settings

• Does your app work with swyping or voice input?


8/20/2013

10

Display settings

Consider whether a user has made device settings that can impact how your application appears.


Battery conditions

• Have you tested your app under low battery conditions?

• Is your application responsible for battery drainage?

- See Julian Harty’s work

- See Android Forum


8/20/2013

11

Battery conditions: power save


Is this a variable you need to consider in testing?

Task Killers


• What background tasks are running?

• What data is being synced?

• Does a Task Killer help or not?

Android

http://www.pcworld.com/article/244067/taskkiller_apps_will_they_help_or_hurt_your_battery_life.html

IOS

http://www.youtube.com/watch?v=nEJA1yUgR80









8/20/2013

12

Exercise: Device Settings

• What device settings make an impact on the app you’re testing?

• What app permissions does your app require?

• What settings make an impact on mobile browser testing?

1. Identify what settings need to be factored into testing or configuring a device.

2. Build a mind map or outline structure to group these settings together in a logical way.

3. Should the settings be grouped according to device manufacturers? Such as grouping Nokia settings vs. Samsung settings?


8/20/2013

1

The Competitive Mobile Marketplace

section 5


What do mobile apps compete on?

According to uTest’s white paper from the product Applause, there are intuitive metrics such as:

• content

• elegance

• pricing

• usability

• and satisfaction


8/20/2013

2

How steep is the competition?

• Apple Store: 700,00+ apps

• Google Play: 700,000+ apps

• Blackberry App World: 100,000+ apps

• By 2015, the International Data Corporation (IDC) predicts 182.7 billion mobile apps will be downloaded

• In 2010, 10.7 billion apps were downloaded

- source: uTest Applause white paper


MARKET RANKING


8/20/2013

3

Market Rankings

Applyzer is a product that watches market rankings. Notice how they segregate data by country and mobile app category.


Market Rankings

Distimo is another market watching application.


8/20/2013

4

Market Rankings


Market Rankings

RankedNow is another product tracking the competitive market.


8/20/2013

5

MOBILE COMPETITORS


Competitive Analysis – Vertical Competition mobile app to app

AccuWeather The Weather Channel


8/20/2013

6

Competitive Analysis – Vertical Competition tablet app to app


Competitive Analysis – Mobile Competition the app Shazam vs. other ways to spend time

Notice how competitors might not be offering the same product but compete equally well for a user’s time, purchase and use.


8/20/2013

7

Mobile competition: Shazam’s “competitors”

See how this “competitor” isn’t offering the same product but is vying for the user’s time.

This competitor is a more typical “competitor.”


Mobile competition: a sample of differences by store

Competitors vary from store to store.


8/20/2013

8

MARKET COMMENTS


Charmin – Sit or Squat

This Charmin app was liked until the app was tied to Facebook and requires a FB Account in order to login or comment. Users have scathing comments about the change.


8/20/2013

9

Tide Stain Brain

The Tide app made a change when the company removed user comments on how to Remove stains and in turn infuriated the user community.


Chlorex myStain

The app gains respect and ratings by now being solely focused on pushing their product.


8/20/2013

10

Out of Milk

Note the number of users for the free version. And also note the overall ratings by users.


Spotify

Note how users comment on updates that cause issues or comment on missing functionality.


8/20/2013

11


Competitive Analysis Exercise: 1. Select two mobile applications that

would seem to be competitors such as AccuWeather and the Weather Channel.

2. Review each app for the “intuitive metrics” listed as factors.

3. Make notations on each of the factors for both apps.

4. For satisfaction, you can use either your satisfaction with the app, app store comments or a combination of both.

5. If you were to apply a scoring to each app, which app would you choose over the other app?


build your mobile testing expertise

Technology