preparing your qa team for mobile testing
DESCRIPTION
Rewriting your Master Testing Strategy is in order when you look to tackle your first mobile initiative. This includes getting your test environments ready for testing. Often test environments have been on closed internal networks that are not accessible from the internet. Delivering and testing a mobile application on a 3G or 4G network changes all that. Coming up with a supported platform strategy is also paramount to success. One needs to take network speed, device features, screen size and resolution as well as multi-tasking life-cycles into effect when testing. QA testers will need to become familiar on how to service the devices they test with as well. Additionally the team will need to incorporate field testing prior to delivery. This presentation will organize the challenges a modern QA team has to contend with, and make some strong suggestions on how to craft a respectable Mobile Test Strategy.TRANSCRIPT
![Page 1: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/1.jpg)
Preparing Your QA Team for Mobile TestingWhere Reality and New Beginnings Meet
![Page 2: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/2.jpg)
@ggeoffreggeoffre.com
20 years of experience in the development, marketing, sales, and leadership of computer
software
A C T S
![Page 3: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/3.jpg)
Where did the term "bug" come from?66 years ago on September 9th 1947, operators of the Mark II Aiken Relay Computer being tested at Harvard University, found something curious trapped between points at Relay #70, Panel F.
NOT ENTIRELY TRUE
![Page 4: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/4.jpg)
Bugs not Defects!
![Page 5: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/5.jpg)
Bugs not Defects!
![Page 6: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/6.jpg)
Bugs not Defects!
![Page 7: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/7.jpg)
Bugs not Defects!
![Page 8: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/8.jpg)
Bugs not Defects!
Outside the scope of most acceptance or unit testing
Tend to design around them not to them
Defensive programming techniques employed
More likely hardware based than software based
Environment plays a major role discovering them
Magnified greatly with mobile based development
![Page 9: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/9.jpg)
Ubiquitous
(not just mobile)
![Page 10: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/10.jpg)
Three Eras of Computing
mainframe personal
ubiquitous
![Page 11: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/11.jpg)
Three Human to Computer Ratios
many one one one
one many
: :
:
![Page 12: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/12.jpg)
Leaving Other Eras Behind
Single, closed computing systems
Three "major" user upgradable operating systems
Maybe five “major” browsers
Virtualization possible to contain testing options
![Page 19: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/19.jpg)
Mobile Testing Challenge
Device Manufactures and Device Models
Carrier Networks and Network Speeds
OS Versions and API Updates
Different Features for Different Phones
![Page 20: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/20.jpg)
Android Fragmentation 11,868 Distinct Android devices seen this year
3,997 Distinct Android devices seen last year
682,000 Devices surveyed for this report
47.5% Samsung's share of those devices
8 Android versions still in use
37.9% Android users on Jelly Bean
http://opensignal.com/reports/fragmentation-2013/
![Page 21: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/21.jpg)
Android Fragmentation 18,796 Distinct Android devices seen this year
11,868 Distinct Android devices seen last year
682,000 Devices surveyed for this report
43% Samsung's share of those devices
7 Android versions still in use
46% Android users on Jelly Bean
http://opensignal.com/reports/2014/android-fragmentation/
UPDATEDUPDATED
![Page 23: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/23.jpg)
http://opensignal.com/reports/2014/android-fragmentation/
UPDATEDUPDATED
![Page 24: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/24.jpg)
http://developer.android.com/about/dashboards/index.html
API Versions
![Page 25: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/25.jpg)
http://developer.android.com/about/dashboards/index.html
API VersionsUPDATEDUPDATED
![Page 27: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/27.jpg)
UPDATEDUPDATED
http://opensignal.com/reports/2014/android-fragmentation/
![Page 29: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/29.jpg)
UPDATEDUPDATED
http://opensignal.com/reports/2014/android-fragmentation/
![Page 31: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/31.jpg)
UPDATEDUPDATED
http://opensignal.com/reports/2014/android-fragmentation/
![Page 32: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/32.jpg)
NEW!NEW!
http://opensignal.com/reports/2014/android-fragmentation/
![Page 33: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/33.jpg)
http://developer.android.com/about/dashboards/index.html
Screen Resolutions
![Page 34: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/34.jpg)
http://developer.android.com/about/dashboards/index.html
Screen ResolutionsUPDATEDUPDATED
![Page 35: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/35.jpg)
![Page 36: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/36.jpg)
iOS Omneity 12* Distinct iOS devices seen this year
8 Distinct iOS devices seen last year
10M Events/hr surveyed for this report
100% Apple's share of those devices
3 iOS versions still in use
80.7% iOS users on iOS 7 (97.1% iOS 6+7)
http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
![Page 37: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/37.jpg)
iOS Omneity 17* Distinct iOS devices seen this year
12 Distinct iOS devices seen last year
10M Events/hr surveyed for this report
100% Apple's share of those devices
3 iOS versions still in use
80.7% iOS users on iOS 7 (97.1% iOS 6+7)
http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
UPDATEDUPDATED
![Page 38: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/38.jpg)
http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
iOS Adoption Rates
![Page 39: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/39.jpg)
http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
iOS Adoption RatesUPDATEDUPDATED
![Page 40: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/40.jpg)
iOS Usage
http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
![Page 41: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/41.jpg)
iOS Usage
https://www.fiksu.com/resources/ios_8_tracker
UPDATEDUPDATED
![Page 42: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/42.jpg)
iPhone Usage
http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
![Page 43: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/43.jpg)
iPhone UsageUPDATEDUPDATED
https://www.fiksu.com/resources/ios_8_tracker
![Page 44: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/44.jpg)
iPad Usage
http://www.fiksu.com/iOS-7-iPhone-5s-5c-Usage-Tracker
![Page 45: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/45.jpg)
iPad UsageUPDATEDUPDATED
https://www.fiksu.com/resources/ios_8_tracker
![Page 46: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/46.jpg)
NEW!NEW!
![Page 47: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/47.jpg)
700 Million iOS Devices
![Page 48: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/48.jpg)
800 Million iOS DevicesUPDATEDUPDATED
![Page 49: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/49.jpg)
900 Million Android Devices
![Page 50: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/50.jpg)
1Billion Android DevicesUPDATEDUPDATED
![Page 51: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/51.jpg)
Smartphone Markershare
![Page 52: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/52.jpg)
Smartphone MarkershareUPDATEDUPDATED
Worldwide
USA
comscore.comidc.com
![Page 53: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/53.jpg)
Planning for the Experience
![Page 54: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/54.jpg)
NFRs of MobileImmutable requirements set before the project even begins
Dictated to you by many different outside organizations
Typically misunderstood during the design phase
Often ignored by functional based acceptance testing
Not typically caught by code quality unit testing
Highly improbable if not impossible to avoid absolutely
Things that every mobile user deals with every single day
![Page 55: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/55.jpg)
Examples of Mobile NFRs
Wireless Network Access, Latency and Speed
Screen Size, Aspect Ratio, Resolution and Orientation
Multi-tasking Interruptions, Time Limits and Unexpected Shutdowns
User Specified Configurations and Privacy Settings
Processor Performance and Availability of On-Device Storage
![Page 56: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/56.jpg)
Examples of Mobile NFRs
GPS Speed, Accuracy and Availability
Photo Resolution, Focal Length, Lowlight Ability
Platform Versions, Releases and Updates
Code Signing, Packaging and Distribution Methods
![Page 57: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/57.jpg)
![Page 58: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/58.jpg)
Network Reachability
Degraded performance depending on network latency
Slows down performance
Some areas are not connected at all
Stops functionality cold
Bandwidth utilization and data plans
Frustrates users and impacts runtime costs
Users are actually Mobile and moving around!
Don’t always access the network from the same WiFi
![Page 59: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/59.jpg)
![Page 60: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/60.jpg)
Network Speeds
http://www.pcmag.com/fastest-mobile-networks
![Page 61: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/61.jpg)
![Page 62: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/62.jpg)
Screen Size and Orientation
Layout of Information
Can all of the information be viewed
Scrolling To Access Information
Usability of hidden navigational components
Logic Triggered by Switching
Refresh screen data with each rotation
Deployment to Tablet Devices
![Page 63: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/63.jpg)
![Page 64: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/64.jpg)
Multitasking Interruptions
Interruption from other Apps
Incoming Phone Calls
Tweets, Posts, Email etc…
Background Operations
Downloads, Tracking, Music
External Connectivity
Bluetooth, WiFi Screen Sharing
![Page 65: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/65.jpg)
![Page 66: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/66.jpg)
Battery Life
Using lots of device features
GPS, Network, Video, etc...
User disabling device features to prolong battery life
Affects use cases and therefore should affect testing scenarios
User deletes Apps to find out which one is causing problems
![Page 67: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/67.jpg)
![Page 68: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/68.jpg)
Accessing Online APIs
Keep in mind that users do not have to update
Loosely coupled APIs with optional fields helps with flexibility
Layers of services may simplify end user support, but complicates operations (Seperate API Layer for each User Group or Platform)
Large APIs accounting for all needs get updated more frequently than specialized smaller APIs for a single Apps needs.
![Page 69: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/69.jpg)
![Page 70: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/70.jpg)
Rewrite Master Testing Strategy
Connected and disconnected states
Internet facing test environments
Multiple sized devices for each platform version
Multitasking scenarios per platform
Track battery life in different use case scenarios
Plan for live field testing to gauge responsiveness
Review instrumentation results for memory use
![Page 71: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/71.jpg)
Impacted Testing AreasLevels
Unit
Integration
System
Acceptance
Methods
Static
Dynamic
White-Box
Black-Box
Gray-Box
Types
Installation
Compatibility
Smoke
Regression
Usability
Accessibility
Security
Internationalization and Localization
Performance
![Page 72: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/72.jpg)
Primary Test Cases
Rotate the device (on every single screen)
Receive a phone call (at the worst possible times)
Switch to/from another app (at the worst possible times)
Resolution Specific Content and Graphics
Disconnect from the network entirely (airplane mode)
1
![Page 73: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/73.jpg)
Edge Test Cases
Switch between wi-fi and wireless networks (device specific)
Battery dies (device shuts off instantly)
Apply an update (with data change)
2
![Page 74: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/74.jpg)
Special Test Cases
Background processing of developed app
App to app communications
Specific interaction with device hardware
Notification or event based triggering
Specific interaction with platform APIs
3
![Page 75: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/75.jpg)
Frustration comes from…
Tester can no longer reproduce the error
Tester can reproduce on physical device, but developer cannot in the simulator
Developer requires physical access to testers device in order to reproduce and diagnose the error
API environment is down or inaccessible
App needs to be removed and re-installed
Device needs to be reset (power, network, settings, full)
![Page 76: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/76.jpg)
Learning How Mobile Works
Shutdown Apps for a Fresh Start
Hard Reset to Recover from Catastrophic Failures
Back-up and Restore the Device to Factory Settings
Delete and Re-Install Apps
How to Create Multi-Tasking Interruptions
Understanding of the Platform’s Notification System
Privacy and Accessibility Settings
![Page 77: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/77.jpg)
Some things that can help
Version number of build, traceable back to the snapshot of the code branch, located within the deployed app on the test device
Screen shots and even video of error occurring on the test device
Simplistic API Test Interface to verify if it is the backend or the app
Adequate supply of physical test devices that can be checked in/out by all team members
![Page 78: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/78.jpg)
Some things that can help
Add validation criterial to every story card
Make version control and change control information of APIs available to all team members
Create an API up/down status page
![Page 79: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/79.jpg)
Important Communications
Report the Device Type - Hardware and OS Versions
Capture the complete software version information at time of failure
App version traceable back to code branch
Fresh install or update from prior version?
API Environment and Version
Perform a Standardized Smoke Test First
Steps to Reproduce
Reproducible on second like device
Reproducible on second un-like device
Reproducible within Simulator/Emulator
![Page 80: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/80.jpg)
Agile Considerationsintroduction of non-functional story cards
addition of non-functional tests to story cards
SCRUM Considerationsco-location and field testing is not practical together
everyone plays every role (anyone can play any role)
personal devices may not be fair game test devices
![Page 81: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/81.jpg)
BETA
What about beta testers?
They are dynamic and certainly fall into the system based compatibility testing space, but…
are more often than not selective function based testers with their own agenda
non-dedicated resources that have higher priorities in life outside of testing your app
in a controlling position as some of your most valuable customers
![Page 82: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/82.jpg)
Manual is too expensive…
On the day of the launch, purchase the app from the store and…
ensure that all of the last minute content changes were applied correctly
access the system and log on to a real account
download the updated version from the store to see if all of your previously stored data is still accessible
![Page 83: Preparing your QA team for mobile testing](https://reader038.vdocuments.mx/reader038/viewer/2022110306/554f455cb4c905524c8b456d/html5/thumbnails/83.jpg)