innovations in mobile testing: expanding your test plan

T14TrackSession4/21/20162:00PM

"InnovationsinMobileTesting:ExpandingYourTestPlan"

Presented by:

Melissa Tondi

Denver Automation and Quality Engineering

Broughttoyouby:

340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com

Melissa Tondi Denver Automation and Quality Engineering

Melissa Tondi has spent most of her career working within testing teams- concentrating on functional, performance, security, and mobile testing techniques. In her current roles, she is the founder of Denver Mobile and Quality (DMAQ) and, as Head of SQE at ShopatHome.com, she is assisting teams to continuously improve the design, build, test, and delivery of quality software. Throughout her career in the software test and quality engineering fields she has focused on organizing testing teams around three major tenets—efficiency, innovation, and culture. Melissa’s previous roles have included Vice President of Professional Services for a Mobile-focused startup, Director of Software Quality Engineering in the world’s leading education company; QA consultant for health care, finance, and software-as-a-service industries; and president of the Software Quality Association of Denver.

4/8/16

1

Mobile Dev+Test 2016 - Innovations in Mobile Testing: Expanding Your Test Plan

Melissa Tondi

Takeaways and Agenda

→  Level Setting: Your current (or new) mobile testing strategy –  Understanding your Mobile Solutions

•  Mobile Web: RWD and AWD •  Apps: Native and Hybrid, Near Native •  IoT (a brief definition)

→  Expanding your Test Plan - Test Types that Mobile has Introduced

→  Expanding your Test Plan - Accessibility →  Expanding your Test Plan - Using Simulators and Emulators →  Device proliferation and using the Device Matrix →  Environments and Tools – Proxies, Performance, CI

4/8/16

2

Level Setting - Understanding your Mobile Solution: Mobile Web

What is “Mobile”? Mobile web sites designed to match every web-enabled device. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to an M Site or similar.

–  Responsive Web Design – an approach aimed at building sites to provide an appropriate viewing experience (reading and navigation with minimal resizing, panning, and scrolling) across a wide range of devices. It adapts the layout to the viewing environment.

•  No matter how much you resize the screen, that same layout will automatically respond to that size

•  A single codebase for a website can serve both mobile and desktop users. So instead of having two separate versions of your site (one desktop site and one mobile site), you will have a single responsive site that will accommodate simultaneously mobile and desktop users


What is “Mobile”? –  Adaptive Web Design – uses the server to detect the device that’s

being used. A separate template is maintained for each device that is supported. There are several distinct layouts for multiple screen sizes and the layout used depends on the screen size used.

•  The goal is a highly differentiated experience that is built for the specific intent of the mobile customer

•  It allows you to specifically capture user intent more precisely on a mobile device

4/8/16

3


Responsive Web Design Pros and Cons –  Harder to build initially, but is flexible as new devices and IoT

(Internet of Things) expand users’ experiences –  Images are scaled down rather than resized or eliminated

altogether therefore, they generally take longer to load

Adaptive Web Design Pros and Cons –  Faster to initially build, but requires maintenance as new devices

are supported –  Because they are created for mobile users they have smaller and

optimized images. Images are loaded to meet the device resolution and screen size, making page load time more efficient.

Level Setting - Understanding your Mobile Solution: IoT

A Brief Definition… IoT (Internet of Things): a network of physical objects that feature an IP address for internet connectivity and the communication that occurs between these objects and other Internet-enabled devices and systems

→  Our current, common IP Addresses are based on IPv4. It uses 32-bit addresses and is limited to around 4.3 billion addresses

→  IPv6 is slated to replace IPv4 and uses 128-bit addresses with 3.4×1038 unique addresses of capability

4/8/16

4

Level Setting - Understanding your Mobile Solution: Things to Know

RWD and AWD →  Use CSS (Cascading Style Sheets) to adjust layout →  The spots where it changes are called “breakpoints” →  Functionality may be hidden, or workflows may be adjusted based upon

size →  Usually adjusted to vertically scroll Tools →  BrowserStack

–  https://www.browserstack.com/ →  Mobile-Friendly Test

–  https://www.google.com/webmasters/tools/mobile-friendly/ →  responsive test

–  http://responsivetest.net/#u=http://www.zootemplate.com|1024|768|1 →  Responsinator

–  https://www.responsinator.com/


IoT →  Gray Box Testing

–  Understanding the Architecture, OS, Third Party Hardware, Apps, Firmware, Connectivity Protocols

→  Real Time Operating System (RTOS) –  Operating Systems that serve real-time application process data as it

comes in, typically without buffering delays →  Service Virtualization

–  Emulates only the behavior of the specific dependent component. Instead of virtualizing an entire database, you monitor how the application interacts with the database and then replicate the related database behavior.

4/8/16

5

Level Setting - Understanding your Mobile Solution: Apps

Native Apps: Are coded with a specific programming language. They are fast, reliable, and powerful, and are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform. These apps are written in the language specified by the platform →  Objective-C or Swift for iOS using the iOS SDK (Software

Developer Kit) →  Java for Android using the Android SDK


Hybrid Apps: These rely on development frameworks and are written with the same technology used for websites and mobile web implementations. They are hosted or executed inside a native container on a mobile device. They ensure cross-platform compatibility and can access the phone’s hardware (camera, GPS, user’s contacts, etc.) These are regarded as web applications wrapped inside of a mobile app. The mobile app wrapper itself uses a web viewer (a scaled down version of a browser) to display the web pages. Web pages are developed with the same technologies that are used for both traditional and mobile web like:

–  CSS –  HTML –  JavaScript

4/8/16

6


Near Native: These have business logic core written in a language other than what is provided by the platform. The UI and other aspects make full use of native components and APIs. The result is an app that can be targeted to multiple platforms but run natively on each. There is a somewhat high percentage of code reuse across platforms. Example: Apps written in Appcelerator's Titanium. They are generally written in Javascript, but make use of all native components. The app can be targeted to multiple platforms but it runs natively on each


→  Apps require much greater hardware testing –  3D graphics –  Device sensors

→  Apps need to work on all screen sizes –  Avoid wasted space

→  Apps define XML layouts similar to HTML –  Node Structure –  Static Content (Images) –  Dynamic Content (Color, text)

4/8/16

7

Level Setting - Understanding your Mobile Solution: Operating Systems

Android: →  Custom version of Linux →  Apps built in java →  Chrome Browser

(WebKit) →  Over 20,000 different

Android devices →  Custom Manufacturer

code →  Forked versions of

Android (like the Kindle)

iOS: →  Proprietary Apple OS

built on BSD (Berkeley Software Distribution)

→  Only works on Apple hardware

→  Apps built in Objective C and Swift

→  Safari browser (WebKit) →  Less than 20 different

devices →  Low OS variance

Expanding your Test Plan – Test Types

Peripheral –  Wireless Testing – NFC (Near Field Communication), Bluetooth/

Bluetooth LE Accessory, Stylus –  Wired –  Internal to Device – Headphone Jack, Keyboard –  External to Phone - CC Readers, Bar Code Scanners

Connection Testing –  USB Power/Data –  4G/LTE/CDMA+/Wimax –  3G/GSM/CDMA, 2G/Analog, WiFi, Hotspot Generation –  Computer Tethering –  Carrier –  Throttle – Limiting bandwidth and measuring an App’s performance

4/8/16

8


Gestures –  Swipe, tap, double tap, drag, pinch, spread expand, flick, shake,

orient, tilt, press, press and hold, press and tap, press and drag, swipe and hold, rotate.

Interruption Tests –  Controlled - Plug in/out USB, power, and headphone, Home Button,

Power Button, Navigating to another App, Save State –  Uncontrolled/”Elevator” test - SMS, Phone Call, Notifications


Syncing –  Updating two or more locations to ensure applicable files are

equivalent. –  App to Cloud, App to Computer, App to App, Device to Device

Internal Hardware Integration –  Camera, GPS, Accelerometer, Battery Drain, SIM Card, Volume,

SMS, Microphone, Speakerphone

4/8/16

9

Expanding your Test Plan – Accessibility

Worldwide Web Consortium (W3C) –  http://www.w3.org/TR/WCAG20/#guidelines - Web

https://www.w3.org/TR/wcag2ict/ - Mobile You’ll want to determine which Level is appropriate for your organization and users, but here’s the gist:

–  Level A – A good place to start. Great for organizations that already have a product in use and who want to establish a baseline for accessibility conformance.

–  Level AA – The next step. This level means that most people will be able to use your site/product in most situations. Many education and government agencies require this level.

–  Level AAA – The most difficult to achieve and maintain. In rare situations, this may be required, but the W3C makes it clear that it is not possible to satisfy all Level AAA success criteria for some content.

Expanding your Test Plan – Accessibility

Exercise – Android

Settings > My Device >Accessibility > Vision > TalkBack > Turn TalkBack on

Exercise - iOS For iOS phones, go to Settings > General >Accessibility > VoiceOver > Turn VoiceOver on

4/8/16

10

Expanding your Test Plan – Accessibility: Things to Know (Android)

Android Gestures All gestures for TalkBack require only 1 finger. →  Touch or move around the screen – speaks and select the item under

your finger →  Swipe right or down – move to next item on screen →  Swipe left or up – move to previous item on screen →  Double-tap – select focused item →  Swipe up then down – move to first item on screen →  Swipe down then up – move to last item on screen →  Swipe right then left – scroll forward on page or move slider →  Swipe left then right – scroll back on page or move slider →  Swipe up then left – Home button →  Swipe down then left – Back button

Expanding your Test Plan – Accessibility: Things to Know (iOS)

iOS Gestures →  1 finger gestures:

–  Touch or move around the screen – speaks and select the item under your finger

–  Tap 2 times fast – activate the selected item –  Swipe left or right – selects the previous or next item –  Swipe up or down – move to the selected rotor option

→  2 finger gestures: –  Swipe up – read all from first object in selected area –  Swipe down – read all from selected item –  Scrub back and forth – go back, cancel, close pop-up –  Pinch out/in – select/deselect –  Rotate to open the web rotor and select rotor items (see the last point

for 1 finger gestures) →  3 finger gestures:

–  Swipe left or right – move to previous or next horizontal screen/page –  Swipe up or down – move/scroll down or up

4/8/16

11

Expanding your Test Plan– Emulators, Simulators

Choosing Emulators and Simulators

–  A lab is not complete without a mixture of both

•  A Simulator mimics only the software. iOS has one, but it does not simulate any of the iPhone hardware layer

•  An Emulator mimics both software and hardware. Android has one, but only emulates CPU and Memory

Expanding your Test Plan– Emulators, Simulators

When to use Simulators/Emulators

→  Testing the Function •  Broken buttons, missing images, correct formatting for vertical

and horizontal modes •  RWD/AWD or mobile site elements display correctly

→  Previewing the Form

•  Demo basic design and layout mockups before significant development investment is spent on polishing the UI

•  See the application within the frame of a real device, compared to the window of a development program

4/8/16

12

Expanding your Test Plan – Emulators, Simulators

When to use Simulators/Emulators

→  Accelerating Development Time •  For simple visual checks, opening an app in a device emulator

can be as simple as switching windows, compared to fully loading a physical device

→  Test more frequency with quick checks on an emulator, thus finding bugs faster and with greater context

Expanding your Test Plan – Emulators, Simulators: Things to Know

Simulators 16-Bit Display Operating System User Interface

Emulators Keyboard and Buttons WiFi Simulated SIM Card System on a Chip Sound Chip Flash Memory

4/8/16

13

Expanding your Test Plan – Device Testing Things to Know

24-Bit Display Touchscreen Multi-Touch Camera & Video Voice Recognition True Sound Quality USB Connections Bluetooth SD Cards Network Data Real Phone Calls Dual SIM Cards

Battery State Device Temperature Real GPS & Sensors Pre-Installed Apps App Store Access App Installation Email Sync Contacts Sync Calendar Sync App Log Encryption Network Security Fingerprint Scanner

Build Your Lab

4/8/16

14

Build Your Lab – Getting Started

Determining your Device Matrix →  Operating System

–  OS customizations, missing libraries, driver issues →  Screen Size

–  Rendering issues, usability, missing layouts →  Pixel Density

–  Density Independence, missing layouts →  Aspect Ratio

–  X,Y calculations, overlapping panels, display issues →  System on a Chip (SoC)

–  Hardware performance, Instruction set, battery, signal →  Carrier

–  Network protocol, speed, responsiveness, packet loss


Determining your Device Matrix – The Goal →  Efficiency, not necessarily 100% coverage →  Build your lab to cover both apps and mobile web →  Define a list of categories of devices →  Adjust the number of devices based on reasonable coverage

4/8/16

15


Determining your Device Matrix – Example Categories →  Common: Matches the Most Common Display Configuration →  Newest: Latest OS, Largest Screen, Highest Resolution →  Oldest: Oldest, smallest, slowest device →  Abnormal: Non-standard OS, Aspect Ratio, Orientation, and

Size →  Popular: In Terms of Sales →  Budget: Low-priced New Model →  Flagship: By Manufacturer (for example)

–  iPhone 7 (TBD) –  Galaxy Note –  Google Nexus –  LG G5

→  Catch-All

Building your Lab – Android

Category DeviceName OS Size Density Resolu8on DPI AR Protocol SystemonChip

Newest Sam.GalaxyS 4.2.2 Normal xhdpi 1920x1080 441 16:9 LTE/GSM QualcommS4

Flagship LGNexus4 4.2 Normal xhdpi 768x1280 318 3:5 GSM QualcommS4

Oldest HTCTaNoo 1.6 Small ldpi 320x240 143 4:3 GSM QualcommS1

Popular Sam.GalaxyS3 4.1.2 Normal xhdpi 1280x720 326 16:9 CDMA SamsungExynos

Common Mot.Droid3 2.3.4 Normal hdpi 960x540 275 16:9 GSM CortexA9

Abnormal LGOpTmusVU 4.0 Large xhdpi 1024x468 256 4:3 GSM NvidiaTegra3

Budget DellVenue 2.2 Normal mdpi 480x800 228 3:5 GSM SnapdragonS2

Catch-all SonyXperiaP 2.3 Normal hdpi 960x540 275 16:9 GSM SonyNovaThor

4/8/16

16

Building your Lab - iOS

Device

Name

OS Display Aspect SoC Carrier

Newest iPhone6 8 5.5”1920x1080ppi 9:16 A8Chipwith*

MoTon

Coprocessor

T-Mobile

Oldest iPhone3g 6 3.5”320x480165ppi 2:3 AppleA3 AT&T

Common iPhone5s 7/8 4”1136x640326ppi 9:16 AppleA5 Verizon

Popular iPhone6+ 6 3.5”640x960330ppi 2:3 AppleA4 Sprint

iPad

(Re8na)

iPadAir 7 9.7”2048x1536264ppi 3:4 AppleA7 Verizon

iPod iPodTouch

(4thgen)

5 3.5”640x960326ppi 2:3 AppleA4 WiFi

Mini iPadMini 8 7”1024x768162ppi 3:4 AppleA5 AT&T

Build Your Lab – Summary

Determining your Device Matrix – End Result →  Logically Select Devices →  A Large Number of Permutations can be Covered with a

Handful of Devices →  As New Devices are Announced, Run them through the DM

to Determine whether to Purchase

4/8/16

17

Environments

Test Environments –  Browserstack.com or SauceLabs.com

•  Online Simulators for any OS or mobile device •  Useful for troubleshooting and regression testing •  Defects aren’t always reproducible on real devices •  Automation Cloud (supports WebDriver) •  Quick rendering comparisons across multiple devices

–  Hardware Rack •  High Power USB Hub necessary to keep tablets charged •  Stands to maintain device orientation

Tools – Things to Know

Mobile Automation Tools - Noteworthy →  Mobile automation is still in its infancy →  Mobile devices lack good automation hooks →  Most tools can only control an app →  Many automation tools require modification of the app

–  Custom libraries, recompilation, or external application

→  Many automation tools can’t automate the stock web browsers –  safari, chrome

→  Most tools can’t automate outside the app (OS, configuration, etc.)

4/8/16

18

Tools - Open Source vs “Closed”

→  Open Source is customizable, and maintainable by anyone trained

→  Open source tools typically require a custom framework

→  Anyone with a coding background can work on the tests

→  WebDriver is the W3C standard for web application testing

→  Closed tools are more usable for those without development backgrounds

→  Closed tools are expensive

→  Closed tools offer better reporting and integrations out of the box

→  Closed tools mean it may be hard to find people with experience in specific tools

Tools - Open Source Open Source Options →  Selenium

–  Multiple side projects support mobile testing –  Selenium experience is very common

→  Appium, Android Driver, Robotium –  Appium being merged as part of Selenium 3.0 –  Acts as a remote webdriver node –  Run tests on simulators or physical devices (USB) –  Don’t have to change the app –  Can automate default browsers (chrome / safari) –  Can automate android / iOS apps –  Same libraries used for desktop automation –  Used in Browserstack and SauceLabs

→  MonkeyTalk –  GUI based –  Selenium based –  Sort-of Free and Open source

4/8/16

19

Tools – “Closed”

→  EggPlant –  Image based

automation tool –  Can control full device

(not just apps) →  HP UFT

–  Large Market Share –  GUI Based –  Expensive

→  Ranorex –  GUI Based –  Supports android 2.2+

→  TestComplete –  GUI or Code driven –  Multiple language

choices –  Also supports windows

applications

Other Tools

Proxies →  A proxy intercepts all HTTP traffic from a computer or

mobile device →  Traffic can be inspected, paused, re-executed, and

manipulated →  Valuable for debugging purposes →  Can be included with defect reports →  Similar to chrome dev tools →  Examples

–  Fiddler (windows) –  Charles (Mac) –  BrowserMobProxy (Selenium)

4/8/16

20

Tools – Other Considerations

Performance Testing No such thing as client side performance testing →  You can do memory, battery, CPU, RAM profiling during

normal testing activities →  Generating Server load

–  HTTP Traffic

→  Most load testing tools will support mobile –  Configure it as a proxy –  Record traffic –  Parameterize –  Generate Load

Tools – Other Considerations

Continuous Integration →  A build server (Jenkins, TeamCity, Bamboo, CircleCI) kicks

off tests periodically →  Can view pass/fail status, troubleshoot, etc. →  Can be hooked up with most automation tools →  Tests can be executed against physical devices, local

simulators, or cloud simulators →  Speed up time through parallel execution →  Setup a local GRID cluster for faster results →  High power USB hub for tablets

4/8/16

21

Contact Information

→ email: [email protected]

→ Twitter: @melissatondi

→ Blog: MelissaTondi.blogspot.com

→ LinkedIn: Melissa Tondi