the basics of mobile app testing & test automation

webinar

The Basics of Mobile App Testing & Test Automation

26 March 2014

Ville-Veikko HelppiTechnical Product Managerville-veikko.helppi@bitbar.com

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

Challenges in App Developmentwebinar

Fact: Most typical application fails when tested across different device variants used by consumers

• Failures: Both SW and HW cause failures• Emulators do not offer realistic platform for dev• Even app installation fails – not tested at all • Application crashes, leaks memory, consumes too

much resources, behaves not as meant to behave• UI scaling/display/resolution – common problem• Application performance is bad, draining battery

Why Apps Fail?webinar

Testdroid Research Available Soon…webinar

What Improves the App Quality?• Automate generic things as much as you can• During the development your app changes –

the same must apply for testing!• Carefully select (testing) technology & partner• Use all 24 hours per day with test automation• Cloud-based platform is the only solution to

get you quickly covered in the global markets

webinar

Mobile Apps for Different Verticalswebinar

Mobile

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• Testdroid has helped thousands of app developers in these subverticals!

• What are the critical elements in each these verticals?

Mobile

Utilities & Tools

Banking & Payment

Retail & Travel

• User Experience!• Resource consumption &

validation (CPU, Mem…)• Fully utilizing hardware –

e.g. Touch screen• Mobile games require

access to graphics APIs (e.g. OpenGL ES)

Mobile

Utilities & Tools

Banking & Payment

Retail & Travel

• Usability!• Functional and

Behavioral testing • Metrics analysis for all

captured data (e.g. logs, screenshots, perf stats)

• Connection with Social Media + other apps

Mobile

Utilities & Tools

Banking & Payment

Retail & Travel

• Performance!• Connectivity, robustness

and durability• Screen orientation

(portrait vs. landscape)• Graphics quality,

streaming capabilities• User profiles

Mobile

Utilities & Tools

Banking & Payment

Retail & Travel

• Security!• Secure connectivity with

back-end systems• Top Quality – extremely

brand sensitive vertical• Compliances and

verification between real devices and infrastructure

Mobile

Utilities & Tools

Banking & Payment

Retail & Travel

• Data!• Connectivity and data

connection with back-end systems

• Configurability of the app• Bad quality can hurt the

brand and leave you without customers

webinar

Prerequisites for Mobile App Testing• Testing environment as authentic as possible– Real devices <-> emulators/simulators

• Test coverage needs to be adequate– Software is not identical in all Android/iOS devices– Hardware is even more fragmented landscape

• Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes• Test automation framework that works for you

webinar

Trade-offs – Manual vs. Automation• Costs & Assets– Time, People, Infrastructure, Tools, Training

• App Development & Testing Cycle – Together– Testing after Daily Builds (Continuous Integration)

• Tools & Technology – Frameworks– Myriad of frameworks available to help you

• Test Coverage & Reusability– Global coverage is CRITICAL to SUCCESS

• Outcome – Efficiency, Effectiveness and Results

webinar

Manualwebinar

Automation

Smaller coverage, More money burnt & time wasted, Error-prone

Large coverage, quickly

completed, Less money & time wasted, Exact

results.

What Framework Works You The Best?• High-level & declarative test automation

frameworks work well when app is under dev• Easy to integrate with continuous integration• Distance between use case and actual test• What do you build: Native, Hybrid or Web app?• The technical competence in your organization• Desired outcome from test automation

webinar

Different Ways of Doing Test Automation

webinar

Automatic test exercisers Record and PlaybackHand written test scripts

Benefits:

Accurate, specific to your testing needs, plenty of options with frameworks, tools

Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty

Fastest & extremely automated, excellent for smoke testing/quick testing, availability

Tradeoffs:

Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)

Compelling Recorder+Playback tools available for only few test automation frameworks

Not accurate as real test cases

webinar

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Test Automation During the Dev1

webinar

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Test Automation During the Dev2

Why Real Devices are Must-to-Have?• Emulators/simulators won’t help you to test…– User Experience and Usability– Hardware– Software– Infrastructure

webinar

0 % = the percentage of your app users that use emulator to run your app!

Test Automation Will Deliver You…• App Compatibility & Global coverage

• Reusability• Test efficiency & effectiveness• Faster time-to-market

webinar

Test Automation during the Mobile App Development Process

Smoke testing

Regression testing Regression testing Regression testing Regression testing

Functional & Interoperability testing

Performance & Stress testing Performance & Stress testing

Compatibility testing

webinar

Android Instrumentation Framework

webinar

• InstrumentationTestRunner – the primary plumbing for running tests on Android

• Android Instrumentation Framework is built on top of JUnit – a standard test framework on for any Java development

• Mock objects – methods for creating mock system objects such as content, service and intent

iOS Instruments (with Xcode)webinar

Android Example: The Family Tree of Test Automation Frameworks

webinar

Android Instrumentation Framework

Robotium Espresso

UI Automator

Calabash

Appium

Comparison of Test Automation Frameworks

webinar

Supports Calabash Robotium uiautomator Espresso Appium

Android Yes Yes Yes Yes Yes

iOS Yes No No No Yes

Mobile web Hybrid(webviews)

Yes (Android)

Limited to x.y clicks

No Yes (Android & iOS)

Scripting Language

Ruby Java Java Java Almost any

Test creation tools

CLI Testdroid Recorder

UI Automator viewer

Hierarchy Viewer

Appium.app

Supported API levels

All All 16 => 8, 10, 15-19 All

Community Active Contributors Google Google Active

webinar

Tip #1: Test early, Test oftenwebinar

Tip #2: Plan What to Automatewebinar

People

Infrastructure

Training

Tip #3: Use only Real Deviceswebinar

Tip #4: Use Atomic Test Unitswebinar

Test iteration #1 Test iteration #2

Tip #5: Create Hermetic Testswebinar

Tip #6: Use All Possible Deviceswebinar

Testdroid Cloud’s 300+ Android devices = 93-95% global Android volumes!

Tip #7: Separate Apps and Testswebinar

Your App

Tip #8: Output Everything to Logswebinar

Tip #9: End-to-End Testingwebinar

Tip #10: Integrate Dev & Testingwebinar

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

webinar

the basics of mobile app testing & test automation

Technology

w05 basics of setting up profibus process automation...

app sec - code insecurity basics

understanding the test automation culture of app developers

android app based home automation

memcache basics on google app engine

htce basics app press release - culinate...

android app development (basics)

advanced marketing automation techniques - beyond the basics

cv rao presents mobile app development basics

basics of plc automation

basics of android app development

substation automation basics - the next generation

app academy: distribute your app through automation (october...

developing a comprehensive test automation qa strategy ·...

mobile app analytics basics

smart home: the basics of home automation home: the basics...

marketing automation basics: 17 tips for getting...

information architecture (basics for app building)

web app automation talk

python map automation – beyond the basics of arcpy.mapping