the basics of mobile app testing & test automation
DESCRIPTION
Interested in a presentation video? Sign up at testdroid.com/webinars-archive The name of the game in today’s competitive mobile application landscape is a speed combined with robustness. How can you quickly deliver the best quality apps and how to ensure your app will stay on the top of download rankings at app markets? These questions will be answered as well as a comprehensive approach to improve your development process and app quality will be presented.TRANSCRIPT
webinar
The Basics of Mobile App Testing & Test Automation
26 March 2014
Ville-Veikko HelppiTechnical Product [email protected]
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
2
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
3
Challenges in App Developmentwebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
4
Fact: Most typical application fails when tested across different device variants used by consumers
• Failures: Both SW and HW cause failures• Emulators do not offer realistic platform for dev• Even app installation fails – not tested at all • Application crashes, leaks memory, consumes too
much resources, behaves not as meant to behave• UI scaling/display/resolution – common problem• Application performance is bad, draining battery
Why Apps Fail?webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
5
Testdroid Research Available Soon…webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
6
What Improves the App Quality?• Automate generic things as much as you can• During the development your app changes –
the same must apply for testing!• Carefully select (testing) technology & partner• Use all 24 hours per day with test automation• Cloud-based platform is the only solution to
get you quickly covered in the global markets
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
7
Mobile Apps for Different Verticalswebinar
Mobile
Games
Utilities & Tools
Video Streaming & Multimedia
Banking & Payment
Retail & Travel
• Testdroid has helped thousands of app developers in these subverticals!
• What are the critical elements in each these verticals?
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
8
Mobile Apps for Different Verticalswebinar
Mobile
Games
Utilities & Tools
Video Streaming & Multimedia
Banking & Payment
Retail & Travel
• User Experience!• Resource consumption &
validation (CPU, Mem…)• Fully utilizing hardware –
e.g. Touch screen• Mobile games require
access to graphics APIs (e.g. OpenGL ES)
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
9
Mobile Apps for Different Verticalswebinar
Mobile
Games
Utilities & Tools
Video Streaming & Multimedia
Banking & Payment
Retail & Travel
• Usability!• Functional and
Behavioral testing • Metrics analysis for all
captured data (e.g. logs, screenshots, perf stats)
• Connection with Social Media + other apps
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
10
Mobile Apps for Different Verticalswebinar
Mobile
Games
Utilities & Tools
Video Streaming & Multimedia
Banking & Payment
Retail & Travel
• Performance!• Connectivity, robustness
and durability• Screen orientation
(portrait vs. landscape)• Graphics quality,
streaming capabilities• User profiles
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
11
Mobile Apps for Different Verticalswebinar
Mobile
Games
Utilities & Tools
Video Streaming & Multimedia
Banking & Payment
Retail & Travel
• Security!• Secure connectivity with
back-end systems• Top Quality – extremely
brand sensitive vertical• Compliances and
verification between real devices and infrastructure
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
12
Mobile Apps for Different Verticalswebinar
Mobile
Games
Utilities & Tools
Video Streaming & Multimedia
Banking & Payment
Retail & Travel
• Data!• Connectivity and data
connection with back-end systems
• Configurability of the app• Bad quality can hurt the
brand and leave you without customers
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
13
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
14
Prerequisites for Mobile App Testing• Testing environment as authentic as possible– Real devices <-> emulators/simulators
• Test coverage needs to be adequate– Software is not identical in all Android/iOS devices– Hardware is even more fragmented landscape
• Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes• Test automation framework that works for you
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
15
Trade-offs – Manual vs. Automation• Costs & Assets– Time, People, Infrastructure, Tools, Training
• App Development & Testing Cycle – Together– Testing after Daily Builds (Continuous Integration)
• Tools & Technology – Frameworks– Myriad of frameworks available to help you
• Test Coverage & Reusability– Global coverage is CRITICAL to SUCCESS
• Outcome – Efficiency, Effectiveness and Results
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
16
Manualwebinar
Automation
Smaller coverage, More money burnt & time wasted, Error-prone
Large coverage, quickly
completed, Less money & time wasted, Exact
results.
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
17
What Framework Works You The Best?• High-level & declarative test automation
frameworks work well when app is under dev• Easy to integrate with continuous integration• Distance between use case and actual test• What do you build: Native, Hybrid or Web app?• The technical competence in your organization• Desired outcome from test automation
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
18
Different Ways of Doing Test Automation
webinar
Automatic test exercisers Record and PlaybackHand written test scripts
Benefits:
Accurate, specific to your testing needs, plenty of options with frameworks, tools
Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty
Fastest & extremely automated, excellent for smoke testing/quick testing, availability
Tradeoffs:
Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)
Compelling Recorder+Playback tools available for only few test automation frameworks
Not accurate as real test cases
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
19
webinar
Developers
Source code
Repository
Build failed
Polling
Notifying
Successful build
QA & Testers
Building
Error reporting
Test Cases
Application
Users / Consumers
Rele
ase
Com
mit
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
20
Test Automation During the Dev1
webinar
Developers
Source code
Repository
Build failed
Polling
Notifying
Successful build
QA & Testers
Building
Error reporting
Test Cases
Application
Users / Consumers
Rele
ase
Com
mit
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
21
Test Automation During the Dev2
Why Real Devices are Must-to-Have?• Emulators/simulators won’t help you to test…– User Experience and Usability– Hardware– Software– Infrastructure
webinar
0 % = the percentage of your app users that use emulator to run your app!
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
22
Test Automation Will Deliver You…• App Compatibility & Global coverage
• Reusability• Test efficiency & effectiveness• Faster time-to-market
webinar
Test Automation during the Mobile App Development Process
Test
requ
irem
ents
/ P
lann
ing
Smoke testing
Regression testing Regression testing Regression testing Regression testing
Functional & Interoperability testing
Functional & Interoperability testing
Functional & Interoperability testing
Performance & Stress testing Performance & Stress testing
Compatibility testing
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
23
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
24
Android Instrumentation Framework
webinar
• InstrumentationTestRunner – the primary plumbing for running tests on Android
• Android Instrumentation Framework is built on top of JUnit – a standard test framework on for any Java development
• Mock objects – methods for creating mock system objects such as content, service and intent
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
25
iOS Instruments (with Xcode)webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
26
Android Example: The Family Tree of Test Automation Frameworks
webinar
JUnit
Android Instrumentation Framework
Robotium Espresso
UI Automator
Calabash
Appium
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
27
Comparison of Test Automation Frameworks
webinar
Supports Calabash Robotium uiautomator Espresso Appium
Android Yes Yes Yes Yes Yes
iOS Yes No No No Yes
Mobile web Hybrid(webviews)
Yes (Android)
Limited to x.y clicks
No Yes (Android & iOS)
Scripting Language
Ruby Java Java Java Almost any
Test creation tools
CLI Testdroid Recorder
UI Automator viewer
Hierarchy Viewer
Appium.app
Supported API levels
All All 16 => 8, 10, 15-19 All
Community Active Contributors Google Google Active
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
28
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
29
Tip #1: Test early, Test oftenwebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
30
Tip #2: Plan What to Automatewebinar
People
Infrastructure
Tools
Training
Time
MAN
UAL
AUTO
MATED
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
31
Tip #3: Use only Real Deviceswebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
32
Tip #4: Use Atomic Test Unitswebinar
Test iteration #1 Test iteration #2
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
33
Tip #5: Create Hermetic Testswebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
34
Tip #6: Use All Possible Deviceswebinar
Testdroid Cloud’s 300+ Android devices = 93-95% global Android volumes!
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
35
Tip #7: Separate Apps and Testswebinar
Your App
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
36
Tip #8: Output Everything to Logswebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
37
Tip #9: End-to-End Testingwebinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
38
Tip #10: Integrate Dev & Testingwebinar
Com
mit
Developers
Source code
Repository
Build failed
Polling
Notifying
Successful build
QA & Testers
Building
Error reporting
Test Cases
Application
Rele
ase
Users / Consumers
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
39
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
40
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
41
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
42
Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
43
webinar
© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.
44