the basics of mobile app testing & test automation

44
webinar The Basics of Mobile App Testing & Test Automation 26 March 2014 Ville-Veikko Helppi Technical Product Manager [email protected]

Upload: bitbar

Post on 08-May-2015

1.811 views

Category:

Technology


2 download

DESCRIPTION

Interested in a presentation video? Sign up at testdroid.com/webinars-archive The name of the game in today’s competitive mobile application landscape is a speed combined with robustness. How can you quickly deliver the best quality apps and how to ensure your app will stay on the top of download rankings at app markets? These questions will be answered as well as a comprehensive approach to improve your development process and app quality will be presented.

TRANSCRIPT

Page 1: The Basics of Mobile App Testing & Test Automation

webinar

The Basics of Mobile App Testing & Test Automation

26 March 2014

Ville-Veikko HelppiTechnical Product [email protected]

Page 2: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

2

Page 3: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

3

Page 4: The Basics of Mobile App Testing & Test Automation

Challenges in App Developmentwebinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

4

Fact: Most typical application fails when tested across different device variants used by consumers

• Failures: Both SW and HW cause failures• Emulators do not offer realistic platform for dev• Even app installation fails – not tested at all • Application crashes, leaks memory, consumes too

much resources, behaves not as meant to behave• UI scaling/display/resolution – common problem• Application performance is bad, draining battery

Page 5: The Basics of Mobile App Testing & Test Automation

Why Apps Fail?webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

5

Page 6: The Basics of Mobile App Testing & Test Automation

Testdroid Research Available Soon…webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

6

Page 7: The Basics of Mobile App Testing & Test Automation

What Improves the App Quality?• Automate generic things as much as you can• During the development your app changes –

the same must apply for testing!• Carefully select (testing) technology & partner• Use all 24 hours per day with test automation• Cloud-based platform is the only solution to

get you quickly covered in the global markets

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

7

Page 8: The Basics of Mobile App Testing & Test Automation

Mobile Apps for Different Verticalswebinar

Mobile

Games

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• Testdroid has helped thousands of app developers in these subverticals!

• What are the critical elements in each these verticals?

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

8

Page 9: The Basics of Mobile App Testing & Test Automation

Mobile Apps for Different Verticalswebinar

Mobile

Games

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• User Experience!• Resource consumption &

validation (CPU, Mem…)• Fully utilizing hardware –

e.g. Touch screen• Mobile games require

access to graphics APIs (e.g. OpenGL ES)

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

9

Page 10: The Basics of Mobile App Testing & Test Automation

Mobile Apps for Different Verticalswebinar

Mobile

Games

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• Usability!• Functional and

Behavioral testing • Metrics analysis for all

captured data (e.g. logs, screenshots, perf stats)

• Connection with Social Media + other apps

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

10

Page 11: The Basics of Mobile App Testing & Test Automation

Mobile Apps for Different Verticalswebinar

Mobile

Games

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• Performance!• Connectivity, robustness

and durability• Screen orientation

(portrait vs. landscape)• Graphics quality,

streaming capabilities• User profiles

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

11

Page 12: The Basics of Mobile App Testing & Test Automation

Mobile Apps for Different Verticalswebinar

Mobile

Games

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• Security!• Secure connectivity with

back-end systems• Top Quality – extremely

brand sensitive vertical• Compliances and

verification between real devices and infrastructure

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

12

Page 13: The Basics of Mobile App Testing & Test Automation

Mobile Apps for Different Verticalswebinar

Mobile

Games

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• Data!• Connectivity and data

connection with back-end systems

• Configurability of the app• Bad quality can hurt the

brand and leave you without customers

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

13

Page 14: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

14

Page 15: The Basics of Mobile App Testing & Test Automation

Prerequisites for Mobile App Testing• Testing environment as authentic as possible– Real devices <-> emulators/simulators

• Test coverage needs to be adequate– Software is not identical in all Android/iOS devices– Hardware is even more fragmented landscape

• Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes• Test automation framework that works for you

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

15

Page 16: The Basics of Mobile App Testing & Test Automation

Trade-offs – Manual vs. Automation• Costs & Assets– Time, People, Infrastructure, Tools, Training

• App Development & Testing Cycle – Together– Testing after Daily Builds (Continuous Integration)

• Tools & Technology – Frameworks– Myriad of frameworks available to help you

• Test Coverage & Reusability– Global coverage is CRITICAL to SUCCESS

• Outcome – Efficiency, Effectiveness and Results

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

16

Page 17: The Basics of Mobile App Testing & Test Automation

Manualwebinar

Automation

Smaller coverage, More money burnt & time wasted, Error-prone

Large coverage, quickly

completed, Less money & time wasted, Exact

results.

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

17

Page 18: The Basics of Mobile App Testing & Test Automation

What Framework Works You The Best?• High-level & declarative test automation

frameworks work well when app is under dev• Easy to integrate with continuous integration• Distance between use case and actual test• What do you build: Native, Hybrid or Web app?• The technical competence in your organization• Desired outcome from test automation

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

18

Page 19: The Basics of Mobile App Testing & Test Automation

Different Ways of Doing Test Automation

webinar

Automatic test exercisers Record and PlaybackHand written test scripts

Benefits:

Accurate, specific to your testing needs, plenty of options with frameworks, tools

Fast to create, accurate, not as sensitive to human-errors as hand-written tests, tools avail’ty

Fastest & extremely automated, excellent for smoke testing/quick testing, availability

Tradeoffs:

Takes a lot of time, ties resources to write test cases/scripts, error-prone (humans)

Compelling Recorder+Playback tools available for only few test automation frameworks

Not accurate as real test cases

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

19

Page 20: The Basics of Mobile App Testing & Test Automation

webinar

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Rele

ase

Com

mit

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

20

Test Automation During the Dev1

Page 21: The Basics of Mobile App Testing & Test Automation

webinar

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Rele

ase

Com

mit

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

21

Test Automation During the Dev2

Page 22: The Basics of Mobile App Testing & Test Automation

Why Real Devices are Must-to-Have?• Emulators/simulators won’t help you to test…– User Experience and Usability– Hardware– Software– Infrastructure

webinar

0 % = the percentage of your app users that use emulator to run your app!

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

22

Page 23: The Basics of Mobile App Testing & Test Automation

Test Automation Will Deliver You…• App Compatibility & Global coverage

• Reusability• Test efficiency & effectiveness• Faster time-to-market

webinar

Test Automation during the Mobile App Development Process

Test

requ

irem

ents

/ P

lann

ing

Smoke testing

Regression testing Regression testing Regression testing Regression testing

Functional & Interoperability testing

Functional & Interoperability testing

Functional & Interoperability testing

Performance & Stress testing Performance & Stress testing

Compatibility testing

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

23

Page 24: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

24

Page 25: The Basics of Mobile App Testing & Test Automation

Android Instrumentation Framework

webinar

• InstrumentationTestRunner – the primary plumbing for running tests on Android

• Android Instrumentation Framework is built on top of JUnit – a standard test framework on for any Java development

• Mock objects – methods for creating mock system objects such as content, service and intent

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

25

Page 26: The Basics of Mobile App Testing & Test Automation

iOS Instruments (with Xcode)webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

26

Page 27: The Basics of Mobile App Testing & Test Automation

Android Example: The Family Tree of Test Automation Frameworks

webinar

JUnit

Android Instrumentation Framework

Robotium Espresso

UI Automator

Calabash

Appium

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

27

Page 28: The Basics of Mobile App Testing & Test Automation

Comparison of Test Automation Frameworks

webinar

Supports Calabash Robotium uiautomator Espresso Appium

Android Yes Yes Yes Yes Yes

iOS Yes No No No Yes

Mobile web Hybrid(webviews)

Yes (Android)

Limited to x.y clicks

No Yes (Android & iOS)

Scripting Language

Ruby Java Java Java Almost any

Test creation tools

CLI Testdroid Recorder

UI Automator viewer

Hierarchy Viewer

Appium.app

Supported API levels

All All 16 => 8, 10, 15-19 All

Community Active Contributors Google Google Active

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

28

Page 29: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

29

Page 30: The Basics of Mobile App Testing & Test Automation

Tip #1: Test early, Test oftenwebinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

30

Page 31: The Basics of Mobile App Testing & Test Automation

Tip #2: Plan What to Automatewebinar

People

Infrastructure

Tools

Training

Time

MAN

UAL

AUTO

MATED

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

31

Page 32: The Basics of Mobile App Testing & Test Automation

Tip #3: Use only Real Deviceswebinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

32

Page 33: The Basics of Mobile App Testing & Test Automation

Tip #4: Use Atomic Test Unitswebinar

Test iteration #1 Test iteration #2

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

33

Page 34: The Basics of Mobile App Testing & Test Automation

Tip #5: Create Hermetic Testswebinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

34

Page 35: The Basics of Mobile App Testing & Test Automation

Tip #6: Use All Possible Deviceswebinar

Testdroid Cloud’s 300+ Android devices = 93-95% global Android volumes!

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

35

Page 36: The Basics of Mobile App Testing & Test Automation

Tip #7: Separate Apps and Testswebinar

Your App

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

36

Page 37: The Basics of Mobile App Testing & Test Automation

Tip #8: Output Everything to Logswebinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

37

Page 38: The Basics of Mobile App Testing & Test Automation

Tip #9: End-to-End Testingwebinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

38

Page 39: The Basics of Mobile App Testing & Test Automation

Tip #10: Integrate Dev & Testingwebinar

Com

mit

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Rele

ase

Users / Consumers

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

39

Page 40: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

40

Page 41: The Basics of Mobile App Testing & Test Automation

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

41

Page 42: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

42

Page 43: The Basics of Mobile App Testing & Test Automation

Agenda• App Quality and Mobile App Testing• How To Get Started with Test Automation?• Test Automation Frameworks• 10 Tips for Successful App Testing• Testdroid Update• Demonstration• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

43

Page 44: The Basics of Mobile App Testing & Test Automation

webinar

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

44