build your own mad-lab - for mobile test automation

MAD-LABMobile Automation Devices LAB

Anand BagmarSoftware Quality Evangelist

About Me

@BagmarAnand

Blog – essenceoftesting

about.me/anand.bagmar

What do you expect from this session?

Context

EcosystemPlatforms

• Android (> v4.4.2)

• iPhones (> v5)

• Desktop / WAP

Regions• Indonesia

• Malaysia

• India

• Middle-East

• ...

Dynamic Content

Search

Download

Offers

ADs

What this means ... Reality!• Unique installs (Android) - > 1.75M

• Unique # of devices (Android) - > 13K

What this means ... Product!• One product, One experience

• Unique / Regional / Local content

• User detection & offers based on origin

• Region

• Carrier network

• ...

What this means ... Testing!• Video Quality, Dynamic Content, Download Expiry

• Simulating Carriers / Networks

• Consumer focus!• Experience

• Device / OS combinations

Business Requirements• Provide visibility into -

• What is Tested

• Overall Quality of Product

APIs

Payment partners

CMS

....

....

Architecture

APIs

Payment partners

CMS

....

....

Where are these?

How would you approach

Automation for such a product?

What Metrics make sense for

determining Quality?

Automation Strategy

Automation Criteria• E2E tests should focus on Business Rules & User Scenarios

• Reports should reflect the above

• Understood by all Business & Technical stakeholders

• Generated automatically

• Should run frequently

• Every new change in the product-under-test

• On a variety of device / OS combinations

What to Automate?Validate what was working, continues to work!

• Identify user journeys / scenarios

• Prioritize it!

Run Automation against...Usage patterns from Analytics / PlayStore / App Store

• Identify devices that matter

• Identify devices that are “automation-friendly”

Automation Tech Stack• Cucumber-JVM à For specifying Business Rules

• Appium à To allow integration with Android / iOS

• Gradle / Groovy à As build tool scripting language

• Jenkins à Continuous Integration Server

• TTA - Test Trend Analyzer à Trend & Failure Analysis

MVP / POC

Scale&

Repeat

Experiment, Learn & Evolve

Experiments• Run tests on Emulators

• Use Cloud-based services (Local Vs Remote)

• SauceLabs

• TestObject

• Amazon Device Farm

• pCloudy

• Devices

Some Experiments didn’t work• Emulators

• Video Player

• Location

• Cloud-based services• Location

• Tech Stack

• Devices

MAD-LAB

MAD-LAB – Mobile Automation Devices LAB

Innovations in MAD-LAB• Device management (selection, cleanup, app install and

uninstall)

• Parallel test execution (at Cucumber scenario level) -maximising device utilisation)

• Appium server management

• Adb utilities (periodic ADB server disconnects)

• Custom reporting using cucumber-reports

• Video recording of each scenario and embedding in the custom reports

• Analytics Automation

Conclusion

Learnings (so far)Blue-print is important!

• Stepping-stone approach

Experiment!

• Know your objective(s)

• Fail-fast!

Make the Product Testable!!

References• Sample code

• https://github.com/anandbagmar/cucumber-jvm-appium-infra

• Blog posts• https://essenceoftesting.blogspot.com/search/label/madlab

• TTA (Test Trend Analyzer)• https://essenceoftesting.blogspot.com/search/label/tta

Thank you!

@BagmarAnand

Blog – essenceoftesting

about.me/anand.bagmar