mobile app testing with xamarin test cloud - sec.ch9.ms · pdf filemobile app testing with...

Mobile App Testing with Xamarin Test CloudMartin Šimeček

martin.simecek@microsoft.com

@deeedx

Mobile Quality Challenges

Device Fragmentation

App Complexity

Fast Release Cycles

Short Sessions

High Mobile User Expectations

Results! The Differences in Time to Value

19K unique

Android devices available on the market

20 iPhone and

iOS configuration combinations on the market

In the United States, a business must test on

134 different devices

80% of devices in use

to cover

The Manual Approach

Limited device reach, based on user availability

Slower and less accurate

Delayed feedback collection

Inconsistent

The Device Remoting Approach

Prone to human error

Limited test complexity

No simultaneous testing slows

time to identify issues

The Simulator Approach

Limited real world applicability

Virtual, not real devices

Limited accuracy

Xamarin Test Cloud Approach

AUTOMATED WITH MANAGED DEVICES

Consistent, scalable and accurate tests

Simultaneous testing across device platforms, operating systems and form factors

Real usage scenarios, including complex gestures

Secure with device wipe after each test run

Xamarin Test Cloud

Xamarin Test Cloud

2,000+ devicesAutomate your app testing and run them on over

2,000 different real devices. Test everything users do,

as well as any performance problems with step-by-

step memory and performance tracking.

Real devices, real quality• Ensure higher quality by testing on real devices

• Automate app testing on over 2,000 real devices

• Use C#, Ruby, or Cucumber to build automated

tests

• Integrates with any continuous integration system

Tap Scroll Swipe Pinch Multi finger

Text Entry Rotation GPS

Examples of User Interactions

Visually compare test results against

hundreds of devices at a time

• Zoom in and full screen resolution of all

screens

• Video playback coming in 2015

Review each step of a test

case and compare visual

results on all devices

Access precise device specifications,

full-size screenshots, and the native

device log

Understand CPU and memory usage

during every step

Object-level user

interface testing

Tests adapt as user

interface changes

Tests Adapt to UI Changes

March 2015

How does it work?

Xamarin Test Cloud

1. Create a Xamarin Test Cloud account

2. Prepare tests

3. Deploy to XTC

4. Integrate with your release pipeline

1. Create Xamarin Test Cloud Account

free trial

– 30 days

– max. 3 concurrent devices

– max. 3 testing hours per day

DEMO

Registering XTC account

2. Prepare tests

UI Tests in a separate project

using NUnit

can be hand-written

can be recorded using Test Recorder

iOS tests cannot be run on Windows

REPL

Read-Eval-Print-Loop

useful when defining tests

app.Repl()

– stops running test and opens a console window

tree

– lists the current view‘s structure

copy

– copies all commands to clipboard

Commands (Methods)

app.Flash()

– „blinks“ an element

– app.Flash(c => c.Class(„EditText“))

app.EnterText(c => c.Marked(„creditCardNumberText“), „12345“)

– app.ClearText(...)

app.Tap(c => c.Marked(„validateButton“))

app.WaitForElement(c => c.Marked(„label“).Text(„ABC“))

– 1 minute in test cloud

– 15 seconds locally

– can be changed with a parameter

DEMO

Working with Xamarin REPL

Test Recorder

available as a preview extension for Visual Studio

https://developer.xamarin.com/guides/testcloud/testrecorder

/visual-studio/

3. Deploy to Xamarin Test Cloud

via web portal

from Visual Studio / Xamarin

Studio

from command line

DEMO

Deploy to XTC from VS

4. Integrate with your release pipeline

Visual Studio Team Services

Xamarin Test Cloud as one of the build steps

DEMO

VSTS & Xamarin Test Cloud

Mobile App Testing with Xamarin Test CloudMartin Šimeček

martin.simecek@microsoft.com

@deeedx