automate your build acceptance testing

26
Automate Your Build Acceptance Testing for Mobile Apps Rachel Obstler VP Mobile Testing Chris Karnacki Solutions Lead

Upload: keynote-mobile-testing

Post on 15-Apr-2017

392 views

Category:

Mobile


3 download

TRANSCRIPT

Page 1: Automate Your Build Acceptance Testing

Automate Your Build Acceptance Testing for Mobile Apps

Rachel ObstlerVP Mobile Testing

Chris KarnackiSolutions Lead

Page 2: Automate Your Build Acceptance Testing

Your Participation

Open and close your control panel

Join audio:• Choose Mic & Speakers to use VoIP• Choose Telephone and dial using the

information provided

Submit questions and comments via the Questions panel

Note: Today’s presentation is being recorded and will be posted to keynote.com fro on-demand viewing.

Your Participation

Page 3: Automate Your Build Acceptance Testing

Automate Your Build Acceptance Testing for Mobile Apps

Rachel ObstlerVP Mobile Testing

Chris KarnackiSolutions Lead

Page 4: Automate Your Build Acceptance Testing

Mobile and Agile Change the Test Process

Software Diversity

Hardware Diversity

Faster Release Cycles

Agile Processes

Today’s users expect apps to just work on their device

2-week sprints monthly releases

Automation

Page 5: Automate Your Build Acceptance Testing

A. >75%B. 50-75%C. 25-50%D. <25%E. Not sure

Poll: Of your organization’s mobile development projects, what % have build acceptance testing fully automated?

Page 6: Automate Your Build Acceptance Testing

Why Aren’t More Companies Automating Mobile Testing?

ComplexitiesDesktop Skillsets

Access to Devices

Time

Geographical Diversity

% Automation (mean across responses)

Unit tests 53%

Functional tests (GUI) 45%

Functional tests (Beyond GUI, e.g., APIs) 42%

Continuous testing — tests are kicked off at each code check in and integration (CI) 42%

We automate nonfunctional (performance) 38%

We automate nonfunctional (integration) 35%

We automate test environment provisioning 30%

Base: 155 professionals with knowledge of their firm’s Agile practices. Source: Forrester's Q2 2015 Global Agile Software Application Development Online Survey

“Using your best estimate, how much do you automate the following?”

CurrentRelease

FutureReleasePlanning

Page 7: Automate Your Build Acceptance Testing

Automating Build Acceptance Testing Potential Gains Over 2 weeks – Savings of 67

hours of person/testing time (equal to a 200 test case regression test)

Ongoing you could allocate 83% of a person to another project, or reduce your release time by up to 1 week

And… Reduced the time to give

development feedback on the release

Reduced rework for the development team

Small contained project Highest return on automated testing Huge benefit in a short period of time

# of Testers 0 1 2 3 4Elapsed Hours to Validate

Build (Manual) 6.7 3.3 2.2 1.7

Elapsed Hours to Validate Build (Automated) 1.0

Simple Example20 # of Build Acceptance Tests4 # of Target Platforms5 Average # of minutes to manually execute a test case3 Average # of minutes to automatically execute a test caseDaily Frequency of a BAT run

Page 8: Automate Your Build Acceptance Testing

Kick-Start Your Mobile Automation Program

Define the Build Acceptance

Program

Expand When Appropriate

Continuous Integration

Architecture

Execution – Script and Test

Page 9: Automate Your Build Acceptance Testing

Kick-Start Your Mobile Automation Program

Define the Build Acceptance

Program

Page 10: Automate Your Build Acceptance Testing

Defining the Program – Creating Optimized Device ListAdvantages

Ensures coverage across device OSs and form factors

Minimizes the set of devices necessary to test across

iOS Android

Why Real Devices vs Emulators? Different screen sizes/resolutions

cause blurring, sizing issues, items that fall off screen

On-screen, physical controls that function differently

Impacts of low memory or other services running on device

Page 11: Automate Your Build Acceptance Testing

Defining the Program – Build Acceptance Test DefinitionCreate your BAT list, making sure:1. All tests are automatable on real

devices2. All tests validate major functional

areas, such that failure means the build is rejected

3. Most, if not all, basic functionality is covered – use the 80/20 rule

4. The BAT will not run >1 hour

Goals of BAT Save test team effort to setup and

test a build when major functionality is broken

Give development team immediate feedback on the build so they can fix it immediately

Page 12: Automate Your Build Acceptance Testing

Kick-Start Your Mobile Automation Program

Define the Build Acceptance

Program

Continuous Integration

Architecture

Page 13: Automate Your Build Acceptance Testing

Continuous Integration Architecture – The Basics Architecture must enable true

continuous integration Determine what works best based

on current systems and skills: Which build system is utilized Where the app resides Notification plan for BAT completion Team programming or automation skills

And don’t forget to consider security!

BAT Must HavesFor BAT automation to work, the build must trigger both the app to be automatically loaded on the target devices, and the automated tests to run.

Page 14: Automate Your Build Acceptance Testing

CI Architecture and Data Flow DiagramDevelopment Test

Review CodePush Code

Bug Reports

BAT Test Results

Build AutomationVersion Control Real Mobile DevicesBAT Build on pushNightly

regression

Page 15: Automate Your Build Acceptance Testing

A. JenkinsB. Bamboo C. TeamCityD. OtherE. None

Poll: What build automation application do you use for continuous integration?

Page 16: Automate Your Build Acceptance Testing

Example Integration with Jenkins

Page 17: Automate Your Build Acceptance Testing

Kick-Start Your Mobile Automation Program

Define the Build Acceptance

Program

Continuous Integration

Architecture

Execution – Script and Test

Page 18: Automate Your Build Acceptance Testing

Execution – Implement Automated BATs & CI IntegrationThe implementation phase includes:

Writing scripts Testing scripts Integration with your build process End-to-end validation of app build to

report

Object-Level Scripting Easily re-useable Works across devices Remain resilient to ongoing app

changes Creates a framework for future

regression test automation

Page 19: Automate Your Build Acceptance Testing

Sample Report Quick view of

overall BAT success By test pass/fail Ability to dive into

failed test results quickly and share results with the agile team

Page 20: Automate Your Build Acceptance Testing

Kick-Start Your Mobile Automation Program

Define the Build Acceptance

Program

Expand When Appropriate

Continuous Integration

Architecture

Execution – Script and Test

Page 21: Automate Your Build Acceptance Testing

Ongoing Management and ExpansionOnce the project is implemented, you must:

Review the test plan Maintain the scripts Ensure multiple team members are trained Review of CI architecture and process with

the entire agile team Review reports

Then consider expansion, but continue to be prudent on defining manageable projects!

Success includes Training so team can easily maintain

and add to existing scripts Framework that enables easy

expansion of automation program Freeing tester time!

Page 22: Automate Your Build Acceptance Testing

Demo

Page 23: Automate Your Build Acceptance Testing

Keynote Mobile TestingMobile app testing platform for agile development teams Integrates into Your Build Process Comprehensive Cloud-Based

Library of Real Mobile Devices Manual or Fully Automated options Flexible Test Scripting Options Powerful Test Automation Engine Intelligent Reporting

Interactive and automated testing for any mobile app or website on hundreds of mobile phones and tablets

300+ shared devicesPrivate devices

Page 24: Automate Your Build Acceptance Testing

Keynote Mobile Testing Build Acceptance Testing Program

Program Stage Details

1. Define BAT plan and Goals Review or recommend target devices Review or help define list of Build Acceptance tests Create project schedule

2. Define CI Environment and Requirements

Document the different components of the build process, and how BAT automation fits in

Document the data flow and information sharing of results3. Implement Automated BAT Integrate with existing build process

Build and test automated scripts Test end-to-end process

4. Training and Handoff for Ongoing Management

Training / handoff to internal team to maintain, expand program

Page 25: Automate Your Build Acceptance Testing

Questions?

Page 26: Automate Your Build Acceptance Testing

Learn More

About Keynote Mobile Testing: www.keynote.com BAT Automation Program: www.keynote.com/now/bat More questions?

[email protected] +1 (650) 403-2451