Infosys & Experitest Co-webinar:
Total Quality in Mobile Apps -Continuous
Integration & Performance Testing
Nov 2014
Part I
How To Use Continuous Integration for Mobile
Speaker: Guy Arieli , Experitest CTO
Agenda
Why is it important?
How is it different from regular Continuous Integration?
What is the recommended flow for Continuous Integration in mobile?
Case study: major bank
Demo: mobile continuous integration Jenkins
Continuous Integration is especially important for mobile
1
2
3
4
5
What Is Continuous Integration?
Software
development practice
where members of a
team integrate their
work frequently
Significantly reduces
integration problems
and allows a team to
develop cohesive
software rapidly
Each person
integrates at least
daily – leading to
multiple integrations
per day
CI significantly reduces integration problems and allows a team to develop cohesive software rapidly
YOURAPP
Developers Source Code
Repository
CI Server
Why is CI especially important for mobile apps?
Mobile app development is faster
End-user expectation for fast response time
Frequent external environment changes
New OS versions
New devices
High adoption of Agile methodology in R&D
Fast app release
Many (hundreds) devices to be tested
Several mobile OS to be tested
Huge device matrix
Higher ROI because of mega matrix
How is mobile CI different?
Parallel execution
Complex management
of Continuous
Integration
Smart device allocation
Dual application mode
(instrumented, non-instrumented)
Continuous Integration For Mobile
Developers
Source Code
Repository
CI Server
YOURAPPExecuter Agent
Execute Test On Devices
x
Test Results
Recommended process for mobile Continuous Integration
Build the application
Prepare the application for automation
Reserve the devices to be used
Deploy application and device preparation
Tests Execution
View and analyze reports
Build the Application
For example: If it’s an Android
build, you will get a URL for
downloading this APK file
Build the Application
Source Code
Repository
CI Server
YOURAPP
New Build of Your App
Trigger the entire
execution
Recommended process for mobile Continuous Integration
Build the application
Prepare the application for automation
Reserve the devices to be used
Deploy application and device preparation
Tests Execution
View and analyze reports
Prepare the Application
YOURAPP
New Build of Your App
YOURAPP
YOURAPP
Instrumentedmore control and more
capabilities over the
application
Non- instrumented
(optional step)
Application under test
may have two modes:
Tips for
Implementation
Unattended, on-the-fly
instrumentation
Test automation tool should enable:
1
2 Perform the instrumentation
on the same machine that is
being used for automation
3 Windows machine to perform all
instrumentation (inc. on-the-fly)
(also in the case of an iOS app)
Dynamic, smart reservation of the Devices to be used
@os=‘android’ and @versionnumber > 4.2
and
@manufacture=‘htc’
Dynamic,
smart
reservation
Execute your regression of tests on as many devices as you can
Cloud of Devices
Smart Device Reservation
Recommended process for mobile Continuous Integration
Build the application
Prepare the application for automation
Reserve the devices to be used
Deploy application and device preparation
Tests Execution
View and analyze reports
Deploy Application and Device Preparation
**These operations should be available and should be able to run entirely unattended.
Device reboot
Application clear data
Uninstallation of applications
Installation of applications
Launch of applications
Have full control of
the device and of
the application
installation
Working with the Settings
Recommended process for mobile Continuous Integration
Build the application
Prepare the application for automation
Reserve the devices to be used
Deploy application and device preparation
Tests Execution
View and analyze reports
Tests Execution
CI Server
Executer Agent
Execute the Devices
Remote Mobile
device Lab
T1
T2
T2
T2
…
Tn
Actions for making
Test execution
efficient:
Capability to run the tests in parallel on different devices
Decupling between the CI server and the execution machine
Queue of tests that are waiting for devices the be available
Recommended process for mobile Continuous Integration
Build the application
Prepare the application for automation
Reserve the devices to be used
Deploy application and device preparation
Tests Execution
View and analyze reports
Recommended process for mobile Continuous Integration
Demo How to use Jenkins for mobile
Continuous integration
Case Study
Headquartered in APAC
Spans over 50 countries across Europe, America, Africa & Asia
~200,000 employees
Mobile applications in tens of languages
Ranked in top 10 largest banks worldwide
Revenue more than $30 Bn.
Customer Profile:
Multinational banking and financial services company
Major global financial services provider engaged in personal
banking, credit cards, corporate and investment banking and
wealth and investment management.
The bank’s Personal and Business
accounts management apps
Wallet apps
Trading apps
Mortgages apps
BankCard
Many more…
The Need
Testing end users' apps Testing the Internal bank
Employees’ Apps
Loan qualification form
Mortgage requirements form
Etc.
Applications used by The bank
employees providing services to
customers such as:
The Requirements
On-premise devices, all within the bank VPN
Cross-platform testing on all OS: Android, iOS,
WindowsPhone8 & BlackBerry
No need for access to application source code for testing
(on-the-fly instrumentation)
Support for multi-field form filling including scrolling in
lists, invisible objects, etc.
Ability to test on devices located in one of the bank facility
by testers located in other locations
</>
Implementation
Years working with the tool: 3
Over 100 SeeTestAutomation stations
Number of Automation engineers working on the project: 200
4 geographical locations around the world (US, Europe, Africa and China)
Number of tests before: 0
Number of tests today: ~20,000
Number of executed tests: ~200,000 (each test can run on more than one device)
Onsite deployment with integration into the existing test environments
QTP, Junit, Eclipse
QC and Jenkins
Parallel execution of tests
Coverage
Total coverage: ~90%.
Number of executions of the full cycle: Once a day
Number of apps tested: +20, (different application in different regions)
Coverage of both customer ’s apps and internal systems apps
Full app test coverage for all OS-Cross platform testing of
Android
iOS
WindowsPhone8
BlackBerry
Full end-to-end coverage of test scenario
Support for multiple languages (including English, Spanish, French)
What if we did it manually?
A single QA engineer: 56 man years to finish the cycle (lifetime).
For one day we would need 20,000 trained manual QA engineers.
Another Advantage
The R&D engineers can work faster and take more risks
According to the bank R&D managers the development speed is
ROI
3-4 times faster!
The Tools Used
• Automation tool
• Same test runs on different devices
• Plugs into all testing & CI environments:
• UFT/Selenium/ Microsoft Visual Studio/
RFT/Java/C#/TestComplete//Perl/Python/Ruby
• QC/TFS/Jenkins/TeamCity/RQM
• Manual testing tool
• Mini-recorder for automating short,
repetitive, manual scenarios
• Report with screenshots of each step
• Device Lab located onsite (in the customer VPN)
• Devices can be reserved and accessed by any remote user
• Used in conjunction with SeeTestAutomation/Manual for testing of remote devices
SeeTestAutomation SeeTestManual
Free
Tool!
SeeTestCloud
Manual to Automation Script Convertor
Thank you
Infosys Point of View (POV) on
Mobile Application Performance Testing
17 December, 2014
Performance is key, for a Mobile user
Poor App Store ratings
… are often linked with poor app
performance
• Frequent crashes and
glitches
• Slow loading times
• High battery drain
• Unreliability
• Inefficient start and exits …
Forrester: “Mobile apps live
and die by their ratings in an
App Store… When the rating
suffers, customer adoption
suffers.”
Analysis reveal that
majority of 1-star
reviews are related
to performance
issues, proving that
Quality is key for an
end-user.
Apigee: “44% of users delete
a mobile app immediately if it
didn’t perform as expected;
Performance is reported to be
especially important for
banking (74%) apps
IMS, Goldsmiths: "Users
experience a lot of negative
emotions and frustrations
when trying to complete some
digital tasks and apps are slow
to load and respond"
realBusiness: “Consumers are
unforgiving towards slow and
poor – performing apps.”
52%
0
76%
48%
0
15%
0
9%
CRITICAL, 1-STAR REVIEWS
PERFORMANCE RELATED, CRITICAL, 1-STAR
REVIEWS
% of Critical, 1-Star Reviews For Top iOS Apps
Performance Related
Other
Crash, Bug
Lagging, Performance, Hot, Battery DrainSlow, Long Load, Load Fails
Source: appurify
If left unaddressed, the impacts are significant
Lost transactions not only result in lost revenue for banks, but have a detrimental impact on customer loyalty and brand reputation – it's a lose-lose situation.
38 percent try another app when faced with a problem, 34 percent stop using the app, and 19 percent complain to friends and family
Majority users delete or uninstall mobile app because of performance problems. About 18% delete the app immediately if it froze for just 5 sec. 38% delete the app if it freezes for longer than 30 seconds
30% of users would spend more money with an organization that had a good mobile app
29 percent would pay more for a product or service if the organization’s app performed better than its competitors
Nearly one third of smartphone and tablet owners would change banks if a mobile app wasn’t up to par
Addressing Customer Reviews on Mobile apps..33
• One in six customers who switch banks indicate a poor
mobile banking experience prompted the switch
• Between 51% and 76% of mobile users use their mobile
apps to check balances, transfer money, pay bills, and
look up branch locations and hours. Hence reduced
performance leads to greater & immediate customer dissatisfaction
Problem Cause Solution
Application is buggy (doesn’t
open, crashes, freezes often).
Sloppy code, varying internet
connection, poor memory
management.
Thorough testing of the app
under multiple conditions and
variables to ensure that an
app is devoid of issues
Infosys’ Performance Testing Solution
Client Application Performance Server Side Performance Network Impact on Performance
Device Impact Testing / Client Application Performance34
Key Measurement Parameters:
• Battery Consumption
• Device CPU and Memory constraints
• Device I/O – Assess speed for read and
write to devices
Tools:
- Xcode Instruments (memory
allocation, leaks, CPU usage …)
- Android ADB (battery consumption,
CPU usage / processing speed)
- Android DDMS ( memory usage,
device I/O, track running thread info ..)
- Quadrant (only for Android)
- Neoload
Server Side Performance / Load Testing35
Key Measurement Parameters:
• Response Time
• Error Rate
• Throughput
• Server CPU utilization
• Server Memory utilization
Tools:
- LoadRunner
- Neoload
- CloudTest
Mobile Performance Tool Recommendation:
• Ability to support multiple devices
• Ability to support native applications as well as browser-based applications
• Ability to record through the device using agents, or through emulators, as
well as directly from a browser to reduce complexity
• Ease of use and fast turnaround time
Network Impact Testing36
Network Emulation Options:
Network Coverage - Occasional disconnection of the network.
Packet Loss & Duplication – Packet loss & duplication for an application
session by dropping packets based on preset values
Delay and Jitter - Delay of data originating from app .
Multi-Network testing (Roaming) –Switching from one type of network to
another type of network e.g. (3G, 4G, LTE)
Key Measurement Parameters:• Response Time
• Exception Handling
• UI Issues
• Consistency
Tools: - Infosys Network Impact Testing Solution (iNITS)
- Shunra
- Neoload
Mobile Performance Tool Recommendation:• Ability to dynamically emulate various network conditions
• Ability to generate and or emulate load from various locations, leveraging the
cloud
• Ability to test regardless of the protocol the application uses
Some of the Performance Testing Best Practices for Mobile Apps
Test early: Do quick check on mobile web application performance, using page analysis tools like
‘PageSpeed Insights’, Yslow etc. For native apps, developers can perform regular checks with
Instruments / DDMS tools
Verify & validate the performance on a real device
Perform both modular and system performance tests
Correlate server-side metrics with client-side response times. It helps nailing down the root cause
May look at reusing the functional scripts for load Testing (e.g. SeeTestLoad -LoadRunner)
Look for consistency in test results, during multiple test iterations
Ensure complete performance test coverage, executing Load, Endurance, Stress and Single-user
test cycles
37
© 2013 Infosys Limited, Bangalore, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to changewithout notice. Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Exceptas expressly permitted, neither this documentation nor any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing,photocopying,recording or otherwise, without the prior permission of Infosys Limited and/ or any named intellectual property rights holders under this document.
Thank You