techniques, tips & tools for mobile app testing
Post on 03-Aug-2015
54 Views
Preview:
TRANSCRIPT
© 2015 SOASTA. All rights reserved. April 15, 2023 1
Brad JohnsonVP Product Marketing
@bradjohnsonsv
Session W12
Techniques, Tips and Technology for
Better Mobile App Testing
Vancouver, BCJune 24, 2015
48%TIME
37%EXPERTISE
35%PROCESS
?Says who
you.
250 Testers from 6 Continents
Mobile is your chance to make a CHANGE
Learn new TECHNIQUES
April 15, 2023 10© 2015 SOASTA. All rights reserved.
Seek TIPS from experts
Embrace new TECHNOLOGY
Learn new TECHNIQUES
REALITY always trumps speculation
Static aggregates (ok)
REALITY always trumps speculation
Real Time (best)
REALITY always trumps speculation
Data science - better than best!
April 15, 2023 16© 2015 SOASTA. All rights reserved.
ASAP
Simple
Time Consuming
Mundane
Rhythm
Goal : Continuous Integration execution
Know how to AutomateReturn person days to your test cycle
April 15, 2023 17© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of DeliveryWhat’s not to like?
April 15, 2023 18© 2015 SOASTA. All rights reserved.
o Small Batch Sizes
What’s not to like?
Continuous Integration Offers Speed of Delivery
April 15, 2023 19© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
What’s not to like?
April 15, 2023 20© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
What’s not to like?
April 15, 2023 21© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
What’s not to like?
April 15, 2023 22© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
What’s not to like?
April 15, 2023 23© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
What’s not to like?
April 15, 2023 24© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
What’s not to like?
April 15, 2023 25© 2015 SOASTA. All rights reserved.
Continuous Integration Offers Speed of Delivery
o Small Batch Sizes
o Comprehensive Version Control
o Simple Branch Strategies
o Automation
o Continuous Feedback
o Emphasis on working builds
o Consistent environments
o Developer – Tester Collaboration
What’s not to like?
April 15, 2023 26© 2015 SOASTA. All rights reserved.
Device reaction to app functionality Battery, CPU, Memory
Dependent services and 3rd party apps/ads
Separate back and front end performance
User experience under every traffic condition
Performance to revenue correlation
Performance Is Everything!Every strategy needs perf coverage
April 15, 2023 27© 2015 SOASTA. All rights reserved.
Seek TIPS from experts
April 15, 2023 28© 2015 SOASTA. All rights reserved.
7 Steps to Pragmatic Mobile Testing
April 15, 2023 29© 2015 SOASTA. All rights reserved.
RetestTest Development Infra setup RepeatFixTest Analyze
TIME
Beware Automation Solutions that Hinder Speed
Developers code tests, but not faster.
The same delays with different people?
April 15, 2023 30© 2015 SOASTA. All rights reserved.
RetestTest Development Infra setup RepeatFixTest Analyze
TIME
Analysis is still slow & manual
The same delays with different people?
Beware Automation Solutions that Hinder Speed
April 15, 2023 31© 2015 SOASTA. All rights reserved.
Get Testers TESTING!
RetestTest DevelopmentInfra setup RepeatAnalyze
Develop plans Post-test ValidateTraditional Approach
Test Fix
/
Analyze
Fix
Test Dev Test/
Analyze
Fix
Test Dev Test/ Analyze
Fix
Test Dev Test/Analyze
Fix
Test Dev TestModern Approach
OR/
Analyze
Fix
Test Dev Test /
Analyze
Fix
Test Dev Test
AUT/device setup Manual
April 15, 2023 32© 2015 SOASTA. All rights reserved.
250 hours of manual tests compressed to 1 hour
Regression tests not run until feature complete now run nightly 4 months to overnight
25% faster Time To Market
April 15, 2023 33© 2015 SOASTA. All rights reserved.
“(Mobile test automation) has been critical in improving both the speed to deliver and the quality of our mobile apps.”
“We decreased our test cycles from three weeks to a couple of days.”
3 WEEKS 2 Days
Embrace new TECHNOLOGY
RUM = “What to
Test”
(real user monitoring)
April 15, 2023 36© 2015 SOASTA. All rights reserved.
Mobile Test Automation Criteria
o Rapid & accurate test development
o Reliable, maintainable test automation
o Stability across mobile OS platforms
o Device performance measurement
o Real-Time feedback (via Continuous Integration is a +)
Keeping up with mobile development
April 15, 2023 38© 2015 SOASTA. All rights reserved.
Rank active devices
Device usage
Weight matrix to device
Know problem areas per device
Device Access Private Public
Know your devicesBuilding a Matrix
April 15, 2023 39© 2015 SOASTA. All rights reserved.
April 15, 2023 40© 2015 SOASTA. All rights reserved.
Source: Mobile Test Coverage Index (Perfecto)
April 15, 2023 41© 2015 SOASTA. All rights reserved.
Private Device Clouds
o Behind the firewall
o Always available
o Online access for all your teams
o 24x7, secure, remote access to devices
o Manual & automated testing Hands on when you need them.
April 15, 2023 42© 2015 SOASTA. All rights reserved.
Public Device Cloudso Dedicated
o Latest in-market devices
o Maintained for you
o Secure
o API’s to install apps, reboot/wake-up, launch tests
Perfect for automated mobile regression testing
April 15, 2023 43© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CDNCredit Card
Web analytic
s
Other Svcs…
Test the Back End
Performance Tester(s)
Virtualized services
3rd party services
Load Testing Tool
April 15, 2023 44© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
Analytics
Load GeneratorLoad
GeneratorLoad Generator
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CDNCredit Card
Web analytic
s
Other Svcs…
Performance Tester(s)
Virtualized services
3rd party services
Cloud Servers
Test the Back End
Load Testing Tool
April 15, 2023 45© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
Analytics
Load GeneratorLoad
GeneratorLoad Generator
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CDNCredit Card
Web analytic
s
Other Svcs…
Performance Tester(s)
Virtualized services
3rd party services
Cloud Servers
Test the Back End
Load Testing Tool
April 15, 2023 46© 2015 SOASTA. All rights reserved.
Analytics
Database
Main
SUT/AUT
Web Server
Web Server
App ServerWeb Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Analytics
Load GeneratorLoad
GeneratorLoad Generator
CDNCredit Card
Web analytic
s
Other Svcs…
Performance Tester(s)
Data sources: Native (SSH), JMX, PerfMon, CA Introscope, AppDynamics, New Relic, CloudWatch,….
3rd party services Virtualized services
Cloud Servers
Real time analytics
Test the Back End
Load Testing Tool
April 15, 2023 47© 2015 SOASTA. All rights reserved.
Continuous Mobile Testing
Utilize Real User data to drive dev & test specifications
April 15, 2023 48© 2015 SOASTA. All rights reserved.
Continuous Mobile Testing
I JUST MADE A CODE CHANGE
CI runs unit tests
Utilize Real User data to drive dev & test specifications
April 15, 2023 49© 2015 SOASTA. All rights reserved.
Continuous Mobile Testing
I JUST MADE A CODE CHANGE
CI runs unit tests
Utilize Real User data to drive dev & test specifications
Spin up suitable, consistent test environments in the cloud
or the lab
April 15, 2023 50© 2015 SOASTA. All rights reserved.
Continuous Mobile Testing
Mobile automation runs in remote or on premise devices
I JUST MADE A CODE CHANGE
CI runs unit tests
Utilize Real User data to drive dev & test specifications
Spin up suitable, consistent test environments in the cloud
or the lab
April 15, 2023 51© 2015 SOASTA. All rights reserved.
Continuous Mobile Testing
Iterative load tests run as CI tasks
Mobile automation runs in remote or on premise devices
I JUST MADE A CODE CHANGE
CI runs unit tests
Utilize Real User data to drive dev & test specifications
Spin up suitable, consistent test environments in the cloud
or the lab
April 15, 2023 52© 2015 SOASTA. All rights reserved.
Continuous Mobile Testing
Iterative load tests run as CI tasks
Mobile automation runs in remote or on premise devices
I JUST MADE A CODE CHANGE
CI runs unit tests
Daily (or more frequent) updates keep teams quality focused
Embedded failure details & bottleneck data isolate issues
Utilize Real User data to drive dev & test specifications
Spin up suitable, consistent test environments in the cloud
or the lab
April 15, 2023 53© 2015 SOASTA. All rights reserved.
Continuous Mobile Testing
Iterative load tests run as CI tasks
Mobile automation runs in remote or on premise devices
I JUST MADE A CODE CHANGE
CI runs unit tests
Manage to a performance
baseline
Daily (or more frequent) updates keep teams quality focused
Embedded failure details & bottleneck data isolate issues
Utilize Real User data to drive dev & test specifications
Spin up suitable, consistent test environments in the cloud
or the lab
April 15, 2023 54© 2015 SOASTA. All rights reserved.
?EXPERTISE
April 15, 2023 55© 2015 SOASTA. All rights reserved.
?TIME
April 15, 2023 56© 2015 SOASTA. All rights reserved.
?PROCESS
April 15, 2023 57© 2015 SOASTA. All rights reserved.
Mobile Test Automation
Techniques, Tips & Tools for Mobile
April 15, 2023 58© 2015 SOASTA. All rights reserved.
Mobile Test Automation
Real TestDevices
Techniques, Tips & Tools for Mobile
April 15, 2023 59© 2015 SOASTA. All rights reserved.
Mobile Test Automation
Performance Testing
Real TestDevices
Techniques, Tips & Tools for Mobile
April 15, 2023 60© 2015 SOASTA. All rights reserved.
Mobile Test Automation
Performance Testing
Real TestDevices
Solid Integration
Techniques, Tips & Tools for Mobile
Mobile is your chance to make a CHANGE
You CAN do it!
April 15, 2023 63© 2015 SOASTA. All rights reserved.
Resources:
SOASTA.com
o Whitepaper: 7 Steps to Pragmatic Mobile Testing• http://info.soasta.com/Whitepaper-7-Steps-to-Pragmatic-Mobile-Testing.html
o July 7 Webinar “Strategies for Mobile Automation Success”• http://info.soasta.com/strategies-for-mobile-test-automation-success.html
o Recorded webinars - e.g. “Pragmatic Mobile Testing”• http://www.soasta.com/webinars/
Others
o “Future of Testing” (search Stickyminds/Techwell)
o “Mobile Test Coverage Index” (search Perfecto)
See “7 Steps” Session @ 3:00pm!
April 15, 2023 64© 2015 SOASTA. All rights reserved.
@bradjohnsonsv bjohnson@SOASTA
Brad Johnson
top related