continuous quality as a key for a 5 star mobile apps delivery
TRANSCRIPT
Continuous Quality As a Key for A 5 Star Mobile Apps Delivery
@esconfs#esconfs
Eran Kinsbruner Perfecto Mobile, Israel
About MeEran Kinsbruner is a director & mobile technical evangelist at PerfectoMobile, one of the leading mobile cloud and automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke & Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing blog at ek121268.wordpress.com.
Contact Me: [email protected]
Agenda
3
Mobile marketST
ART Mobile
Testing Coverage
Mobile ProjectsChallenges
Mobile Test
automationCI & Best Practices
Easily Applied for Browsers …
Mobile Create Challenges …
UIIntegration
Unit Testing
UIIntegration
Unit Testing
Performance Security
Mobile Requires Shifting the Plan …
Target Devices
TargetEnv.OS
OS versionsForm factorsCPU & MemoryManufacturersWearable
NetworksOrientationEventsDevice eventsLocationsNotifications
Common Mobile Project Challenges/Pains
6
7
• Pain:• Market pressure compressing delivery schedules• Growing App complexity driving test case growth• Rapidly evolving test matrix
• Goal:• Deliver feedback to developers faster• Complete test coverage within each sprint cycle
• Strategy• Increase automation• Adopt continuous integration
The Challenges – Velocity vs. Quality
And
Or
What does it take to cover the REALend user experience?
v
Device & Platform
Model
OS Version
Screen Size
Form Factors v
Environment
Location
Network
v
Conditions
Signal Strength
Load
Others
Q1 2015 – EU5 Index50% Preview
Supports:• Sufficient Device Mix
• Legacy, Reference, New• Allows sufficient OS coverage• Support mix of Screen Sizes• Support mix of hardware’s
Application name position
Labels
Menu panel
Button images
Text colors and images
Cross Platform Differences
Functional Automation – Mobile Objects
100% accurate native object rec.
Much faster than OCR
Support all languages
Does not often detect GUI defects
Imperative for enduser’s experience
Imperative for GUI glitches
Implement theWYSIWYG
Slower then objectlevel analysis
Dependent on 3rd party
Does not support all languages
Object level analysis
Visual analysis (OCR)
May require Instrumentation
Mobile Test Automation That Works
• Develop for testability• Object Recognition
– Mix ‘n match methods• Image recognition• DOM• Object id• XPath• CSS
• Robust– Error handling– Conditional flow– Reusable functions– Data driven
ReliableRepeatable Maintainable
Scalable
CI Blockers: Inefficient & Brittle AutomationW
hat
• Automated testing takes too much time to develop
• Constant maintenance
Why
• Advance coding skills • No cross platform tests• Lack coding guidelines
How
• Build stable framework • Use predefined code
functions and existing product
• Separate test code from execution mgt
CI Blockers: Poor Execution OptimizationW
hat • Automated testing run
slow• Too many tests, not
enough time• Not stable on mobile
devicesW
hy • Lack of execution management
• Single threaded automation
• Device errors
How
• Parallel execution • Tune the code• Apply automation • best practices• Manage tests like code
Error handling Remove CI/Automation Blockers
System level control
Application control
Device control
Mobile Continuous Integration Recommended Work Flow
Dev Team
Source Control
Code Commit
CI Server
ReportingUnit Tests
Launch Perfecto Job (Release Package)
Support today: PM Native &
Selenium
Automated App Deployment On Real Devices
Functional TestingIntegration Testing
Performance Testing
Continuous Feedback
Test Planning & Test Execution
Bug Reporting
Detailed reports
Planning optimization for next build
Manage the Time Budget …
Build Frequency
TestDuration
Test ScopeDevice
CoverageUnit System Smoke Regression Performance
Hourly 20-40 Min ±4
3-4 times a day 30-60 Min ±6
Nightly 2-7 Hours ±10
Weekend 10-48 Hours ±20
Velocity! Quality!
Strategies for achieving 5 stars
1. Assure device coverage
Real Devices
Optimal mix to cover
market
Max Elasticity
My app analytics Mobile Market
My Space
App Control
System Control
Device control
Network Control
Burst up/down
Always available
1. Assure device/OS coverage
April OCTOBERJUNE JULY AUGUST SEPTEMBER
Sony Xperia Z4 Tablet
New iPhones & iPadsSamsung Note 5
Motorola Moto X (New)
iOS 9 BetaWindows 10for Phones
DEVICES
PLATFORM
Mobile Market Calendar 2015
Android M ?
MAY
Apple Watch Release
Samsung Galaxy S6Samsung Galaxy S6 EdgeLG G4Nokia Lumia 640 XL
ZTE Blade S6 PlusHuawei P8Lenovo A6000 Plus
New trending devices,First Samsung’s to run Android L OTS
You can already start Beta testing in June!
Seek for Emulators of iPhone 6S and more close to the release date!
Seek for early Beta through Android StudioNew nexus devices
Perfecto Mobile’s Q2 Index Release!!
2. Built an adaptive test plan
Update your test plan according to customer feedback
Automate to run regularly
Mobile Ready Automation
Fast Impact utilizing existing
knowledge
Test on real devices
Sustainable – doesn’t break
CI Ready
Device Agnostic, any
app type
Unattended
3. Mobile-Ready Automation is Key
Become a Jedi Master you will
Does your Lab come with “Environment Controls”?
Are you testing “in the Wild”?
4. Test in Real-World Conditions
Build Your Mobile App Test Plan ProperlyFunctional• Validation of Functionality• Smoke / Regressions Testing• Offline access testing• Negative Testing
Non Functional• Network Strength / Outage /
Recovery• Different NW Types• Peripheral Testing• Services
Interoperability (IOP)• Voice / SMS interrupts• Notifications• Battery /Cable Removal
Memory Leak• Memory Usage• Memory Leaks• Garbage Collection
Installation Testing• New App Install• Uninstall and Reinstall• Upgrade testing
Language Testing• Validation for Locales (French
& English)• Images and Text• Currencies, time zones etc.• Context
Performance Testing• CPU Usage testing• Network Usage• Page Render time or activity
Render time
Security Testing• OWASP Vulnerabilities• Sensitive data (App/Device)• Dynamic Testing• Static Code Analysis• Data Encryption
Usability Testing• User Experience• Competitive Analysis• Expert Review
Get 15% off using Discount Code:
MWS15(until June 30th)