continuous quality for a 5 star mobile apps delivery
TRANSCRIPT
Mobile Test Automa/on & The Need for Con/nuous Tes/ng to Ensure High, Mul/pla=orm Performance
June, 2015
Eran Kinsbruner
Presented By:
Gidi Pridor
Introducing Perfecto Mobile
GeHng No/ced
Good Friends
At a glance
enabling manual and automated mobile applica0on tes0ng as well as applica0on monitoring for increasingly agile delivery processes.
“Perfecto Mobile marks shift in mobile apps QA with Continuous Quality Lab”
• Founded in late 2006, Business HQ in Boston and R&D in Israel • Proven management team with over 200 employees globally • Funded by leading VCs, Total -‐ $57.8M
Agenda
3
Mobile market ST
ART Mobile
TesQng Coverage
Mobile Projects Challenges
Mobile Test automaQon
CI & Best PracQces
Easily Applied for Browsers …
Mobile Create Challenges …
UI
IntegraQon
Unit TesQng
UI
IntegraQon
Unit TesQng
Performance Security
Mobile Requires ShiQing the Plan …
Target Devices
Target Env. OS
OS versions Form factors CPU & Memory Manufacturers Wearable
Networks OrientaQon Events Device events LocaQons NoQficaQons
6
• Pain: • Market pressure compressing delivery schedules • Growing App complexity driving test case growth • Rapidly evolving test matrix
• Goal: • Deliver feedback to developers faster • Complete test coverage within each sprint cycle
• Strategy • Increase automaQon • Adopt conQnuous integraQon
The Challenges – Velocity vs. Quality
And
Or
What does it take to cover the REAL end user experience?
v
Device & Pla=orm
Model
OS Version
Screen Size
Form Factors v
Environment
LocaQon
Network
v
Condi/ons
Signal Strength
Load
Others
Q1 2015 – EU5 Index 50% Preview
Supports: • Sufficient Device Mix
• Legacy, Reference, New • Allows sufficient OS coverage • Support mix of Screen Sizes • Support mix of hardware’s
UK – Q2 Sneak Preview Index
Smartphones: 1. iPhone 5S 2. iPhone 5C 3. Samsung Galaxy S4 4. iPhone 6 5. iPhone 6 Plus 6. Sony Xperia Z3 7. HTC One M8 8. Nexus 6 9. Samsung Galaxy S5 10. Samsung Galaxy Note 4 11. Samsung Galaxy A5 12. Samsung Galaxy S6 13. Samsung Galaxy S6 Edge 14. HTC One M9 15. LG G4 16. LG G3
Tablets: 1. Apple iPad Mini 3 2. Apple iPad Air 2 3. Apple iPad Air 4. Samsung Galaxy Tab 4 – 7.0’’ 5. Samsung Galaxy Tab S 10.5'' 6. Alcatel One Touch Pop 7 S 7. Sony Xperia Z3 Tablet 8. Google Nexus 9 (HTC)
Application name position
Labels
Menu panel
Button images
Text colors and images
Cross Plakorm Differences
Func/onal Automa/on – Mobile Objects
100% accurate naQve object rec.
Much faster than OCR
Support all languages
Does not onen detect GUI defects
ImperaQve for end user’s experience
ImperaQve for GUI glitches
Implement the WYSIWYG
Slower then object level analysis
Dependent on 3rd party
Does not support all languages
Object level analysis
Visual analysis (OCR)
May require InstrumentaQon
Mobile Test Automa/on That Works
• Develop for testability • Object RecogniQon
– Mix ‘n match methods • Image recogniQon • DOM • Object id • XPath • CSS
• Robust – Error handling – CondiQonal flow – Reusable funcQons – Data driven
Reliable Repeatable Maintainable
Scalable
Error handling Remove CI/Automa/on Blockers
System level control
ApplicaQon control
Device control
Mobile Con/nuous Integra/on Recommended Work Flow
Dev Team
Source Control
Code Commit
CI Server
ReporQng Unit Tests
Launch Perfecto Job (Release Package)
Support today: PM Native & Selenium
Automated App Deployment On Real Devices
Functional Testing Integration Testing
Performance Testing
Con/nuous Feedback
Test Planning & Test Execu/on
Bug ReporQng
Manage the Time Budget …
Build Frequency
Test Dura/on
Test Scope Device
Coverage Unit System Smoke Regression Performance
Hourly 20-‐40 Min ±4
3-‐4 /mes a day 30-‐60 Min ±6
Nightly 2-‐7 Hours ±10
Weekend 10-‐48 Hours ±20
Velocity! Quality!
Strategies for achieving 5 stars
1. Assure device coverage
2. Lab Sizing
Me
AnalyQcs
Teams Projects
SDLC Cycle
Requirements
v
Market
My Space
Popular & emerging
The Index
1. Coverage
Industry trends & insights
1. Assure device/OS coverage
April OCTOBER JUNE JULY AUGUST SEPTEMBER
Sony Xperia Z4 Tablet
New iPhones & iPads Samsung Note 5
Motorola Moto X (New)
iOS 9 Beta Windows 10 for Phones
DEVICES
PLATFORM
Mobile Market Calendar 2015
Android M ?
MAY
Apple Watch Release Samsung Galaxy S6 Samsung Galaxy S6 Edge LG G4 Nokia Lumia 640 XL ZTE Blade S6 Plus Huawei P8 Lenovo A6000 Plus
New trending devices, First Samsung’s to run Android L OTS
You can already start Beta tesQng in June!
Seek for Emulators of iPhone 6S and more close to the release date!
Seek for early Beta through Android Studio New nexus devices
Perfecto Mobile’s Q2 Index Release!!
2. Built an adap/ve test plan
Ø Update your test plan according to customer feedback
Ø Automate to run regularly
Mobile Ready
AutomaQon
Fast Impact uQlizing exisQng
knowledge Test on real
devices
Sustainable – doesn’t break
CI Ready
Device AgnosQc, any app type
Unaxended
3. Mobile-‐Ready Automa/on is Key
Become a Jedi Master you will
Does your Lab come with “Environment Controls”?
Are you tesQng “in the Wild”?
4. Test in Real-‐World Condi/ons
Build Your Mobile App Test Plan Properly Functional • Validation of Functionality • Smoke / Regressions Testing • Offline access testing • Negative Testing
Non Functional • Network Strength / Outage /
Recovery • Different NW Types • Peripheral Testing • Services
Interoperability (IOP) • Voice / SMS interrupts • Notifications • Battery /Cable Removal
Memory Leak • Memory Usage • Memory Leaks • Garbage Collection
Installation Testing • New App Install • Uninstall and Reinstall • Upgrade testing
Language Testing • Validation for Locales (French &
English) • Images and Text • Currencies, time zones etc. • Context
Performance Testing • CPU Usage testing • Network Usage • Page Render time or activity
Render time
Security Testing • OWASP Vulnerabilities • Sensitive data (App/Device) • Dynamic Testing • Static Code Analysis • Data Encryption
Usability Testing • User Experience • Competitive Analysis • Expert Review
Eran Kinsbruner Dir, Technical Evangelist Perfecto Mobile [email protected]
Gidi Pridor VP, Strategic MarkeQng Perfecto Mobile [email protected]