dealing with device fragmentation in mobile games testing · ui scripts difficult game engines like...
TRANSCRIPT
![Page 1: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/1.jpg)
Dealing with Device Fragmentation in
Mobile Games Testing
Ru Cindrea - Altom Consulting
![Page 2: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/2.jpg)
About meand Altom
● started as a tester in 2002
● partner and software tester at Altom since 2008○ software testing services○ testing training - BBST series online
● into mobile testing, mobile automation and mobile app development
● lately worked with Bitbar on using Testdroid Cloud for mobile games testing using image recognition
@ru_altom
![Page 3: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/3.jpg)
Testdroid Cloud from Bitbar
● real devices in the cloud● support for most common platforms● over 500 unique devices ● remote access as well as running scripts
● working with mobile games companies ● helping them develop a test framework
that allows for fast checking of new builds
![Page 4: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/4.jpg)
Mobile Games:The Context and The Challenges
![Page 5: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/5.jpg)
![Page 6: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/6.jpg)
➔ Lessons learned from Web and Mobile App testing to deal with fragmentation:
◆ use scripts to automate repetitive checks◆ choose most common combinations
Fragmentation
![Page 7: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/7.jpg)
A Definition “In computer programming, fragmentation is when a
combination of software and hardware do not provide a consistent, top-level experience for the vast majority of its user-base.”
![Page 8: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/8.jpg)
Most Common Where?
![Page 9: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/9.jpg)
Most Common Based on What?
![Page 10: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/10.jpg)
Most Common Based on What?
➔ Graphics/performance can make or break a game➔ Factors:
◆ OS Versions◆ Memory ◆ Resolution◆ Chipsets◆ OEM
➔ Low and mid range devices are important
![Page 11: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/11.jpg)
Incredibly Competitive Market
![Page 12: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/12.jpg)
Over 6 billion USD revenue for the Top 10 Mobile Games in 2015 (https://www.superdataresearch.com/blog/us-digital-games-market)
Ilkka Paananen - Supercell CEO:
● Top 10 companies make 50% of all mobile games revenue
● 10000 new games are submitted monthly
![Page 13: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/13.jpg)
➔ Lessons learned from Web and Mobile App testing to deal with fragmentation:◆ use scripts to automate repetitive checks◆ choose most common combinations
Fragmentation
no longer restricted to X most common devices, we can test on over 400 of them
![Page 14: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/14.jpg)
Automation Challenges in Mobile Games
![Page 15: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/15.jpg)
Automated UI
ScriptsDifficult
Game Engines like Unity
● exported binaries for iOS, Android, etc.=> game is one big canvas
● game engine tools are mostly focused on unit testing and require instrumentation
● “click at coordinates” not feasible
![Page 16: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/16.jpg)
Automated UI
ScriptsDifficult
Game Engines like Unity
● exported binaries for iOS, Android, etc. => game is one big canvas => image processing/recognition?
● game engine tools mostly focused on unit testing and require instrumentation
● “click at coordinates” not feasible=> unless we know the exact coordinates all the time?
![Page 17: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/17.jpg)
Framework Using
OpenCV & Appium
● use OpenCV to find objects on screen
● create a test framework around it that allows clients to easily develop their own scripts
● worked with Bitbar team on developing this framework
● have scripts runnable in Testdroid Cloud
![Page 18: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/18.jpg)
We don’t want pixel perfect or exact matches
![Page 19: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/19.jpg)
OpenCV: Feature Matching + Homography to find Objects with Akaze Algorithm
![Page 20: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/20.jpg)
OpenCV Akaze Algorithm: Resolution Agnostic, works rotated and stretched items
![Page 21: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/21.jpg)
OpenCV Akaze Algorithm: Resolution Agnostic, works rotated and stretched items
![Page 22: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/22.jpg)
OpenCV Akaze Algorithm: Resolution Agnostic, works rotated and stretched items
![Page 23: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/23.jpg)
OpenCV Akaze Algorithm: Resolution Agnostic, works rotated and stretched items
![Page 24: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/24.jpg)
Set of Query Images = Mobile Elements
![Page 25: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/25.jpg)
![Page 26: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/26.jpg)
Mobile Elements defined with query images
![Page 27: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/27.jpg)
Mobile Elements defined with query images
![Page 28: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/28.jpg)
Mobile Elements defined with query images
![Page 29: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/29.jpg)
Mobile Elements defined with query images
![Page 30: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/30.jpg)
![Page 31: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/31.jpg)
Appium
![Page 32: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/32.jpg)
![Page 33: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/33.jpg)
Now the same thing in the Testdroid Cloud
![Page 34: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/34.jpg)
![Page 35: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/35.jpg)
● Create new groups of devices○ filters
■ platform■ API Levels■ resolutions■ etc
![Page 36: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/36.jpg)
![Page 37: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/37.jpg)
![Page 38: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/38.jpg)
![Page 39: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/39.jpg)
Conclusions
![Page 40: Dealing with Device Fragmentation in Mobile Games Testing · UI Scripts Difficult Game Engines like Unity exported binaries for iOS, Android, etc. => game is one big canvas game engine](https://reader030.vdocuments.mx/reader030/viewer/2022041117/5f2c9ff151e488788b423594/html5/thumbnails/40.jpg)
● very reliable● types of problems found:
○ out of memory○ crashes○ graphics missing/not displayed correctly
● start with simple scenarios● allow for fast checking by a person after each run rather than trying to
verify everything automatically
Some conclusions