The Way to Production:Game Testing & Analitics

Lecture 04

2016-2017Mobile Games Development & Promotion, Maksimenkova

Olga, FCS, SSI1

Today

Testing

• What to test in games?

• How to test?

• Features for mobile games?

Statistics

• Metrics

• Tools

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI2

Game testingTesting and QA

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI3

Beta testing

• Closed (internal) beta

• Open (public/external) beta

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI4

What should be tested in a game?

• Balance (balance testing)• Fairness, a gameplay is fair to both the human and AI player alike• Difficulty balance• Levels balance• Maps, weapons,….• The guidelines for balancing a game see in (The many faces of testing: the

game life cycle, Game Development Essentials: Game Testing & QA by L. Levy, J. Novak)

• Hardware compatibility (compatibility testing)• Video subsystem• Audio subsystem• Controllers (input/output devices)• Network… • See a sample checklist at bitbar (http://bitbar.com/mobile-game-testing-

part-4-test-for-connectivity/)

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI5

What should be tested in a game?

• Compliance testing • Meeting standards, specifications, requirements

• Tolerance

• Warnings

• Copyrights, ….

• Internationalization (Localization Testing)

• Funniness (Playtesting)

• Usability (usability testing)• UI/UX

• Tfortesting. Testing make things perfect, Test Cases For Games Apps / Checklist for Games Apps (https://tfortesting.wordpress.com/2012/10/04/test-cases-for-games-apps-checklist-for-games-apps/)

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI6

Testing techniques

• Ad-Hoc• Finding bugs by playing.

• Automated• Let computer do repetitive work for you

• Functionality• Make sure that game functions properly well according to your expectations• Functionality requirements see in (Race to the Finish Line: elite bug hunting,

Game Development Essentials: Game Testing & QA by L. Levy, J. Novak)

• Progression• Seeking possibly interruptions in a gameplay

• Regression• Checking the results of old bugs fixing

• Mobile game testing see at bitbar (http://bitbar.com/mobile-game-testing-the-importance-and-difference-from-app-testing/)

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI7

Study

• Find and compare definitions of crowdsourcing• Please avoid using only Wikipedia

• What is the difference between crowdsourcing and outsourcing?

• What is crowdsourcing testing?• Give a list of features with descriptions and links

• Explore the types of software testing, compare them with game testing.• Give a table of comparison

• Use all above and sources from the References to build testing strategies of your game.• Describe stages, game testing scenarios

• Prepare testing artifacts like checklists, testing requirements document, software testing plan, game testing plan

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI8

Crowdsourced testing

• The Definitive Guide to Crowdsourced Testing (for Testers and Companies) (http://www.softwaretestinghelp.com/guide-to-crowdsourced-testing/)

• Traditional Testing vs. Crowdsource Testing vs. Freelance Testing (https://www.qualitestgroup.com/blog/test-methodologies-and-philosophies/traditional-testing-vs-crowdsource-testing-vs-freelance-testing/)

• Outsource Vs. Crowdsource Testing (http://www.base36.com/2013/05/outsource-vs-crowdsource-testing/)

• Crowdsourced Testing: A New Way to Do Mobile App Testing (https://dzone.com/articles/crowdsourced-testing-a-new-way-to-do-mobile-app-te)

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI9

Game analyticsMetrics and Statistics

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI10

Basic metrics

• Relationship between User and a Game• DAU (Daily Active Users)

• The number of unique users that start at least one session in a game on any day.• DAU may be called a measure of “stickiness” of a game

• Sessions• Sessions number is increased every time user opens a game.• Brings an information as a ratio with DAU

• DAU/MAU• MAU is Mothly Active Users• DAU/MAU ratio may show how well a game retains users

• Retention• Churn

• How many users who downloaded and may be set up a game are no longer playing?

• Money• Conversion Rate

• measures the percentage of unique users that have made a purchase out of the total number of users during that time period.

• ARPDAU (Average Revenue Per Daily Active User)• Allows to measure how a game performs.• Useful if tracked before and after user completes scenario/campaign/level.

• ARPPU (Average Revenue Per Paying User)• ARPDAU only for paid users

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI11

In-Game Metrics

• Source-Sink-Flow• Source means primary and secondary currency

• Sink the ways of spending these types of currency

• Flow defines the balance between gaining and spending currency

• Start-Fail-Complete (progression metrics)• Start is about starting new levels. It measures the number of times a player

starts a new level.

• Fail measures the number of times when user started but did not complete level.

• Complete measures the number of times when user started and complete level

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI12

Reference Study (Metrics)

• Study the following links:• 15 Metrics All Game Developers Should Know by Heart

(http://www.gameanalytics.com/blog/metrics-all-game-developers-should-know.html)

• Glossary of Metrics (https://unity3d.com/ru/learn/tutorials/topics/analytics/glossary-metrics)

• The Game Life Cycle & Game Analytics: What metrics matter when? (http://www.slideshare.net/TomSente/casualconnect2012-honeytracks-game-lifecycle-kpis)

• How to calculate Mobile Game Retention and LTV using Google Analytics (http://measureofluck.com/calculate-retention-ltv-using-google-analytics/)

• Complete the list below with new metrics and their descriptions

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI13

Reference Study (Analytics Tools)

• Study the following links:• What is the best analytics tool for online mobile game?

(https://www.quora.com/What-is-the-best-analytics-tool-for-online-mobile-game)• Analytics that empower you (http://www.gameanalytics.com/features)• Help your players level up their game

(https://www.helpshift.com/gaming/?utm_source=google&utm_medium=cpc&utm_campaign=Mobile+Gaming&utm_term=%2Bmobile%20%2Bgame%20%2Banalytics&mt=b&ad=108252874846&gclid=CjwKEAiA94nCBRDxismumrL83icSJAAeeETQxdtomAPDzvBEfxGEUImyKbnvLFbzY5AFNHvdJ-HFORoC_yzw_wcB)

• The ten best free analytics tools for mobile app developers: from Google Analytics toFabric, Firebase, Flurry and DevMate (http://www.techworld.com/picture-gallery/apps/eight-best-free-analytics-tools-for-mobile-app-developers-3636407/)

• Prepare a list of analytics tools for mobile games, what metrics they allow to measure.

• Select one tool suitable for the genre and monetizing strategy of your game-project.• Study it carefully and justify its application in your project. • Describe scenarios with screenshots of how will you measure your game metrics.• Give the examples of metrics for your game.

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI14

References (Testing)

• L. Levy, J. Novak, Game Development Essentials: Game QA & Testing (2010).

• C.P. Schultz, R.D. Bryant, Game Testing: All in One (2011)

• J. Hoberg, Difference between software testing and game testing (http://www.gamasutra.com/blogs/JohanHoberg/20140721/221444/Differences_between_Software_Testing_and_Game_Testing.php)

• J. Carrera, Test Planning and Execution in a Mobile Development Project using SCRUM (https://www.scribd.com/document/53989828/Test-Planning-and-Execution-in-a-Mobile-Game-Development-Project-using-SCRUM)

• Game testing methodology (https://www.scribd.com/doc/6555758/Game-Testing-Methodology)

• 17 Strategies for End to End Mobile Testing on both iOS and Android (https://www.ymedialabs.com/17-strategies-for-end-to-end-mobile-testing-on-both-ios-and-android/)

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI15

References (Metrics)

• Mobile Game Metrics: Why Focus on a Single KPI Can Be Dangerous (https://sweetpricing.com/blog/2016/08/mobile-game-metrics/)

• Tracking the metrics that matter in social and mobile gaming (http://www.develop-online.net/opinions/tracking-the-metrics-that-matter-in-social-and-mobile-gaming/0196581)

• A comprehensive list of metrics for free-to-play games (http://www.gamesbrief.com/2014/05/a-comprehensive-list-of-metrics-for-free-to-play-games/)

• Free to play game metrics for beginners (http://www.gdcvault.com/play/1015473/Free-to-Play-Game-Metrics)

• 9 Mobile App KPIs to Know (http://mashable.com/2013/09/04/mobile-app-metrics/#OlOMPY9cGkq1)

• Mobile Game Analytics Example (https://calq.io/docs/resources/track/mobile-game)

• New mobile game statistics every game publisher should know in 2016 (https://www.surveymonkey.com/business/intelligence/mobile-game-statistics/)

• Retention and LTV as Core Metrics to Measure Mobile Game Performance (http://www.gamasutra.com/blogs/IgorMatrofailo/20151221/262214/Retention_and_LTV_as_Core_Metrics_to_Measure_Mobile_Game_Performance.php)

2016-2017Mobile Games Development & Promotion,

Maksimenkova Olga, FCS, SSI16