1Methods of Social Inquiry

for Game Design:Some Initial Thoughts

Elizabeth “Ellie” BartelsSenior Associate, Caerus Associatesebartels@caerusassociates.com

Overview

Purpose and Motivation Current Design Process and Critique Social Science and Game Design Best Practices for Game Design Broader Considerations

2

3

Key Research Questions

How can we build games to facilitate meeting our objectives?

How can we assess the validity of findings from games?

How can sponsors and clients use these tools to understand findings of the games we build for them?

4

Goal of Improving Game Design Methods

Reality of games is that they will always be imperfect instruments Witches of Wargaming Logistical constraints

Huge strength of gaming is flexibility, cookie cutter approach not the answer

Point is not to say anything short of a perfectly designed game is unacceptable, its to be able to talk about how these limitations on design limit our findings Set expectations before the game Defend findings after the game

5

Objectives Research

Game Structure, Scenario, Rules, and

Roles Developme

nt

Game Execution

Post Game Analysis

Standard Game Design Process

6

On Game Design (Chap 5 Art of Wargaming) Focus on crafting game objectives based on sponsor

requirements How do you translate broad objective (ex. “Understand XX

phenomenon”) into concrete focus areas? World building more art than science, no real formalisms

available How do you decide which world is most useful to build?

Define Information Requirements, “all necessary information and no unnecessary information” How do I know what is necessary? What is the potential

cost of getting it wrong?

7

Question of Structure Often talk about desire to

increase game structure, but we mean two different things:1. Structured Methods ask

standardized questions that guide and standardize data collection, making systematic comparisons possible

2. Structured Problems have clearly defined boundary, highlighting specific variables and strong hypothesis about their causal relationship

Structured

Problem

Unstructured Problem

Structured

Method

Unstructured Method

Matrix Gaming

Mods & Sims

Reporting

Wastes of Time

How to Improve Structure

8

Structured Problem

Unstructured Problem

Structured MethodUnstructured Method

Most Current Gaming

Where We Need to

Go

Where We Talk about

Going

9

Proposed Game Design ProcessBased on Social Science Research Methods

Objectives Define mental model/hypothesis/theory Strategy for instantiating the model Method of representation (Game Structure, Scenario,

Rules, and Roles) Game execution Post-game analysis

10

Gaming in Social Science

Gaming is used by some social scientists, but generally as a pedagogical tool (ex. Asal and Brynen) or as part of experimental treatment (ex. McDerrmott) Use is limited by perceived lack of rigor

Game theory related, but often too simplified to do more than point to abstract types of systems (ex. Axelrod)

Very limited published work situating gaming among other techniques at a theoretical level

11

Social Science in Gaming

Social science is used by some gamers, as a supplement to more standard gaming techniques (ex. Wong and Cobb, Brightman)

Use of social science often linked to study of Irregular Warfare issues

Limited published work on applying social science thinking to overarching game design (ex. McCown, and Bartels, McCown, and Wilkie)

12

Introduction to Social Science Approaches

Limiting to neopositivist approaches

Three categories: Formal Models: Creation of theorized mechanism in artificial environment that can

be compared to real world events

Statistics: Measuring causal effect through correlation between large numbers of examples using quantitative measurements

Case Studies: Measuring causal mechanism through tracing sequence of events across small number of rich qualitative examples

All require: Developing a logically consistent model, deriving observable implications, testing those observables in the real world, and using results of test to infer about world more broadly

13

Games and Case Studies Case studies a promising, but relatively untapped methodology

for structure research of problem at a range of different levels of structure

Appropriate to exploring complex causal relationships through structured, but flexible, approach

Methodology supports not only theory testing, but also the emergence of new theories and variables over the course of research

Primary Reference: Alexander George and Andrew Bennett, Case Studies and Theory Development in the Social Sciences, MIT Press, 2005

14

Logic of Case Study Research Case(s) should be examples of a class or sub-class of

phenomenon Historical events are usually examples of multiple classes, so we

need to be specific about what lens we are using

Method for structured, focused analysis: Structured: Questions, based on objective, are asked of each

case to guide and standardize data collection, which makes systematic comparison possible

Focused: Motivated by a specific research question Both can be harder to achieve in collaborative environment,

important to ensure all parties share understanding

15

Logic of Case Study Research for Games

Games can most often be thought of a single case study, where the comparison is: Shown by variation over time (process tracing) Counter factual analysis to posit a control

Can get stronger findings when there are clear comparisons between the game and another case, but not as common Historical reality Paired Games

16

Best Practices for the Research Design Process

Case Study Term Gaming EquivalentProblem and Research Question Purpose and Objectives

Research Strategy and Specification of Variables

Game Concept

Case Selection Scenario Setting

Describe Variance in Key Variables

Defining Scenario, Roles, and Rules

Data Requirements Data Collection Plan

17

Best Practices for Objectives Case study objectives derive from research problem, just as game objectives

derive from sponsor problem

Central problem can be well structured or poorly structured, which will shape objectives

6 broad types of objectives of case studies, which can also apply to games:

Descriptive

Pattern detection

Identify variables, hypotheses, causal mechanism, and paths

Plausibility probes of new hypotheses

Testing validity and scope of theory

Explain case with existing theory

18

Best Practices for Game Concept: Research Strategy

Hypothesis: What is the current understanding of what is occurring Because games are often on un- or poorly-structured problems, often

these are messier in games than in social science Gamers frequently skip this step, but this is what drives all future

decisions about what is, and is not, important to include in the game Critical when linking games to other analysis, as need to ensure

consistent definitions and assumptions are in place to make compatible

In single cases (which include most games) important to consider all available alternative hypotheses, including those suggested by participants, to ensure “left out” options don’t threaten validity

19

Best Practices for Game Concept: Variable Definition

Definition of variables: Outcome to be explained or predicted (dependent variable) Inputs and intervening variables that might cause outcome Which variables are we interested in variation, and which should

be constant

Confounding variables are factors that might produce the outcome being studied but are not the input of interest. Ideally, these are constant across the case, but often need to show through process tracing that while present, they don’t appear at the right time point in events to create the outcome

20

Best Practices for Game Concept: Decisions as a Key Variable

Games by definition have player decisions as a key variable, but depending on objectives they are different types of variables

Independent: Participant decisions produce what outcomes? These games are focused on the product of decision

Dependent: What decisions are made by participant given specific inputs? These games are focused on the process of decision making

21

Best Practices for Game Concept: Other Key Variables

Other variables in games: Environmental factors included (or excluded) in the scenario Actors represented (or not) by the players Rule formalized (or not) in the rules

When possible, define variables clearly ahead of game play: Outcome to be explained/predicted (Dependent) Variables that contribute to the outcome (Independent) Constants

22

Best Practices for Scenario Setting First criteria should be relevance of case to research question

Case/setting must be an example of a class of events being studied “Cuban missile crisis” can be used as a case of deterrence, or coercive diplomacy,

crisis management, etc.

Studying most and least likely cases, or “deviant” cases more likely to yield interpretable results Most likely: favorable case, possible to strongly undermine hypothesis Least likely: unfavorable case, possible to generate convincing support for hypothesis

Good tests are often unrepresentative, as a result it is important not to overgeneralize

Often case is starting point, rather than theory, but still requires clear objectives and research strategy (including clearly stated hypothesis)

23

Best Practices for Defining Scenario, Roles, and Rules

Understanding range of variation that can occur is important for specifying what causal relationships are illustrated Think carefully about what variation you need in the case Cases with clear variances are more important for theory testing

Limits in variation considered by designer is based on habit as well as practical constraints Environment limited by time and data available Player roles limited by available participants, number of cells by space Rules limited by staff available to support communication and adjudication

Critical to be honest about how these choices restrict game findings

24

Best Practices for Defining Scenario, Roles, and Rules: Variation in between Game and Reality has

a…Decisions are Independent Variable/Process Games

Decisions are Dependent Variable/Product Games

Roles Medium Impact on Validity • Scope (who is involved in

decision making)• Level of analysis (Essence of

Decisions)

Medium Impact on Validity • Scope (who is involved in decision

making)• Level of analysis (Essence of

Decisions)

Environment Small Impact on Validity • Information availability

Large Impact on Validity • Problem type• Timelines• Information availability• Context

Rules Large Impact on Validity • Planning paradigms• Information sharing

Small Impact on Validity • Information sharing

25

Best Practices for Defining Scenario, Roles, and Rules

Critical to be honest about how these choices restrict game findings Internal Validity: how strong a claim can you make about

causality External Validity: how much can you generalize the findings of

the game to real world cases

Common problems: Elimination of alternative explanations for the outcome Failure to consider alternative hypothesis Limited applicability to other cases

26

Best Practices for Data Collection Plan

Consistent, thoughtful data collection plan provides the structure needed to support systematic analysis

Generalized questions to be asked: Not a mechanical process, questions must be integrated

with other decisions made during research design Does not preclude collecting more data

Better defined research design supports more precise data collection

27

Consequences for Games as Part of Larger Studies

Focus on good design can make it easier to link games to other research projects

Allows better variable matching by making sure consistent variables are actually comparable across studies

Often there’s a desire to move from unstructured to more structured problems, or from the specifics of a particular empirical case to findings that have a more general scope, this method highlights those opportunities

28

Abductive, Inductive, or Deductive? We are only able to undertake the process of simplifying the

complexity of the world into the artificial form of game by making choices about what to include or exclude

We make these choices based on our existing hypothesis or mental models of the problem set

We as designers have a theory, and use that theory to shape the game world deductively, we just generally don’t acknowledge it

Deductive design method forms boundary conditions for the abductive or inductive analytic outputs of games