1Methods of Social Inquiry
for Game Design:Some Initial Thoughts
Elizabeth “Ellie” BartelsSenior Associate, Caerus [email protected]
Overview
Purpose and Motivation Current Design Process and Critique Social Science and Game Design Best Practices for Game Design Broader Considerations
2
3
Key Research Questions
How can we build games to facilitate meeting our objectives?
How can we assess the validity of findings from games?
How can sponsors and clients use these tools to understand findings of the games we build for them?
4
Goal of Improving Game Design Methods
Reality of games is that they will always be imperfect instruments Witches of Wargaming Logistical constraints
Huge strength of gaming is flexibility, cookie cutter approach not the answer
Point is not to say anything short of a perfectly designed game is unacceptable, its to be able to talk about how these limitations on design limit our findings Set expectations before the game Defend findings after the game
5
Objectives Research
Game Structure, Scenario, Rules, and
Roles Developme
nt
Game Execution
Post Game Analysis
Standard Game Design Process
6
On Game Design (Chap 5 Art of Wargaming) Focus on crafting game objectives based on sponsor
requirements How do you translate broad objective (ex. “Understand XX
phenomenon”) into concrete focus areas? World building more art than science, no real formalisms
available How do you decide which world is most useful to build?
Define Information Requirements, “all necessary information and no unnecessary information” How do I know what is necessary? What is the potential
cost of getting it wrong?
7
Question of Structure Often talk about desire to
increase game structure, but we mean two different things:1. Structured Methods ask
standardized questions that guide and standardize data collection, making systematic comparisons possible
2. Structured Problems have clearly defined boundary, highlighting specific variables and strong hypothesis about their causal relationship
Structured
Problem
Unstructured Problem
Structured
Method
Unstructured Method
Matrix Gaming
Mods & Sims
Reporting
Wastes of Time
How to Improve Structure
8
Structured Problem
Unstructured Problem
Structured MethodUnstructured Method
Most Current Gaming
Where We Need to
Go
Where We Talk about
Going
9
Proposed Game Design ProcessBased on Social Science Research Methods
Objectives Define mental model/hypothesis/theory Strategy for instantiating the model Method of representation (Game Structure, Scenario,
Rules, and Roles) Game execution Post-game analysis
10
Gaming in Social Science
Gaming is used by some social scientists, but generally as a pedagogical tool (ex. Asal and Brynen) or as part of experimental treatment (ex. McDerrmott) Use is limited by perceived lack of rigor
Game theory related, but often too simplified to do more than point to abstract types of systems (ex. Axelrod)
Very limited published work situating gaming among other techniques at a theoretical level
11
Social Science in Gaming
Social science is used by some gamers, as a supplement to more standard gaming techniques (ex. Wong and Cobb, Brightman)
Use of social science often linked to study of Irregular Warfare issues
Limited published work on applying social science thinking to overarching game design (ex. McCown, and Bartels, McCown, and Wilkie)
12
Introduction to Social Science Approaches
Limiting to neopositivist approaches
Three categories: Formal Models: Creation of theorized mechanism in artificial environment that can
be compared to real world events
Statistics: Measuring causal effect through correlation between large numbers of examples using quantitative measurements
Case Studies: Measuring causal mechanism through tracing sequence of events across small number of rich qualitative examples
All require: Developing a logically consistent model, deriving observable implications, testing those observables in the real world, and using results of test to infer about world more broadly
13
Games and Case Studies Case studies a promising, but relatively untapped methodology
for structure research of problem at a range of different levels of structure
Appropriate to exploring complex causal relationships through structured, but flexible, approach
Methodology supports not only theory testing, but also the emergence of new theories and variables over the course of research
Primary Reference: Alexander George and Andrew Bennett, Case Studies and Theory Development in the Social Sciences, MIT Press, 2005
14
Logic of Case Study Research Case(s) should be examples of a class or sub-class of
phenomenon Historical events are usually examples of multiple classes, so we
need to be specific about what lens we are using
Method for structured, focused analysis: Structured: Questions, based on objective, are asked of each
case to guide and standardize data collection, which makes systematic comparison possible
Focused: Motivated by a specific research question Both can be harder to achieve in collaborative environment,
important to ensure all parties share understanding
15
Logic of Case Study Research for Games
Games can most often be thought of a single case study, where the comparison is: Shown by variation over time (process tracing) Counter factual analysis to posit a control
Can get stronger findings when there are clear comparisons between the game and another case, but not as common Historical reality Paired Games
16
Best Practices for the Research Design Process
Case Study Term Gaming EquivalentProblem and Research Question Purpose and Objectives
Research Strategy and Specification of Variables
Game Concept
Case Selection Scenario Setting
Describe Variance in Key Variables
Defining Scenario, Roles, and Rules
Data Requirements Data Collection Plan
17
Best Practices for Objectives Case study objectives derive from research problem, just as game objectives
derive from sponsor problem
Central problem can be well structured or poorly structured, which will shape objectives
6 broad types of objectives of case studies, which can also apply to games:
Descriptive
Pattern detection
Identify variables, hypotheses, causal mechanism, and paths
Plausibility probes of new hypotheses
Testing validity and scope of theory
Explain case with existing theory
18
Best Practices for Game Concept: Research Strategy
Hypothesis: What is the current understanding of what is occurring Because games are often on un- or poorly-structured problems, often
these are messier in games than in social science Gamers frequently skip this step, but this is what drives all future
decisions about what is, and is not, important to include in the game Critical when linking games to other analysis, as need to ensure
consistent definitions and assumptions are in place to make compatible
In single cases (which include most games) important to consider all available alternative hypotheses, including those suggested by participants, to ensure “left out” options don’t threaten validity
19
Best Practices for Game Concept: Variable Definition
Definition of variables: Outcome to be explained or predicted (dependent variable) Inputs and intervening variables that might cause outcome Which variables are we interested in variation, and which should
be constant
Confounding variables are factors that might produce the outcome being studied but are not the input of interest. Ideally, these are constant across the case, but often need to show through process tracing that while present, they don’t appear at the right time point in events to create the outcome
20
Best Practices for Game Concept: Decisions as a Key Variable
Games by definition have player decisions as a key variable, but depending on objectives they are different types of variables
Independent: Participant decisions produce what outcomes? These games are focused on the product of decision
Dependent: What decisions are made by participant given specific inputs? These games are focused on the process of decision making
21
Best Practices for Game Concept: Other Key Variables
Other variables in games: Environmental factors included (or excluded) in the scenario Actors represented (or not) by the players Rule formalized (or not) in the rules
When possible, define variables clearly ahead of game play: Outcome to be explained/predicted (Dependent) Variables that contribute to the outcome (Independent) Constants
22
Best Practices for Scenario Setting First criteria should be relevance of case to research question
Case/setting must be an example of a class of events being studied “Cuban missile crisis” can be used as a case of deterrence, or coercive diplomacy,
crisis management, etc.
Studying most and least likely cases, or “deviant” cases more likely to yield interpretable results Most likely: favorable case, possible to strongly undermine hypothesis Least likely: unfavorable case, possible to generate convincing support for hypothesis
Good tests are often unrepresentative, as a result it is important not to overgeneralize
Often case is starting point, rather than theory, but still requires clear objectives and research strategy (including clearly stated hypothesis)
23
Best Practices for Defining Scenario, Roles, and Rules
Understanding range of variation that can occur is important for specifying what causal relationships are illustrated Think carefully about what variation you need in the case Cases with clear variances are more important for theory testing
Limits in variation considered by designer is based on habit as well as practical constraints Environment limited by time and data available Player roles limited by available participants, number of cells by space Rules limited by staff available to support communication and adjudication
Critical to be honest about how these choices restrict game findings
24
Best Practices for Defining Scenario, Roles, and Rules: Variation in between Game and Reality has
a…Decisions are Independent Variable/Process Games
Decisions are Dependent Variable/Product Games
Roles Medium Impact on Validity • Scope (who is involved in
decision making)• Level of analysis (Essence of
Decisions)
Medium Impact on Validity • Scope (who is involved in decision
making)• Level of analysis (Essence of
Decisions)
Environment Small Impact on Validity • Information availability
Large Impact on Validity • Problem type• Timelines• Information availability• Context
Rules Large Impact on Validity • Planning paradigms• Information sharing
Small Impact on Validity • Information sharing
25
Best Practices for Defining Scenario, Roles, and Rules
Critical to be honest about how these choices restrict game findings Internal Validity: how strong a claim can you make about
causality External Validity: how much can you generalize the findings of
the game to real world cases
Common problems: Elimination of alternative explanations for the outcome Failure to consider alternative hypothesis Limited applicability to other cases
26
Best Practices for Data Collection Plan
Consistent, thoughtful data collection plan provides the structure needed to support systematic analysis
Generalized questions to be asked: Not a mechanical process, questions must be integrated
with other decisions made during research design Does not preclude collecting more data
Better defined research design supports more precise data collection
27
Consequences for Games as Part of Larger Studies
Focus on good design can make it easier to link games to other research projects
Allows better variable matching by making sure consistent variables are actually comparable across studies
Often there’s a desire to move from unstructured to more structured problems, or from the specifics of a particular empirical case to findings that have a more general scope, this method highlights those opportunities
28
Abductive, Inductive, or Deductive? We are only able to undertake the process of simplifying the
complexity of the world into the artificial form of game by making choices about what to include or exclude
We make these choices based on our existing hypothesis or mental models of the problem set
We as designers have a theory, and use that theory to shape the game world deductively, we just generally don’t acknowledge it
Deductive design method forms boundary conditions for the abductive or inductive analytic outputs of games