decidarch: playing cards as software architects · decidarch: playing cards as software architects...
TRANSCRIPT
DecidArch: Playing Cards as Software Architects
Patricia Lago & Jia F. CaiVrije Universiteit Amsterdam
[email protected]; [email protected]
Remco C. de BoerArchiXL
Philippe KruchtenUniversity of British Columbia, Canada
Roberto VerdecchiaGran Sasso Science Institute & VU Amsterdam
Abstract
Teaching software architecture is a challenge
because of the difficulty to expose students to actual
meaningful design situations. Games can provide
a useful illustration of the design decision making
process, and teach students the power of team
interaction for making sound decisions.
We introduce a game –DecidArch– developed to
achieve three learning objectives: 1) create awareness
about the rationale involved in design decision
making, 2) enable appreciation of the reasoning
behind candidate design decisions proposed by others,
and 3) create awareness about interdependencies
between design decisions.
The game has been played by 22 groups with a
total of 83 players, all of them students of the VU
software architecture course. We present some of
the lessons learned, both from our observation and
through participant survey. We conclude that the
game well supports our three learning objectives, and
we identify several improvement points for future
game editions.
1. Introduction
Teaching software architecture to universitystudents is particularly challenging as this is a topicthat demands experience, technical competence,domain knowledge and software development skills -all characteristics that one acquires with time andpractice rather than within the walls of a classroom.
Case studies, term-long projects, presentationsof architectural designs by actual architects fromindustry, experiments with design tools, etc. canall contribute to some aspects of this learning. Inthis paper we present our attempt to devise a simplecard game –DecidArch– to let players experience theconcept of architectural design decision making.
DecidArch is not a computer game or a computer
simulation. It is more akin to Monopoly or Risk,and can be played around a table in about an hour.
In this paper we describe our learning objectivesaround architecture decision making, the design ofthe game, and the validation experiment conducted.
2. Related Work
Over the past 15 years, there has been a lot ofinterest in games to support software engineeringeducation, as testified by the series of ICSEworkshops on the topic: GAS: Games and SoftwareEngineering from 2011 onward e.g. [1]. In theirextensive literature review [2], Kosa and colleaguesshow that the large majority (88%) of these gamesare digital, that is, they involve some ratherencompassing simulation of a software developmentsituation, and are mostly played by students againstthe computer (e.g. SimSE1).
To illustrate one situation or one concept,simpler board games or card games are oftenpreferable, and there are numerous examples ofsuccessful ones by Taran at CMU [3] and thesame group at UC Irvine [4]. “Bringing their ownunique attributes, non-digital games may provideadditional benefit. For instance, educators mayutilize them, harnessing their social characteristics(e.g. face-to-face interaction) especially in teachingcomplex subjects that involve social aspects andthat are hard to teach without simulating” [2]. Suchgames, where the students play against or with eachother rather than against or with a computer, havebeen developed before, for instance, to illustratearchitectural technical debt [5], to teach softwareproject management [6], agile development [7], andprocess issues in software engineering [4]. To the bestof our knowledge, there are two games that, similarto ours, target architecture design reasoning but havevery different learning objectives: Schriek et al [8]
1https://goo.gl/3fVEey
specifically focus on design reasoning and their cardsacts as triggers for the students to explore the designspace by applying the reasoning techniques suggestedby the selected card; Cervantes et al [9] focus onteaching the architecture design process by usingthe Attribute-Driven Design method [10]. Whiletheir cards are much more sophisticated than ours(including technologies, tactics and patterns), theyare specific to big data systems. Also, the authorsaim to lead the players to make “smart decisions”.We rather do not enter the merit of the qualityof the decisions themselves, but focus on makingstudents appreciate the dynamics of architecturedesign, which entails collaboration and consensus,reasoning and among alternative options.
It must be noted that the literature alreadyreports on courses designed to at least teachsome aspects of such dynamics. Lago and vanVliet [11], for instance, focused on aspects ofsoftware architecture where communication is key,like defining views that frame stakeholder concerns,trade-off analysis and architecture assessment. DeBoer et al. [12] emphasized collaboration andbuilt a course that would let the students act asarchitects and experience the multiple and oftenconflicting variables influencing architecture designdecisions. The game we present here, DecidArch,differs by zooming into one single course topic,namely architecture design decision making, andleveraging the examples cited above by emphasizingcollaboration (the students play our game as teamsof architects) and bringing together various skillsand competences (by applying theory into practice).
3. Study Design
DecidArch is a game to be played by studentsin the context of our software architecture course.There, the students attend the traditional lecturesexplaining the theory related to architecture designdecision making; in parallel they work in teams on apractical project proposed by one of our industrialpartners. At the mid-point of their project, weask them to play our game. In this way, theyalready have some practical experience in SoftwareArchitecture Design Decision Making (SADDM), sothat they can transfer better what they learn duringthe game onto their own project.
Our study addressed the following main ResearchQuestion (RQ): Can we teach SADDM through a
card game?. To answer this main research question,we defined four sub-questions, which we addressed infour subsequent study phases as illustrated in Fig. 3:
Phases 1-3 focused on building the game, and Phase4 on applying the game in our software architecturecourse. In particular:
Figure 1. Study design: activities and research questions.
Phase-1 By studying the literature, we identifiedtopics, a.k.a. knowledge elements [13], thatare needed in SADDM and that we wantto include in the game. The correspondingresearch sub-question is RQ1 - What topicsare included in the reasoning process whenmaking architecture design decisions?
Phase-2 By using the results of Phase-1, wedesigned a first prototype of the gameaddressing two research sub-questions: RQ2 -What are the target learning objectives? andRQ3 - How will we design the card game?We used the literature and related works toidentify our learning objectives.
Phase-3 We addressed RQ3 by evaluating ourgame in a pilot where we asked practicingsoftware architects to play the game prototypeand to provide feedback on the learningobjectives. We used these pilot results toimprove the game and create a revised versionof DecidArch.
Phase-4 Finally, we used the improved versionof DecidArch to address our last researchsub-question: RQ4 - How do we evaluatethe game on its ability to teach the learningobjectives? We conducted an experimentwith the Master students of the softwarearchitecture course at our university. Asa result of Phase-4, we were able to drawconclusions on our main research question, RQ.
4. Study Execution
4.1. Phase-1: The Selected SADDMTopics
The literature provides a quite rich and maturebody of knowledge in SADDM, which informedour answer to RQ1. As our game is targeting
novice software architects (i.e. university students)in time-bounded sessions, it must be simple andself-contained, yet cover the dynamics and key topicsat the basis of design reasoning. Therefore, startingfrom the ISO/IEC/IEEE 40201:2011 standard [14]on Software architecture representation, and theseminal work of Babar et al. [15], we adopted thetopics illustrated in Fig. 2. In particular:
Figure 2. SADDM topics and their mapping on gamecards
Purpose. As pointed out by Tang & van Vliet [16],a system has business goals, and a higher-levelpurpose that helps framing the context of thedecisions by stating what the system is for.In [14], the system purpose is a key stakeholderconcern. In our game, we mapped it on aspecific type of card (Project Card) because itprovides the overarching context for the wholegame session.
Stakeholders. Depending on their specificconcerns, each stakeholder can be concernedwith a different list of Quality Attributes(QAs), or they may assign different prioritiesto different QAs.
Functional requirements. Among the variousdesign concerns influencing the design decisionmaking, we need to provide functionalrequirements to reason about the behavior ofthe designed system, and its quality. They“state what the system must do, and how itmust behave or react to runtime stimuli” [17].
Quality Attribute requirements. A QA is a“measurable or testable property of a systemthat is used to indicate how well the system[will] satisfy the needs of its stakeholders” [17].QA requirements, in turn, state the conditionsthat a system is expected to satisfy in termsof quality concerns [18, 19].
Design options. Given a system stakeholders anddesign concerns, SADDM is the processof making design decisions and justifyingthem [16,20]: each design decision is chosenamong alternative available design options,as based on a certain rationale (or designreasoning), which involves quality trade-offs(as each design option offers a viable solutionbut yields different quality properties, i.e.strengths and weaknesses). The incrementalnature of this process is captured in Fig. 2 bythe cyclic relations between Design concernand Design decision: a concern is addressedby a certain decision, which when made, canraise new concerns. Further, by lookingat software architecture as a set of designdecisions [21], each decision also depends uponother decisions.
4.2. Phase-2: The Card GamePrototype
The literature study2 carried out in Phase-1allowed us to gain synthetic understanding of the keyfactors in SADDM. Based on such understanding,we formulated the game learning objectives anddesigned the prototype of the card game that realizesthem. Similarly, we also included a gaming objectiveaddressing the user experience (UX). Below wedescribe first the learning/gaming objectives and thegame prototype, and then how the game elementsaddress the objectives.
4.2.1. The Learning Objectives. Weformulated the following target learning objectivesthat also provide our answer to RQ2:
• LO1-Reasoning: Create awareness about the
rationale involved in making trade-offs and
choosing design options among alternatives.
Explicitly or implicitly, design decisions arechosen among alternative design options. Thischoice depends on the rationale behind thedecision. The literature shows that designspace exploration and making explicit thereasoning and rationale helps in makingbetter-informed design decisions [20,22]. Asa valuable co-product, it also offers softwarearchitecture documentation for future change.Accordingly, the game should make thestudents aware of the topics playing a role
2We performed a simple literature review starting frompilot studies suggested by three experts followed by a roundof snowballing.
in design reasoning, and that the underlyingrationale involves making trade-offs whenchoosing design options among alternatives.This should improve their competence indesign reasoning.
• LO2-Differences: Enable the appreciation
of the design decisions proposed by others.
Typically, SADDM is a collaborative effort.However, different architects may makedifferent design decisions due to for examplebiases, differences in experience, technicalcompetence or even design reasoning.Similarly, because of such differences, thestudents can have different ideas that resultin different design decisions, too. If ourgame makes the students aware of thesedifferences, and makes them appreciate thatdiversity contributes to a high-quality softwarearchitecture design [23], this should improvetheir competences in collaborating as a teamby discussing their design ideas.
• LO3-Reconsideration: Create awareness about
the dynamics of software architecture design.
Design decisions may change and this mayresult in inconsistencies with other designdecisions they depend upon. Therefore uponchanging a decision, the architect shouldreconsider the other design decisions to checkif inconsistencies have arisen, and resolve them.The later such inconsistencies are addressed,the more expensive is their resolution. If ourgame makes the students aware that changingdesign decisions may result in changes toother decisions, this should improve theircompetence in design reasoning.
4.2.2. The Gaming Objective. From a purelyUX perspective, the game should be fun to play[24,25] so that the players are engaged (essentialfor teamwork) yet challenged (i.e. the game shouldbe easy to understand and easy to play so that theattention of the payers is fully on decision making).Also, they should be rewarded for the effort, i.e. bescored on the quality of the produced architecturedesign decisions. We called this objective Playability.
4.2.3. How the game works. We designedDecidArch with a deck of playing cards, which coverthe topics identified in Section 4.1. Examples ofplaying cards types in the game are shown in Fig. 3.
The Project card describes the name and purpose
CONCERN CARD Concern-ID: 1
CONCERNThe users need to be able to upload articleson the platform. How will the system store
these articles?
1 Store the articles in multiple distributeddatabases owned by the Owner.
Security (+), Performance (+), Availability (+), Maintainability (-), ...
2 Upload the articles in the cloud owned byan external cloud provider.
Security (-), Performance (+), Availability (+), Maintainability (++), ...
3 Store the articles in a single local databaseowned by the Owner.
Security (+), Performance (-), Availability (-), Maintainability (+), ...
OPTIONS
STAKEHOLDER CARD
PROJECT Social News Platform
STAKEHOLDER UserGOAL
The User will read news articles that arewritten or shared by other users. Thosewho write and share articles are oftenly
journalists or activists.
QUALITY ATTRIBUTES
Usability - The system should be easy touse. (QA-Priority: +) Security - The system should protect theinformation of the user from unauthorizedaccess. (QA-Priority: +)
PROJECT CARD
PROJECT Social News Platform
PURPOSE
The social news platform is adigital system that allows usersto write, share and read news
articles.
EVENT CARD
TITLE:
Change in data protection regulation.
DESCRIPTION
Due to a change in data protectionregulation, recommendationsregarding data storage has changed.
Use of 3rd party cloud storageproviders to store sensitive personalinformation impacts Security with (--).
Figure 3. Examples of cards.
of the system whose software architecture isthe game focus. This provides the decisionmaking process its context for a completegame session.
A Stakeholder card describes the role of acertain stakeholder in terms of his or her rolename, goal, and the list of QAs the stakeholderis concerned with. Each QA in the list isdescribed by a short definition and the levelof importance (or QA-priority) for the specificstakeholder.
A Concern card describes a concern that needsto be addressed with a design option. Itcan be a functional requirement or a QArequirement. The card also includes aninitial list of alternative design options; eachoption is illustrated by a short descriptionand an associated simplified representation ofits quality impact, i.e. the list of QAs eachwith a logical value from “++”, “+”, to “−”and “−−”, which, respectively, indicate if thedesign option would contribute to that QAvery positively, positively, negatively or verynegatively. In addition, a neutral value “0”stands for “no significant impact”.
Concern cards are the key game design element:when one is drawn, the players must addressthat concern, and make a design decisionby choosing one among the possible designoptions. Such a choice entails a line ofreasoning: using the quality impacts of eachalternative, therefore making trade-offs.
Event cards describe events especially designed todisrupt the current set of design decisions, andput the players in a new, changed situationwhere the design decisions made so far mustbe reconsidered and some of them possiblychanged. Event cards are introduced toaddress LO3-Reconsideration.
The playing cards have been populated withthe contents from the student projects proposed byour industrial partners in the software architectureMaster course at the Vrije Universiteit Amsterdam,editions 2015 (9 project reports) and 2016 (14project reports). For details about how the datahas been extracted, we refer the reader to the onlinematerial [26].
Further, Fig. 4 shows how the game is played. Inthe Preparation phase (Fig. 4.(a)), the players sitat the table as a team of software architects thatcollaborate to design a software system architecturefor a project and its stakeholders. The game sessionis set up by a Project card and the Stakeholdercards corresponding to that project3. In addition,the players prepare two shuffled stacks of cards (facedown), one of Concern cards and one of Event cards.
Finally, each player gets a Decision Preparation
Template where, for each design concern drawnduring the game, he or she will incrementally recordthe design option s/he would personally choose toaddress that concern. Each team also gets a sharedDecision Taking Template where, for each designconcern, the design option decided upon by the teamof architects is recorded. In the respective templates,the players (and the team) also briefly indicate therationale for the proposed (resp. decided-upon)design decision.
As illustrated in Fig. 4.(b), playing the gameconsists of multiple rounds. For each round eachplayer, in turn, (i) draws a Concern card; (ii)individually records his/her suggested option for thatconcern along with his/her rationale; (iii) followedby a team discussion of all suggestions, rationalesand trade-offs; and (iv) a team decision recordedon the Decision Taking Template. At each turn the
3The number of Stakeholder cards may vary depending onthe drawn Project card.
players need to consider during their SADDM thedecisions made until that point. When a round iscompleted (i.e. all players drew a concern card),(v) an Event card is drawn: this card describessome change for the project. Depending on thechange, the team has to collaboratively reconsiderthe addressed Concern cards to make sure that thechange is properly covered. The changes due to thedrawn Event cards last for the entire game.
The team of players carries out a number ofrounds. The game finishes when either (i) allConcern cards have been addressed, or (ii) theallotted time limit (e.g. 30 minutes) is expired.In both cases, the players get a last chance to revisetheir set of design decisions, given the list of concernsand the list of events occurred during the game.
Each game has a reward: winning the game!In our case, we have been looking for a scoringmechanism that would be simple to calculate, andmake DecidArch more fun by rewarding the playersfor doing well, and at the same time by encouragingthem to produce the best possible design withinthe allotted time. The scoring elements we havebeen looking at, are: (i) address all concerns andevents, yet (ii) satisfy all project stakeholders, and(iii) produce a set of design decisions with the bestquality impact for the target Quality Attributes.The details of the scoring mechanism are providedin the online material [26].
4.2.4. How the Game addresses the
Learning/Gaming Objectives. The elementsof the game and the way it works are especiallyconceived to accommodate the target objectives, asexplained in the following.
• In a game session, the Project card andStakeholder cards provide the needed contextfor the players to carry out SADDM. The waygame rounds are organized should facilitatekey aspects of the reasoning process, henceLO1-Reasoning. In particular, the Concerncards with their list of design options prompteach player to consciously explore the designspace, reason about the associated qualityimpacts and make QA trade-offs (that alsoconsider the QA-priority in the Stakeholdercards). When players were not satisfied withthe suggested design options on the card, theycould also propose their own. This realizesthe SADDM process illustrated by the whiteelements in Fig. 2. In addition, the use of thetwo templates help make the design decision
Figure 4. (a) Game preparation and (b) How to play the game.
and rationale explicit at both individual andteam level.
• To address LO2-Differences, we separatedSADDM in two steps: whenever a Concerncard is drawn, first each player selects thedesign option s/he wants to propose andwrites (in the individual Decision PreparationTemplate) his/her rationale; then all proposalsare discussed in the team where the playerscan make their case for their own proposalbut also appreciate the reasoning of theother players for alternative options. In thisway, the collaborative nature of SADDMis enforced, as well as the appreciation ofdifferent perspectives and rationales. The useof a shared Decision Taking template alsofacilitate team building, hence helping theplayers overcome individual competitiveness.
• As change is a fact of life, effective designdecision making must include continuousreconsideration of past design decisions. Whilethe game already enforces changes due tothe incremental iterative decisions embeddedby multiple rounds and individual turns,these changes might not challenge pastdecisions enough. To ensure effectivelyaddressing LO3-Reconsideration, Eventcards introduce unforeseen changes placingplayers in disruptive situations where, e.g.,decisions become invalid, or new concernsimpact the whole design. Also, thanksto the templates (that enforce makingdecisions and rationales explicit), checkingpast decisions against new events becomeseasier. This creates further awareness of thesilent added-value of expliciting architecturalknowledge.
• To address Playability, we designed the cardsto be as self-explanatory as possible, and thegame rules to be short and easy to grasp.Also, we devised a mechanism to score thedecisions made by the team of players tobalance simplicity (e.g. easy scoring to satisfythe stakeholders’ needs and QA trade-offs) andsignificance (e.g. decisions’ consistency andsatisfaction of all stakeholders).
4.3. Phase-3: The Pilot
We piloted the game prototype with 7 softwarearchitects from the firm ArchiXL4. After a briefintroduction to the game, the architects split intwo teams and played. Thereafter, we carriedout a discussion session in which we presentedthe learning objectives and gathered the architects’feedback on the extent to which the gameaddressed these objectives, and any additionalcomments. For research purposes, the whole pilotwas video-recorded.
We collected valuable suggestions forimprovement, which we used to revise thegame. To address LO1-Reasoning, the participantssuggested to introduce the templates (absent in theprototype) that would help making explicit therationale behind a design decision and use it laterto justify it in the team discussion. Not to overloadthe game dynamics, simplicity was mentioned as key.To keep rationale short and simple, the templatesoffer a bounded space to write it down.
LO2-Differences was considered insufficientlycovered. To correct this weakness, the participantssuggested to introduce two different types of
templates that would put explicit emphasis onindividual contributions (e.g. for players to voice
4www.archixl.nl/en
their own design ideas in spite of the perceived “timepressure”) and teamwork.
Finally, in the discussion aroundLO3-Reconsideration the participants observed thatthe events foreseen in the prototype (representingincidents like a fire in a data center) had aninsufficient impact on the players’ past designdecisions, hence not putting enough pressure forreconsideration. To address this comment, weintroduced new Event cards especially engineeredto either influence the quality impact of the designoptions (e.g. a new privacy law that forces a higherlevel of security), or change the quality demands ofsome stakeholders.
5. Experiment Results
5.1. Experiment context
The experiment has been conducted as part of our2017-2018 Software Architecture master course. Thetarget audience for this course is computer scienceand information science students. Participation inthe experiment was compulsory for students enrolledin the course. The students received the gamemanual beforehand, and were asked to prepareby studying the game rules. At the start of theexperiment, instructors briefly reiterated the rules.It was made clear that, while participation wasmandatory, performance in the game would notimpact the course grade. Instructors were presentto clarify any questions that arose during the game.
A total of 83 master students, organized in 22groups, played the game. When asked about theirprofessional experience, 74 students indicated theypossessed some experience in software engineering- either as software developer, software engineerand/or software architect - while 19 indicated theyhad none.
5.2. Learning Objective 1: Reasoning
Fig. 5 shows the responses to survey statementsabout Learning Objective 1: create awareness
in the students about the rationale involved in
making trade-offs and choosing design options among
alternatives. The chart shows that, by and large,the game supports those aspects that we considerindicative for achieving this learning objective.
Almost all students indicated they reflectedon how a design option could impact stakeholderconcerns (statement S1.1), and that they had toconsider quality attribute trade-offs (statement S1.2).To a somewhat lesser extent, but still in majority,
students indicated they were motivated by thegame’s templates to document the reasons behindthe chosen design options (statement S1.3) and thatthe quality impacts on the game’s concern cardsmotivated them to explicitly reason about thesetrade-offs (statement S1.4).
Most noticeable are the relatively large numberof students who indicated they felt neutraltowards statement S1.3, as well as those who feltneutral towards or disagreeing with statement S1.4.Regarding the use of templates (statement S1.3),remarks from these students indicate that they wouldhave preferred less writing and more interaction.Regarding the quality impacts (statement S1.4),students remarked that they did not agree withthe impacts that were provided on the cards, thatthey “had their own beliefs”, or already knew enoughabout the design options provided.
Figure 5. Diverging stacked bar chart for surveystatements related to LO1-Reasoning
Is it interesting to note that the introduction ofthe templates had been prompted by the feedbackfrom the pilot. Whether or not the use oftemplates to structure discussions is valued moreby experienced architects than by novices is notsomething we’ve investigated further, but thereappears to be a difference in this respect betweenthe two player populations. Perhaps some furtherfine-tuning is needed to find a sweet spot betweenrecording in-game design decisions and discussingthem. It is well-known from knowledge managementstudies that ‘recording what you know’ (e.g. designrationale, code documentation, and knowledge ingeneral) is not something people tend to do withouta very clear reason to do so [27].
5.3. Learning Objective 2:Appreciating Other Decisions
Fig. 6 shows the responses to survey statementsabout Learning Objective 2: enable the appreciation
of the design decisions proposed by others. Theoverall impression from the diagram is that the gamesupports aspects relevant to LO2.
Almost all students indicated that, while playingthe game, they discussed design ideas with the otherplayers (statement S2.1). For most student teams,the differences between players in the team led todiscussions about different design ideas (statementS2.2). However, about half of the students wereneutral to or disagreeing with the statement that thedecision preparation template gave an opportunityto contribute new design ideas (statement S2.3).
The comments that students left for statementS2.3 point to three reasons for the lower scores: (1)the templates are good for traceability, but not somuch for creativity (for which direct discussion wasdeemed more suitable, cf. LO1); (2) the use of thetemplate is costly (write down your own idea, discuss,write down the consensus); and (3) some teams didnot feel the need to introduce any new design ideas,since the options provided on the cards were alreadysufficient.
Figure 6. Diverging stacked bar chart for surveystatements related to LO2-Difference
The idea behind the decision preparationtemplate had been that individual team memberscould, in relative quiet, first think about theirown position and arguments before ‘defending’ thatposition in a group discussion. Apparently, not allplayers did (or needed) that.
5.4. Learning Objective 3: Awarenessof the Dynamics of Design
Fig. 7 shows the responses to survey statementsabout Learning Objective 3: create awareness about
the dynamics of software architecture design. Thediagram shows that the game’s support for aspectsrelated to LO3 is not as high as for the other twolearning objectives. The root cause can probably befound in the responses to statement S3.3: “During
the game, we had to change past design decisions”.Responses to this statement were predominantlynegative. The dynamics we intended to happen inthe game did apparently not always occur.
Students remarked that they ‘got lucky’, ‘(byaccident) did everything right’, that ‘event cardsdid not have an impact’, that in the allotted timethey ‘reached only one event card’, or ‘none at all’,even that they ‘didn’t know [changing decisions] waspossible’. Some others, however, had to change ‘twoout of the four decisions we made before’ and otherseven had to ‘change completely all design decisions.
We conclude that the support for LO3 is presentin the game, but depends too much on chance.In a next iteration of the game, the impact andfrequency of events should be reconsidered. In short,players should run into enough events with enough(potential) impact so that the chance they have torevisit earlier design decisions increases.
Figure 7. Diverging stacked bar chart for surveystatements related to LO3-Reconsideration
5.5. Gaming Objective: Playability
Fig. 8 shows the responses to survey statementsabout the playability of the game. The majorityof students who played the game agree that thegame was fun (statement S4.1), that the cards wereeasy to understand (statement S4.2), the rules wereclear (statement S4.3) and the scoring sheet wasclear (statement S4.4). Overall, playability of thegame is sufficient although some remarks indicatethat the scoring sheet could be improved. Anadditional suggestion was to explicitly incorporatethe ‘consistency’-aspect (cf. LO3) in the scoring.
6. Threats to Validity
In this section we present potential threats tovalidity of our study results and our strategies tomitigate them.
Internal Validity refers to the ability to draw
Figure 8. Diverging stacked bar chart for surveystatements related to Playability
correct conclusions from the experiment andthe collected data. To avoid threats tointernal validity we designed a priori asurvey. The survey covered the envisionedlearning objectives through a list of statements(reported in Figs. 5-7). To avoid biasesintroduced by the adoption of the Likert scaleto evaluate the statements, each statement wasaccompanied by an open question, in whichparticipants could add additional informationon their scaled responses.
An additional threat to internal validityis resulting from the background of therespondents. As they are students enrolledin a university course, the answers couldbe biased due to their concerns regardingpotential relations between their answers andtheir academic performance. To mitigate thisthreat, before the survey was filled in, studentswere assured that their answers would notaffect in any way their course grade and itwould be purely used for research purposes.
External Validity refers to the ability togeneralize the results of the experiment. Inour case, we want to generalize the findingsto students learning software architecture.This threat, therefore, mainly relates to thepossible homogeneity of the respondents (allstudents of the Software Architecture courseat our university). However, this threatis mitigated by the multidisciplinary andmulticultural nature of our student population:about 50% is international, and the studentswere attending three different programs (CS,SE and IS) and eight different study tracks,ranging from Software Engineering and Green
IT and Business Information Systems toHigh Performance Computing and Theoretical
Computer Science. The students also differedin terms of professional experience, as reportedin Section 5.1.
Construct validity refers to potential threatsrelated to the extent to which the datacollected is suited to answer our researchquestions. A potential threat to constructvalidity lies in the data source we adopted. Infact, it is crucial that the survey respondentsare representative of the typical card gameusers. The demographics of the survey wereall students of a university course focusingon Software Architecture. As the teachingobjective is specific to this topic, we deemthe vast majority of respondents as beinginterested in the topic and hence representativeplayers.An additional threat to construct validity isconstituted by the environment in which thegame was conducted. In fact, the game wascarried out under the supervision of instructorswho explained the game and intervened ifclarifications were asked for. This couldpotentially lead to a different interactionbetween players playing with and withoutsupervision.
Conclusion validity refers to the ability todraw correct conclusions from analyzing thegathered data. A first threat to conclusionvalidity lies in the statements adopted toevaluate our learning objectives. In order tomitigate this threat, we defined the survey a
priory. The survey was then jointly discussedby all researchers. This was required in orderto ensure that the survey covered all aspects ofthe game that were needed to comprehensivelyanswer the research questions.Another potential threat lies in the qualityof the answers provided by the surveyrespondents. To mitigate this threat, eachLikert scale response was accompanied withan open answer. Regarding further threatsresulting from potential biased interpretationsof the survey results, the collected data wasanalyzed by all authors. When necessary,results were jointly discusses in order to alignthe interpretation of the data and provide anobjective analysis.
7. Conclusions
DecidArch is a board game that is inexpensiveand easy to introduce in the classroom to teach
university students about the concept of softwarearchitecture design decision making: examiningtradeoffs and compromises among stakeholders’demands, and major quality attributes, in the faceof some moderate uncertainty.
The answer to our main research question:Can we teach SADDM through a card game?
is a: yes. DecidArch well supports our threelearning objectives, although the support for learningobjective 3 (Reconsideration) currently dependstoo much on chance. Besides addressing thisaspect through higher impact and frequency of theevent cards, a future improvement we envisage forthe game is a more balanced use of the decisionpreparation templates. These changes do not changethe nature nor the design of the game, and are onlytweaking the game’s behavioral parameters.
The game DecidArch is available along with theexperiment replication data [26].
References
[1] Proc. 5th International Workshop on Games andSoftware Engineering (GAS), ACM, 2016.
[2] M. Kosa, M. Yilmaz, R. V. O’Connor, andP. M. Clarke, “Software engineering education andgames: A systematic literature review,” Journalof Universal Computer Science, vol. 22, no. 12,pp. 1558–1574, 2016.
[3] G. Taran, “Using games in software engineeringeducation to teach risk management,” in 20thConference on Software Engineering EducationTraining, pp. 211–220, 2007.
[4] A. Baker, E. Oh Navarro, and A. van der Hoek,“An experimental card game for teaching softwareengineering processes,” Journal of Systems andSoftware, vol. 75, no. 1-2, pp. 3–16, 2005.
[5] T. Grant, “Dice of debt game.” www.agilealliance.org/dice-of-debt-game. Accessed on: 2018-03-13.
[6] P. Kruchten and J. King, “Mission to Mars: Anagile release planning game,” in 24th Conferenceon Software Engineering Education and Training(CSEET), pp. 552–552, 2011.
[7] Agile42, “Kanban pizza game.” www.agile42.com/en/training/kanban-pizza-game. Accessed:2018-03-13.
[8] C. Schriek, J.-M. E. van der Werf, A. Tang, andF. Bex, “Software architecture design reasoning: Acard game to help novice designers,” in SoftwareArchitecture, LNCS, pp. 22–38, Springer, 2016.
[9] H. Cervantes, S. Haziyev, O. Hrytsay, andR. Kazman, “Smart decisions: An architecturaldesign game,” in Proceedings of the 38thInternational Conference on Software EngineeringCompanion (SEET Track), ICSE, pp. 327–335,ACM, 2016.
[10] R. Wojcik, F. Bachmann, L. Bass, P. Clements,P. Merson, R. Nord, and B. Wood,“Attribute-driven design (ADD), version 2.0,”tech. rep., 2006. CMU/SEI-2006-TR-023.
[11] P. Lago and H. van Vliet, “Teaching a courseon software architecture,” in 18th Conference onSoftware Engineering Education Training (CSEET),pp. 35–42, 2005.
[12] R. de Boer, R. Farenhorst, and H. van Vliet,“A community of learners approach to softwarearchitecture education,” in Conf. on SoftwareEngineering Education & Training, 2009.
[13] M. A. Babar and P. Lago, “Design decisionsand design rationale in software architecture,”Journal of Systems and Software, vol. 82, no. 8,pp. 1195–1197, 2009.
[14] International Organization for Standardization(ISO), “Systems and software engineering– Architecture description,” standard, 2011.ISO/IEC/IEEE 42010:2011.
[15] M. A. Babar, T. Dingsoyr, P. Lago, and H. van Vliet,eds., Software Architecture Knowledge Management- Theory and Practice. Springer, 2009.
[16] A. Tang and H. van Vliet, “Software architecturedesign reasoning,” in Software ArchitectureKnowledge Management: Theory and Practice,pp. 155–174, Springer, 2009.
[17] L. Bass, P. Clements, and R. Kazman, SoftwareArchitecture in Practice. SEI Series in SoftwareEngineering, Pearson Education, Inc., 3 ed., 2012.
[18] E. Hochmuller, “Towards the proper integrationof extra-functional requirements,” AustralasianJournal of Information Systems, vol. 6, no. 2, 1999.
[19] L. Chung and J. C. S. do Prado Leite,“On non-functional requirements in softwareengineering,” in Conceptual Modeling: Foundationsand Applications, pp. 363–379, Springer, 2009.
[20] M. Razavian, A. Tang, R. Capilla, and P. Lago,“In two minds: how reflections influence softwaredesign thinking,” Journal of Software: Evolutionand Process, vol. 28, no. 6, pp. 394–426, 2016.
[21] A. Jansen and J. Bosch, “Software architecture asa set of architectural design decisions,” in WorkingIEEE/IFIP Conf. on Software Architecture, 2005.
[22] A. Tang and M. F. Lau, “Software architecturereview by association,” Journal of systems andsoftware, vol. 88, no. 2, pp. 87–101, 2014.
[23] M. Razavian and P. Lago, “Feminine expertise inarchitecting teams,” IEEE Software, vol. 33, no. 4,pp. 64–71, 2015.
[24] A. Iosup and D. Epema, “An experience report onusing gamification in technical higher education,”in Technical Symposium on Computer ScienceEducation, SIGCSE, pp. 27–32, ACM, 2014.
[25] M.-B. Ibáñez, A. Di-Serio, and C. Delgado-Kloos,“Gamification for engaging computer sciencestudents in learning activities: A case study,” IEEETransactions on Learning Technologies, vol. 7, no. 3,pp. 291–301, 2014.
[26] J. F. Cai, P. Lago, R. C. de Boer, P. Kruchten, andR. Verdecchia, “DecidArch Replication Package,”https://github.com/DecidArch/HICCS2018.
[27] J. F. Hoorn, R. Farenhorst, P. Lago, and H. vanVliet, “The lonesome architect,” The Journal ofsystems and software, vol. 84, no. 9, 2011.