Abstract Reasoning in Collaborative Modeling

Ilona Wilmont Erik Barendsen Stijn Hoppenbrouwers

Institute for Computing and Information Sciences Radboud University Nijmegen

Nijmegen, The Netherlands {i.wilmont, e.barendsen, s.hoppenbrouwers}@cs.ru.nl

Sytse Hengeveld

CCV Holland B.V. Arnhem, The Netherlands

s.hengeveld@nl.ccv.eu

Abstract This paper reports on a case study of abstract

reasoning in a real collaborative modeling setting. The study was conducted according to the behavioral observation principles of human ethology. Our findings indicate a relation between an individual’s executive functioning and his ability to do abstract reasoning. Furthermore, we find individual differences in these abilities, and our results suggest that lack of certain skills pushes a session back to its initial phase until a shared conception of what is being modeled is achieved. These findings further our understanding of the process of collaborative modeling; how the qualities and behaviors of an individual modeler influence the interactive modeling process and its final outcomes. 1. Introduction Collaborative modeling is a complex task, requiring a variety of skills. Our general aim is to understand these skills and their mutual dependencies, in order to design appropriate tools and methods to facilitate modelers. The present study is part of this larger research project. We focus on abstraction and executive functioning. Abstract reasoning is vitally important in formal modeling of any kind, whether it is done individually or in collaboration with peer modelers. We view abstractions as inherent to modeling. A model provides a precise and unambiguous description of a designated part of reality, written in a certain language, which can be formal or natural. The use of any language implies abstraction. Also, models have a certain level of abstraction determined by the goal of the model and the scope of the domain it describes. Many aspects associated with successful collaborative modeling performance can be related to an individual’s executive functioning. Executive control functions are defined as “control functions that are responsible for directing and regulating

cognitive as well as social behavior” [25]. In the case of collaborative modeling, these functions appear in the way interaction is monitored for correctness, goal-relatedness and mutual understanding. In order to distinguish phases in the modeling process, we view the construction of a model as a special case of problem solving. We will first state our research questions, before elaborating on the necessary background information.

1. How do transitions between abstraction levels proceed in interaction for modeling?

2. How are abstract reasoning, executive functioning, and the process of problem solving related in context of collaborative modeling?

3. How are individual differences in abstract reasoning expressed during collaborative modeling?

With regard to question 1, evidence from neuroscience shows that the human brain can indeed deal with at least three levels of abstraction [5,6]. We need to find out how these relate to abstraction as used in practice. Transitions between levels of abstraction should follow logically out of interaction, during which certain situations are highly likely to lead to shifts in abstraction level. For instance, a failure to comprehend a formed abstraction in most cases evokes an illustration at a concrete (‘instance’) level. Regarding question 2, we expect executive functioning to play a key role in abstract reasoning, and also that abstract reasoning is necessary for successful modeling performance. Question 3 investigates more closely the individual differences in abstract reasoning capacities, and the associated behaviors exhibited. Neuroscientific findings [5,6] indicate that cognitive processing power increases as the abstraction level increases.

We expect that some modelers are more capable than others of forming and comprehending abstractions, and that those modelers who are more capable of understanding and forming abstractions are also more capable of monitoring whether the abstractions lead to the desired goal. However, this does not imply that these differences are static. Abstraction appears to be trainable to a certain extent, but we still expect individual differences to remain. Apart from exploring the relations between abstraction and cognitive skills in modeling, the present study serves as a pilot in the development of a research instrument for qualitative analysis of collaborative modeling sessions. This paper is organized as follows. We develop a conceptual framework for our analysis. In order to show that our classifications make sense, we refer to underlying evidence from neuroscience and cognitive psychology. Then we continue with a description of our case study and the results we have obtained. We conclude with suggestions and plans for future research. 2. Conceptual Framework Collaborative modeling happens by making abstractions. Modelers are trying to define a system, and create a description for it. A description is abstract by definition because of the abstraction inherent to the use of language. We see that things go wrong when formation of abstractions fails: failure to understand, erroneous interpretation, or several modelers continuing along different lines of reasoning. We will illustrate this with examples from our observations. The ability to monitor both formation and flow of abstractions, and also the concepts and relations used in forming abstractions, seems to be one of the key skills to good modeling performance. It is through this perspective that we construct our framework of the cognitive processes facilitating abstract reasoning. Monitoring is an important skill being studied in the fields of metacognition, problem solving and executive functioning. Each of these areas of research provides valuable insights for understanding abstract reasoning. First we discuss different theories of abstraction, and how different levels of abstraction are represented in the brain. This three-level representation serves as a basis for our conceptualization of abstraction. Then we examine how abstractions are facilitated

behaviorally. For this, we apply Barkley’s model of executive functioning [1], which includes the role of working memory, selective attention and metacognitive behaviors, to the mechanism of abstraction. Finally, we discuss how a perspective of problem solving can help provide structure to a collaborative modeling session to facilitate analysis. 2.1. Abstraction The essence of an abstraction in collaborative modeling is to omit information irrelevant to the situation at hand, but to never lose the link with the concrete situation. This is also known as information hiding [7]. When this occurs, relations are drawn between different concrete concepts: concepts with strong imagery that people can immediately form associations with and provide examples of. These relations often specify a generalization. They are based on some attribute the concepts have in common. Generalization relations can occur on different levels of abstraction complexity, depending on how many earlier abstractions are included in the relation. Some abstraction (information hiding) is involved in every utterance, since one never describes, for instance, each and every letter that is typed in a form, what texts are read etc., but this level of abstraction is so basic and natural to the use of language itself that we do not consider it here. The earliest theories of abstraction emerged in the field of philosophy, as early as the time of Plato, but George Berkeley (1685 - 1753) was the first to propose a notion that still remains central in many theories of abstraction today. Berkeley argued that abstraction occurred through a "shift in attention"; it is possible to focus on a particular feature of a single object, and let that feature represent a whole group of objects [3]. This notion is known as selective attention. Abstraction has recently entered the field of cognitive neuroscience and psychology, which has led to ideas such as attention, perception and neural connectivity being included in theories of abstraction. Barsalou has proposed a three-stage theory of abstraction in which connections between concrete and abstract concepts are “direct and nonmetaphorical” [3]. The first stage places that which is perceived in context, by means of direct association with objects or actions associated with the concept. Secondly, selective attention selects the features that are relevant to forming an understanding of the concept, but the emphasized features in one's personal understanding are based on prior beliefs about the concept. In the third stage, introspective perceptual systems are employed to interpret the

selective concept attributes. This includes emotional states and cognitive operations such as search and comparison; these are unique to the individual systems of perceptual interpretation [3]. Barsalou's three stages of developing an abstract representation of a concept correspond well to how abstraction is represented in the brain [5]. Christoff & Keramatian [5] examine brain activation in rule-guided behavior, and provide evidence that the human brain responds to three different levels of abstraction: concrete instances, half-abstract concepts and highly abstract concepts. We classified abstractions occurring in observed modeling interactions according to these levels. The objects used by [5] among which to define relations were 2-dimensional circles with a black side and a white side, and words with a certain level of abstraction as defined by Paivio [20]: Concrete: the orientation of the black side of the circle is {up, left, down, right}. Examples of words for the concrete condition are {desk, bottle, motor}. First-order abstraction: the orientation of a pair of circles is {same, different}. Words used in this condition were {trip, dance, symbol}. Second-order abstraction: two pairs of circles were {related (same-same, different-different), unrelated (same-different, different-same)}. Some words in this condition were {myth, appeal, grace}. Reaction and processing time are observed to increase with increasing abstraction, but Christoff & Keramatian [5] have eliminated the possibility that this increase is due to the influence of increasing task difficulty by controlling for the latter in a consecutive study. They observed no difference in reaction time and accuracy between different conditions, but they did find that each prefrontal area was activated consistently according to its corresponding level of abstraction [5,6]. This implies that abstraction level rather than the degree of complexity controls which brain area is activated. 2.2. Executive Functioning and Metacognition Abstraction involves different basic cognitive processes, which have also been associated with prefrontal cortex activity [8,16,24,25]. The prefrontal cortex generally has been found to participate in “higher aspects of motor control and the planning and execution of behavior, tasks that require the

integration of information over time” [12], such as executive functioning and metacognitive monitoring. Metacognition can be seen as knowledge of an individual’s own cognitive processes and behaviors, the ability to monitor and control these mentally, and to act accordingly in the individual’s overt behavior. Based on the many definitions found in the literature, consensus has generally been achieved that monitoring and control are the most important aspects of metacognition [10,11,16,4]. The first researchers to write about executive functioning [17,18] focused mostly on the planning and regulatory aspects of executive function. Later theories, such as Barkley’s model [1], introduced the importance of working memory in executive function. According to [1], executive control functions are heavily dependent on both verbal and non-verbal working memory capacity. Barkley furthermore defines self-regulation of mood, motivation and level of arousal, as well as problem solving as essential for executive functioning. In his model, non-verbal and verbal working memory interact intensely to facilitate self-regulation, which is in turn essential for flexible problem solving to take place. Ylvisaker et al. [25] continue with a more operational definition of what executive functions should comprise, many of which are essential skills for modeling: knowing what is easy and what is difficult, goal setting, planning behavior to achieve the goal, initiating behavior towards achievement, inhibiting interfering behavior, monitoring behavior, strategic thinking and flexible problem solving performance. Many aspects of this definition can be observed first-hand in collaborative modeling sessions. In practical tasks, selectivity of attention, or what to attend to, is typically determined by the task performer himself, based on his personal intentions and the goal he desires to achieve. Most practical tasks involve divided attention as opposed to directed attention, because there are always multiple distracting stimuli in a task environment. This also holds for collaborative modeling, in which distracters take the form of other modeling participants and multiple sources of information. This top down control of attention heavily involves working memory and strategy, both executive aspects of control. In particular, the quality of divided attention depends on two factors: processing capacity and control processes, whereby processing capacity is largely determined by the speed of information

processing [23]. Goal pursuit and active information seeking are both proposed to guide selective attention [14,22]. Barsalou argues that once selective attention has focused on a particular feature multiple times, a perceptual representation or symbol will develop for it, and then selective attention will keep focusing on that particular aspect as formed through experience [3]. Whereas we do see that attention is strongly guided by goals, it is our belief that individual differences in comprehension ability and executive functioning determine which particular aspects attention focuses on within those goals. 2.3. Problem Solving Problem solving literature has focused on defining phases of problem solving, during which different aspects of behavior are emphasized corresponding to the solution phase the problem solver is in. While different authors name the phases differently, the main flow boils down to the solver first forming a mental representation of the problem. When this is clear he moves on to developing a solution strategy, and finally evaluates the solution [15,21]. This process can be applied to a modeling session, during which we see modelers going through similar phases. Also, the cognitive processes discussed above can be observed within these phases. 3. Method The case study is part of a larger research project on collaborative modeling that was conducted at a Dutch organization. The organization is currently involved in charting its current business processes and designing new ones in order to develop a new automated information system. The techniques used for this are collaborative modeling workshops and analyst modeling sessions involving the following stakeholder roles: project manager, business analyst, business architect, change manager, 2 heads of departments, 2 supervising seniors, internal auditor. The minimum group size in our study was two. 3.1. Data collection One researcher has spent three months at the company, being present at relevant sessions, and recording them in audio format initially, but as the stakeholders became more accustomed to the researcher’s presence, a video camera was installed in the workshop room and video recordings were made in addition to audio. The stakeholders indicated not to be bothered by its presence. Additional time was spent getting to know the stakeholders, but care

was taken not to talk about the research objectives to avoid introducing research bias. The modeling sessions and stakeholder workshops all took place in the same project room, which was equipped with a beamer and two flip chart boards. The models under discussion had been printed and were attached to the walls. During the stakeholder workshops, the modelers presented the models to the stakeholders and these were required to respond to certain issues or things that appeared odd to them. In some cases, bits of model were explicitly shown, in other cases, issues were formulated in natural language. During the analyst-only modeling sessions, heavy use was made of the flip charts, and interaction was not explicitly structured. Models were adapted and contradictory issues discussed. Our case study covers three collaborative modeling sessions, two of which were ‘business analysts only’, and one involved stakeholders. The sessions with the stakeholders were meant to gather information about the business processes in practice, and the ‘business analysts only’ sessions aimed to develop new processes and integrate the existing ones. We made sure to select modeling sessions that were ‘meaningful’, in the sense that the discussions remained relevant to the modeling goal(s), progress was made towards the goal(s), and abstractions were used to manage model complexity and concept redundancy throughout the session. Note that we specify meaningfulness in terms of the modeling process rather than the modeling result. 3.2. Coding Based on the theoretical considerations in Section 2, we developed an initial codebook with codes classifying abstraction levels, executive functions and problem solving. The initial codebook was tested and modified by coding random 15-minute samples of the recordings, checking completeness w.r.t. the observed interaction, mutual exclusion, and agreement between the researchers. The resulting classification is described below. 3.2.1. Abstraction We classified abstractions occurring in the modeling interactions based on the levels described by Christoff & Keramatian [5].

• Concrete In modeling interactions, concrete

descriptions are those of daily activities from the stakeholders’ practices relevant to the process being modeled. Instance-level objects and actions are used when describing such scenarios.

• First-order Abstractions of this kind appear in modeling sessions in the form of the ability to recognize patterns between concrete objects and actions, such as similarities, differences, types etc.

• Second-order Second-order abstractions concern the recognition of relations among the patterns identified in first order abstract relations, which we could call ‘patterns of patterns’, allowing for a further generalization in representation.

3.2.2. Executive Functions In order to code executive functioning behavior, the operational definition by [25] proved to be the most useful. The codes we used were:

• Goal setting • Planning behavior to achieve the goal • Initiating behavior towards achievement • Inhibiting interfering behavior • Monitoring behavior

We do assign an interpretation to inhibiting interfering behavior and monitoring behavior, since in collaborative modeling, participants do not only monitor their own behavior, but also, if possible, the behavior of other modelers. We take knowing what is easy and what is difficult, strategic thinking and flexible problem solving to follow directly from the behaviors we coded, since these three are not immediately observable in the individually transcribed interactions. 3.2.3. Problem Solving In order to get a complete idea of how abstractions develop during collaborative modeling interactions we view the whole session as a problem solving process. We base the phases we observe on those as defined by [21], with minor modifications. These are:

• Recognize or identify the problem • Organize knowledge about the problem • Define and represent the problem mentally • Develop a solution strategy

• Allocate mental and physical resources for solving the problem

• Formulate a possible solution • Evaluate the solution for accuracy

For this study we did a qualitative analysis, focusing on the suitability of our initial codebook to address our questions. We interpreted the transcripts with three coders according to the indicators specified in our codebook. We initially coded individually, and afterwards came together to compare our coding results and discuss the adequacy of the codebook. Throughout the process, we used memos to keep track of the context of the interaction. 4. Results Our analysis shows that there are several frequently recurring patterns of abstract reasoning behavior to be discerned from the collaborative modeling interactions. 4.1. Relations Between Abstraction Levels The three different levels of abstraction are recognizable and observable. First-order abstractions and concrete scenarios are clearly dominant in the sessions we analyzed, and there is much interaction between these two levels, largely verification-oriented. It seems that abstractions are built on each other; that first order abstractions can only be made after the concrete representation is clear. We provide an example below (translated from Dutch to English): M1: a reminder, a note [concrete] M2: yes, it could be that you’ll make a call [inaudible] eh… to the employers…. [concrete] M1: yes but a reminder…. guarding is checking whether… there has been a delivery... [identifies discrepancy in scope; monitoring; first-order abstraction to define scope] M2: yes. M1: and…. reminding is a note, or a phone call…. Or whatever….. [continues to define scope with concrete example] M2: yes…. Guarding can also be….. okay, we have only 4 out of 5 expected deliveries.... [redefines first-order abstraction with concrete example] …..

M1: yes both…. Both from ‘signal’ and to ‘receive delivery’ goes to ‘guarding’ …. that is the only way to put together that list…. [second order abstraction, relation between abstract concept ‘signal’ and ‘guarding’]. We tried to observe how abstraction steps are made. This seems to be largely an individual process, with one modeler proposing an abstraction step and then collaborative testing with concrete examples and counter-examples, rather than a collaborative discussion in which both participants contribute to coming to an abstraction. In this case, we suspect this might be due to individual modeler capacities, but we need more data to verify this. When a walk-through of the process being modeled was done, the utterances were related to concrete mental representations the modelers had of the actions and objects associated with the process. When abstractions were being made they were mostly of the first order. These appear to be two main types of abstraction: content related abstractions, which make generalizations over the actual content of the process, and model syntax related abstractions, which group process concepts into modeling concepts such as activities, triggers and decisions. The observations suggest that model syntax related abstractions were easier to make than content related abstractions for all modelers observed. Only when scope and definition of what is being discussed are clear on a concrete level, can an abstraction be made and can a relation between two abstract concepts be drawn. As long as this is not clear, there appears to be no intention to make the abstraction or draw the relation and the modelers automatically deviate towards testing with scenarios. This does not appear to be a conscious process, an interpretation confirmed by [13]. 4.2. Abstraction and Executive Functioning We observed that the problem solving phases appeared to correspond to the levels of abstraction being used and the modelers’ comprehension and executive functioning. Each problem solving phase seems to involve specific levels of abstraction to achieve the goal of the phase. Participants move back and forth between phases, constantly being monitored and adjusted accordingly. In general, we see a cycle of problem solving that we might categorize as 'explore the problem, formulate a solution, evaluate the solution', where exploration of the problem is being done using concrete examples and counter-examples in an attempt to falsify the situation. Then,

based on this knowledge, a tentative solution is formulated, in which first order abstractions are being made to create structure in the solution, and even second order ones, but in this particular case this failed because one of the participants could not achieve sufficient comprehension to reason at this level. The solution is then evaluated, first by using first-order abstractions but also with concrete instances if it is perceived that a full shared understanding has not been achieved. An example of going back to the ‘definition of problem’ phase after ‘evaluating the solution’ follows below: M2: ehmmmm…. Let me see….. yes well making everything digital… yes…. Feedback notes and everything, then I think we have enough but…. Eh… shall we put this into our tool? This way? [evaluate solution based on the diagram, concrete] M1: what if….. eh…. Goal of the process is to register the details about the wages…..[first-order abstraction, monitoring behavior pertaining to goal, initiating behavior towards achievement, recognize problem, has to make it clear to peer] M2: yes… M1: in any case.. eh…. Eh…. Preferably on time, complete and correct, but in any case complete and correct….. [first-order abstraction, represent problem] M2: yes…. M1: what if we eh…. Guarding….. huh…. We send a reminder, hey good friend, eh…. Eh…. You have not sent us anything yet…. [concrete illustration, represent problem, organize knowledge] M2: yes… M1: we get no reply…. [continues with concrete illustration in phase of representing problem and organizing knowledge] M2: yes.. M1: what happens then? [asks for scenario test, does not specify abstraction level, continues to monitor progress] M2: there is no reply, then we receive nothing…[concrete filling in of representation] M1: right, then we receive nothing [monitoring] M2: we can’t process anything [concrete illustration]

M2: we can’t process anything, so as a final result you have no complete and correct information about the wages of that employee [monitoring, and concrete mention of the consequences] M1: yes… M2: what happens then? [again asks for scenario test, after having clarified point with concrete illustration] M1: then you do not achieve the goal, in fact? [problem defined and understood by peer] M2: yes.. [monitoring] With regard to problem solving, in the ‘identify, define and represent the problem’ phases, as well as when ‘organizing one’s knowledge’, mostly scenarios were used to illustrate what was going on: reasoning at the concrete level. Some first-order abstractions, mostly model syntax related ones, were made to clarify what belonged with what and to identify scope. When moving towards a ‘solution strategy and monitoring progress’, modelers became more liberal in using first and even second-order abstractions, but as soon as the monitoring pinpointed a discrepancy in what phase each participating modeler was in, they returned to the ‘define and represent problem’ phase with its characteristic concrete scenario-based illustrations. An example of our memos on the interaction during the stakeholder modeling session: 52:45 a sticker session is started, involving 4 stakeholders, 1 analyst and 1 project leader. The stickers with requirements on the brown paper are all concrete instances of what the stakeholders encounter in their daily practice. They reason about them by testing them with concrete scenarios. 57:30 a stakeholder has mentioned that a rule should be put on a sticker. She does not know exactly how to formulate it. The 2 analysts help her with this first order abstraction, and in the end the analyst guards the scope and it gets definitive form. 1:00:14 a requirement is mentioned: readable, no holes, no staples. Concrete. 1:05:45 a stakeholder again tests a scenario. The stakeholders discuss it, and they remain mostly on the concrete level. The analyst makes a minor abstraction, one stakeholder catches on, but then another responds with a concrete notion. The first one sketches a first order abstraction scenario (can supply wrong details) and gets a

questioning look. It is clarified with a concrete instance (filling in a 1 instead of a 10). 1:14:21 the analyst tries an abstract 1st order conclusion. The stakeholders merely agree, this implies that they understand it. He continues to define scope. One stakeholder understands it and pushes to start describing process steps. Then the project leader mentions they were gathering input, on which the stakeholder says that they have been gathering input for ages. Mismatch in project scope and focus of the session. 1:34:49 ‘correctness’ is questioned by one of the stakeholders: what is it? Another one cuts in by saying that that is too much detail to elaborate. However, the same stakeholder has no trouble defining 'authorisation'; apparently she has a clear conception from her experience what authorisation is. As for the criterion of correctness: they all seem to know what ‘correct’ is, but when something is sufficiently 'correct' seems to be harder to specify. The most prominently observed pattern of behavior seems to be a cycle of monitoring, detecting errors, testing with scenarios, and monitoring again. As long as monitoring, as part of executive functioning, occurred accurately, modeling interactions remained relevant to the goal. Also, the formed abstractions had to be extensively tested with scenarios before they were either accepted or rejected. Lack of monitoring led to the discussion straying from focus, sometimes taking some time before one of the modelers noticed the irrelevance of the discussion. Modelers got back on track after metacognitive prompts and illustrations with concrete scenarios by their peers. 4.3. Individual Differences As soon as a modeler capable of monitoring noticed that definitions or actions stated in terms of first order abstractions were not being understood, a switch was made to clarifying the definition or action with a concrete scenario. This occurred not only for definitions, but also for elaborations on definitions and the identification of new process steps. The reconceptualization often continued until the modelers perceived there to be a shared understanding of the concept in question. The behaviors exhibited by the modelers in the sessions involving stakeholders differ mostly in the sense that they focus much more on concrete scenarios; the goal is to gather input, and the input largely takes the form of scenario descriptions. The analysts do remain in the lead.

The modelers were most of the time not in synchrony with regard to in which problem solving phase they were thinking. The more capable ones were quick to ‘identify, represent and define the problem’, and ‘organize their knowledge’ around it, so that they could start ‘monitoring their progress’ and move towards a ‘solution strategy’. However, the less capable ones stayed longer in the ‘define and represent’ phase, struggling to ‘organize their knowledge’ and never truly moving towards a ‘solution strategy’, much less engaging in ‘monitoring their behavior’ and scope of the process being modeled. There appeared to be no significant differences in the ability to comprehend concrete scenarios. This is consistent with the literature [5,6,19]. First-order abstractions appeared comprehensible to all our modelers, but the speed with which they were grasped differed. In contrast, abstractions of the second order presented more trouble. They did not appear very often in the recorded sessions, but in one instance we noticed that the modelers having trouble to comprehend these abstractions appeared to meet their cognitive limits. Irrespective of the amount of time they were exposed to these abstractions they could not achieve sufficient comprehension to reason on this level. This appears to be highly individual and independent of the role within the project. Abstraction appears to be trainable to a certain extent, but eventually an individual will be constrained by his or her cognitive limits, which was later confirmed by further interviews with analysts about their experiences. With regard to monitoring behaviors, not all modelers were equally able to perform the monitoring cycle. It seems as though there are certain prerequisites for this, such as having a clear mental model of the process being modeled, and its scope. The modelers less capable to perform this cycle had a strong tendency to forget about the goal of the process and get lost in irrelevant details. We did observe that some modelers appeared able to do this mainly for their own behavior, whereas others were capable of doing this for both their own behavior and the behavior of their peer modelers. Most stakeholders appear to be able to make first order abstractions about their own domains, of which they have very clear mental representations. However, when abstractions were made by the analysts across different processes, in some cases the stakeholders merely agreed and no interpretation can be made as to whether they fully understood them or

not. In other cases they explicitly stated that comprehension failed. 5. Discussion We used a perspective on abstraction based on neuroscientific evidence. This perspective has proven very useful for testing the theory in practice, because abstractions are made and used in this way. However, there are considerable differences between individuals concerning the ability to do abstract reasoning. Abstraction appears to be trainable to a certain extent, but still individual differences remain, allowing some to go further than others. The question remains, what is the cause of this, and could we possibly facilitate or stimulate required capacities in some way? Is abstraction as a learning process visible within sessions, or across sessions? In our observations we encountered an interesting issue: can people become so used to reasoning with abstract concepts that they come to view them as concrete ones? Many stakeholders appear to relate the abstraction before them to knowledge of their daily practice in a very specific way: if they can recognize parts or keywords in it, they will understand it, no matter how complex the abstraction is. The hypothesis here is that they actually may not fully understand the abstraction; they simply recognize keywords and fill in the blanks with their daily experience. From this observation follows the question whether people then use them in the same way mentally as concrete concepts, or whether the concepts keep their level of abstractness, and people simply become more proficient in reasoning with abstract concepts? Results by [5] suggest the latter case. In this study we cannot make any claims as to brain area activation, but it would be interesting to investigate how people perceive whether they are reasoning with concrete or abstract concepts. A related issue is the question of what constitutes an abstract and a concrete concept. On a very basic level, abstractions are inherent to using language, and we can therefore say that a reference to something as apparently concrete as a plant is an abstraction, because we do not mention leaves, stem, flowers, or maybe even chloroplasts and nuclei. Concrete and abstract differ largely per context, and we need to find out what people view as concrete and abstract in every domain of investigation. We can ask ourselves what further qualities we require of our modelers and our stakeholders. Can they make their own abstractions, or achieve a

sufficient understanding of them so as to be able to reason with them? Do we even wish to burden our stakeholders with abstractions, or is it sufficient that the analyst adapts his behavior when dealing with stakeholders? 6. Conclusion Our method turned out to be an effective way of exploring the relations between the different cognitive processes of collaborative modeling, and it is worthwhile to further develop the codebook. Question 1. We found that first-order abstractions and concrete scenarios are dominant in collaborative modeling interaction. Question 2. A cycle of problem solving can be observed: the exploration of a problem, followed by the formulation of a solution and its evaluation, with extensive switching when necessary between these phases, guided by monitoring of progress and behavior. Question 3. There are individual differences in abstraction abilities, with some individuals reasoning and comprehending better than others. 7. Future Research The perspective of abstraction and how executive functioning facilitates abstraction in a modeling session has proved to be viable and interesting, and we intend to continue our research in this direction. More sessions with more different modelers will have to be recorded and analyzed, and be made more precise with measurements of executive functioning. This should show both how modeling performance with regard to abstract reasoning and executive functioning affects the resulting model quality, and the understandability of the model for the stakeholders. Monitoring as a part of executive functioning has proved to be vital in achieving the right focus and the set goal, and abstract reasoning facilitates the process. If we want to support collaborative modeling, we have to look further into what executive processes are most important for modeling. We intend to prepare our codebook for quantitative analysis, elaborating it with explicit criteria for when to code a phenomenon, and to compute the inter-coder reliability for it. We need a characterization of

instances where abstractions fail, by means of interactions typical for them. Concerning the transitions between abstraction levels, we need to more explicitly know how the relations change, and of which order of complexity they are. We will also be looking into other known influencing factors on human group performance to create a holistic understanding of the process of conceptual modeling. Individual differences in cognitive capacities, as well as influencing factors such as experience, background, attitudes and creativity have been left out of scope in this paper, but are well worth investigating, particularly when considering sessions with stakeholders. For instance, if a stakeholder is capable of doing something, and she perceives the modeling task to have a useful outcome, this may move her towards a positive attitude, which may ultimately motivate her to participate more actively. Further investigation of the individual and what leads and drives her to make proper abstractions will provide more insight in why modelers do certain things, and how they could be guided in this. 8. Acknowledgements The authors would like to thank three anonymous reviewers for their detailed and extremely helpful comments. This paper results from the Agile Service Development project (http://www.novay.nl/okb/projects/agileservice-development/7628), a collaborative research initiative focused on methods, techniques and tools for the agile development of business services. The project consortium consists of BeInformed, BiZZdesign, CRP Henri Tudor, Everest, HU University of Applied Sciences Utrecht, IBM, Novay, O&i, PGGM, RuleManagement Group, Radboud University Nijmegen, Twente University, Utrecht University, and Voogd & Voogd. The project is part of the program Service Innovation & ICT of the Dutch Ministry of Economic Affairs.

9. References [1] Barkley, R.A., ADHD and the nature of self-control, Guilford Press, New York, 1997. [2] Barsalou, L.W., Perceptual Symbol Systems, Behavioral and Brain Sciences, Vol.22, 1999, pp. 577-609. [3] Berkeley, G., A Treatise Concerning the Principles of Human Knowledge: Introduction, Oxford University Press, Oxford, UK, 1734/1998. [4] Brown, A.L., Metacognition, executive control, self-regulation, and other more mysterious mechanisms, Metacognition, Motivation, and Understanding, Vol.65, 1987, pp. 116.

[5] Christoff, K. & Keramatian, K., Abstraction of Mental Representations: Theoretical Considerations and Neuroscientific Evidence. In: Perspectives on Rule-Guided Behavior, Eds: Bunge, S.A. & Wallis, J.D., Oxford University Press, 2007. [6] Christoff, K., Keramatian, K., Gordon, A.M., Smith, R., & Burkhard, M., Prefrontal organization of cognitive control according to levels of abstraction, Brain Research, Vol. 1286, 2009, pp. 94-105. [7] Colburn, T. & Shute, G., Abstraction in Computer Science, Minds and Machines, Vol. 17, No. 2, Springer, 2007, pp. 168-184. [8] Connor, P. D., Sampson, P. D., Bookstein, F. L., Barr, H. M., & Streissguth, A. P., Direct and indirect effects of prenatal alcohol damage on executive function, Developmental Neuropsychology, Vol. 18, 2000, pp. 331–354. [9] Eibl-Eibesfeldt, I., Human Ethology, Aldine de Gruyter, New York, 1989. [10] Flavell, J.H., Metacognitive aspects of problem solving, The Nature of Intelligence, Vol. 12, 1976, pp. 231-235. [11] Flavell, J.H., Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry, American Psychologist, Vol. 34, No. 10, 1979, pp. 906-911. [12] Gazzaniga, M.S., Ivry, R.B. & Mangun, G.R., Cognitive Neuroscience: The Biology of the Mind, W.W. Norton & Company, Inc., New York, 2002. [13] Hazzan, O., Reflections on teaching abstraction and other soft ideas, ACM SIGCSE Bulletin, Vol. 40, No. 2, ACM, 2008, pp. 40-43. [14] Hochberg, J., Gestalt theory and its legacy: Organization in eye and brain, in attention and mental representation. In: Perception and cognition at century’s end: Handbook of perception and cognition, 2nd edition, Ed: Hochberg, J., Academic Press, 1998. [15] King, A., Discourse Patterns for Mediating Peer Learning. In: Cognitive Perspectives on Peer Learning, Eds: O’Donnell, A. & King, A., Lawrence Erlbaum Associates, Inc., Mahwah, 1999. [16] Kluwe, R.H., Cognitive Knowledge and Executive Control. In: Human mind – Animal mind, Ed: Griffin, D., Springer, New York, 1982. [17] Lezak, M.D., Neuropsychological assessment, Oxford University Press, New York, 1995. [18] Luria, A., Higher cortical functions in man, Tavistock, London, 1966.

[19] Markovits, H., Doyon, C. & Simoneau, M., Individual differences in working memory and conditional reasoning with concrete and abstract content, Thinking and Reasoning, Vol. 8, No. 2, 2002, pp. 97-107. [20] Paivio, A., Yuille, J.C., Madigan, S.A., Concreteness, imagery, and meaningfulness values for 925 nouns., Journal of Experimental Psychology, Vol. 76, No. 1, Supplement, 1968, pp. 1–25. [21] Pretz, J.E., Naples, A.J. & Sternberg, R.J., Recognizing, Defining and Representing Problems. In: The Psychology of Problem Solving, Eds: Davidson, J.E. & Sternberg, R.J., Cambridge University Press, Cambridge, 2003. [22] Toomela, A., How culture transforms mind: A process of internalization. Culture and Psychology, Vol. 2, 1996, pp. 285–305. [23] Van Zomeren, E. and Eling, P., Aandacht en executieve functies. In: Klinische Neuropsychologie, eds. Deelman, B., Eling, P., De Haan, E. and Van Zomeren, E., Uitgeverij Boom, Amsterdam, 2004. [24] Watson, S.M.R. & Westby, C.E., Strategies for addressing the executive function impairments of students prenatally exposed to alcohol and other drugs, Communication Disorders Quarterly, Vol. 24, No. 4, 2003, pp. 194. [25] Ylvisaker, M., Szekeres, S. F., & Feeney, T. Cognitive rehabilitation: Executive functions. In: Traumatic brain injury rehabilitation: Children and adolescents, Ed: Ylvisaker, M., Butterworth-Heinemann, Newton, 1998.