Seediscussions,stats,andauthorprofilesforthispublicationat:https://www.researchgate.net/publication/222545926

Recoveringarchitecturalassumptions

ARTICLEinJOURNALOFSYSTEMSANDSOFTWARE·APRIL2006

ImpactFactor:1.35·DOI:10.1016/j.jss.2005.10.017·Source:DBLP

CITATIONS

29

READS

12

3AUTHORS,INCLUDING:

PatriciaLago

VUUniversityAmsterdam

188PUBLICATIONS1,827CITATIONS

SEEPROFILE

HansVanVliet

VUUniversityAmsterdam

195PUBLICATIONS2,768CITATIONS

SEEPROFILE

Allin-textreferencesunderlinedinbluearelinkedtopublicationsonResearchGate,

lettingyouaccessandreadthemimmediately.

Availablefrom:PatriciaLago

Retrievedon:09February2016

www.elsevier.com/locate/jss

The Journal of Systems and Software 79 (2006) 552–573

Recovering architectural assumptions

Ronny Roeller, Patricia Lago, Hans van Vliet *

Computer Science Department, Vrije Universiteit, De Boelelaan 1081a, 1081 HV Amsterdam, The Netherlands

Received 16 June 2005; received in revised form 30 September 2005; accepted 24 October 2005Available online 13 December 2005

Abstract

During the creation of a software architecture, the architects and stakeholders take a lot of decisions. Many of these decisions can bedirectly related to functional or quality requirements. Some design decisions, though, are more or less arbitrarily made on the fly becauseof personal experience, domain knowledge, budget constraints, available expertise, and the like. These decisions, as well as the reasonsfor those decisions, are often not explicit upfront. They are implicit, and usually remain undocumented. We call them assumptions. Thereis no accepted way to document assumptions, and the relation between the software architecture and these assumptions easily gets lost,becomes hidden in the girders of the architecture. They are rediscovered at a later stage, when the software evolves and assumptionsbecome invalid or new assumptions contradict earlier ones. In this paper, we develop a method to recover such assumptions from anexisting software product. We illustrate the method by applying it to a commercial software product, and show how the results can helpassess the evolutionary capabilities of its architecture.� 2005 Elsevier Inc. All rights reserved.

Keywords: Software architecture; Architectural knowledge; Assumptions; Architecture recovery

1. Introduction

From a software engineering perspective, software thatdoes not change, is not very interesting. We need notbother about the code�s comprehensibility, because noone has to read it any more. The quality of the architectureis not important, because no one will change it. The com-plexity of the system is irrelevant, because it bothers noone.

Software is of interest only when it does change. Thenissues like comprehensibility, quality, complexity, suddenlydo become important. Conversely, when developing sys-tems, we should not only bother about today�s require-ments but also, and maybe even more so, about therequirements of tomorrow.

This inevitable evolution is one of the main reasons forthe importance of software architecture, as for instance sta-ted by Bass et al. (2003): a software architecture manifests

0164-1212/$ - see front matter � 2005 Elsevier Inc. All rights reserved.

doi:10.1016/j.jss.2005.10.017

* Corresponding author.E-mail address: hans@cs.vu.nl (H. van Vliet).

the early design decisions. These early decisions determinethe system�s development, deployment, and evolution. Thearchitectural phase is the earliest point at which these deci-sions can be assessed.

There are many definitions of architecture. Many talkabout components and connectors, or the high-level con-ception of a system. This high-level conception then is sup-posed to capture the major design decisions (Bass et al.,2003). Whether a design decision is major or not reallycan be ascertained only with hindsight, when we try tochange the system. Only then it will show which designdecisions were really important. A priori, it is often notat all clear if and why one design decision is more impor-tant than another (Fowler, 2003).

Since the quality of a system is to a large extent deter-mined by the changes we can still make to that systemwe, as designers and architects, go to great lengths to tryto predict the future user requirements. Any change weforesee, we build into the system, through a proper decom-position, the right kind of interfaces, parameterization, andso on. By doing so, we make assumptions about what will

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 553

change and, mutatis mutandis, about what will not change.At a later stage, we may get into trouble if somethingchanges that was not catered for. Then, maintenancebecomes expensive, the next release takes too much time,or the architecture is found to be too inflexible.

We conjecture (Lago and van Vliet, 2005) that it is use-ful not only to explicitly model variability at softwarearchitecture time, but that it is useful also to explicitlymodel invariability, i.e. the assumptions we make aboutthe system and the environment in which the system isgoing to function. In a sense, we argue for explicitly mod-eling negative as well as positive variability, assuming thatnegative variability concerns things that we assume will notchange.

Usually, assumptions are not explicitly modeled. Yet,architects base design decisions on them. At a later stage,it becomes difficult to trace the reasons for certain designdecisions. Architectural decisions are like material floatingin a pond. When not touched for a while, they sink and dis-appear from sight. These sunken decisions are the most dif-ficult to change at a later stage. In particular, duringevolution one may stumble upon these design decisions,try to undo them or work around them, and get into trou-ble when the change turns out to be very costly if notimpossible. The future evolutionary capabilities of a systemcan be better assessed if these assumptions were explicit.This paper gives a method to make assumptions explicit.

In this paper, we discuss RAAM, a method to recoverassumptions of an operational system. RAAM stands for‘‘Recovering Architectural Assumptions Method’’. Theassumed basis for the method is free access to the sourcecode of the software product and to statistics about itsdevelopment process. The method is best used when histor-ical information on the evolution of a system is available.Further, the possibility to interview developers of the pro-ject is needed. The method provides a set of tools that hasto be tailored to special situations, i.e. to the data sourcesavailable. The set of assumptions recovered is part of theknowledge an organization needs for the successful evolu-tion of the system. We illustrate the use of RAAM withan extensive case study.

The research described in this paper takes place in thecontext of the GRIFFIN project. The goal of the GRIF-FIN project is to develop notations, tools and associatedmethods to extract, represent and use architectural know-ledge that currently is not documented or represented inthe system (van Vliet and Lago, 2005).

The remainder of this paper is organized as follows. Sec-tion 2 describes related work on assumptions, knowledgemanagement, and architecture reconstruction. Section 3introduces and discusses RAAM, our method for recover-ing architectural assumptions. The next sections apply theknowledge gained to a case study. Section 4 introducesthe case, Section 5 unveils its assumptions, and Section 6provides an outlook onto their influence on the possiblefuture of the system. Sections 7 and 8 end the paper withlessons learned and conclusions.

2. Related work

The discussion of related work in this section concernsthree topics: assumptions, knowledge management, andarchitecture reconstruction.

2.1. Assumptions

It is quite tricky to draw the line between assumptions,requirements and constraints. We often use these termsinterchangeably, and we can always find some example thatcontradicts a definition. But we may argue as follows.

Requirements are demands on a system, and can be bothfunctional and non-functional.

Constraints are limitations on a system. We tend tothink of constraints as non-functional requirements, likethe use of a certain database technology, or the implemen-tation in a pre-defined programming language. However,they may be functional too, e.g. a required integration withthe functionality of an existing sub-system.

Requirements and constraints are always consciousdesiderata on the system. They are always stated by the cli-ent or user of the system, and therefore they must be expli-cit. Otherwise we will never know them.

We use the term assumptions as a general denominatorfor the forces that drive architectural design decisions. Justlike it is difficult to distinguish between the what and how insoftware development, so that one person�s requirements isanother person�s design, it is also difficult to distinguishbetween assumptions and decisions. Here too, from oneperspective or stakeholder, we may denote something asan assumption, while that same thing may be seen as adesign decision from another perspective. This paper takesthe perspective of the architect.

We distinguish four types of assumptions:

• Implicit and undocumented: the architect is unaware ofthe assumption, or it concerns of course knowledge.Examples include earlier experience, tacit company pol-icies to use certain approaches, standards, and the like.

• Explicit but undocumented: the architect takes a decisionfor a very specific reason, e.g. the decision to use a cer-tain user interface policy because of time constraints.The reasoning is not documented and thus is likely tovaporize over time.

• Explicit, and explicitly undocumented: the reasoning ishidden. There may be tactical company reasons to doso, or the architect may have personal reasons, e.g. toprotect his position.

• Explicit and documented: this is the preferred, but oftenexceptional, situation.

When left undocumented, assumptions are much more dif-ficult because they remain tacit, or become so over time.They are more interesting as well, as they define the hiddenlong-term invariabilities the system has to adhere tobeyond requirements and constraints.

554 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

Making assumptions is an inherent part of the dailywork in the software industry. Thus, even if they are notaware about it, architects and developers regularly haveto fight with its consequences. Although none of the papersmentioned below presents a ready, complete solution forhow to handle assumptions, some authors do mentionthe problem and give suggestions how to investigate them.

Lago and van Vliet (2004) state that most assumptionsare captured only in the mind, but nonetheless influencethe architecture. If these undocumented assumptions arenot made explicit and formalized in the software descrip-tion, they lead to problems when artifacts of that softwaresystem are to be reused later on, because assumptions inthe core system may mismatch the new target environment.They suggest to find a model that includes, besides variabil-ities, also invariabilities in software architecture descrip-tions. They later (Lago and van Vliet, 2005) present amodel to represent assumptions explicitly.

Yakimovich et al. (1999) look at the issue from a differ-ent perspective, viz. that of creating systems composed ofcommercial-of-the-shelf (COTS) products. The paper pre-dicts integration problems if a component depends onassumptions that involve a conflicting-assumption basedinteraction with an environment. This situation leads tointeraction protocol mismatches. The main focus of thepaper is a method to estimate these integration costs ofcommercial-of-the-shelf products. It turns out that strictassumptions have a major impact on integration costs. Thistheory is also put forth in (Garlan et al., 1995) which dis-cusses a particular COTS-integration project. After invest-ing five man-years, the system described supported allrequired features, but was lacking important quality prop-erties. The system was huge, slow, inefficient, and hard tomaintain. The authors investigated the reasons for thispoor quality and found that conflicting assumptionsthwarted them. To overcome these limitations, the papersuggests making architectural assumptions explicit. Thepaper emphasizes that not only does the widespread cultureof lacking documentation cause the problems, but also themissing formal vocabulary and the missing structuringtechniques to express assumptions, i.e. an adequate model-ing language.

Ecklund et al. (1996) analyze the reasons for assump-tions and reveal that they are not always made withoutpurpose. Developers often have to balance between multi-ple aims. One of the main reasons for intentional assump-tions is the reduction of development costs. The paper callsthis factor the budgetary cut off: architects can add featuresonly until they reach the budget line. As financial limits area matter of fact, architects have no choice but to makeassumptions at various points. The paper points out thatit is essential to improve documentation. One should doc-ument where limitations influence the architecture and towhich concrete assumptions they lead.

A serious problem occurs with assumptions if one triesto integrate two components that have conflicting assump-tions. Uchitel and Yankelevich (2000) is purely dedicated

to enhance the architectural mismatch detection withassumptions. Although the authors do not suggest a con-crete notation, they state that current architecture descrip-tion languages do not allow capturing assumptions. Fromtheir point of view two aspects have to be modeled: theassumptions themselves and their relations to the behaviordescription. Baum et al. (2000) suggest a notation to easilyfind mismatches.

Assumptions can be classified by different discriminatingfactors besides the explicit vs implicit and documented vsundocumented categorizations given above. Lago andvan Vliet (2004) choose the source of assumptions as differ-entiating factor, i.e. the reason why an assumption is made.They distinguish three sources of assumption:

• Technical assumptions are the most obvious category.They concern the technical environment in which theapplications run, e.g. database systems, network infra-structures, middleware software, and programminglanguages.

• Organizational assumptions describe the influence of thecompany�s structure on the software architecture. Basset al. (2003) discuss an example from the aircraft indus-try, where a flight simulator is developed by a group ofindustrial partners. The companies involved insist on ahigh degree of protection of their intellectual property.Consequently, the software architecture must reducethe interaction between the different enterprises andallow them to independently develop their parts againsta public interface.

• Management assumptions are made to achieve businessobjectives. A manager who has successfully carried outa project will try to reapply his successful strategies.Also assumptions about further market developmentare subsumed by this category.

This kind of categorization helps to find assumptions. Itis the one used in this paper. It supports analyzing the pos-sible reasons for an assumption, e.g. to ask the projectmanager if he reapplied old strategies. Another point ofview might be helpful to model the impact of assumptions,by finding out which parts of a system are affected by them.Garlan et al. (1995), a paper that mainly focuses on possi-ble mismatches, offers such a categorization, mainly fortechnical assumptions. Uchitel and Yankelevich (2000)suggest a similar classification based on the target of theassumption. Categories are based on structure, regularity

distribution, burst of service calls and data.The literature about variability management is related to

that of invariability management. Variability points areplaces where the architect builds in a foreseen change.The rationale for the decision whether a variability pointis inserted in the system or not, is often left implicit, andthen is an assumption. Gomaa and Webber (2004) presenta classification of variabilities that can, slightly modified,also be applied to assumptions. The paper mentions twomajor types of variants: internal and external variants.

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 555

Internal variants are created and maintained by the coredevelopers of the system. Users just select them. In con-trast, external variants are created by users. An analogyexists for assumptions: internal assumptions influence thedevelopment of the core system; external assumptions affectthe users when they generate conflicts with the clients� envi-ronment. For example, a system may internally communi-cate via Java RMI, but externally via XML-SOAP. Thatmeans the system assumes all components of the core sys-tem are programmed in Java. Therefore, although coredevelopers cannot add C++ or Ada components directlyto the core, users could use any language that supportsSOAP to integrate the whole core system in their environ-ment. So internal assumptions are only relevant to the coredevelopers, while external assumptions also affect the usersof the system. Such a categorization may be helpful toclearly separate between the inter-company interests andthe information that has to be delivered to re-users.

2.2. Knowledge management

The main value of a software company is its intellectualcapital. As Rus and Lindvall (2002) state: The major prob-

lem with intellectual capital is that it has legs and walks

home every day. This phenomenon is not only a problemif a key developer goes on holiday or even quits his job,but also when the company educates new staff. Therefore,it is in the interest of companies to transform the implicitknowledge, such as assumptions, in developers� minds toexplicit knowledge on paper. The individual experts shouldshare their knowledge between each other and with thecompany. This concept is called the learning-organization

concept. The field of research that handles these topics iscalled knowledge management.

The knowledge evolution cycle in Rus and Lindvall(2002) describes the way from tacit knowledge to explicitknowledge. The five steps are (1) individual members ofthe organization create the knowledge by learning, problemsolving, creativity, or importing from outside sources; (2)members acquire and capture this information in explicitform; (3) organizations organize and transform the know-

(2) Transform tacit to explicit knowledge

(3) Write explicit knowledge down

(4) Deploy and distribute knowledge

(5) Apply knowledge to new projects

Knowledge Evolution Cycle

(1) Individuals capture tacit knowledge

Fig. 1. Positioning RAAM in the contex

ledge in written material or in a knowledge base; (4) orga-nizations deploy and distribute the knowledge throughtraining programs, automated knowledge based systems,or expert systems; (5) all members of the organization use

and apply the knowledge in other projects. As illustratedin Fig. 1, the assumption recovery method in this paperfocuses on steps (2) and (3): discovering assumptions andwriting them down.

Apart from assumptions as a piece of knowledge, there isanother way in which knowledge management is relevantfor this research. The developed method has to be tailoredand fine-tuned to organizational needs. Using the experi-ence from one project may help to get better results foranother project in the same organization. This experiencereuse refers e.g. to estimating the adulterating by humansor reuse of checklists that worked well. Such a knowledgebase can even be used to avoid repeating error-proneassumptions in new projects. These topics are outside thescope of this paper and are not further discussed. Rusand Lindvall (2002) provide an introduction to the stateof knowledge management and explains the concept ofexperience factories, which might be suitable to store thiskind of experience concerning the usage of RAAM.

2.3. Architecture reconstruction

Software architecture reconstruction is a special form ofreverse engineering. Specific methods for architecturereconstruction are described in Krikhaar (1997), Guoet al. (1999), Ding and Medvidovic (2001) and Bass et al.(2003, chapter 10). These methods are all based on analy-sing the source code of a system. They semi-automaticallyderive architectural knowledge from this analysis. In con-trast, we have a more limited goal: finding assumptionsunderlying the architecture as-is, rather than the completearchitecture. Also, we more heavily rely on analysing �sec-ondary� sources such as versioning data for retrieving thoseassumptions.

van Deursen et al. (2004) discuss Symphony, a verygeneral model for architecture reconstruction. Symphonyhas two stages: reconstruction design, and reconstruction

(1) Discovering assumptions

(2) Modeling assumptions

Developed Method

no support

no support

no support

t of the knowledge evolution cycle.

556 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

execution. The reconstruction design stage results in a well-defined procedure for reconstructing the architecture. Thisstage specifically addresses the problem that triggers thereconstruction process, and determining which informationis needed to answer this question, as well as how to get thatinformation. The reconstruction execution step next yieldsthe architectural description. This reconstruction step is a,generally iterative, process of data gathering, knowledgeinference, and interpretation of the information obtained.

Our RAAM method can be seen as an instance of thisarchitecture reconstruction metamodel. Our trigger is torecover architectural assumptions. We identified a numberof sources that may help to do so, and defined steps to actu-ally retrieve those assumptions. This reconstruction designphase is described in Section 3. For the case study, thereconstruction execution phase is described in Section 5.

3. RAAM: recovering architectural assumptions method

Fig. 2 gives an overview of the recovery process. Thefirst analysis steps constitute kind of a tool box, i.e.depending on the available data the analyst performs theanalyses. The information thus gained acts as input toguided interviews with the architects, which then result ina list of assumptions.

For recovery, RAAM provides a tool box which ana-lyzes different information sources: starting with free inter-views, to financial reports and version control, to thesource code. The underlying idea is to find suspiciouseffects in the evolution of the product. A suspicious effectis a potential bad smell (Fowler, 2000) at the architecturallevel. Where a bad code smell is a concrete indication for arefactoring need, a suspicious effect in our context denotesan aspect of a system�s evolution that deserves further scru-tiny. Examples are a file with very many revisions, a com-ponent that many authors have contributed to, or acomponent that scores badly on a given set of quality met-rics. These suspicious effects are the basis for the guidedinterviews. Faced with these suspicious effects, architectsand developers are triggered to explain the design anddevelopment of key parts of the product. This elucidationin turn may lead the analyst to assumptions that acted asimplicit rationale for these design decisions.

Free interviews

Financial reports

Documentation

Version Control

Source codes

Guided interviews

List of assumptions

List of suspicious

effects

Fig. 2. Overview of RAAM.

There are two extreme approaches to recover assump-tions from a product already developed: we can rely on atool that generates a list of assumptions straight from thesource code, or we can interview the developers and archi-tects. The first approach obviously does not work. We can-not expect a tool to recover tacit knowledge. One mayexpect that interviewing developers yields useful resultswhen done shortly after development. But a test in thecompany that provided the case study showed that the abil-ity of also this approach is limited. The developers couldnot even come up with obvious assumptions that we easilyidentified after a short analysis of the product.

That means some intermediate approach has to be fol-lowed: combining structural analysis with interviews. Butthe thought experiment with the two poles also shows thatit is very unlikely that 100% of the assumptions will even berecovered.

Next to code and people, there is a third valuable sourceof information: data about the development process, asstored in the project memory. Again, there are many possi-ble resources: statistics from the version control system (likeCVS), the source code of older versions, the documenta-tion, and even the financial department may have detailedreports about who was working when on which component.The use of these different sources of information is dis-cussed in the following subsections.

Fig. 2 suggests a linear process, in which all suspiciouseffects are collected first, after which the guided interviewstake place. This is a simplification. In practice, there willlikely be feedback loops. For example, insights gained froma guided interview about financial reports may allow one tobetter understand certain elements from the CVS analysis,which may lead to additional suspicious effects, etc.

3.1. Free interviews

In contrast to the guided interviews discussed in Section3.6, free interviews are not based on previous research ofother information sources. Instead, architects and develop-ers are directly asked if they recall assumptions they made.As assumptions are still more or less unfamiliar to peopleoutside research, it helps to explain the concept of assump-tions at the start of the interview. It should be emphasizedthat assumptions are not a sign of bad work, but a necessarypart of the architect�s life. It may be difficult for the intervie-wee to just give a list of assumptions. For that reason, theinterviewer may ask about the history of the developmentand try to ask further as soon as the interviewee mentionsproblems or other irregularities. These free interviews canalso be used to get a better understanding of the currentarchitecture, something that will pay off later when lookingat the version control information and the source code.

3.2. Financial reports

Financial reports include data like hours spent per devel-

oper per module or ratio between hours spent for develop-

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 557

ment, testing, maintenance, etc. Analyzing these data mayyield valuable information. For example, modules thathave a high ratio between hours spent on maintenanceand hours spent on development may indicate that assump-tions had to be removed or, more likely, that a work-around was introduced.

It depends on the company�s structure what kind offinancial information is available. Small companies withan informal way of working may hardly have any accuratereports stating which developers were working on whichcomponents. In contrast, big companies with a big finan-cial department may have very precise information. Basi-cally, the attraction of financial reports comes from thefact that they are non-technical. This diversification maypay off. Similar information sources are likely to retrievesimilar assumptions. Analyzing a different view on thedevelopment process may lead to totally new aspects.

Although financial reports can be a valuable source, it iswise to interview clerks on how timely the data for thefinancial reports are to pay attention to how reliable thesekind of data are. If developers fill out a form about theirworking hours only once in a while, the results may be veryinaccurate.

3.3. Documentation

The system�s documentation may be a rich source ofinformation. If the documentation is accurate, it refers tothe actual state of the source code. But too often the officialdocumentation states how the system should look ratherthan how it really looks. Discussing the differences betweenthe documentation and the code, and the reasons for thesedifferences, may give valuable hints. A walkthrough type ofinterview can be used to find these differences, which poten-tially hide assumptions. Again, the amount of availabledocumentation depends on the organization�s culture. Bigcompanies tend to focus on codification, and mostly havea very detailed policy of how to create documentation.Small companies often focus on personalization, andmainly document who can be asked about what. Obvi-ously, the focus chosen influences the amount of availabledocumentation.

3.4. Version control

Information about the development process, as capturedin a version control system, is a valuable source from whichtools like StatCVS (Pekacki and Cyganiak, 2004) generatestatistics. Interpreting this summarized information shedslight on critical aspects of a project, e.g. a high rate of filerevisions may indicate an error-prone class. As the result ofthe quantitative analysis, a list of potentially interestingaspects can be made. These next have to be augmentedby a qualitative analysis to finally assess the underlyingassumptions.

Analyzing version control data often is the best way togain quantified information about the development pro-

cess. Nonetheless, these data have to be reviewed carefully.Many factors influence the statistics, and sometimes evendestroy the whole picture. For example, the time span aclass or module is part of the project influences many mea-sures. A file that has been part of the project for manyyears most likely has undergone more revisions than a filethat was added just a week ago. On the other hand, onemay argue that newer files potentially have more revisions,because the system grows over time, more developers areinvolved, and thus more coordination is required. Forexample, if at the start only two developers work on thesystem, they can avoid merge conflicts by frequently talk-ing to each other. With 10 developers, this way of workingis no longer feasible. Instead developers will check in theirchanges with a higher frequency, which ultimately leads toa higher apparent revision rate.

Another important factor is the different working styleof developers. For example, one developer may prefer towrite extremely compact source code, while another devel-oper tends to emphasize clearness and readability. So,although the function of a class may be the same, theamount of code may differ. In fact, the more readablebut larger class may even be less influenced by assump-tions, because more people understand it. The frequencyof commits also influences the statistics. One typical devel-oper prefers to commit his changes only once a day, whileanother commits every small change directly. In the casestudy discussed in Section 5 for instance, the averageamount of added lines of code (LOCs) per revision variesfrom 10 to 57 depending on the developer.

The fact that version control analysis tools like StatCVS

treat files without any semantic understanding is a two-edged sword. On one hand, this unawareness of semanticsaffects the result, since for example comments and evendead code are counted as source code. Even worse, if thedirectory structure does not match the component struc-ture, it may be very hard to identify the error-prone com-ponents. But having no semantic analysis is also anadvantage. Every part of the project is taken into account,not only the subset that the analysis tool understands. Notonly Java and C++ code are taken into account, but alsoHTML files and Java Server Pages (JSP).

When analyzing version control information, oneshould always take into account that suspicious effects neednot be caused by assumptions, but may also be caused byspurious statistics. Nevertheless, version control informa-tion provides useful input to the guided interviews, whichfinally lead to the exposure of the assumptions that weremade during the development process.

3.5. Source code

The version control analysis sees directories and files asthey are: directories and files, as collections of bytes. Butfor the source code also its internal structure can be investi-gated, e.g. classes, modules, and packages. Investigation ofthe internal structure leads to more accurate results,

Table 1Foci of measures

Focus Measures

Procedural metrics MVGObject oriented design CBO, WMC1Structural metrics IF4

Table 2Correlation between measures

MVG WMC1 CBO IF4

MVG 1.0000 0.7127 0.4142 0.1721WMC1 0.7127 1.0000 0.6483 0.3433CBO 0.4142 0.6483 1.0000 0.8034IF4 0.1721 0.3433 0.8034 1.0000

558 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

because such an investigation also detects connectionsbetween different classes and how often a class is used. Onthe other hand, the approach is limited, and informationfrom the version control approach has to be added. Forexample, analysis tools may be able to handle only commonlanguages like Ada, C++, or Java, but not less structuredlanguages like PHP. The approach may also lose importantinformation that is not saved in a programming language atall, such as configuration files and HTML pages.

To analyze the source code, one may basically use anycode analyzer that generates reasonable information. Forillustration purposes, we have used the open-source toolCCCC (Littlefair, 2004). This tool analyzes C++ and Javaprograms and generates various metrics. For the case studyin Section 5, the following four measures are used asindicators:

• The McCabe’s Cyclomatic Complexity (MVG) measuresthe decision complexity at the procedural level. It isdefined as the number of linearly independent pathsthrough a directed acyclic graph which depicts the flowof control of a subprogram. A function withoutbranches has an MVG of 1. Every if, case, catch, for,

while, do statement increases the MVG (McCabe, 1976).• The coupling between objects measure (CBO) counts the

clients and suppliers of the analyzed class. A high valuefor CBO indicates a weak encapsulation in the objectoriented design and may result in problems to reusethe module.

• The weighted methods per class value (WMC1) is thenumber of methods of a class. WMC1 is a measure forthe size of a class. The assumption is that larger classesare in general less desirable.

• The information flow measure (IF4) is a structural mea-sure computed from the square of the product of thefan-in and fan-out. Fan-in is the number of modulesthat pass data into the analyzed module, while fan-outis the number of modules that receive information fromthe module. To calculate these values the public inter-faces are analyzed (Henry and Kafura, 1981).

There are yet other tools that generate other statistics.Basically, one has to choose indicators that quantify howimportant a class is. Ideally, the chosen indicators are sta-tistically independent or have at most a weak causal con-nection. For example, number of methods, number of codelines, and average number of code lines per method is abad choice, because the latter can be derived from the for-mer two. Table 1 indicates the different foci of the measureswe use. Despite the different foci, these measures do have asignificant correlation in our case. Table 2 gives the corre-lation coefficients for one release; the others look similar. Italso shows visually in Figs. 7–11, where the top items in thelist generally score high on most measures.

The tool CCCC generates a list of all classes with theirassociated measures. In the next step, these classes aresorted using the following algorithm:

1. For each of the four measures (MVG, CBO, MWC1,IF4) sort the classes in descending order, giving fourlists.

2. For each list, the class ranked as number 1 gets 20points, the class ranked as number 2 gets 19 points,etc. Classes ranked as number 21 or higher get 0 points.

3. For each class, add up the points from the four lists.4. Sort the classes in descending order of the number of

points assigned to them.

A high rank indicates that the class is likely to bemore important for the whole system. This ranking ensuresthat we concentrate the analysis on really importantclasses. Even more interesting is to compare the rankingsof different versions. Suspicious changes like dramaticincreases or decreases in the ranking, new classes, disap-pearing classes, etc. may be caused by assumptions. To findthe reasons behind these changes, the architects and devel-opers of the corresponding components have to be inter-viewed.

3.6. Guided interviews

Although assumptions are not directly implemented inthe architecture, they are an important factor that influ-ences the evolution of the architecture. That is, an assump-tion may influence the development in very specific ways.From the opposite viewpoint, a suspicious effect in thedevelopment may be caused by an assumption. The aimof our method is to find these suspicious effects and usethem as input to guided interviews. In these guided inter-views the architects and developers of the system areinvited to explain what caused the observed effects. So thediscovery is split in two parts: a quantitative analysis usingmeasurement techniques, and a qualitative analysis usingquestioning techniques. The interviewer uses the indica-tions of suspicious effects gained from the quantitativeanalysis to direct the qualitative analysis.

Table 3Changes between major versions

Version Modifcations

1 Added: Java applicationChanged: –Removed: –

2 Added: multi-lingualChanged: rewrite in Java, database model,C++ for core engineRemoved: –

3 Added: no functional logicChanged: –Removed: –

4 Added: EJBs, rule system in scripting language,XML importerChanged: configuration in XML instead of binary files,faster build system, J2EERemoved: –

5 Added: using XSL and Cocoon, full EJB/J2EE,data updates during run-timeChanged: supports also NT (not only Linux)Removed: –

6 Added: business properties, integrate external products,configuration tool v.1Changed: –Removed: EJBs

7 Added: support for Dynamo, WebSphere, and JBoss,partly configuration tool v.2(more oriented on business user requests)Changed: –Removed: Cocoon

8 Added: reportingChanged: finish configuration tool v.2Removed: –

Suspicious effect: Before Version 7, the softwaresupported only Orion as its J2EE application server.Later also WebSphere, JBoss, and Dynamo could beused by customers. In the same step, Cocoon wasremoved.

Interview: Before the usage of EJBs in Version 4,only the servlet container Tomcat (Moczar, 2004) wasused. Version 4 runs on Orion, because at that timeOrion was the fastest and cheapest available J2EE ser-ver. Subsequent customers had to integrate the applica-tion into their WebSphere and Dynamo environments.Then, all Orion-specific solutions had to be replacedby J2EE-compatible approaches. These changes affectalso Cocoon, because the JSP-engine of WebSphere

and Jboss conflicted with the used Cocoon version.Assumption: Orion is the fastest and cheapest appli-

cation server (Assumption A6), J2EE is easy to main-tain (Assumption A5).

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 559

4. Case study

The developed method was applied to a real-worldapplication to investigate its behavior. The analyzed prod-uct improves e-commerce channels of major retail compa-nies in the USA, UK, and continental Europe. Theapplication is implemented with Java/J2EE. Within thedeployed application, two major components can be iden-tified: the core engine and the configuration tool withwhich sales managers manage the operational aspects.We�ll denote the company that provide the case study asSalesPlus. This is a fictitious name. To preserve anonimity,we have renamed also the version numbers of releases andthe actual names of the modules of the product.

The evolution of the analyzed product has been full ofturns—like a slalom. The dynamic and flexible business fieldof an e-business company is full of challenges: new businessrequirements, new technologies, and new major trends. Theevolution reflects all these influences of the last years. Some-times changes could be adopted easily; but sometimes evencrucial parts of the application had to be rewritten, oftenbecause its assumptions did not hold any longer.

5. Recovering assumptions

In this section we describe the application of RAAM tothe SalesPlus application. All available information sourceswere analyzed and a list of assumptions was generated.Although the different sources emphasize different aspectsof the system, there is some overlap, and certain assump-tions break the surface through more than one informationsource. Each assumption is mentioned only once, in thesection discussing the source in which the assumptionwas found first. For each suspicious effect found in a cer-tain step, we give its description together with additionalinformation gathered during the guided interview and theassumption identified, if any. If no assumption is identified,the assumption field is marked ‘‘none’’.

Note that it is not our goal to discuss each and everyassumption in the case study. Only assumptions that helpto illustrate the use of the method are described.

5.1. Free interviews

Asking directly for assumptions retrieved hardly anyassumption. Since architects and developers were not famil-iar with the concept of assumptions, they could not readilyidentify them. To avoid this direct question, they were askedfor differences between major product versions. That turnedout to be a good start to get an overview of the system�s evo-lution. Table 3 shows the differences detected between theversions. Especially of interest are situations in which a fea-ture occurred in one version, but disappeared later on. Forexample, Cocoon (Ziegeler and Langham, 2002) is intro-duced in Version 5, but removed again in Version 7. Thisremoval was necessary because Cocoon turned out to betoo complex and not compatible with the various applica-

tion servers. In terms of a suspicious effect, the informationgained from discussions with stakeholders, and the assump-tions derived, this analysis can be phrased as follows:

560 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

Although the free interviews did not yield many assump-tions, they did help to identify the major versions. Forexample, in the first attempt, also other versions wereassumed to be major versions. During the interviews itturned out that most improvements had been introducedin Versions 6, 7, and 8. For that reason, the analysis ofthe other information sources concentrates on only theseversions.

5.2. Documentation

The documentation policy of SalesPlus focused on spec-ifications and JavaDoc in the source code. The fast changesin the software made it very hard to keep architecturaldescriptions up-to-date. The detailed documentation andthe architecture were not consistent. This made it difficultto use the existing documentation to identify assumptions.

5.3. Version control and financial reports

SalesPlus uses the Concurrent Versions System (CVS)(Vesperman, 2003) to manage its development process.CVS logs every change that is made to a file of the projectand allows to roll-back to previous stable states.

The open-source tool StatCVS (Pekacki and Cyganiak,2004) is used to analyze the CVS archive and to generatevarious statistics. Table 4 shows the raw data for the largermodules, which is reused in subsequent diagrams. Thistable gives the number of revisions and the lines of code(LOCs) per module. As mentioned in the discussion ofRAAM in Section 3.4, the results are not based on compo-nents but on directories. For example, one might deducefrom Table 4 that the configuration tool in the directory/configtool/ is the heart of the application, because itincludes 16% of all LOCs. But in the guided interviews it

Table 4CVS statistics of modules

CVS-module Revisions(%)

LOCs(%)

configtool/ 9.3 16.2src/net/product/util/ 1.1 3.9src/net/product/field/type/ 2.2 3.9src/net/product/comp/ 1.3 3.8extendertool/src/ 0.1 3.2configtool/js/ 0.1 2.7src/net/product/index/ 0.7 2.6components/bo/src/net/product/bo/model/ 1.5 2.3src/net/product/field/ 1.3 2.2src/net/product/viewer/ 0.9 2.1config/ 3.1 1.9src/net/product/product/ 2.0 1.9configtool/i18n/ 0.6 1.8importer/src/net/product/importer/domain/ 0.0 1.7components/database/src/ 0.4 1.7exporter/src/net/product/xmlexport/ 0.2 1.7components/util/src/net/product/util/ 0.3 1.6src/net/product/engine/ 2.1 1.6html/sample/ 1.0 1.6src/net/product/service/display/slicerules 1.1 1.5

became clear that this directory is just badly structured.For example, all parts of the configuration tool are in asingle directory, while other components are split intodifferent directories.

To obtain better information, it is useful to depict thenumbers in relation to each other. Fig. 3 shows the relationof revisions to module size, measured in lines of code. Onemay expect that larger modules have more changes. Thecorrelation is significant (Pearson�s product–moment corre-lation is 0.89, p-value 9.39e�8), but only if we include allvalues. If we drop the data for /configtool/, the correlationis not significant (Pearson�s correlation is 0.12, with a p-value of 0.64). Some of the suspected values in Fig. 3 areexplained by the nature of the module. E.g. config/ includesthe configuration files, which are updated very frequently.Also /configtool/i18n is easily explained. ‘‘i18n’’ means‘‘internationalization’’, because between the initial ‘‘i’’and the final ‘‘n’’, there are 18 letters. /configtool/i18n con-tains the specialization to English, German, and French forthe configuration tool. So each time a functionality isadded, also the corresponding strings have to be put inthese three files.

Fig. 4 shows the ratio of the number of lines of code thathave ever been written for a module to the number ofLOCs that this module currently consists of. A high ratioindicates a lot of rewrites, possibly caused by assumptions.Although this figure looks very promising, during theguided interviews no related assumptions were recovered.

The three most significant measures are summarized inFig. 5: lines of code (LOCs), revisions per LOC, and addedLOC per LOC. All modules are sorted in each category, thetop value gets 20 points, the next 19, down to 1 point for thelowest value. Finally, the three values are added up andsorted by their sums. This simple algorithm gives all mea-sures the same weight. The algorithm easily normalizes thedata, i.e. one outlier does not influence the result too much.

Next to the configuration directory, the configurationtool configtool/ scores highest. To further investigate thedevelopment of the configuration tool, the time sheets fromthe financial department are analyzed. Unfortunately, onlythe data for the year 2003 were completely available. Fig. 6shows the relation of development time spent in the config-uration tool and the time invested in the browser engine.During May and July, the developers spent more than halfof their working time in the configuration tool. For thenext three months this value flips back to around 20%,and for the last two months of the year, it increases againto around 40%. The decline is caused by the finalization ofVersion 6 in October 2002, in which the configuration toolwas introduced. But sales managers could still not manageall settings with the configuration tool, although they werealready part of the product. This limitation was eliminatedin the next few months, resulting in the higher percentageof development work for the configuration tool near theend of the year.

The interviews revealed a close coupling between thedesign of the configuration tool and its developers. The final

Fig. 3. Revisions per module vs. module size (logarithmic chart).

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 561

version had to be part of Version 8 for business reasons.Driven by the very high time pressure, the responsible archi-tect chose a design that reflects the available humanresources. A lot of logic was outsourced to the XSL andJavaScript level, such as verifying submitted data from theuser (Assumption A17) and generating the XML blocks(Assumption A20). With this approach, XSL developerscould take over a larger share of the work. Furthermore,the architect favored JSP over native Java, because JSPcan be changed easily at runtime and thereby allows for fas-ter development. So the time pressure and the skills of theavailable developers directly influenced the design chosen.The architect assumed a certain mix of developers whenchoosing a certain architectural solution (Assumption A19).

Another assumption in this context is that sales manag-ers at the client company will use the configuration tool atall (Assumption A16), since the graphical user interface isthe only feasible way to change business rules. Althoughit is possible to change the XML configuration files directly

with a text editor, the verification of business rules is doneonly in the web pages. Managers decided that this solutionis acceptable, but they overlooked another important con-sequence. It turned out that big companies are not willingto train their sales people for different user interfaces.Instead, they prefer their own meta tools that combinethe configuration tools of all subsystems under one userinterface. That means, in the future the company may nolonger be able to assume that no central configuration layeris needed (Assumption A18), but will have to provide apublic API for configuration purposes. This configurationlayer has also to verify the business rules.

StatCVS also generates some statistics that are basednot on the directory level but on single files. Tables 5 and6 show some of them. The first shows the files that havebeen changed most frequently. The latter shows the fileson which the largest number of authors worked. Theimpact of these statistics is shown for one example.Table 5 indicates that most revisions relate to a file called

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

conf

igto

ol/

src/

net/p

rodu

ct/u

til/

src/

net/p

rodu

ct/fi

eld/

type

/

src/

net/p

rodu

ct/p

rodu

ct/

exte

nder

tool

/src

/

conf

igto

ol/js

/

src/

net/p

rodu

ct/in

dex/

com

pone

nts/

bo/s

rc/n

et/p

rodu

ct/b

o/m

odel

/

src/

net/p

rodu

ct/fi

eld/

src/

net/p

rodu

ct/v

iew

er/

conf

ig/

src/

net/p

rodu

ct/p

rodu

ct/

conf

igto

ol/i1

8n/

impo

rter

/src

/net

/pro

duct

/impo

rter

/dom

ain/

com

pone

nts/

data

base

/src

/

expo

rter

/src

/net

/pro

duct

/xm

lexp

ort/

com

pone

nts/

util/

src/

net/p

rodu

ct/u

til/

src/

net/p

rodu

ct/e

ngin

e/

htm

l/sam

ple/

Rat

io

Added LOCs/LOCs

Fig. 4. Ratio of changed LOC vs. actual LOC.

Suspicious effect: Class Category has the highestscore in all versions.

Interview: A category is the most complex type ofinformation, because it is hierarchical. Over the evolu-tion of the application, this class was rewritten severaltimes to make it more efficient.

Assumption: none.

Suspicious effect: Classes Registry and BPParser-

TokenManager disappear both in subsequent versions.Interview: These classes are part of a third party

framework, on which the application was based inthe very beginning. After Version 3, these parts havebeen rewritten because the used framework turnedout to be too complex and too slow regarding thestartup time. Furthermore, the company wanted to

562 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

Bridge.java. The same file also ranks in the top group inTable 6. That means a lot of people did a lot of work on thatfile. Interviewing the developers revealed the cause hereof.Because of two conflicting assumptions ‘‘One responsibledeveloper per module’’ (Assumption A10) and ‘‘Centralizedinterface can be kept up-to-date’’ (Assumption A11)the Bridge between the core engine and the web interfacecreated a lot of trouble.

5.4. Source code of major versions

This section shows the charts for all major versions,mentions suspicious classes, and reports on the results ofthe guided interviews (Figs. 7–11). Finally, the underlyingassumptions are explained. Some of the suspicious effectsare not visible by comparing only two charts. Instead, thehistory over a longer time gives the whole picture of theevolution. In these cases, the suspicious effect and its expla-nation is mentioned only once at the first appearance of theeffect. For example, the first EJB classes are found in Ver-sion 4, Version 6 was pure EJB, and Version 8 was againEJB-free. In this case, the problem is discussed only inthe section for Version 4.

5.4.1. Version 3

become independent of closed source code that isowned by third parties, to avoid legal and technicalproblems in the future.

Assumption: Closed source code from third partiescauses problems (Assumption A1).

Fig. 5. Summary of the three measures.

Table 5Most CVS-revisions per file

File #Rev.

src/net/product/jsp/Bridge.java 295

src/net/product/engine/CoreEngine.java 291configtool/common.xsl 198src/net/product/product/EntryManager.java 188src/net/product/product/Entry.java 168src/net/product/index/Index.java 166configtool/task_feature_edit.xsl 162components/bo/src/net/product/bo/manager/

EntryManagerEJB.java 150

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 563

5.4.2. Version 4

Suspicious effect: Classes YYparse and YYlex wereintroduced with Version 4 and immediately becamethe most complex part according to metric MVG.

Interview: Both classes are part of the rule compiler.This system compiles a proprietary description of thebusiness rules into Java code. In Version 6, the com-piler was removed and replaced by an open scriptinglanguage, because maintaining an own compiler isvery complex and costly. The scripting language offersthis functionality cheaper and more user-friendly.

Assumption: Users accept the proprietary rule sys-tem (Assumption A12), Own rule-compiler offers morefreedom (Assumption A13).

configtool/taskbar.xml 143html/sample/common.xsl 137

Table 6Most authors per file

File #Authors

html/sample/common.xsl 10config/system.default.xml 10src/net/product/util/prefs/BusinessProperties.java 9src/net/product/product/EntryManager.java 9src/net/product/jsp/Bridge.java 9

src/net/product/engine/CoreEngine.java 9src/net/product/field/type/SQLFieldTypeManager.java 9src/net/product/field/CountryManager.java 9src/net/product/field/CategoryManager.java 9src/net/product/field/FieldManager.java 9

Fig. 6. Development effort for config tool vs. engine.

Fig. 7. Most suspicious classes in Version 3.

564 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

Suspicious effect: Classes EntryManagerEJB andFields are introduced in Version 4 and become moreand more important until Version 6, their rankingsdrop in Version 7, and they finally totally disappearin Version 8.

Interview: In Version 4 the first occurrences of EJBsappear, because clients requested so. Version 6 isbased totally on EJBs. The full-EJB solution turnedout to be too resource intensive when a clientrequested to manage more than 50,000 products. Forthat reason, in Version 7 many components are madeEJB-free again, and finally Version 8 is totally free ofEJBs again.

Assumption: Users want EJBs (Assumption A7),EJBs are fast enough (Assumption A8).

Suspicious effect: Class Universe appears new in theranking.

Interview: Although the concept of multi-universeswas already introduced in Version 5, it gained moreand more importance.

Assumption: none.

Suspicious effect: Class Index appears new in theranking.

Interview: The Java-class Index is a redesign of theold C++ storage. The main reason for the change isto get rid of C++ code and to provide a more efficientaccess.

Assumption: Java is too slow (Assumption A3),100% Java is easy to maintain (Assumption A4).

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 565

5.4.3. Version 6

Suspicious effect: Class User disappears, although ithas the 4th highest score in Version 5.

Interview: The application became stateless in Ver-sion 6. Session tracking did not work any longerbecause end-users became more aware about privacyand therefore switched off cookies. Stateless accessoffers similar functionality and even eases integrationwith third party products that depend on sessiontracking.

Assumption: End-users accept session tracking(Assumption A9).

Suspicious effect: Class SaveEntryAction scores veryhigh, but disappears in Version 7.

Interview: SaveEntryAction is a handler for the first,Struts-based, configuration tool with which the salesmanagers define business rules. Later the Struts-framework (Cavaness, 2002) was removed, because itwas too complex to handle.

Assumption: Struts is easy to maintain (AssumptionA2).

Suspicious effect: Classes Value and SetValue disap-pear, although both score very high in earlier versions.

Interview: Before the EJB era, Value and SetValue

stored the values of fields. In the pure-EJB Version6, these storages have been removed.

Assumption: none.

Suspicious effect: Class Bridge disappears, althoughit has the 5th highest score in Version 5.

Interview: The Bridge was intended to offer a uniqueinterface for Java Server Pages (JSP) to the coreengine. Unfortunately, the Bridge never did so com-pletely—it is a leaky abstraction (Spolsky, 2002). So,the developers decided to access the core functionalitydirectly from JSPs.

Assumption: One responsible developer per module(Assumption A10), Centralized interface can be up-todate (Assumption A11).

5.4.4. Version 7

Suspicious effect: Class Index grows extremely incomplexity, as measured by MVG, from the 17th tothe 4th position.

Interview: Index became the core of the product.Therefore, a lot of optimization, e.g. caching, tookplace in Version 7.

Assumption: none.

Suspicious effect: Class BitSetType appears new inthe ranking.

Interview: BitSetType allows to store sets very effi-ciently in a bit pattern structure. This solution waschosen to improve performance.

Assumption: none.

Fig. 8. Most suspicious classes in Version 4.

Suspicious effect: Class Field raises strongly in theranking.

Interview: With Version 7, the location string wasconnected to fields, which gives a much higher degreeof structure to the class Field.

Assumption: none.

Suspicious effect: Class FieldMap appears new in theranking.

Interview: FieldMap adds a more abstract way tohandle fields. Before, fields were handled indepen-dently, and differently, in each component.

Assumption: none.

566 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

5.4.5. Version 8

Suspicious effect: Classes ExtendedProductEngine

and AdvancedProductEngine appear new in theranking.

Interview: Both classes existed before. They allowplugable third party engines that take over a part ofthe functionality. In Version 8, it turned out that thesystem has to check which third party product is used,because the way of accessing is different for them. Thatmeans the abstract layer in between is by-passed.

Assumption: All relevant third party engines can becontrolled in a similar way (Assumption A15).

Suspicious effect: Class Universe has many moremethods than in former versions.

Interview: The Universe no longer delivers all avail-able fields but can be configured to show only subsetsof them. This solution improves performance.

Assumption: none.

5.5. List of assumptions

The following list provides for each discovered assump-tion a short explanation of its nature and its impact on the

Fig. 9. Most suspicious classes in Version 6.

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 567

architecture. The list gives for each assumption, its nameand its classification as a technical assumption (Tech),organizational assumption (Org), or management assump-tion (Man). See Section 2 for a discussion of these classes.

A1: Closed-source code from third parties causes problems(Man): Managers expect to be confronted with legaland technical problems if closed-source modulesfrom third parties are integrated in the application.Technical problems may occur if a bug is expectedin a third party module, but the bug cannot be fixedbecause the source code is not accessible. Therefore,only free and commercial open-source modules areintegrated.

A2: Struts is easy to maintain (Tech): Version 6 intro-duced the first version of the configuration tool whichprovides a graphical user interface to manage busi-ness rules. After market research, the architectassessed that Struts offers an easy way to create aweb-based application for user inputs. Later, thedevelopers struggled with the unnecessary complex-ity, the leaking documentation, and the poorly con-ceived software design of Struts.

A3: Java is too slow (Tech): In 2000, mainly the JavaDeveloper Kit (JDK) 1.2 was used by clients. Asthe documentation for the JDK 1.4.2 (Sun, 2003)shows, major performance improvements took placewith the introduction of the JDK 1.3 and the JDK1.4. However, JDK 1.2 had such a poor performancethat it was impossible in 2000 to run the Java-basedrule-system with it. For that reason the rule-compilerwas reimplemented in C++.

A4: 100% Java is easy to maintain (Tech): The first versionof the product was bought from another company asa pure Java application. Architects and developersagreed that the language is easier to maintain thanC++, which was the other alternative. Only perfor-mance critical parts like the query engine had beenimplemented in C++ to use its more efficientcompiler.

A5: J2EE is easy to maintain (Tech): After SalesPlusdecided to no longer use proprietary middleware,they had to find a new one. J2EE offers all necessaryfeatures, has a big developers community, and isthe preferred middleware platform for Java appli-cations.

Fig. 10. Most suspicious classes in Version 7.

568 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

A6: Orion is the fastest and cheapest application server

(Tech): The analyzed product uses Java Server Pages(JSP) to generate XML-pages which are finally pro-cessed by the client�s web application. Therefore, aJ2EE servlet engine is needed. The developers decidedto use a complete application sever. Three years ago,the fastest and cheapest J2EE server on the marketwas Orion. This assumption revealed its impact laterwhen developers started to use Orion-specificfeatures.

A7: Users want EJBs (Man): In 2001–02, Enterprise JavaBeans (EJB) were one of the hypes in Java develop-ment. Every e-commerce company that wanted tobe taken seriously had to use EJBs one way or theother. The experience of a lot of companies soonshowed that an EJB-for-everything approach doesnot work well because of the significant overheadand the resulting bad scalability.

A8: EJBs are fast enough (Tech): From the introductionof EJBs in Version 4 until its peak in Version 6, archi-tects and developers believed that they could achieveall quality properties with this approach. Then a new

customer requested to manage more than 500,000entries. This new requirement invalidated theassumption.

A9: End-users accept session tracking (Man): An E-Com-merce website should provide information to endus-ers that fit best their desires. For that reason, sucha system has to learn about its users, e.g. via user pro-filing. The architects assumed that cookies would bethe best solution to capture user data. But after somewebsites abused cookies, and the media discussed thetransparent client, end-users became more awareabout privacy. Finally, a lot of customers switchedoff cookies, the main source of data for user profiling.That is why the developers replaced the cookie-basedsystem.

A10: One responsible developer per module (Org): SalesPlusis quite a small software company. Each part of thesystem is informally attached to one developer whois responsible for keeping an eye on that component.This way other developers always have an expert whocan provide them with the internals of the componentin question.

Fig. 11. Most suspicious classes in Version 8.

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 569

A11: Centralized interface can be up-to-date (Org): Thearchitects planned to not let the Java Server Pagesdirectly access the core engine. Instead, an intermedi-ate layer, the Bridge class, should provide an interfacethat can be called by the JSPs and that forwards therequests to the core. It turned out that the people-centric development process (Assumption A10) madeit impossible to keep the interface up-to-date, becausenone of the programmers has a deep enough under-standing about all parts of the system, and theresponsible developers regularly forgot to updatetheir parts of the Bridge.

A12: Customers accept the proprietary rule system (Man):Business rules describe how the product behaves.With Version 3, a proprietary rule-compiler wasintroduced, which generates Java code from client�sconfiguration files. It turned out that customers arenot willing to learn a proprietary language to describetheir business purposes. For that reason, the rule-compiler was replaced by a scripting language-basedsystem in Version 4.

A13: Own rule-compiler offers more freedom (Tech): In thebeginning the product provided only very trivial set-tings to describe the behavior. Architects expected

this basic system too limited for further projects. Thislimitation should be overcome by the proprietaryrule-compiler which processed the business rules.Looking back, one can see that the compiler offersa lot of unnecessary freedom for a very high pricein development and maintenance costs. The currentversion has a much simpler rule system based on ascripting language, which offers nonetheless enoughfreedom.

A15: All core engines can be controlled in a similar way(Tech): The application traditionally comes with justone core engine. Through the requests of customers,SalesPlus offers now also another, more powerful,engine for certain environments. To let customerschoose which solution they want to use, the systemneeds a transparent way of accessing the core engine.Developers assumed that a simple interface will solvethe problem. Later it turned out that each core enginehas not only very different options for tuning butpartly even a different way of integration.

A16: Customers will use the configuration tool (Man): Theconfiguration tool provides also non-technical per-sons such as sales managers the opportunity tochange business rules. To react directly to market

570 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

changes, this feature is essential. The managers atSalesPlus assumed that companies are willing to traintheir sales people to use this web-based tool. It turnedout that some clients plan to develop meta tools thatcombine all subsystems under one user interface inthe future.

A17: XML verification in JS only is adequate (Tech): Thebusiness rules are saved in XML format. The config-uration tool already generates these XML snippets.Since only the JavaScript (JS) in the configurationtool generates these XML blocks (AssumptionA20), the architect assumes that only the JavaScripthas to check if the submitted values are valid, i.e.the back-end never checks the generated XML.

A18: No centralized configuration layer is necessary (Tech):Based on Assumption A16, the architects did notexpect customers to integrate the configuration man-agement into their own meta tools. For that reason,no independent configuration layer was necessary.Instead, the configuration tool triggers the compo-nents directly to propagate the changes.

A19: Specific mix of Java/XSL developers (Org): At thetime of designing the architecture of the configurationtool, SalesPlus was faced with a high time pressure,but too few Java developers. For that reason, partof the development effort had to be transferred toavailable XSL/JavaScript developers. This transferfinally led to the current design, which moves a lotof logic like validation of form values and XML gen-eration to JavaScript.

A20: Only JavaScript creates XML (Tech): Based on thedesign of the configuration tool, which is highly influ-enced by Assumption A19, only JavaScript in theusers� browsers shall generate XML snippets thatdescribe the business rules.

Some of the assumptions, like ‘‘EJBs are fast enough’’(Assumption A8) or ‘‘End-users accept session tracking’’(Assumption A9) did not come as a big surprise to manag-ers and architects at the company. They drew two majorconclusions from the above list. First, the configurationtool seems to be the weakest part of the system from anarchitectural point of view (Assumptions A16–A20). Tak-ing into account that the configuration tool is one of themost important components from a business perspective,there is an urgent need for action. Second, none of thearchitects or developers had been aware of the discrepancybetween the developer-centric work style and the central-ized responsibility. The conflict between Assumption A10and Assumption A11 is an obvious example hereof. Butalso the design of the configuration tool has the sameshortcoming. The configuration tool does not delegatethe logic to related components, but enforces the develop-ers of the configuration tool to implement the whole logic.

Many of the technical assumptions concern qualityissues, such as maintainability, performance, or flexibility.In their description, we often use qualitative terms, like

�fastest�, �cheapest�, �easy�. The value of these descriptionslies not in their literal interpretation, but in the issues beingconveyed.

6. Assumptions and SalesPlus� future

This section uses the discovered assumptions to discusssome of the challenges SalesPlus may be faced with in thefuture. We discuss two possible change scenarios, likelyfuture situations. For both scenarios, we discuss conflictswith the assumptions recovered.

The Configuration-Live-Reporting Cycle scenario isinteresting because it illustrates the interplay between atechnical assumption—functionality concentrated in a cer-tain component—with an organizational one—the way ofworking in the organization. The second scenario illus-trates even more complex dependencies between assump-tions. These examples show how knowledge aboutassumptions helps discussions about the further evolutionof the system. They also illustrate the need for moreadequate assumptions modeling tools.

6.1. Configuration-live-reporting-cycle

Scenario: A customer wants to optimize his settings.

Therefore, he sets a different value for each week and tries

to find the rate which generates the maximum profit.

The configuration tool is a milestone in letting the salesmanagers make configuration changes without the helpof their technical support. They can change all relevant set-tings themselves, an important factor in fast businesses likee-commerce. Reporting is the next big wave in this develop-ment. The configuration tool already provides basicreports. The vision is to extend the reports to finally closethe cycle: a sales manager configures the product, analyzesthe influences after some days, and fine-tunes the settingsfurther.

This scenario is threatened from two sides. If sales man-agers refuse to use the configuration tool (AssumptionA16), it is a bad idea to bind the cycle functionality purelyto the configuration tool and not in a more abstract layer.The other problem may derive from the person-centricdevelopment (Assumption A10). On one hand, the config-uration tool couples the functionality of the different com-ponents tighter and tighter, but on the other hand, theknowledge about these sub-fields is decentralized. Onemay solve the problem with a different developmentapproach, in which different developers work tightertogether. This seems a rather theoretical solution, becausechanging the development process at such a fundamentalpoint is unrealistic. A more elegant approach is to let theconfiguration tool focus purely on front-end issues withoutunderstanding what it is managing. The concrete parts ofthe system can be plugged into this managementframework.

The recovery of these connections helped the developersto design their new concept of an independent configura-

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 571

tion layer. The revealed assumptions convinced managersthat Assumption A16 had to be dealt with, i.e. users mayprefer not to use the configuration tool. Therefore, Sales-Plus intends to offer a public API for the configurationlayer. Even more importantly, architects became moreaware of the conflict between the people-centric develop-ment (Assumption A10) and the centralized configuration.For this reason, the configuration layer will provide onlyan abstract management interface, but the concrete logicwill be placed in the search, browse, and promotioncomponents.

6.1.1. Business management in client’s application

Scenario: A customer wants to manage the applicationtogether with other subsystems of his or her e-commerce

solution in a central administration tool.

A big company normally combines a lot of different sub-systems, such as inventory control, customer relationshipmanagement (CRM), etc., in its online channels. Trainingsales managers for all different products is a very expensiveprocess. For that reason, such a company plans to createmeta tools, which control all subsystems through a uniqueinterface. As a consequence, SalesPlus has to provide notonly the proprietary configuration tool but also an openinterface. This demand conflicts with the assumptionthat ‘‘all changes go through the configuration tool’’(Assumption A16), which necessitates further changes.For example, the new abstract configuration layer can nolonger delegate the validation of configuration values tothe front end.

<<Split Java/JSP/XSL/JS>>

<<no verification in JSP

<<direct access to conf-f

<<implementation>>

<<leads>>

<<leads>>

<<org assumption>>

Specific mix Java/XSL developers (A19)

<<tech assumption>>

Only JavaScript creates XML (A20)

<<component>>

Config Tool

Fig. 12. Assumptions in t

Fig. 12 shows the assumptions that influence the config-

uration tool (CT). The two basic assumptions are the oneabout the available human resources at the time of devel-opment (Assumption A19) and the claim that the CT willbe the only user interface to manage business rules(Assumption A16). All the other assumptions are derivedfrom these two. The assumption that the CT is the onlyuser interface leads to the assumption that no additionalconfiguration will be necessary (Assumption A18). Instead,the CT on its own has the capability to write business rulesto the depending configuration files. Based on the assump-tion about the available resources (Assumption A19), adesign was chosen that kept a big share of the logic inthe front-end. Namely the HTML forms are already veri-fied for valid values in JavaScript code. If this check is suc-cessful, an XML block is generated and sent to the server.As the application assumes that the CT is the only sourceof configuration changes and that the related XML textis purely generated by JavaScript (Assumption A20), onecan assume that testing the XML on JavaScript level isadequate (Assumption A17). Based on the relation betweenassumptions shown in Fig. 12, one can see that the invali-dation of Assumption A16 also invalidates AssumptionsA17, A18, and A20.

7. Lessons learned

A major problem in the beginning of the analysis isexpert identification. At SalesPlus, it was often unclearwho is responsible for which part of the system. The story

>>

iles>>

<<leads>>

<<leads>>

<<man assumption>>

Users will use the CT (A16)

<<tech assumption>>

XML verification only in JS is adequate (A17)

<<tech assumption>>

No centralized conf. layer is necessary (A18)

he configuration tool.

572 R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573

becomes even more complex when these responsibilitieschange over a component�s lifetime, e.g. if the originalauthor leaves the company, the actual responsible devel-oper may not know the details of early decisions. Thechanging responsibilities lead to problems since part ofthe knowledge may be totally lost. During our research,it paid off to always ask the interviewees also when andhow they had been involved.

Although it was not the aim of our research to recoverall assumptions, we did face the problem of how to guess

the impact of an assumption. Potentially, there is anassumption behind everything. It is not feasible to first cap-ture all possible assumptions, and then filter out the reallyirrelevant ones. This way we will end up classifying far toomany assumptions. Although RAAM helps to focus on thereally change-prone modules, it remains hard to identifythe really relevant assumptions in these modules duringthe guided interviews. This process needs a deep under-standing of the architecture being analyzed, even aboutparts that are free of assumptions. It is a moot questionwhether an external consultant would be able to makethese decisions without first analyzing the whole systemin detail.

Another problem with the case study, one which mayhold for all fast and dynamic companies, is that the soft-ware system being analyzed is a moving target. Informationthat had been captured near the beginning of the investiga-tion was partly out-dated at the end of the research project.To identify relevant modifications, it is necessary to regu-larly consult the change log of the system. The highlydynamic nature of the SalesPlus system adds an additionallevel of complexity to the interviews. Sometimes, intervie-wees spoke about different states of development, becausethey had not updated their knowledge to the current state.The problem of the moving target emphasizes again howimportant personalization is next to codification, i.e. notonly to capture what the assumptions are, but also whom

to ask for more information.Tailoring the developed method to the concrete situation

can be done in several ways. The case study demonstratesthat the method works also if only a subset of the informa-tion sources is available. For example, in the case study,hardly any financial reports exist about the working pro-cess, and useful documentation was not available at all.The very rich data gained from version control and sourcecode compensated for this lack of other sources.

Fig. 12 gives one example of how relations between

assumptions can be modeled. There is considerable poten-tial for improvement in this area. One may conceive of arich set of relationships, including relationships like leadsto and overpowers. One may think of possibilities to usethese relationships to answer questions like Why are two

assumptions in conflict? or What changes if one assumption

turns out to be invalid?Next to graphical notation, a mathematical notation will

also be helpful. The former already depicts the assumptionsin their context, but the latter would allow creating auto-

matic analyzing tools like mismatch detectors. For exam-ple, Mei et al. (2003) offer a good notation for variationconstraints and features. Such a mathematical model ofthe assumptions also eases integration in software designtools. Aside from automatic mismatch detection, thesetools may improve traceability between assumptions andarchitectural artefacts.

Finally, the modeling shown in Fig. 12 hints at yetanother problem. This figure only describes the state atone point of time. But assumptions change over time.Not only may the status change from valid to invalid orvice-versa, but also dependency relations between assump-tions may change. It may be helpful to develop a notationthat models these evolutionary aspects of assumptions.

8. Conclusions

The case study showed that the developed method is apowerful tool to recover and model assumptions. A seriousproblem during these investigations was that the researcherneeded to acquire a deep understanding of the software sys-tem to identify the relevant assumptions. In general, oneneeds to find the right people that have a deep understand-ing of the system and are able to transfer that knowledge,and people that have the knowledge to support the recov-ery of implicit assumptions.

RAAM supports reconstruction of the assumptions pic-ture of a system. This assumptions picture is part of theknowledge a learning organization needs to successfullysupport the evolution of a system. In Section 6 we useassumptions to assess the evolutionary capabilities of a sys-tem. In doing so, we are able to assess the cost of changingthe system. By identifying the assumptions involved in achange scenario, we gain insight into also how the set ofassumptions changes over time.

Further research is aimed at developing a mathematicalmodel to describe assumptions. Such a model is a goodbase to integrate assumptions to software architecturetools. We will also apply the method to other software pro-jects. The aggregation of the experience gained willimprove the method and serve as a base for a best practicedatabase architects can use during architecture develop-ment.

Acknowledgements

This research has partially been sponsored by the Dutch

Joint Academic and Commercial Quality Research & Devel-

opment (Jacquard) program on Software Engineering Re-search via contract 638.001.406 GRIFFIN: a GRId For

inFormatIoN about architectural knowledge. This researchalso received partial support from the Dutch Ministry ofEconomic Affairs via contract SENTER-TSIT3018CALCE: Computer-Aided Life Cycle Enabling of SoftwareAssets. We thank the anonymous referees for their helpfulsuggestions, and Joost Schalken for doing the statisticalcomputations.

R. Roeller et al. / The Journal of Systems and Software 79 (2006) 552–573 573

References

Bass, L., Clements, P., Kazman, R., 2003. Software Architecture inPractice, second ed. Addison-Wesley Professional.

Baum, L., Becker, M., Geyer, L., Molter, G., 2000. Mapping requirementsto reusable components using design spaces. In: IEEE InternationalRequirements Engineering (ICRE2000), Schaumburg, Illinois, pp.159–167.

Cavaness, C., 2002. Programming Jakarta Struts. O�Reilly.Ding, L., Medvidovic, N., 2001. A light-weight, incremental approach to

software architecture recovery and evolution. In: Proceedings WorkingConference on Software Architecture (WICSA). IEEE ComputerSociety, pp. 191–200.

Ecklund, E.F., Delcambre, L.M.L., Freiling, M.J., 1996. Change cases:use cases that identify future requirements. SIGPLAN Not. 31 (10),342–358.

Fowler, M., 2000. Refactoring: Improving the Design of Existing Code.Addison-Wesley.

Fowler, M., 2003. Who needs an architect. IEEE Software 20 (5), 11–13.Garlan, D., Allen, R., Ockerbloom, J., 1995. Architectural mismatch or

why it is hard to build systems out of existing parts. In: Proceedings ofthe Seventeenth International Conference on Software Engineering.ACM Press, pp. 179–185.

Gomaa, H., Webber, D.L., 2004. Modeling adaptive and evolvablesoftware product lines using the variation point model. In: Proceedingsof the Proceedings of the 37th Annual Hawaii International Confer-ence on System Sciences (HICSS�04)—Track 9. IEEE ComputerSociety.

Guo, G., Atlee, J., Kazman, R., 1999. A software architecture recon-struction method. In: Proceedings Working Conference on SoftwareArchitecture (WICSA). Kluwer Academic Publishers, pp. 15–33.

Henry, S., Kafura, K., 1981. Software structure metrics based oninformation flow. IEEE Transactions on Software Engineering 7,510–518.

Krikhaar, R., 1997. Reverse architecting approach for complex systems.In: Harrold, M., Visaggio, G. (Eds.), Proceedings InternationalConference on Software Maintenance. IEEE Computer Society, pp.4–11.

Lago, P., van Vliet, H., 2004. Observations from the recovery of asoftware product family. In: Third Software Product Line Conference(SPLC)LNCS, vol. 3281. Springer-Verlag, pp. 214–227.

Lago, P., van Vliet, H., 2005. Explicit assumptions enrich architecturalmodels. In: Proceedings 27th International Conference on SoftwareEngineering. IEEE Computer Society, pp. 206–214.

Littlefair, T., 2004. Cccc 3.pre84. Software. Available from: <http://sourceforge.net/projects/cccc>.

McCabe, T.J., 1976. A complexity measure. In: Proceedings of the 2ndInternational Conference on Software Engineering. IEEE ComputerSociety Press, pp. 308–320.

Mei, H., Zhang, W., Gu, F., 2003. A feature oriented approach tomodeling and reusing requirements of software product lines. In: 27thAnnual International Computer Software and ApplicationsConference.

Moczar, L., 2004. Tomcat 5 Unleashed. Sams.Pekacki, L., Cyganiak, R., 2004. Statcvs 0.2.2. Software. Available from:

<http://statcvs.sourceforge.net>.Rus, I., Lindvall, M., 2002. Guest Editors� introduction: knowledge

management in software engineering. IEEE Software 19, 26–38.Spolsky, J., 2002. The law of leaky abstractions. Available from: <http://

www.joelonsoftware.com/articles/LeakyAbstractions.html>.Sun, 2003. JavaTM 2 SDK, Standard Edition Documentation, Version

1.4.2. Sun Microsystems, Inc.Uchitel, S., Yankelevich, D., 2000. Enhancing architectural mismatch

detection with assumptions. In: 7th IEEE International Conferenceand Workshop on the Engineering of Computer Based, Edinburgh,Scotland, pp. 138–147.

van Deursen, A., Hofmeister, C., Koschke, R., Moonen, L., Riva, C.,2004. Symphony: view-driven software architecture reconstruction. In:Proceedings 4th Working IEEE/IFIP Conference on Software Archi-tecture (WICSA). IEEE Computer Society, pp. 122–132.

van Vliet, H., Lago, P., 2005. GRIFFIN—a GRid For inFormatioNabout architectural knowledge. Available from: <http://www.cs.vu.nl/~se/griffin>.

Vesperman, J., 2003. Essential CVS. O�Reilly.Yakimovich, D., Bieman, J.M., Basili, V.R., 1999. Software architecture

classification for estimating the cost of cots integration. In: Proceed-ings of the 21st International Conference on Software Engineering.IEEE Computer Society Press, pp. 296–302.

Ziegeler, C., Langham, M., 2002. Cocoon: Building XML Applications.New Riders.