multicriteria decision making software french2005

8/18/2019 Multicriteria Decision Making Software French2005

1/16

JOURNAL OF MULTI-CRITERIA DECISION ANALYSIS

J. Multi-Crit. Decis. Anal. 13: 65–80 (2005)

Published online in Wiley InterScience (www.interscience.wiley.com) DOI: 10.1002/mcda.372

Comparison Study ofMulti-attribute DecisionAnalytic Softwarez

SIMONFRENCH*andDONG-LINGXUy

Manchester BusinessSchool (MBS) ,TheUniversityofManchester, BoothStreetWest,ManchesterMl5 6PB,UK

ABSTRACT

In this paper, we discuss the functionality and interfaces of five MCDM software packages. We recognize that nosingle package is appropriate for all decision contexts and processes. Thus our emphasis is not so much to comparethe functionality of the packages per se, but to consider their fit with different decision making processes. In doingso, we hope to provide potential users with guidance on selecting a package that is more compatible with their needs.Moreover, we reflect on the further functionality which we would believe should be developed and included inMCDM packages. Copyright# 2006 John Wiley & Sons, Ltd.

KEY WORDS: decision processes and culture; multi-attribute decision analytic software

1. INTRODUCTION

With the advance of modern computing technol-ogy, there are more and more software packagesavailable offering to support decision analysis. InOR/MS Today’ 2002 biennial survey 27 softwarepackages were listed; in the 2004 survey there were45 packages (Maxwell, 2002, 2004). All thepackages are seemingly versatile, offer user-friendly graphical interfaces and more thansufficient power to tackle substantial problems.

With such choice, it is natural to ask what are thedifferences between the packages. Are some morepowerful, more intuitive or whatever than others?In our case we are concerned with the fit of MCDM packages with the culture and form of thedecision process. Some decision analyses arecarried out on a desktop for a client, some in adecision support room with a group of decisionmakers present, and some ‘off-line’ for an organi-zation with the results being fed back via a reportor decision seminar. Belton and Hodgkin (1999)remark similarly on different contexts of use andtheir requirements implications for decision sup-

port software. Are some packages more suited for

one or more of these processes than others?Moreover, the designer of the decision analyticsoftware will inevitably have his or her ownworldview and philosophy and that may reducethe value of the software for others with differentworldviews. Clearly, this is obviously the case if wecompare software emanating from distinct schoolsof decision analysis such as the multi-attributevalue analysis (MAVT) (Keeney, 1992; Keeneyand Raiffa, 1976), the French outranking ap-proach (Roy, 1996) and analytical hierarchy

process (AHP) (Saaty, 1980). But within schoolsthere are also subtle}and not so subtle!}differ-ences; and there are those who straddle two ormore schools.

In this paper we make a small step towardsexploring the differences between five softwarepackages, taking a perspective that is a littlebroader than the purely that of functionality andsoftware design. Broadly, our objectives are to:

* compare methods used for problem structur-

ing, value elicitation, sensitivity analysis and

presentation within the packages;* identify their weaknesses and strengths, such

as limitations, user friendliness, information

handling, and flexibilities, paying regard to

their use in different decision making pro-

cesses;and thus both to:

* help potential users identify the functionality

that they may need in an MCDM package;

Copyright # 2006 John Wiley & Sons, Ltd.

*Correspondence to: Manchester Business School(MBS), The University of Manchester, Booth StreetWest, Manchester Ml5 6PB, UK.E-mail: [email protected]: [email protected] study is based upon the second author’s MBAdissertation at Manchester Business School.


2/16

* suggest new functionality that may be

incorporated into MCDM packages.

However, we recognize that these objectives arehopelessly ambitious given the enormous number

of different possible ways in which these packagesmay be used. Thus we have a more focusedobjective of contributing to the discussion of methodologies for evaluating decision analyticsoftware for different purposes by providingexemplar comparisons. We should also be clearabout what we are not seeking to accomplish. Wedo not intend to provide a comprehensive surveyof the functionality of the packages: the AidingInsight series of surveys in OR/MS Insight areavailable for that (Maxwell, 2002, 2004). We seeour work as complementary to that of Belton andHodgkin (1999).

The paper is organized as follows. In the nextsection, we note the variety of contexts for decisionmaking and different decision support processes.We then describe the methodology for comparingthe software and the packages themselves. Finally,we turn to the comparison and some conclusions.We do not summarize the underlying theories andtechnical details of decision analytic methodolo-gies in the following, referring instead to theliterature (Belton and Stewart, 2002; French, 1988;Goodwin and Wright, 2003; Keeney, 1992; Saaty,1980; Watson and Buede, 1987).

2. DECISION CONTEXTS, PROCESSES ANDANALYSES

No two situations which call for a decision are everidentical; they differ due to a wide range of factors:

* Problem context: For example, what are the

external characteristics of the problem; is it

well structured; is uncertainty present; how

many options and possibilities need to be

considered?* Social Context: For example, what are the

characteristics of the social organization in

which the decision has to be made; who are

the decision makers and how many are there;

what are their responsibilities and account-

abilities; who are the stakeholders?* Cognitive factors of the DMs: For example,

how intelligent, imaginative, knowledgeable

are the decision makers; can they live with

risk and uncertainty; which behavioural

‘biases’ and ‘heuristics’ do they exhibit.

For discussion of such factors, we refer to, interalia, (Bazerman, 2002; Belton and Stewart, 2002;

French and Geldermann, 2005; French and RiosInsua, 2000; Kleindorfer et al ., 1993; Watson andBuede, 1987).

In addition to the variety in decision contexts,there is only slightly less variety in decisionprocesses. For instance:

* Who are the different players: decision

makers, experts, stakeholders and analysts

(French and Rios Insua, 2000)? When and

how do they become involved? On some

occasions not all may be involved and some

individuals may serve in two or more roles.

* Is there a single decision maker to whomultimately all decision analyses are addressed

and what decision analytic methodologies do

her1 worldview favour? Or are there several

with conflicting beliefs, preferences and,

perhaps more fundamentally, different

worldviews?* How much time is available for the analysis?

It may be anything from a couple of hours to

a couple of months or, occasionally in the

case of societal decisions, seemingly decades.* Will the analyses have to be communicated

to stakeholders as part of a subsequentcommunication implementation process?

* Will the analysis be conducted by the

decision maker themselves? Will an impartial

analyst be involved? Will the analyst and

decision makers work together throughout

or will the analyst take the analysis away and

report back to the decision makers with a

solution. If there are a group of decision

makers, how will the analyst interact with

them: one to one or in a plenary group?

Given this enormous range of contexts anddecision processes, it would be surprising if decision analytic software came in ‘one size fitsall’ packages}although the vendors of thepackages might try to persuade you otherwise.Moreover, it would be equally surprising if we

1We refer to decision makers in the feminine anddecision analysts in the masculine.

S. FRENCH AND D.-L. XU66

Copyright # 2006 John Wiley & Sons, Ltd. J. Multi-Crit. Decis. Anal. 13: 65–80 (2005)


3/16

could analyse all software packages from all theseperspectives. Thus we shall be more selective in ourexemplar comparisons and consider three context-process pairs which to some extent span the rangeof possibilities. But only some extent: we do not

explore any serious approach to modelling un-certainty such as found in decision tree andinfluence diagram packages. We focus on multi-attribute problems with conflicting objectives.Thus we consider the software’s suitability foruse in:

A. A single user context in which a singledecision maker analyzes the problem forherself on her own computer: a self-help orDIY context (Belton and Hodgkin, 1999).

B. A group meeting context in which thedecision makers gather with an analyst andanalyse the problem in a plenary decisionconference (Belton and Hodgkin, 1999;Eden and Radford, 1990; French, 1988;Phillips, 1984), i.e. in which the analyst runsthe software and the results are projected forthe group to see together.

C. A consultancy role in which the analystmeets with the decision makers}perhaps asa group, perhaps one-to-one}and thenanalyzes the issues ‘off-line’ reporting backto the decision makers with recommenda-tions in due course.

As noted in Belton and Hodgkin (1999), thesecontexts lead to different requirements. Forinstance in case A, the decision maker may be anon-expert user and will need more support in theform of help and other explanation features thanan expert analyst. In case B, the software shouldproject well and offer the audience clear, unclut-tered screens without distracting information andoptions that would only be of use to an expertuser; however, since the decision makers do notoperate the software nor interpret the analysisthemselves, there need be less in the form of helpand explanation. In case C, the software may offer

complexities to an expert user, but should alsooffer report writing features and simple plots toenable the insights gained from its use to beconveyed to a wider audience. In case A, thesoftware may need to help the non-expert decisionmakers structure the problem quickly and effec-tively; whereas in cases B and C the analyst mayuse his expertise to structure the problem beforeinputting it into the model format assumed by the

software. In all cases, the means of inputting judgemental information may need to support theelicitation process itself.

Note that in case B, we have very much in minda decision conference with plenary analysis of the

models. There is also a group meeting context inwhich the decision makers use networked softwareto explore the problem both individually and inplenary as a group (Nunamaker et al ., 1988). Wedo not discuss this case because it requires thesoftware to be group enabled. While some of thepackages we consider either have some of thisfunctionality or offer a version which has, we didnot have the means to explore their use in such agroup context.

3. STUDY METHODOLOGY AND CHOICEOF PACKAGES

We chose to look at five packages: HiView,V.I.S.A, Web-Hipre, Expert Choice and Logical Decisions. Our choice was in part guided byavailability and our experience with some of thesoftware. But we also sought to consider twoMAVT packages and three hybrids betweenMAVT and AHP, although we recognize thatthese distinctions are becoming blurred as thepackages develop and functionality is added. Wedid not look at outranking packages, being aware

that the philosophy and perspective behind themwas far removed from our own and thus we couldnot hope to provide a fair evaluation.

To compare the packages in detail, we firstexplored the functionality of each package firstlyusing a common example relating to the choice of a motorcycle (Isitt, 1990). As all the five testedpackages use a weighted sum method for attributeaggregation (albeit with different interpretations)and certain core functionalities, such as ranking,are the same, they will not be discussed further.Instead we concentrate on those aspects in whichthey differ, in particular:

* problem structuring, such as attribute hier-archy construction and modification;

* weight and value elicitation;* data presentation and sensitivity analysis.

Note that by problem structuring we mean theinitial stages of an analysis in which the decisionmakers’ perceptions of the issues and concernsfacing them are brought to the surface and

MULTI-ATTRIBUTE DECISION ANALYTIC SOFTWARE 67



4/16

modelled as a decision table. This process is oftensupported by a range of soft modelling andbrainstorming methods (Pidd, 1996, 2004; Rosen-head and Mingers, 2001). Of course, we recognizethat the need for formulation varies from context

to context. In some cases, the attributes andalternatives are all but explicit; in others there is

just what Ackoff termed a mess (Ackoff, 1974;Pidd, 1996). We would note, however, that whensubstantial formulation is necessary then we leantowards value-focused methodologies (Keeney,1992).

We recognize that terminology in the MCDMfield is not entirely standard: cf. Belton andStewart (2002), Goodwin and Wright (2003) andKeeney (1992). We use attribute and criteriaessentially synonymously, tending to use theformer, to describe characteristics of the alter-

natives, e.g. cost. An objective is an attribute plusan imperative, e.g. minimize cost. We use the termvalue tree generally, recognizing that others mightuse attribute tree or criteria hierarchy.

One final point: our more generic aim of discussing the appropriateness of different MCDMsoftware for providing support in different deci-sion making contexts means that sometime wedescribe and discuss a feature in one package atsome length, thus giving it apparent emphasis forthat package. That feature may be present inseveral or all the other packages, but since we donot discuss it in detail again, we may seem to imply

that their implementations are less effective. Thatis not our intention: absence of comment shouldnot be taken as an indication that the feature isabsent from a package.

4. DESCRIPTION OF THE PACKAGES

4.1. HiViewHiView is one of the earliest software package forsupporting MCDM. It implements standardMAVT approaches to supporting ranking deci-sions. It was developed in the 1980s, initially for

facilitating decision conferencing by Larry Phillipsand Scott Barclay at the London School of Economics (Barclay, 1984). The original versionworked under MS DOS and used special fontsdesigned for legibility via three-lens data projec-tors. The functionality and screen design was keptsimple in order to focus conference participantsattention on the core analysis needed in a 2-dayconference. Simplicity of use with more complex

functionality hidden from immediate view to leavea clean screen remain very much part of HiView’sdesign, consistent with the underlying requisitemodelling methodology championed by Phillips(1984). Nonetheless, some complexity has been

added in the developments of later versions.2

Forinstance, originally HiView had no pairwisecomparison options for eliciting values andweights: values and weight were input numericallyor via thermometer scales. The latest versionincludes the Macbeth qualitative pairwise compar-ison approach (Bana e Costa and Vansnick, 2000).No networked version of the software is available.HiView does have an easy-to-use reporting func-tion. The generated report has both graphics andtext. The report can be viewed through webbrowsers and will help in developing reports,especially necessary in case C.

4.1.1. Problem structuring. Attributes can beadded, moved and linked easily to form a valuetree: and the linking may be performed after theattributes have been brainstormed. The finishedvalue tree can be displayed either vertically orhorizontally}and the package will re-organize thetree neatly. There is no technical reason why thevalue tree cannot be very large, but the windowshave clearly been designed to display mosteffectively trees with 5–15 attributes, i.e. the scaleof tree that can be built and analysed easily in a2-day decision conference.

4.1.2. Value and weight elicitation. Relative orabsolute values for leaf attributes can be enterednumerically: support for their elicitation is pro-vided graphically via a thermometer or histogram,or verbally via the Macbeth module. The attri-butes can be rescaled subsequently via a non-linearvalue function: piecewise linear, discrete (forverbal grades) and logarithmic value functionsare currently supported. The graphic window forvalue function definition is simple and straightfor-ward to use. For piecewise linear value function,

new points can be added at any location of theconsidered attribute value range.

HiView supports numerical weight elicitation bydirect assignment and swing weighting (Figure 1);

2The current third version, HiView 3, is built by IPLInformation Processing Ltd., marketed by CatalyzeLimited (www.catalyze.co.uk) in association withEnterprise LSE.




5/16

and the Macbeth module provides support forverbal pairwise elicitation. The direct weight

assignment is relatively simple and easy to operate.It considers one family of attributes at a time. Theweights can be assigned either by typing or mousedrag-dropping in the interactive graphics window.For a relatively large tree, the swing weightwindow does not display all leaf attributes atonce; again there is a presumption that the valuetree will be of the moderate size common indecision conferences.

4.1.3. Sensitivity analysis. HiView provides anoverview picture of the local sensitivity of the

ranking, showing when changes in the weights of individual attributes will lead to a change in thehighest ranked alternative: see Figure 2. This is aunique and very valuable feature of HiView. Itidentifies sensitive weights and guides furtherinvestigation. In the picture, the attribute will becoloured red if less than 5% change of its weightwill cause a change in alternatives ranking, amberif 5–15%, and green if above 15%. There is also asorts function which identifies the relative advan-tages and disadvantages in terms of weighted orunweighted attributes scores between any pair of alternatives. While not strictly a sensitivity techni-

que, this can guide exploration and certainly aidsunderstanding of the reasons that certain alter-natives rank highly.

4.2. V.I.S.AV.I.S.A stands for Visual Interactive SensitivityAnalysis for Multi-criteria Decision Making. Itwas developed by Valerie Belton (Belton, 1984;Belton and Vickers, 1988). The first version

appeared in 1988; its latest release is V.I.S.A5.0.3 Originally written for MS DOS, it is now afully compliant MS Windows program, whichimplements standard MAVT approaches to sup-porting decisions. There is also a group version

Figure 1. Support for swing weighting in HiView.

Figure 2. Overview of local sensitivity to weights on allattributes.

3Developed and marketed by Visual Thinking (www.vi-sualt.com).




6/16

allowing networking and a variety of joint working.In many respects V.I.S.A is more a research andeducation tool than a widely marketed MCDMpackage. Valerie Belton and her team at StrathclydeUniversity are continually exploring and evaluatingextra functionality in prototype versions.

The opening screen looks busier than some,though there are in truth no more buttons andmenus than average, because there is a watermarkexplaining how to get started and get help and anempty window for inputting alternatives. Those of us persuaded by value-focused thinking arguments(Keeney, 1992) may be a little discomforted by thisinitial subliminal lead towards alternative-focusedthinking; but it is more than possible to build

the value tree first and look to creating alternativeto evaluate. In use it is both easy and productiveto have many sub-windows open leading to amuch busier screen than some of the otherpackages. Thus more skill is necessary in usingV.I.S.A with groups lest they are distracted by

windows which are not the current centre of attention. Furthermore, font sizes tend to be smallby default though there are zoom buttons to resizethe tree quickly. For individual use they are fine,but in projection to moderate-sized groups, therecan be difficulties (see Figure 3). There are noreport writing features per se; the user is left to cutand paste plots using the standard Windowsclipboard.

Figure 3. The main window in V.I.S.A.




7/16


8/16

line at the end of each bottom-level attribute orthey can be set in a sub-window using histogramsor thermometers. The weights of higher-levelattributes are calculated automatically by additionand renormalization. The within-family method

breaks the tree into families of attributes, eachconsisting of a parent and its immediate children.In each family, the relative importance of a childattribute is considered in relation to the parent anda weight assigned accordingly. This process isrepeated for all the families. Unlike some otherpackages, e.g. Expert Choice and HiView, V.I.S.Ainitially defaults to equal weights. This has theadvantage that users can begin exploring themodel as soon as a few criteria and alternativesare defined; but there are disadvantages.

Firstly, the exploration might distract usersbefore they have finished brainstorming attributes

and building the tree. Secondly, to suggest anyweights risks anchoring biases (Bazerman, 2002;Kahneman et al ., 1982). Thirdly, an explorationprocess in which the final ranking may visible asweights are set is open to manipulation, deliberateor subconscious.

4.2.3. Sensitivity analysis. V.I.S.A’s strength is itsinteractive sensitivity analysis functions: they aresimple, flexible yet very powerful. Indeed, thedevelopment of V.I.S.A was stimulated by thedesire to provide such interactive exploration of sensitivity arising from Belton’s early work (Bel-

ton, 1984). For some time this interactivity was theprogram’s unique selling point. Specifically, sensi-tivity analysis is carried out by displaying perfor-mance scores of all alternatives (i) on a selectedattribute; (ii) on two selected attributes on a two-dimensional plane; and (iii) on a selected attributeas a function of a selected attribute weights. Theeffects of changing any weight or score on a leaf attribute is shown dynamically on the graphs; i.e.the effects of any changes are automaticallyupdated on all current displays, at all levels of the hierarchy. Recent work by Hodgkin et al .(2002) has explored novel sensitivity plots which

one hopes will soon be incorporated into the nextversion of V.I.S.A.

4.3. Web-HipreWeb-Hipre has been developed by Raimo Ha ¨ ma ¨ -la ¨ inen and programmed by Jyri Mustajoki. It isbased upon an earlier MS DOS package, Hipre3+. It implements both MAVT and AHPmethodologies for supporting decisions. As its

name suggests, Web-Hipre is a web-based packagewritten in Java and part of the Decisionarium site(Ha ¨ ma ¨ la ¨ inen, 2003).5 As a web-based application,there is no need to download and install anysoftware in order to use it. It can be accessed from

Internet anywhere in the world. Although it hasbeen used in many applications, see, e.g. Musta-

joki et al . (2002), the package, as is the case for thewhole decisionarium site, has been built with astrong emphasis as a research and education tool.Web-Hipre’s great advantage is that it is deployedvia the Web. Anyone with a browser can run it.This alone makes it excellent for teaching. Forconsultancy and decision conferencing, it is betterto buy and run the program locally, as interactingwith local files is less straight forward from thewebsite. In one sense, the help function is one of the most substantial available, since the decisio-

narium site offers e-learning tools in addition tomore conventional help facilities within Web-Hipre itself. There are no report writing featuresat the time when the test was carried out(November 2003): however some have been builtinto a version of Web-Hipre used in nuclearemergencies (Bertsch et al ., 2005).

4.3.1. Problem structuring. Layout of the valuetree construction window looks like a table with-out grids. Each tree element occupies one tablecell. Although elements can be moved from onecell to another, the software limits the layout of thetree much more than the other packages weexamined. There is no support for problemstructuring other than the fact that attributes canbe placed onto the screen and linked into the valuetree later. The tree has to be built left to right withleft most element always representing the ‘top’attribute. Moreover, the alternatives are includedas the lowest level}rightmost}nodes in the tree.In this respect, Web-Hipre shares a perspective ondecision modelling with AHP. Moreover, thealternatives must each be linked to every leaf

attribute, which can make the tree look very busyif not messy (see Figure 5). Interestingly, Web-Hipre allows more than one top goals. This meansthat Web-Hipre allows two or more value trees tobe built on the same screen. Those trees can havesome common attributes and alternatives. It is aunique feature not found in other packages.

5www.decisionarium.tkk.fi




9/16

4.3.2. Value elicitation. Attribute scores are inputinto Web-Hipre via a table and there is little directsupport for elicitation. It is possible to assign amarginal value function. The shape of a valuefunction can be linear, piecewise linear or expo-nential. Also value functions can be non-mono-tonic. In contrast with some other packages, it ispossible to set points on the value function exactly,but this operation is somewhat less intuitive for

beginners. Web-Hipre does not allow attributes tobe scored with qualitative value judgements,except that in the sense of paired comparisons inthe AHP screens.

The input to Web-Hipre supports five methodsfor assigning attribute weights: Direct, Smart,Swing, Smarter and AHP. It is possible to mixAHP and MAVT elicitation, reflecting the view of Raimo Ha ¨ ma ¨ la ¨ inen that the two methodologies

have more in common than some other may think(Salo and Ha ¨ ma ¨ la ¨ inen, 1997).

4.3.3. Sensitivity analysis. There is only one typeof graph for sensitivity analysis in Web-Hipre. Thegraph shows the relationships between the scoresof alternatives on a selected attribute and theweight of the selected sub-attribute: see Figure 6for a similar plot from Expert Choice. The current

weight is marked with a fixed vertical line. Clickingthe mouse anywhere on the graph will generateanother vertical line at the clicked position tosimulate the effect of the weight change. However,the package lacks the interactivity of V.I.S.A onthe one hand, and the clarity of screen design of HiView, on the other. Notably, Web-Hipre is theonly one of the five packages not to providedPareto plot functionality; a tool that many of us

Figure 5. The value tree plus linked alternatives in Web-Hipre.




10/16

use to explore issues such as cost benefit compro-mises (see e.g. French et al ., 1992).

4.4. Expert ChoiceThe first version of Expert Choice6 was developed

by Professors Forman and Saaty in 1983. Theyrealized that the rate of adoption of AHP, thedecision analytic methodology developed by Saaty(1980), would be greatly enhanced by the avail-ability of user-friendly software. They were right intheir assessment and one of the great strengthsthat the AHP community have had over the yearsis the availability of excellent software. ThusExpert Choice has a long history and a large usercommunity which have ensured that now it is avery mature and well designed piece of software.Now in addition to AHP, Expert Choice offerssupport to a wider range of linear weighting

methods, including direct weighting and non-linear attribute value functions.

4.4.1. Problem structuring. Expert Choice has twodifferent interfaces for structuring a value tree: theTreeView pane and the ClusterView pane. In bothpanes, attributes can be easily sorted, automati-cally organized and moved around into differentbranches of the tree. It also has a feature which isnot seen in many other packages: the ProConpane. This pane is used to support the formationof the value tree. Pros and Cons can be enteredinto the ProCon pane at any location and sortedautomatically or moved around manually afterbeing added. Then the pros and cons can beconverted into attributes. The conversion ismanual. To convert, the ProCon pane and oneof the TreeView and ClusterView panes aredisplayed side by side, the pros and cons can bedrag-dropped into TreeView or ClusterView paneto become an attribute in the value tree. Users areprompted to rename the Pro or Con into a properattribute name during the process. Finally, forthose of us susceptible to losing data in the highlyunstructured process of problem formulation,Expert Choice has excellent data backup function-alities to prevent accidental data losses, such asfrom program crash, which is an important featureto corporate users.

4.4.2. Value elicitation. There are three differentinterfaces for facilitating pairwise comparisons:

ratio (such as attribute x being as three times moreimportant as attribute y), verbal expression (suchas attribute x being moderately important thanattribute y), and two sliding bars with onerepresenting the importance of attribute x and

the other that of attribute y. For scoring analternative on a leaf attribute, in addition topairwise comparison, the user can directly score analternative on an attribute by using a numberbetween 0 and 1, or enter a number in originalattribute unit or a verbal rating using a pre-definedgrade. For those attributes assessed by methodsother than pairwise comparison and direct scoring,value functions need to be defined and ExpertChoice provides such interfaces to do so. Thosefeatures show that Expert Choice is no longer justan implementation of AHP. For eliciting theattribute weights, the pairwise comparison inter-

face is automatically presented to users; but othermethods are also available, such as direct assign-ment of weights. Throughout, the weights andscores are clearly treated and presented separately.

4.4.3. Sensitivity analysis. Expert Choice providesseveral graphs for analysing the effects of weightchanges on the ranking. Some are standard, seeFigure 6. As pioneered by V.I.S.A, some areinteractive: e.g. weights, represented by bars orlines, can be dragged to desired levels, while thescores of the alternatives on a selected attributewill follow the weight changes dynamically. There

are two further types of graphs which also respondto the weight changes. One is for comparing twoalternatives head-to-head on a group of attributes.The other is for comparing all alternatives on twoattributes in a two-dimensional plane, each dimen-sion representing an attribute, which is ideal forcost–benefit or cost effectiveness analysis.

4.5. Logical DecisionsLogical Decisions for Windows7 was one of thefirst packages to provide support for both MAVTand AHP methodologies. It also was one of the

first to allow for some uncertainty in the attributevalues via probabilistic approaches and implemen-ted a multi-attribute utility model, includingassessment of interaction weights (French andRios Insua, 2000; Keeney, 1992; Keeney andRaiffa, 1976). While all the other four packageshave very strong connections with active research

6www.expertchoice.com 7www.logicaldecisions.com




11/16

groups in academia; Logical Decisions is essen-tially a commercial package produced by anindependent software house and consultancy.

4.5.1. Problem structuring. The package uses verydifferent terminology from much of the MCDMliterature, referring to all (leaf) bottom levelattributes as measures and middle layer attributesin a value tree as goals. It uses a level of a measureinstead of a value of an alternative on an attribute.It calls a value function a common unit. It islogically clearer than the other packages todistinguish bottom-level attributes from middle

layer attributes as they are indeed different and aretreated differently in all packages and underlyingdecision analytic methodologies.

In structuring a value tree, which is plotted leftto right, attributes can be copied and pasted, butcannot be moved by dragging and dropping. Mostof the operations related to the tree structuringhave to be done in other windows, taking more‘clicks’ to build the same tree than in otherpackages tested. There is support for zooming,which is necessary since the goals and measures aredisplayed in quite large text boxes, which never-theless cannot display long attribute names very

well, see Figure 7. This makes the tree difficult toview in decision conferences and also makes largetrees difficult to include in reports.

4.5.2. Value elicitation. The values of alternativeson bottom-level attributes (measures) are enteredthrough a matrix, which is similar to the V.I.S.Apackage. Unlike the other packages, it allows notonly a single number as an element of the matrix,

but also a probability distribution. If an alter-native has such uncertain attribute values, MonteCarlo simulation is used in the evaluation. Theshape of value functions in Logical Decisions canbe linear, a default setting, piecewise linear, orcurved, but it has to be monotonic. Valuefunctions can be input through several interfaces,according to the value function elicitation methodused, such as direct assessment, the bisectionmethod, and AHP. Compared with the otherpackages, the shape of a value function is less easyto control when using the direct method.

There are six methods for eliciting attribute

weights: Tradeoffs, Direct, Smart, Smarter, Pair-wise and AHP. The Tradeoffs method is not seenin any other packages. It asks the decision makerhow much she would sacrifice on one attribute inorder to gain a certain amount on another. It thencalculates the weight ratio of the two attributesfrom the tradeoffs and by taking into account thevalue range of the two attributes. It captures thespirit of the value trading approaches, AHP inLogical Decisions is applied differently to otherpackages. Instead of considering a family of attributes, it considers all bottom-level attributessimultaneously, which makes the comparison

matrix very large and much less user-friendly.In Logical Decisions, different decision makers

are able to have different sets of attribute weightsand different value functions. This is useful whenthere are multiple stakeholders involved in adecision problem. However, the current versiondoes not provide a synthesized group opinion, butthere is a full Logical Decisions for Grouppackage.

Figure 6. A sensitivity plot produced in Expert Choice.




12/16

Figure 7. A hierarchy in Logical Decisions.




13/16

4.5.3. Sensitivity analysis. Logical Decisions pro-vide similar types of graphs for sensitivity analysisas in Expert Choice, but only one of the types isinteractive, which uses interactive bars to representweights on the left pane and alternative scores on

the right. The other types do not facilitate userinteractions to change weights.

5. COMPARISON AND DISCUSSION

Perhaps our most immediate conclusion fromexploring all five packages}and a few othersbesides}is that all provide excellent support forthe decision analytic process beginning withproblem formulation and continuing through toevaluation and report writing. Much functionalityis common between the packages and there has

been convergence in terms of the methodologiessupported, most notably in the joint support of AHP and MAVT analyses. Even HiView whichdoes not have any AHP functionality per se, nowsupports pairwise comparisons in the form of theMacbeth approach to elicitation. Only V.I.S.Afocuses solely on support for MAVT methodolo-gies. And that perhaps is an unfair statement,because Valerie Belton and her students haveexperimented with many extensions to V.I.S.A,including a fuzzy version (Koulouri and Belton,1998). Nonetheless, there are distinctions betweenthe packages and their origins still shows through

in some of their functionality and interfaces. Let usconsider the fit with the three cases of decisionprocesses identified in Section 2.

5.1. Case A: analysis conducted by the decisionmakers themselvesIf the decision maker is skilled in decision analysis,each package provides substantial support, andthere is little to choose between them. For theuntrained decision maker, things are not so clear.The methodology of decision analysis has asophistication that belies the simplicity of theweighted sum of attribute scores model which lies

at the heart of decision analysis. It is too easy toinput numbers, obtain a ranking and adopt thehighest ranking alternative with only a perfunctoryexploration of the sensitivity of the ranking to theinputs. Sensitivity analysis and Pareto plots,indeed all the techniques of decision analysisshould be used to challenge thinking and catalysecreativity (French, 2003; Phillips, 1984). OnlyWeb-Hipre via the decisionarium website provides

substantial training support for decision makersnew to decision analysis. Some of the functional-ities offered by, e.g. Web-Hipre or LogicalDecisions, can be inconsistent because they allowthe user to mix MAVT and AHP without checking

the theoretical compatibility of the implied opera-tions. Modern object-oriented programming meth-ods should enable the compatibility of methodsused in a decision analysis to be policed effectively(Liu and Stewart, 2003). Then there is the issue of whether the approach should be value focused oralternative focused (Keeney, 1992; Wright andGoodwin, 1999). The opening screens of ExpertChoice lead one into a brainstorming sessionwhich may avoid too great an early focus onalternatives. The other packages are effectivelyneutral in supporting both approaches. In short,all the packages provide excellent support for

decision analytic calculations but little in the wayof support of the decision making process itself.This is perhaps surprising because MAUD, one of the earliest MCDM packages, did emphasizeprocess support (Humphreys and McFadden,1980). Klein (1994) has developed the theoryfurther, providing an automated explanation of the implications of decision analytic calculations:see also Papamichail and French (2003). Theimportance of providing guidance and explana-tions in decision support software which is drivenby the decision makers themselves has been shownby Benbasat and his co-workers (Dhaliwal and

Benbasat, 1996; Mao and Benbasat, 2000). Thuswe would hope that one of the next areas of functionality to be included into MCDM packagesis support for the decision making process itself:see also Papamichail and Robertson (2004).

5.2. Case B: decision conferencingIn the case of decision conferencing when theanalyses are run in plenary by the facilitator/analyst and his team, the requirements on thesoftware are subtly different. The analyst team willhave the expertise to run the software and thefacilitation of the process means that the software

need not support the process itself. However, thefact that the software is projected for all the groupto see does have implications. The plots and textneed to be easy to read and the screens should beclear of distractions. Those of us who have workedwith groups and projected complex software knowthe difficulties caused if the screen becomescluttered with windows and there are optionsavailable behind unexplained buttons. One is




14/16

forever pointing at the right place on the screenand dragging attention back to the issue at hand.In this respect, HiView still shows its origins as asoftware tool purpose designed to support decisionconferences, though in fairness, the distinction in

this respect between the five packages is not sogreat as it was in previous versions. Requisitemodelling involves beginning simple and addingcomplexities later; thus there is need for thesoftware to allow the model to be edited andnew attributes and alternatives added. Thesepackages meet that requirement to varying degreesas noted earlier.

It is worth noting that none of the software hasreally been designed for projection. What happensis that the PC’s screen is simply diverted to a dataprojector. What the group sees is precisely whatthe analyst sees. One development that would be

very welcome is specific windows were sent to thedata projector without all the gory details of menus and buttons on the analyst’s screen. Theanalyst could have complex analyses available tohim without distracting the group with all thedetails, showing them only the plots that will trulyinform their understanding. This is eminentlypossible, but not available in any of the currentversions of the software.

5.3. Case C: use in consultancyAs in case B, the software will be driven by aexperienced decision analyst. Thus there is little

need to either support the process or avoiddistraction by keeping the screens clear. What isneeded is sufficient functionality for the analysisitself and the means to include that in theconsultancy report. Some packages, as noted,provide a reporting function; others rely onexporting to Excel spreadsheets or the Windowsclipboard to provide the means of developingtables and plots for the report. The route via Excelis useful because Excel offers a range of plots tosupplement those in the software itself. The routevia the clipboard can be problematic because itoften requires far more Mb than are strictly

needed and even in the latest versions of WindowsXP sometimes the clipboard loses lines. Web-Hipre has the greatest difficulty in this respectbecause it is Java based and cannot write to theuser’s filestore at all: the clipboard provides theonly route. When reporting functions do exist,they are useful but tend to reproduce outputavailable onscreen. It would be useful to askwhether the printed medium, being non-interac-

tive, requires a different range of plots and tablesto those use in live analysis.

5.4. Concluding remarksThere are two general areas in all packages coulddo rather more. The first is in problem formulationand structuring. While Expert Choice does providesupport for brainstorming a list of entities for theanalysis and then building the model, none of thepackages really draw upon the growing range of problem formulation methodologies, often re-ferred to as soft OR (Belton and Stewart, 2002;French et al ., 1998; Pidd, 1996, 2004; Rosenheadand Mingers, 2001). We have noted that Beltonet al . (1997) have investigated the interfacebetween a formulation phase supported primarilyby cognitive modelling followed by an evaluationphase supported by V.I.S.A, but there is poten-tially much more that can be done in this regard.The second point, which perhaps reflects theparticular interests of one of us (French, 2003), isthat none of the packages explores the full range of sensitivity analysis that is possible (Hodgkin et al .,2002; Mateos et al ., 2003; Rios Insua, 1990, 1999).Perhaps all packages could move in these direc-tions: certainly it seems to us that identifyingpotentially optimal alternatives would be a veryuseful feature.

Four of the packages have strong academicconnections: HiView, V.I.S.A, Expert Choice and

Web-Hipre. This seems to manifest itself in acertain vibrancy in the way in which newfunctionality has been introduced at each version.Logical Decisions began over a decade ago withthe most advanced functionality in that it allowednon-independent trade-offs, but it has developedlittle in functionality since then.8 It seems to usthat the next phase of development should be todevelop functionality relating to different contextsof use, either marketing different versions of thesoftware or providing different ‘skins’ that can beadopted according to how the software is beingused.

We would advise the potential user of MAVTsoftware to give careful consideration to theintended context of use. There is much more tosuch software than the calculation of weightedsums and the presentation of attractive plots.Their purpose is to support the process of decision

8To be fair, we should remember that there is a group-enabled version which we have not examined.




15/16

analysis from problem formulation to reportwriting and implementation. Different contextsrequire different emphases on the different stagesand thus some software fits better with one contextthan others. Evaluation of MAVT software

requires a much broader perspective than onefocused on detailed functionality: see also Beltonand Hodgkin (1999). We hope that this paper’svalue will stem in part not just from the detailsabove, but its approach and recognition thatdecision analytic software need do more thanfacilitate calculation and analysis. It must alsocohere with the decision culture and process. Wehope too that our work will contribute to thedesign of future decision analytic software.

ACKNOWLEDGEMENTS

We are grateful to many people for discussions: wemention Valerie Belton, Raimo Hamalainen,Larry Phillips, Theo Stewart and Jian-Bo Yang;and thank them along with many others. Refereesoffered very helpful advice on an earlier version of this paper. However, the interpretations we offerare ours alone}as are the errors. Moreover, wetrust that we do not offend the developers of thedifferent packages by our reflections on themotivation and origins of the packages. Thepackages evaluated were in some cases, evaluationdownloads provided on vendor’s websites.

REFERENCES

Ackoff RL. 1974. Redesigning the Future: A SystemsApproach to Societal Planning. Wiley: New York.

Bana e Costa CA, Vansnick J-C. 2000. Cardinal valuemeasurement with Macbeth. In Decision Making:Recent Developments and Worldwide Applications,Zanakis SH, Doukidis G, Zopounidis C (eds).Kluwer: Dordrecht, 317–329.

Barclay S. 1984. Hiview Software Package. Decision

Analysis Unit, London School of Economics.Bazerman M. 2002. Managerial Decision Making. Wiley:

New York.Belton V. 1984. The use of a simple multi-criteria model

to assist in selection from a short-list. Journal of theOperational Research Society 36: 265–274.

Belton V, Ackermann F, Shepherd I. 1997. Integratedsupport from problem structuring through to alter-native evaluation using COPE and VISA. Journal of Multi-Criteria Decision Analysis 6: 115–130.

Belton V, Hodgkin J. 1999. Facilitators, decisionmakers, D.I.Y. users: is intelligent multi-criteriadecision support for all feasible or desirable. EuropeanJournal of Operational Research 113(2): 247–260.

Belton V, Stewart TJ. 2002. Multiple Criteria DecisionAnalysis: An Integrated Approach. Kluwer AcademicPress: Boston.

Belton V, Vickers S. 1988. VISA Software Package.Department of Management Science, University of Strathclyde.

Bertsch V, French S, Geldermann J, Ha ¨ ma ¨ la ¨ inen RP,Papamichail KN, Rentz O. 2005. Multi-criteriadecision support and evaluation of strategies forenvironmental remediation management. Omega,accepted 2005.

Dhaliwal JS, Benbasat I. 1996. The use and effects of knowledge-based system explanations: theoreticalfoundations and a framework for empirical evalua-tion. Information Systems Research 7(3): 342–362.

Eden C, Ackermann F. 1998. Making Strategy: The

Journey of Strategic Management. Sage: London.Eden C, Radford J (eds). 1990. Tackling Strategic

Problems: The Role of Group Decision Support. Sage:London.

French S (ed.). 1988. Readings in Decision Analysis.Chapman & Hall: London.

French S. 2003. Modelling, making inferences andmaking decisions: the roles of sensitivity analysis.TOP 11(2): 229–252.

French S, Geldermann J. 2005. The varied contexts of environmental decision problems and their implica-tions for decision support. Environmental Science and Policy 8: 378–391.

French S, Kelly GN, Morrey M. 1992. Decision

conferencing and the International Chernobyl Project.Radiation Protection Dosimetry 12: 17–28.

French S, Rios Insua D. 2000. Statistical DecisionTheory. Arnold: London.

French S, Simpson L, Atherton E, Belton V, Dawes R,Edwards W, Ha ¨ ma ¨ la ¨ inen RP, Larichev O, LootsmaFA, Pearman AD, Vlek C. 1998. Problem formulationfor multi-criteria decision analysis: report of a work-shop. Journal of Multi-Criteria Decision Analysis 7:242–262.

Goodwin P, Wright G. 2003. Decision Analysis forManagement Judgement. Wiley: Chichester.

Ha ¨ ma ¨ la ¨ inen RP. 2003. Decisionarium}aiding deci-sions, negotiating and collecting opinions. Journal of

Multi-Criteria Decision Analysis 12: 101–110.Hodgkin J, Belton V, Koulouri A. 2002. Intelligent user

support for MCDA: a case study. Working Paper,Department of Management Science, University of Strathclyde, Glasgow Gl 1 QE, UK.

Humphreys PC, McFadden W. 1980. Experiences withMAUD: aiding decision structuring versus bootstrap-ping the decision maker. Acta Psychologica 45: 51–69.

Isitt T. 1990. The sports tourers. Motor Cycle Interna-tional 64: 18–27.




16/16

Kahneman D, Slovic P, Tversky A (eds). 1982.Judgement under Uncertainty. Cambridge UniversityPress: Cambridge.

Keeney RL. 1992. Value-Focused Thinking: A Path toCreative Decision Making. Harvard University Press:Cambridge, MA.

Keeney RL, Raiffa H. 1976. Decisions with MultipleObjectives: Preferences and Value Trade-offs. Wiley:New York.

Klein DA. 1994. Decision-Analytic Intelligent Systems:Automated Explanation and Knowledge Acquisition.Lawrence Erlbaum Associates: New Jersey.

Kleindorfer PR, Kunreuther HC, Schoemaker P. 1993.Decision Sciences. Cambridge University Press:Cambridge.

Koulouri A, Belton V. 1998. Fuzzy multi-criteria DSS:their applicability and their practical value. ResearchReports, Department of Management Science, Uni-versity of Strathclyde, Glasgow.

Liu D, Stewart TJ. 2003. Integrated object-oriented

framework for MCDM and DSS modelling. DecisionSupport Systems 38: 421–434.

Mao J-Y, Benbasat I. 2000. The use of explanations inknowledge-based systems: cognitive perspectives anda process-tracing analysis. Journal of ManagementInformation Systems 17(2): 153–179.

Mateos A, Jiminez A, Rios Insua S. 2003. Modellingindividual and global comparisons for multi-attributepreferences. Journal of Multi-Criteria Decision Analy-sis 12: 177–190.

Maxwell D. 2002. Aiding Insight VI. OR/MS Today29(3): 44–51.

Maxwell D. 2004. Aiding Insight VII. OR/MS Today31(3): 44–55.

Mustajoki J, Ha ¨ ma ¨ la ¨ inen RP, Marttunen M. 2002.Participatory multi-criteria decision analysis withWeb-Hipre: a case of lake regulation policy. SystemsAnalysis Laboratory, Helsinki University of Technol-ogy. http://www.sal.hut.fi/Publications/pdf-files/mmusb.pdf (Visited on 28/10/2003).

Nunamaker JF, Applegate LM, Konsynski BR. 1988.Computer-aided deliberation: model managementand group decision support. Operations Research 36:826–848.

Papamichail KN, French S. 2003. Explaining and justifying the advice of a decision support system:a natural language generation approach. ExpertSystems with Applications 24: 35–48.

Papamichail KN, Robertson I. 2005. Integrating deci-sion making and regulation in the managementcontrol process. Omega 33(4): 319–332.

Phillips LD. 1984. A theory of requisite decision models.Acta Psychologica 56: 29–48.

Pidd M. 1996. Tools for Thinking: Modelling in Manage-ment Science. Wiley: Chichester.

Pidd M (ed.). 2004. Systems Modelling: Theory and Practice. Wiley: Chichester.

Rios Insua D. 1990. Sensitivity Analysis in Multi-Objective Decision Making. Springer: Berlin.

Rios Insua D (ed.). 1999. Sensitivity analysis in

MCDA. Journal of Multi-Criteria Decision Analysis8(3): 117–187.

Rosenhead J, Mingers J (eds). 2001. Rational Analysis for a Problematic World Revisited . Wiley:Chichester.

Roy B. 1996. Multi-Criteria Modelling for DecisionAiding. Kluwer Academic Publishers: Dordrecht.

Saaty TL. 1980. The Analytical Hierarchy Process.McGraw-Hill: New York.

Salo A, Ha ¨ ma ¨ la ¨ inen RP. 1997. On the measurement of preferences in the analytical hierarchy process (withdiscussion). Journal of Multi-Criteria Decision Analy-sis 6(6): 309–343.

Watson SR, Buede DM. 1987. Decision Synthesis:

The Principles and Practice of Decision Analysis.Cambridge University Press: Cambridge.

Wright G, Goodwin P. 1999. Rethinking value elicita-tion for personal consequential decisions (with dis-cussion). Journal of Multi-Criteria Decision Analysis8(1): 3–30.



multicriteria decision making software french2005

Documents