exploring multicriteria elicitation model based on ... · exploring multicriteria elicitation model...

Research ArticleExploring Multicriteria Elicitation Model Based onPairwise Comparisons Building an Interactive PreferenceAdjustment Algorithm

Giancarllo Ribeiro Vasconcelos and Caroline Maria de Miranda Mota

Programa de Pos-Graduacao em Engenharia de Producao (PPGEP) Universidade Federal de Pernambuco (UFPE)Recife 50740-550 Brazil

Correspondence should be addressed to Giancarllo Ribeiro Vasconcelos giancarllounirvedubr

Received 8 March 2019 Accepted 1 June 2019 Published 18 June 2019

Guest Editor Love Ekenberg

Copyright copy 2019 Giancarllo Ribeiro Vasconcelos and Caroline Maria de Miranda Mota This is an open access article distributedunder the Creative Commons Attribution License which permits unrestricted use distribution and reproduction in any mediumprovided the original work is properly cited

Pairwise comparisons have been applied to several real decision making problems As a result this method has been recognized asan effective decision making tool by practitioners experts and researchers Although methods based on pairwise comparisons arewidespread decisionmaking problems withmany alternatives and criteria may be challengingThis paper presents the results of anexperiment used to verify the influence of a high number of preferences comparisons in the inconsistency of the comparisonsmatrixand identifies the influence of consistencies and inconsistencies in the assessment of the decision-making process The findingsindicate that it is difficult to predict the influence of inconsistencies and that the priority vector may or may not be influencedby low levels of inconsistencies with a consistency ratio of less than 01 Finally this work presents an interactive preferenceadjustment algorithmwith the aim of reducing the number of pairwise comparisons while capturing effective information from thedecision maker to approximate the results of the problem to their preferences The presented approach ensures the consistency ofa comparisons matrix and significantly reduces the time that decision makers need to devote to the pairwise comparisons processAn example application of the interactive preference adjustment algorithm is included

1 Introduction

Preference judgment is a key issue in multicriteria decision-making (MCDM)methodsMCDM is a generic term given toa collection of systematic approaches andmethods developedto support the evaluation of alternatives in a context withmany objectives and conflicting criteria [1 2]

Multicriteria methods can be classified into three groupsfirst aggregation methods based on a single criterion ofsynthesis whose main representatives are the multiattributeutility theory the analytic hierarchy process (AHP) andMACBETH The second group is outranking methods suchas the PROMETHEE and ELECTRE methods Finally thethird group consists of the interactive methods such asmultiobjective linear programming (PLMO) [3 4]

Different types of cognitive and behavioral biases playan important role in decision-making (DM) The MCDM

aims to help people make strategic decisions according totheir preferences and an overarching understanding of theproblem [2] and it is subject to various cognitive andprocedural deviations [5] These deviations can occur at allstages of the decision-making process (problem structuringevaluation criteria and alternatives and sensitivity analysis)although they can lead to incorrect recommendations

In additive models inconsistency can be perceived intwo distinct situations of decision maker (DM) judgmentsintercriteria and intracriteria evaluations The intercriteriaevaluation involves the elicitation procedure for determiningthe weights of the criteria where inconsistencies have beenreported such as ratio [6] swing [7] trade-off [2 8 9]and fitradeoff [10] In the intracriterion evaluation a valuefunction is determined for each criterion At this stage somemethods consider a simplistic approach by assuming a linearvalue function such as in Smarts and Smarter [7] or a

HindawiMathematical Problems in EngineeringVolume 2019 Article ID 2125740 14 pageshttpsdoiorg10115520192125740

2 Mathematical Problems in Engineering

Table 1 Number of pairwise comparisons based on the number of criteria and alternatives

Number of Alternatives Number of Alternatives

Number of Criteria

2 3 4 5 6 7 8 9 10 2 3 4 5 6 7 8 9 102 2 4 6 8 10 12 14 16 18 2 2 6 12 20 30 42 56 72 903 3 6 9 12 15 18 21 24 27 3 3 9 18 30 45 63 84 108 1354 4 8 12 16 20 24 28 32 36 4 4 12 24 40 60 84 112 144 1805 5 10 15 20 25 30 35 40 45 5 5 15 30 50 75 105 140 180 2256 6 12 18 24 30 36 42 48 54 6 6 18 36 60 90 126 168 216 2707 7 14 21 28 35 42 49 56 63 7 7 21 42 70 105 147 196 252 3158 8 16 24 32 40 48 56 64 72 8 8 24 48 80 120 168 224 288 3609 9 18 27 36 45 54 63 72 81 9 9 27 54 90 135 189 252 324 40510 10 20 30 40 50 60 70 80 90 10 10 30 60 100 150 210 280 360 450

Reciprocal Transitive Matrix Traditional Method

more sophisticated approach to build a utility function thatexpresses risk attitudes [2] Another group of methods buildsthe value function based on pairwise comparisons (PCs)of several preference statements (questioning) such as theanalytic hierarchy process [11ndash13] and Macbeth [14 15]

A common cognitive deviation in MCDM methodsoccurs when there is a PC of a large number of alternativeswhich requires a great deal of cognitive effort by the DM[16ndash20] In this case several studies have reported concernsrelated to the applicability of this type of procedure insituations where the number of criteria and alternatives isquite large [17 18 20 21] In such situations the numberof PCs made by the DM grows at an alarming rate Thetime that analysts spend with DMs is increasingly scarceand convincing a top executive to spend hours or evendays making PCs of alternatives and criteria is often notfeasible

This paper focuses on the preference judgment of a DMbased on PCs of a qualitative criterion to build a valuefunction For this purpose an analysis was conducted basedon the PC process The AHP is one of the best knownmulticriteria decision aid (MCDA) approaches and is thuswidely used [1 22] AHP is an additive method proposedby Saaty [11ndash13] and since its introduction has attractedincreasing attention from researchers [22ndash26] The methodconverts subjective assessments of relative importance to avector of priorities (value function of one criterion) basedon PCs performed within each criterion The comparativejudgments are made using the fundamental scale devised bySaaty [11] and a consistency logic is applied to check theDMrsquosjudgments [27]

In this article we explore the influence of consistency andinconsistency in preference assessmentsmdashbased on PCsmdashbyperforming an experiment with several individuals to assesstheir preferences based on a given situation A high levelof inconsistency has been verified within the literature (ref-erence) Additionally we evaluate an alternative procedureto assess the DMrsquos preferences The preferences of a DMare assessed based on an interactive procedure of askingquestions and adjusting preferences thus reducing the timespent by the DM and assuming an acceptable level of possibleinconsistencies

This paper is organized as follows The next sectionpresents a review of the literature concerning the causes ofinconsistencies in PCs as well as several solutions Section 3describes the experiment conducted to verify the influenceof inconsistency Lastly a preference adjustment algorithm inPCs is presented

2 Literature Review

Benitez et al [28] have found that human beings make moregood decisions than bad ones throughout their lifetime andthat irrationality is a common cause of bad decisions Theythus proposed a model based on the most likely choice tocapture shifts in the decision-making (irrationality) processto reduce biases caused by inconsistent judgments

A PC uses human abilities such as knowledge andexperience to compare alternatives and criteria in a pairwisemanner and assemble a comparisons matrix [29] Inconsis-tency arises when some opinions of a comparisons matrixcontradict othersTherefore it is important to check the con-sistency of opinions when performing a series of calculationsto arrive at the value of the consistency ratio (CR) whichindicates the consistency of the comparisons matrix For thePCs matrix it is desirable that the CR for any comparisonsmatrix be less than or equal to 010 [25 30]

The number of n PCs is 119899(119899 minus 1)2 hence groupingand hierarchy structure should be used for larger n [21]Bozoki et al [20] performed a controlled experiment withuniversity students (N = 227) which enabled them to obtain454 PCsmatricesThey conducted experimentswithmatricesof different sizes from different types of multicriteria decisionsupport methods and found that the size of a matrix affectsthe DMrsquos consistency (measured by inconsistency indices)An increasing matrix size leads to greater inconsistency

Consider the multicriteria problem where the DM mustmake a PC alongside five criteria and ten alternatives In sucha situation the DM will have to allow time to perform 225evaluations (Table 1) to compare the alternatives within eachcriterion If we insert twomore criteria this number increasesto 315 evaluations Ten criteria with ten alternatives require495 evaluations The number of alternatives is certainly thelargest source of comparisons whereas all alternatives are

Mathematical Problems in Engineering 3

compared considering each criterion Equation (1) calculatesthe number of comparisons (CN) in the traditional method[21]

119862119873 = 1198962 [119899 (119899 minus 1)] (1)

where119862119873 is the number of PCs of the traditionalmethod119896 is the number of criteria and 119899 is the number of alternatives

Alternatively Equation (2) calculates the number of com-parisons in the Reciprocal Transitive Matrix (RTM) Methodthis method was initially proposed by Koczkodaj and Szy-bowski [31] and called Pairwise comparisons simplified

119862119873lowast = 119896119899 minus 1 (2)

where 119862119873lowast is the number of PCs of the RTMMethodPrevious studies have already proposed several solutions

to this problem Weiss and Rao [32] for example haveproposed reducing the required number of questions to eachDM through the use of incomplete blocks administered todifferent responders Furthermore Harker [33] developedan incomplete PC technique that aims to reduce this effortby ordering the questions in decreasing informational valueand terminating the process when the value of additionalquestions decreases below a certain level

A comparisons matrix with a CR equal to zero is repre-sentative of a fully consistent DM this matrix is known as aRTM or a consistent matrix [31 34] The process of buildinga reciprocal transitive comparisons matrix requires the DMto conduct comparisons of only one line of the comparisonmatrix The remaining values are determined by obeying themathematical assumptions of a RTM The resulting matrixcomparison is consistent based on the comparison made bythe DMrsquos criteria or alternatives with respect to a preselectedcriterion or alternative

The process of building a RTM dramatically reduces thenumber of comparisons performed by the DM as shown inTable 1 For example consider a problem with seven criteriaand eight alternatives Using the traditional method theDM performs 196 comparisons while using the process ofbuilding a RTM reduces the number of comparisons to 49

The effort of DMs is greatly reduced when building aRTM which may reflect a more careful evaluation of PCsOn the other hand an error or very imprecise evaluation ofthe initial comparison can cause distortion in the decisionprocess [21] Kwiesielewicz and Uden [35] for examplehave found that the relationship between inconsistent andcontradictorymatrices of data exists as a result of the PCTheconsistency check is performed to ensure that judgments areneither random nor illogical The authors reveal that even ifa matrix successfully passes a consistency test it can be con-tradictory Thus an algorithm for checking contradictions isproposed

21 Issues with Pairwise Comparisons Numerous studieshave examined problems in the use of the PC [20 36ndash38] Some authors have dedicated their research to theseproblems Some problems are related to this work especially

in the use of ratio scales and eigenvalue as a measure ofinconsistency

With regard to the ratio scale problem Ishizaka et al [38]Salo and Hamalainen [39] Donegan et al [40] and Lootsma[41] proposed new ratio scales to solve problems associatedwith the use of this type of scale Goepel [37] and Koczkodajet al [41] moreover conducted research comparing thescales but unanimously determined that the scales havelimitations in their maximum value which restricts theinterpretation of the DM However using larger scales mayincrease uncertainties

In some cases an unlimited scale is required especiallywhen comparing measurable entities such as distance andtemperature [42]

Another issue is the eigenvalue problem Some authorsagree that while the eigenvalue is used as a good approxi-mation for consistent matrices there are expressive resultsregarding the existence of better approximations such asgeometric means [42ndash44] Some recent studies have foundthat the use of geometric means [45] is the only method thatsatisfies the principles of consistency and is immune to theproblem of reversal order [45]

A PC matrix refereed here as M presents the relationsbetween n alternatives M is reciprocal if 119898푖푗 = 1119898푗푖 forall 119894 119895 = 1 119899 M is consistent if it satisfies the transitivityproperty 119898푖푗 = 119898푖푘119909 119898푘푗 for all 119894 119895 119896 = 1 119899 [36 46ndash48] Note that while every consistent matrix is reciprocal theconverse is in general false [31]

When an n times n matrix119872 is not consistent it is necessaryto measure the degree of inconsistency One popular incon-sistency index proposed by Saaty [11] is defined as follows

119868119888 (119872) = 120582푚푎푥 minus 119899119899 minus 1 (3)

120582푚푎푥 is the principal eigenvalue ofMLet 119877119868 denote the average value of the randomly obtained

inconsistency indices which depends not only on 119899 but on themethod of generating random numbers tooThe consistencyratio (CR) ofM indicating inconsistency is defined by [11 13]

119862119877 (119872) = 119868119888 (119872)119877119868 (4)

If the matrix is consistent then 120582푚푎푥 = 119899 so 119868119888(119872) = 0and 119862119877(119872) = 0

Saaty [11 13] introduced eigenvalues to verify the consis-tency of PC matrices Furthermore Saaty considers a matrixto be consistent when 119862119877(119872) le 01 meaning that 10 of thedeviation of the largest eigenvalue of a given matrix from thecorresponding eigenvalue of a randomly generated matrixSaatyrsquos definition of consistency is good for any array orderThe main disadvantage of Saatyrsquos definition of consistency isthe rather unfortunate threshold of 10 [47]

Koczkodaj [47] introduced a new way of measuringinconsistencies of a PCs matrix based on the measure of thelowest deviation of an 119872 matrix in relation to a consistentRTM The interpretation of the measure of consistencybecomes easier when we reduce a basic reciprocal matrix toa vector of three coordinates [119886 119887 119888] We know that 119887 = 119886 lowast 119888


is valid for each RTM thus we can always produce threeRTM (vectors) by calculating a coordinate of the combinationof the two remaining coordinates These three vectors are[119894 119887 119888] [119886 119886 lowast 119888 119888] and [119886 119887 119905] The consistency measure(CM) is defined as the distance relative to the nearest RTMrepresented by one of these three vectors for a given metricConsidering Euclidean (or Chebysheff) metrics we have [47]

119862119872 = min(111988610038161003816100381610038161003816100381610038161003816119886 minus11988711988810038161003816100381610038161003816100381610038161003816 1119887 |119887 minus 119886119888|

111988810038161003816100381610038161003816100381610038161003816119888 minus11988711988610038161003816100381610038161003816100381610038161003816) (5)

Koczkodajrsquos consistency index [36 42 46 47 49ndash54] notonly measures the inconsistency but it also shows where it islarger thus guiding the DM to reassess and correct the incon-sistency Additionally Bozoki and Rapcsak [46] investigatedsome properties of the PCs matrix inconsistencies of Saatyand KoczkodajThe results indicate that the determination ofinconsistency requires further study

Considering Saatyrsquos inconsistency index some questionsremain to be answered [46]What is the relationship betweenan empirical matrix of human judgments and a randomlygenerated one Is an index obtained from several hundredrandomly generated matrices the correct reference pointto determine the level of inconsistency of the matchedcomparisons matrix constructed from human decisions fora real decision problem How can one take matrix size intoaccount more precisely

ConsideringKoczkodajrsquos consistency index an importantissue seems to be the elaboration of the thresholds in higherdimensions or the replacement of the index with a refinedrule of classification [46]

22 Correcting Inconsistencies Wadjdi et al [16] investigatedthe importance of data collection using forms to ensure theconsistency of comparisonmatricesThe authors have proventhat the proposed form can guarantee data consistency inthe PCs matrix The form can also be used in any othertechniques such as Fuzzy-AHP TOPSIS and Fuzzy-TOPSIS

Xu [55] defined the concepts of incomplete reciprocalrelation additive consistent incomplete reciprocal relationand multiplicative consistent incomplete reciprocal relationand subsequently proposed two goal programming modelsbased on additive consistent incomplete reciprocal relationand multiplicative consistent incomplete reciprocal relationrespectively to obtain the incomplete reciprocal relationpriority vector

Pankratova and Nedashkovskaya [56] employed a com-puter simulation to compare these methods of consistencyimprovement without the participation of a DM It has beenfound that taking an inadmissible inconsistent matrix ofcomparison with the consistency ratio equal to CR = 02or CR = 03 consistency improvement methods can helpdecrease inconsistency up to admissible level CRle01 for nge5However the results reveal that these methods are not alwayseffective Drawing near to admissible inconsistency does notguarantee closeness to the real priority vector of decisionalternatives

A method for constructing consistent fuzzy preferencerelations from a set of n - 1 preference data was proposed by

[57] The authors stated that with this method it is possibleto ensure better consistency of the fuzzy preference relationsprovided by the DMs However this approach differs fromour interactive preference adjustment insofar as our analysisseeks to obtain information from the DMs whereas theprevious approach was mathematical

Voisin et al [58] have noted that several consistencyindices for fuzzy PC matrices have been proposed within theliteratureHowever some researchers argue thatmost of theseindices are not axiomatically grounded which may lead todeviations in the results The authors of the present paperovercome this lack of an axiomatically grounded index byproposing a new index referred to as the knowledge-basedconsistency index

Benıtez et al [59] proposed the use of a technique thatprovides the closest consistent matrix given inconsistentmatrices using an orthogonal projection on a linear space Inanother paper [60] the same authors proposed a frameworkthat allows for balancing consistency and DMsrsquo preferencesfocusing specifically on a process of streamlining the trade-off between reliability and consistency An algorithm wasdesigned to be easily integrated into a decision support sys-tem This algorithm follows a process of interactive feedbackthat achieves an acceptable level of consistency relative to theDMsrsquo preferences

Benıtez et al [61] also proposed a method for achievingconsistency after reviewing the judgment of the comparisonsmatrix decider using optimization and discovering that itapproximated the nearest consistent matrixThis method hasthe advantage of depending solely on n decision variables (thenumber of elements being compared) being less expensivethan other optimization methods and being easily imple-mented in almost any computing environment

Motivated by a situation found in a real application ofAHPNegahban [62] extends previouswork on improving theconsistency of positive reciprocal comparative matrices byoptimizing its transformation into almost consistent matri-ces An optimization approach is proposed and integratedinto the Monte Carlo AHP framework allowing it to solvesituations where distinct or almost insufficient matricesare generated through the direct sampling of the originalpaired comparison distributionsmdasha situation that prohibitssignificant statistical analysis and effective decision-makingthrough the use of the traditional AHP Monte Carlo

Brunelli et al [63] found evidence of proportionalitybetween some consistency indices used in the AHP Hav-ing established these equivalences the authors proposed aredundancy elimination when checking the consistency ofthe preferences

Xia and Xu [64] proposed methods to derive intervalweight vectors from reciprocal relations to reflect the incon-sistency that exists when the DMsrsquo preferences are taken intoaccount for alternatives (or criteria) The authors presentedprogramming models to minimize the inconsistency basedon multiplicative and additive consistency

Benıtez et al [27] again proposed a formula thatprovides in a very simple manner the consistent matrixclosest to a reciprocal (inconsistent) matrix Additionallythis formula is computationally efficient since it only uses


Choose the holiday destination

C1 Costs C3 Tourist AttractionsC2 Hotel Features

A5 NatalA4 SalvadorA3 FortalezaA2 FlorianopolisA1 Rio de JaneiroAlternatives

Criteria

Objective

Figure 1 Criteria hierarchy of the holiday destination choice problem

sums to perform the calculations A corollary of the mainresult reveals that the normalized vector of the vector whosecomponents are the geometric means of the rows of acomparisons matrix only produces the priority vector forconsistent matrices

The evaluation of consistency has also been studied usingimprecise data Thus the theory of fuzzy numbers has beenapplied to MCDA methods to better interpret the judgmentof DMs The fuzzy AHP has been widely applied and theevaluation of consistency in such situations has been theobject of research Bulut et al [65] proposed a fuzzy-AHPgenericmodel (FAHP)with pattern control consistency of thedecision matrix for group decisions (GF-AHP)The GF-AHPimproved performance using direct numerical inputswithoutconsulting the decider In practice some criteria can easily becalculated In such cases consulting with the DM becomesredundant

Ramık and Korviny [66] presented an inconsistencyindex for matrices of PC with fuzzy entries based on theChebyshev distance However Brunelli [67] showed that theChebyshev distance may fail to capture inconsistency and asa result should be replaced by the most convenient metricLiu et al [68] reported that the study of consistency is veryimportant since it helps to avoid erroneous recommenda-tions Accordingly they proposed a definition of reciprocalrelationships that privileged triangular fuzzy numbers thatcan be used to check fuzzy-AHP comparison matrices

Xu and Wang [69] extended the eigenvector method(EM) to prioritize and define a multiplicative consistencyfor a list of incomplete fuzzy prioritizations The authorspresented an approach to judge whether this relationship isacceptable or not and subsequently developed a prioritizationratio of consistency to fuzzy incomplete similar to the oneproposed by Saaty

Koczkodaj et al [49 51 54] used incomplete comparisonmatrices to create a system that monitors the inconsistency ofthe DM at each step of the PC process informing the DM if itexceeds a certain threshold of inconsistencyThe algorithm isconstructed by locating themain reason for the inconsistencyOur research differs from the system proposed in that itreduces the space in which the DM would be inconsistent byintroducing an interactive process where in the first stagethe DM is asked to complete only a single line of the matrixFurthermore with the help of an algorithm we have captured

and corrected deviations from the matrix in relation to theDMrsquos preferences

A recent study examined the notion of generators of thePCsmatrix [31]The proposedmethod decreased the numberof PCs from 119899(119899 minus 1)2 to 119899 minus 1 which is similar to ourapproach except that we use an algorithm that identifies andcorrects deviations in the PCs matrix to better capture theDMrsquos preferences

3 Checking the Influence of Inconsistency

We have performed an experiment of a decision-makingsituation among students and staffs of a university using aprocedure based on the PCs matrix The experiment aimsto identify the influence of the number of PCs on the com-parisons matrix consistency in accordance with the researchconducted by Bozoki et al [20] To measure consistency weuse the consistency ratio [11ndash13]

The experiment consisted of presenting different locationoptions for a summer holiday to DMs DMs completed acomparisons matrix whose number of alternatives grew asthe DM concluded a comparative stageThe alternatives werethe cities of Rio de Janeiro Florianopolis Salvador NatalFortalezaMaceio Joao Pessoa Aracaju Vitoria Recife PortoAlegre and Curitiba as presented in Figure 1

While the experiment surveyed a sample of 180 peopleonly 76 answered it completely Of the 76 who responded infull only 30 dedicated the answersTheothers only completedthe questionnaire without any criteria or attention and werevery inconsistent even in the initial stages when the numberof alternatives and the cognitive effort were minimal

The evaluation process began with the comparison ofthree possible alternatives (cities) Then 4 alternatives 5alternatives and eventually up to 10 alternatives were evalu-ated Individuals reported that as the number of alternativesincreased their ability to discern the difference betweenthem decreased Lastly the surveyed individuals were askedwhether they agreed with the ranking obtained 67 of theindividuals reported that they had a different perspective thanthe presented results

The results of the experiment indicate that inconsistencyin the comparisons matrix increases as the number ofalternatives increases that is as the comparisons increase (seeFigure 2)


026024022020018016014012010008006004002

Mean Plot of Consistency Ratio grouped by Number of AlternativesSpreadsheet20 10vlowast358cMean = 00575+00128lowastx

Number of AlternativesMeanMeanplusmn095 Conf Interval

3 4 5 6 7 8 9 10 11 12

Con

siste

ncy

Ratio

Figure 2 Behavior of the consistency ratio in relation to the number of alternatives

Table 2 Results of all evaluations of alternatives

Cities CR =0Rank CR=002Rank CR=004Rank CR=006Rank CR=008Rank CR=010RankRio de Janeiro 0263 1 0260 1 0257 1 0253 1 0249 1 0251 1Florianopolis 0052 6 0051 6 0051 6 0050 6 0051 6 0057 6Recife 0131 3 0139 3 0138 3 0137 3 0136 3 0137 3Salvador 0037 8 0037 8 0036 8 0036 9 0038 10 0046 8Natal 0065 5 0064 5 0063 5 0063 5 0062 5 0063 5Fortaleza 0258 2 0252 2 0249 2 0248 2 0249 1 0228 2Maceio 0087 4 0091 4 0090 4 0089 4 0089 4 0089 4Joao Pessoa 0033 9 0035 9 0035 9 0034 10 0040 8 0040 9Aracaju 0044 7 0043 7 0048 7 0048 7 0047 7 0050 7Vitoria 0029 10 0028 10 0032 10 0039 8 0038 9 0038 10

The findings indicate that comparisons of 3 4 and 5alternatives do not usually generate problems of consistencywhen considering CR lt01 as recommended by Saaty whenusing PCs However matrices with 6 or more alternativesusually generate CRgt 01 thus extrapolating the limit

The results further confirm the research by Bozoki etal [20] The cognitive effort to complete a comparisonsmatrix withmore than 5 alternatives is significant and fifteensuccessive comparisons are repeated as the problem increasesthe number of criteria Thus we propose a solution based onthe reduction in the number of comparisons verifying andcorrecting inconsistencies through an interactive algorithm

The results from the previous experiment were used toexplore the influence of inconsistency on the PCsMatrix Weselected 10 alternatives (cities) in which we have observeda high rate of inconsistency in all results The matrix valueswere randomly changed so as not to reverse the DMrsquospreferences We analyzed the inconsistency in each exampleand then reproduced similar situations of inconsistency for aselected DM

For example if the DM compares A1 and A2 suchas 4 and A2 and A3 as 2 to maintain consistency and

transitivity the comparison between A1 and A3 will be 6Theinconsistency introduced does not change the order A1 gtA2 gt A3 however although it would change the comparisonbetween A1 and A3 to 5 or 7

To verify the influence of inconsistencies we created aconsistent matrix formed by the RTM to serve as a refer-ence point for each decision-maker Therefore a consistentcomparisons matrix was created using selected responsesprovided by the decision-makers ie the minimal neces-sary information By including the additional informationfrom the PC that was already provided by the decision-maker we were able to generate different levels of inconsis-tency

The results of a consistentmatrix were comparedwith fiveother situations of inconsistency all within an accepted levelIn all situations the matrices of comparison maintained thefirst line of evaluation completed by the DMs but in eachcase allowed for a different level of inconsistency (from 0 to01)

The final ranking of the situations was analyzed for agiven decision-maker The results are presented in Table 2


Table 3 Representation of the comparison indices that will be evaluated

n odd n evenAlternatives Comparison indices Alternatives Comparison indicesA2 A3 I23 or I32 A2 A3 I23 or I32 An-3 An-2 In-3n-2 or In-2n-3 An-2 An-1 In-2n-1 or In-1n-2An-1 An In-1n or Inn-1 An-1 An In-1n or Inn-1Legend Ai ith alternative Iij comparison index of alternative Ai in relation to alternative Aj n number of alternatives

An evaluation of the alternatives was created without incon-sistencies and keeping the first matrixrsquos row fixed four otherevaluations were performed with CR = 002 004 006 008and 010 (Table 2)

A straightforward verification of the alternatives positionin Table 2 shows that the variation of the CR can influencethe final results Considering that CR is equal to 0 001 and002 there is no change in ranking although there are slightvariations in the priority vector With CR equal to 004 006008 and 010 however changes in ranking occur This facthas been observed for all DMs and it is an indication thatinconsistencies directly influence the result this finding isespecially important for additive methods

A similar result has been observed for all DMs whoparticipated in the present experiment In all cases at leastrank changed positions This result leads to the conclusionthat we cannot rely only on CR to measure the consistency ofthe comparisons matrix

Inconsistency can affect the recommendations made tothe DM and can even be detrimental resulting in a wrongdecision For example in all ranks Rio de Janeiro was rankedfirst however when CR = 008 Rio de Janeiro was tied withFortaleza This tie could generate doubt in the DM and causehim to choose Fortaleza In this example a wrong decisionwill most likely not have detrimental consequences but forstrategic decision-making an error could provoke seriousconsequences

4 Pairwise Comparisons withPreference Adjustment

All research related to the consistency of a comparisonsmatrix stems from the impossibility of ensuring that DMs areconsistent with their value judgment whenmaking PCs of thealternatives

In the previous section we attempted to show how theinconsistency of DMs increases as the number of alter-natives increases and how this inconsistency influencesthe overall results Additionally inconsistency is influencedby the bias in the process of DMrsquos elicitation of prefer-ences

Thus we suggest a method that seeks to reduce andcorrect inconsistencies caused by DMs in the qualitativeassessment of PCs

This method is based on two procedures First the DMmust access the preferences of a set of alternatives for a givencriterion by comparing the set of alternatives to a reference

alternative The compassion process only occurs betweenthe reference alternative for instance the best alternativeand the set of available alternatives Thus only one line ofthe comparisons matrix needs to be completed In a secondstep the remaining lines will be filled in with the help ofthe mathematical assumptions of the RTM This ensures theconsistency of the array and reduces the number of com-parisons The procedure uses the bases of the mathematicalpresuppositions of the RTM to verify acceptable values ofinconsistencies Later the interactive algorithm will identifyand correct the inconsistencies of the DM

At this point we assume that the restructuring process hasalready been undertaken and that the criteria and alternativeshave already been identified Decision makers subsequentlysort alternatives from better to worse

The proposed procedure reduces the number of prefer-ence comparisons demanded by theDMwhile increasing theaccuracy of their preferences The DM fills in only the firstrow of the comparisonmatrices and a few pairs of alternativesof the other lines This procedure results in a significantreduction of the number of PCs

The interactive algorithm selects a few pairs of availablealternatives to be evaluated ie additional comparisonsbetween the alternatives to confirm the preference judgmentof the DM with an estimated value in situations withoutinconsistency

When DMs complete one row of a comparisons matrixthere are no inconsistencies caused by a lack of reciprocityonly deviations from judgment can occur Nevertheless theDM may not have been consistent with his or her own pref-erences Thus a preference adjustment in the comparisonsmatrix may be performed to avoid possible deviations Oneway to accomplish this is proposed below

41 Preference Adjustment Algorithm The preference adjust-ment is based on questions posed to the DMThese questionsreview a comparison between two alternatives These twoalternatives should be chosen according to the number ofalternatives

The number of PCs has been presented previously toensure that the minimum number of required assessments isperformed Only then can there be a good preference adjust-ment Indeed complete consistency can only be ensured ifall comparison indices are evaluated However evaluating allcomparison indices may be too expensive thereby diminish-ing the advantage of reducing the number of PCs

Thenumber of revisions required depends on the numberof alternatives (Table 3)


Table 4 Proposed standard questions to assess consistency

Indices Range Question1 le 퐼119894119895 le 1 5 Considering the Ck criterion Do the alternatives Ai and Aj also contribute to the objective1 5 lt 퐼119894119895 le 2 5 There was indecision between the previous question and the next question2 5 lt 퐼119894119895 le 3 5 Considering Ck criterion Does your judgment slightly favor the alternative Ai on the alternative Aj3 5 lt 퐼119894119895 le 4 5 There was indecision between the previous question and the next question4 5 lt 퐼119894119895 le 5 5 Considering the Ck criterion Does your judgment strongly favor the alternative Ai over alternative Aj5 5 lt 퐼119894119895 le 6 5 There was indecision between the previous question and the next question

6 5 lt 퐼119894119895 le 7 5 Considering the Ck criterion Does your judgment reveal that the alternative Ai is very strongly favored overalternative Aj its domination of importance is demonstrated in practice

7 5 lt 퐼119894119895 le 8 5 There was indecision between the previous question and the next question

8 5 lt 퐼119894119895 le 9 Considering the Ck criterion Does your judgment show that the Ai alternative overcomes the Aj alternativewith the highest degree of certainty

(i) For a comparisons matrix with an even numberof elements n Do not conduct assessments withcomparison indices between the first row and the firstcolumn there will be an odd number of elements toevaluate It is not possible to form pairs with an oddnumber of elements without repetition so n2 + 1evaluations would be needed

(ii) For a comparisons matrix with an odd number ofelements n Again do not conduct assessments withcomparison indices between the first row and the firstcolumn there will be an even number of elementsto evaluate In this case n2 evaluations would beneeded

Initially the preference adjustment may seem costly andthe number of evaluations may seem as large as it is in thetraditional PCsMatrix However this is not trueThe solutionto the problem with ten criteria and ten alternatives usingRTM would require 145 evaluations (Table 1) 90 PCs and55 preference adjustments whereas in the traditional PCsmatrix there would be 450 PCs

The number of comparison indices is always greater thanzero The comparison indices of the main diagonal may notbe evaluated as each receives a value of 1 The comparisonindices of the first row and first column of the comparisonsmatrix must be completed directly by the DMwhen he or shemakes the PCs where the first column is a direct result of thiscomparison

As the PCs made by the DM are based on an interpreta-tion of Saatyrsquos fundamental scale the preference adjustmentwill also rely on this scale A standard set of questions basedon the fundamental scale of Saaty is proposed in Table 4Thequestions are presented only for comparison scores higherthan zero

As previouslymentioned to ensure an intuitive judgmentby theDM it is important to rank the alternatives frombest toworst based on the criterion in questionThus theDMshouldonly be asked to compare ordered alternatives in such a waythat the best alternative is always positioned in the first row

The assessment procedure by the DM is formalizedconsidering the whole procedure of preference adjustment A

standard algorithm that structures all steps of the analysis wascreated (Figure 3) The algorithm consists of six operationswhich are presented below

(1) Identify and order the comparison indices to be evalu-ated Let n be the number of alternatives To identifythe comparison indices the rules below should befollowedThe first comparison index to be evaluated is alwaysthe one that compares the alternatives A2 and A3 it is11986823 or 11986832The second comparison index to be evaluated isalways the one that compares alternatives A4 and A5Repeat this procedure until the nth alternative If nis positive the last comparison index necessarily rep-resents the comparison between An-1 and An alsoAn-1 does not change since it has been previouslyevaluated (Table 3)

(2) Loop repeat After ordering the comparison indicesto be evaluated initiate the evaluation process ofordering until the last index comparison is evaluatedat which point the procedure ends

(3) Verification of rangeThe beginning of the assessmentconsists in determining which range from Table 4will be used to evaluate the comparison index

(4) Question Ask the DM the questions listed in Table 4

(i) If the answer is yes go back to step (2) to startthe next evaluation

(ii) If the answer is no proceed to step (5)(5) Reevaluation of index of the first lineThe DM should

reevaluate the indices 1198681푗 and 119868푖1 update the compar-isons matrix accordingly

(6) Determine if there was a change in the range Check ifthe index 119868푖푗 changed in terms of range as shown inTable 4If there was no change in terms of range considerthat the DMwas given the opportunity to reassess the


no

yes

k = 1

no

yesk = k +1

Identify and sort the comparison indices tobe evaluated

The DM should reevaluate the indices I 1j and I i1 update the

comparison matrix

yes

nok = k +1

end

If n is even check if k is greater

than n 2 +1

If n is odd check if n is greater than

n 2

Check that the kth range belongs to Iij

Ask the decision maker

questions to evaluateconsistency

Check if the index Iij changed in

terms of range

k counter

Figure 3 Preference adjustment algorithm

comparison index of the first line and that despitehis or her negative response in step (4) the range didnot change enough to change the landscape of theinitial evaluation Return to step (2) to start the nextevaluation

If there was a change in terms of range go back to step(3) to conduct a new evaluation of the same index

An important issue to be considered is the automaticchange in another comparison index when the DM revisesand changes an index of the first row Similarly it is wisefor researchers to consider that some indices of comparisonas assessed by the DM will be subjected to change Whenthese changes occur they must be reevaluated However thealgorithm solves this problem

If an index in the first row is changed the other indexesthat depend on it are automatically corrected considering themathematical assumptions of the RTM Thus it will correctthe deviations in the DMrsquos preferences and simultaneouslymaintain the consistency of the matrix

As we did not perform evaluations with A1 start with11986823 or 11986832 If the DM changes the value of 11986812 and 11986813 theunderlined comparison indices will change

Without repeating the lines already evaluated the DMmust evaluate 11986845 or 11986854 which were not altered by previousreevaluation If the DM changes the value of 11986814 and 11986815 thecomparison indices underlined in Table 5 will change It is

Table 5 Change in the comparisons matrix due to the preferenceadjustment of 11986814 and 11986815

A1 A2 A3 A4 A5 A6A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866

important to note that none of the possible indices previouslyevaluated 11986823 or 11986832 will have changed

Finally no index containingA6 was evaluated and there isno alternative to pair with A6 In this case we must evaluate11986856 or 11986865 as shown in Table 3 For this situation (even n)the DM can only change the value of 11986816 since 11986815 haspreviously been reevaluated and any change in 11986815 would leadto a reassessment of 11986854 because its value will be changedIf the DM changes the value of 11986816 the comparison indicesunderlined in Table 6 will change It is important to notethat none of the possible indices previously evaluated (1198682311986832 11986845 and 11986854) will have changed thus they do not requirereassessment


Table 6 Change in the comparisons matrix due to the preference adjustment of 11986816A1 A2 A3 A4 A5 A6

A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866

Table 7 Preference evaluation of the holiday destination choiceproblem for the criterion Tourist Attractions

A1 A2 A3 A4 A5A1 100 300 300 500 300

5 Application of the Procedure

We have applied the procedure for two situations First weused the problems explored in the experiment conducted onitem 3 (tourist cities) Then we tested the procedure with10 DMs to determine the inconsistency and agreement levelwith the selected alternative

51 Summer Holiday Case We applied the proposed proce-dure to the case of selecting the next city to spend the summerholiday The alternatives were Rio de Janeiro FlorianopolisSalvador and Natal as presented in Figure 1

The procedure involved one DM and the support of ananalyst to run the interactive algorithmWewill assume threecriteria with the following weightsW1 = 060W2 = 010 andW3 = 030 as previously defined

Details of the procedure are presented with the TouristAttractions criterion (Tables 7 and 8) For this situationalternative A1 Rio de Janeiro is considered the referencealternative Continuing the procedure should be performedwith the alternativersquos comparison matrices considering thethree criteria

Evaluate 11986823 = 1 Question Considering the TouristAttractions criterion do alternatives A2 and A3 also con-tribute to the goal The answer is yes so the Matrix was notupdated

Evaluate 11986854 = 1 67 Question Considering the TouristAttractions criterion do alternatives A5 and A4 also con-tribute to the goal or do your experience and judgmentslightly favor alternative A5 over alternative A4 Are you sureabout both situations The DM says that alternatives A5 andA4 also contribute to the goal 11986854 assumes value 1 and weupdate the comparisons matrix11986854 change in the range repeat the question to assess 11986854 =1 Ask the DM Considering the Tourist Attractions criterion

do alternatives A5 and A4 also contribute to the goal or doyour experience and judgment slightly favor alternative A2over alternative A4The answer is yes so the Matrix was notupdated At the end of the preference settings the priorityvectors are presented in Table 9

The results presented in Table 10 indicate that the pref-erence adjustment is an important opportunity for the DMto identify deviations in the comparisons matrix and thuscorrect them It was found that the results with and with-out adjustment differ from one another in three rankingpositions Fortaleza moved from third position to fourthSalvador from fourth to fifth position and Florianopolis fromfifth to third position

52 Additional Experiment In the experiment ten DMs ofthe very inconsistent group were chosen to test the PCswith preference adjustment The DMs presented CR higherthan 02 and thus their results were not considered in theexperiment

We begin the experiment reassessing the DMrsquos prefer-ences for the first line of the matrix comparison Then weapply the preference adjustment algorithm

At the end of the process we asked two questions Do youconsider the effort and time spent applying the method to bereasonable Do you consider that the result represents yourpreferences For both questions a five-point scale was usedas an option for response (1)No not at all (2)No not much(3)More or less (4) In general yes (5) Yes of course

For question 1 seven DMs responded ldquoYes for surerdquoand three responded ldquoUsually yesrdquo This indicates that effortdeclined considerably compared to the traditional method

For question 2 five DMs responded ldquoYes for surerdquo threeresponded ldquoUsually yesrdquo and two responded ldquoMore or lessrdquoThis indicates that the results can be considered satisfactoryin 80 of the cases

The application of the proposed approach reveals that theeffort required by the DMs to evaluate alternatives and toassess their preferences clearly decreased while attempting tomaintain an acceptable level of inconsistency

6 Conclusion

In this study we demonstrate that the inconsistency of PCmatrices even within acceptable limits can influence theresults of a decision processThus we proposed the followingapproach First the PC comprises only the first row of thematrix while the other lines are filled in based on theassumptions of the RTM Second we use an algorithm toidentify and correct deviations in the preferences of the DMsin the matrices


Table 8 Additional preference information for the comparisons matrix of the holiday destination choice problem for the criterion TouristAttractions

A1 A2 A3 A4 A5A1 100 300 300 500 300A2 033 100 100 167 100A3 033 100 100 167 100A4 020 060 060 100 060A5 033 100 100 167 100Bold cell representing the additional value required by DM

Table 9 Priority vectors of the alternatives for choosing the holiday destination

Costs Hotel Features Tourist AttractionsWeight 060 010 030

With adjustment Without adjustment Without adjustment With adjustment Without adjustment With adjustmentRio de Janeiro 0071 0053 0506 0466 0455 0484Florianopolis 0100 0158 0169 0233 0152 0161Fortaleza 0166 0158 0169 0155 0152 0161Salvador 0166 0158 0101 0093 0091 0097Natal 0489 0474 0056 0052 0152 0097

Table 10 Comparison of the problemrsquos results of choosing the holiday destination without preference adjustment and with preferenceadjustment

Alternatives Without adjustment With adjustmentPriority Vector Ranking Priority Vector Ranking

Rio de Janeiro 0243 2∘ 0223 2∘

Florianopolis 0124 5∘ 0166 3∘

Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘

The process of building a RTM when applied to the PCsmatrix reduces the DMrsquos effort in the PCs The significantreduction in a DMrsquos effort stems from the building of a RTMwhich entails a totally consistent evaluation TraditionallyPCs allow for certain level of inconsistency CR less than orequal to 010

The most important issue concerns the measurement ofdeviations when inconsistencies are allowed The simulationand the experiments in this work clearly indicate that allow-ing inconsistencies achieves different results than when usingconsistent DM evaluations although this is not always thecase This result does not nullify the value of building theRTM nor does it discourage the use of a traditional PCsmatrix

Ultimately the most important aspect is to check theconsistency of the evaluation results to confirm that the DMrsquospriorities are reflected in his or her judgment If the answeris negative it may be necessary to reassess In fact thisprocess is already included in virtually all MCDA methodsyet in the case of the PCs matrix for which there are manyalternatives this would make decision making even moredifficult High cognitive efforts generate inconsistencies inaddition to requiring a significant amount of time Thus it is

important to use tools that reduce cognitive effort and ensuresatisfactory results

The preference adjustment algorithm aims to comple-ment the information provided by the DM The adjustmentsapproximate the results of the problem to the preferencesof the DM The procedures presented in this paper reducethe cognitive effort of the DM eliminate inconsistencies inthe comparison process and present a recommendation thatreflects the preferences of the DM

More research is needed however to verify whether theresults that are overly sensitive to inconsistency are linkedto the scale used in the assessment are caused by smalldifferences between alternatives or are simply the result oferrors caused by the DMrsquos high cognitive effort With regardto the application of the proposed algorithm an importantissue is to examine the use of other scales and the consistencyindex associated with these scales such as the scale proposedby Koczkodaj [46]

Data Availability

The data included in the study rdquoExploring MulticriteriaElicitation Model Based on Pairwise Comparison Building


an Interactive Preference Adjustment Algorithmrdquo are theresponsibility of authors Giancarllo Ribeiro Vasconcelos andCaroline Maria de Miranda Mota who collected the data inan open databaseThere are no restrictions on access to them

Conflicts of Interest

The authors declare that they have no conflicts of interest

References

[1] M Marttunen J Lienert and V Belton ldquoStructuring problemsfor multi-criteria decision analysis in practice a literaturereview of method combinationsrdquo European Journal of Opera-tional Research vol 263 no 1 pp 1ndash17 2017

[2] H Keeney and L R Raiffa Decisions with Multiple ObjectivesPreferences and Value Tradeoffs New York NY USA 1976

[3] H Polatidis D A Haralambopoulos G Munda and RVreeker ldquoSelecting an appropriate multi-criteria decision anal-ysis technique for renewable energy planningrdquo Energy SourcesPart B Economics Planning and Policy vol 1 no 2 pp 181ndash1932006

[4] J Figueira S Greco andM Ehrgott ldquoMultiple criteria decisionanalysis state of the art surveysrdquo Multiple Criteria DecisionAnalysis State of the Art Surveys vol 78 no 1 p 1045 2005

[5] G Montibeller and D vonWinterfeldt ldquoCognitive and motiva-tional biases in decision and risk analysisrdquo Risk Analysis vol 35no 7 pp 1230ndash1251 2015

[6] W Edwards ldquoHow to use multivariate utility measurement forsocial decision makingrdquo IEEE Transactions on Systems Manand Cybernetics vol 7 no 5 pp 326ndash340 1977

[7] W Edwards and F H Barron ldquoSMARTS and SMARTERimproved simple methods for multiattribute utility measure-mentrdquo Organizational Behavior and Human Decision Processesvol 60 no 3 pp 306ndash325 1994

[8] R L Keeney ldquoValue-focused thinking a path to creativedecisionmakingrdquo 1992

[9] R L Keeney ldquoCommon mistakes in making value trade-offsrdquoOperations Research vol 50 no 6 pp 935ndash1077 2002

[10] A T de Almeida J A de Almeida A P C S Costa and A T deAlmeida-Filho ldquoA newmethod for elicitation of criteria weightsin additive models flexible and interactive tradeoffrdquo EuropeanJournal of Operational Research vol 250 no 1 pp 179ndash191 2016

[11] T L Saaty ldquoA scaling method for priorities in hierarchicalstructuresrdquo Journal of Mathematical Psychology vol 15 no 3pp 234ndash281 1977

[12] T L Saaty ldquoThe analytic hierarchy process in conflict manage-mentrdquo International Journal of Conflict Management vol 1 no1 pp 47ndash68 1990

[13] T L Saaty ldquoHow to make a decision the analytic hierarchyprocessrdquo European Journal of Operational Research vol 1 pp19ndash43 1990

[14] C A Bana e Costa and J-C Vansnick ldquoA critical analysis of theeigenvalue method used to derive priorities in AHPrdquo EuropeanJournal of Operational Research vol 187 no 3 pp 1422ndash14282008

[15] C A B E Costa J-M De Corte and J-C Vansnick ldquoMAC-BETHrdquo International Journal of Information Technology ampDecision Making vol 11 no 2 pp 359ndash387 2012

[16] A FWadjdi EM Sianturi andN Ruslinawaty ldquoDesign of datacollection form to ensure consistency in AHPrdquo in Proceedingsof the 2018 10th International Conference on Information Tech-nology and Electrical Engineering (ICITEE) pp 529ndash533 KutaIndonesia July 2018

[17] L Wu X Cui and R Dai ldquoJudgment number reduction anissue in the analytic hierarchy processrdquo International Journal ofInformation TechnologyampDecisionMaking vol 9 no 1 pp 175ndash189 2010

[18] R R Tan and M A B Promentilla ldquoA methodology foraugmenting sparse pairwise comparison matrices in AHPApplications to energy systemsrdquo Clean Technologies and Envi-ronmental Policy vol 15 no 4 pp 713ndash719 2013

[19] H Raharjo and D Endah ldquoEvaluating relationship of consis-tency ratio and number of alternatives on rank reversal in theAHPrdquo Quality Engineering vol 18 no 1 pp 39ndash46 2006

[20] S Bozoki L Dezso A Poesz and J Temesi ldquoAnalysis ofpairwise comparison matrices an empirical researchrdquo Annalsof Operations Research vol 211 no 1 pp 511ndash528 2013

[21] D Ergu and G Kou ldquoQuestionnaire design improvement andmissing item scores estimation for rapid and efficient decisionmakingrdquo Annals of Operations Research vol 197 pp 5ndash23 2012

[22] L Huo J Lan and Z Wang ldquoNew parametric prioritizationmethods for an analytical hierarchy process based on a pairwisecomparison matrixrdquo Mathematical and Computer Modellingvol 54 no 11-12 pp 2736ndash2749 2011

[23] H Hou and H Wu ldquoWhat influence domestic and over-seas developersrsquo decisionsrdquo Journal of Property Investment ampFinance vol 37 no 2 pp 153ndash171 2019

[24] W Gaul andD Gastes ldquoA note on consistency improvements ofAHP paired comparison datardquo Advances in Data Analysis andClassification ADAC vol 6 no 4 pp 289ndash302 2012

[25] C Lin G Kou and D Ergu ldquoAn improved statistical approachfor consistency test in AHPrdquo Annals of Operations Research vol211 no 1 pp 289ndash299 2013

[26] S-W Lin and M-T Lu ldquoCharacterizing disagreement andinconsistency in expertsrsquo judgments in the analytic hierarchyprocessrdquo Management Decision vol 50 no 7 pp 1252ndash12652012

[27] J Benıtez J Izquierdo R Perez-Garcıa and E Ramos-Martınez ldquoA simple formula to find the closest consistentmatrix to a reciprocal matrixrdquo Applied Mathematical ModellingSimulation andComputation for Engineering and EnvironmentalSystems vol 38 no 15-16 pp 3968ndash3974 2014

[28] X Su ldquoBounded rationality in newsvendor modelsrdquo Manufac-turingamp Service OperationsManagement vol 10 no 4 pp 566ndash589 2008

[29] T Toma and M R Asharif AHP Coefficients OptimizationTechnique Based on GA Beppu Japan 2003

[30] M Brunelli L Canal and M Fedrizzi ldquoInconsistency indicesfor pairwise comparisonmatrices a numerical studyrdquoAnnals ofOperations Research vol 211 pp 493ndash509 2013

[31] W W Koczkodaj and J Szybowski ldquoPairwise comparisonssimplifiedrdquoAppliedMathematics and Computation vol 253 pp387ndash394 2015

[32] E N Weiss and V R Rao ldquoAHP design issues for large-scalesystemsrdquo Decision Sciences vol 18 no 1 pp 43ndash61 1987

[33] P T Harker ldquoIncomplete pairwise comparisons in the analytichierarchy processrdquoAppliedMathematical Modelling Simulationand Computation for Engineering and Environmental Systemsvol 9 no 11 pp 837ndash848 1987


[34] B Golany andM Kress ldquoA multicriteria evaluation of methodsfor obtaining weights from ratio-scale matricesrdquo EuropeanJournal of Operational Research vol 69 no 2 pp 210ndash220 1993

[35] MKwiesielewicz and E vanUden ldquoInconsistent and contradic-tory judgements in pairwise comparison method in the AHPrdquoComputers amp Operations Research vol 31 no 5 pp 713ndash7192004

[36] S Bozoki J Fulop and W W Koczkodaj ldquoAn LP-basedinconsistency monitoring of pairwise comparison matricesrdquoMathematical and ComputerModelling vol 54 no 1-2 pp 789ndash793 2011

[37] J Reichert ldquoOn the complexity of counter reachability gamesrdquoFundamenta Informaticae vol 143 no 3-4 pp 415ndash436 2016

[38] K D Goepel ldquoComparison of judgment scales of the analyticalhierarchy process mdash a new approachrdquo International Journal ofInformation Technology amp Decision Making vol 18 no 2 pp445ndash463 2019

[39] A A Salo and R P Hamalainen ldquoOn the measurementof preferences in the analytic hierarchy processrdquo Journal ofMulti-Criteria Decision Analysis vol 6 no 6 pp 309ndash319 1997

[40] HADonegan F J Dodd andT BMcMaster ldquoAnew approachtoAHPdecision-makingrdquoTheAmerican Statistician vol 41 no3 p 295 2006

[41] F A Lootsma ldquoScale sensitivity in the multiplicative AHP andSMARTrdquo Journal of Multi-Criteria Decision Analysis vol 2 no2 pp 87ndash110 1993

[42] W W Koczkodaj L Mikhailov G Redlarski et al ldquoImportantfacts and observations about pairwise comparisons (the specialissue edition)rdquo Fundamenta Informaticae vol 144 no 3-4 pp291ndash307 2016

[43] T L Saaty ldquoOn the measurement of intangibles A principaleigenvector approach to relative measurement derived frompaired comparisonsrdquo Notices vol 60 no 2 pp 192ndash208 2013

[44] R J Hill ldquoA note on inconsistency in paired comparisonjudgmentsrdquo American Sociological Review vol 18 no 5 p 5642006

[45] J Barzilai ldquoDeriving weights from pairwise comparison matri-cesrdquo Journal of the Operational Research Society vol 48 no 12pp 1226ndash1232 1997

[46] S Bozoki and T Rapcsak ldquoOn Saatyrsquos and Koczkodajrsquos incon-sistencies of pairwise comparison matricesrdquo Journal of GlobalOptimization vol 42 no 2 pp 157ndash175 2008

[47] W W Koczkodaj ldquoA new definition of consistency of pairwisecomparisonsrdquo Mathematical and Computer Modelling vol 18no 7 pp 79ndash84 1993

[48] MW Herman andWW Koczkodaj ldquoAPL2 implementation ofa new definition of consistency of pairwise comparisonsrdquo ACMSIGAPL APL Quote Quad vol 24 no 4 pp 37ndash40 2007

[49] W W Koczkodaj M W Herman and M Orłowski ldquoUsingconsistency-driven pairwise comparisons in knowledge-basedsystemsrdquo in Proceedings of the Sixth International Conference onInformation and Knowledge Management pp 91ndash96 Las VegasNv USA November 1997

[50] E Dopazo and J Gonzalez-Pachon ldquoConsistency-drivenapproximation of a pairwise comparison matrixrdquo Kybernetikavol 39 no 5 pp 561ndash568 2003

[51] W W Koczkodaj K Kułakowski and A Ligeza ldquoOn thequality evaluation of scientific entities in Poland supported byconsistency-driven pairwise comparisonsmethodrdquo Scientomet-rics vol 99 no 3 pp 911ndash926 2014

[52] W W Koczkodaj and J Szybowski ldquoThe limit of inconsistencyreduction in pairwise comparisonsrdquo International Journal ofApplied Mathematics and Computer Science vol 26 no 3 pp721ndash729 2016

[53] R Janicki and W W Koczkodaj ldquoA weak order solution to agroup ranking and consistency-driven pairwise comparisonsrdquoAppliedMathematics and Computation vol 94 no 2-3 pp 227ndash241 1998

[54] W W Koczkodaj P Dymora M Mazurek and D StrzałkaldquoConsistency-driven pairwise comparisons approach to soft-ware product management and quality measurementrdquo in Con-temporary Complex Systems and Their Dependability vol 761of Advances in Intelligent Systems and Computing pp 292ndash305Springer International Publishing Cham Swizerland 2019

[55] Z S Xu ldquoGoal programming models for obtaining the priorityvector of incomplete fuzzy preference relationrdquo InternationalJournal of Approximate Reasoning vol 36 no 3 pp 261ndash2702004

[56] N Pankratova and N Nedashkovskaya ldquoMethods of evaluationand improvement of consistency of expert pairwise comparisonjudgmentsrdquo International Journal ldquoInformation Theories andApplicationsrdquo vol 22 no 3 pp 203ndash223 2015

[57] E Herrera-Viedma F Herrera F Chiclana and M LuqueldquoSome issues on consistency of fuzzy preference relationsrdquoEuropean Journal of Operational Research vol 154 no 1 pp 98ndash109 2004

[58] A Voisin J Robert E H Viedma Y Le Traon W Derigentand S Kubler ldquoMeasuring inconsistency and deriving prioritiesfrom fuzzy pairwise comparisonmatrices using the knowledge-based consistency indexrdquo Knowledge-Based Systems vol 162pp 147ndash160 2018

[59] X Benıtez X Delgado-Galvan J Izquierdo and R Perez-Garcıa ldquoAchieving matrix consistency in AHP through lin-earizationrdquo Applied Mathematical Modelling Simulation andComputation for Engineering and Environmental Systems vol35 no 9 pp 4449ndash4457 2011

[60] J Benıtez X Delgado-Galvan J A Gutierrez and J IzquierdoldquoBalancing consistency and expert judgment in AHPrdquo Mathe-matical and Computer Modelling vol 54 no 7-8 pp 1785ndash17902011

[61] J Benıtez XDelgado-Galvan J Izquierdo andR Perez-GarcıaldquoImproving consistency in AHP decision-making processesrdquoAppliedMathematics and Computation vol 219 no 5 pp 2432ndash2441 2012

[62] A Negahban ldquoOptimizing consistency improvement of posi-tive reciprocal matrices with implications for Monte carlo ana-lytic hierarchy processrdquo Computers amp Industrial Engineeringvol 124 pp 113ndash124 2018

[63] M Brunelli A Critch and M Fedrizzi ldquoA note on theproportionality between some consistency indices in the AHPrdquoApplied Mathematics and Computation vol 219 no 14 pp7901ndash7906 2013

[64] M Xia and Z Xu ldquoInterval weight generation approaches forreciprocal relationsrdquo Applied Mathematical Modelling vol 38no 3 pp 828ndash838 2014

[65] E Bulut O Duru T Kececi and S Yoshida ldquoUse of consistencyindex expert prioritization and direct numerical inputs forgeneric fuzzy-AHP modeling a process model for shippingasset managementrdquo Expert Systems with Applications vol 39no 2 pp 1911ndash1923 2012


[66] J Ramık and P Korviny ldquoInconsistency of pair-wise compar-ison matrix with fuzzy elements based on geometric meanrdquoFuzzy Sets and Systems vol 161 no 11 pp 1604ndash1613 2010

[67] M Brunelli ldquoA note on the article ldquoInconsistency of pair-wisecomparison matrix with fuzzy elements based on geometricmeanrdquo [Fuzzy Sets and Systems vol 161 pp 1604-1613 2010]rdquoFuzzy Sets and Systems vol 176 no 1 pp 76ndash78 2011

[68] F Liu W G Zhang and L H Zhang ldquoConsistency analysisof triangular fuzzy reciprocal preference relationsrdquo EuropeanJournal of Operational Research vol 235 no 3 pp 718ndash7262014

[69] Y Xu and H Wang ldquoEigenvector method consistency testand inconsistency repairing for an incomplete fuzzy preferencerelationrdquo Applied Mathematical Modelling vol 37 no 7 pp5171ndash5183 2013

Hindawiwwwhindawicom Volume 2018

MathematicsJournal of


Mathematical Problems in Engineering

Applied MathematicsJournal of


Probability and StatisticsHindawiwwwhindawicom Volume 2018

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of



Engineering Mathematics

International Journal of


Operations ResearchAdvances in

Journal of


Function SpacesAbstract and Applied AnalysisHindawiwwwhindawicom Volume 2018

International Journal of Mathematics and Mathematical Sciences


Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018

Hindawiwwwhindawicom Volume 2018Volume 2018

Numerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisAdvances inAdvances in Discrete Dynamics in

Nature and SocietyHindawiwwwhindawicom Volume 2018

Hindawiwwwhindawicom

Dierential EquationsInternational Journal of

Volume 2018


Decision SciencesAdvances in


AnalysisInternational Journal of


Stochastic AnalysisInternational Journal of

Submit your manuscripts atwwwhindawicom


Table 1 Number of pairwise comparisons based on the number of criteria and alternatives

Number of Alternatives Number of Alternatives

Number of Criteria

2 3 4 5 6 7 8 9 10 2 3 4 5 6 7 8 9 102 2 4 6 8 10 12 14 16 18 2 2 6 12 20 30 42 56 72 903 3 6 9 12 15 18 21 24 27 3 3 9 18 30 45 63 84 108 1354 4 8 12 16 20 24 28 32 36 4 4 12 24 40 60 84 112 144 1805 5 10 15 20 25 30 35 40 45 5 5 15 30 50 75 105 140 180 2256 6 12 18 24 30 36 42 48 54 6 6 18 36 60 90 126 168 216 2707 7 14 21 28 35 42 49 56 63 7 7 21 42 70 105 147 196 252 3158 8 16 24 32 40 48 56 64 72 8 8 24 48 80 120 168 224 288 3609 9 18 27 36 45 54 63 72 81 9 9 27 54 90 135 189 252 324 40510 10 20 30 40 50 60 70 80 90 10 10 30 60 100 150 210 280 360 450

Reciprocal Transitive Matrix Traditional Method

more sophisticated approach to build a utility function thatexpresses risk attitudes [2] Another group of methods buildsthe value function based on pairwise comparisons (PCs)of several preference statements (questioning) such as theanalytic hierarchy process [11ndash13] and Macbeth [14 15]

A common cognitive deviation in MCDM methodsoccurs when there is a PC of a large number of alternativeswhich requires a great deal of cognitive effort by the DM[16ndash20] In this case several studies have reported concernsrelated to the applicability of this type of procedure insituations where the number of criteria and alternatives isquite large [17 18 20 21] In such situations the numberof PCs made by the DM grows at an alarming rate Thetime that analysts spend with DMs is increasingly scarceand convincing a top executive to spend hours or evendays making PCs of alternatives and criteria is often notfeasible

This paper focuses on the preference judgment of a DMbased on PCs of a qualitative criterion to build a valuefunction For this purpose an analysis was conducted basedon the PC process The AHP is one of the best knownmulticriteria decision aid (MCDA) approaches and is thuswidely used [1 22] AHP is an additive method proposedby Saaty [11ndash13] and since its introduction has attractedincreasing attention from researchers [22ndash26] The methodconverts subjective assessments of relative importance to avector of priorities (value function of one criterion) basedon PCs performed within each criterion The comparativejudgments are made using the fundamental scale devised bySaaty [11] and a consistency logic is applied to check theDMrsquosjudgments [27]

In this article we explore the influence of consistency andinconsistency in preference assessmentsmdashbased on PCsmdashbyperforming an experiment with several individuals to assesstheir preferences based on a given situation A high levelof inconsistency has been verified within the literature (ref-erence) Additionally we evaluate an alternative procedureto assess the DMrsquos preferences The preferences of a DMare assessed based on an interactive procedure of askingquestions and adjusting preferences thus reducing the timespent by the DM and assuming an acceptable level of possibleinconsistencies

This paper is organized as follows The next sectionpresents a review of the literature concerning the causes ofinconsistencies in PCs as well as several solutions Section 3describes the experiment conducted to verify the influenceof inconsistency Lastly a preference adjustment algorithm inPCs is presented

2 Literature Review

Benitez et al [28] have found that human beings make moregood decisions than bad ones throughout their lifetime andthat irrationality is a common cause of bad decisions Theythus proposed a model based on the most likely choice tocapture shifts in the decision-making (irrationality) processto reduce biases caused by inconsistent judgments

A PC uses human abilities such as knowledge andexperience to compare alternatives and criteria in a pairwisemanner and assemble a comparisons matrix [29] Inconsis-tency arises when some opinions of a comparisons matrixcontradict othersTherefore it is important to check the con-sistency of opinions when performing a series of calculationsto arrive at the value of the consistency ratio (CR) whichindicates the consistency of the comparisons matrix For thePCs matrix it is desirable that the CR for any comparisonsmatrix be less than or equal to 010 [25 30]

The number of n PCs is 119899(119899 minus 1)2 hence groupingand hierarchy structure should be used for larger n [21]Bozoki et al [20] performed a controlled experiment withuniversity students (N = 227) which enabled them to obtain454 PCsmatricesThey conducted experimentswithmatricesof different sizes from different types of multicriteria decisionsupport methods and found that the size of a matrix affectsthe DMrsquos consistency (measured by inconsistency indices)An increasing matrix size leads to greater inconsistency

Consider the multicriteria problem where the DM mustmake a PC alongside five criteria and ten alternatives In sucha situation the DM will have to allow time to perform 225evaluations (Table 1) to compare the alternatives within eachcriterion If we insert twomore criteria this number increasesto 315 evaluations Ten criteria with ten alternatives require495 evaluations The number of alternatives is certainly thelargest source of comparisons whereas all alternatives are



119862119873 = 1198962 [119899 (119899 minus 1)] (1)



119862119873lowast = 119896119899 minus 1 (2)













119868119888 (119872) = 120582푚푎푥 minus 119899119899 minus 1 (3)



119862119877 (119872) = 119868119888 (119872)119877119868 (4)






119862119872 = min(111988610038161003816100381610038161003816100381610038161003816119886 minus11988711988810038161003816100381610038161003816100381610038161003816 1119887 |119887 minus 119886119888|

111988810038161003816100381610038161003816100381610038161003816119888 minus11988711988610038161003816100381610038161003816100381610038161003816) (5)




















Criteria

Objective
















026024022020018016014012010008006004002



3 4 5 6 7 8 9 10 11 12

Con

siste

ncy

Ratio























































no

yes

k = 1

no

yesk = k +1



comparison matrix

yes

nok = k +1

end


than n 2 +1


n 2





terms of range

k counter









A1 A2 A3 A4 A5 A6A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866





A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866


A1 A2 A3 A4 A5A1 100 300 300 500 300
















6 Conclusion












Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018










119862119873 = 1198962 [119899 (119899 minus 1)] (1)



119862119873lowast = 119896119899 minus 1 (2)













119868119888 (119872) = 120582푚푎푥 minus 119899119899 minus 1 (3)



119862119877 (119872) = 119868119888 (119872)119877119868 (4)






119862119872 = min(111988610038161003816100381610038161003816100381610038161003816119886 minus11988711988810038161003816100381610038161003816100381610038161003816 1119887 |119887 minus 119886119888|

111988810038161003816100381610038161003816100381610038161003816119888 minus11988711988610038161003816100381610038161003816100381610038161003816) (5)




















Criteria

Objective
















026024022020018016014012010008006004002



3 4 5 6 7 8 9 10 11 12

Con

siste

ncy

Ratio























































no

yes

k = 1

no

yesk = k +1



comparison matrix

yes

nok = k +1

end


than n 2 +1


n 2





terms of range

k counter









A1 A2 A3 A4 A5 A6A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866





A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866


A1 A2 A3 A4 A5A1 100 300 300 500 300
















6 Conclusion












Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018










119862119872 = min(111988610038161003816100381610038161003816100381610038161003816119886 minus11988711988810038161003816100381610038161003816100381610038161003816 1119887 |119887 minus 119886119888|

111988810038161003816100381610038161003816100381610038161003816119888 minus11988711988610038161003816100381610038161003816100381610038161003816) (5)




















Criteria

Objective
















026024022020018016014012010008006004002



3 4 5 6 7 8 9 10 11 12

Con

siste

ncy

Ratio























































no

yes

k = 1

no

yesk = k +1



comparison matrix

yes

nok = k +1

end


than n 2 +1


n 2





terms of range

k counter









A1 A2 A3 A4 A5 A6A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866





A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866


A1 A2 A3 A4 A5A1 100 300 300 500 300
















6 Conclusion












Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018












Criteria

Objective
















026024022020018016014012010008006004002



3 4 5 6 7 8 9 10 11 12

Con

siste

ncy

Ratio























































no

yes

k = 1

no

yesk = k +1



comparison matrix

yes

nok = k +1

end


than n 2 +1


n 2





terms of range

k counter









A1 A2 A3 A4 A5 A6A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866





A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866


A1 A2 A3 A4 A5A1 100 300 300 500 300
















6 Conclusion












Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018









026024022020018016014012010008006004002



3 4 5 6 7 8 9 10 11 12

Con

siste

ncy

Ratio























































no

yes

k = 1

no

yesk = k +1



comparison matrix

yes

nok = k +1

end


than n 2 +1


n 2





terms of range

k counter









A1 A2 A3 A4 A5 A6A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866





A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866


A1 A2 A3 A4 A5A1 100 300 300 500 300
















6 Conclusion












Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018



















































no

yes

k = 1

no

yesk = k +1



comparison matrix

yes

nok = k +1

end


than n 2 +1


n 2





terms of range

k counter









A1 A2 A3 A4 A5 A6A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866





A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866


A1 A2 A3 A4 A5A1 100 300 300 500 300
















6 Conclusion












Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018










A1 11986811 11986812 11986813 11986814 11986815 11986816A2 11986821 11986822 퐼23 11986824 11986825 11986826A3 11986831 퐼32 11986833 11986834 11986835 11986836A4 11986841 11986842 11986843 11986844 11986845 11986846A5 11986851 11986852 11986853 11986854 11986855 11986856A6 11986861 11986862 11986863 11986864 11986865 11986866


A1 A2 A3 A4 A5A1 100 300 300 500 300
















6 Conclusion












Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018


















Fortaleza 0162 3∘ 0159 4∘

Salvador 0135 4∘ 0133 5∘

Natal 0336 1∘ 0318 1∘







Data Availability






References















































































Journal of












Journal of







Volume 2018






Volume 2018












References















































































Journal of












Journal of







Volume 2018






Volume 2018





















































Journal of












Journal of







Volume 2018






Volume 2018








exploring multicriteria elicitation model based on ... · exploring multicriteria elicitation model...

Documents