interactive requirements prioritization using search based optimization technique and constraint...

Interactive Requirements Prioritization Using Search Based Optimization Technique and Constraint Solver

Acknowledgement: Dr. Paolo Tonella and Dr. Angelo Susi

University of Trento, Italy Fondazione Bruno Kessler (FBK), Trento, Italy

Francis Palma [email protected]

Ecole Polytechnique de Montreal

PhD. Supervisor: Dr. Yann-Gaël Guéhéneuc, Ecole Polytechnique Co-supervisor: Dr. Naouel Moha, UQAM

Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions

Problem: ‘Prioritization of Requirements’ To find the best ordering of requirements in each successive release to ensure quality & value of the delivered system, trade-off constraints & end-user satisfaction

Do Trade-off between user needs & real constraints

Now, highest quality & best valued system!

Extracted from Requirement Documents

Problem Description: An Example Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions

2 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver

of 63

q  Problem description

q  Related works

q  Genetic Algorithm

q  Approach A (Search Based Technique)

q  Case study

q  Results A

q  Approach B (Constraint Solver: SMT)

q  Results B

q  Conclusions



of 63

Requirements Prioritization?

A task performed in Software Project Management to determine which candidate requirements should be included in a certain release, negotiating all technical & non-technical constraints.

Requirements Prioritization matters, why?

Expectations ~ high; Timelines ~ short; Resources ~ limited, Make sure product contains the most essential functions, maximizing customer satisfaction and best resource utilization.

Prioritization dimensions may include:

Stakeholder expectations, business value, risk, cost, difficulty, time-constraints, dependencies among requirements etc.

Requirements Prioritization Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

(1) Acquisition and coding of set of Requirements and Domain Knowledge; (2) Apply A Prioritization Technique (3) Output of the ranking

Problem Description (cont.)

Find it! Why Ordered List? Order of implementation satisfy the developers’ constraints & delivers maximum value to the user.



of 63

req/software analyst

Classification: State-of-the-Art approaches

Prioritization Approaches Classification

using user knowledge either performing pair-wise comparison or not

using domain knowledge

using both

ü  Domain Knowledge: Encoded, reusable requirements details or requirement specifications ü  User Knowledge: When Requirement Analyst aware of complete domain knowledge



of 63

State-of-the-Art approaches (cont) Pairwise Comparison based approaches Analytic Hierarchy Process (AHP): comparing all unique pairs of requirements to determine which of the two is of higher priority, and to what extent [T.L. Saaty, ‘80]

Bubble Sort: compares two requirements & swap them if they are in the wrong order [Karlsson, ‘98] Cost-Value Approach: each requirement is determined on (i)  the value to the users (ii)  the cost of implementing, also uses the AHP technique [Karlsson & Ryan, ‘97] Case Based Ranking (CBRank): exploits a machine learning algorithm to guide the elicitation of user preferences during the prioritization process [Paolo Avesani et al ‘05]



of 63

State-of-the-Art approaches (cont.)

Non-Pairwise comparison based approaches Numerical Assignment: grouping requirements in different priority groups [Bradner, ‘97] MoScoW: groups all requirements into four priority groups MUST have, SHOULD have, COULD have, and WON’T have [Tudor and Walter, ‘06] Simple Ranking: requirements are ranked from integer 1 to N [Berander & Andrews, ‘05] Binary Search Tree: each node represents a requirement, lower priority requirements placed in left subtree, higher priority ones in right subtree, than node priority [Karlsson, ‘98] $100 Method: each stakeholder is asked to assume having $100 to distribute over the requirements in a ratio scale [Berander & Andrews ‘05]

Combining Techniques based approaches Planning Game: combination of two prioritization techniques i.e. Numerical Assignment & Simple Ranking [Beck, ‘99]

Domain Knowledge based approaches Priority Groups: dividing requirements into separate groups. then groups are ranked by using AHP [Joachim Karlsson] Genetic Algorithm: optimization is an application of GA & used in the problem of requirements prioritization too; uses domain knowledge [John Holland]



of 63

State-of-the-Art approaches (cont)

Summary Approach Cons

AHP Scalability Bubble Sort Scalability, time CBRank unable to accept constraints i.e. Dep Cost-Value Approach Time consuming

BST Sensitivity; a single error may build wrong tree GA Can’t resolve contradictory; $100 Method Longer; less confidence, biased MoScoW Ambiguous final ordering Simple Ranking unable handling complex scenarios

Scalability & handling of technical dependencies are the very common problems of all state-of-art approaches!



of 63

Approach A



Domain Knowledge: Prio & Dep

Domain knowledge includes two precedence graphs •  Prio •  Dep

How to build them? (encoded graph)

Prio: For 1st release, initial priorities Dep: For 1st release, dependencies

Prio

Dep



of 63

Genetic Algorithm 1. Acquisition and coding of set of Requirements and Domain Knowledge 2. Apply Genetic Algorithm (no user knowledge) 3. Output of the ranking (the most promising individual)



of 63

Genetic Algorithm Pseudo Code

The Canonical GA (a simple pseudo code is presented here):

1. choose initial population 2. evaluate each individual’s fitness REPEAT: 3. select best-ranking individuals to reproduce 4. apply crossover operator 5. apply mutation operator 6. evaluate each individual’s fitness until terminating condition (e.g. until at least one individual has the

desired fitness or enough generations have passed)



of 63

What is Population & Individuals?

Indv. ID Individual Disagree Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 >

Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 >

Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 >

Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 >

Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 >

Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 >

Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 >

Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 >

Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 >

Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 >

R1 R3 R2 R4 R5 R6 R7 R8 R9

Population!

A set of solution candidates

Pr1

An individual, also a solution candidate A requirement as a gene



of 63

Defining Individual’s Fitness?



Domain Knowledge Encoding Transform the domain knowledge into graphs

Req Priorities Dependencies R1 10 R2, R3, R7 R2 20 R3 R3 10 R4 20 R5 30 R4, R8 R6 40 R7 30 R3 R8 20 R9 50 R8, R5

Prio

Dep



of 63

Definition: Disagreement Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions

Disagreement: Between a pair of ordering (i.e. Pr1 and Prio | Dep | Eli), disagreement is the count of pairs that are inverted in two orderings. §  Lower disagreement defines higher fitness

Formally define:

17 Francis Palma - 'Interactive Requirements Prioritization; Using Search Based Optimization Techniques and Constraint Solver !

dis(pr1, pr2) = {(r,s)" pr1* | (s,r)" pr2*}

of 63

Disagreement Calculation Indv. ID Individual Disagr

Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 4 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 >

Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 >

Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 >

Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 >

Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 >

Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 >

Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 >

Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 >

Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 >

R1 R3 R2 R4 R5 R6 R7 R8 R9

Conflicts = {(R5, R8), (R6, R7), (R6, R8), (R7, R8)}

Prio

For Pr1



of 63

Disagreement Calculation (cont) Indv. ID Individuals Disagr.

Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 4 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 >

Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 >

Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 >

Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 >

Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 >

Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 >

Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 >

Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 >

Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 >

R1 R3 R2 R4 R5 R6 R7 R8 R9


Dep

For Pr1



of 63

Disagreement Calculation (cont)

Indv. ID Individuals Disagr Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 8 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 > 8

Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 > 16

Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 > 15

Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 > 23

Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 > 29

Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 > 30

Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 > 29

Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 > 17

Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 > 13

Total Conflicts = 4+4 = 8 {(R5, R8), Prio (R6, R7), (R6, R8), (R7, R8), (R1, R3), Dep (R1, R7), (R1, R2), (R5, R8)}

and so on…



of 63

GA Crossover Operator We used cut-head/fill-in-tail and cut-tail/fill-in-head…

R2 R3

Positions 5-6 as cut points

R8

Pr2

Pr3

Pr2’

R4 R1 R5 R8 R6 R7 R9

R5 R8 R1 R3 R7 R2 R9 R4 R6

R7 R9 R6 R2 R3 R4 R1 R5

- variation allows searching out different available niches, find better fitness values and subsequently better solutions - never produce chromosomes containing duplicate genes.



of 63

GA Mutation Operator

Mutation is basically a swap operator, we used requirement-pair-swap

R2 R3

Pr2 R4 R1 R5 R8 R6 R7 R9

Pr2’ R2 R3 R6 R1 R5 R8 R4 R7 R9

- mutation causes movement in the search space - may produce a stronger chromosome.



of 63

GA Selection Operator We used the Tournament Selection for our approach

- allows the selection pressure to be easily adjusted - faster than other selection operators i.e. Roulette Wheel Selection - better convergence



of 63

Our IGA Approach 1. Acquisition and coding of set of Requirements and Domain Knowledge 2. Apply Interactive Genetic Algorithm (exploiting User Knowledge) 3. Output of the ranking (the most promising individual)



of 63

Our Approach: Pairs to be Elicited Indv.

ID Individuals Disagree

Pr1 < R1, R3, R2, R4, R5, R6, R7, R8, R9 > 8 Pr2 < R2, R3, R4, R1, R5, R8, R6, R7, R9 > 8 Pr10 < R1, R4, R2, R3, R5, R6, R9, R8, R7 > 13

Pr4 < R4, R5, R6, R3, R2, R1, R8, R9, R7 > 15

Pr3 < R5, R2, R1, R3, R7, R8, R6, R9, R4 > 16

Pr9 < R1, R3, R5, R7, R9, R2, R4, R6, R8 > 17

Pr5 < R7, R8, R6, R5, R2, R3, R4, R9, R1 > 23

Pr8 < R8, R9, R6, R7, R4, R5, R2, R3, R1 > 29

Pr6 < R5, R6, R7, R8, R9, R1, R2, R3, R4 > 29

Pr7 < R9, R8, R7, R6, R5, R4, R3, R2, R1 > 30

Ranked individuals w. r. t. disagreement

Pr1 = < R1, R3, R2, R4, R5, R6, R7, R8, R9 > vs Pr2 = < R2, R3, R4, R1, R5, R8, R6, R7, R9 >

TIE PAIRS

Pr1, Pr2 (R1, R3), (R2, R3), (R6, R8), (R7, R8)

Candidate pairs to be asked to the decision maker



of 63

Our Approach: User Feedback

Ties appears in population, - Contradictory information w.r.t. the initial constraints - Nothing is expressed explicitly in the initial constraints. etc... ... Simple example: Why (R7, R8)? Contradictory or Ambiguous w.r.t. Prio & Dep..

TIE PAIRS

Pr1, Pr2 (R1, R3), (R2, R3), (R6, R8), (R7, R8)

Prio

Dep



of 63

Our Approach: User Feedback TIE PAIRS

Pr1, Pr2 (R1, R3), (R2, R3), (R6, R8), (R7, R8)

Eli

User Preference Graph eliOrd

Experience &

knowledge ‘<‘ or ‘>’

So, user knowledge is playing important role…



of 63

Our Approach: New Round with New Constraints

§  The new evolved population after using GA operators on population:

ü  Crossover ü  Mutation ü  Selection

§  Compared against the new set of constraints graphs

Eli Dep Prio



of 63

Evaluation of the Approach A



The Case Study: Target System (ACube) §  Prioritize requirements for a real software system, as part of the project *ACube (Ambient Aware Assistance)

designing a highly technological monitoring environment to be deployed in nursing homes to support medical and assistance staff

§  After end-user requirements analysis phase, 60 user requirements and 49 technical requirements Four macro-scenarios have been identified.

ID Macro-Scenario # of requirements

FALL Monitoring falls 26

ESC Monitoring escapes 23

MON Monitoring dangerous behavior 21

ALL The three scenarios 49

*ACube is a social welfare project coordinated by Fondazione Bruno Kessler (FBK) and funded by Autonomous Province of Trento under Bando Grandi Progetti, 2006. *http://acube.fbk.eu/



of 63

Case Study: The Overall Process Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


§  Collected user and technical requirements §  Identified macro-scenarios §  Tech. requirements has priority values (range 1-500) defined as a function that indicates priority-level of the tech. requirement w.r.t. the priority of the user requirements §  Software architect analyzed the requirement documents & extracted requirement implementation dependencies §  Software architect also defined a Gold Standard (GS) i.e. the ordering that would be followed at the end --- ---

§  With Prio info, Dep info and GS, we applied IGA Algorithm and generated Eli info during runtime, and evolve… §  IGA returns an ordering applying GA operators with user’s (Requirements Analyst) assistance. §  We then measure fitness against Gold Standard (GS)

of 63

Gold Standard (GS)

§  For each of the four macro-scenarios, we obtained the Gold Standard (GS) prioritization from the Software Architect of the ACube project

ü  The GS prioritization is the ordering given by the software architect to the requirements when planning their implementation during the ACube project.

Why Gold Standard?

ü  To measure disagreements with respect to GS. ü  To evaluate our approach in terms of disagreement against other non-interactive approaches using the same GS.



of 63

Research Questions



Research Question - 1 RQ1 (Convergence) Can we observe convergence with respect to the finally elicited fitness function?

-  Convergence is not obvious immediately, as Eli Graph is evolving at early stages. -  Although the full fitness function is known only at end of elicitation process.

YES!



of 63

Research Question - 2 RQ2 (Role of interaction) Does IGA produce improved prioritizations compared to non-interactive requirements ordering?

IGA outperforms substantially GA (and RAND), especially when a higher number of pairwise comparisons can be carried out

YES!



of 63

Research Question - 3 RQ3 (Role of initial precedence constraints) How does initial availability of precedence constraints affect the performance of IGA?

- Different types of Domain Knowledge affect IGA significantly - The improvement of IGA over GA is even higher when limited ranking information is available

Improves!



of 63

Research Question - 4 RQ4 (Robustness) Is IGA robust with respect to errors committed by the user during the elicitation of pairwise comparisons?

IGA can tolerate user errors up to 20%.

YES!



of 63

General Discussion §  Up to now we propose an Interactive approach for requirements prioritization

§  We experimented on a real project i.e. ACube

§  We also verified robustness w.r.t other non-interactive approaches Analytical Hierarchy Process (AHP):

- Most common approach for requirements prioritization… - Decision maker decides for every unique pairs based on different criteria… - For a system with N requirements and C criteria, AHP will have, total C × (C − 1) + C × N × (N − 1) / 2 comparisons (a huge manual task!) So, only with 25/50/100 (max) pairwise comparisons and acceptable performance IGA makes AHP obsolete



of 63

Approach B Using Constraints Solver: SMT



•  Based on Sa$sfiability Modulo Theory* and on Interac9ve Pairwise User Feedback – aims at minimizing the disagreement between

•  a total order of prioritized requirements (Gold Standard) •  constraints that are either encoded with the requirements

or expressed iteratively by the user during prioritization process

* a decision problem for logical formulas expressed in classical

first-order logic with equality or inequality

We try to overcome some of the limits of other approaches: exploiting user knowledge, minimizing the requests to users to pay attention to scalability problems, considering the possibility of arbitrary constraints and assuring robustness to errors

Our Approach Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

•  A set of Requirements

•  Domain Knowledge: available as requirements documenta9on (e.g., cost of the implementa9on, value for the stakeholders, dependencies between requirements) that can be converted into total or par9al rankings of the requirements

Req Cost Dependencies

R1 High R2, R3 R2 Low R3 R3 Low

R4 Medium R3 R5 Medium

•  Evaluation from users in terms of orderings between pairs of requirements

The Input Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

1. Acquisi9on and coding of Domain Knowledge for the set of requirements under analysis

2. Computa9on of solu9ons via SMT, also exploi9ng evalua9ons from decision makers

3. Output of the ranking

R3 R2

R1 R4

R1 R2

R3

R4

R5

Domain knowledge (constraints) User feedback

R1 R2 R3 R4

Set of requirements

Interac<ve SMT

R2

R1

R3

R4

<,> ?

The Main Approach Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

2.1. Computa9on of a set of solu9ons via SMT solver (Yices* tool) 2.2. If the number of solu9ons is grater than 1

–  then: iden9fica9on of conflicts (9es) between the solu9ons and the constraints

–  else: exit and return the solu9on 2.3. Request of knowledge to users to decide about conflicts 2.4. If max number of itera9ons / max pair comparison

–  then: exit and return the solu9on –  else: 2.1

The SMT Interactive Algorithm

*h^p://yices.csl.sri.com/



of 63

Approach Details



Req Value (Priori<es) Dependencies

R1 High R2, R3 R2 Low R3 R3 Low

R4 Medium R3 R5 Medium

Transform the domain knowledge into graphs

R3 R2

R1

R5 R4

Priorities

R3 R2

R1 R4

Dependencies

Domain Knowledge coding into Graphs Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

requirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminateswhen a unique solution at minimum disagreement is foundor the maximum allowed number of pairwise comparisons isreached.

Req Prio DepsR1 High R2, R3

R2 Low R3

R3 LowR4 Medium R2

R5 Medium

4 5

1

2 3

R

R R

R R

3 2

1 4

R R

R R

Prio Deps

Table 2: Requirements with priority and dependen-cies

Let us consider the five requirements listed in Table 2.For each requirement we consider the priority expressed bythe requirement engineer who collected them and the depen-dencies the requirement may have with other requirements.For conciseness, in Table 2 we omit other important ele-ments of a requirement (e.g., the textual description). Con-straints upon requirements can be represented by means ofa constraint graph. In a constraint graph, an edge betweentwo requirements indicates that, according to the relateddependency or priority, the requirement associated with thesource of the edge should be implemented before the targetrequirement. Edges may be weighted, to actually quantifythe strength or importance of each constraint. When en-coded in the input language of the SMT solver, such con-straints will be retractable constraints which are given theweight labeling the edge (or 1 if no weight is given). Aninfinite weight is used for constraints that must necessarilyhold in the final ordering of the requirements. These will benon-retractable constraints for the SMT solver.

At the right of Table 2, we show the constraint graphinduced by the priority (Prio) property of the requirementsand the constraint graph induced by the dependencies (Deps)between requirements. Requirements with High priority shouldprecede those with Medium priority. Hence the edges (R1, R4)and (R1, R5) in the graph Prio. Similarly, the precedence be-tween Medium and Low priority requirements induces thefour edges at the bottom of the constraint graph Prio. Re-quirement R1 depends on R2, R3, hence the implementationof R2, R3 should precede that of R1, which gives raise to theedges (R2, R1) and (R3, R1) in Deps.

It is possible to encode the requirements prioritizationproblem as a MAX-SAT2 problem. An SMT solver willfind an integer assignment which maximizes the weight ofthe retractable constraints that are satisfied by the solution(minimum cost of unsatisfied constraints). The encoding isdescribed formally in the next section. Intuitively, it con-sists of an assignment of positions (integers between 1 and2Technically we are exploiting a MAX-SMT solver.

(set-evidence! true)(define x::(-> nat nat))(define N::nat 5)

;; Only permutations of positions are allowed(assert (forall (i::(subrange 1 N)) (and (>= (x i) 1) (<= (x i) N))))(assert (forall (i::(subrange 1 (- N 1)))

(forall (j::(subrange (+ i 1) N)) (/= (x i) (x j)))))

;; Prio(assert+ (< (x 1) (x 4)) 1)(assert+ (< (x 1) (x 5)) 1)(assert+ (< (x 1) (x 2)) 1)(assert+ (< (x 1) (x 3)) 1)(assert+ (< (x 4) (x 2)) 1)(assert+ (< (x 4) (x 3)) 1)(assert+ (< (x 5) (x 2)) 1)(assert+ (< (x 5) (x 3)) 1)

;;Deps(assert+ (< (x 3) (x 2)) 1)(assert+ (< (x 3) (x 1)) 1)(assert+ (< (x 3) (x 4)) 1)(assert+ (< (x 2) (x 1)) 1)(assert+ (< (x 2) (x 4)) 1)

(max-sat)

Figure 1: Encoding of the constraints in Table 2 forthe SMT solver Yices

N , for N requirements) to the components of an integerarray x of size N . The assignment must define a permu-tation of the positions, hence there should be no repetitionof positions. This is easily encoded as a set of inequalitiesbetween pairs of positions (x[i] != x[j] " i != j). The otherconstraints (from the constraint graphs) are encoded as in-equalities (e.g., x[1] < x[4] and x[1] < x[5] for the two edgesat the top of the constraint graph Prio). The solutions tothe MAX-SAT problem produced by the SMT solver are allrequirement orderings that violate the minimum number ofretractable constraints (e.g., Deps and Prio), or have min-imum cost of violation, in case di!erent weights are givento di!erent constraints. The encoding of the requirementprioritization problem in Table 2 for the SMT solver Yices3

is shown in Figure 1.

Id Reqs DisagreePr1 < R1, R5, R4, R3, R2 > 4Pr2 < R1, R4, R5, R3, R2 > 4Pr3 < R1, R5, R3, R4, R2 > 4Pr4 < R1, R5, R3, R2, R4 > 4

Table 3: Prioritized requirements and related mini-mum disagreement

After encoding the constraint graphs in Table 2 as re-tractable assertions (assert+ in Figure 1) and running anSMT solver (e.g., Yices) on them, we obtain the four priori-tized lists of requirements shown in Table 3 (the solver mustbe run four times to obtain the complete set of prioritiza-tions, each time negating the previously found solutions).These are all prioritized lists having the same, minimumcost of retracted assertions (i.e., 4). Such a cost is called dis-agreement and it represents the number of constraints thatare not respected by the resulting requirement positions.The first two prioritizations Pr1 and Pr2 are in complete

3http://yices.csl.sri.com/

R3 R2

R1 R4

Dependencies

R3 R2

R1

R5 R4

Priorities

Retractable assertions

Problem SMT Encoding

Retractable assertions: assert+ can be retracted. (details on http://yices.csl.sri.com/)



(assert+ [expr] [weight])

of 63

requirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminateswhen a unique solution at minimum disagreement is foundor the maximum allowed number of pairwise comparisons isreached.

Req Prio DepsR1 High R2, R3

R2 Low R3

R3 LowR4 Medium R2

R5 Medium

4 5

1

2 3

R

R R

R R

3 2

1 4

R R

R R

Prio Deps

Table 2: Requirements with priority and dependen-cies

Let us consider the five requirements listed in Table 2.For each requirement we consider the priority expressed bythe requirement engineer who collected them and the depen-dencies the requirement may have with other requirements.For conciseness, in Table 2 we omit other important ele-ments of a requirement (e.g., the textual description). Con-straints upon requirements can be represented by means ofa constraint graph. In a constraint graph, an edge betweentwo requirements indicates that, according to the relateddependency or priority, the requirement associated with thesource of the edge should be implemented before the targetrequirement. Edges may be weighted, to actually quantifythe strength or importance of each constraint. When en-coded in the input language of the SMT solver, such con-straints will be retractable constraints which are given theweight labeling the edge (or 1 if no weight is given). Aninfinite weight is used for constraints that must necessarilyhold in the final ordering of the requirements. These will benon-retractable constraints for the SMT solver.

At the right of Table 2, we show the constraint graphinduced by the priority (Prio) property of the requirementsand the constraint graph induced by the dependencies (Deps)between requirements. Requirements with High priority shouldprecede those with Medium priority. Hence the edges (R1, R4)and (R1, R5) in the graph Prio. Similarly, the precedence be-tween Medium and Low priority requirements induces thefour edges at the bottom of the constraint graph Prio. Re-quirement R1 depends on R2, R3, hence the implementationof R2, R3 should precede that of R1, which gives raise to theedges (R2, R1) and (R3, R1) in Deps.

It is possible to encode the requirements prioritizationproblem as a MAX-SAT2 problem. An SMT solver willfind an integer assignment which maximizes the weight ofthe retractable constraints that are satisfied by the solution(minimum cost of unsatisfied constraints). The encoding isdescribed formally in the next section. Intuitively, it con-sists of an assignment of positions (integers between 1 and2Technically we are exploiting a MAX-SMT solver.

(set-evidence! true)(define x::(-> nat nat))(define N::nat 5)

;; Only permutations of positions are allowed(assert (forall (i::(subrange 1 N)) (and (>= (x i) 1) (<= (x i) N))))(assert (forall (i::(subrange 1 (- N 1)))

(forall (j::(subrange (+ i 1) N)) (/= (x i) (x j)))))

;; Prio(assert+ (< (x 1) (x 4)) 1)(assert+ (< (x 1) (x 5)) 1)(assert+ (< (x 1) (x 2)) 1)(assert+ (< (x 1) (x 3)) 1)(assert+ (< (x 4) (x 2)) 1)(assert+ (< (x 4) (x 3)) 1)(assert+ (< (x 5) (x 2)) 1)(assert+ (< (x 5) (x 3)) 1)

;;Deps(assert+ (< (x 3) (x 2)) 1)(assert+ (< (x 3) (x 1)) 1)(assert+ (< (x 3) (x 4)) 1)(assert+ (< (x 2) (x 1)) 1)(assert+ (< (x 2) (x 4)) 1)

(max-sat)

Figure 1: Encoding of the constraints in Table 2 forthe SMT solver Yices

N , for N requirements) to the components of an integerarray x of size N . The assignment must define a permu-tation of the positions, hence there should be no repetitionof positions. This is easily encoded as a set of inequalitiesbetween pairs of positions (x[i] != x[j] " i != j). The otherconstraints (from the constraint graphs) are encoded as in-equalities (e.g., x[1] < x[4] and x[1] < x[5] for the two edgesat the top of the constraint graph Prio). The solutions tothe MAX-SAT problem produced by the SMT solver are allrequirement orderings that violate the minimum number ofretractable constraints (e.g., Deps and Prio), or have min-imum cost of violation, in case di!erent weights are givento di!erent constraints. The encoding of the requirementprioritization problem in Table 2 for the SMT solver Yices3

is shown in Figure 1.

Id Reqs DisagreePr1 < R1, R5, R4, R3, R2 > 4Pr2 < R1, R4, R5, R3, R2 > 4Pr3 < R1, R5, R3, R4, R2 > 4Pr4 < R1, R5, R3, R2, R4 > 4

Table 3: Prioritized requirements and related mini-mum disagreement

After encoding the constraint graphs in Table 2 as re-tractable assertions (assert+ in Figure 1) and running anSMT solver (e.g., Yices) on them, we obtain the four priori-tized lists of requirements shown in Table 3 (the solver mustbe run four times to obtain the complete set of prioritiza-tions, each time negating the previously found solutions).These are all prioritized lists having the same, minimumcost of retracted assertions (i.e., 4). Such a cost is called dis-agreement and it represents the number of constraints thatare not respected by the resulting requirement positions.The first two prioritizations Pr1 and Pr2 are in complete

3http://yices.csl.sri.com/

R3 R2

R1 R4

Dependencies

R3 R2

R1

R5 R4

Priorities

Completing the Encoding Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

solu<on Requirements rankings Disagree

Pr1 < R1,R5,R4,R3,R2 >

Pr2 < R1,R4,R5,R3,R2 >

Pr3 < R1,R5,R3,R4,R2 >

Pr4 < R1,R5,R3,R2,R4 >

R1

R5

R4

R3

R2


!


R3 R2

R1 R4

Dep

R1

R5

R4

R3

R2

4

4

4

4

Production of Solutions & Identification of Conflicts via SMT



Prio

of 63

solu<on Requirements rankings Disagree

Pr1 < R1,R5,R4,R3,R2 >

Pr2 < R1,R4,R5,R3,R2 >

Pr3 < R1,R5,R3,R4,R2 >

Pr4 < R1,R5,R3,R2,R4 >

TIE PAIRS

Pr1, Pr2, (R4, R5)

Pr1, Pr3 (R3, R4)

… …

PR1 = < R1,R5,R4,R3,R2 > vs PR2 = < R1,R4,R5,R3,R2 > (R4,R5)

Candidate pairs to be asked to decision maker

Solutions at minimum disagreement

4

4

4

4

Pairs to be Evaluated from the Decision Maker Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

R3 R2

R1

R5 R4

Priorities

R3 R2

R1 R4

Dependencies

Why (R4,R5) ? Nothing is said about (R4,R5) in the Priorities and Dependencies graphs

Why (R3,R4) ?

Contradiction in ordering (R3,R4)

TIE PAIRS

Pr1, Pr2, (R4, R5)

Pr1, Pr3 (R3, R4)

… … <>?

R3

R4 R4

R5


< or >

User Feedback Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

R3 R2

R1

R5 R4

Priorities

R3 R2

R1 R4

Dependencies

R3

R4

R5


New Graph eliOrd from User Feedback Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

Evaluation of Approach B

52

•  Priori9ze requirements for a real soaware system, as part of the project ACube (Ambient Aware Assistance) –  designing a highly technological monitoring environment to be deployed in nursing

homes to support medical and assistance staff •  Aaer user requirements analysis phase,

–  60 user requirements where retrieved that produced 49 technical requirements –  Four macro-‐scenarios have been iden9fied.

Id Macro-‐scenario # of requirements

FALL Monitoring falls of pa9ents 26

ESC Monitoring escapes of pa9ents 23

MON Monitoring dangerous behavior of pa9ents 21

ALL The three scenarios 49

Case Study: ACube Project Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

•  Two sets of technical constraints: –  Prio graph: a func9on that associates technical requirement to a number

indica9ng the priority of the technical requirement with respect to the priority of the user requirements it is intended to address

–  Dep graph: is defined on the basis of the dependencies between technical requirements

•  For each of the four macro-‐scenarios, we considered as the Gold Standard (GS) the priori9za9on from the soaware architect of the ACube project –  The GS priori9za9on is the ordering given by the soaware architect to the

requirements when planning their implementa9on during the ACube project

•  Comparison with “Incomplete Analy9c Hierarchy Process” and “Interac9ve Gene9c Algorithm”

Ingredients for the Evaluation Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

•  Is a variant of the Analy9c Hierarchy Process (AHP) •  Pairwise comparison itera9ve process in which the decision

maker is required to specify a preference between two requirements Ri and Rj as an integer value pij∈[1 . . . 9]

coming from other sources.We evaluated our SMT approach comparing it with other

state-of-the-art interactive prioritization techniques (IAHPand IGA in particular). Results indicate that SMT substan-tially outperforms them, while keeping the user e!ort (interms of number of elicited pairs) acceptable. Moreover, wecompared interactive and non-interactive prioritization, ob-serving an increased performance when interaction with thedecision maker takes place. We also evaluated the robust-ness of our method with respect to decision maker’s errors.

This paper is structured as follows: in Section 2 we de-scribe the background for our work, Section 3 presents theproposed approach and algorithm. In Section 4 we commenta set of experimental evaluations on e!ectiveness, robust-ness and e"ciency of the approach. The empirical assess-ment was conducted on a set of requirements from a realhealthcare project. Relevant related works are discussed inSection 5. Conclusions and future work are presented inSection 6.

2. BACKGROUNDHere we briefly introduce two quite successful, state-of-

the-art methods for interactive prioritization of requirements:Incomplete Analytic Hierarchy Process (IAHP) [4] and In-teractive Genetic Algorithm (IGA) [14]. Both methods havethe objective of synthesizing an approximation of the rank-ing of a set of N requirements, minimizing the e!ort asso-ciated with the interactive input required from the decisionmaker (consisting of pairwise comparisons).

2.1 Incomplete AHPIAHP [4] is a variant of the Analytic Hierarchy Process

(AHP) [12] that can deal with incomplete information (pair-wise comparisons) from the user. It was designed to supportscalability of AHP to requirements whose size make AHPprohibitively expensive (the number of pairs that must beelicited to apply AHP is quadratic with the number of re-quirements).

In particular, AHP implements a pairwise comparison it-erative process in which the decision maker is required tospecify a preference as an integer value pij ! [1 . . . 9], be-tween two requirements Ri and Rj . The value can be mappedto a qualitative measure of the preference relation, as sum-marized in Table 1. After assigning a preference value to oneof the two requirements (say, Ri) compared to the other one(Rj), the preference value for the least preferred requirement(Rj) is the reciprocal of pij , so, pji = 1/pij . When all thepairs have been elicited, a ranking is synthesized throughthe computation of the principal eigenvector of the matrixpij , the components of which determine the rank of eachrequirement.

Preference pij Definition

1 Ri Equally important to Rj

3 Ri Moderately more important than Rj

5 Ri Strongly more important than Rj

7 Ri Very strongly more important than Rj

9 Ri Extremely more important than Rj

2, 4, 6, 8 For compromise between the above values

Table 1: A possible fundamental scale in the interval[1 . . . 9] used with AHP

IAHP overcomes the AHP scalability problem by mini-mizing the number of pairs elicited from the decision maker,while maintaining a good trade-o! between the precision ofthe final solution and the e!ort of the decision maker. Thisis done by calculating, at each iteration of the elicitationprocess, a prediction of the next most promising pair to beasked to the decision maker in order to find a stable andgood approximation of the target ranking, early before elic-iting all the N(N "1)/2 pairs for the set of N requirements.

The high level steps of the IAHP process are: (i) thedecision maker provides N "1 judgments which form a con-nected graph; (ii) using the available pairwise comparisons,the missing comparisons are estimated by taking the geomet-ric mean of the intensities over a spanning tree (this resultsin a weight matrix); (iii) the derivatives of the weight matrixwith respect to the missing matrix elements are computedand the next pairwise comparison to ask is determined basedon such derivatives; (iv) if the stopping criteria are met (e.g.,maximum number of elicitations or convergence metrics be-low threshold), then the computation stops and a rank isproduced by calculating the principal eigenvector of the es-timated matrix; otherwise the selected comparison is elicitedand the algorithm iterates.

2.2 Interactive GAIn [14] we presented an Interactive Genetic Algorithm

(IGA) to achieve the minimization of the disagreement be-tween a total order of prioritized requirements and the var-ious constraints coming from the domain knowledge, thatare either encoded with the requirements or expressed iter-atively by the user during the prioritization process.

In the approach we exploit the interactive input from theuser whenever the fitness function cannot be computed pre-cisely based on the information available. Specifically, eachindividual in the population being evolved by the genetic al-gorithm represents an alternative prioritization Pri of theN requirements. Two examples of such individuals are:Pr1 = #R1, R5, R4, R3, R2$, or Pr2 = #R1, R4, R5, R3, R2$.When individuals having a high fitness (i.e., a low disagree-ment with the constraints) cannot be distinguished, sincetheir fitness function evaluates to a plateau, user input isrequested interactively, in the form of preferences betweenpairs of requirements, so specifying if Ri is before or afterRj in the user ideal ranking, so as to make the fitness func-tion landscape better suited for further minimization. Inthis way, knowledge about the constraints in the domaincan help the method to minimize the amount of knowledgeelicited from the decision maker, so decreasing her/his ef-fort. The prioritization process terminates when a low dis-agreement is reached, a time out is reached or the allocatedelicitation budget is over.

3. APPROACHThe prioritization approach we propose aims at minimiz-

ing the disagreement between a total order of prioritizedrequirements and the various constraints that are either en-coded with the requirements or that are expressed itera-tively by the user during the prioritization process. Weuse SMT solvers to achieve such a minimization, takingadvantage of interactive input from the requirement engi-neer whenever the solver produces more than one prioritizedlist of requirements having the same disagreement with theavailable constraints. The prioritization process terminates

**P. T. Harker. Incomplete Pairwise Comparisons in the Analytic Hierarchy Process. Math. Modelling, 9(11):837 – 848, 1987 **T. L. Saaty and L. G. Vargas. Models, Methods, Concepts & Applications of the Analytic Hierarchy Process. Kluwer Academic, 2000

I-AHP Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

•  The IGA method consists in a Gene<c Algorithm associated to a Pairwise Interac<ve and Itera<ve process to acquire user preferences.

•  Differently from IAHP, the priori9za9on elicited by the user is strict (Ri less or more important than Rj, no range as to be specified)

P. Tonella, A. Susi, and F. Palma. Using Interactive GA for Requirements Prioritization. In 2nd International Symposium on Search Based Software Engineering, pages 57–66. IEEE, 2010

Interactive Genetic Algorithm Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

RQ1 (Comparison) Does the SMT-based method produce improved prioritizations compared to IAHP and IGA?

Figure 2: Disagreement with GS after eliciting atmost 25 (top), 50 (middle) and 100 (bottom) pairsfrom the user in the ALL scenario

RQ3 (Robustness).In Figure 4, we show that even if the performance of the

SMT-based interactive approach degrades at increasing usererror rates, it always produces an improved prioritizationcompared to IGA and IAHP with a user who does not makeany mistake. We show this for 50 elicited pairs, but sim-ilar results are available for a di!erent number of elicitedpairs. With no error SMT produces a prioritization with adisagreement of 90. At 5%, 10% and 20% error rate, thedisagreement becomes 92, 95 and 98 respectively. On theother hand, even if the user makes no error (i.e. error-freeuser), the disagreement of the prioritizations produced byIGA and IAHP is 120 and 208. We obtained similar plotsfor the other three macro scenarios (MON, FALL, ESC).

In Figure 5 we compare the robustness of SMT with thetwo other approaches considered in this work, IGA and IAHP.Here we show the results obtained for ALL. Similar resultshave been obtained for the other macro scenarios. The SMT-based interactive approach is substantially more robust thanthe others. At increasing user error rates, the performanceof IGA and IAHP is degrading to a higher degree. SMT

Figure 3: Disagreement (top) and average dis-tance (bottom) with GS for interactive and non-interactive SMT after eliciting 25, 50, 100 pairs fromthe user in the ALL scenario

has also a degrading performance at increased error rates,but it outperforms IGA and IAHP to a large extent. For anerror rate of 5% SMT, IGA and IAHP have a disagreementof 92, 120 and 236 respectively. For a 10% error rate, theseare 95, 119 and 277. For an error rate of 20%, the disagree-ments with GS become 98, 123 and 331 respectively. So,clearly the SMT-based interactive approach is more robustthen IGA and IAHP.

RQ4 (Performance).Table 8 shows the comparison among execution times of

SMT, IGA and IAHP. All execution times in the table areobtained from the system time of each execution and aremeasured in seconds, with a very small observable variabil-

Figure 4: Robustness of SMT: Disagreement forSMT at di!erent user error rates; IGA and IAHPwith error-free user, after eliciting 50 pairs in theALL scenario

!


RQ1: Comparison SMT, IGA, IAHP Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

RQ2 (Role of interaction) Does SMT-based interactive prioritization produce improved prioritizations compared to non-interactive SMT-based prioritization?










!


RQ2: Role of Interaction Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

RQ3 (Robustness) Is the SMT-based method more robust than IAHP and IGA with respect to errors committed by the user during the elicitation of pairwise comparisons?










!


RQ3: Robustness Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

•  Again cost/benefit trade off offered by Interac9ve SMT as compared to IAHP and IGA is more acceptable

Threats

•  External validity (generaliza9on of the findings): –  we used only 1 real project with 4 sub-‐scenarios

•  Construct Validity: –  Other performances measures should be considered (not only disagreement)

– More complex user error models should be considered

Discussion Outline Problem RelatedWorks GeneticAlgo ApproachA CaseStudy ResultsA ApproachB ResultsB OngoingWorks Conclusions


of 63

Algorithm selection

Experiment parameters setting

Pairwise elicitation

The Prioritization Tool & Technical Report

TR-FBK-SE-2011-3: Francis Palma, Angelo Susi, Paolo Tonella: Using an SMT Solver for Interactive Requirements Prioritization. Tech. Rep. FBK-SE (Mar. 2011) 61 of 63

Conclusions & Future Works q  We proposed an Interactive Genetic Algorithm & an Interactive SMT to collect pairwise information useful to prioritize the requirements

q  We also verified the robustness of the algorithms with respect to increased error-prone responses from analysts

q  We evaluated the approach in a real project (ACube)

q  In summary, we contributed NOVEL APPROACH to prioritize requirements What’s Next? Experiment

- On-line: Empirical Study with real object (i.e. human/analyst)



of 63

Publications

- Paolo Tonella, Angelo Susi, and Francis Palma. Using Interactive GA for Requirements Prioritization. In 2nd International Symposium on Search Based Software Engineering (SSBSE 2010), pages 57–66. IEEE, 2010 (Under minor review for the Journal of Information and Software Technology (IST)) - Francis Palma, Angelo Susi, Paolo Tonella, Using an SMT solver for interactive requirements prioritization, in Tibor Gyimothy and Andreas Zeller, SIGSOFT FSE, ACM, 2011, pp. 48-58 (19th ACM SIGSOFT Symposium on the Foundations of Software Engineering, Szeged, Hungary)da 09/05/2011 a 09/09/2011



of 63

Thank You! …and

Questions?


64