on the value of user preferences in search-based software engineering
DESCRIPTION
Paper presented at the 35th International Conference in Software Engineering, San Francisco, USA. May 2013TRANSCRIPT
On the Value of User Preferences in Search-Based Software Engineering:
A Case Study in Software Product Lines
Abdel Salam Sayyad Tim MenziesHany Ammar
West Virginia University, USA
International Conference on Software EngineeringMay 23rd, 2013
2
Sound bites
Feature-based SERepresentation and validation: importantMulti-objective Configuration: interesting
Stop tinkering with small stuff!Minor improvements welcome…
… but not insightful!
Enough with the usual suspects: NSGA-II, SPEA2, etc.
If user preferences matterThen the best optimizer understands preferences the best
Roadmap
① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion
Roadmap
① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion
5
Welcome to the new world!Old world:
product-based SE
e.g. Microsoft office
New world: app-based SE
e.g. Apple app store
6
The times, they are a changin’
Old world: product-based SEe.g. Microsoft office • Vendors tried to retain their
user base via some complete ecologies
• One software solution for all user needs (e.g. Microsoft Office).
• Large, complex, software platforms, – very slow to change.
7
The times, they are a changin’
New worlde: app-based SE • Smart phones and tablet-
based software
• Users choosing many numbers of small apps from different vendors, – each performing a specific
small task.
• Vendors must quickly and continually reconfigure apps – To retain and extend their
customer base.
e.g. Apple app store
8
Feature–oriented domain analysis [Kang 1990]
• Feature models = a lightweight method for defining a space of options
• De facto standard for modeling variability
Cross-Tree Constraints
Cross-Tree Constraints
9
What are the user preferences?• Multi-objective optimization = navigating competing concerns
– Success criteria = choose features that achieve the user preferences!• Suppose each feature had the following metrics:
1. Boolean USED_BEFORE?2. Integer DEFECTS3. Real COST
• Show me the space of “best options” according to the objectives:1. That satisfies most domain constraints (0 ≤ #violations ≤ 100%)2. That offers most features3. That we have used most before4. Using features with least known defects5. Using features with least cost
10
Diving deeper• Much prior work in SBSE (*)
--------------------------(*) Sayyad and Ammar, RAISE’13
explored tiny objective spaces (2 or 3 objectives)
Used NSGA-II
Didn’t state why!
Roadmap
① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion
12
Multi-objective Optimization
Higher-level Decision Making
The Pareto Front
The Chosen Solution
13
Survival of the fittest(according to NSGA-II [Deb et al. 2002])
Boolean dominance (x Dominates y, or does not):- In no objective is x worse than y- In at least one objective, x is better than y Crowd
pruning
14
Tinkering with small stuff…
ssNSGA-II: steady state [Durillo ‘09]
Other Algorithms: Ranking criteria:NSGA-II
SPEA2 [Zitzler ‘01] More focus on diversity, with a new diversity measure.
FastPGA [Eskandari ‘07]Borrowed criteria from NSGA-II, and diversity measure from SPEA2
MOCell (Cellular GA) [Nebro ‘09] NSGA-II
NSGA-IIMOCHC (re-designed selection, crossover, and mutation) [Nebro ‘07]
Boolean Dominance
Boolean Dominance
Boolean Dominance
Boolean Dominance
Boolean Dominance
15
Indicator-Based Evolutionary Algorithm (IBEA) [Zitzler and Kunzli ‘04]
1) For {old generation + new generation} do– Add up every individual’s amount of dominance with
respect to everyone else
– Sort all instances by F– Delete worst, recalculate, delete worst, recalculate, …
2) Then, standard GA (cross-over, mutation) on the survivors Create a new generation Back to 1.
Continuous Dominance
16
An important property of IBEA
• Allows solutions to crowd close to the zero point.
• In our problem: Allows many solutions that achieve violations equal or close to zero… then allows optimization in the other 4 dimensions…
• Crowd pruning can hurt!
• Using real-valued benchmark functions, Wagner et al. (EMO 2007) confirm that the performance of NSGA-II and SPEA2 rapidly deteriorates with increasing dimension, whereas indicator-based algorithms (e.g. IBEA) cope very well with high-dimensional objective spaces.
Roadmap
① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion
18
2 Case studies from SPLOT
Cross-treeconstraints
Algorithms
• 7 Algorithms from jMetal: http://jmetal.sourceforge.net/ – IBEA, NSGA-II, ssNSGA-II, SPEA2, FastPGA, MOCell, MOCHC.
19
• Features are bits in the “product” string.
• Same settings for all algorithms.
20
4 studies:Two, three, four, five- objectives
The five objectives:1. satisfy most domain constraints (0 ≤ violations ≤ 100%)2. offer most features3. most used before4. least known defects5. least cost
Two objectives = 1,2Three objectives = 1,2,5Four objectives = 1,2,3,5Five objectives = 1,2,3,4,5
21
HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints
22Comment 1: all about the same for the 2-objective problem
HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints
23Comment 2: IBEA is better on HV.
HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints
24Comment 3: IBEA has no spread operators, but gets best spread
HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints
25Comment 4: All the non-IBEA algorithms are very similar
HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints
26
HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints
Comment 5: IBEA does much, much better on constraints
27
HV = hypervolume of dominated regionSpread = coverage of frontier% correct = % of solutions that fully satisfy the constraints
Comment 6: IBEA’s advantage is more striking with E-Shop
28
Why is this interesting?
Other Algorithms• The usual suspects are widely,
uncritically used in many applications– E.g. especially NSGA-II and SPEA2
• Focused on internal algorithmic tricks– Techniques for
• improving spread • Improving HV• Avoid overlaps in cross-over of
dominated space• etc.
• And the net effect of all those differences?– Not much
IBEA• Rather stupid on those
internal tricks– Just does same old crossover
mutate GA– Plus: aggressive exploration
of the preference space
• And the net effect of all those differences– Better spreads– Better HV– Fewer constraint violations
Conclusion: preference is power
Roadmap
① Feature-based SE② Evolutionary Algorithms③ Experiment & Results④ Conclusion
30
Current Achievements and Future Work
• ICSE’13: initial results proving IBEA’s advantage.• CMSBSE’13: IBEA runtime improvement.• (Under review): – mutation operator tailored to feature tree. 2-3
orders-of-magnitude improvement in runtime.– Scalability to 6000+ real feature model using
innovative “population seeding” method.
• Future Work:– Further scalability.– Interactive configuration.
31
Sound bitesThe new age of the app
In this new world: use FODA(feature-oriented domain analysis)
Stop tinkering with small stuff Many MEOAs have strikingly similar performance
Enough with the usual suspects: NSGA-II, SPEA2, etc.Too much uncritical application of these algorithms
If user preferences matterThen the best optimizer understands preferences the bestIBEA: aggressive preference exploration
AcknowledgmentThis research work was
funded by the Qatar National Research Fund
(QNRF) under the National Priorities Research Program
(NPRP) Grant No.: 09-1205-2-470.