Which Feature Location Technique is Better?

Emily Hill, Alberto Bacchelli, Dave Binkley, Bogdan Dit,

Dawn Lawrie, Rocco Oliveto

Motivation: Differentiating FLTs

Totally unrelated

In vicinity

Precision = 0.20 Precision = 0.20

Example• Developer works down ranked list• At each item can explore or not• When exploring structure, can bail

at any time

Proposed Approach: Rank Topology

• Use evaluation measures that consider the likelihood of a developer finding fix locations

• Use textual information to approximate developer’s interest (i.e., likelihood) of following “trail” in structural topology, starting from ranked list

• Rank topology = inverse of the number of hops in topology

Example• Developer works down ranked list• At each item can explore or not

• 3rd rank result + 4 structural hops = 7 total hops

• Rank topology metric = 1 / 7

• No discrimination: explores everything

How “smart” is the user?

• Semi-intelligent: only follows a structural hop if the next method exhibits textual clues– Rank topology uses VSM cosine similarity (tf-idf)– Structural edge added if both methods > median

scores for query– Supported by user studies of information foraging

theory [Lawrance, et al TSE 2013]

• Omniscient: makes no wrong choices, exploring only those ranks and structural hops that lead to a bug

Preliminary Study: Distinguish QLM from Random

Ranked list of results all have same bug fixes at exactly the same ranks

Conclusion• Rank topology differentiates between

randomly ordered lists and a state of the art IR technique (QLM) with relevant results at the exact same ranks

• Future work– How well does rank topology mimic developer

behavior in practice?– How closely can/should we model user behavior?

• Our question: Does the research community need to revise how we evaluate FLTs?

Preliminary Study

• Effect of program structure on the rank topology metric for each JabRef bug used in the case study.