digging for diamonds: identifying valuable web automation programs in repositories jarrod jackson 1,...
TRANSCRIPT
Digging for diamonds: Identifying Digging for diamonds: Identifying valuable web automation programs valuable web automation programs
in repositoriesin repositories
Jarrod Jackson1, Chris Scaffidi2, Katie Stolee2
1 Oregon State University2 University of Nebraska - Lincoln
22
Web scripts:Web scripts:Enabling users to enhance the browserEnabling users to enhance the browser
IBM CoScripter Web Macro
Problem Approach Evaluation Conclusion
33
Web scripts:Web scripts:Enabling users to enhance the browserEnabling users to enhance the browser
Yahoo Pipe
Problem Approach Evaluation Conclusion
44
Web scripts:Web scripts:Enabling users to enhance the browserEnabling users to enhance the browser
GreaseMonkey UserScript
Problem Approach Evaluation Conclusion
55
Repositories of end-user code:Repositories of end-user code:The good, the great, and the “other”The good, the great, and the “other”
C. Bogart, et al. End-User Programming in the Wild: A Field Study of CoScripter Scripts. VL/HCC 2008.
Previous study:
Of 1445 web macros…~ 10% had many runs~ 10% had many users~ 80% were “other”
This is the largest web macro repository> 6000 users, > 3000 “public” scripts
Problem Approach Evaluation Conclusion
66
What if our repositories could…What if our repositories could…
• … omit pieces of code from search results if they are unlikely to be reused, anyway?
• ... provide a UI for administrators to review (and remove?) old code that’s unlikely to be used?
• … advise programmers, when they upload code, about how to improve the reusability of their code?
Problem Approach Evaluation Conclusion
77
Needed: a model for predicting reuseNeeded: a model for predicting reuse
• Key questions for discovering such a model…– What information about the code indicates reusability?– How do we combine this information to predict reuse?
• Similar models have been successful on OO code– Predicting reuse based on coupling & cohesion– Predicting bugginess based on code complexity metrics,
information about code authors, code churn, …
Web scripts are much simpler (don’t call each other, don’t have inheritance, etc)… we need different information here.
Problem Approach Evaluation Conclusion
88
Prior work found 35 traits (in 8 Prior work found 35 traits (in 8 categories) statistically related to reusecategories) statistically related to reuse• Mass appeal – eg popular keywords• Language – eg data values are in English• Annotations – eg comments• Flexibility – eg parameterization (variables)• Length – eg small # distinct lines of code• Author information – eg early adopter?• Advanced syntax – eg “control-click” keyword• No Preconditions – eg no cookies needed
All traits are computed automatically from one of four sources: executable code statements, URLs referenced, annotations, code history.
Problem Approach Evaluation Conclusion
99
• Given a binary measure of reuse, for each trait– Find the threshold that optimally divides the reused
scripts from the un-reused scripts
Model that we developed Model that we developed (in words & pictures)(in words & pictures)
Tra
it le
vel
Threshold
Problem Approach Evaluation Conclusion
1010
Predicting if a macro will be reusedPredicting if a macro will be reused
• Count how many predictors are satisfied
• Predict that the macro will be reused if this count exceeds some minimum– A tunable parameter– A higher minimum implies a higher bar that a script
must overcome to be predicted as to be reused• Fewer false positives, higher false negatives
Problem Approach Evaluation Conclusion
1111
ExampleExample
• E.g.: Suppose that our measure of reuse is “script is reused more than 75% of other scripts”
• Suppose that based on this measure of reuse, the best thresholds for four predictors are…
comments ≥ 3 lines_of_code ≥ 40 prev_created ≥ 10 literals ≤ 4
• The model would predict that some other script would satisfy the reuse measure criterion if the script satisfies at least n of these predictors
Problem Approach Evaluation Conclusion
1212
How well does this approach work…How well does this approach work…
• … for different kinds of web scripts?
• … for different reuse measures?
• … when predicting future reuse based on past data?
• … when only a subset of traits are available?
Problem Approach Evaluation Conclusion
1313
Scripts and measures Scripts and measures for our evaluationfor our evaluation
Problem Approach Evaluation Conclusion
Measure cutoff
1414
Accuracy varied little by measure or Accuracy varied little by measure or script type (e.g., TP ≥ 0.7 at FP = 0.4)script type (e.g., TP ≥ 0.7 at FP = 0.4)
Problem Approach Evaluation Conclusion
1515
Yahoo Pipe accuracy slipped a bit Yahoo Pipe accuracy slipped a bit when using past to predict futurewhen using past to predict future
Problem Approach Evaluation Conclusion
1616
Code-based traits gave nearly the full Code-based traits gave nearly the full accuracyaccuracy
(History, URL, Annotations, Code)
Problem Approach Evaluation Conclusion
1717
ConclusionsConclusions
• Model is equally accurate for a range of uses– And might only require code-based traits
Problem Approach Evaluation Conclusion
1818
ConclusionsConclusionsand future workand future work
• Model is equally accurate for a range of uses– And might only require code-based traits– But can we improve accuracy by using information
available after reuse is attempted?– Can we also predict how happy people will be when
reusing different pieces of code?
• And now to put the model to work…– Improving search engines– Providing UI for administrators to review macros– Giving programmers advice automatically
Problem Approach Evaluation Conclusion
1919
Thank YouThank You
To ICISA for this opportunity to present this paper
Problem Approach Evaluation Conclusion
2020
So how do we separate the So how do we separate the wheat from the chaff?wheat from the chaff?
• Providing such features requires predicting whether code will ever be reused
– Without relying on information that’s available after code is reused (“chicken and egg”)
• Ratings, reviews, etc…• (For some features, of course, we can always add this
information in later.)
– With a fairly simple model for making predictions• So that predictions can be explained to users• Especially when we’re advising users about how to improve
reusability of their code!!!!!
Problem Approach Evaluation Conclusion