generating tests for mobile apps from natural …€¦ · this is fu*king annoying. fix it already!...
TRANSCRIPT
GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG
REPORTS AND APP REVIEWS
Partially supported by: NSF, IBM, MSR, and Google
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
Joint work with Marcelo d’Amorim and Paola Spoletini
SNEAK PREVIEW
SNEAK PREVIEW
GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG
REPORTS AND APP REVIEWS
Partially supported by: NSF, IBM, MSR, and Google
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
Joint work with Marcelo d’Amorim and Paola Spoletini
SNEAK PREVIEW
GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG
REPORTS AND APP REVIEWS
Partially supported by: NSF, IBM, MSR, and Google
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
Joint work with Marcelo d’Amorim and Paola Spoletini
Field failures are
unavoidable!
SNEAK PREVIEW
GENERATING TESTS FOR MOBILE APPS FROM NATURAL LANGUAGE BUG
REPORTS AND APP REVIEWS
Partially supported by: NSF, IBM, MSR, and Google
Alessandro (Alex) OrsoSchool of Computer Science – College of Computing
Georgia Institute of Technology
Joint work with Marcelo d’Amorim and Paola Spoletini
Field failures are
unavoidable!
TYPICAL DEBUGGING PROCESS
Very hard to(1) reproduce(2) debug
BugRepository
TYPICAL DEBUGGING PROCESS
Very hard to(1) reproduce(2) debug
BugRepository
OVERARCHING GOAL: help developers (1) investigate field failures,
(2) understand their causes, and(3) eliminate such causes.
Recent survey of Apache, Eclipse, and Mozilla developers:
Information on how to reproduce field failures is the most valuable, and difficult to obtain, piece of information for investigating such failures.[Zimmermann10]
OUR WORK SO FARRecording and replaying executions[icsm 2007, icse 2007]
Input anonymization [icse 2011]
Input minimization [woda 2006, icse 2007]
✘
Mimicking field failures [icse 2012, ase 2013, icst 2014]
Locating and explaining field failures [issta 2012, issta 2013, ASE 2014, TOSEM 2015, TR]
MIMICKING FIELD FAILURESUser run (R) Relevant events
(breadcrumbs)Mimicked run (R’)
MIMICKING FIELD FAILURESUser run (R) Relevant events
(breadcrumbs)Mimicked run (R’)
Twitter crashes when I: 1) Open the app 2) Select a tweet with an image 3) Retweet the selected tweet
MIMICKING FIELD FAILURESUser run (R) Relevant events
(breadcrumbs)Mimicked run (R’)
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
OVERALL VISION
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Reviews Classifier
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
App Test Generator
App Analyzer
Code Analyzer
GUI Analyzer
Bug Reports
Test Steps
App StoreReviews
App Code
App GUI
Domain Knowledge Actions
DictionaryTest Cases
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Reviews Classifier
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
App Test Generator
App Analyzer
Code Analyzer
GUI Analyzer
Bug Reports
Test Steps
App StoreReviews
App Code
App GUI
Domain Knowledge Actions
DictionaryTest Cases
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Reviews Classifier
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Reports
App StoreReviews
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
Bug Reports
Test Steps
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
Bug Reports
Test Steps
IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
Bug Reports
Test Steps
IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
Bug Reports
Test Steps
IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
Bug Reports
Test Steps
IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
IT FU*KING CRASHES!!! The app stillcrashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
Bug Report Miner
Bug Reports
Test Steps
IT FU*KING CRASHES!!! The app still crashes every single time I retweet atweet that contains an image.This is fu*king annoying. Fix it already!
BEEP
BEEP
1) Open twitter2) Select tweet with image3) Retweet➡ Crash
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
reply
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
reply
retweet
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
reply
retweet
like
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
reply
notifications
retweet
like
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
reply
notifications
retweet
like
image
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
reply
notifications
retweet
like
image
<action1> <parameter*> <trigger><action2> <parameter*> <trigger>…
App Analyzer
Code Analyzer
GUI AnalyzerTest Steps
App Code
App GUI
Domain Knowledge Actions
Dictionary
reply
notifications
retweet
like
image
<action1> <parameter*> <trigger><action2> <parameter*> <trigger>…
<open twitter> <> <start com.twittwer. app.twitter><retweet> <current tweet> <click >…
App Test Generator
Test Steps
ActionsDictionary
Test Cases
App Test Generator
Test Steps
ActionsDictionary
Test Cases
1) Open twitter2) Select tweet with image3) Retweet➡ Crash
App Test Generator
Test Steps
ActionsDictionary
Test Cases
1) Open twitter2) Select tweet with image3) Retweet➡ Crash
<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…
App Test Generator
Test Steps
ActionsDictionary
Test Cases
1) Open twitter2) Select tweet with image3) Retweet➡ Crash
<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…
App Test Generator
Test Steps
ActionsDictionary
Test Cases
1) Open twitter2) Select tweet with image3) Retweet➡ Crash
<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…
App Test Generator
Test Steps
ActionsDictionary
Test Cases
1) Open twitter2) Select tweet with image3) Retweet➡ Crash
<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…
App Test Generator
Test Steps
ActionsDictionary
Test Cases
1) Open twitter2) Select tweet with image3) Retweet➡ Crash
<open twitter> <> <start com…twitter><retweet> <current tweet> <click >…
CURRENT AND FUTURE WORK
CURRENT AND FUTURE WORKCURRENT AND FUTURE WORK
CURRENT AND FUTURE WORKCURRENT AND FUTURE WORK
• Review Classifier : reuse• Bug Report Miner: very preliminary version• App Analyzer : previous work on feature identification and
cross-platform issues• App Test Generator: previous work on Android testing
• Is it feasible? What are the metrics of success?• What technologies are best suited for the tasks involved? Can
we use them in a black-box fashion?• Can we leverage multiple similar reviews/bug reports?• Can we do this interactively (as the users enter their reviews)?
OPEN QUESTIONS/CHALLENGES