![Page 1: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/1.jpg)
Help, It looks Confusing:GUI Task Automation Through Demonstration and Follow-up Questions
Thanapong Intharah,
Daniyar Turmukhambetov
Gabriel J. Brostow
University College London
![Page 2: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/2.jpg)
People Who Want To …
Target Users:
![Page 3: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/3.jpg)
Target Users: People Who Want To …
(Linear) Have a macro/script where one click does
many actions.
Example: switch everything to high contrast mode.
![Page 4: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/4.jpg)
Target Users: People Who Want To …
(Linear) Have a macro/script where one click does
many actions.
Example: switch everything to high contrast mode.
(Loop) Do iterative tasks, automatically.
Example: make each photo in this folder into a
separate slide in PowerPoint.
![Page 5: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/5.jpg)
Target Users: People Who Want To …
(Linear) Have a macro/script where one click does
many actions.
Example: switch everything to high contrast mode.
(Loop) Do iterative tasks, automatically.
Example: make each photo in this folder into a
separate slide in PowerPoint.
(Monitoring) React the same way each time …
Example: click the “skip ad” during a youtube video.
![Page 6: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/6.jpg)
Before HILC
![Page 7: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/7.jpg)
• Nearest neighbors : Programming by Demonstration
– Sikuli, Sikuli Slides (Yeh et al 2009).
– Koala (Little et al 2007), CoScripter (Leshed et al 2008).
– More systems can be found in “Watch what I do” (Cypher
and Halbert 1993).
Before HILC
![Page 8: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/8.jpg)
Feature Comparison
Domain
Independent
No Programming
Skill Required
Deal with
Non-linear Tasks
Sikuli
Sikuli Slides
Koala, CoScriptor
Sheepdog,
Familiar
HILC
![Page 9: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/9.jpg)
Feature Comparison
Domain
Independent
No Programming
Skill Required
Deal with
Non-linear Tasks
Sikuli
Sikuli Slides
Koala, CoScriptor
Sheepdog,
Familiar
HILC
![Page 10: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/10.jpg)
Feature Comparison
Domain
Independent
No Programming
Skill Required
Deal with
Non-linear Tasks
Sikuli
Sikuli Slides
Koala, CoScriptor
Sheepdog,
Familiar
HILC
![Page 11: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/11.jpg)
Feature Comparison
Domain
Independent
No Programming
Skill Required
Deal with
Non-linear Tasks
Sikuli
Sikuli Slides
Koala, CoScriptor
Sheepdog,
Familiar
HILC
![Page 12: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/12.jpg)
Feature Comparison
Domain
Independent
No Programming
Skill Required
Deal with
Non-linear Tasks
Sikuli
Sikuli Slides
Koala, CoScriptor
Sheepdog,
Familiar
HILC
![Page 13: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/13.jpg)
Feature Comparison
Domain
Independent
No Programming
Skill Required
Deal with
Non-linear Tasks
Sikuli
Sikuli Slides
Koala, CoScriptor
Sheepdog,
Familiar
HILC
![Page 14: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/14.jpg)
• Nearest neighbours : Programming by Demonstration
– Sikuli, Sikuli Slides (Yeh et al 2009).
– Koala (Little et al 2007), CoScripter (Leshed et al 2008).
– More systems can be found in “Watch what I do” (Cypher
and Halbert 1993).
• Differences :
– HILC is software-agnostic;
no need for Accessibility API’s and works across applications.
- Inputs users feed to HILC resemble how a human
teaches another to complete a task.
- Able to deal with non-linear task.
Before HILC
![Page 15: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/15.jpg)
Phase A. HILC observes a user1 demonstrating a task.
Phase B. HILC asks a user2 to clarify confusing steps, if any,
and then synthesizes a task script.
Phase C. A user3 triggers the script to complete the task.
From User’s Perspective
![Page 16: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/16.jpg)
Phase A. HILC observes a user1 demonstrating a task.From User’s Perspective
![Page 17: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/17.jpg)
Phase B. HILC asks a user2 to clarify confusing steps.From User’s Perspective
![Page 18: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/18.jpg)
Phase C. A user3 triggers the script to complete the task.From User’s Perspective
![Page 19: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/19.jpg)
Linear Looping Monitoring
A: Demonstration
B: Teaching
C: Running
From development perspective
Normal +Signal +Signal
Supporters
Positives
False positives
Supporters
Visual Cues
Execute Execute Execute
Task Classification
Phase
s
![Page 20: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/20.jpg)
Phase A : Demonstration Phase
• Sniffer
• Video
Log-file
Sequence of basic actions
![Page 21: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/21.jpg)
HILC gets follow-up input from user (a.k.a. Teacher):
• more positive examples
• remove false positive examples
• point out the visual cues
• add supporters
Phase B : Teaching phase
![Page 22: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/22.jpg)
Phase B : Teaching phase
HILC allows users to add more positive examples, remove
false positive examples, point out the visual and add
supporters
Supporters
![Page 23: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/23.jpg)
When and What to ask?
Teaching phase – How did we do it?
(Algorithm)
![Page 24: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/24.jpg)
When and What to ask? (Linear tasks)
(Algorithm)
![Page 25: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/25.jpg)
When and What to ask? (Looping task)
(Algorithm)
![Page 26: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/26.jpg)
When and What to ask? (Monitoring task)
(Algorithm)
![Page 27: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/27.jpg)
Phase C : Running phase
![Page 28: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/28.jpg)
Scenarios and experiments
![Page 29: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/29.jpg)
• HILC is not aware of off-screen objects.
because it only looks at a screenshot image.
• Scrolling is currently not supported.
• HILC is not aware of system state, hence it only
relies on a fixed time delay between basic
actions.
• HILC relies on having a sniffer, but we have
preliminary results from just videos of the GUI.
Limitations
What can you do with this project?.
![Page 30: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/30.jpg)
• Computer Vision.
• 3 Phases.
• Follow-up questions.
Take home messages
![Page 31: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/31.jpg)
Thank you
![Page 32: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/32.jpg)
Q & A
For more information and code please visit http://visual.cs.ucl.ac.uk/pubs/HILC/
![Page 33: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/33.jpg)
Extra slides
![Page 34: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/34.jpg)
![Page 35: Help, It looks Confusingvisual.cs.ucl.ac.uk/pubs/HILC/IUI17_present_finallized.pdf · – Sikuli, Sikuli Slides (Yeh et al 2009). – Koala (Little et al 2007), CoScripter (Leshed](https://reader033.vdocuments.mx/reader033/viewer/2022043022/5f3e06b412ac1e6a39588302/html5/thumbnails/35.jpg)
• The log file unifies inputs from both video and
sniffer
• Transcribe input log-file
– Viterbi Algorithm
• Unary potential is computed using SVM + Random Forest
• Pairwise potential encourages segments from the same basic
action to follow the order
- - - - - - L L L - - - - L - - - - - - - - - - - - - - - - - - - - - - - R - - - - - - - - - - - - - - -
Basic actions = {Click, Double Click, Click Drag, Right Click}
Key frame
Demonstration Phase – How did we do it?
(Algorithm)
Segment
Time interval
Input to SVM