practical sikuli: using screenshots for gui automation and testing

PRACTICAL SIKULIUSING SCREENSHOTS FOR GUI AUTOMATION AND TESTING

Tsung-Hsiang (Sean) ChangMIT CSAIL

1

Collaborators: Tom Yeh, Rob Miller

WHAT IS SIKULI?2

WHAT IS SIKULI?2

1

2

3

3

Low-level API?Mac’s system design?

4

EMPTY THE TRASH CAN5

OUTLINE

• What Sikuli is

• How Sikuli works

• Sikuli for GUI automation

• Sikuli for GUI testing

• Conclusions

6

How Sikuli Works

7

TEMPLATE MATCHING

click(                            )

8

TEMPLATE MATCHING

click(                            )

8

SYSTEM DESIGN

9

1

2

3

DEMO 1:EMPTY TRASH CAN

10

REGION AND MATCH

11

match = region.find( or “text pattern”)

Region

12

• basic attributes: x, y, width, height

• Match is a Region

• Screen is also a Region

• chained methods: Screen(0).find(W).click(X)

REGION

KEY METHODS OF REGION

• find, findAll, exists, wait, waitVanish

• Mouse Actions

click, doubleClick, rightClick, hover, dragDrop

mouseDown, mouseUp

• Keyboard Actions

type(text), paste(text)

keyDown, keyUp

13

EX2: FACEBOOK STALKER

14

EX2: FACEBOOK STALKER

14

RESTRICT SEARCH REGION

15

But the Region constant is bad

16

APP CLASS

17

• firefox = App.open(“Firefox”)

• firefox.focus(), firefox.close()

• firefox.window([n])

• returns the Region of Firefox’s n-th window

FIND IN CERTAIN APP

18

EX3: MUTE SPEAKERS

19

EX3: MUTE SPEAKERS

19

SPATIAL OPERATORS

20

SPATIAL OPERATORS

20

SPATIAL OPERATORS (2)

21

TEXT MATCHING AND RECOGNITION

22

• click(“OK”), find(“Cancel”)

• region.text() - read the text in a region

VISUAL-EVENT-DRIVEN PROGRAMMING

23

• observes visual events in any Region

• Region.onAppear(Pattern, handler)

• Region.onVanish(Pattern, handler)

• Region.onChange(Pattern, handler)

VISUAL-EVENT-DRIVEN PROGRAMMING

24

VISUAL-EVENT-DRIVEN PROGRAMMING

24

VISUAL DICTIONARY

25

GUITESTING

26

VISUAL ASSERTION

• make sure the appearance of visual feedback

27

VISUAL ASSERTION

• make sure the appearance of visual feedback

27

JUNIT INTEGRATION

28

SIKULI IN COMMAND LINE

29

usage:Sikuli-­‐IDE  [-­‐-­‐args  <arguments>]  [-­‐h]  [-­‐r  <sikuli-­‐file>]  [-­‐s]  [-­‐t  <sikuli-­‐test-­‐case>]  -­‐-­‐args  <arguments>                          specify  the  arguments  passed  to  Jython's  sys.argv  -­‐h,-­‐-­‐help                                            print  this  help  message  -­‐r,-­‐-­‐run  <sikuli-­‐file>                  run  .sikuli  or  .skl  file  -­‐s,-­‐-­‐stderr                                        print  runtime  errors  to  stderr  instead  of  popping  up  a  message  box  -­‐t,-­‐-­‐test  <sikuli-­‐test-­‐case>      run  .sikuli  as  a  unit  test  case  with  junit's  text  UI  runner

CONCLUSIONS

30

PLATFORM INDEPENDENCE

• Works on ANY GUI that can be displayed on Windows/Linux/Mac

• Virtual machines

• Remote desktop

• Mobile simulators: Android, iPhone

• Web: Flash, HTML+Javascript

31

PROGRAM AGAINST UI

32

Sikuli programs are written against the user interface instead of an API

UI: visible, familiar, always exists API: faster, probably more stable

FUTURE WORK

• fast and accurate OCR on screen

• Accessibility API integration

33

Thanks!

http://sikuli.org

New Sikuli X 1.0-rc2 released.

Follow us on twitter @sikuli

Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)

or, just google “sikuli”

34