State-of-the-Art in Automated GraphicalUser Interface Testing

Zebao Gao, Gregory Kramida, Andrew Ruef, and Weiwei YangUniversity of Maryland, College Park

People rely on and use software for manytasks, ranging from launching spacecraft,writing reports, and communicating with

other people. But, because of bugs, people arenot always happy with software. Sometimes,software does not do what it should and this canmake users unhappy or sometimes have veryserious effects on the world. Software shouldbe a tool that people can rely on, but often itis not.

The (in)famous blue screen of deathcame to symbolize software errors.

Traditional Software Testing

So how can we ensure reliability of software?For testing software at a low level, there areadvanced techniques and tools for software de-velopers to use. However, these tools operateon the software at a much lower level than usersof the software. Users typically interact withsoftware via a Graphical User Interface (GUI),

and errors there cannot easily be detected bythe bug-finding software currently in use by in-dustry or prepared by the research community.

The traditional industry approach is to per-form manual testing on graphical applications,which is expensive, time consuming, and oftenunreliable. This increases the cost of softwareand the time to market. A study conducted byNIST in 2002 reports that software bugs costthe U.S. economy $59.5 billion annually. Morethan a third of this cost could be avoided ifbetter software testing was performed.

Enter GUITAR

How can we improve this? Many test-ing frameworks either require humantesters to manually record/script work-

flows (e.g., Selenium1, HP UFT2) or randomlytrigger controls in the application (e.g. Mon-key3). GUITAR4 (GUI Testing FrAmewoRk),designed and developed by Dr. Memon andhis research team at University of Maryland,is more systematic and efficient: GUITAR au-tomatically creates an Event Flow Graph, orEFG, to represent a user’s interaction with anapplication.

The graph represents the connections be-tween possible events in the GUI, for exampleclicking “File” and then “Open”. The EFGcan then be converted directly and automat-ically into test cases. And more importantly,the entire process is fully automated.

1http://www.seleniumhq.org/2http://www8.hp.com/us/en/software-solutions/unified-functional-automated-testing3http://developer.android.com/tools/help/monkey.htm4http://sourceforge.net/projects/guitar/

February 2015 Page 1 of 3

This bar-chart compares the OME* technique with and without context-aware mapping(green and blue bars, respectively) to baseline GUITAR and random path testing.

The OME* paradigm

This is not the end of the story. Soft-ware continues to grow in complexity,including interactive complexity. The

automatic construction of the EFG is impor-tant to the scalability of the testing technique,but the size and complexity of some applica-tions make this difficult. Until recently, it wasimpossible for GUITAR to build a precise andcomplete EFG in one run of programs withcomplex GUIs. Some software is so complexthat no state-of-the-art planning techniques areable to depict a full picture of their EFG due

to lack of domain knowledge.

Researchers in Dr. Memons lab workingon GUITAR have defined a new technique,Observe-Model-Exercise* (OME*). They no-ticed that one major limitation in GUITAR islack of context information during executiontime. By keeping a context-aware mappingtable, this problem is greatly alleviated. In ad-dition, instead of conceptually separating theEFG modeling and test case execution proce-dures, they introduce a ciclic workflow whichobserves new widgets and events during execu-tion of test cases.

The event space for a budgeting program Buddi grows from the small model obtained byGUITAR without OME* (a), to an expansive network after five iterations with OME*.

February 2015 Page 2 of 3

References

[1] Nguyen, B. N., and Memon, A. M. (2014). An Observe-Model-Exercise* Paradigm toTest Event-Driven Systems with Undetermined Input Spaces. Software Engineering, IEEETransactions on, 40(3), 216-234.

February 2015 Page 3 of 3