domain-specific languages in automated testing tools mickey phoenix senior software engineer...
TRANSCRIPT
![Page 1: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/1.jpg)
Domain-Specific Languagesin Automated Testing Tools
Mickey PhoenixSenior Software Engineer
Solutions IQSeattle, Washington
Rand HusoSenior Software Engineer
Solutions IQSeattle, Washington
Thursday March 6, 20088:30 - 10:00 am
Presenters:
![Page 2: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/2.jpg)
What is a Domain-Specific Language?
• Concise and Expressive• Specific • Consistent in Granularity
![Page 3: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/3.jpg)
Concise and Expressive• Written in the user's language
• Relative to the level the user is thinking at
• Thorough coverage at that level
![Page 4: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/4.jpg)
Specific
• Domain-specific; often, project-specific
• Portability and re-usability are not goals
• Customized for each client, project, and domain
![Page 5: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/5.jpg)
Consistent Granularity
• Assists logical decomposition• Avoids getting lost in the details• Matches the way people give instructions to each
other– “How do you brush your teeth?”
![Page 6: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/6.jpg)
. . . Oooops.
![Page 7: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/7.jpg)
Customers think at multiple levels at different times, so a single Domain-Specific Language
will not suffice!
But…
![Page 8: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/8.jpg)
Why do we want multiple levels of DSL?
• Test at multiple levels
• Avoid hiding too much information inside DSL commands
• Compose higher-level DSL commands from lower-level DSL commands
– not from the native commands of the testing tool
• Keep the core testing tool simple at alllevels
![Page 9: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/9.jpg)
Selling Domain-Specific Languages
• Up-front costs to the customer
• Additional effort by the developer
How do we get Customer and Developer Buy-In?
![Page 10: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/10.jpg)
Why Customers Love DSLs
• Customers can participate fully
• Customers can write tests in their own language (no lossy translations)
• Customers can validate tests just by reading them -- “executable documentation”
![Page 11: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/11.jpg)
Customers Are Full Participants• Customers pair with developers to write
tests• Customers understand developer-written
tests just by reading them
![Page 12: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/12.jpg)
Tests in Customer Language
Translations are lossy:
We want as few translations aspossible between the customer's"language of thought" and the test.
“The vodka is good, but the meat is spoiled.”
“The spirit is willing, but the flesh is weak.”
![Page 13: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/13.jpg)
Tests As Executable Documentation
• The goal of Agile testing is "executable documentation“
• A test written in a DSL is like self-documenting code
– It requires minimal comments, because the test is written in a language the users can read
![Page 14: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/14.jpg)
Why Developers Love DSLs
• Easier and faster to write at appropriate granularity
• Abstraction reduces test maintenance cost
• Keeps the core testing tool simple
![Page 15: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/15.jpg)
Appropriate GranularityMakes Tests:
• Faster to Write – fewer, bigger statements
• Easier to Read – comprehension by “chunking”
• Easier to Verify – don’t get lost in the details
• Easier to Maintain – all of the above
Think OOD
![Page 16: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/16.jpg)
AbstractionReduces Maintenance Cost
• Tests must change with changes to the spec
• Abstracting out common actions keeps changes to a single point
– “once and only once”
![Page 17: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/17.jpg)
Core Testing Tool Stays Simple
• Every client and project will have some unique needs
• Easy per-client and per-project extensibility keeps changes local to those who need them
Extensibility minimizes feature creepin the core tool
![Page 18: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/18.jpg)
What Do We Expect of an Automated Testing Tool?
• Presupposed: a testing tool which has a written "script" – as opposed to a keystroke/click/etc. recorder
• Provides extension mechanisms – allows us to define new statements in the DSL(s) that can
then be used in the test scripts
• Provides hierarchical abstraction mechanisms – the ability to define test script "subroutines" which
can be invoked by name
![Page 19: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/19.jpg)
Extension Mechanisms
• Used by the developer– Need not be accessible to the customer
• Allow multiple implementation languages where practical (plugins, etc.)
• Support several granularities of DSL
• Can be customized and overridden at the client and project levels independently
![Page 20: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/20.jpg)
Hierarchical Abstraction
WALK:Step forward with right foot
Step forward with left foot
STEP FORWARD:Shift weight to opposing foot
Bend knee
Swing foot forward
Set foot down in advance of body’scenter of mass
![Page 21: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/21.jpg)
DSL Examples
• A low-level test written without DSL• The same low-level test written in a DSL• The definition of the low-level DSL• A mid-level test written in a low-level DSL• The same mid-level test written in a mid-level
DSL
We will review:
![Page 22: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/22.jpg)
Low-Level Test (no DSL)
falsedocument.getElementById("field_one_mandatory_flag").style.visibility
assertJavascriptValue
truedocument.getElementById("field_one").style.editable
assertJavascriptValue
truedocument.getElementById("field_one").style.visibility
assertJavascriptValue
Assert that field_one is “enabled" (visible and editable, but not mandatory)
comment
![Page 23: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/23.jpg)
Low-Level Test (with DSL)
field_oneassertEnabled
![Page 24: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/24.jpg)
Custom DSL Definition(Low-Level)
Selenium.doAssertEnabled = new function(fieldName){ var field = document.getElementById(fieldName); var mandatoryFlag = document.getElementById(
fieldName + "_mandatory_flag"); if (!field.style.visibility){
throw new Error("Field is not visible."); } if (!field.style.editable) {
throw new Error("Field is not editable."); } if (mandatoryFlag.style.visibility){
throw new Error("Field is mandatory."); }}
![Page 25: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/25.jpg)
[no body text for thisentry]
assertTextPresent
link=This is a test entry subject
clickAndWait
link=This is a test entry subject
assertElementPresent
This is a test entry subjectassertTextPresent
Link=List EntriesclickAndWait
Mid-Level Test (Low-Level DSL)
This is a test entry subjecttype
link=Create New EntryclickAndWait
button=LoginclickAndWait
passwordtestPasswdtype
usernamebsmithtype
button=Create EntryclickAndWait
![Page 26: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/26.jpg)
Mid-Level Test (Mid-Level DSL)
!include LoginStandardUser
!include CreateEntry !include VerifyEmptyEntry
entrySubjectThis is a test entry subject
store
![Page 27: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/27.jpg)
Example Extensibility Framework• STIQ – StoryTestIQ
– Free open-source project at SourceForge (http://storytestiq.sourceforge.net/)
• Extensibility mechanisms evolved along with the tool, in response to need
• Framework-level and client-level extensibility contributed the most to power
• Project-level and hierarchicalextensibility contributed the most to readability
![Page 28: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/28.jpg)
Framework-Level Extensibility
• Java code changes on the server (e.g. SQL support)
– Requires re-compilation of the local (STIQ) project
– Written in Java (may not be as familiar to developers in a .NET shop)
![Page 29: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/29.jpg)
Client-Specific Extensibility
• client-extensions.js Allows corporate-level testing support people to add features that are useful for all projects
– Requires re-compilation of the local (STIQ) project
– Written in Javascript (familiar to all web developers)
![Page 30: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/30.jpg)
Project-Specific Extensibility
• project-extensions.jsAllows individual projects to have added features (in the browser) that will be useful only for their project– Requires reloading the browser page, but
does not require re-compiling– Written in Javascript (familiar to all web
developers)
![Page 31: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/31.jpg)
Hierarchical Extensibility
• !include syntax
allows hierarchical decomposition of tests, including tests written in low-level DSL terms defined in: – client-extensions.js source– project-extensions.js source
![Page 32: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/32.jpg)
Conclusion
• Providing DSL support in your automated testing tool has many advantages – to you– to the project developers and testers– to the user
• ...?
• Profit!
![Page 33: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/33.jpg)
Questions?
![Page 34: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/34.jpg)
Small-Group Discussions
![Page 35: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/35.jpg)
Discussion Questions
• One implementation language or many implementation languages? – You can write Java code on the server, or
Javascript code or SQL on the client, to augment the tests (in STIQ). Should a testing tool provide customization support in multiple implementation languages?
![Page 36: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/36.jpg)
Discussion Questions
• Are some implementation languages more suited to certain levels of granularity than others? – Will some levels of DSL be easier to
implement/customize in low-level vs. high-level languages? Which languages are most suitable to implement which levels of DSL, and why?
![Page 37: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/37.jpg)
Discussion Questions
• Should a customizable testing tool define its own, custom implementation language, or should it use a standard language like Java or Javascript?
![Page 38: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/38.jpg)
Discussion Questions
• Which agents should be able to customize? Testers? Developers? Only the team tasked with supporting the testing tool?
![Page 39: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/39.jpg)
Discussion Questions
• How do DSLs affect customer transparency? – A single-line DSL statement makes the test more
readable, but also obscures the actions taking place within that single-line command, which may potentially be quite complex. Doesn't this just bring us back to the same old problem of the customer having to "take it on faith" that the test asserts the behavior they desire?
![Page 40: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/40.jpg)
Discussion Time
![Page 41: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/41.jpg)
Working Group Presentations
![Page 42: Domain-Specific Languages in Automated Testing Tools Mickey Phoenix Senior Software Engineer Solutions IQ Seattle, Washington mphoenix@solutionsiq.com](https://reader035.vdocuments.mx/reader035/viewer/2022070418/56649f065503460f94c1b21a/html5/thumbnails/42.jpg)
Authors:
Michele CoxRand HusoMichael Phoenix