test automation patterns

nt Session

Presented by:

Dorothy Indepen ultant

Brought to you by:

340 Corporate Way, Suite Orange Park, FL 32073 888‐2

W3 Concurre4/9/2014 10:30 AM

“Test Automation Patterns”

Graham

dent Testing Cons

300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

http://starcanada.techwell.com/�

Dorothy Graham Independent Test Consultant

In software testing for forty years, Dorothy Graham is coauthor of four books—Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation—and is currently working with Seretta Gamba on a new book on test automation patterns. A popular and entertaining speaker at conferences and seminars worldwide, Dot has attended STAR conferences since the first one in 1992. She was a founding member of the ISEB Software Testing Board and a member of the working party that developed the ISTQB Foundation Syllabus. Dot was awarded the European Excellence Award in Software Testing in 1999 and the first ISTQB Excellence Award in 2012. Learn more about Dot at DorothyGraham.co.uk.

http://members.techwell.com/sme-profiles/dorothy-graham�

http://www.dorothygraham.co.uk/

[email protected] [email protected]

© Seretta Gamba and Dorothy Graham 2014 www.DorothyGraham.co.uk

1

Test Automation Patterns

Prepared and presented by

© Seretta Gamba, Dorothy Graham 2014

Dorothy Graham

@DorothyGraham [email protected] www.DorothyGraham.co.uk

2

Contents Background

Issues and examples Test automation Patterns Wiki

Conclusion




3

Origin of patterns in software – 1970s: architect Christopher Alexander describes

patterns for buildings (e.g. light from 2 walls) – 1987: Kent Beck & Ward Cunningham

experimented & presented at OOPSLA – 1994: “Gang of Four” wrote a book applying

pattern principles to software design •  “Design Patterns: elements of reusable object-oriented

software”, Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides

– now widely accepted in software development community

4

What is a pattern? – a generally reusable solution to a commonly

occurring problem within a given context – a description or template for how to solve a

problem that can be used in many different situations

– a pattern is NOT •  a finished solution that can be ‘plugged in’ or

transformed directly into code •  prescriptive •  a set of steps

– a pattern needs to be implemented within context



5

Where did these automation patterns come from?

– Experiences book published 2012 – Seretta Gamba contributed Ch 21

•  Automation through the back door, by supporting manual testing

– Seretta is a developer, used to using patterns

– when she read the other chapters, she thought: “Patterns! Patterns!”

– she wrote up test automation patterns as a book & asked me to join her

– Word didn’t work very well, so we made a wiki

6

Introduction to the wiki

•  the Test Automation Patterns wiki –  ISSUES – describe problems in automation – PATTERNS – proven ways to overcome problems – other things: acknowledgements, references, some

failure patterns •  in this presentation

–  I am using an off-line version of the wiki for demonstration purposes

–  later I will tell you how you can gain access to the “real” wiki



7

Test automation patterns wiki

•  purpose: share information, ideas and experiences about test automation, using issues and patterns

•  mind-maps summarise issues and patterns (not clickable as links)

•  issues and patterns are classified into sub-sections (e.g. management, process, etc.)

•  “diagnostics” help you find your way into the most relevant issues for you (new)

8

Contents Background


Conclusion




9

Issues (problems)

•  what problems affect you in test automation? – what causes the most trouble? – what takes longer than it should? – what is holding you back in your automation? – what are you struggling with? – what would you most like to improve?

10

Test automation Issues: 4 examples •  HIGH MAINTENANCE COST

–  Maintenance of the test automation scripts takes a long time and costs more than it’s worth.

•  NO PREVIOUS TEST AUTOMATION –  Your first time automating, or automation has failed and

you are starting again. •  STALLED AUTOMATION

–  Automation has been tried, but it never “got off the ground” or has now fallen into disuse.

•  UNREALISTIC EXPECTATIONS –  Management has unrealistic expectations about what

test automation can & can’t do.



11

Issue: HIGH MAINTENANCE COST

•  Maintenance of the test automation scripts takes a long time and costs more than it’s worth

•  Questions to ask: – what cost (or time) would be acceptable? – how is the cost measured? – Note: answers may be different in different

contexts

12

Issue: HIGH MAINTENANCE COST 1

•  What are your symptoms? – When software changes, e.g. window sequence is

different, tests fail. More effort to edit the existing tests than to start again. (Capture/replay used)

– Diagnostic question: Are the scripts structured and reusable? •  1) No, scripts are mainly “stand-alone” without much

reuse, automators are “re-inventing” •  2) Yes, structured and reusable scripts, but it needs

automators to add new tests, and we are short of time and/or automators

visit offline wiki



13

Issue: HIGH MAINTENANCE COST 2

•  What are your symptoms?

– No time allowed to maintain the automated tests

– Tests are written in the tool’s language, but the tool has changed, or we have to change tools

14

HIGH MAINTENANCE COST (Tour) - 1

•  Symptom: SUT changes -> large cost to update – MAINTAINABLE TESTWARE – Symptom: Scripts are mainly “stand-alone” without

much reuse, automators are re-inventing •  GOOD PROGRAMMING PRACTICES

– DESIGN FOR REUSE, OBJECT MAP

– Symptom: Structured & reusable scripts, automator / time shortage •  DOMAIN-DRIVEN TESTING

–  ABSTRACTION LEVELS, KEYWORD-DRIVEN TESTING



15

HIGH MAINTENANCE COST (Tour) - 2

•  Symptom: No time to maintain automated tests (we know we should) – MANAGEMENT SUPPORT

•  Explain technical debt, SELL THE BENEFITS

•  Symptom: Scripts in tool language, tool changed or having to change tools – TOOL INDEPENDENCE – ABSTRACTION LEVELS

16

DOMAIN-DRIVEN TESTING

•  Uses a “Domain-specific test language” – keywords from the application domain

•  insurance: New Policy or Make Claim •  mobile: Make Call or Update Contacts

– each keyword is processed by a script that then processes any data for that keyword

–  tests in spreadsheet or databases •  make it easy for business testers to add new tests •  needs technical support (to set up the TESTWARE

ARCHITECTURE and when things go wrong)



17

Testware architecture: abstraction levels abstraction here:

easier to write automated tests: widely used DOMAIN-DRIVEN

TESTING

abstraction here: easier to maintain, and change

tools: long life GOOD PROGRAMMING

PRACTICES, GOOD DEVELOPMENT

PROCESS

testware archite

cture

Testers

Test Execu/on Tool runs scripts

High Level Keywords

Structured Scripts

structured testware

Test Automator(s)

write tests (in DSTL)

18

Some other patterns

•  FAIL GRACEFULLY – make sure a test “cleans up” after itself even if it

fails (so subsequent tests can run without worry) – alternative: FRESH SETUP, where each tests re-

sets its initial state, no matter what happened before, or SHARED SETUP

•  KILL THE ZOMBIES – make sure tests are actually doing something;

delete or archive tests no longer in use



19

Contents Background


Conclusion


20

About these patterns (and the wiki)

•  the wiki has a limited set of patterns – and a limited “depth” for each pattern

•  patterns use other patterns –  it can get rather complex

•  we are focusing on system-level automation rather than unit test automation – See Gerard Meszaros’ book “xUnit Test Patterns:

Refactoring test code” for unit test patterns •  these are test automation patterns, not test

patterns



21

Diagnostics (new)

•  modeled on a doctor’s questions to a new patient – ask questions, and ask more questions, depending on the answers

•  First level – new to automation? Go to NO PREVIOUS TEST

AUTOMATION – completely happy? giving up (good riddance)?

•  share your story – want to improve or revive your automation?

•  see next level

22

Improve or revive automation

•  Most pressing issue: –  lack of support (from mgt, testers, dev) –  lack of resources (staff, s/w, h/w, time) –  lack of direction (what to aut, what architecture) –  lack of specific knowledge (of SUT, tool,

maintainability) – unsatisfactory quality of automation (ROI,

maintenance, unreliable) •  each takes you to more detail in the wiki, and

points you to an issue with resolving patterns



23

Lack of support

•  different kinds of support may be lacking – managers pay “lip service” – don’t see the value – managers have UNREALISTIC EXPECTATIONS –  testers don’t help the automators – developers don’t help the automators – specialists don’t help the automators (DB,

networks) – no one helps the newbies

24

Contents Background


Conclusion




25

Conclusion •  we hope our wiki will help you identify and

understand problems (issues) and give you ideas (patterns) for dealing with them

•  the wiki is free, by invitation – email me at [email protected] for an

invitation –  if you already have a wikispaces account, also give

me your wiki user name – an invitation will be sent from the wiki giving you

instructions for how to set up your user name & password

26

Finally

•  Thank you for coming today •  Any questions? •  Contact us:

–  [email protected] – [email protected]

•  We welcome your feedback •  We welcome your stories & experiences •  All the best with your test automation!

test automation patterns

Technology