automated acceptance test practices and pitfalls
DESCRIPTION
Key impediments and pitfalls people run into when trying to use AATs, and how to get over themTRANSCRIPT
AUTOMATED ACCEPTANCE TESTINGGOOD PRACTICES AND PITFALLSWyn Van [email protected]@wynv
Agenda
• Part I: Overview• Part II: Pros
• Why even talking about them
• Part III: Pitfalls & Solutions • Impediments
• Resistance/Bias/Value• Bandwidth/Velocity• Learning curve/Experience• Getting specs
• Pitfalls • Speed• Maintainability • Poor adoption
Part I
AAT Overview
**Legacy Nightmare <div class="pdl13 pdr15"> <div class="fll w744"> <div class="wp100 bdr02 bgc03"> <div class="pd08 mostDownload01_l vat"> <div class="bgc02 vat"> <div class="mostDownload_m bgc02 vat"> <div class="wp100"> <div class="fll bgc05 pdr03 h20">
Automated Acceptance Tests (AATs)
Given – When – Then
Given I have no foodWhen I buy 2 pickles on AmazonThen I have some food
End-to-end UI, Integration tests
Acceptance
Component
Unit
Black box
White box
End-to-end Business facing
Localized Technology facing
Manual Checking
End-to-end
UI
Component
Unit
SOURCE: http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid
Manual Checking
Inverting the Testing Pyramid
UI
Acceptance, Integration
Unit
-Exploratory testing
Acceptance Test Strategy
• Happy paths• Major unhappy paths• Legacy• Regression
Who writes acceptance tests?
What can you write them in?Vim
Sublime Text
Part II
How they work
Start with a storyUser story As an internet user,I want to search for “ALT.NET”And get valid results
• Acceptance criteria 1: I get at least 100k results• Acceptance criteria 2: Results ordered by relevance …
Scenarios
Feature
Tracking AATs
Tracking AATs
Various Frameworks
Browser driver
Browser automation
Test runner/harness
BDD Framework
.NET
Selenium, Phantom.js
Selenium, Waitn
NUnit, xUnit, MSTest
SpecFlow,Fitness (.NET runner), Cuke4Nuke, MSpec
JavaScript
Selemium, Phantom.js
Phantom.js/Casper.js
Mocha, Jasmine
Chai, Jasmine
Java/Ruby
Selenium, Phantom.js
Selenium, Waitr
JUnit, test-unit
JBehave, Fitness, Cucumber/RSpec/Capybara
Why Have AATs? (Pros)
Communication• Helps specify behavior of the system in plain text• Provides a medium for non-tech and devs to agree on
Are we talking about the right system?
Seams, unit test mistakes
Automation
“There’s no place for human beings to be doing regression testing manually.”
-Jez Humble
Speedier deployments
Save resources
More testing during development
Why NOT Have AATs?
Why don’t more people use them?
Impediments • Poor adoption • Bandwidth/Velocity• Learning Curve/Experience• Business users won’t write specifications
Why NOT have AATs?• Maintenance • Speed• High false negatives, non-determinism
Questions?
Part IV
Overcoming the Impediments
Minimize # of end-to-end tests• AATs should be a small portion of your test suite• Is your new feature entirely new?• Balance high # of unit tests + selected end-to-end & acceptance
Concise Specs• Declarative• Reuse steps • Concise, many methods, fat fixtures
Bad:• Click on Log In button• Click username box and type ‘myUsername’ and click ‘password’ box and type
‘myPassword’• Click on link for Transfer Payment• Click box for amount and type 400• Click ‘Transfer’ button• Assert success message
Good:• Login as ‘MyAccountName’• Navigate to Transfer Payment page• Transfer 400 dollars • Assert success message
Use a headless browser
Don’t go through the UI• Controller down, service down• Use mocks, stubs
When Acceptances Tests catches a bug
• See why bug got through unit/integration tests• Add unit, integration tests • Prune AAT?
Unit
Organize your tests…Use Page Objects
Keeps each page’s tests • Encapsulated • Readable • Reusable
public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page}
Domain
Service
UI
GoogleHomepage
+search(query)+clickSearch()+clickGettingLucky Data Access
GoogleHomepageService
+search(query)+clickSearch()+clickGettingLucky
IGoogleHomepage
+search(query)+clickSearch()+clickGettingLucky
Page Objects
Page ObjectsFaster test creation – like Legos
• Keeps step definitions very readable and thin
Hides details• Abstracts SpecFlow from Selenium• Keeps infrastructure at a lower level
Keeps changes minimal• Changes in the application should require minimal changes in the tests, and not in the specs
Easier to write new tests• Constructing page objects • Can be used to create a DSL
One way[When(@"I log in with the teacher account '(.*)'")]public void WhenILogInWithTheTeacherAccount(string orgCode){
var driver = new FirefixDriver();
driver.Navigate().GoToUrl("http://www.google.com");
var userTb = Driver.FindElement(By.Name("username”));var passwordTb = Driver.FindElement(By.Id("password"));userTb.SendKeys(username);passwordTb.SendKeys(password);
var loggedInLink = Driver.FindElement(By.LinkText("Log Out"));Assert.That(loggedInLink != null);
}
Better way – page objects public class LogInPage : PageBase { public PageBase LogIn(string username, string password) { Driver.FindElement(By.Id("UserName")).SendKeys(username); Driver.FindElement(By.Id("Password")).SendKeys(password); Driver.FindElement(By.CssSelector("input[type=\"submit\"]")).Click();
return GetInstance<AccountLandingPage>(Driver); }
public void HasExpectedTab(string tabName) { var link = Driver.FindElement(By.LinkText(tabName)); Assert.AreEqual(tabName, link.Text); } }
Other things • Managed test data• Distributed execution
Tips
Developer tips• Don't run in same assembly as other tests, so these can be run separately since they're very slow (i.e. nightly)
• Navigation
• Create different levels of suites depending on depth/level of feedback desired:• Smoke, Current iteration/sprint, Regression
• Run AATs as close to the real environment as possible
GherkinSpecs shouldn’t have much setup code
Given I am on the homepageAnd I click loginAnd I enter username of ‘username’ and password of ‘password’When clicking ‘Log In’Then take me to my landing page
Given I am on the homepageWhen logging in with ‘username’ and ‘password’Then take me to my landing page
UI tests• Sometimes tests will fail if the page doesn’t have enough time to load
• Capture screen shots when tests fail• Run nightly• Run across multiple machines
Anti-patterns• Developers writing acceptance tests by themselves, for themselves
• Tying AATs to an implementation - unit tests are implementation specific, AATs are NOT
• Too much repetition in the gherkin
• Too many UI tests
• UI tests are failing too easily when changing the UI
Conclusion
Where we can be• Reliable tests• Reusable test code (even a DSL)• Separation of concerns• Short and clear tests• Fast test creation • Quick regression feedback• Tests that are understood by non-technical people• Fewer bugs
Resources:Tools used here:• SpecFlow – specflow.org• PhantomJS – phantomjs.org• Selenium WebDriver – seleniumhq.org
Books:• Specification By Example, Gojko Adzic• Continuous Delivery, Jez Humble, David Farley• Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce
Articles:• Automated Acceptance Tests, http://www.thoughtworks.com/insights/articles/automated-acceptance-tests• BDD with SpecFlow and ASP.NET MVC, http
://blog.stevensanderson.com/2010/03/03/behavior-driven-development-bdd-with-specflow-and-aspnet-mvc/• Using Gherkin Language in SpecFlow,
https://github.com/techtalk/SpecFlow/wiki/Using-Gherkin-Language-in- SpecFlow
• BDD with SpecFlow, MSDN, http://msdn.microsoft.com/en-us/magazine/gg490346.aspx• Using SpecFlow with the Page Object,
http://blogs.lessthandot.com/index.php/EnterpriseDev/application-lifecycle-management/using-specflow-to
• Problems with Acceptance Tests, http://xprogramming.com/xpmag/problems-with-acceptance-testing• Top 10 reasons why teams fail with Acceptance Testing,
http://gojko.net/2009/09/24/top-10-reasons-why-teams-fail-with-acceptance-testing/• Maintaining Automated Acceptance Tests (ThoughtWorks),
http://www.youtube.com/watch?v=uf0EVbH5hdA• Creating Maintainable Automated Acceptance Test Suites, Jez Humble, • http://www.youtube.com/watch?v=v-L_2y6g5DI
Happy Three Amigos
Slides: http://www.slideshare.net/wynvandevanter/automated-acceptance-tests-in-net
Code:https://github.com/Wyntuition/AATsInSpecFlowDemos
[email protected]@wynv