automated acceptance test practices and pitfalls

AUTOMATED ACCEPTANCE TESTINGGOOD PRACTICES AND PITFALLSWyn Van Devanterwyn.vandevanter@excella.com@wynv

Agenda

• Part I: Overview• Part II: Pros

• Why even talking about them

• Part III: Pitfalls & Solutions • Impediments

• Resistance/Bias/Value• Bandwidth/Velocity• Learning curve/Experience• Getting specs

• Pitfalls • Speed• Maintainability • Poor adoption

Part I

AAT Overview

**Legacy Nightmare <div class="pdl13 pdr15"> <div class="fll w744"> <div class="wp100 bdr02 bgc03"> <div class="pd08 mostDownload01_l vat"> <div class="bgc02 vat"> <div class="mostDownload_m bgc02 vat"> <div class="wp100"> <div class="fll bgc05 pdr03 h20">

Automated Acceptance Tests (AATs)

Given – When – Then

Given I have no foodWhen I buy 2 pickles on AmazonThen I have some food

End-to-end UI, Integration tests

Acceptance

Component

Unit

Black box

White box

End-to-end Business facing

Localized Technology facing

Manual Checking

End-to-end

UI

Component

Unit

SOURCE: http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid

Manual Checking

Inverting the Testing Pyramid

UI

Acceptance, Integration

Unit

-Exploratory testing

Acceptance Test Strategy

• Happy paths• Major unhappy paths• Legacy• Regression

Who writes acceptance tests?

What can you write them in?Vim

Sublime Text

Part II

How they work

Start with a storyUser story As an internet user,I want to search for “ALT.NET”And get valid results

• Acceptance criteria 1: I get at least 100k results• Acceptance criteria 2: Results ordered by relevance …

Scenarios

Feature

Tracking AATs

Tracking AATs

Various Frameworks

Browser driver

Browser automation

Test runner/harness

BDD Framework

.NET

Selenium, Phantom.js

Selenium, Waitn

NUnit, xUnit, MSTest

SpecFlow,Fitness (.NET runner), Cuke4Nuke, MSpec

JavaScript

Selemium, Phantom.js

Phantom.js/Casper.js

Mocha, Jasmine

Chai, Jasmine

Java/Ruby

Selenium, Phantom.js

Selenium, Waitr

JUnit, test-unit

JBehave, Fitness, Cucumber/RSpec/Capybara

Why Have AATs? (Pros)

Communication• Helps specify behavior of the system in plain text• Provides a medium for non-tech and devs to agree on

Are we talking about the right system?

Seams, unit test mistakes

Automation

“There’s no place for human beings to be doing regression testing manually.”

-Jez Humble

Speedier deployments

Save resources

More testing during development

Why NOT Have AATs?

Why don’t more people use them?

Impediments • Poor adoption • Bandwidth/Velocity• Learning Curve/Experience• Business users won’t write specifications

Why NOT have AATs?• Maintenance • Speed• High false negatives, non-determinism

Questions?

Part IV

Overcoming the Impediments

Minimize # of end-to-end tests• AATs should be a small portion of your test suite• Is your new feature entirely new?• Balance high # of unit tests + selected end-to-end & acceptance

Concise Specs• Declarative• Reuse steps • Concise, many methods, fat fixtures

Bad:• Click on Log In button• Click username box and type ‘myUsername’ and click ‘password’ box and type

‘myPassword’• Click on link for Transfer Payment• Click box for amount and type 400• Click ‘Transfer’ button• Assert success message

Good:• Login as ‘MyAccountName’• Navigate to Transfer Payment page• Transfer 400 dollars • Assert success message

Use a headless browser

Don’t go through the UI• Controller down, service down• Use mocks, stubs

When Acceptances Tests catches a bug

• See why bug got through unit/integration tests• Add unit, integration tests • Prune AAT?

Unit

Organize your tests…Use Page Objects

Keeps each page’s tests • Encapsulated • Readable • Reusable

public class GoogleHomepage: PageBase, IGoogleHomepage {

// Flow for this page}

Domain

Service

UI

GoogleHomepage

+search(query)+clickSearch()+clickGettingLucky Data Access

GoogleHomepageService

+search(query)+clickSearch()+clickGettingLucky

IGoogleHomepage

+search(query)+clickSearch()+clickGettingLucky

Page Objects

Page ObjectsFaster test creation – like Legos

• Keeps step definitions very readable and thin

Hides details• Abstracts SpecFlow from Selenium• Keeps infrastructure at a lower level 

Keeps changes minimal• Changes in the application should require minimal changes in the tests, and not in the specs

Easier to write new tests• Constructing page objects • Can be used to create a DSL

One way[When(@"I log in with the teacher account '(.*)'")]public void WhenILogInWithTheTeacherAccount(string orgCode){

var driver = new FirefixDriver();

driver.Navigate().GoToUrl("http://www.google.com");

var userTb = Driver.FindElement(By.Name("username”));var passwordTb = Driver.FindElement(By.Id("password"));userTb.SendKeys(username);passwordTb.SendKeys(password);

var loggedInLink = Driver.FindElement(By.LinkText("Log Out"));Assert.That(loggedInLink != null);

}

Better way – page objects public class LogInPage : PageBase { public PageBase LogIn(string username, string password) { Driver.FindElement(By.Id("UserName")).SendKeys(username); Driver.FindElement(By.Id("Password")).SendKeys(password); Driver.FindElement(By.CssSelector("input[type=\"submit\"]")).Click();

return GetInstance<AccountLandingPage>(Driver); }

public void HasExpectedTab(string tabName) { var link = Driver.FindElement(By.LinkText(tabName)); Assert.AreEqual(tabName, link.Text); } }

Other things • Managed test data• Distributed execution

Tips

Developer tips• Don't run in same assembly as other tests, so these can be run separately since they're very slow (i.e. nightly)

• Navigation

• Create different levels of suites depending on depth/level of feedback desired:• Smoke, Current iteration/sprint, Regression

• Run AATs as close to the real environment as possible

GherkinSpecs shouldn’t have much setup code

Given I am on the homepageAnd I click loginAnd I enter username of ‘username’ and password of ‘password’When clicking ‘Log In’Then take me to my landing page

Given I am on the homepageWhen logging in with ‘username’ and ‘password’Then take me to my landing page

UI tests• Sometimes tests will fail if the page doesn’t have enough time to load

• Capture screen shots when tests fail• Run nightly• Run across multiple machines

Anti-patterns• Developers writing acceptance tests by themselves, for themselves

• Tying AATs to an implementation - unit tests are implementation specific, AATs are NOT

• Too much repetition in the gherkin

• Too many UI tests

• UI tests are failing too easily when changing the UI

Conclusion

Where we can be• Reliable tests• Reusable test code (even a DSL)• Separation of concerns• Short and clear tests• Fast test creation • Quick regression feedback• Tests that are understood by non-technical people• Fewer bugs

Resources:Tools used here:• SpecFlow – specflow.org• PhantomJS – phantomjs.org• Selenium WebDriver – seleniumhq.org

Books:• Specification By Example, Gojko Adzic• Continuous Delivery, Jez Humble, David Farley• Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce

Articles:• Automated Acceptance Tests, http://www.thoughtworks.com/insights/articles/automated-acceptance-tests• BDD with SpecFlow and ASP.NET MVC, http

://blog.stevensanderson.com/2010/03/03/behavior-driven-development-bdd-with-specflow-and-aspnet-mvc/• Using Gherkin Language in SpecFlow,

https://github.com/techtalk/SpecFlow/wiki/Using-Gherkin-Language-in- SpecFlow

• BDD with SpecFlow, MSDN, http://msdn.microsoft.com/en-us/magazine/gg490346.aspx• Using SpecFlow with the Page Object,

http://blogs.lessthandot.com/index.php/EnterpriseDev/application-lifecycle-management/using-specflow-to

• Problems with Acceptance Tests, http://xprogramming.com/xpmag/problems-with-acceptance-testing• Top 10 reasons why teams fail with Acceptance Testing,

http://gojko.net/2009/09/24/top-10-reasons-why-teams-fail-with-acceptance-testing/• Maintaining Automated Acceptance Tests (ThoughtWorks),

http://www.youtube.com/watch?v=uf0EVbH5hdA• Creating Maintainable Automated Acceptance Test Suites, Jez Humble, • http://www.youtube.com/watch?v=v-L_2y6g5DI

Happy Three Amigos

Slides: http://www.slideshare.net/wynvandevanter/automated-acceptance-tests-in-net

Code:https://github.com/Wyntuition/AATsInSpecFlowDemos

wyn.vandevanter@excella.com@wynv