reliable acceptance testing

43
Dagfinn Reiersøl, ABC Startsiden 1 Reliable acceptance testing

Upload: dagfinnr

Post on 11-May-2015

1.383 views

Category:

Technology


1 download

TRANSCRIPT

Page 1: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 1

Reliable acceptance testing

Page 2: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 2

Manual testing is immoral

• Not what humans are good at

• Time consuming = expensive

• High stress

• Error-prone

• Exception: exploratory testing

Page 3: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 3

Types of automated testing

• Unit tests: clean code

• Acceptance tests: communicate with user / customer

• Other functional tests: ensure that everything works

Page 4: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 4

High-level challenges of acceptance testing

• Getting the Product Owner involved

• Test Doubles

• Not over-testing the user interface

Page 5: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 5

Unit tests

• Keep code clean

• Avoid “hidden” bugs

• Aid design

• Test-driven development

Page 6: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 6

Scrum Product Owner

• Represents customer(s) and other stakeholders

• Owns the product backlog

• Is not a developer

Page 7: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 7

Acceptance tests

• For the Product Owner or possibly testers

• Goal: Make sure we implement the right features

• Ideally editable by a non-programmer

• Tests features, not modules

• Make sure you're done

Page 8: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 8

Other functional tests

• Goal: Make sure the implementation works

• Could be testing the same thing as acceptance tests, but

• need not be business-readable

Page 9: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 9

The Test Double pattern

Page 10: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 10

Surrounded by “Test Doubles”

Page 11: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 11

Web testing: advantages

• You can test exactly what the user sees

• There are tools to test JavaScript-heavy / Ajax apps

• There are tools to record and play back tests

• “It seemed like an good idea at the time”

Page 12: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 12

Web testing: disadvantages

• Fragile: Small UI changes break tests

• Fragile: External services break tests

• Slow

• Harder to replace external services with fakes

Page 13: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 13

Strategies for robustness

• Test Bus

• Test patterns

• Test refactoring

• Making tests unnecessary

Page 14: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 14

Web testing: essentials

• Must be in a programming language (conditionals, variables)

• Language not necessarily PHP, since HTTP allows communication between languages

• “Business-readable” programming language (FitNesse wiki, Gherkin...)

Page 15: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 15

Web testing tips

• Test the HTML output using regex, XPath, CSS selector

• Use element IDs or names to test links, forms and fields

• Log HTTP requests in the application

Page 16: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 16

How to use the request log

Automated test fails but not manual run from browser

• Run the automated test, find the failing request in the log.

• Run the app manually, find the successful request in the log.

• Compare.

Page 17: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 17

Using SimpleTest

<?phprequire_once 'simpletest/autorun.php';require_once 'simpletest/web_tester.php';error_reporting (E_ALL);

class JoomlaSimpleTestTest extends WebTestCase {

function testJoomla() { $this->get('http://localhost/joomla'); $this->assertResponse(200); $this->assertTitle('Welcome to the Frontpage'); $this->assertText('Welcome to the Frontpage'); $this->click('Welcome to Joomla!'); $this->assertPattern('/Joomla! is a free.*content

publishing system/is'); }}

Page 18: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 18

Using PHPUnit with Selenium

<?phprequire_once 'PHPUnit/Extensions/SeleniumTestCase.php';class JoomlaPhpUnitTest extends

PHPUnit_Extensions_SeleniumTestCase { protected function setUp(){ $this->setBrowser('*firefox'); $this->setBrowserUrl('http://localhost/'); } public function testJoomla() { $this->open('http://localhost/joomla/'); $this->assertTitle('Welcome to the Frontpage'); $this->assertTextPresent('Welcome to the Frontpage'); $this ->clickAndWait( '//ul[@class=\'latestnews\']/li[4]/a'); $this ->assertTextPresent('Joomla! is a free open'); }}

Page 19: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 19

SimpleTest vs. PHPUnit / Selenium

Simpletest:

– Faster

– Easier to use

PHPUnit with Selenium

– Heavier but more advanced

– Can test JavaScript-heavy / Ajax apps

– Can test cross-browser compatibility

Page 20: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 20

Don't overdo Selenium

“He ate a lot of brazil nuts which is a big deal because they contain selenium, which in high doses causes fatigue, vomiting, skin irritation, discharge from the fingernail beds and hair loss.”

- House, M.D.

Page 21: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 21

Acceptance test frameworks

• FitNesse: has specific support for PHP

• Cucumber: has PHP-related documentation

• Robot Framework

• Concordion

Page 22: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 22

Using FitNesse with SimpleTest

Page 23: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 23

PhpSlim Fixture for FitNesse test (simplistic!)

<?phprequire_once 'simpletest/browser.php';

class SimpleTestFixture { function __construct() { $this->browser = new SimpleBrowser; }

function goToPage($url) { $this->content = $this->browser->get($url); return !empty($this->content); }

function pattern($pattern) { return preg_match("/$pattern/is",$this->content) > 0; }}

Page 24: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 24

Using Cucumber with Webrat

• joomla.feature (Cucumber Gherkin business-readable)

• webrat_steps (Ruby code testing via HTTP)

Feature: Joomla welcome In order to understand what Joomla is As a potential user I want to be able to read an introduction to Joomla

Scenario: Welcome page Given I am on the home page When I follow "Welcome to Joomla!" Then I should see "Joomla! is a"

Page 25: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 25

Webrat steps for Cucumber

• Ruby code runs Webrat which runs the web app using HTTP

Given /^I am on (.+)$/ do |page_name| visit path_to(page_name)end

When /^I follow "([^\"]*)"$/ do |link| click_link (link)end

Then /^I should see "([^\"]*)"$/ do |text| response_body.should contain(text)end

Page 26: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 26

Avoiding difficult tests

• It's hard to test behaviors that cut across the HTTP request

• Improve architecture so these behaviors can't fail

• Example: duplication between input control name and HTTP variable name

Page 27: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 27

Hard to test: name duplication

<input type="text" name="address1" value="<?= $address1 ?>">

// Oops...

echo $_POST['address_1'];

Page 28: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 28

Getting rid of name duplication

• Use a form handling package

• Or do it yourself. Simplistic version here:

<?php define('ADDRESS1_NAME', 'address1');$address1_value = $_POST[ADDRESS1_NAME];

<html>...<input name="<?= ADDRESS1_NAME ?>" value="<?=

$address1_value ?>">

Page 29: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 29

Refactoring tests

• Refactor or be doomed to drown in duplicate code

• Especially true of web tests

• Most important: eliminate duplication

Page 30: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 30

BDD-style test names

class JoomlaSimpleTestTest extends WebTestCase {

function setUp() { $this->get('http://localhost/joomla'); }

function testFrontPageIsValid() { $this->assertResponse(200); $this->assertTitle('Welcome to the Frontpage'); $this->assertText('Welcome to the Frontpage'); }

function testWelcomePageIsValid() { $this->click('Welcome to Joomla!'); $this->assertPattern('/Joomla! is a free.*content

publishing system/is'); }}

Page 31: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 31

Using custom assertions instead

class JoomlaSimpleTestTest extends WebTestCase { function testJoomla() { $this->get('http://localhost/joomla'); $this->assertValidHomePage(); $this->click('Welcome to Joomla!'); $this->assertValidWelcomePage(); } function assertValidHomePage() { $msg = 'Error on home page'; $this->assertResponse(200,$msg); $this->assertTitle('Welcome to the Frontpage',$msg); $this->assertText('Welcome to the Fontpage',$msg); } function assertValidWelcomePage() { $this->assertPattern( '/Joomla! is a free.*content publishing system/is', 'Error on welcome page'); }

Page 32: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 32

ComposedRegex

http://martinfowler.com/bliki/ComposedRegex.html

• This regex tolerates markup (or anything else) between the content items

• Yes, I know the email regex is simplistic

$email = '.*\w+@\w+\.\w+.*'; $firstname = '.*\w.*'; $lastname = '.*\w.*'; $date = '.*20\d\d-\d\d-\d\d( |&nbsp;|T)\d\d:\d\d.*'; $regex = 'Email address:'. $email . 'Name: ' . $firstname . $lastname . 'Date:' . $date;

Page 33: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 33

Re-using setup

• Refactor gradually as needed:

– When test class gets big, split it and...

– ...extract setup method into a separate class

– When setup is common to many tests...

– ...extract setup class into a separate file

class JoomlaTestSetup { function setUp() { $this->get('http://localhost/joomla'); }}

class JoomlaSimpleTestTest extends JoomlaTestSetup..

Page 34: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 34

Useful types of classes in web testing

• Scraper

• Domain Object

• Specification

Page 35: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 35

Domain object test version

class UserTest extends WebTestCase { function setUp() { $this->user = new TestUser( 'Jane','Doe','[email protected]'); }

function testCanRegisterUser... $this->setFieldByName( 'firstname', $this->user->firstname );

Page 36: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 36

Given-when-then in PHP

class JoomlaSimpleTestTest extends WebTestCase { function setUp() { $this->pages = array( 'JoomlaHomePage' => 'http://localhost/joomla'); } function testWelcomPageIsValid() { $this->givenIAmOn('JoomlaHomePage'); $this->whenIFollow('Welcome to Joomla!'); $this->thenIShouldSee('Joomla! is a free'); } function givenIAmOn($name) { $this->get($this->pages[$name]); } function whenIFollow($text) { $this->clickLink($text); } function thenIShouldSee($pattern) { $this->assertPattern("/$pattern/is"); }

Page 37: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 37

Test bus

http://www.martinfowler.com/ieeeSoftware/testBus.pdf

Page 38: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 38

Recommendations from Uncle Bob

• Test the features through the test bus

• Test the user interface separately in isolation

• Run a few end-to-end tests to test the “plumbing”

Page 39: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 39

Test bus

Where is the test bus? Some possbilities:

• Test the Model

• Test the Controller

• Test the server side of an Ajax or “thin server” application

• Test a public API

Page 40: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 40

Test bus with presentation tests

Page 41: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 41

Fake web service

• Easier to do when not testing via HTTP

• Sensing/verification is very hard to do

Page 42: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 42

Self-initializing fake

http://martinfowler.com/bliki/SelfInitializingFake.html

Page 43: Reliable acceptance testing

Dagfinn Reiersøl, ABC Startsiden 43

Self-initializing fake

http://martinfowler.com/bliki/SelfInitializingFake.html