continuous quality assurance using selenium webdriver

72
Quality Assurance Continuous …using Selenium WebDriver San Francisco Selenium Meetup Group - May 16, 2013

Upload: aoe

Post on 08-May-2015

7.045 views

Category:

Technology


0 download

DESCRIPTION

San Francisco Selenium Meetup, May 2013

TRANSCRIPT

Page 1: Continuous Quality Assurance using Selenium WebDriver

Quality Assurance Continuous

…using Selenium WebDriver

San Francisco Selenium Meetup Group - May 16, 2013

Page 2: Continuous Quality Assurance using Selenium WebDriver

Fabrizio Branca Lead System Developer at

Page 3: Continuous Quality Assurance using Selenium WebDriver

San Francisco, CA

Janine

Fiona

that’s me

It’s a boy!

Page 4: Continuous Quality Assurance using Selenium WebDriver

E-Commerce: Magento

CMS: TYPO3

Portals: ZF, FLOW,…

Mobile Searchperience: SOLR

120 people in 7 offices world-wide

High Performance

/Scale

Global Enterprise Projects

Page 5: Continuous Quality Assurance using Selenium WebDriver

Continuous Integration

Deployment

Development

Delivery

Improvement

Page 6: Continuous Quality Assurance using Selenium WebDriver

Quality Assurance Continuous

Everything sounds better with “continuous”

Page 7: Continuous Quality Assurance using Selenium WebDriver

“Continuous” == “Doing stuff over and over again”

Builds Deployment Tests Infrastructure

Setup

Devbox Setup

== “Automation”

Page 8: Continuous Quality Assurance using Selenium WebDriver

tim

e s

pe

nt

task size

does it manually

does it manually

gets annoyed

writes script to automate

runs script

wins

loses

Geeks vs. Non-Geeks

Makes fun of geek’s complicated method

Page 9: Continuous Quality Assurance using Selenium WebDriver

— Ovidiu Platon

We are not shipping your machine! if it works on your machine! I don’t care

Page 10: Continuous Quality Assurance using Selenium WebDriver

Picture The BIG

Page 11: Continuous Quality Assurance using Selenium WebDriver

Vagrant Boxes

Page 12: Continuous Quality Assurance using Selenium WebDriver

Awesome stuff

Every single line potentially

can break production!

Bratwurst (Germans)

Donut (Americans)

Pizza (Italians)

Page 13: Continuous Quality Assurance using Selenium WebDriver

Version Control

GIT, SVN,…

PHP lint

Page 14: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews

“Gerrit”

Page 15: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Triggered by commit, by time

or manually

Jenkins

minify js/css

.tar.gz

“virtual package” (revision)

Page 16: Continuous Quality Assurance using Selenium WebDriver

Multiple Release Branches?*

Multiple Build Pipelines!

… *http://nvie.com/posts/a-successful-git-branching-model/

Page 17: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Page 18: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

CodeSniffer

php-pmd

php-cpd

php-depend

Page 19: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration”

First system where code from all

developers/teams meets each other

unstable system. Might be broken. Will be rebuild

without warning

Page 20: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests

PHPUnit

Code coverage

Page 21: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests

AOE PMD (for Magento)

broken links, HTTP headers, W3C validity,…

Test features that rely on other parts (e.g. framework, external

services,…) External components (SSL, Redis, Varnish,…)

Basically this is also PHPUnit, but

addressing other stuff

Page 22: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Here we go:

Selenium!

Page 23: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Behat (Cucumber)

Same setup as production, but different machines

Client QA / approval

Page 24: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning

Stress

Tests

Jmeter

Spin up EC2 instances in the cloud to produce

traffic*

Page 25: Continuous Quality Assurance using Selenium WebDriver

Again: multiple servers or autoscaling setup

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Stress

Tests

Install on

“Production”

A/B Deployments on different

scales

Plan B: rollback

Zero Downtime

Page 26: Continuous Quality Assurance using Selenium WebDriver

Varnish Array EC2

Frontend Array Backend Array EC2 EC2

EC2

Worker Array EC2 EC2

Deployment x

ELB Load Balancer

Route 53 DNS-Service

shop.angrybirds.com

Deployment

Page 27: Continuous Quality Assurance using Selenium WebDriver

Varnish Array EC2

Frontend Array Backend Array EC2 EC2

EC2

Worker Array EC2 EC2

Deployment x

ELB Load Balancer

Varnish Array EC2

Frontend Array Backend Array EC2 EC2

EC2

Worker Array EC2 EC2

Deployment x+1 Release Y

ELB Load Balancer

Route 53 DNS-Service

shop.angrybirds.com

Deployment

Page 28: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Stress

Tests

Install on

“Production”

Page 29: Continuous Quality Assurance using Selenium WebDriver

Deployment != Feature Release

Page 30: Continuous Quality Assurance using Selenium WebDriver

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Stress

Tests

Install on

“Production” Enable Features

Page 31: Continuous Quality Assurance using Selenium WebDriver

Version

Control Code

Reviews Build Static Code

Analysis

Install on

“Integration”

Unit Tests Integration

Tests

Acceptance

Tests

Install on

“Staging”

Install on

“Production” Enable

Features

Page 32: Continuous Quality Assurance using Selenium WebDriver

Pipeline Visualization

#284

#283

#282

#281

#280

Build Static Code

Analysis Unit Tests

Install on “Integration”

Integration Tests

Acceptance Tests

#285

for team dashboards

Page 33: Continuous Quality Assurance using Selenium WebDriver

Acceptance Tests

Page 34: Continuous Quality Assurance using Selenium WebDriver

test automation is a developer’s task

we need to provide them some good tools…

us

Page 35: Continuous Quality Assurance using Selenium WebDriver

Selenium 1 vs. Selenium 2

Page 36: Continuous Quality Assurance using Selenium WebDriver
Page 37: Continuous Quality Assurance using Selenium WebDriver

What’s out there? Selenium for PHP

PHPUnit_Selenium (Sebastian Bergmann)

WebDriver-PHP (Chibimagic)

Php-webdriver-bindings (Lukasz Kolczynski)

php-webdriver

PHPUnit_Selenium (Giorgio Sironi)

Page 38: Continuous Quality Assurance using Selenium WebDriver

php-webdriver Selenium 2

08/2011 Facebook/php-webdriver (Justin Bishop)

Element-34/php-webdriver (Adam Goucher)

instaclick/php-webdriver (Anthon Pang)

Page 39: Continuous Quality Assurance using Selenium WebDriver

php-webdriver

“A very thin wrapper of WebDriver”

Page 40: Continuous Quality Assurance using Selenium WebDriver

So what?

A thin layer is good! But not enough…

no test logic

no higher level components

dealing with Selenium specific tasks

not compatible to legacy test cases

No abstraction / structure

Page 41: Continuous Quality Assurance using Selenium WebDriver

Architecture

HTTP AUT

No direct access (DB, classes,…)

REST* (over HTTP)

*http://code.google.com/p/selenium/wiki/JsonWireProtocol

Page 42: Continuous Quality Assurance using Selenium WebDriver
Page 43: Continuous Quality Assurance using Selenium WebDriver

Project test cases

PHPUnit

Project Components

Components Library

Menta

php-webdriver

reusable

project specific

Page 44: Continuous Quality Assurance using Selenium WebDriver

Menta Selenium 2 Testing Framework for PHP

Who is “Menta”?

What is “Menta”?

Page 45: Continuous Quality Assurance using Selenium WebDriver

Why “Menta”?

Page 46: Continuous Quality Assurance using Selenium WebDriver
Page 47: Continuous Quality Assurance using Selenium WebDriver

Features Menta The framework.

Not the cat…

Components Session Management

? Bonus Features :) Event / Observer Configuration Management

PHPUnit integration

Page 48: Continuous Quality Assurance using Selenium WebDriver

Why?

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

Data Providers

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

Dependent Tests

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

✔ ✔ ✔

Test Infrastructure

Page 49: Continuous Quality Assurance using Selenium WebDriver

Menta for PHPUnit Abstract test class

PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2 ProjectY_Abstract_TestCase

Very thin layer on top of PHPUnit

ProjectY_TestCase_Registration ProjectY_TestCase_Checkout the “magic” happens in the components

Page 50: Continuous Quality Assurance using Selenium WebDriver

Legacy Selenium 1 tests PHPUnit_Framework_TestCase PHPUnit_Extensions_SeleniumTestCase

ProjectY_TestCase_Registration ProjectY_TestCase_Checkout

Page 51: Continuous Quality Assurance using Selenium WebDriver

Legacy Selenium 1 tests PHPUnit_Framework_TestCase Menta_PHPUnit_Testcase_Selenium2

ProjectY_TestCase_Registration ProjectY_TestCase_Checkout

Menta_PHPUnit_Testcase_Selenium1

Selenium1Facade

Compatible with PHPUnit_Extensions_SeleniumTestCase

Page 52: Continuous Quality Assurance using Selenium WebDriver

HTML Report Screenshots (on demand

and on failure)

test description from phpdoc

additional messages

error details

Page 53: Continuous Quality Assurance using Selenium WebDriver

HTML Report “Duration Heatmap”

shows long running tests

Page 54: Continuous Quality Assurance using Selenium WebDriver

Menta for PHPUnit Text Result Printer

PHPUnit 3.7.20 by Sebastian Bergmann. Configuration read from /var/www/Tests/Acceptance/conf/devfb.ff.vmhost.xml > SUITE: > SUITE: > SUITE: Acceptance_Tests_Account_Register > TEST: Acceptance_Tests_Account_Register::register SUCCESS. (Duration: 8.18 sec) > TEST: Acceptance_Tests_Account_Register::login SUCCESS. (Duration: 6.5 sec) > TEST: Acceptance_Tests_Account_Register::testRegistrationMail FAILURE: Searching for mail with subject 'Welcome, Test_705b077a962977e1a412696ea42249ae User_705b077a962977e1a412696ea42249ae' timed out (Duration: 22 sec) < Duration: 36 sec < Duration: 36 sec > SUITE: > SUITE: Acceptance_Tests_Checkout_CheckoutMultiAddressTest > TEST: Acceptance_Tests_Checkout_CheckoutMultiAddressTest::putProductsInCart FAILURE: Element "//table[@id="shopping-cart-table"]//h3[contains(concat(' ', @class, ' '), ' attentionText ')]" does not contain text "Bulb Vase" Failed asserting < Duration: 9.87 sec > SUITE: Acceptance_Tests_Checkout_CheckoutTest > TEST: Acceptance_Tests_Checkout_CheckoutTest::categoryView SUCCESS. (Duration: 1.79 sec) > TEST: Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart FAILURE: Failed asserting that two strings are equal. (Duration: 3.55 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::putPlatterInCart" to pass. (Duration: 5.34 sec) SKIPPED: This test depends on "Acceptance_Tests_Checkout_CheckoutTest::checkout" to pass. (Duration: 5.34 sec)

Page 55: Continuous Quality Assurance using Selenium WebDriver

Components?

Components!

Page 56: Continuous Quality Assurance using Selenium WebDriver

Manager Component

Menta_ComponentManager::get(‘PageObject_Cart’) PageObject_Cart

Page 57: Continuous Quality Assurance using Selenium WebDriver

Rewrites Component

PageObject_Cart ProjectZ_PageObject_Cart

extends

change locators

add functionality

change behavior

overwrite labels

Page 58: Continuous Quality Assurance using Selenium WebDriver

Manager Component

Menta_ComponentManager::get(‘PageObject_Cart’) ProjectZ_PageObject_Cart

Menta_ComponentManager::addRewrite(

‘PageObject_Cart’,

‘ProjectZ_PageObject_Cart’

);

Page 59: Continuous Quality Assurance using Selenium WebDriver

Components

Common Assert Wait Screenshot Selenium1Facade

Imap WebService W3C Validator Curl …

Some might not even require a Selenium session

Page 60: Continuous Quality Assurance using Selenium WebDriver

Gallery Screenshot

Page 61: Continuous Quality Assurance using Selenium WebDriver

Components! PageObjects?

PageObjects Components

{ } }⊇{

PageObjects are

components

Page 62: Continuous Quality Assurance using Selenium WebDriver

PageObjects?

represents services offered by a page or feature

is the only thing with a deep knowledge of

the HTML

clickCheckoutButton() isInCart($product) getTotals()

✔ ✔ ✔

Page 63: Continuous Quality Assurance using Selenium WebDriver

Component Libraries

Menta

Magento

(your library)

Project-

specific

Magento

(community-provided)

General

(your library)

Page 64: Continuous Quality Assurance using Selenium WebDriver

Session Management

Let Menta manage your sessions

auto-close

auto-start

incl. on exceptions and ctrl+c

Reusing sessions

Cleaning sessions

Page 65: Continuous Quality Assurance using Selenium WebDriver

Configuration Management

Fallback mechanism for global settings

Create multiple configurations for different browsers,

application instances or Selenium Servers

Compatible to PHPUnit (xml)

Page 66: Continuous Quality Assurance using Selenium WebDriver

Event / Observer Menta_Events::dispatchEvent('after_session_create');

Menta_Events::addObserver('after_session_create', function($session) {

$session->window('main')->postSize(array('width' => 1024, 'height' => 768));

}

Page 67: Continuous Quality Assurance using Selenium WebDriver

Bonus Features

Page 68: Continuous Quality Assurance using Selenium WebDriver

Sauce Labs Support

*configuration: http://username:[email protected]:80/wd/hub

works fine*

update test result (REST)

Page 69: Continuous Quality Assurance using Selenium WebDriver

Sauce Labs Support

Speaking name Test results

Page 70: Continuous Quality Assurance using Selenium WebDriver

Perceptual diffs Build X

Build X-1

Perceptual Diff

Page 71: Continuous Quality Assurance using Selenium WebDriver

github.com / AOEmedia / Menta github.com / AOEmedia / Menta_SampleProject

It’s composer based and you’ can get started in a few minutes!

Page 72: Continuous Quality Assurance using Selenium WebDriver

Thank you! Any questions?

http://www.aoemedia.com

http://www.fabrizio-branca.de

@fbrnc Follow me on twitter!

My blog