argus eyes

argus eyesBran van der Meer, 18-08-2015

Agenda

1. Regression2. Tooling3. Argus Eyes

Regression

1

re·gres·sionnoun; /rɪˈɡrɛʃ(ə)n/1 a situation in which things get worse rather than better2 a return to a former state

Regression testing

Regression testing is a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional areas of a system after changessuch as enhancements, patches or configuration changes, have been made to them.

Visual Regression

• We’re not functionally testing the application• We’re only testing the implemented design• CSS is “global”, and thus risky• Automation because continuous bug delivery• Automation because humans are not robots

Spot the differences

Humans fail at tiny details

Why in the front-end?

• Fail fast• We know what is visually right and wrong• We need to fix the visual bugs anyway

Tooling

2

Most tools

• Need a config (json/yaml/dsl)• Capture baseline• Compare current state with baseline

Requirement: Fail fast

• Early dropout, unit-test style• CLI reports• Possibly pre-commit or pre-push

Requirement: Component-based

• Page-based gives too many false positives• No duplicate selectors in config

Requirement: Gitflow

Requirement: Async loading

• When has the page finished loading?• Fixed timeout?• Event fired?• Attribute available?• Custom conditions?• Require.js& Conditioner.js

Requirement: Flexible

• Generate the config (or partially)• Set baseline (or multiple)• Compare (any two sets of screenshots)

Existing tools

• PhantomCSS• BackstopJS• Succss• Wraith• Galen• Applitools

Existing tools → Meh

• PhantomCSS → Buggy• BackstopJS → No Gitflow support• Succss → Immature• Wraith → Ruby• Galen → Too complex• Applitools → Not open source

“It can’t be that difficult, right?”— Famous last words.

Argus Eyes

3

http://github.com/arguseyes/argus-eyesFuture: http://arguseyes.io/

Flexible

• Simple CLI tool, Unix philosophy• Distilled down to just two actions:

• Capture a new set, save as any name• Compare any two sets

Walkthrough

1. Create a config file2. Set a baseline for comparison3. Add commits4. Add the current state for comparison5. Test for visual differences between the two sets

Create a config file

• Tell argus eyes what screen sizes, pages and components you have

• Filename: argus-eyes.json• Pages have a name and url• Pages have components• Components have a name and css selector

Example config file{

"sizes": ["320x480", "768x1024"],"pages": [ {

"name": "homepage","url": "http://localhost:3000/","components": ["header", "footer"]

} ],"components": [ {

"name": "header","selector": ".header"

}, {"name": "footer","selector": ".footer"

} ]}

Set baseline for comparison

argus-eyes add baseline

1 Set of screenshots in cache.argus-eyes/

baseline/

320x240/

homepage/

header.png

footer.png

768x1024/

homepage/

header.png

footer.png

Add current state for comparison

argus-eyes add current

2 Sets of screenshots in cache.argus-eyes/ .argus-eyes/

baseline/ current/

320x240/ 320x240/

homepage/ homepage/

header.png header.png

footer.png footer.png

768x1024/ 768x1024/

homepage/ homepage/



Test for visual differences

argus-eyes compare baseline current

Succeeded compare

Failed compare

Found a difference

Comparison results also in cache.argus-eyes/ .argus-eyes/ .argus-eyes/

baseline/ current/ diff_baseline_current/

320x240/ 320x240/

homepage/ homepage/



768x1024/ 768x1024/ 768x1024/

homepage/ homepage/ homepage/


footer.png footer.png footer.png

Gitflow examplegit checkout masterargus-eyes add master

git checkout developargus-eyes add developargus-eyes compare master develop

git checkout feature/developargus-eyes add feature/developargus-eyes compare master feature/developargus-eyes compare develop feature/develop

Feature: ignore elements

Feature: ignore elements

Hide a child element of a component (display:none):

{

"components": [

{

"name": "happy-hour",

"selector": ".happy-hour",

"ignore": ".counter"

}

]

}

Feature: finished-when

We’re not visually finished when the DOMContentLoadedand window.onload events are fired!

Feature: finished-when

Provide a function to specify when it’s finished loading:

{

"finished-when":

"return document.body.hasAttribute('data-conditioner-initialized');"

}

{

"finished-when":

"setTimeout(function() {

var timeout = true;

}, 2500);

return timeout;"

}

Feature: threshold

Set the threshold percentage for comparison:

argus-eyes compare develop current--threshold=5

Try to solve every false positive with ignore first!

Feature: configtest

Run a config file syntaxtest:

argus-eyes configtest

$ npm i argus-eyes -g

$ argus-eyes --versionv0.4.0

Pull Request incoming

• Generate & update config file:gulp visual-regression-update-config

• Add baseline:gulp visual-regression-add

• Compare current with baseline:gulp visual-regression-test

• Add set with any name:gulp visual-regression-add --name=develop

• Compare any two sets:gulp visual-regression-test

--left=master --right=develop

Thoughts?

argus eyes

Software