seconf2015: advanced automated visual testing with selenium
Post on 22-Jan-2018
4.289 Views
Preview:
TRANSCRIPT
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com
ADVANCED AUTOMATED VISUAL TESTING WITH
SELENIUM
YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
Q & A
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
A VISUAL BUG
AND ANOTHER…
AMAZON PRIME DAY – JULY 15th, 2015
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO COVER MANUALLY Web browsers Devices Operating systems Screen resolutions Responsive design L10n 3rd Party upgrades
WHY SHOULD IT BE AUTOMATED?
WHY SHOULD IT BE AUTOMATED?
NATIVE / HYBRID MOBILE APPS Harder to roll back changes
Can’t push daily
Updates take battery and data
Higher quality bar
WHY SHOULD IT BE AUTOMATED?
Many are already doing it…
PhantomCSSFighting Layout Bugs
CSS Critc Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
EyesHuxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare
kobold
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
Q & A
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
SCREENSHOTS
Full page? Regions? Frames? Device pixel ratio? Rotation? Viewport size? Page preparation? Page stabilization?
QUICK FEEDBACK TOOLS
MOTIVATION
Get fast feedback on code
changes
Run tests locally in the
background
SETUP
Render screenshots with a
headless browser
PhatomJS, SlimerJS
Configuration file driven tests
CONS
Partial coverage
Chrome, IE?
Limited navigation
Tests stale browser versions
VISUAL COVERAGE TOOLS
MOTIVATION
Verify that the app / site looks right
in all real execution environments
Test as many UI states as possible
Execute many tests in parallel
SETUP
Render screenshots using real
browsers on a variety of
operating systems and devices
in parallel
WebDriver, Grid
Code driven tests
WebDriver, DSLCONS
Requires test infrastructure
Implement and maintain test
code
SELENIUM BASED TOOLS
CODE / SCRIPT
Needle (Python WD)
WebDriverCss (JS WebDriverIO)
Gemini (JS DSL)
Selenium Visual Diff (Java WD)
VisualCeption (PHPCodeCeption)
Pix-Diff (JS Protractor)
Shoov (JS WebDriverIO)
Vizregress (.NET WD)
Rspec Page Regression(Capibara)
Applitools Eyes (All WD + Appium)
Fighting Layout Bugs (JavaWebDriver)
Huxley (Record Playback)
CONFIGURATION
Viff
CSS Visual Test
GreenOnion
Wraith-Selenium
NON-SELENIUM BASED TOOLS
CODE / SCRIPT
PhantomCSS (JS CasperJS)
Specter (JS DSL)
FBSnapshotTestCase (XCTest)
CONFIGURATION
Wraith
Dpdxt
Grunt PhotoBox
Grunt-Vigo
Snap And Compare
BackstopJS
CSSCritic
Kobold
DiffCop
SUCCSS
MANUAL
Eyes Express (Chrome)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
WHY NOT?
FALSE POSITIVES
ANTI-ALIASING?
ANTI ALIASING 1/2
ANTI ALIASING 2/2
BRIGHTNESS 1/2
BRIGHTNESS 2/2
TEXT POSIONING 1/2
TEXT POSIONING 2/2
IMAGE SCALING 1/2
IMAGE SCALING 2/2
ANDMORE…
1 pixel offsets in element positioning
Dynamic content
Moving elements
Images of different size
Performance
Image Comparison APIs
ImageMagick A powerful command line tool for image processing.
APIs are available for most programming languages.
Fuzzing is used to eliminate slight color differences
An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
The Javascript Engines Resemble.js
Pixel by pixel + error ratio + anti-aliasing
http://huddle.github.io/Resemble.js
Blink-DiffPixel by pixel + error ratio + anti-aliasing + perceptual color distance computation
https://github.com/yahoo/blink-diff
Looks-SamePixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same
Applitools Eyes A specialized image processing stack designed to compare computer
generated UI images
Handles anti-aliasing, pixel offsets, color similarity, and image scaling
Dynamic and moving content
Compare images of different sizes
No error ratio configuration required
Validates full UI pages
Fast!
Supports layout matching
DEMO
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
REPORT DIFFERENCESAs files on the file system (combined with source control)
REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
UPDATE THE BASELINE
Rename or commit individual image files
GUI (Gemini GUI)
UPDATE THE BASELINE
Overwrite mode
Automatic maintenance (Applitools Eyes)
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
Q & A
WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA
Visual testing of frontend components by
frontend developers
WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others
Full or partial validation of application
screens
A collaboration tool
WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA
Validate your staging deployment using your
production deployment as a baseline
WHERE DOES IT FIT?
• Monitoring
• Ops
• QA
No missing resources in production
No corruption due to 3rd party data
No corruption due to browser / OS upgrades
QUESTIONS?
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com
THANK YOU
top related