qaops - nyc selenium users meetup - 03/15/12
Post on 08-May-2015
2.262 Views
Preview:
DESCRIPTION
TRANSCRIPT
Christopher Hoey
■ Senior IT Manager - Wireless Generation
■ Started in QA
■ Switched over to Operations
■ Selenium abuser since 2006
■ Got to see the Selenium vs Webdriver Cage Fight live
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Founded 12/14/06 Closed 07/16/09
Thanks!
Ashley WilsonLindsey BaronSauce LabsConductor
NYC Metro Selenium Users
Selenium Developers
Jmeter Developers
Show of hands
Numbers collected and updated before sharing slides
About 35 attended and participated in informal poll
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - primary job title
Quality Assurance ?12
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - primary job title
Developer ? 10
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - primary job title
Operations ? 5
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - experience with Selenium
Have not used it?5
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - experience with Selenium
Tried Selenium IDE?
8Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - experience with Selenium
Mastered Selenium IDE?
1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - experience with Selenium
Tried Selenium RC/Webdriver?
13Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - experience with Selenium
Mastered Selenium RC/Webdriver?
6Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - experience with Selenium
Selenium tied to CI build system?
10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - experience with Selenium
Running a Selenium Grid?
7Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - use of Selenium
Functional Testing New Features?
9Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - use of Selenium
Regression Testing?
10Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - use of Selenium
Load/Stress Testing? really?
1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - use of Selenium
Benchmarking Performance?
1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - use of Selenium
Automating Data Entry?
1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - use of Selenium
Monitoring for Operations?
1Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Show of hands - use of Selenium
Something Strange?
2 Filling out forms to renew trial licenses when expired
Running build jobs for a build tool that does not have a cli
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
DisclaimersNot a talk about best practices
Forcing Selenium to do things not originally intended
A lot of talk about Selenium IDE QAOps term kind of meant as a joke but might have meaning? Times have changed, tools improved and storms have passed
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Dev + Ops = DevOps --> QA + Ops = QAOps
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
About Wireless Generation
● Create tools that make teacher's jobs easier
● Primarily focused around assessment and reporting apps
● Mix of mature code/platforms and new development projects
● Experiencing rapid growth year over year
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Faith in monitors pretty low Manual configuration management Serious bugs in third party tools Large increases in user traffic Lots of new projects to juggle while keeping existing work online Sysadmins manually checking site =(
Reuse anything that helps!
Going MacGyver in early days of Ops
Selenium to the rescue!
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
● Use Selenium IDE so anyone in Operations with Firefox can use
● Suites are happy path smoke tests that are not brittle ● Focus on touching all moving parts of infrastructure
● All suites can run with any username
● All suites can run on any environment
● Sysadmins running Selenium tests -- still done manually =/
Suites of Smoke Tests for Operations
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Suites for Operations
DEMO TIME
Removing brittleness from Selenium IDE tests against a site with difficult
locators
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Suites for Operations - DEMO TIME
All we really want to know is if we threw any errors while loading pages that touch on all parts of our platform. Two simple regex searches for every page gets us the desired minimal coverage
Yes that looks way too simple but it did the job very well.
These tests started finding issues before our customers did.
Confidence increased dramatically!
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Suites for Operations - DEMO TIME
https://zzz.yyy.net/wgen/tpri/LoadTpriClasseSummary.do?ppkDefaultVersionId=7&ppkPeriodId=20&ppkClassId=550987013&ppkCompetencyDisplayId=1&ppkInstId=33645685&ppkSchoolGroupId...
We know the starting URL pattern.
The rest of the URL is really dynamic and the page locators that happen to be available are not Selenium friendly. We could use XPath but that has proven to be really brittle for us. Solution - Store the html source and use storeEval with a regex to find the correct link every time
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Automation - judges.py
■ Headless Selenium servers
■ Suites of tests for each production webapp using smoke tests
■ Run through suites serially in a never ending loop
■ Any failure in suites marks the status as failed for that server
■ Look at each servers total result time running the
suite based on thresholds
■ Running helps see what server is currently being tested
■ Results projected on wall in Ops area
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
We have a fundamental problem
Sysadmins Customers
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Solution - Any browser any platform
Bromine
Grid
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting Selenium vs JMeter
● Lots of start up overhead
● ~30 concurrent sessions before strange things happen
● Lots of requirements / coordination headless is a hack
● Loads quickly
● ~300 concurrent sessions before strange things happen
● Java and a jar, headless is easy
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter
Queries to find interesting users
Scripts to replace variables for Selenium suites
Playback Selenium scripts through JMeter proxy
Selenium Scripts IN
JMeter Scripts OUT
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter
DEMO TIME
Setting up jmeter as a proxy recorder for Selenium
playback
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Start JMeter and create a new basic WorkBench
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Add an HTTP Proxy Server: Edit -> Add -> Non-Test Elements -> HTTP Proxy Server
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Set your port, any record filters of what to include or exclude and click start to fire up the proxy
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Set Firefox network settings to use localhost and the port you configured JMeter proxy to use
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Playback your tests using Selenium and watch as JMeter records the hits
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Add a thread group by right clicking Test Plan -> Add -> Threads -> Thread Group
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Name your thread group, pick how many threads you want, ramp up time and how many loops
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Loadtesting with JMeter - DEMO TIME
Run your tests and set servers ablaze. Run -> Start
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Finding hard bugs
Uh oh! Just throwing more memory at the problem will not solve this!
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Finding hard bugs
Did we see this in the environments below Production? Not very noticeable unless looking hard for it.
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Finding hard bugs
Ramping up JMeter threads of a specific user in an endless loop reproduces issue in every environment
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Dashboard Setup?
Use Selenium to login and navigate to your various dashboards for TV's and sharing with teams.
QAOps?
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Getting most of the focus these days. Certainly best of all worlds coming together. Takes a while to be a good practitioner based on tech debt | servers | timeboxes | complexity | culture
A lot of quick wins can be made here.
Selenium IDE makes it trivial for Ops to start running and developing tests
Questions?
We are hiring!
http://www.wirelessgeneration.comDev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
Picture Credits
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
http://www.russpainter.com/shop/images/clip.jpg
http://g-ecx.images-amazon.com/images/G/01/stores/sport-goods/1209-huntsman.jpg
http://uploads.neatorama.com/wp-content/uploads/2010/06/macgyver.jpg
http://www.agilegeoscience.com/storage/post-images/python-logo-small.png?__SQUARESPACE_CACHEVERSION=1316752792052
http://www.packagingsuppliesbymail.com/images/C/Duct%20Tape.jpg
Picture Credits
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
http://upload.wikimedia.org/wikipedia/en/0/07/Jakarta_jmeter_logo.jpg
http://cdn.babble.com/momcrunch/files/2012/02/SauceLabs_Logo_Vert_150px.gif
http://www.pptbackground.net/plog-content/images/powerpoint/education-backgrounds/raising-hands-together-powerpoint-backgrounds.jpg
http://gigaom2.files.wordpress.com/2011/01/firefox-ie.png
http://seleniumhq.org/images/selenium-ide-logo.png
Picture Credits
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
http://www.portlandmercury.com/imager/dr-strangelove/b/original/3791589/80b9/mwbw-drstrangelove.jpg
http://stylreesta.files.wordpress.com/2011/07/caution_big.png
http://3.bp.blogspot.com/_t5Xqp0AYeg4/S74FdH-SO-I/AAAAAAAAAOU/OlMrI9m8VIE/s320/smileyface1.jpg
http://en.wikipedia.org/wiki/File:Devops.png
http://blogs.babycenter.com/wp-content/uploads/2008/02/crying-face.jpg
Picture Credits
Dev + Ops = DevOps --> QA + Ops = QAOpsNYC Metro Selenium Users Meetup March 15, 2012
http://seleniumhq.org/images/selenium-grid-logo.png
http://www.studiosysadmins.com/site_media/attachments.gallery/2009/05/12/ServerRoomFireWeb.jpg
http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Tux.png/220px-Tux.png
http://blogs.splunk.com/wp-content/uploads/2010/02/Picture-65.png
http://www.wisedatasecurity.com/imagenes/ie-8.jpg
top related