mozilla: continuous deploment on sumo

MOZILLA Continuous Deployment on Sumo

Matt Brandt, QA Engineer

[email protected]

support.mozilla.com

Tuesday, November 22, 2011

mailto:[email protected]


MOZILLA Continuous Deployment on Sumo

[Nearly]

Matt Brandt, QA Engineer

[email protected]

support.mozilla.com




Bring questions


Bring questions

What’s wrong with our current small iterations?


Bring questions


Trust developers to test the app??!!


Bring questions



What happens to our Selenium test suite?


Bring questions




We’ll be relegated to manual testers?


Bring questions




We’ll be relegated to manual testers?

How do we measure quality?


But first a little history...


Short Iterations


SimplifiedWhat the team did before

Trunk/Dev

Stage Prod



Trunk/Dev

Stage Prod

•Verify new features

•Is this the right thing?

•Automate



Trunk/Dev

Stage Prod



•Automate

•IT pushes

•Regression test

•Exploratory Testing




Trunk/Dev

Stage Prod



•Automate

•IT pushes

•Regression test

•Exploratory Testing


•IT pushes

•Verify

•Wahoo!


The brave new worldContinuous deployment

http://cliptank.com/funny/dog-costumes.php




Tools & attitude


Discuss what is acceptable risk

https://wiki.mozilla.org/QA/Execution/Web_Testing/Continuous_Deployment#Risks_.26_Plans_.26_Tools


https://wiki.mozilla.org/QA/Execution/Web_Testing/Continuous_Deployment#



Risk: Less critical areas will not be covered by automation







•If the feature breaks, it is acceptable to wait up to an hour for a fix.







•If the feature breaks, it is acceptable to wait up to an hour for a fix.

•Example: the ability to answer a question.





Acceptable risk


What’s our manual strategy?


Manual strategy


Manual strategy

Feature verification & exploratory testing


Manual strategy


Testing can occur in production


Manual strategy


Testing can occur in production

Community involvement on new features


Automation is fun

https://github.com/mozilla/sumo-tests




Automation strategy


Automation strategy

Selenium tests for areas that are clumsy to cover with unit tests or QUnit


Automation strategy


Qualify as deployment blockers: run in dev & staging envs


Automation strategy



Non-volatile production tests verify env & services work


Automation strategy



Non-volatile production tests verify env & services work

[All] Tests will be deleted if they don’t cover critical areas


WafflingFailure is always an option


Control


Gathering data


Now

Trunk/Dev

Stage Prod

•Automation runs

•Fast iterations

•Early stage testing


Now

Trunk/Dev

Stage Prod

•Automation runs

•Fast iterations


•Auto-updated

•No post-commit hooks

•Bug & feature testing


Now

Trunk/Dev

Stage Prod

•Automation runs

•Fast iterations


•Auto-updated

•No post-commit hooks

•Bug & feature testing

•Not push-button

•Automation

•Monitoring

•Feedback tool

•Wahoo!


Questions


•https://support.mozilla.com•https://wiki.mozilla.org/QA/Execution/Web_Testing/Continuous_Deployment•https://github.com/mozilla/sumo-tests•http://coffeeonthekeyboard.com/the-future-of-sumo-development-511/•http://coffeeonthekeyboard.com/acronyms-you-should-know-mttd-and-mttr-597/•http://www.slideshare.net/stephendonner/continuous-deployment-9363895

•http://docs.jquery.com/QUnit•https://github.com/jsocol/django-waffle•http://coffeeonthekeyboard.com/introducing-waffle-for-django-541/•https://github.com/etsy/statsd•http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

References


https://support.mozilla.com

https://support.mozilla.com



http://coffeeonthekeyboard.com/the-future-of-sumo-development-511/




http://coffeeonthekeyboard.com/acronyms-you-should-know-mttd-and-mttr-597/




http://docs.jquery.com/QUnit

http://docs.jquery.com/QUnit

http://coffeeonthekeyboard.com/introducing-waffle-for-django-541/




https://github.com/etsy/statsd

https://github.com/etsy/statsd

mozilla: continuous deploment on sumo

Technology