Getting By Without “QA”

San Francisco Baking in the Quality Meetup 05/12/2015 Dave King

About Me

• Programming 20+ years

• Penn State PhD 2009

• Team Lead at Tilt

• Here’s a picture of my dog

What is Tilt?

Tilt is the the easiest way to collect money from a group.

We Are Hiring!

tilt.com/learn/jobs

Tilt Engineering

• ~80 people at company, ~30 engineers

• CI/CD - ship code to production daily

Is QA Necessary?

• No QA specialists on engineering team

• “Quality is everyone’s job”

• We’ve seen both sides of the “No QA” coin

Is QA Necessary?Good Bad

Team responsible for quality Quality is defined by the team

QA doesn’t bother you about bugs CEO bothers you about bugs

Ship code to production fast Ship bugs to production fast

Nobody tells you to write tests Nobody tells you to write tests

Our Quality Journey

• A year ago, many defects in production

• “[one] week where every single deploy had to either be rolled back or have a fast follow”

• Today - very few escaped production defects, many small uneventful deploys

One Year Ago

• Tests run on every branch prior to merge - takes ~30 minutes

• Unstable tests - “rerun until green” mentality

One Year Ago

• Single team member fixed test suite when it broke

One Year Ago

• Very small smoke suite run in staging prior to production deploy

• Tested only 3 core flows

• Tests written in Perl

Today

• Tests rewritten using Nightwatch.js

• Entire team responsible for expansion/maintenance of test suite

Today• Selenium Suites for 15 core flows

• ~60-70 scenarios

• “Create Tilt”, “Contribute to Tilt”, “Auth”, etc

• Run every 10 minutes vs staging

• All passing when I left for this meetup 😎

What’s Been Helpful?

“Bad” Tests

“Good” Tests

Lessons Learned

• You need to invest in test improvement

Investment is Important

• Can’t say “fix the tests” and expect results

• Team may not know the technologies involved

• Team may not feel like they have time to investigate and learn

How Can You Invest?

• Hire people who have done it before

• Give the team exploratory time to improve tests

• Represent functional tests in JIRA: “Write Test” stories

Lessons Learned

• Continually run tests against staging

Test Against Staging

• Dev environments have low fidelity

• A failing test is everyone’s problem

• Tests rerun themselves to green, smoothing out “flappers”

Lessons Learned

• Choice of technology matters

Choice of Technology Matters

• Past job - write tests in Ruby, write code in JavaScript/Python

• Nobody knows the tools involved, language idioms, etc

• “Copy-paste” mentality, “lol what’s a Gemfile”

Choice of Technology Matters

• Tilt uses Nightwatch.js, written in JavaScript

• Add libraries to tests using npm

• We use babel.js to make our tests even better

Choice of Technology Matters

• register babel in nightwatch.conf.js

• Now, enjoy ES6 in your tests!

var createTestUser = (appId) => { return (token) => { return request({ method: 'POST', url: `https://graph.facebook.com/${appId}/accounts/test-users?access_token=${token}`, json: true }); }; };

Choice of Technology Matters

• Nightwatch.js provides a good foundation for a functional test suite

• Page objects and commands are “built in”

• Hand-rolled suite needed to reinvent these

Lessons Learned

• Split tests into a separate repository

Separate Test Repository

• Easier to see what’s important for things to work

• Slightly looser code review standards 😉

Lessons Learned

• Improving your product improves test stability

Improve Your Product

• User experience is stochastic

• Tests set an upper bound, “wait 5 seconds…”Test fails if user experience is

worse than predefined wait time!

Improve Your Product

• You can improve unstable tests by changing the waits to more accurately reflect your application’s behavior

• You can also improve the behavior of the application!

Improve Your Product

• After rewriting tests, some jobs were still only 65-70% stable

• Hard to know if a build on staging was good or not!

Improve Your Product

• Over the last three months, we’ve made big steps to improve the product:

• Rewrote contribution lightboxes in React.js

• Massively sped up our campaign page load

• Removed page refresh after facebook login

Improve Your Product

• Now all tests 95%+ reliable, very few “flappers”

• But we didn’t really spend time tunneling to “improve the tests”

• Benefit to no-QA - team members can invest time into product improvements too!

Our Next Steps

• Continue to expand test suite to cover all core flows

• Add functional coverage for tracking calls

• Add cross-browser coverage

• Investigate visual testing (screenshot diffs)