are your continuous tests too fragile for agile?
Post on 16-Jul-2015
550 Views
Preview:
TRANSCRIPT
Paraso& Proprietary and Confiden1al 1
11/13/14
Are Your Con*nuous Tests Too Fragile for Agile
Arthur Hicken Paraso& Evangelist
Paraso& Proprietary and Confiden1al 2 Paraso& Proprietary and Confiden1al 2
Agenda
§ Challenges of Con1nuous Agile § Tes1ng defined § Tes1ng is hard (no one wants to do it) § Best prac1ces § A modest proposal: Spring cleaning
Paraso& Proprietary and Confiden1al 3 Paraso& Proprietary and Confiden1al 3
The role of QA in Agile
§ QA is part of development, not an a&erthought
§ Review developer tes1ng efforts § Addi1onal tes1ng of requirements § Itera1ve during development § Fast discovery and repor1ng of defects § Inves1gate root cause of defects § Influence development policy
Paraso& Proprietary and Confiden1al 4 Paraso& Proprietary and Confiden1al 4
Challenges of Agile
§ Need to be fast § Need to be flexible § Tes1ng can be overlooked if not baked in § Architectural compromises from schedule
Paraso& Proprietary and Confiden1al 5 Paraso& Proprietary and Confiden1al 5
Challenges of Con1nuous
§ Tests must produce binary decision go/no-‐go § Reuse unit test and func1onal tests from dev to QA
§ High level of automa1on § Requires disciplined mature process § Tes1ng must automa&cally answer
§ Is it stable § Will it do what it’s supposed to do
Paraso& Proprietary and Confiden1al 6 Paraso& Proprietary and Confiden1al 6
Importance of Tes1ng
§ How do you know when you're done? § How do you know if a fix for a minor bug broke a major func1on of the system?
§ How can the system evolve into something more than is currently envisioned?
§ Tes1ng, both unit and func1onal, needs to be an integrated part of the development process.
Paraso& Proprietary and Confiden1al 7 Paraso& Proprietary and Confiden1al 7
Tes1ng delivers
A maintained suite of unit tests: § Represents the most prac1cal design possible § Provides the best form of documenta1on for classes
§ Determines when a class is "done" § Gives a developer confidence in the code § Is a basis for refactoring quickly
Paraso& Proprietary and Confiden1al 8 Paraso& Proprietary and Confiden1al 8
Why Tes1ng is O&en Ignored
§ Perceived as “burdensome” § Too much effort § Too much 1me § We’re on a schedule – no 1me to test.
§ Why is there always 1me to do it over, but never to do it right?
§ Developer Ego § “I don’t need to test my code – I don’t make mistakes.”
§ Tes1ng means I have to admit that I might have made a mistake.
§ I don’t like to test – it’s beneath me. § Tes1ng isn’t fun – I just want to code! § That’s for the QA dept.
Paraso& Proprietary and Confiden1al 9 Paraso& Proprietary and Confiden1al 9
Defini1ons
§ Unit § Func1onal § Regression
Paraso& Proprietary and Confiden1al 10 Paraso& Proprietary and Confiden1al 10
Unit Tes1ng
Unit tests that validates your method behaves a certain way given a specific parameters
§ Regressions tes1ng § Validate behavior of exis1ng code hasn’t changed § Scheduled and run automa1cally
§ Test Driven Development § Validate behavior of new code to be wrijen § Created before developing, turned into regression tests a&er
Paraso& Proprietary and Confiden1al 11 Paraso& Proprietary and Confiden1al 11
Func1onal Tes1ng
§ Incarna1ons of func1onal tests § Unit tests (Junit, CppUnit, Nunit, etc) § Integra1on tests § Scripted tests § Manual tests
§ Ways to create func1onal tests § Manually wrijen § Machine generated from applica1on tracing § Machine generated from capturing traffic § Machine generated from recording clicks
Paraso& Proprietary and Confiden1al 12 Paraso& Proprietary and Confiden1al 12
Regression Tes1ng
§ Contributors to the regression test suite § Func1onal unit tests that pass § Machine generated unit tests for current behavior § Unit test wrijen by developers and testers § Tests wrijen to validate bug fixes
§ Regression test failures § Unwanted change in behavior – fix the code § Inten1onal change in specifica1on – update test controls
Paraso& Proprietary and Confiden1al 13 Paraso& Proprietary and Confiden1al 13
What’s the Difference?
Unit Tests • The code is doing things right • Wrijen from programmers perspec1ve • Confirm code gives proper output based on specific input
Func1onal Tests • The code is doing the right thing • Wrijen from users perspec1ve • Confirm the system does what the user expects it to do
Regression Tests • It s1ll works
Paraso& Proprietary and Confiden1al 14 Paraso& Proprietary and Confiden1al 14
Why do Unit Tes1ng
So&ware bugs are inserted during coding phase, but o&en aren’t found un1l func1onal tes1ng, or worse they are found by customers. The longer a bug stays in the system, the more expensive it is to fix it.
Unit tes1ng is a strategy that allows you to find out if your system is behaving
the way you expect early in the SDLC. Unit tes1ng allows for easy regression tes1ng, so you can find out quickly
whether you have introduced some unexpected behavior into the system.
Paraso& Proprietary and Confiden1al 15
The Cost of Defects
Paraso& Proprietary and Confiden1al 16 Paraso& Proprietary and Confiden1al 16
Best Prac1ces
Paraso& Proprietary and Confiden1al 17 Paraso& Proprietary and Confiden1al 17
What’s Important
§ Unit tests require a LOT of care § Unit tes1ng is a con1nuous opera1on § Test suite maintenance is a commitment § Control data (asser1ons) in the code
Paraso& Proprietary and Confiden1al 18 Paraso& Proprietary and Confiden1al 18
Test incorpora1on
§ Take exis1ng unit tests and schedule them into build process
§ Run and fix tests as part of build process § Ideal goal: 0 failures and 100% code coverage § Realis1c goal: 10% test failures and 80% code coverage
§ If the unit test suite is noisy, don’t qualify the build based on asser1on failures
Paraso& Proprietary and Confiden1al 19 Paraso& Proprietary and Confiden1al 19
Unit Test Crea1on
§ Maintainability § Proper asser1ons § Nightly runs § Tests should be:
§ Automa1c (no human input or review) § Repeatable (no strange dependencies)
Paraso& Proprietary and Confiden1al 20 Paraso& Proprietary and Confiden1al 20
Legacy Test Suites
§ Leverage your exis1ng suite § Clean up incrementally § Set policy based on current results § Tighten policy when in compliance
Paraso& Proprietary and Confiden1al 21 Paraso& Proprietary and Confiden1al 21
Maintaining and Crea1ng UT
§ Maintaining is more of a problem than crea1ng.
§ Don’t create a test if you can’t or don’t plan to maintain it.
§ Crea1ng as you go creates a bejer test suite § Tests not run regularly become irrelevant
Paraso& Proprietary and Confiden1al 22 Paraso& Proprietary and Confiden1al 22
Write tests with the code
§ Create the test when you’re wri1ng the code § Have a test for every task you complete § Increases understanding § Improve asser1ons
Paraso& Proprietary and Confiden1al 23 Paraso& Proprietary and Confiden1al 23
Connect Unit Tests
§ Associate tests with Code and Requirements § Naming Conven1ons should have meaning § Use code comment tags @task @pr @fr § Use check-‐in comments @task @pr @fr § Improve change-‐based tes1ng
Paraso& Proprietary and Confiden1al 24 Paraso& Proprietary and Confiden1al 24
Specific Genera1on
§ Don’t be lured by 100% coverage § Auto asser1ons are not meaningful unless validated
§ Review the tests as you work on the code
Paraso& Proprietary and Confiden1al 25 Paraso& Proprietary and Confiden1al 25
Code Review Test
§ Code for unit tests should be reviewed § Consider doing test-‐design review § Ensures you’re tes1ng what you think you are § Helps reduce noise § Helps improve understanding § Review the code and the test code together
Paraso& Proprietary and Confiden1al 26 Paraso& Proprietary and Confiden1al 26
Tes1ng Excep1on Handling
§ Frequently Skipped § Prevents crashes § Prevents death spiral § Rule of thirds
§ 1/3 basic func1onality § 1/3 advanced features / infrequent serngs § 1/3 error handling
Paraso& Proprietary and Confiden1al 27 Paraso& Proprietary and Confiden1al 27
All About Asser1ons
§ Verifies that the code behaves as intended § True/false § Null or not null § Specific values § Complex condi1ons
Paraso& Proprietary and Confiden1al 28 Paraso& Proprietary and Confiden1al 28
Why Assert?
§ Excuses: § My code already has console output § I can check it visually/manually
§ Binary output (pass/fail) § Part of overall test suite § No human interven1on
Paraso& Proprietary and Confiden1al 29 Paraso& Proprietary and Confiden1al 29
Asser1on failures
Introduced coding error • Fix the new code
Func1onality changed • Update the asser1on
Improper asser1on • Comment it out
Paraso& Proprietary and Confiden1al 30 Paraso& Proprietary and Confiden1al 30
Asser1on Policy
§ Assign failures as tasks § Author of the test or of the changed code § Try to reach developer that has the highest chance to understand the change
§ Developer assesses failure: § Bug (fix) § Changed func1onality (update asser1on) § Bad asser1on (improve or remove) -‐ don’t delete; instead, comment out
Paraso& Proprietary and Confiden1al 31 Paraso& Proprietary and Confiden1al 31
What to assert
§ Ease of crea1ng asser1ons leads to poor choices
§ Anything CAN be checked – but should it be? § Asser1ons should be connected to what you’re trying to test
§ Asser1ons should be logically invariant § Check values by range, not equivalence § “What do I expect here logically?”
Paraso& Proprietary and Confiden1al 32 Paraso& Proprietary and Confiden1al 32
Asser1on Example: Java
public Integer getNumber() { Scanner input = new Scanner( System.in ); System.out.print( "Enter a number between 0 and 10: " ); int number = input.nextInt(); // assert that the absolute value is >= 0 assert ( number >= 0 && number <= 10 ) : "bad number: " + number; return number
}
Paraso& Proprietary and Confiden1al 33 Paraso& Proprietary and Confiden1al 33
How to recognize bad asser1ons
§ How o&en do you WANT to manually verify an asser1on?
§ Daily or near-‐daily checking indicates poor asser1on.
§ Troublesome possibili1es: § Date § User or machine name § OS § Filesystem names and paths
Paraso& Proprietary and Confiden1al 34 Paraso& Proprietary and Confiden1al 34
Missing Asser1ons
§ Error was found a&er code changed § Test suite never caught the error § Update unit-‐test with new asser1on § Write new unit tests as necessary
Paraso& Proprietary and Confiden1al 35 Paraso& Proprietary and Confiden1al 35
Cleaning It Up
Paraso& Proprietary and Confiden1al 36 Paraso& Proprietary and Confiden1al 36
The Challenge: Management’s Perspec1ve
§ Need tests to reduce risks § Tes1ng takes 1me from
features/fixes § Not enough 1me § Not enough people § Tests and reports lack context
Paraso& Proprietary and Confiden1al 37 Paraso& Proprietary and Confiden1al 37
The Challenge: Development’s Perspec1ve
§ Hundreds of failing or broken tests
§ Test requirements unclear § Exis1ng tests are dubious § Deadline pressure
Paraso& Proprietary and Confiden1al 38 Paraso& Proprietary and Confiden1al 38
Spring Cleaning Solu1on
§ Throw out the useless, old junk § Iden1fy and eliminate “noisy” tests
§ Organize the remaining chores § Priori1ze unit test tasks
§ Keep it clean § Establish and follow a rou1ne test maintenance plan
Paraso& Proprietary and Confiden1al 39 Paraso& Proprietary and Confiden1al 39
Throw out the Junk: Iden1fying Noisy Tests
§ Tests that are fragile § Pass/Fail/Incomplete results frequently change § Frequent changes are required to keep up-‐to-‐date
§ Tests that are outdated § More than “n” releases have occurred despite this test failure § The test applies to legacy code that is frozen § The test has not been executed for “x” period of 1me
§ Tests that are unclear § It takes more than 5 minutes to understand why the test exists § The failure message is incorrect/unhelpful/irrelevant § There is no associated requirement or defect
§ Be merciless!
Paraso& Proprietary and Confiden1al 40 Paraso& Proprietary and Confiden1al 40
Organize Chores: Priori1ze Tasks with Policy
§ All tasks are created equal § Some tasks are more equal than others § Policies establish priori1za1on and assignment
Paraso& Proprietary and Confiden1al 41 Paraso& Proprietary and Confiden1al 41
Keep it Clean: Establish a Rou1ne Process
§ Management and Development need to co-‐define a Policy § Clear tes1ng goals that 1e tests to business needs § Clear delinea1on of responsibility
§ Test crea1on and dele1on § Test failure resolu1on § Naming conven1ons § Peer review
§ Clear and frequent communica1on § Understand the technical and business impact § Enhance traceability to project requirements § Ask ques1ons
§ Focus on safety/quality, not coverage § Keep the number of tests minimal § Run all tests on a regularly scheduled, frequent basis § Clean as you go
Paraso& Proprietary and Confiden1al 42 Paraso& Proprietary and Confiden1al 42
Spring Cleaning Recap
§ Find and delete “noisy” tests § Use policies to priori1ze, assign, and plan § Co-‐design and follow a process to keep it clean
Paraso& Proprietary and Confiden1al 43 Paraso& Proprietary and Confiden1al 43
§ Web § hjp://www.paraso&.com/jsp/resources
§ Blog § hjp://alm.paraso&.com
§ Social § Facebook: hjps://www.facebook.com/paraso&corpora1on
§ Twijer: @Paraso& @MustRead4Dev @CodeCurmudgeon
§ LinkedIn: hjp://www.linkedin.com/company/paraso&
§ Google+: +Paraso& +ArthurHickenCodeCurmudgeon § Google+ Community: Development Tes1ng Because its Bejer
top related