build the right regression suite with behavior-driven testing
DESCRIPTION
Manual functional testing is a slow, tedious, and error prone process. As we continue to incrementally build software, the corresponding regression test suite continues to grow. Rarely is time allotted to consolidate and keep these test cases in sync with the product under development. If these test cases are used as the basis for automation, the resulting suite is composed of very granular tests that are often quite brittle in nature. Using a case study, Anand Bagmar describes how behavior-driven testing (BDT) can be applied to identify the right type of test cases for manual and automated regression testing. Learn how the BDT technique can be applied in your context and domain, regardless of the tools and technologies used in your project and organization.TRANSCRIPT
![Page 1: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/1.jpg)
T14
Test Techniques
5/8/2014 1:30:00 PM
Build the Right Regression Suite
with Behavior-Driven Testing
Presented by:
Anand Bagmar
ThoughtWorks
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
![Page 2: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/2.jpg)
Anand Bagmar
ThoughtWorks
Anand Bagmar is a hands-on and result-oriented software quality evangelist with seventeen years in the IT field. Passionate about shipping quality products, Anand specializes in building automated testing tools, infrastructure, and frameworks. He writes testing-related blogs and has built open-source tools related to software testing-WAAT (Web Analytics Automation Testing Framework), TaaS (for automating integration testing in disparate systems), and TTA (Test Trend Analyzer). Anand is the lead organizer for vodQA, the popular testing conference in India. Follow him on Twitter @BagmarAnand, email him at [email protected], or read his Essence of Testing blog.
![Page 3: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/3.jpg)
Building the “right” regression suite using Behavior Driven Tes5ng
(BDT) Anand Bagmar
So#ware Quality Evangelist Principal Consultant, ThoughtWorks
![Page 4: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/4.jpg)
• “Succeeding with Agile” – Mike Cohn
• MarCn Fowler – Test Pyramid – hFp://marCnfowler.com/bliki/TestPyramid.html
The Test Pyramid
![Page 5: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/5.jpg)
Func%onal Tests (GUI) • Developers & / QA team
Web Service Tests • Developers & / QA team
Integra%on tests • Developers Component tests • Developers
Unit Tests • Developers
Test AutomaCon Pyramid
Manual / Exploratory TesCng
Product under test
Ideal Test Pyramid
Business-‐facing Tests
Technology-‐facing Tests
• Cost • Effort • Time
![Page 6: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/6.jpg)
![Page 7: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/7.jpg)
Test Pyramid reality
Development team Test Pyramid QA team
Test Pyramid
![Page 8: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/8.jpg)
Func%onal Tests (GUI) • Developers & / QA team
Web Service Tests • Developers & / QA team
Integra%on tests • Developers Component tests • Developers
Unit Tests • Developers
Manual / Exploratory TesCng
Test Ice-‐cream cone
Business-‐facing Tests
Technology-‐facing Tests
Test Pyramid – AnC PaFern
Product under test
![Page 9: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/9.jpg)
Func%onal Tests (GUI) • Developers & / QA team
Web Service Tests • Developers & / QA team
Integra%on tests • Developers Component tests • Developers
Unit Tests • Developers
Manual / Exploratory TesCng
Product under test
Ideal Test Pyramid
Business-‐facing Tests
Technology-‐facing Tests
IdenCfy the right type of tests that sit on top of the Test Pyramid
![Page 10: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/10.jpg)
Case Study
• MulCple, long running projects
• Legacy applicaCons
• Integrated
![Page 11: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/11.jpg)
• Limited AutomaCon • Long regression cycle • Huge cost of fixing defects
TesCng Challenges
![Page 12: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/12.jpg)
• No visibility into what is tested • Outdated • BriFle • LiFle / less value • Expensive • Maintenance nightmare
As a result
![Page 13: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/13.jpg)
Biggest problem
• Trust deficit on the team – In exisCng AutomaCon – In manual tesCng
• Finger poinCng / blame game
![Page 14: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/14.jpg)
![Page 15: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/15.jpg)
![Page 16: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/16.jpg)
• Remove the ambiguity • Same visibility for all
![Page 17: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/17.jpg)
EvoluCon of Test Frameworks Behavior Driven
Development (BDD)
OpCmized scripts
Record & Playback
Core Framework
![Page 18: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/18.jpg)
Behavior Driven Development (BDD)
![Page 19: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/19.jpg)
RED
GREEN REFACTOR
![Page 20: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/20.jpg)
Behavior Driven TesCng (BDT)
![Page 21: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/21.jpg)
The Big Picture
• How is this funcConality going to be used by the end-‐user?
• What is their thought process going to be when using this product?
• What is the “core-‐business-‐value” I am delivering to the end-‐user?
![Page 22: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/22.jpg)
Func%onal Tests (GUI) • Developers & / QA team
Web Service Tests • Developers & / QA team
Integra%on tests • Developers Component tests • Developers
Unit Tests • Developers
Manual / Exploratory TesCng
Product under test
Ideal Test Pyramid
Business-‐facing Tests
Technology-‐facing Tests
IdenCfy the right type of tests that sit on top of the Test Pyramid
![Page 23: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/23.jpg)
Expected Func%onality:
The customer should be able to withdraw money from his account via an ATM machine
An example
![Page 24: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/24.jpg)
Given the account has balance of INR 5000 When the customer requests INR 1000 Then the account is debited by INR 1000
BDD Example:
![Page 25: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/25.jpg)
Given the account is in credit
And the dispenser contains cash
When the customer requests cash
Then ensure the account is debited
And ensure cash is dispensed
And ensure the card is returned
BDT Example:
![Page 26: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/26.jpg)
New Expected Func%onality:
The customer should be able to withdraw money from his account only if his card is valid
![Page 27: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/27.jpg)
Given the account is in credit And the card is valid And the dispenser contains cash When the customer requests cash Then ensure the account is debited And ensure cash is dispensed And ensure the card is returned
BDT Example -‐ updated:
![Page 28: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/28.jpg)
Test specificaCon styles
• ImperaCve • DeclaraCve
![Page 29: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/29.jpg)
• Write user flows based on personas, and how they use the business funcConality, or,
• Write business flows, and how different user personas can use that
![Page 30: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/30.jpg)
Another example
For Indigo Airlines website –
• Guest User is able to search for a flight for a single traveller
• Ability to specify contact informaCon for person booking the flight
![Page 31: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/31.jpg)
Search Search Results
Contact Informa%on
![Page 32: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/32.jpg)
ImperaCve style Given I am a guest user on the Indigo home page And I select “round” trip opCon And I select “Pune” from the origin dropdown And I select “Bangalore” from the desCnaCon drop down And I select departure date as “5 December 2013” And I select returning date as “25 December 2013” When I click on Search Then I should see the search results page And I should see at least 1 opCon for my criteria ...
![Page 33: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/33.jpg)
… When I select the “first” opCon Then I am on the Contact InformaCon page When I enter first name as “foo” And I enter last name as “bar” … … And I click the “Select and Con%nue” buFon Then I should be on the next page
![Page 34: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/34.jpg)
Given I am a guest user When I search for flight opCons for a “one-‐way” trip for “1” “Adult” from “Delhi” to “Bangalore” And I select the “first” flight And I enter “valid” contact details for “traveller1” Then I am able to Save and ConCnue
DeclaraCve style
![Page 35: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/35.jpg)
What’s next?
![Page 36: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/36.jpg)
DO NOT AUTOMATE • Low Value, High Cost
• Manual Regression!
2nd candidate for automa%on • High Value, High Cost
Automate ONLY if you have %me • Low Value, Low Cost
• ? Manual Regression?
1st candidate for automa%on • High Value, Low Cost
Cost
Value
![Page 37: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/37.jpg)
Func%onal Tests (GUI) • Developers & / QA team
Web Service Tests • Developers & / QA team
Integra%on tests • Developers Component tests • Developers
Unit Tests • Developers
BDT helps iden5fy the “right” type of regression tests!
Manual / Exploratory TesCng
Product under test
Ideal Test Pyramid
![Page 38: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/38.jpg)
A real world Case Study
![Page 39: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/39.jpg)
![Page 40: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/40.jpg)
How did we accomplish this?
![Page 41: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/41.jpg)
![Page 42: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/42.jpg)
Who is doing the acCon?
What is the business funcConality?
AcCons and verificaCons happen implicitly Depicts a user flow
Given auc%oneer creates a sale And 5 items are added to the sale When auc%oneer starts the sale And auc%oneer sets a starCng bid of 5000 And buyer_1 bids And buyer_2 bids Then auc%oneer Sells the item And buyer_2 wins the item
![Page 43: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/43.jpg)
Final state Tests running on Hudson / Jenkins Since 1st test was automated
Feature files 33
Scenarios 65
Smoke tests execuCon Cme 15 minutes
Regression tests execuCon Cme 45 minutes
Test case repository (manual + automated)
Cucumber .feature files with appropriate tagging
Project Management tool Mingle
![Page 44: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/44.jpg)
?
Incrementally build User Flows
Regress required Product Business
Value
Tests in-‐sync
EffecCve Tests
Remove ambiguity
Understand new
funcConality
Living DocumentaCon
Onboarding
![Page 45: Build the Right Regression Suite with Behavior-Driven Testing](https://reader034.vdocuments.mx/reader034/viewer/2022052619/5550baeab4c905fa618b4e32/html5/thumbnails/45.jpg)
Tools to enable BDD / BDT
• Cucumber • JBehave • SpecFlow • Twist • Concordion