AT10 Concurrent Session 11/8/2012 3:45 PM 

       

"Specification-by-Example: A Cucumber Implementation"

   

Presented by:

Mary Thorn Deutsche Bank

        

Brought to you by:  

  

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Mary Thorn Deutsche Bank

VP of QA at Deutsche Bank Global Technologies in North Carolina, Mary Thorn has a broad testing background that spans automation, data warehouses, and web-based systems in a wide variety of technologies and testing techniques. During her more than fifteen years of experience in healthcare, HR, and SaaS-based products, Mary has held manager and contributor level positions in software development organizations. She has a strong interest in agile testing methodologies and direct experience leading agile teams through Scrum adoption and beyond. Mary is a Certified ScrumMaster and an active participant in Triangle Institute of Software Quality Assurance.

.

 

1

Mary Thorn

QA Manager at Deutsche Bank.

Over 15 years of QA experience working in the following areas: Healthcare HR Agriculture Financial and SaaSareas: Healthcare, HR, Agriculture, Financial and SaaS-based products.

Have a broad testing background that spans automation, data warehouses to web-based systems in a wide variety of technologies and testing techniques.

Heavy interest in Agile Testing Methodologies and directHeavy interest in Agile Testing Methodologies and direct experience leading Agile teams in the SCRUM Software Development Lifecycle (SDLC).

2

• Introduction

• The Communication Problem

• What are Acceptance Tests?

• Simple Example

• Acceptance Testing with Cucumber

• Feature FilesFeature Files

• Wrap up

Goal: Presenter would like to have a successful presentation

Acceptance Criteria: 1. Verify the attendees stay awake2. Verify the attendees are happy with the presentation

3

The Challenge:Today, the most difficult problem in software development is knowing what to build, not howto build it.

Knowing what to build, requires knowing why it is being built. In other words, understanding the goal.

The Problem:Stories typically concentrate on the what and the how, not the why. Goals and examples are usually missing.

Stories are usually a list of imperative requirements (acceptance criteria).

Imperative requirements are usually interpretedImperative requirements are usually interpreted in subtly different ways by each person. If goals and examples are absent, misinterpretation is much more likely.

4

Example:Requirement: Draw a star with 10 points.

OR

Solution: Specification by ExamplePeople understand requirements best usingPeople understand requirements best using concrete examples.

Examples clear up ambiguity and misunderstandings.

Examples expose missing requirements.

Examples force everyone to think harder about a problem.

5

Specification by ExampleThis is the basis of acceptance testing andThis is the basis of acceptance testing and Behavior Driven Design (BDD).

It is critical that everyone participates in creating the examples so that the understanding is shared.

Ultimately, acceptance testing is less about the example itself and more about the conversation required to create the example.

What does this have to do with automated

testing?

6

In a nutshell, automated t t tacceptance tests are

executable examples that define the system's

specificationsspecifications.

Acceptance Tests areFormal examples that define requirementsFormal examples that define requirements.

Executable.

Self-explanatory and readable by all team members.

Tests of business rulesTests of business rules.

A living specification of system behavior.

7

Acceptance Tests do NOTTest pieces of code in isolation They are not aTest pieces of code in isolation. They are not a substitute for unit and integration tests.

Verify the entire system.

Verify all permutations of a piece of functionality.

Use mock objects except to remove aUse mock objects except to remove a dependency on an external 3rd party.

Benefits:Higher quality b/c everyone shares the same understanding of the requirements and builds it right the first time.

Tests become the system's regression suite.

Create an objective verification of “done-ness” for a story. A story is done when all tests pass.

Create transparency into progress on a story.p y p g y

Manual testers are part of the automation process.

Allows for more exploratory testing b/c happy path is automated

8

Acceptance Criteria: Transactions are rounded to the nearest cent.

Original Value Rounded Value$0.021$0 025

$0.02

$$0.025$0.029

$0.02$0.02

Why was there confusion?

The goal was missing...

Goal: Create a currency conversion system.

The Concrete Example clarified what was meant by the acceptance criteriaby the acceptance criteria.Refined Acceptance Criteria: Transactions are always rounded down to the nearest cent.

9

This simple example is a REAL EXAMPLE in a system that was well-tested and approved by all y pp yparties.

Attacker was able to steal > $15,000 starting with a single cent using this process...

$0.01 → 0.0051 € rounded to 0.01 €

0.01 € → $0.0196 rounded to $0.02

A seemingly minor ambiguity in the requirements was very expensive.

Concrete examples drive understanding and consensus.

To create good representative examples, everyone must understand the goal of the story.

For technical people to properly understand the goal, they must understand the business domaindomain.

10

What does at doesCucumber have to

do with this?

• Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language and serves as documentation of requirements, automated regression tests and development-aid - all rolled into one format.

• Cucumber works with Ruby, Java, .NET, Flex y, , ,or web applications written in any language. It has been translated to over 40 spoken languages.

11

Components:

• Cucumber (Framework)• HTMLUnit (Browser Emulator)• Selinium/Watir (Browser Emulator API)• Features (Tests written in sentence format)• Step Definitions (Backend code to support

Features)Features)

Command line –Rake TAGS -@abc

Cucumber Framework - Tags

Feature File -Sentence

Step Definition File –Matching Sentence

Terminal Browser Emulator Selinium/Watir –Terminal Result Output-

Browser Emulator Selinium/Watir Execute Command

12

Process:• Product Owner writes acceptance tests in p

feature file scenarios format in the story. Worst case, they at least start the Scenario:

• During grooming the team discusses the scenarios and asks questions. Team can add/remove scenarios during grooming or team updates feature file afterteam updates feature file after.

• Once Sprint has started QA and Dev work on getting the feature file 90% complete before the developer starts coding.

Process:• The developers code the scenarios in theThe developers code the scenarios in the

feature file in a TDD manner.• QA expands the scenarios in the feature files

after development if there are new findings.• When testing is complete PO acceptance is

done by running the acceptance tests.

13

Two Components:• The Feature files which contain the test• The Feature files which contain the test

Scenarios.

• The Step Definition files which contain the Ruby “glue” code that connects the tests to the application.

Feature File Overview:Written in plain textWritten in plain text

Define the test suite and contain the actual tests which are called Scenarios

Written collaboratively by everyone on the team (PO, QA, and Dev)

QA can run these manually.

14

Feature:Suite is defined by keyword “Feature:” followedSuite is defined by keyword Feature: followed by a description. This states the goal for the Feature File. When the Feature File is complete, a tag is included at this level so that all of the test Scenarios contained within the file can be run from a single command.g

Feature: This tests the currency conversion system

Background:“Background:” is run before each test so thatBackground: is run before each test so that you don’t have to repeat steps. Equal to a setup step.

Background:Given I login with the test accountAnd I navigate to the Transaction ScreenAnd I navigate to the Transaction ScreenThen I should see the Transaction logo

15

Scenario:Individual tests are defined by “Scenario:”Individual tests are defined by Scenario:followed by a unique name. Scenarios are concrete examples of how we want the software to behave. The first line states the goal of the given Scenario, while every line thereafter lists the steps needed to complete the test.p p

Scenario: Verify Transactions are rounded to the nearest cent.

Scenario Outline:“Scenario Outline:” is used for data driven testsScenario Outline: is used for data driven tests.

Scenario Outline: Verify Transactions are rounded to the nearest cent.

Given I am on the order screenWhen I make a <transaction> of valuesThen the <values> are rounded

Examples:|t ti | l ||transaction|values||.021|.02|

|.025|.02|

|.029|.02|

16

Keywords:• Tests have reserved words that begin each line:Tests have reserved words that begin each line:

Given, When, Then, And, But.• Use Given to state a precondition• Use When & And when taking an action• Use Then when asserting a validation

Scenario Outline: Verify Transactions are rounded to the nearest cent.

Given I am on the order screenWhen I make a transaction of 9.025And save the transactionThen the value should equal 9.02

Tags:• Tags a powerful tool to organize tests and• Tags a powerful tool to organize tests and

scenarios. A Scenario or feature can have as many tags as you like. Just separate them with spaces.

• A note to the wise: Keep it simple. @tag @smoke_test @component

17

@CucumberpresentationFeature: User would like to have a successful presentationTests the following acceptance criteria: 1. Verify the attendees are awake2 Verify the attendees are happy with the presentation2. Verify the attendees are happy with the presentation

@AttendeeawakeScenario: Verify the attendees are awake

Given that I am an attendeeWhen I have drunk caffeineThen I will be able to listen to the presenter

@AttendeeishappyS i V if Att d hScenario: Verify Attendees are happy

Given I am the presenterWhen I talk slowAnd my content is interestingThen the attendees will be happy

Cucumber is a functional test automation tool for lean and agile teams. It supports behavior-driven development, specification by example and agile acceptance testing. p g p gYou can use it to automate functional validation in a form that is easily readable and understandable to business users, developers and testers.It helps teams create executable specifications, that are a goal for development, acceptance criteria and functional regression checks for future changes.Allows teams to create living documentation, a singleAllows teams to create living documentation, a single authoritative source of information on system functionality that is always up-to-date.Allows manual testers to be involved in automation, as well as, frees them up from testing the “happy path” to be able to real exploratory testing.

18

QUESTIONS?