SE 3800NOTE 5

BEYOND FUNCTIONAL TESTING

Dr. Rob Hasker

Dr. Brad Dennis

ReliabilitySystems Engineering: • Linear System: response to

complex input can be described as a sum of responses to simpler inputs• Eg: Wave propagation• Nonlinear examples: AC power

flow, almost any complex system

• Reliability for linear systems: product of reliability of each component

A: 90% Reliable

B: 90%

C: 90%

Reliability: .93 = .73

Reliability in Software

Car that works 73% of the time: find a shop or dealer

How many components in software?

How can we achieve high reliability for SW?

Review – what is

Integration, system, acceptance testing? What is performance testing?

How do you do it?What can you learn from this type of testing?What can't you learn from this type of testing?

Stress testAnswers what?What do you want to happen?

○ Goal: graceful degradation - bend but don't break

More testing methods

How do you test security? Websites:

Database injection, cross-site scriptingFuzz checkers – inserting random data

How to test usability? In general, how to test non-functional

requirements?Must identify testable component!

Testing strategy Why might we not be able to run every test

every time?How should we prioritize tests?

Regression set minimization: identifying a subset that would reveal most bugsResearch question!Should be able to run all unit tests, majority of

system tests Less frequently: performance, other tests

Maybe these are just run once per day

Qualities of good tests

Thorough – coverageStart with statement coverage

RepeatableGenerate small data sets that have

predictable outputsTest runs should not depend on each other

UnderstandableMaintainableEasy to identify failed case

Testing

What about “grandma testing”?

Testing

What about “grandma testing”?Sexist? Ageist?

"Local grandmother finally uses printer"

CES 2009: hall for products for retirees

Testing

What about “grandma testing”?Sexist? Ageist?

"Local grandmother finally uses printer"

Is grandma-using-the-computer themodern equivalent of racist jokes?

CES 2009: hall for products for retirees

Testing

What about “grandma testing”?Sexist? Ageist?Fuzz testing tools

Testing

What about “grandma testing”?Sexist? Ageist?Fuzz testing tools

What if need large data sets?Test oraclesVersion matching

Testing

What about “grandma testing”?Sexist? Ageist?Fuzz testing tools

What if need large data sets?Test oraclesVersion matching

Key: differentiate between robustness, correctness

System Testing Focus: functional testing

that is, are there errors in features? In some sense, this is the only test that

matters tests all functionality seen by customerwhy shouldn't we do all testing at this level?

○ harder to test deeply○ harder to know how to fix system if test fails

Acceptance testing: system testing w/ focus on common scenarios

Functional/unit testing

How is functional testing different from unit testing?Scope: unit test exercises pieces, functional

is end-to-end How are the the same?

Written by developer organizationLikely to use mocks to isolate components

Acceptance/functional testing

How does acceptance testing differ from functional testing?Functional:

○ Internal, verify featuresAcceptance:

○ External, validate requirements○ Rare to use mocks – only for very low

probability/high impact errors that must be tested○ Production environment (as close as possible)

The Real Differences Lies in What You Are Asking Does this feature meet the customer

requirement?Acceptance Test.

Does this feature work? Functional Test.

Does this feature work with the database?Integration Test.

Does this method do what I want it to do?Unit Test.

What does a manual test look like?Step Req Pass Conditions Pass?

1. Select the Biology program.

UIR-2

System displays biology classes w/ first class BIOLOGY 1150, Section 01, Title GENERAL BIOLOGY, Instructor Block, Anna, Filled/Seats 52/53, Class# 1311, Credits 5, Meets BOE 0221 MWF 8:00-8:52

P / F

2. Double-click on Class# 1330

UIR-1System includes Class# 1330 in schedule at bottom

P / F

3. Scroll down to Class# 1331 (BIOLOGY 1650, Section 01)

UIR-9System displays Class# 1331 with a pink background

P / F

4. UIR-9All sections listed between #1311 and #1331 have a white background

P / F

5. Select the GENENG program.

UIR-2 System displays general engineering courses P / F

6. UIR-9

GENENG 1030 sections 01-07 have a pink backgroundall other sections of GENGENG 1030 have a white background

P / FP / F

7. UIR-9All sections of GENGENG 1000 have a white background

P / F

What does a manual test look like?Step Req Pass Conditions Pass?

1. Select the Biology program.

UIR-2

System displays biology classes w/ first class BIOLOGY 1150, Section 01, Title GENERAL BIOLOGY, Instructor Block, Anna, Filled/Seats 52/53, Class# 1311, Credits 5, Meets BOE 0221 MWF 8:00-8:52

P / F

2. Double-click on Class# 1330

UIR-1System includes Class# 1330 in schedule at bottom

P / F

3. Scroll down to Class# 1331 (BIOLOGY 1650, Section 01)

UIR-9System displays Class# 1331 with a pink background

P / F

4. UIR-9All sections listed between #1311 and #1331 have a white background

P / F

5. Select the GENENG program.

UIR-2 System displays general engineering courses P / F

6. UIR-9

GENENG 1030 sections 01-07 have a pink backgroundall other sections of GENGENG 1030 have a white background

P / FP / F

7. UIR-9All sections of GENGENG 1000 have a white background

P / F

• Lots of detail

• Must document passes

manually

• Maintenance issue when

steps depend on each other

Who should write acceptance tests? Obvious answer: client

Probably manual testsNot much help to development team

Alternative: behavior-driven developmentFunctional, acceptance tests written up frontSimple model: acceptance criteria for each

story = collection of system-level tests

Getting clients to write tests Why can’t clients write our tests?

Barrier: testing language Solution: Cucumber

Testing language: specify start conditions, expected results

Supports tables when have several examples

Developer: provides hooks defining how the tests are executed

Client-written tests Wins:

Test documents what system supposed to do in a verifiable manner!

Expected behavior is very concreteCustomer gets to control behavior from start

ProblemsExtra time to write tests – if client not writing tests,

probably should use other methods○ But then client may not really want system!

Must maintain tests○ But manual tests need maintenance, too

Acceptance tests in

https://cucumber.io/ Supports Ruby, Java, .NET, C++, all

web applications Underlying language: Gherkin

Steps: 1. describe behavior

Feature: AdditionIn order to avoid silly mistakesAs a math-challenged personI want to be told the sum of two numbers

Scenario: Add two numbersGiven I have entered 50 into the calculatorAnd I have entered 70 into the calculatorWhen I press addThen the result should be 120 on the

screen

Steps: 1. describe behavior

Feature: AdditionIn order to avoid silly mistakesAs a math-challenged personI want to be told the sum of two numbers

Scenario: Add two numbersGiven I have entered 50 into the calculatorAnd I have entered 70 into the calculatorWhen I press addThen the result should be 120 on the

screen

These actually all have

the same effect.

2. Step definition in Ruby

Given /I have entered (.*) into the calculator/ do |n| calculator = Calculator.new calculator.push(n.to_i)end

Run and watch it fail:

2. Write code to make it pass

class Calculator def push(n) @args ||= [] @args << n endend

Run see step pass:

3. Repeat until “green like a cuke”

Behavior-driven development Basic model: test-driven development Great for starting, probably not for

regular workEspecially friendly to people who are

learning about syntax

Gherkin Syntax

Each feature described in single .feature file

3 elements:Feature: a name – free form syntaxDescription: documentation

○ Any line not starting w/ a keywordScenario: concrete example

Gherkin Syntax Keywords:

FeatureScenarioGiven, when, then,

and, but: stepsBackgroundScenario outlineExamples

Feature: Refund item

Sales associates required by law to refund purchases

Scenario: Jeff returns a faulty microwave Given Jeff has bought a microwave for $100 And he has a receipt # don’t assume this is true! When he returns the microwave Then Jeff should be refunded $100

Gherkin Syntax Keywords:

FeatureScenarioGiven, when, then,

and, but: stepsBackgroundScenario outlineExamples

Feature: Refund item

Sales associates required by law to refund purchases

Scenario: Jeff returns a faulty microwave Given Jeff has bought a microwave for $100 And he has a receipt # don’t assume this is true! When he returns the microwave Then Jeff should be refunded $100

On same line

Documentation- no keyword, multiline

Can also comment with #

Gherkin Syntax Keywords:

FeatureScenarioGiven, when, then,

and, but: stepsBackgroundScenario outlineExamples

Feature: Refund item

Sales associates required by law to refund purchases

Scenario: Jeff returns a faulty microwave Given Jeff has bought a microwave for $100 And he has a receipt # don’t assume this is true! When he returns the microwave Then Jeff should be refunded $100

• No distinctions between these• Chose for readability• Single when (or break up)• Then: compare actual to expected

Gherkin Syntax Keywords:

FeatureScenarioGiven, when, then,

and, but: stepsBackgroundScenario outlineExamples

Background: Given a $100 microwave was sold on 2015-08-26 And today is 2015-08-26

Steps which are assumed before each scenario

Scenario outline Parameterize values, specify with table:

Scenario Outline: feeding a suckler cow Given the cow weighs <weight> kg When we calculate the feeding requirements Then the energy should be <energy> MJ And the protein should be <protein> kg

Examples: | weight | energy | protein | | 450 | 26500 | 215 | | 500 | 29500 | 245 | | 575 | 31500 | 255 | | 600 | 37000 | 305 |

Scenario outline Parameterize values, specify with table:

Scenario Outline: feeding a suckler cow Given the cow weighs <weight> kg When we calculate the feeding requirements Then the energy should be <energy> MJ And the protein should be <protein> kg

Examples: | weight | energy | protein | | 450 | 26500 | 215 | | 500 | 29500 | 245 | | 575 | 31500 | 255 | | 600 | 37000 | 305 |

One or more examples sections

Required header row

‘|’ is a key symbol

Implementing tests

C++ code implementing this step:

Scenario: Given I have 48 cukes in my belly

Given(“^I have (\\d+) cukes in my belly$”) { REGEX_PARAM(int, cukes); USING_CONTEXT(MyAppCtx, context); std::cout << “Cukes: “ << cukes;}

Implementing tests

C++ code implementing this step:

Scenario: Given I have 48 cukes in my belly

Given(“^I have (\\d+) cukes in my belly$”) { REGEX_PARAM(int, cukes); USING_CONTEXT(MyAppCtx, context); std::cout << “Cukes: “ << cukes;}

Capture groupUsually regular expressiond+: one or more digitsMatches and, then, etc.

Declaration

Boiler plate

Key: regular expressions

See n05-3800-regex.html Flexible tool for capturing words with

specific patterns Will generally use several simple cases:

\d+, \w+^, $: anchor to start and endbackslash to escape special characters

Review

Unit vs. functional vs. acceptance testing

Testing non-functional requirements: performance, security, etc.

Behavior-driven development with Cucumber

Next class period: exercise on writing tests in Cucumber