test requirements for sqa

Test Requirements:The Basis of Testing

Capital Terefe

Capital Terefe 1

AgendaDefining Test Requirements (TR)

What, Why, Where

Fitting TR’s into the testing pictureWhat’s within our testing processGenerating TR’s

Organizing & Decomposing TR’s

Test Requirements Hierarchy Samples

Setting the stage for measuring test coverage

Capital Terefe 2

Background“Test Requirements Hierarchy” is a term

coined from Rational’s SQA Team Test software.

The principle of identifying, organizing, and measuring test requirements is universal to many test processes and methodologies

Much of this in-service is distilled from:Rational methodologies (we are an SQA Team Test

house after all)QAI Workbench modelSeminar topics from

19th annual International Software Testing Conference Star ‘98 West Conference

Ed Kit’s “Software Testing in the Real World”

Capital Terefe 3

Test Requirements

What exactly is a Test Requirement?

Why identify Test Requirements?

Where does a Test Requirement come from?

Capital Terefe 4Defining TR’s: What, Why, Where

What exactly is a Test Requirement?

Identifies the “WHAT” of testing What needs to be tested, AND What are you going to validate about it

Includes both normal and error conditions

Covers business rules, functionality, non-functional standards

Do NOT have case specific data values assigned to them yet (data appears in test cases, the “How” of testing) examples…


Example 1: Testing the inserting of a record to a table

“Validate that you can insert an entry”

“Validate that insertion fails if entry already present”

“Validate that insertion fails if table already full”

“Validate that you can insert an entry to an empty table (initial)”

These are test requirements NOT tests because they do not describe the data element being inserted

The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements

“Validate you can insert ‘John Doe’” is a test case not a test requirement

Capital Terefe 6

Test Requirements Identified (among others):

Defining TR’s: What, Why, Where

Why identify Test Requirements? It’s what QC “owns” in our workbench:

Requirements-based or Function-based testing

It’s the basis for establishing the completion of testing

Helps determine the scale of the testing effort

Governs the types of resources you will need

Serves to identify automation strategies you can use

Becomes a roadmap for your testing effort

Can be a tool for leverage and dialog with developers and business analysts

Dev Team can sign off on them (Verification!)Capital Terefe 7

Defining TR’s: What, Why, Where

Where does a TR come from?

Traditional: Business Requirements, functionality, internal logic… Marketing specs, Functional specs, Technical specs

Reality: “Interview Analysis”, Non-Functional Checklists

(standards & compliance), Use Cases (from business scenarios and users), Discovery during testing, any other deliverables from previous workbenches (diagrams, modeling, flowcharts, etc.)


How do Test Requirements

relate to the Test Plan?Traditionally, the Test Plan has represented

“what” was going to be tested, even including test cases.

Paradigm is shifting: Test Plan should relate what your testing process (and deliverables) will be for a particular project.

A Test Plan should build confidence in the testing process for a project, making approaches clear.

A Test Plan can include the Test Requirements

However, if the Test Requirements are too lengthy, they should become their own document: A Test Requirements HierarchyCapital Terefe 9Defining TR’s: What, Why, Where

AgendaDefining TR’s: What, Why, Where





Capital Terefe 10

Drilling down: Where test requirements fit into the

picture

Capital Terefe 11

BusinessRequirement

TestRequirement

TestScenarios/Cases

TestProcedure/Script

Fitting TR’s into the testing picture

Generates

1

M

Generates

1

M

Executes/Runs

1

M

Drilling Down

Capital Terefe 12

BusinessRequirement

TestRequirement

TestScenarios/Cases



First, Let’s look

at this relationship:

What’s within our

testing process

Then we’ll look

at this relationship:

Gernerating TR’s

from what feeds into

our testing process

ATM Example: Practice Writing Test Requirements

Business Requirements:

- “ATM must do withdrawals”

- “Withdrawals are between $20-$300”

- “Withdrawals are in $20 multiples”

Group Exercise!

1. Limit the scope to these 3 requirements.

2. What will you validate (test for)?

3. Are there any implied requirements that may not be written out?

Capital Terefe 13What’s within our testing process

Example 2: Testing Withdrawals on an ATM

“Validate that a withdrawal option is available”

"Validate that a withdrawal of a multiple of $20, between $20-$300 can be done"

"Validate that <$20 is not allowed"

"Validate that >$300 is not allowed"

"Validate that $20 multiples >$300 is not allowed"

"Validate that non-$20 multiples between $20-$300 not allowed"

"Validate strange numeric amounts/combinations not allowed (all zero's, all 9's, 20.0000)"

“Validate that the withdrawal received is equal to the amount requested”

"Validate that a valid withdrawal amount must be below the account balance”

These are test requirements NOT tests because they do not describe the data element being used (like $20, $40, $60, $1)

The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements

Capital Terefe 14

Test Requirements Identified (among others):

What’s within our testing process

Test Scenarios/Cases for - “Validate that a withdrawal of a multiple of $20,

between $20-$300 can be done”

Capital Terefe

15

Case # P/F $ entered ExpectedResults

ActualResults

WD01 Pass 20 $20 withdrawnWD02 Pass 40 $40 withdrawnWD03 Pass 60 $60 withdrawnWD04 Pass 80 $80 withdrawn

WD05 Pass 100 $100 withdrawn: : : :

WD13 Pass 260 $260 withdrawnWD14 Pass 280 $280 withdrawnWD15 Pass 300 $300 withdrawn


Test Procedure & Script for previous example

Step 1: Insert Card

Step 2: Enter PIN

Step 3: Select Withdraw option

Step 4: Enter dollar amount

Step 5: Validate amount received

Do until EOF ‘until end of data file

Input data record

Senddata CARDINFO to “Cardfield”

Senddata “Enter”

Senddata PIN to “PINFfield”


Senddata “W ” to “SelectionField”

Senddata AMOUNT to “DollarField”


If ErrorMsg > 0 then print ErrorMsg

Print “DollarAMTgiven”

Loop

Capital Terefe 16

Procedure: Script: (in pseudo-code )


ThinkThinkManual !

Manual !ThinkThink

Automated

Automated

!!






Capital Terefe 17

The Workbench Concept

Capital Terefe 18

DODO Check

Standards

Tools

Rework

Entrance

CriteriaExit Criteria

Product Input

Product Output

Generating TR’s

Our workbench is called “Generating Test Requirements”

Entrance Criteria for Business Requirements to generate Test

Requirements

Visible ? Clear? (unambiguous) Complete? Consistent? (conflicting requirements must be prioritized) Reasonable? (achievable) Measurable? (quantifiable) Modifiable? (will it change or is it stable?) Traceable? (the source is known) Dependent requirements identified? Testable? (given current environment, resources, skills)

Capital Terefe 19Generating TR’s

Exit Criteria for Test Requirements

Can another tester create test cases/scenarios from these? Does a Test Requirement specify what is being tested and what

about it we are validating? (Clear?) Are the Test Requirements…

- Complete?

- Consistent? (conflicting requirements must be prioritized)

- Reasonable? (achievable)

- Measurable? (quantifiable for measuring test coverage)

- Modifiable? (will it change or is it stable?)

- Traceable? (the source is known)

- Testable? (given current environment, resources, skills)

Do the test requirements cover the complete scope of the project?

Are all the test requirements verified and signed off by the Development Team?


When creating Test Requirements (“Do”)...

Use “action” verbs & words- “Validate that…”- “Check for…”- “Test that…”

Trace them back to the source

Remember that different applications arrange in different ways

- Think of MF, batch, C/S, web, e-commerce, GUI, etc.- Use “testing considerations” checklists that generally cover

what kinds of things should be considered when testing your specific situation

Make your Test Requirements document a “living document”


Also...Maintain a level of balance between too much & too

little- Too High level: won’t be useful, vague, can’t generate test

cases from it.

- Too low level: Over-process, over documentation, no productivity

- General rule: 5-7 levels deep in an outline format

Organize them!- Outline/Hierarchy format recommended

- Look at how the BA breaks down the project into areas

- Look at how the PA breaks down the project into areas

- Organize by functional areas

- Organize by “types” of testing (Function vs. System vs. Non-Functional)







Capital Terefe 23

Distinguishing the types of testing….

I. Function-Based Tests

II. User Interface Tests

III. Security Tests

IV. Installation Tests

V. Configuration Tests

VI. Performance Tests (Response)

VII. Load Tests (simultaneous users, lots of small transactions)

VIII. Volume Tests (Big transactions)

Capital Terefe 24

IX. Stress Tests (breaking point: memory, resources)

X. Resource Usage Tests

XI. Documentation Tests

XII. Compatibility Tests

XIII. Recovery Tests

XIV. Serviceability Tests

and others…

*III - XIV are all “Systems-based tests”

Organizing TR’s

Organizing by Functional areas….

Most testers perform function-based, or requirements-based tests

- Tests on functionality that the system will provide

- Usually scenario/case based- includes normal and alternate

(invalid) cases

Capital Terefe 25Organizing TR’s

Organizing by Functional areas….

Most testers also perform User Interface Style Tests

- These are generally separate from the functionality that the software will provide

- Usually encompass the architectural standards & compliance (like Windows Design Standards)

- Also includes tests of navigation, menus, admin functions (like printing, saving)

Capital Terefe 26Organizing TR’s






Capital Terefe 27

Remember this?…Drilling down

BusinessR equirem ent

TestRequirement

TestScenarios/Cases


Capital Terefe 28Fitting TR’s into the testing picture

Decomposing: Drilling down within a Test Requirement

Capital Terefe 29

BusinessRequirement

TestRequirement

TestScenarios/Cases



Keep the

function-based

perspective

in mind!

Business Function

Tasks within theFunction

Data Entry Typesfor transactions

Transactions toperform a task

Field Validation

Test Requirement Decomposition

Capital Terefe 30

Business Function

Tasks with in theFunction

D ata Entry Typesfor transactions


F ie ld Validation

The "what" that the software is in tended todo, like "Rent a Car to a custom er"

The tasks involved: 1) query car rates,2) check custom er policy coverages, ....

The database transactions necessaryto perform the task.

The edits on the data included w ith inthe d ifferent types

The types of data entry thetransaction can process: Add,

Change, Delete, Inquire

Decomposing TR’s


Capital Terefe 31

BusinessFunction

Tasks within theFunction

Data Entry Typesfor transactions


Field Validation

High level Functional Areas:

usually from

“Functional Spec” type

docs, or BA work

Lower level

Functional Areas:

usually from

“Technical Spec” type

docs regarding

internal logic,

or PA workDecomposing TR’s


You can start on the high level functional areas early into the project & build in lower level areas as they become available

Any level can contain a test requirement which can also be made up of (or broken down into) lower level test requirements

Capital Terefe 32Decomposing TR’s

Business Function LevelTry to identify “groups” of functions or

functions connected by similar themesfile management functions, printing functions, help

functions, car rental functions, reservation functions, ticket purchase functions, claim reporting functions

Be sure all areas of the system are covered. If something is left out or doesn’t fit into a group, it becomes its own group.

It may be easier to identify functional areas by “window” instead of by function.

Capital Terefe 33Decomposing TR’sFunction Task TransactionTrans Data

Type

Field

Validation

Business Function LevelAt this level, the idea is seeing the

connections, integration, and interactions of the system’s functionality.

May not necessarily be identifying a test requirement at this level as much as just identifying the functional area.

Capital Terefe 34Decomposing TR’sFunction Task TransactionTrans Data

Type

Field

Validation

Task LevelBreak down each Function area into the tasks

within the function

For complex tasks, it is possible to break them down further into sub-tasks

Some Business Functions can not decompose into further tasks (example: Check Writing function)

Capital Terefe 35Decomposing TR’sFunctio

nTask Transaction

Trans Data

Type

Field

Validation

Transaction LevelFrom this level down, we start to address the

internal things that occur to make the functions and tasks happen

Identify any logical transactions that ties the task to the database or any other transactions necessary to perform the task.

Identify any data processing, calculation, data formatting type transactions

Note: A screen or window may cause the execution of several different logical transactions


n Task Transaction Trans Data

Type

Field

Validation

Transaction Data Type Level

Identify which of the four types the transaction can become: Add, Change, Delete, Inquire

It is entirely possible that a transaction can be multiple types.

If a transaction is only one type, you can wrap this level up into the higher level.


n Task Transaction

Trans Data

TypeField

Validation

Field Validation LevelMost testers like to jump directly to this level.

It’s the most obvious at times.

Field validation covers all edits & cross edits on fields and data.

Be careful of the detail you document at this level. Remember the separation between test requirement and test case.


n Task TransactionTrans Data

Type

Field

Validation

Field Validation LevelNot all test requirements (especially at this level)

fit well to be documented in this manner.- Example: Testing all the stated properties of windows

objects

- Use “Validate that the properties of all the objects in this window match the properties listed on the Object Properties Reference Table in Appendix B upon initialization of the window”

- Don’t list each property check as a separate test requirement if it can be summarize under one test requirement

- This is a judgement call YOU make for your given project.


n Task TransactionTrans Data

Type

Field

Validation

Example 3: Rental Car Application1. Validate that a Rental can occur.

1.1 Check Customer policy coverage

1.2 Query Car availability

1.3 Query Car rates

1.4 Open a Rental ticket

1.4.1 Validate that a customer record can be entered

1.4.2 Validate that credit card approval is obtained

1.4.3 Validate that status on the car record is changed

from “waiting” to “rented”

2. Billing Function

3. Reservation FunctionCapital Terefe 40

Let’s look at

the lower levels

for this one

Decomposing TR’s

Then we’ll try it

on this one



1.4.1 Validate that a customer record can be entered

1.4.1.1 Validate that a new customer can be added to the customer table

1.4.1.1.1 Validate that the first name is all alpha

1.4.1.1.2 Validate that the age is > 21.

1.4.1.1.3 Validate that the phone number is numeric

1.4.1.1.4 Validate area code is an existing area code number.

1.4.1.2 Validate changing an existing customer





…fill in the lower level test requirements!

First, Identify any sub-areas (further tasks, or even

separate transactions within this)

Then, Identify the lowest level field validation test

requirements (think about what is typically

involved with credit card authorizations)


What did you come up with?1. Validate that a Rental can occur.



_________________________________________

_________________________________________

_________________________________________

_________________________________________

_________________________________________

_________________________________________

_________________________________________

_________________________________________


Possible Test Requirements...1. Validate that a Rental can occur.



1.4.2.1 Validate the expiration date is a valid future date

1.4.2.2 Validate the expiration date is not within 1 month of

expiring.

1.4.2.3 Validate that the CC# is 12 digits

1.4.2.4 Validate that the $ amount is <= credit balance available

1.4.2.5 Validate that an authorization # is received.Capital Terefe 44Decomposing TR’s

Function

Task Transaction






Capital Terefe 45

TRH Samples

Let’s look at a few examples of how test requirements can be organized:

- Sample 1: Excerpt from a Personal Finance app

- Sample 2: actually organizing Function requirements, but good representative of the concept

- Sample 3: QBS, from Rational’s sample project (adds types of testing into the hierarchy)

- Sample 4: another view of the Personal Finance app

- Sample 5: A mainframe batch system

Capital Terefe 46

Test Coverage MeasuresTest Requirements are the “what” of testing &

are the basis for establishing the completion of testing

TR’s give us the point of measurement for test coverage

Each TR should receive a Priority, Risk, and Weight

Each TR should be tracked for Verification () and Validation (%)

Capital Terefe 47Test Coverage Measures

Summary & RecapDefined: What, Why, Where

Described: How TR’s “fit” into the big picture

Decomposed: Organizing TRs & Generate TR’s into a measurable format

Demonstrated: Sampling some TRH

Determined: The Measurement Connection

Capital Terefe 48

test requirements for sqa

Software

test plan

test cases

test requirements tr

andmeasuring test requirements

test requirementcome

test coveragecapital

test requirementcapital

testing withdrawals