testing tuesdays - capgemini sogeti danmark a/s · 1) the technical test analyst’s tasks in...

1

Testing Tuesdays Technical Tests

Ole Chr. Hansen Managing Consultant

© CapgeminiSogeti 2

Presenter

• Ole Chr. Hansen • Managing Consultant • Fellow – SogetiLABS – Global Innovation Team

Blog - http://ochansen.blogspot.com LinkedIn: www.linkedin.com/in/ochansen Twitter: www.twitter.com/Ole_Chr_Hansen

• ISTQB Expert Level – Improving the Test Process (Assessing) • ISTQB Accredited Trainer in Software Testing • ISEB Practitioner Certificate in Software Testing • ISTQB Foundation Certificate in Software Testing • TMap NEXT® Test Engineer Certified • TPI NEXT® Foundation Certificate • PRINCE2 Foundation Certificate • Certified Scrum Master • Certified Lead Assessor (ISO 9000)

• Sogeti, ATP, Nordea, BRFkredit, WM-data, CRI, LEC • 18+ years in Test Management, 10+ years in Project Management

© CapgeminiSogeti

Introduction – Certification Scheme


ATTA Contents

1) The Technical Test Analyst’s Tasks in Risk-Based

Testing

2) Structure-Based Testing

3) Analytical Techniques

4) Quality Characteristics for Technical

Testing

5) Reviews

6) Test Tools and Automation


Quality Characteristics for Technical Testing

SYSTEM

What the system does

Stakeholder expectations

Stakeholder compares

The closer they match – the higher the quality


General Planning Issues

• Issues to consider when planning non-functional testing

– Stakeholder requirements

– Required tool acquisition and training

– Test environment requirements

– Organizational considerations

– Data security considerations

The Technical Test

Analyst may be

requested by the

Test Manager….

….to identify the

principal risks for

the relevant

quality…..

….to address any

planning issues

associated with the

proposed tests


STAKE-HOLDERS

Customers

Maintenance

Users

Operators

Helps to ensure no important

requirements are missed

Helps to identify risks and priorities

Stakeholder requirements


Required tool acquisition and training

• Tool support often needed

– Particularly for some performance and security testing

• Tools not available

– Need to acquire tool

– Learn to use the tool

• Training time

• Hire tool specialists (external)

– Implement tool

• Pilot project

• Incremental rollout

– Allow time and ressource

• Learning curve


Test environment requirements

Many technical tests require a production-like test

environment in order to provide realistic measures

e.g., security tests, performance tests

DTAP Concept

DEV TEST ACCEPTANCE PRODUCTION


Organizational considerations

• Technical tests may involve measuring the behavior of several

components in a complete system (e.g., servers, databases,

networks)


Data security considerations

• Access to secure systems

– Physical access / system access

– May require pre-authorization, user id and

password, id card etc.

• Specific security measures implemented for a

system should be taken into account at the

test planning stage to ensure that all testing

activities are possible.

– For example, the use of data encryption

may make the creation of test data and

the verification of results difficult.


Efficiency Testing – Sub-characteristics

Efficiency Testing

Performance Testing

Load Testing

Stress Testing

Scalability Testing

Ressource Utilization


Efficiency Testing

Load testing Stress testing

Scalability testing

Ressource utilization

Efficiency testing

Operational profiles

Tools

Environments

Analysis Reports:

Server

Network Database

Application

Test Plan:

Goal Objectives

Risks Tests

Response Times | Break Point | Peak Load | Memory Leaks | Uptime | Downtime


Types of performance testing

Performance testing

Load testing

Stress testing

Scalability testing


Load testing

• Concurrent users or processes

– Handle increasing levels of

anticipated realistic loads

resulting from the transaction

• Average response times for users

– Under different scenarios of

typical use (operational profiles)

– Measured and analyzed

• Supported by tools

– Generate loads


Stress testing

• Handle peak loads at or beyond the limits

– Anticipated or specified workloads

– Reduced availability of resources

• Assessing risk posed by extreme conditions

• Expect performance levels to be degraded

slowly and predictably

– the functional integrity of the system should be

tested

• The ‘weakest link in the chain’ can be

determine

• Important for web applications

– E.g. self service at the Danish tax authorities



Scalability testing

• Ability of a system to meet future

efficiency requirements

– May be beyond those currently required

– System’s ability to grow

• E.g., with more users, larger amounts of

data stored

• More assessment than testing

• Related to capacity planning

– Simulation of a scaled down production

environment



Resource Utilization Testing

• Sub-characteristic of efficiency

– See appendix for an overview

• Tests relating to resource utilization evaluate the usage of system

resources

– usage of memory

– disk capacity

– network bandwidth

– connections

• Compare with benchmark

– Requirements from user and other stakeholders

– Unnatural growth in usage


Resource Utilization Testing

• Memory resource usage is relevant for real-

time embedded systems

– If the memory footprint exceeds the allowed

measure, the system may have insufficient

memory needed to perform its tasks within

the specified time periods

– This may slow down the system or even lead

to a system crash

• Dynamic analysis may also be applied

– Investigating resource utilization and

detecting performance bottlenecks


Trend in test automation approaches

Capture/Playback

Data-driven

Keyword-driven

Each builds on the previous approaches


Objectives of scripting approaches

• To reduce costs

– Make it easier to build automated tests

– Avoid duplicated specs

– Avoid excessive maintenance costs

• Higher degree of reuse of modular scripting

• Higher RoI

– Better testing support

– Better portability

• Environments and hardware

• Enhance testing capabilities

– Achieve more testing for the same effort (or less)

– Testing beyond traditional manual approaches


Example application

Welcome to COMPANY network

Username

Password OK

Cancel


Captured script

Test Analysts Technical Test Analysts

Record/ Edit

Test Case Test Case Test Case Test Case Test Case Test Case Test Case

Record/ Edit

Test Script Test Script Test Script Test Script Test Script Test Script Test Script

Statements and test data

Manually

Test Tool

SUT


Example captured script

• Test case:

1. Open application login to network

2. Enter username and password

3. Click Ok

• Login screen – scripted by capture/playback tool:

– Login.Username.Enter(”ohansen”)

– Login.Password.Enter(”ata2015”)

– Login.OK.Press


PROS

• No initial effort

• Can start automating

quickly

• No programming skills

required

– Some developer

knowledge would be

beneficial

CONS

• Redundant statements for redundant actions – No reuse

• Difficult to understand – May be in an proprietary

language

• Change to the software can cause many scripts to be updated – Higher maintenance costs

Captured script – Pros/Cons


Data-driven


Test Case Test Case Test Case Test Case

Test Script Library

Low-level statements

Manually

Test Tool

SUT

Create

Data Files Data Files Data Files

Control Scripts

High-level statements


Data-driven

• Test data extracted from test cases / test procedures

– Placed into separate data files

• Control scripts reads data from the data files

– One script implements several tests by reading different data

• Reduces maintenance

• Multiple control scripts needed

– Per type of test

• Possible to automate more tests than it would be possible

to run manually

– Costs of automating some tests may be less than the cost of

running them manually


Example data-driven script

Login screen – test script – data-driven:

DataFile = Openfile(”Logindata”)

Read DataFile.LoginRec // reading the first row in the file

For each LoginRec in DataFile

Login.Username.Enter(LoginRec.Username)

Login.Password.Enter(LoginRec.Password)

Login.OK.Press

Verify (MessageBox.Text = LoginRec.Message)

Read DataFile.Login // reading the next row in the file

End Loop

Username Password Message ohansen atta2016 Welcome hbonde atta1963 Unknown user nlangkilde Xyz346 Incorrect password


PROS

• Reduced build cost

– Faster and easier to

automate similar test

procedures

– Test variations using

different test data

• Tests defined by data files

– Separate from scripts

– Test analysts can create

and maintain data files

CONS

• More initial work is needed – Development the control

and shared scripts

• Developer skills required – Create scripts

• Control/Shared

• Data files may not be an intuitive way to specify tests – Layout and form has to be

appropriate for the tester

Data-driven – Pros/Cons


Keyword-driven


Test Case Test Case Test Case Test Case

Test Script Library

Low-level statements

Manually

Test Tool

SUT

Create

Data Files Data Files Test Files

Control Script

Single control script

High-level and test data


Keyword-driven

• Keyword / Action Word

• Single control script

– ‘Interpreter’

• Test files (or test definitions)

– Independent of scripting language

– ‘Language’ tailored to testers business domain and business

processes


Example keyword-driven script

Keyword Username Password Result

LOGIN_KNOWN_USER ohansen atta2016 Welcome

LOGIN_UNKNOWN_USER hbonde atta1963 Unknown user

LOGIN_INCORRECT_PASSWORD nlangkilde Xyz346 Incorrect password

….. …. …. ….


PROS

• Reduced build cost

– Faster and easier to

automate similar test

procedures

– Test variations using

different test data

• Tests files provide more

natural way to specify the

tests

– Can be tailored

CONS

• More initial work is needed – Development the control

and shared scripts

• Developer skills required – Create scripts

• Control/Shared

• Keyword use needs control – Must avoid too many

Keyword-driven – Pros/Cons


ATTA Contents

1) The Technical Test Analyst’s Tasks in Risk-Based

Testing

2) Structure-Based Testing

3) Analytical Techniques

4) Quality Characteristics for Technical

Testing

5) Reviews

6) Test Tools and Automation


Questions?

testing tuesdays - capgemini sogeti danmark a/s · 1) the technical test analyst’s tasks in...

Documents