Setting up and Managing a Test LabSetting up and Managing a Test Lab

Ruth KeysRuth KeysIndependent Test ConsultantIndependent Test Consultantwww.keyswww.keys--consult.deconsult.deruth@keysruth@keys--consult.deconsult.de

01.12.2004 © Ruth Keys 2004 2

What is a test lab?

“A test laboratory is a place where testing is conducted.”

“Testers must perform controlled experiments, dealing with measurements and known quantities.”

Rex Black, “Managing the Testing Process”

01.12.2004 © Ruth Keys 2004 3

The 5 Ws and an H

01.12.2004 © Ruth Keys 2004 4

First be clear on your goals!

Answer these questions:

WHO will be doing the testing?using the test lab?

WHAT will they be testing?type of testing will be performed?

WHEN will the testing be done?does the test lab need to be available?

WHY do you need a test lab?

01.12.2004 © Ruth Keys 2004 5

WHERE? Make a decision as soon as possible

Are there special requirements which need a dedicated environment?

• Particular hardware requirements?• Particular software requirements?• Security issues?

Are there special requirements which need a dedicated environment?

• Particular hardware requirements?• Particular software requirements?• Security issues?

Virtual or physical?Virtual or physical?

01.12.2004 © Ruth Keys 2004 6

WHAT?What facilities and resources are needed for a dedicated test lab?

01.12.2004 © Ruth Keys 2004 7

WHAT?What facilities and resources are needed for a dedicated test lab?

SpaceOffice furnitureElectric sockets, network connectionsAir conditioning / lightingComputer hardwareOffice resources/stationeryReference materialDoor lock!

01.12.2004 © Ruth Keys 2004 8

WHAT? Establish controlled environment by defining software requirements

Baseline installation reflecting production environment on PC

Operating systemOther applicationsMiddle ware

Test toolsCapture/ReplayTest scriptsBug tracking toolTest management tool

Connection to development environmentTest user accounts

01.12.2004 © Ruth Keys 2004 9

WHAT? Define the requirements for the Application Under Test (AUT)

How many installations of the AUT are required?

Separate performance testing from functional testingSeparate different testing groups

Functional acceptance testersUser acceptance testersOperations acceptance testers

More that one release under test in parallelCan the installation access different databases?

01.12.2004 © Ruth Keys 2004 10

HOW? Well defined processes help maintain a controlled environment

Have processes in place for

Installing a new release of the AUT

Upgrading the baseline installation

Applying patches to the installation

Test data load and maintenance

Backing up the testware

Booking time and resources

01.12.2004 © Ruth Keys 2004 11

HOW? Ensure important information is available to everyone

DocumentationTest strategyProcess descriptionsTest handbookAUT handbook

Check lists (with tick boxes!)How to install a new release of the AUTHow to save a test data setHow to load a test data set

Visualize informationWhiteboard in test lab with „vital“ informationFlip charts, posters, plans on wallsPost-its on PCswhich installation, which operating system etc.

01.12.2004 © Ruth Keys 2004 12

Better Data, Better Testing* Definition of test data

Environmental data*tells the system about its technical environment

Setup data*tells the system about the business rules

Input dataPrimary input dataData directly entered during test executionSecondary input dataData that exists in the system before a test is executed and that affects or is affected by and is a prerequisite for the test execution.

* James Lyndsay, EuroSTAR 2002 Tutorial

01.12.2004 © Ruth Keys 2004 13

HOW? Managing test data is important for reliable test results

How to divide up the test data?Range of values (logical data sets)Data set per testerPartitioning (safe area, change area, scratch area*)

Where to store the data?Directory structuresIn the test environment or outside

How to store the data?Using import/export functions of the databaseInput format for utilitiesNeutral format

* James Lyndsay, EuroSTAR 2002 Tutorial

01.12.2004 © Ruth Keys 2004 14

One solution

Client / server application for a service provider in the travel industry:

Started off as a test lab for

System Test phaseFunctional acceptanceUsabilityPerformance

for one project

Ended up as test lab for

the whole organizationclient/server applicationsfunctional acceptanceuser acceptanceoperations acceptance

01.12.2004 © Ruth Keys 2004 15

WHO tested WHAT?

System test teamFunctional acceptance testsUsability testsPerformance tests

User acceptance test teamBusiness processesIntegrated business suite

Operations acceptance test teamOriginally virtual test labThen own test labLater combined

01.12.2004 © Ruth Keys 2004 16

With WHAT resources?

Virtual test lab without middlewarePhysical test lab with middleware

6 PCs with interchangeable hard disks2 gateways (X25 and X31)1 PC as repository for testware with DAT drive1 PC as “monitor” for serverIsolated section of LAN

Baseline installation as Ghost imagesOperating systemMS OfficeTest tools

01.12.2004 © Ruth Keys 2004 17

HOW did we organize the secondary input data?

Started with one database loaded with test data using a utility provided by the developers

Divided data between the testers by value rangeTests which produced global changes were given exclusive time slots

Improved quality and quantity with migration programNeutral format of input dataEasy to modify and generate volume

Moved to multiple databases / installationsOne per test groupPossible to switch databases within an installation

Finally had scripts to save/load test data sets into the databaseProvided by the OAT test team

01.12.2004 © Ruth Keys 2004 18

Be creative about tapping outside resources

Talk to

Developers Do they have tools, scripts, data you can use?

Administrators Get them to help with automating import/export of test data, restore/recovery of testware

PC Support Groups Often have automated routines to set up baseline PC installation

Operations / OAT group

Will they administer the test lab too?Or at least offer advice!

01.12.2004 © Ruth Keys 2004 19

Lessons learned

Avoid planning paralysis

Start small and evolve with time and experience

Plan, Do, Check, Act

Do

Plan

Check

Act

01.12.2004 © Ruth Keys 2004 20

Aphorism

Deal with the difficult,While it is still easy.Solve large problemsWhen they are still small.Preventing large problemsBy taking small stepsIs easier than solving them.By small actionsGreat things are accomplished.

Lao Tzu