automated client testing within saxess · pdf fileto automate the testing, ... primary system...

NADA

Numerisk analys och datalogi Department of Numerical AnalysisKungl Tekniska Högskolan and Computer Science100 44 STOCKHOLM Royal Institute of Technology

SE-100 44 Stockholm, SWEDEN

Automated Client Testing within Saxess

Sara Néri

TRITA-NA-E02104

Master’s Thesis in Computer Science (20 credits)at the School of Computer Science and Engineering,

Royal Institute of Technology,Supervisor at Nada was Henrik Eriksson

Examiner was Stefan Arnborg

AbstractSaxess is an exchange trading system developed by OM Technology, used by amongothers the Stockholm Stock Exchange. It was launched in 1999 but new versionsare being developed gradually to match requirements from customers. The Admin-istrative Client within Saxess is an interface towards the Saxess database, used toadminister all static data used in a Saxess marketplace.

During the continued development of Saxess, testing of the system is important.Previously, all the clients within Saxess have been tested manually, but lately thetests for one of the clients (Saxess Trade) have been successfully automated. Thereis now an aim to extend the automated testing to comprise the other clients too, tomake the testing of them more effective.

The goal for this project was to create and start up automatic tests for theAdministrative Client. The aim for the future is to use these tests as a complementto the manual tests, for regression testing and retesting of functions after faultcorrection. Three different categories of tests were to be implemented during theproject; Windows testing, Data testing and Savings testing.

Before the testing started, an analysis of QARun, a testing tool from Compuware(used for testing Saxess Trade), was done to evaluate if it was suitable for testingthe Administrative Client. The result of this analysis was that QARun seemed tobe suitable for the project.

A number of test cases from each test category were planned, focusing on threewindows within the Administrative Client; “Submarkets”, “Orderbooks” and “Parti-cipants”. The test cases have been implemented in test scripts and specified in atest specification document.

All planned tests have been possible to implement in QARun. Having this asmain argument we can draw the conclusion that QARun is well suited for supple-mentary testing of the Administrative Client, making regression tests and retestingfunctions after fault correction.

Automatiserad klienttestning inom börshandelssystemetSaxess

Examensarbete

SammanfattningSaxess är ett börshandelssystem utvecklat av OM Technology som används av blandannat Stockholmsbörsen. Det lanserades 1999, men nya versioner utvecklas fortskri-dande för att anpassa systemet enligt olika kunders krav. Den administrativa klienten(AD:n) inom Saxess är ett gränssnitt mot Saxess-databasen, som används för attadministrera alla statiska data som finns på en marknadsplats i Saxess.

Under den fortlöpande utvecklingen av Saxess är det viktigt att testa systemet.Tidigare har alla klienter inom Saxess testats manuellt, men på senare tid har tes-terna för en av klienterna (Saxess Trade) blivit automatiserade, vilket varit lyckat.Det finns nu en målsättning att utöka den automatiserade testningen till att äveninnefatta de andra klienterna, för att effektivisera testningen av dem.

Målet med detta projekt var att skapa och starta upp automatiserade testerför den administrativa klienten, AD:n. Syftet är att använda dessa tester som ettkomplement till de manuella testerna, för regressionstestning och vid omtestningav funktioner efter felrättning. Tre kategorier av tester skulle implementeras underprojektets gång, Windows-testning, Data-testning och Savings-testning.

Innan man började testa gjordes en analys av QARun, ett testverktyg från Com-puware (använt vid testningen av Saxess Trade), för att utreda om detta skulle varalämpligt att använda vid testningen av AD:n. Resultatet av analysen var att QARunverkade lämpligt för projektet.

Ett antal testfall från varje testkategori planerades och fokus lades på de treAD-fönstren ”Submarkets”, ”Orderbooks” och ”Participants”. Testfallen har imple-menterats i testskript och specificerats i ett testspecifikationsdokument.

Alla tester som planerats har kunnat implementeras i QARun. Med detta somhuvudsakliga argument drar vi slutsatsen att QARun är väl lämpat för komplette-rande testning av AD:n, vid regressionstestning och omtestning av funktioner efterfelrättning.

Acknowledgements

First, I wish to thank Joel Brynielsson for his unending support and encouragementthroughout the project. He also helped me using LATEX, and commenting on themanuscript, which resulted in many improvements. I would also like to thank mysupervisor at Nada, Henrik Eriksson, for giving me useful tips and opinions and forreading my manuscript. Margareta Käck helped with proofreading the manuscript.

Second, I would like to thank the people at OM. Simon Oest has been mysupervisor there. Sara Wetterling, Sara Zolfaghari, Josefine Olofsson and ThereseStorheden have encouraged me in my work and patiently helped me by answeringquestions about the Saxess system. My room mate Emil Ek has inspired me withdiscussions on various subjects and answered many questions. Therese Storhedenand Emil Ek have also given me valuable help concerning QARun.

Third, I would like to acknowledge people in my everyday life, helping me invarious ways. Mikael Stralje helped me using LATEX in a Windows environment.Erika Lindroth and Erika Jönsson has encouraged me during many discussions.

Thesis Outline

Chapter 1 contains background information on the origin of this thesis. There isa brief introduction to OM Technology and the history of Saxess. An overview ofthe Saxess system and its different parts is provided along with a more detaileddescription of the Administrative Client. Brief information on the workflow of clienttesting within Saxess and the purpose of a testing tool is given. The goal for theproject together with questions at issue is described.

Chapter 2 gives a summarized description of the basic features and possibilitiesof the testing tool QARun.

Chapter 3 analyzes the testing tool. QARun is compared to some other testingtools to evaluate if QARun seems to be the most suitable to use for testing theAdministrative Client or if another tool should be chosen instead. The concludedopinion is provided.

Chapter 4 discusses how the tests to be implemented were chosen, and containsa description of the three categories of tests that were in focus during the project.

Chapter 5 describes the implementation. There is a brief description of the testplanning. A review of the implementation process and problems encountered duringimplementation is provided. The purpose of the variable database and the testspecification document is presented.

In chapter 6 conclusions are drawn, particularly on using QARun as a testingtool and the possibility of using QARun for testing the chosen categories.

Chapter 7 deals with future aspects of this thesis project, i.e., how the work isto be continued. Some things to consider in the future are proposed.

Appendix A includes the general part from the original test specification docu-ment written during the project.

Contents

1 Introduction 11.1 OM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Saxess History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Saxess System Overview . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3.1 Trading Engine (TE) . . . . . . . . . . . . . . . . . . . . . . . 21.3.2 Common Database (CDB) . . . . . . . . . . . . . . . . . . . . 31.3.3 Client Trade Application (Saxess Trade) . . . . . . . . . . . . 51.3.4 Information Engine and Market Surveillance (IE and Saxview) 51.3.5 Feed (MNF/MIF) . . . . . . . . . . . . . . . . . . . . . . . . 61.3.6 Operation Surveillance (SF) . . . . . . . . . . . . . . . . . . 61.3.7 Trading Control (TC) . . . . . . . . . . . . . . . . . . . . . . 6

1.4 The Administrative Client . . . . . . . . . . . . . . . . . . . . . . . . 71.5 Client Testing within Saxess . . . . . . . . . . . . . . . . . . . . . . . 81.6 Testing Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.7 Goals for the Project . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.8 Questions at Issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 QARun 132.1 Centralized Database . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.2 Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3 Learn and Replay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.4 Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.5 Text Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.6 Attach Names and Object Mapping . . . . . . . . . . . . . . . . . . . 152.7 Event Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.8 Alias Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.9 Image Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.10 Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.11 Identify . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.12 Dialog Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3 Analysis of the Testing Tool 213.1 Rational Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.2 Rational Visual Test . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.3 WinRunner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.4 Concluded Opinion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4 Tests to Implement 25

5 Implementation 275.1 Test Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.2 Implementation of Test Scripts . . . . . . . . . . . . . . . . . . . . . 275.3 Problems during Implementation . . . . . . . . . . . . . . . . . . . . 295.4 Variable Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.5 Test Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

6 Conclusions 316.1 Using QARun as a Testing Tool . . . . . . . . . . . . . . . . . . . . . 316.2 Testing the Chosen Categories . . . . . . . . . . . . . . . . . . . . . . 31

7 Future Work 33

References 35

A Test Specification Administrative Client 37A.1 Window Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

A.1.1 WC01 Change Record, Submarkets . . . . . . . . . . . . . . . 37A.1.2 WC02 Change Record, Orderbooks . . . . . . . . . . . . . . . 37A.1.3 WC03 Change Record, Participants . . . . . . . . . . . . . . 38A.1.4 WC04 Change Record, Submarkets LOOP . . . . . . . . . . . 38A.1.5 WC05 Change Record, Orderbooks LOOP . . . . . . . . . . . 38A.1.6 WC06 Change Record, Participants LOOP . . . . . . . . . . 39

A.2 Edit and Save Checks . . . . . . . . . . . . . . . . . . . . . . . . . . 39A.2.1 ES01 Edit one field, Submarkets: stop loss . . . . . . . . . . . 39A.2.2 ES02 Edit one field, Orderbooks: round lot . . . . . . . . . . 39A.2.3 ES03 Edit one field, Orderbooks: average matching . . . . . . 39

A.3 Add New and Save Checks . . . . . . . . . . . . . . . . . . . . . . . . 40A.3.1 AS01 Add new Person . . . . . . . . . . . . . . . . . . . . . . 40A.3.2 AS02 Delete added Person . . . . . . . . . . . . . . . . . . . . 40A.3.3 AS03 Add new User . . . . . . . . . . . . . . . . . . . . . . . 40A.3.4 AS04 Delete added User . . . . . . . . . . . . . . . . . . . . . 41A.3.5 AS05 Add new Broker . . . . . . . . . . . . . . . . . . . . . . 41A.3.6 AS06 Delete added Broker . . . . . . . . . . . . . . . . . . . . 42A.3.7 AS07 Add User in Participants . . . . . . . . . . . . . . . . . 42A.3.8 AS08 Delete added user account in Participants . . . . . . . . 43

List of Figures

1.1 Saxess system overview. . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Trading with Saxess. . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Trading Engine. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.4 Information Engine and Market Surveillance. . . . . . . . . . . . . . 61.5 The Administrative Client. . . . . . . . . . . . . . . . . . . . . . . . . 71.6 Market structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.1 The Submarkets window. . . . . . . . . . . . . . . . . . . . . . . . . 26

5.1 Relations between person, user and broker. . . . . . . . . . . . . . . 28

All pictures are taken from [13] with permission from OM Technology.

Chapter 1

Introduction

1.1 OM

In 1985, OM started as the world’s first commercially operated electronic exchangein Stockholm, and five years later the exchange was fully electronic[1, 7]. Sincethe start, the company has grown substantially. Today OM is an internationalcompany and one of the world’s leading providers of transaction technology, beingthe partner and leading provider of exchange technology to over 25 internationalexchanges and clearing houses. They develop new marketplaces and make existingones more efficient. OM also owns and operates the Stockholm Stock Exchange(since 1998) and exchanges in Calgary and London. The company has over 1500employees and offices in 10 different countries.

1.2 Saxess History

In 1995 the Stockholm Stock Exchange (SSE) decided to start a new trading systemdevelopment project[6]. The current system, SAX, had been used since 1989 andwas designed to manage 6 000 trades per day (average 1989 was 1 000 trades perday).

During 1992-1994 the demands on the system changed. Financial legislation wasrelaxed as shorting was allowed, turnover tax was withdrawn and the restrictionson members’ proprietary trading were eased. Technological changes included theintroduction of automated trading systems. New instruments and types of membersjoined the exchange.

The new demands made some issues apparent. The lead-time for introducingnew instruments was too long and the dissemination was not flexible enough. Im-plementing new types of functionality was difficult and expensive. By January1997 the capacity for SAX was 50 000 trades per day, so it had scaled well butprice/performance was deteriorating. A new system was needed.

When the Saxess project started, a main goal was increased flexibility. Theability to trade all types of financial instruments was important, as was the ability to

1

manage different market structures (for example price driven trading, calls, acceptorders and bulletin board). It was desirable to get flexible transparency and toprovide support for block trading. The requirements also included new order typessuch as combinations, linked orders and stop loss orders. Continued increase in tradevolumes was foreseen, so performance was an important consideration. The demandon Saxess was to scale to 2 000 transactions per second per partition at constantcost per transaction. Availability had to be the same or better than for SAX.

A market survey was done, indicating that there was no existing applicationfulfilling the demands. Consequently, the project was kept in-house and Saxesswas developed by the IT-department (with some consultant support). The goal forlaunch was set to April 1999.

The development project used iterative development with daily builds. An earlydecision was to strive for platform independence to keep future options open. Testingwas done continuously during development. To automate the testing, specializedtools for simulations and regression tests were developed. An externally availabletest system was provided early on to ensure well-functioning interaction with membersystems.

The current production platform is Sun Solaris, but Saxess is portable to severalother platforms, like AIX, HP-UX, Linux, Solaris, Digital Unix and Windows NT.To ensure stability, Saxess runs on dual geographically separated systems.

Saxess was launched in March 1999, meeting all demands. New versions havebeen developed since then to match new requirements.

1.3 Saxess System Overview

Saxess could be described as a complete system for trading stocks, bonds, derivativesand other financial instruments, where you can trade on multiple exchanges, marketsand currencies[13]. It consists of a Trading Engine, a common database and con-necting points for external applications, which feed the system with orders. Thereis also a number of tools for managing the system. Figure 1.1 shows an overview ofthe system.

The member trading applications connect to the Trading Engine using TCP/-IP[14] as network protocol. The transaction protocol XTP (open eXchange Transac-tion Protocol) is used within the central system for transactions, between the Saxesssystem and Saxess Trade (and other trading applications)[15]. XTP corresponds toOSI layer 7 in the OSI Reference Model[10]. Figure 1.2 shows how trading withSaxess works.

1.3.1 Trading Engine (TE)

The Trading Engine (TE) is written in ANSI C and runs on UNIX platforms thatsupport relevant POSIX standards. It consists of duplicate systems (Primary/Secondary).Under normal circumstances the Primary System receives all requests from the mem-ber trading application. For each request the Primary System issues the request to

2

Trading Control (TC)

Daily Initial Supervisory Functions (SF)

Trading Engine

Feed / Data Warehouse

Market Surveillance (SaxView )& Information Engine (IE)

AdministrativeClient (AD)

SAXESS TradeServer

SAXESS TradeApplication

Common Data Base

During Day

Market Information Feed (MIF)

Figure 1.1. Saxess system overview.

the Secondary System. Both systems then process the request using the same logicand both disseminate any results to its clients. If the Primary System fails, theSecondary System takes over. The take-over is done automatically in a few secondsafter a failure. The Trading Engine then operates in simplex mode until the fail-ing system has recovered. The recovering completes in ten minutes. Operating insimplex mode does not affect the performance but reduces fault tolerance. Sincethe Secondary System has processed the same requests as the Primary System theprocessing can resume without any loss of requests. The transaction flow of theTrading Engine is shown in figure 1.3.

Data are copied from the CDB to the TE every morning. All data are cached inRAM. If the CDB fails, the TE can still continue.

1.3.2 Common Database (CDB)

The Common Database (CDB) is an SQL database implemented in Oracle. It is themain place for storing static data, like exchange members, instruments etc[17]. Everymorning of a trading day, the static data are copied to the Trading Engine as startvalues. During trading, no data is read from the CDB, so the Trading Engine cancontinue trading even if the database fails[13]. Changes from the Trading Control

3

Buyer Seller

SAXESS

Clearing/settlement

Buy!

Sell!Buy!

Done Done

Matching

(VPC)

Figure 1.2. Trading with Saxess.

are saved during the day. At end of day, trades, overnight orders, billing info andend-of-day statistics are saved. The contents of the database are:

• Issuers

• Instruments: Equities, bonds, options, etc.

• Trading hierarchy: Exchange – Market – Submarket – Orderbook

• Parameters:

– Submarkets: opening hours, transparency, price type, etc.

– Orderbooks: lot size, matching rules, etc.

• Participants: Sessions, Users, Applications, Privileges

• Overnight orders

• History: Old trades, Billing info

• Future: Coming updates

Static data is entered mainly using the Administrative Client (AD).

4

Back EndFront End

Back EndFront End

Primary system

Secondary system

ExternalClient

Back EndFront End

Back EndFront End

Primary system

Secondary system

ExternalClient

Figure 1.3. Trading Engine.

1.3.3 Client Trade Application (Saxess Trade)

The Client Trade Application (Saxess Trade) is an application developed in C/C++.It is a complete trading environment for a broker firm trading in Saxess, which isupdated continually when new functionality is added in Saxess. Trading is done ina simple way in the different Saxess markets. Saxess Trade keeps a local copy of theorder books, which minimises the number of queries towards Saxess. The connectionto Saxess is performed using the protocol XTP.

1.3.4 Information Engine and Market Surveillance (IE and Saxview)

Information Engine and Market Surveillance (IE and Saxview) consist of an Apacheweb server and a web browser including all product documentation and the SaxViewapplication. SaxView is a web tool to display trade data reports and statistics. It isused by customer support to display order, trade and trade report information andby the market supervision to monitor the conduct of the market participants[16]. Itis also possible, by session monitoring, to see who is logged on to the system[13].An overview of the IE and Saxview is shown in figure 1.4.

5

TETE

Web server

Web browser

FilesFiles

SaxViewSaxView XML�HTMLXML�HTML

DocsDocs

Figure 1.4. Information Engine and Market Surveillance.

1.3.5 Feed (MNF/MIF)

Nordic Market Feed (NMF) is a convenient way for information vendors to get real-time data from the Nordic exchanges in one single feed. Market Information Feed(MIF) is market information available for information vendors and public. In MIF,historical data from Saxess are stored.

1.3.6 Operation Surveillance (SF)

The Supervisory Functions (SF) application, developed in Java, is used by the Ex-change staff to supervise the processes of the Saxess system[14]. From the SF thefollowing can be monitored:

• all sessions connected to the Trading Engine,

• session related data,

• all processes in Trading Engine,

• process related data such as time for an order to pass through the system,

• time for inter-process communication.

1.3.7 Trading Control (TC)

The Trading Control (TC) is an application developed in Java, used for viewing andupdating the Saxess Trading Engine during the trading day. It uses the externalprotocol XTP[13]. The TC can be run against both a primary and a secondary

6

Trading Engine, but it must run against a primary Trading Engine to be able to up-date the system. From the TC a technical stop, trading halt or release of all objectsin a Saxess exchange (markets, submarkets, orderbooks, participants, etc.) can beimposed. It is also possible to send manual broadcasts to whole markets. Orders canbe cancelled, suspended, flushed and activated; and trades can be cancelled fromhere. After a restart, normal execution of Saxess is started from the TC. It is oneof the most important tools (together with the Administrative Client) used by theservice desk.

1.4 The Administrative Client

The Administrative Client (AD) is an interface (a Windows program) towards theSaxess database (CDB), developed in Delphi. It is a tool used to administer all staticdata used in a Saxess marketplace[13]. The purpose is to enable exchange personnelto display and modify the contents of the Saxess database[11]. Among other things,it is used for setup of the market structure, trading rules, financial instruments,defining new participants, orderbooks and other data used in the Saxess system,authorization of participants and users[14]. A typical AD window is shown in figure1.5.

Figure 1.5. The Administrative Client.

7

There are three main information areas in the AD: Common Information Store(CIS), Financial Instrument Administration (FIA) and Trading Administration(TAD).

The Common Information Store part contains information about organisations,issuers and persons that are of interest to the participating stock exchanges, and amiscellaneous collection of other things[12]. All organization roles/types and persontypes have their own windows, with window specific information, sorted in tabs.Some tabs, for standard organization information are the same for many windows.Organizations and persons have roles such as[14]:

• Exchange

• Member

• Participant

• Issuer

• Clearing institute

• Broker

• User

• Employee

The Financial Information Administration area holds information about all trad-able instruments that are of interest to the stock exchange, including equities, bonds,convertibles, options, warrants, etc.

The Trading Administration area contains information about all parameters fortrading[12], including orderbooks, markets and submarkets. Different parametersapply on each level. A market is a place where buyer and seller meet. A submarketis a logical group of orderbooks that are traded in the same way. The parametershere control the trading characteristics. An orderbook is the smallest tradable objectin which all orders are compiled. Parameters set for an orderbook decide how tradingin a certain instrument should occur . If the market is for example “SSE Equities andrelated” (SSE = Stockholm Stock Exchange), the submarkets for that market couldbe “SSE Equities”, “SSE Convertibles” and “SSE Warrants” and an orderbook in“SSE Equities” could be “ERIC B” (Ericsson B share). Figure 1.6 shows an overviewof the market structure.

1.5 Client Testing within Saxess

The clients within Saxess are tested thoroughly under controlled conditions to verifythat the functionality and characteristics are correct and fulfill the requirementsand user needs[3]. Other reasons for testing are to provide information about theprobability of remaining faults and to give continuous feedback to the developers on

8

ExchangeStockholmsbörsen

ExchangeStockholmsbörsen

ExchangeOslo Børs

ExchangeOslo Børs

SubmarketSSE Warrants

SubmarketSSE Warrants

MarketSSE Equitiesand related

MarketSSE Equitiesand related

InstrumentERIC B

InstrumentERIC B

MarketSSE Bondsand related

MarketSSE Bondsand related

MarketFixed Incomeand related

MarketFixed Incomeand related

MarketCSE Equitiesand related

MarketCSE Equitiesand related

SubmarketSSE Equities

SubmarketSSE Equities

SubmarketCSE Rights

SubmarketCSE Rights

SubmarketSSE Convertibles

SubmarketSSE Convertibles

SubmarketCSE Equities

SubmarketCSE Equities

SubmarketCSE XtraMarked

SubmarketCSE XtraMarked

ExchangeKøbenhavns Fondbørs

ExchangeKøbenhavns Fondbørs

ExchangeVer�bréfaÞing Islands

ExchangeVer�bréfaÞing Islands

OrderbookERIC B

EUR - VPC

OrderbookERIC B

EUR - VPC

OrderbookERIC B

SEK - VPC

OrderbookERIC B

SEK - VPC

Orderbook(ERIC B)

DKK - VPClearing

Orderbook(ERIC B)

DKK - VPClearing

Orderbook(ERIC B)

EUR - VPClearing

Orderbook(ERIC B)

EUR - VPClearing

Figure 1.6. Market structure.

parts of the system reviewed or tested during all phases of the project. Testing isdone according to a test workflow and divided into several test activities that aredescribed below. A more detailed description can be found in [3].

In the test management all system test activities are planned, followed up andcoordinated during the whole system development. The test strategies, processesand methods are specified and tests are planned and reviewed which results in a“test plan”. In the test specification the test cases from the test plan are specifiedand implemented. This activity results in a “test specification” document and testscripts (if automated). In the test execution the system test objects and test casesare executed and the result is analyzed and checked/verified. This results in testrecords/result files or log files. All faults found are reported in SPR’s (SystemProblem Reports). The last activity is the test reporting, where the results of allexecuted system test objects and test cases are analyzed and reported in a “testreport”.

Different test types are used in the test workflow, including “Functional tests”,where the business functionality for a specified client is tested, and “Integrationtests”, where the interaction between sub-systems is tested.

In the test workflow there are also different test strategies. “Weekly build” isused to assure that all functionality developed and the corresponding test cases areexecuted on weekly basis. By doing this weekly, errors can be tracked and correctedimmediately. “Regression testing” is a test strategy in which previously executedtests are re-executed against a new version of the application or system, to ensurethat the quality remains while new features have been added. The purpose of thestrategy “Production-like test environment” is to ensure that all tests are executed

9

in an environment as similar as possible to the production environment used by thecustomer.

Saxess Trade is the only client within Saxess that has been tested automatically.All the other clients are tested manually, following test specifications, which couldbe very time-consuming and ineffective. Hence, the aim is to extend the automatedtesting to also comprise the other clients.

1.6 Testing Tool

A testing tool is a program to make testing more efficient, automating the tests.With a testing tool it is possible to create, modify and run automated test scriptstesting applications. In most testing tools the test scripts can be designed by handin some simple script language or by recording user actions. Every action in the testcase is included in the test script. When the script is run, it carries out the testautomatically.

1.7 Goals for the Project

The purpose of the project was to create and start up automatic tests for the Ad-ministrative Client within Saxess. The automatic tests were to be used as a com-plement to the manual tests for the application, for regression testing and retestingof functions after fault correction. Three different categories of tests were to beimplemented and executed:

Windows testing checking that the window looks and functions as it should whenyou make some change (click somewhere or change size etc.)

Data testing checking that it is possible to go into edit mode and then save. Thepurpose with these tests is to detect corrupt data that could be stored in thedatabase. When saving, the Administrative Client activates checks detectingif there are any corrupt data.

Savings testing Values are fed into the database and it is verified that the newdata can be saved.

Before the testing started, an analysis of QARun, a testing tool from Compuware,was to be done to evaluate if the program would be a suitable tool for implementingthe tests. This analysis included a comparison between QARun and other tools onthe market. If QARun after this analysis or during the forthcoming implementationappears to be unsuitable for the project, a better tool should be suggested.

After the analysis of the tool an evaluation of tests was to be made, investigatingwhich tests that were possible to implement and which tests that would give relevantinformation. The chosen tests were to be implemented, executed and analyzed. Atest specification over the tests was to be written, documenting what is tested ineach test case.

10

1.8 Questions at Issue

Important questions to be answered during this project was:

• Would QARun be a suitable tool to automate the testing of the AD or arethere other tools more suitable for this?

• To what extent would it be possible to automate the testing of the AD? Is itpossible to implement tests within the proposed categories?

• Which tests out of each category are most relevant to implement?

• If there are any difficulties implementing the test cases, then how can they besolved?

11

Chapter 2

QARun

This chapter is a summary of the basic features of the testing tool QARun. A moredetailed description is found in [2].

QARun is a testing tool created by Compuware, for more efficient testing ofapplication software. Testing application software is hard to do thoroughly and verytime consuming. Using a tool like QARun for automating the setup and executionof test scripts makes it possible to increase both the pace of testing and the accuracyof the testing significantly. Automated tests run faster and it is possible to run thetests as many times as you need. And of course, your presence is not called for.All actions performed by the test and all reactions from the tested application arerecorded in a log that can be reviewed anytime.

QARun runs under Microsoft Windows 95 or Windows NT and can be used fortesting most applications accessible from a PC running Windows.

There are two ways to create scripts in QARun. QARun can record user actionsand system responses into scripts, or you can create scripts by writing the codedirectly. The two ways can be mixed, and you can choose whichever way you findmore suitable. The fact that you can record actions into a script makes it easy towrite powerful test scripts even for people with little knowledge of programminglanguages.

When a script is executed in QARun, it looks as if a human tester is performingthe test. The keystrokes and the selection of items from menus and lists are generatedfrom software, but it looks like someone is really doing it.

2.1 Centralized Database

The architecture is based on a multi-user database, which offers centralized controlof users and system access rights under password control. Other benefits are ver-sion management of scripts and checks, user information, change histories, versioncomparison and user-configured archiving.

QARun is a multi-user system, which means that multiple users can work to-gether and share scripts, objects, checks, etc. Records that are being edited by a

13

user are locked for editing for other users and a message will be displayed showingwho is editing the record.

2.2 Scripts

When testing an application manually you use actions, like making menu selectionsor typing data. In a script there are script commands representing these actions.There are many advantages in using scripts:

• You can modify the test if the application changes.

• You can make new tests by copying and modifying existing scripts.

• Scripts can “loop” to repeat a process many times.

• You can build “intelligence” into the tests to handle exceptional situations.

• You can use scripts to document the test process.

• You can begin building test scripts before the application is ready for testing.

The scripts in QARun are written in a script editor, using a simple but powerfullanguage that gives control over any application running in the Windows environ-ment. This makes it possible to write scripts that run automatically at a specifiedtime of the day, run entirely unattended, or interact with users to receive informationlike IDs and passwords.

Every test script can be run individually, but usually they are put together ina suite. A complete test suite includes many individual tests that are executed atdifferent points (test sites) in the application. Apart from the test scripts, there aredriver scripts. The driver script does not actually test the application, but “drives”the application to the proper point where a test should begin, the test site, and thencalls one or several test scripts to begin testing there. After a test is finished, thedriver script takes control again and goes to the next test site.

2.3 Learn and Replay

Using the Learn function, you can create scripts without writing the code yourself.If you switch Learn on, your actions and the responses from the application aretranslated to script commands and pasted into QARun’s Script Editor. Keystrokes,mouse moves and clicks are recorded. QARun works at an object level and objectslike buttons are identified by name, which makes the scripts less sensitive to changesin the application or environment.

The learnt script can be replayed as many times as you want. Replay is alsoobject oriented and the script replays correctly even if there have been environmentalchanges or changes in the application.

14

2.4 Checks

Checks are used to verify that the application is working correctly and respondsproperly to a given input. To test this, you must first know how the applicationshould respond to the given input. Then you compare the actual responses to theexpected responses and record differences as errors. The expected responses arestored in QARun as checks. A check is a definition of what the application shouldbe showing at a particular point. Different types of checks can be defined, forexample:

Clock checks – measures the time the system takes to perform a process.

Form checks – verifies the properties (size, position, ID, active/disabled, etc.) ofthe controls in a dialog against a standard.

List checks – tests the contents (number of items, list content and selection state)of list controls and combo boxes.

Menu checks – verifies the items (text, ID and enable and check states) in a menuat a particular time.

Text checks – compares the textual content (entire content or selected individualareas) of a window to a baseline data. You can check for exact text matches,alphanumeric patterns, dates, times and numbers that can match exactly orfall within ranges.

Window checks – check the status of any window (existence, size and position,state, module and class and whether it is enabled/disabled).

If there is no type of check that suits your testing requirements, it is possible todesign your own check, a so-called User check.

2.5 Text Recognition

QARun has the ability to read text at any size or font from the screen. All text in awindow can be read or just the text from a specific window area. QARun does notrely on optical character recognition, OCR, nor on the possibility to copy charactersto the clipboard.

Text recognition is fundamental for synchronization with multi-user applicationswhere response times can vary. QARun can watch the screen and wait for signsthat the application is ready to continue. In that way it can ensure that the testproceeds as fast as, but not faster than, the application can manage.

2.6 Attach Names and Object Mapping

The technique called object mapping is used to provide simplified aliases for thenames of Window objects. In the Object map, all the aliases are stored and after a

15

window is registered there, all references to it are made by its alias rather than byits attach name.

To control a Windows application you usually click on the window to select it. InQARun, a method called attaching is used. A Windows application can have manydifferent dialog boxes, each containing different menus, edit boxes, lists, buttonsetc. Each dialog and control has a unique name to identify it, and QARun uses thatname to locate the window and attach to it. The attach name of a Windows objectis constructed from five parts:

Type – a single-letter code to specify the type of window. Some windows have atitle bar, but others have no title of their own. To refer to a window withouttitle you must use the name of the parent window (the window in which itresides). The type code indicates if the window has a title or not.

Module – the name of the program that causes the window to display.

Class – describes the purpose of the window (Ex. “Edit”).

Title – the window title. Underlined characters are shown with a “&” precedingthem in their attach name.

Position – distinguishes between windows that have identical module, class andtitle. This part of the attach name is not always needed

A full attach name consists of these five parts (or four, if the position number is notneeded) joined by tilde (~) characters. This is an example:

~N~QADEMO.EXE~ListBox~&New

For a dialog box including two fields, their attach names could look like this:

~P~QADEMO.EXE~Edit~Customer Invoice~1~P~QADEMO.EXE~Edit~Customer Invoice~2

By these attach names you can tell that:

1. Their identities are derived from their parent (~P).

2. The application causing the window to display is QADEMO.EXE(~QADEMO.EXE).

3. They are edit controls (~Edit).

4. The name of the parent window is Customer Invoice (~Customer Invoice).

5. They have identical module class and title, so a position number is needed fordistinction.

16

Logical name (used in scripts) Physical nameCustomer Invoice - Account Name ˜P˜QADEMO.EXE˜Edit˜Customer Invoice˜1Customer Invoice - Account Number˜P˜QADEMO.EXE˜Edit˜Customer Invoice˜2

Table 2.1. Object Map.

The attach names uniquely identify a window, but they are not particularly de-scriptive. Object mapping is used to have a logical name, an alias, associated witha physical window. The Object Map maintains a 1:1 relationship between logicaland physical window names. For the example above, the Object Map could containthe relationships shown in table 2.1.

Two important benefits of object mapping are that the names become moredescriptive and that the scripts are not dependent on actual attach names (whichmay change). If another field is added in the dialog box of the example above, thepositions could change, but using object mapping you do not have to modify thescripts, only the relationship in the Object Map has to be changed.

2.7 Event Mapping

QARun can recognize unexpected conditions in the application and respond to themif they are defined as events. An event is a condition (external to QARun but internalto the computer system) that can be tested by the script. When working with acomputer application you perform some actions and wait for the system to processthem before you can continue. The script must do the same. But the time youhave to wait for the application to process is variable and hard to predict. It wouldbe inefficient and unreliable to just decide in advance how long the script shouldwait. The most efficient solution is to watch the screen for a sign that the system isready to continue. QARun can then recognize unexpected conditions and respondto them. The conditions that QARun must respond to are defined as events. Thefollowing types of events can be defined:

Bitmap – monitors for presence or absence of a graphic.

Date/Time – monitors for when a certain date/time is reached, or a period oftimed elapsed.

Keyboard – monitors for when a specified keystroke combination is typed.

Menu – monitors for when a specific menu item is selected

Mouse – monitors for when a mouse button is clicked/released in a certain window.

Screen – monitors for when a sequence of characters is displayed somewhere in awindow.

Window – monitors for when a window is created, destroyed, moved, sized or justchanged in some other way.

17

Events enable the scripts to act intelligently, making decisions that depend onexternal conditions. There are three important advantages in defining events:

• The scripts are easier to read.

• It allows events to be public, so all users and scripts can use them.

• The event-definition can be changed, applying the change to the event, not tothe scripts referring to it.

2.8 Alias Mapping

When learning a QARun script, any standard control type – a control with a but-ton, edit, list box, radio button, etc. – that is encountered is treated as a Windowsobject. However, some applications have non-standard controls, and those are notrecognized. In this situation QARun stops object recording and only learns key-strokes and mouse clicks, to ensure the correctness of the script. This makes thescript harder to understand, though.

A better solution for this situation is alias mapping. Non-standard controls canbe registered as standard controls in the Alias Map, which cross-references non-standard class controls to standard control types. You only have to ensure that thenon-standard control is comparable to a standard control. A non-standard button,for example, can be registered as a standard button.

Example scriptWithout alias mapping:

Attach ‘‘~P~ADDRESS.EXE~ThunderSSCommand~Address BookVersion 1.0~3’’

MouseClick 26, 18, ’Left Down’MouseClick 26, 18, ’Left Up’

With alias mapping of the button:

Attach ‘‘~P~ADDRESS.EXE~ThunderPictureBox~Address BookVersion 1.0~1’’

Button ‘‘~3’’, ’Left SingleClick’

With alias mapping and object mapping:

Attach ‘‘Address Book Toolbar’’Button ‘‘@Open Database’’, ’Left SingleClick’\\

2.9 Image Mapping

Some Windows applications use bitmap pictures, which are not true Windows ob-jects and have no name of their own; they are just areas within windows. Recording

18

them in QARun normally results in scripts with a bunch of mouse-click commands(compare not using alias mapping when recording non-standard controls).

Image mapping solves this situation. In the Image Map you store capturedbitmap images that are not real Windows objects by descriptive names. After that,when recording a mouse-click on that image, the defined name is used in the script.

2.10 Logs

When you run a script in QARun, all actions and check results are written to a log,where you can see the outcome of the test run. If a check fails, then the expectedand the actual responses are written to the log, so you can compare them.

The tool for viewing and printing logs is called the Log View. There, the logsare displayed in a grid-like format. A log entry contains the name of the script, dateand time each line was executed, the command carried out, the result of checks, etc.You can decide your own log layout (what categories and how the information isshown) and save it. It is also possible to filter the log view to select the entries ofmost interest. A summary of the test showing number of checks passed/failed canbe viewed and printed.

2.11 Identify

This is a utility program that supplies information about windows displayed on thescreen. It can be used to create entries in the Object Map or Alias Map. The mostuseful feature is that you can paste information from Identify into the script editorwhen you need some window data, like the name of the window.

2.12 Dialog Editor

This utility program makes it possible to design dialogs that will be displayed whilerunning the scripts. Dialogs allow you to enter variable or confidential data intoQARun as it is running. The information thus does not have to be hard-coded intothe script. An example is a dialog for login.

19

Chapter 3

Analysis of the Testing Tool

When making my study, I have focused on QARun since that is the program OMhas used/is using to test applications similar to the AD. One of the questions I wassupposed to answer was: Is it possible to use QARun for automatic testing of theAD? During the evaluation of QARun and the possibilities using this tool, I havealso made a small study of some other testing tools for comparison.

3.1 Rational Robot

Rational Robot is an object oriented testing tool, which performs functional tests onWeb, ERP (Enterprise Resource Planning software) and client/server applications,mostly e-commerce and e-business applications[8]. It supports many environmentsand languages including HTML, DHTML, Java, Microsoft Visual Basic and VisualC++, Oracle Developer/2000, Delphi, SAP, PeopleSoft and Sybase Powerbuilder.

With Rational Robot it is possible to create, modify, and run automated func-tional, regression, and smoke tests (the first run of a piece of software after con-struction or a critical change[5]) for e-applications built in a variety of developmentenvironments. Included with Rational Robot are Rational TestManager and Ra-tional SiteCheck[8]. TestManager is a management tool, which gives you controlover and access to all testing planning, assets and activities, etc. SiteCheck is a toolfor Web site link management, which provides a single-click functional test of theentire Web site.

Rational Robot generates test scripts in SQABasic, which is an integrated script-ing environment that makes it possible for you to view and edit your scripts whileyou are recording. You can test each application component under variable condi-tions and the program provides test cases for menus, lists, alphanumeric characters,bitmaps and many other objects. The test results are automatically logged in Ra-tional Repository and the logs are colour coded to give a quick overview. If youdouble-click on an entry you are directly brought to the corresponding line in thescript, where you can correct the possible error.

21

3.2 Rational Visual Test

Rational Visual Test is a tool for automating functional tests of Windows applica-tions created with any development tool[9]. It is integrated with Microsoft DeveloperStudio and has extensive integration with Microsoft Visual C++. The idea usingRational Visual Test, is to get maintainable, reusable and extensible test compon-ents that can be used throughout the development process, and that will make thetest process more efficient and less time-consuming. Tests can be made early inthe process so that the applications can be tested from the beginning and then re-peatedly throughout the development. It is possible to test for control existence andlocation, retrieve property values and update properties[9].

The core of the tool is a Visual Basic-derived programming language, TestBasic,with hundreds of test-specific functions, special constructs to make testing easier,easy access to the Windows API and an open architecture. If a function is neededthat is not included in the language, it is possible to make a new function and placeit in a DLL, and call it from the test program as if it was built in to the TestBasiclanguage. Included in the program is also the Suite Manager, to build, manage andexecute suites of automated tests.

With Rational Visual Test it is also possible to test Web-based applications, bothclient-side and server-side, using the TestBasic functions designed for Web testing.

3.3 WinRunner

WinRunner from Mercury Interactive is a functional testing tool for testing applic-ations of various types. The tool supports the Win95, Win98, WinNT and Win2000platforms and the Netscape and Microsoft Internet Explorer, Java, ActiveX, VisualBasic, C/C++, PowerBuilder and Delphi environments[4]. There are two ways ofcreating tests. Either you can record a typical process by pointing-and-clicking onthe GUI, or you can edit test commands directly. The latter way is more neededwhen making more complex tests. When recording a test, you can insert check-points to compare expected outcome with actual ones. Several types of checkpoints,including text, GUI, bitmap and database can be used.

Apart from creating and running tests, WinRunner can verify database values toensure transaction accuracy. When replaying, the test will verify the actual values inthe database against the expected values. Updated/modified, deleted and insertedrecords are automatically being highlighted.

Using the “DataDriver”, it is possible to test how an application will run withdifferent sets of data. For example it is possible to change hard coded fields, suchas names or numbers, into variables that allow testing the application with multiplevalues.

There is a Function Generator, containing a category list of functions, where youcan find predefined functions and descriptions of them. The function to be used canbe selected from a category list, and the Function Generator provides a description

22

of the selected function. You only have to input the parameters and insert thecomplete function into your test.

To eliminate the special coding requirements for previously unknown objectsWinRunner has a Virtual Object Wizard. Using this, you are presented with a listof object types possible to use to represent your unknown object. You can definethe borders for the object (such as a button), and then enter a name for it. Afterthat WinRunner will refer to the object by this name. This improves the readabilityof the tests.

After creating the tests, they can be run. Executing the tests, WinRunner op-erates the application automatically, as though a user was performing the steps.When the tests are run, WinRunner’s interactive reporting tools helps interpretingthe results, by providing reports listing the errors found in the test and their loca-tions. These contain descriptions of the larger events that occurred during the test,including errors and checkpoints. By clicking a button you can get more details.

Tested applications are over time modified, which means that further testingwill be required. WinRunner enables building reusable tests to use throughout theapplication’s lifecycle, which makes the testing more cost-effective.

3.4 Concluded Opinion

There are many tools for automated tests on the market. I have not been able todo a full evaluation of all the programs that could possibly be useful in my project,since that would take too much time to do. I have made a deeper study of QARun,because OM wanted me to investigate if using QARun would be possible whenautomating the tests for tha AD client. Apart from the study of QARun I havesearched and read about other tools and tried to evaluate if they seem to be betterto use when testing an application like the AD.

My conclusion must be that the most suitable testing tool would be QARun.There are several reasons for this. First, getting acquainted to Saxess, I have studiedtests regarding Saxess Trade (which is another application within Saxess, apart fromthe AD). Doing so I have looked into QARun and how it can be used to make scriptsand build up test suites. In my opinion QARun seems to be a tool appropriate fordoing this kind of testing. The possibilities making scripts, functions, etc. seem tobe enough to be able to automate the testing of an application as complicated asthe AD. Since the testing of Saxess Trade has worked out well using QARun, whyshouldn’t it work with the AD too?

After my small study of other testing tools I have found no indications that anyof the other tools would be more useful than QARun for testing the AD. WinRunnerseems to be a tool equivalent to QARun, from what I have read, so it would possiblybe an alternative, but having two seemingly equivalent tools, I can see no reason tochange to a new one.

Another reason for choosing QARun is that since some of my colleagues use thistool, they have great knowledge of it and, to a large extent, know its possibilities

23

and limitations. If need arises later, I will always have the opportunity to consultthem for help and advice. If one chooses another, new, testing tool, one wouldhave to “start from the beginning”, not being able to use all the knowledge peoplealready possess at OM. I would also have to make a deeper study of the new programbefore I could get started testing. Choosing QARun, I could probably start the testactivities earlier, since the testing tool is already available at OM and since I havelearnt much about the tool and how it works while making my pre-study.

24

Chapter 4

Tests to Implement

To decide which tests to implement during the project, a thorough study of theAdministrative Client was made, with the aim to learn about how the applicationshould be used and all the features included. A test specification for manual testsin the AD that existed before the project was also studied to get some inspirationin planning the tests that were to be implemented. The most important tools inthe analysis of what kind of tests that were possible to create in QARun has beenthe QARun tests made for Saxess Trade and the test specification for those tests.From them much knowledge about the testing of clients in Saxess has been acquiredand the implemented test scripts have been a great introduction in the process oflearning how QARun works as a testing tool.

My focus for the testing has been set on three different groups of tests:

• “Window checks” (Windows testing category)

• “Edit and Save checks” (Data testing category)

• “Add new and Save checks” (Savings testing category)

I have concentrated on the AD windows “Submarkets”, “Orderbooks” and “Parti-cipants”. The Submarkets window is shown in figure 4.1.

The Window checks have been designed to test that the name of the windowschange when changing selected record in the selection list. Edit and Save checks testthat it is possible to edit somewhere in the window and then save the change withoutproblems. The Add new and Save checks test that a new person/user/broker canbe added and saved and also that the added person/user/broker can be deleted.

All the tests I have implemented are further documented in Appendix A, whichis the general part of the test specification document I have produced during thisproject.

25

Figure 4.1. The Submarkets window.

26

Chapter 5

Implementation

This section describes the implementation phase of the project. The implementationprocess was preceded by vast studies of the Saxess system and the testing toolQARun, used for the implementation. The tool study included reading manualsand guides, doing exercises included within the program and looking at test casesimplemented for testing the Saxess Trade client, to study how the program couldbe used for testing. Included in the study of Saxess was an internal OM courseabout the program, “Saxess 3.1 Basic course”. The Administrative Client, whichwas the client to be tested, was also investigated, to give me first hand knowledgeabout all functions and find out what functionality could be tested automaticallyand which tests would be most suitable to do. This study was done reading the“AD Users Guide” and the course material for the internal OM course “Saxess 3.1Administrative Client” and looking around thoroughly within the program to getacquainted with all windows and their functions.

5.1 Test Planning

Before implementing the tests, test cases had to be planned. Suitable test cases werechosen and the scenarios for them were well considered to produce detailed plans ofevery move that was to be realized in each test script. The tests to be implementedwere divided into three groups; Window checks, Edit and Save checks and Add newand Save checks. For each group, a few test cases were to be implemented.

5.2 Implementation of Test Scripts

The first script implemented in QARun was a script starting up the AdministrativeClient window. This script was not part of the planned test cases. It was writtenmerely as a first attempt using QARun, and to start up the AD for being able to runother test scripts. One script called “Global_Functions”, including own functionsthat were to be used in many scripts, was also created.

27

The first group of tests to be implemented was the Window checks. One testscript was done for each window, testing the Submarkets window, Orderbooks win-dow and Participants window. The first versions were quite simple, testing that thewindow name changed correctly, when changing selected record in the selection list.Later these tests were extended with loop tests, looping through the whole selectionlist checking that the window name changes for every selected record.

The next step was to implement the Edit and Save checks. These tests werefound to be more complicated, since what could be edited in the window differedon the currently selected record. Because of this, the tests implemented were verysimple, just making some change (selecting/deselecting a check box, etc.). Testswere written for the Submarkets window and Orderbooks window. In one test, thevalue from a field was cut out, increased by a number and pasted in again.

Before implementing the Add new and Save checks, the relations between “User”,“Broker” and “Person” were investigated. The conclusion was that user and brokerinherit from person and to register a user, a person has to be registered first. Abroker can exist without being a user, but to be active, the broker must be a userwith a user account. The relations are shown in figure 5.1.

PersonPerson

UserUser

BrokerBroker

PersonPerson

UserUser

BrokerBroker

Figure 5.1. Relations between person, user and broker.

The result from the study of these relations was that test cases for adding anddeleting person, user and broker were designed and implemented. These tests usedthe Persons window, the Users window and the Brokers window for adding anddeleting a person/user/broker. In addition, test cases were implemented for addingand deleting a user account in the Participants window. The idea with this “addtest” was to verify that it was possible to activate a new user account for a former

28

user, from the Participants window. The “delete test” was to verify that the addeduser account also could be deleted from the Participants window.

When the implementation of planned Add new and Save checks was finished,a revision of the Window checks followed, where the script code was made easierto read and some dependencies in the code were removed. A similar review of theremaining tests (Edit checks and Add new and Save checks) was also made, whichimproved the script code. In the end of every test script, a function closing allwindows was added, to clean up the AD environment after a test script has beenrun. The “Global_Functions” script was revised, mainly to remove every dependencyin the attach names, for example of object names in the selection list.

5.3 Problems during Implementation

After some slight problems with the license file of QARun, more problematic issuesarose.

A main difficulty in the implementation concerned the id/handle for the ADwindows. It was discovered that the id/handle for every window is dynamic andchanges every time the window is opened. This feature complicated the writingof functions that attached to window id/handles, something that was frequent inthe Saxess Trade tests that inspired me. The problem was solved attaching to thewindow names instead, resulting sometimes in attach names depending on the selec-ted record (like hard coding). But this consequence was later removed exchangingthe part depending on the current selected record in the attach names with the*-symbol. The problem regarding the dynamic id/handle also included every field,button, etc. within the windows, but in most cases this issue could be solved insimilar ways. Another problem that appeared, probably as a consequence of theid/handle problem was that numerous predefined and seemingly useful functions inQARun were not possible to use. As an example, to click a button, the function“Button” could not be used to attach to the button. Instead the button had to beattached by name and the learn-function had to be used to record the coordinatesof the mouseclicks.

A problem, which arose making the Edit and Save checks, was that the windowsdid not always look the same and did not have the same elements for every record,since some properties, for example “average matching”, do not exist for some recordsand since the possibility of editing some properties depend on others, for exampledates.

Implementing the Add new and Save tests, one problem was to get hold of theperson added, when this person was to be made a user. The window for addinga person is at this point already shut down, so it was necessary to find out a wayto remember the id for the added person, to be able to search for him/her. Thesolution to this problem was to use the database variables.mdb. In the script foradding a person, the id was saved in the database. Later, when making that persona user, the id was collected from the database and pasted in the search field, to find

29

the correct added person. The person was easily found and could be made a user.The same solution was used making the test to add a broker and in the tests fordeleting person, user and broker. In these tests the variables in the database werecleared when the person/user/broker had been deleted.

5.4 Variable Database

The variable database, “Variables.mdb”, was created to store useful variables. It wasused mainly in the Add new and Save Checks. The id for a person for example, wasstored, and could then be collected from the database. This made it possible to findthe recently added person and make him/her a user or a broker.

5.5 Test Specification

The test cases implemented during this project are described in the test specificationdocument. This document is divided into several parts. Each test case is describedseparately in both a general and a detailed description. As the general descriptionjust describes the general steps in the test case and what is to be verified, thedetailed description is a complete account of what happens in every little step of thetest case. In this project the test cases are grouped in three, as said earlier. Thisgrouping can be observed also in the Appendix A.

The test specifications are supposed to be written in parallel with the imple-mentation of the test cases. However, in this project the implementation of the testcases has usually preceded the writing of test specifications, especially regarding theAdd new and Save checks. These test cases resulted in very detailed descriptionsdivided in many steps.

The test specifications have different purposes. It is useful if several people areinvolved in the testing, since it is common that one person writes the test specific-ations and others run the tests. The test specifications are also a guarantee for thedevelopers that the test employees really know the functionality. After tests havebeen done, the test specifications are a documentation of which tests have been runand how the test cases have been designed. By reading them it should be possibleto learn what functionality has been tested, and if so, discover when some function-ality that should have been tested, has been missed during the testing. Anotherimportant use for the test specifications is when running regression tests, to be ableto recreate the test cases that have been run earlier.

30

Chapter 6

Conclusions

6.1 Using QARun as a Testing Tool

The primary question at issue for this project was to analyze QARun, to evaluate ifthe program would be a suitable tool to use to implement automated tests for theAdministrative Client (AD) within Saxess. The tests for another application withinSaxess, Saxess Trade, had already been successfully automated using QARun andOM wanted to see if the same tool could be used for the AD as well.

During the project, some difficulties have been encountered using QARun tomake tests for the AD, but they have not been unsolvable. If, for example, some ofthe predefined functions included in the tool, are not possible to use, there are otherways to write the tests. Instead of attaching to the id/handle of a window, which inthe AD changes every time the window is opened, you can use the Attach name, aunique name that every Windows is identified with (see section 2.6 on page 16). Itwill probably not be possible to test every single function in the AD using QARun,but many functions are still possible.

Hence, my opinion must be that QARun would be a useful tool for supplementarytesting of the AD, mainly when making regression tests and when retesting functionsthat have been reported having faults, verifying that they are corrected.

6.2 Testing the Chosen Categories

The testing categories at issue for the project were Windows testing, Data testingand Savings testing. A number of tests have been implemented for every category,choosing the tests that seemed to be most relevant to test. All tests planned to bedone have been implemented successfully. It would have been desirable to implementmore test cases, but the time for the project was limited.

My conclusion from the implementation is that it is possible to automate thetesting of these three categories using QARun. It should be possible to extend theautomated testing of the AD even further, testing most of the functionality includedin the application.

31

Chapter 7

Future Work

The automated testing of the Administrative Client is not finished by the end ofthis project. The idea with the project was to start up the automated testing and tocreate a foundation for the future work. The tests are to be used in regression tests,testing old functionality effectively. The aim for the future is to continue extendingthe automated tests to include more functionality. This development will be donegradually in every project including the Administrative Client and the intention willbe to automate as many manual test cases as possible.

Some things to consider in the future:

• Why are there so many predefined functions in QARun that are not possibleto use?

• If it is possible, it would be useful to make the id:s/handles in the Adminis-trative Client non dynamic.

33

References

[1] Public information about OM. http://www.om.com. Last visited September 2,2002.

[2] Compuware Corporation. QARun User’s Guide, release 4.8, 2001.

[3] Gunnar Hellenius. Core Workflow Test version 1.0. Technical report, OMTechnology, March 2002.

[4] Mercury Interactive. WinRunner: Powerful Test Automation for the Enter-prise. http://www-svca.mercuryinteractive.com. Last visited June 19, 2002.

[5] Jargon dictionary. Available on the Internet: http://info.astrian.net/jargon/terms/s.html#smoke_test. Last Visisted October 14, 2002.

[6] Anders Mårtensson. The Saxess Story. Technical report, OM Technology, 2000.

[7] Sofia Söderström. Usability in Trading Applications: Recommending an evalu-ation method to be used in the development process. Master’s thesis, Departmentof Numerical Analysis and Computer Science, Royal Institute of Technology,Stockholm, Sweden, 2000.

[8] Rational Software. Rational Robot Product Information. http://www.rational.com/products/robot. Last visited May 8, 2002.

[9] Rational Software. Rational Visual Test: Product Information. http://www.rational.com/products/visual_test. Last visited May 8, 2002.

[10] William Stallings. Data and Computer Communications, chapter 13.2. PrenticeHall International Inc., sixth edition, 2000.

[11] Team Documentation and Training, Department Customer Services, BusinessUnit Saxess. AD User’s Guide. OM Technology, 2002.

[12] Team Documentation and Training, Department Customer Services, BusinessUnit Saxess. Saxess 3.1 Administrative Client. Internal course document atOM, 2002.

[13] Team Documentation and Training, Department Customer Services, BusinessUnit Saxess. Saxess 3.1 Basic course. Internal course document at OM, 2002.

35

[14] Team Documentation and Training, Department Customer Services, BusinessUnit Saxess. Saxess Software Product Description. OM Technology, 2002.

[15] Team Documentation and Training, Department Customer Services, BusinessUnit Saxess. Saxess Trade Technical Reference. OM Technology, 2002.

[16] Team Documentation and Training, Department Customer Services, BusinessUnit Saxess. Saxview User’s Guide. OM Technology, 2002.

[17] Dan Tillberg. Introduction for new employees (in Swedish). Technical report,OM Technology, 2002.

36

Appendix A

Test Specification Administrative Client

Includes only the General Test Specification part from the original document writtenduring this project.

A.1 Window Checks

This section describes Window checks, i.e., tests checking that a window looks andfunctions as it should.

A.1.1 WC01 Change Record, Submarkets

Close all windows open within the Administrative Client. Open a “Submarkets”window. Check the attach name for the window (the title). Click the Next(down)button to change record in the selection list and check the attach name again.

Verify:

• The title name of the window changes, to match the Code and Id for theselected record.

A.1.2 WC02 Change Record, Orderbooks

Close all windows open within the Administrative Client. Open an “Orderbooks”window. Check the attach name for the window (the title). Click the Next(down)button to change record in the selection list and check the attach name again.

Verify:


37

A.1.3 WC03 Change Record, Participants

Close all windows open within the Administrative Client. Open a “Participants”window. Check the attach name for the window (the title). Click the Next(down)button to change record in the selection list and check the attach name again.

Verify:


A.1.4 WC04 Change Record, Submarkets LOOP

1. Close all windows open within the Administrative Client. Open a “Submar-kets” window.

2. Check the attach name for the window (the title).

3. Click the Next(down) button to change record in the selection list and checkthe attach name again.

4. Verify:


5. Repeat step 2-4 until the whole selection list has been gone through.

A.1.5 WC05 Change Record, Orderbooks LOOP

1. Close all windows open within the Administrative Client. Open an “Order-books” window.



4. Verify:



38

A.1.6 WC06 Change Record, Participants LOOP

1. Close all windows open within the Administrative Client. Open a “Parti-cipants” window.



4. Verify:



A.2 Edit and Save Checks

This section describes Edit and Save Checks, i.e., tests for verifying that it is possibleto edit data and then save it without problems.

A.2.1 ES01 Edit one field, Submarkets: stop loss

Close all windows open within the Administrative Client. Open a “Submarkets”window. Select a record in the selection list (NOEQ SHR, OSE Shares and PrimaryCapital Cert.). Click for Edit mode. Edit somewhere (click the Stop loss ordercheckbox to select/deselect) and click to Save.

Verify:

• That it is possible to save the edited data.

A.2.2 ES02 Edit one field, Orderbooks: round lot

Close all windows open within the Administrative Client. Open an “Orderbooks”window. Select a record in the selection list (003 BL, SSE Premium Bonds). Clickfor Edit mode. Edit the Round lot field and click to Save.

Verify:


A.2.3 ES03 Edit one field, Orderbooks: average matching

Close all windows open within the Administrative Client. Open an “Orderbooks”window. Select a record in the selection list (003 BL, SSE Premium Bonds). Clickfor Edit mode. Edit somewhere (Average matching - checkbox) and click to Save.

39

Verify:


A.3 Add New and Save Checks

This section describes Add New and Save Checks, i.e., tests for verifying that it ispossible to add information into the database, like new users or accounts and thensave without problems.

A.3.1 AS01 Add new Person

Close all windows open within the Administrative Client. Open a “Persons” window.Click the New button. Fill in the mandatory fields (Exchange, First name, Lastname, Country and Language) and click to Save (If a window is shown saying thata person with the same name already exists, click OK and go on anyway).

Verify:

• That it is possible to add and save a new person.

A.3.2 AS02 Delete added Person

Close all windows open within the Administrative Client. Open a “Persons” window.Click the Edit selections window to add Id.no to the columns shown in the selectionlist and click OK to save. Click the Id.no column name in the selection list to searchby id. Type the id for the recently added person in the search field and click theDelete button to delete the person. Confirm the deletion.

Verify:

• That the added person is deleted.

A.3.3 AS03 Add new User


Open a “Users” window and click the New button. Click the Id.no column namein the “Select Person” window to make the search by Id.no. Type the id for therecently added person in the Find field and click the OK button. Type the signfor the selected participant in the Find field of the “Select Participant” window thatappears and click the OK button. In the “New. . . ” window, type a sign for the Userand click to Save.

Verify:

40


• That it is possible to make the added person a user with user account at aparticipant and save everything.

A.3.4 AS04 Delete added User

Close all windows open within the Administrative Client. Open a “Users” window.Click the Edit selections button to add Id.no to the columns shown in the selectionlist and click OK to save. Click the Id.no column name in the selection list to searchby id. Type the id for the person behind the recently added user in the search fieldand click the Delete button to delete the added user. Confirm the deletion. Closethe “Users” window.

Open a “Persons” window. Click the Edit selections button to add Id.no to thecolumns shown in the selection list and click OK to save. Click the Id.no columnname in the selection list to search by id. Type the id for the person behind theadded, now deleted user in the search field and click the Delete button to delete theperson. Confirm the deletion.

Verify:

• That the added user is deleted.

• That the person behind the added, now deleted user is deleted.

A.3.5 AS05 Add new Broker


Open a “Brokers” window and click the New button. Click the Id.no column inthe “Use an existing person?” window to search by id. Type the id for the recentlyadded person in the Find field and click the Yes button. Make the Trainee checkboxunselected and click to Save.

Open a “Users” window and click the New button. Click the Id.no column in the“Select Person” window to search by id. Type the id for the recently added personin the Find field and click the OK button. Type the sign for the selected participantin the Find field of the “Select Participant” window and click the OK button. In the“New. . . ” window, type a sign for the User and click to Save.

Verify:


• That it is possible to make the added person a broker and save.

41

• That it is possible to make the added person a user and give him/her a useraccount at a participant and save everything.

A.3.6 AS06 Delete added Broker

Close all windows open within the Administrative Client. Open a “Users” window.Click the Edit selections button to add Id.no to the columns shown in the selectionlist and click OK to save. Click the Id.no column name in the selection list to searchby id. Type the id for the person behind the recently added broker in the searchfield and click the Delete button to delete the user account. Confirm the deletion.Close the “Users” window.

Open a “Brokers” window. Click the Edit selections button to add Id.no to thecolumns shown in the selection list and click OK to save. Click the Id.no columnname in the selection list to search by id. Type the id for the person behind theadded broker in the search field and click the Delete button to delete the broker.Confirm the deletion. Close the “Brokers” window.

Open a “Persons” window. Click the Edit selections button to add Id.no to thecolumns shown in the selection list and click OK to save. Click the Id.no columnname in the selection list to search by id. Type the id for the person behind theadded, now deleted broker and click the Delete button to delete the person. Confirmthe deletion.

Verify:

• That the broker is no longer a user.

• That the added broker is deleted.

• That the person behind the added, now deleted broker is deleted.

A.3.7 AS07 Add User in Participants

Close all windows open within the Administrative Client. Run the script Add newUser. Close all open windows again.

Open a “Users” window. Click the Edit selections button to add Id.no to thecolumns shown in the selection list and click OK to save. Click the Id.no columnname in the selection list to search by id. Type the id for the person behind therecently added user in the search field. Click the Delete button to delete the useraccount for the user. Confirm the deletion. Close the “Users” window.

Open a “Participants” window. Type the sign for the chosen participant in thesearch field to select it. Click the Users tab and click the New button. Click theId.no column name in the “Select person” window to make the search by Id.no. Typethe id for the person behind the recently deleted user account in the Find field andclick the OK button. Type a sign for the user in the “New. . . ” window that appearsand click to Save.

Verify:

42

• That it is possible to add a new user account for a person in the “Participants”window and save it.

A.3.8 AS08 Delete added user account in Participants

Close all windows open within the Administrative Client. Open a “Participants”window. Type the sign for the participant where the user recently has been addedin the search field to select it. Click the Users tab. Select the person behind theadded user and click the Edit button. In the account window that appears, clickthe Delete button to delete the user account. Confirm the deletion.

Run the script Delete added Person to delete the person behind the user. Closeall windows open within the Administrative Client.

Verify:

• That it is possible to delete the added user account from the “Participants”window.

43

automated client testing within saxess · pdf fileto automate the testing, ... primary system...

Documents