Test Accelerator for Service Oriented Architectures

(SOA-Test Accelerator)

Instituto Superior Tecnico – Universidade de Lisboa

Joao Filipe Garrett Paixao Florenciojoao.florencio@tecnico.ulisboa.pt

October 2015

Abstract

This document describes the design and development of a tool, “SOA – Test Accelerator”, created toautomate intelligent test creation in service oriented architectures. The need for such a tool comes fromthe increasing difference between testing some independent services and testing their overall interaction.As the system architecture grows in number of services, manually creating test case scenarios becomesan heavy burden. SOA-TA’s ultimate goal is to reduce the time spent on combining and orchestratingservice calls to simulate a business process. The work presented here is based on five stages and theiroutcome depends on the exhaustiveness level chosen by the user. First, the automatic generation of testcases through process descriptions analysis, having business requirements in consideration. Second, thegeneration of the input set required to execute these test cases. Third, the production of specific servicecalls, by means of test scripts to be run on Apache JMeter. Fourth, the execution of these scripts andfifth, showing the results. SOA-TA will be useful for operations that rely on consecutive service calls,and need to ensure the overall system compliance with previously set requirements.

Keywords:SOA testing,test case generation,test case execution,web services,service test-ing,coverage.

1 IntroductionWe cannot solve our problems with the same

thinking we used when we created them.-Albert Einstein

Nobody is perfect. Humans make mistakes allthe time. While some of them are negligible, someare expensive or dangerous. We are used to as-sume that our work may have mistakes, even minorones, and it needs testing. Software testing in par-ticular is now recognized as a key activity of largesystems development. Companies have come to un-derstand that, in order to achieve an higher qual-ity product, efforts on verification and validation,commonly known as V&V [1], save time and money.Service–Oriented Architectures (SOA) [2], as imple-mented by web services, present features that addmuch complexity to the testing burden. SOA, byits dynamic and always-on nature makes most test-ing techniques not directly applicable to it [8]. AsBartolini [10] states, “even if best practices are fol-lowed by the developer to test a service to ensureits quality, nothing guarantees that it will operatesmoothly as part of a dynamic distributed systemmade of multiple orchestrated but autonomous ser-vices.”

As service oriented systems are the pinnacle ofthis loosely coupled kind of systems, we need to en-sure more than just the functional integrity of itsunits, we need to be sure all the components arewell integrated. We are going to focus on the in-

tegration of several individual software modules, inour case web services to be combined and tested as agroup. In order for we to group all the componentsof a service based system we need to create testcases while focusing on the tasks those services willpreform. If the process of deriving test cases couldbe automated and provide requirements-based andcoverage-based test suites that satisfy the moststringent standard dramatic time and cost savingswould be realized [12]. This is what we are aimingto do.

1.1 Goals Motivation

In order to fill the gap between regular testing andmodern service oriented architecture testing, arisesthe SOA-Test Accelerator. The main objective ofour work was to create something that takes the re-sponsibility off the testers of coming up with intelli-gent and smart test paths or scenarios. Commercialtools are mainly software applications where testscan be executed, and results can be seen and ex-tracted. In all of them, the tester has to specificallycreate the test cases, give the inputs, and come upwith a testing strategy suitable for the process. Ourgoal was to create an on-line platform where theuser could enter the BPMN [13] description of thetasks to test and then our Test-Accelerator woulddo the rest. Which means creating the test casesbased on coverage standards, discover intelligently

1

the proper input data, run the tests, and ultimatelyprovide the user an overview of the results.

The main improvement or innovation on SOA-TA is the adaptation of concepts and ideas alreadystudied on graph analysis and satisfiability modulotheories, and bringing them to improve automatictesting. Therefore the main goal was to create asystem that would serve testers and their work, cre-ating value for the company using it.

2 Related Work

Tying the work we have done with previous researchis not an easy job due to the fact that we are join-ing two different study areas, automatic test casegeneration and service architectures testing plat-forms. Hence, the following sections will be dividedin this two major categories, the academic (mostlyresearch work) and the commercial testing frame-works.

2.1 Academic

Rayadurgam [12] worked on a method of automati-cally generate test-cases with model checkers, whichis a technique to exhaustively and automaticallycheck if a certain model meets a given specifica-tion [14]. The authors suggest a formal frameworkto describe both the system’s model and its speci-fications, which is then used alongside the test cri-teria to produce test-cases. The main algorithmrelies on generating a set of properties and thenask the model checker to verify them one by one.This method does not address the obstacle of statespace explosion. Also, creating a formal descrip-tion of simpler systems and models is a task thatnot everyone person could do, besides its neces-sity might be obliterated when using a different ap-proach like ours, in which a simple process descrip-tion in BPMN is used.

The approach chosen by Bai [15] relies on aWSDL document analysis to figure out dependen-cies between operations. The service contract isparsed and test data is generated by examiningthe message data types according to standard XMLschema syntax. Input and output dependencies re-veal which operations can be grouped in the sametest case. Although they are indeed automaticallygenerated and cover all the operations, this methodtests each service independently and not all the ser-vices use in a given process. Unless the whole sys-tem at hand is a unique service, analysing its oper-ation set does not suit integration testing.

2.1.1 Testing SOA

Ribarov et al [8] address testing in three majorfunctions, unit, integration and functional system

testing. In unit testing the authors propose the re-utilization of work already done in components test-ing and generate black box tests from the WSDLdocument, and take advantage of commercial toolspotentials (like the ones mentioned on section 2.2)to generate test-cases.

Integration testing is also challenging mainly dueto third party services dependencies and the pos-sibility of services being missing in the moment oftesting.

The authors state that it has to be accepted thatfor all but the simplest of services, it is very difficultto test exhaustively every input or output. Accord-ing to them, “Up to date there is no end-to-endautomated system testing solution for SOA on themarket.” [8] In Bartolini [10] is presented a solu-tion to go around a common trait existing test ap-proaches share, which is they treat the web servicesas black boxes and focus on the external behaviourignoring internal structure, as our SOA-TA does.

Their approach called Service Oriented Cover-age Testing (SOCT) relies on creating a governanceframework testing at the orchestration level duringvalidation. Using this method, it can monitor whatparts of actual source code are executed but it re-quires services developers to instrument the codeso as to enable target program entities executionmonitoring.

This method of addressing SOA testing showsgood results, nevertheless the need to modify ser-vices code to allow its instrumentation, cannot bedisregarded.

2.2 Commercial Testing Frameworks

In this section we are going review some tools com-mercially available today claiming to address theservice based architectures testing. We are not aim-ing to provide an exhaustive list of every single soft-ware platform that has some testing capabilities, wehave just selected a few of the biggest, most widelyused and most representative of the state-of-the-art.

2.2.1 HP Unified Functional Testing (UFT)

Hewlett-Packard has launched in 2012 a frame-work which provides functional and regression testautomation for software applications and environ-ments [22] It has incorporated a previous web ser-vices specific tool, HP Service test. It enables devel-opers to test, from a single console, all three layers:the interface, the service and the database layer.

As the other testing frameworks/tools presentedin this section, it claims to automate testing. Thisis, at some extent true since it uses a scripting lan-guage to make repetitive tasks automatic. Howeverit only automates independent testing of services,not the whole architecture testing process.

2

2.3 Related Work Critical Analysis 2.3 Related Work Critical Analysis

2.2.2 Oracle Testing Accelerators for WebServices

Oracle has launched in 2011 the 12c version ofits framework, Enterprise Manager, which includesTesting Accelerators for Web Services. It allowstesting the quality and performance of service-oriented architectures based applications directly atthe Web Service interface level. It claims to auto-mate functional and regression testing of servicesand uses an OpenScript platform to allow users togenerate scripts.

It materializes an idea very similar to the oneshown by Rayadurgam [12] aforementioned, testingservices on the interface level.

2.2.3 Apache JMeter

JMeter is a powerful, easy-to-use, and free load-testing tool. Since it is a pure Java application it iscompletely portable between OS’s. Although JMe-ter is known more as a performance testing tool,functional testing elements can be integrated withinthe Test Plan, which was originally designed to sup-port load testing. Many other load-testing toolsprovide little or none of this feature, restrictingthemselves to performance-testing purposes. Jme-ter was created to be first of all a load testingtool. It has evolved to support functional testingnonetheless we can see its performance testing fea-tures are what’s made its success. It does not sup-port WSDL parsing, all web services testing is donemanually, in a sense of knowing what are the typesand operations a web service supports or provides.

2.3 Related Work Critical Analysis

The work shown on 2.1 presents some capablestrategies to automatically generate test cases, how-ever they also present some issues like relying on asystem’s formal description or not having in con-sideration what coverage methods the user wantsto use. In the practical world there are few oper-ations requiring formal validation and verification,while almost every company basing its processes onweb services needs a way to know how reliable thetesting results are. That is what as motivated us touse a simple process description in the test genera-tion process.

Regarding commercial solutions, they all presentsome kind of automation, whether on test creationor test execution. No company presents a way of dy-namically perceive meaningful execution paths frombusiness process descriptions.

Script creation accelerators are also a key featurein some solutions to allow efficient manual creationprocedure. Our approach is based on automaticscrip creation based on the execution graph.

One of the distinguishing features of SOA TestAccelerator is finding and certifying the minimum

number of test scenarios and automatic providingwith proper input. As stated in the beginning, theSOA-TA final stage is the script generation. Thisscript would have to be executed in one of the test-ing tools mentioned in this section. To do this,we have taken into account the financial cost ofsoftware licenses/selling price, the available docu-mentation and market share they have and choseApache JMeter [24] to be the script execution tool.

In conclusion, as we can see that most commercialtools have some common features like facilitatingtest creation by simple users, while giving advancedusers/developers a way to extend them; all rely ona visual interface to simplify test creation. Most ofthem use some scripting language (OpenScript, Vi-sual Basic, BeanShell, etc.). However the most de-cisive factor is that all solutions assume users knowhow to test their system to get an acceptable cov-erage level, while SOA-TA knows exactly what andhow to test in order to comply with formal cover-age rules. It is not focused on test programmingbut on test conceptual creation. In the next sec-tion, we are going to clarify some testing conceptsand standards.

3 Background

In this section, we are going to provide a descriptionof the guiding metrics used in the system develop-ment. As said before, since SOA-TA is a testingtool and since WinTrust, the company funding theproject, has a close relationship with ISTQB (In-ternational Software Testing Qualifications Board),we made an effort to follow its guidelines.

3.1 Coverage Metrics

From our perspective, testing an architecture or-chestration of interconnected services, implies mak-ing sure information produced by one service or op-eration is suitable to serve as input to the next ser-vices in the process. Our first task was to find asolution to accurately model services, its calls, theinformation produced and workflow of a singularor multiple tasks. Then, we would apply coveragerules to check how the task tests should be created.

3.1.1 The Model and Diagram

Service calls specific flow, lead us to choose a state-based testing approach. State-based testing is idealwhen we have sequences of occurring events andconditions that apply to those events [26], mean-ing the proper handling of a situation depends onthe events and conditions that have occurred in thepast.

There are several types of state machines like fi-nite automatons (no guards or actions), Mealy ma-chines (outputs depend on current state and in-

3

puts), Moore machines (outputs depend on currentstate only), state charts (hierarchical states) etc.We have concluded that a Mealy machine is whatdescribes better our system since the input for eachstate will determine output.

Having said this, each service operation in ourmodel will be represented as an individual state.In the case of subsequent service calling (the usualin Service Oriented architectures) all the other ser-vices, which can be called “from” the first, will berepresented as another state.

A classic form of state diagram for a finite statemachine is the directed graph [28]. The notionof graph proved itself very useful since there isa great amount of work already done on extract-ing meaningful information from them (efficientsearches, path-cost analysis, augmentation paths,etc.) [29, 30]. The type of graph we will use is aDirected Graph (DG). In graphs notation there aretwo main concepts, vertices (or nodes) and edges.A vertex of a DG can have incoming vertices andoutgoing vertices [31]. We can describe a graphin mathematical terminology as G = (V,E, V0, Vf ),where

• V is a set of nodes

• V0 is a set of initial nodes, where V0 ⊆ V

• Vf is a set of final nodes, where Vf ⊆ V

• E is a set of edges, where E is a subset of V ×V

3.1.2 Switch Coverage

(N − 1) Switch Coverage, also named“Chow′s switch coverage”, after ProfessorChow, who developed it, states that at least onetest must cover each transition sequence of lengthN or less. Therefore, N represents the length ofthe transition sequence tested. By this descriptionis easy to understand that 1 − switch coverageextends 0− switch coverage. A higher level alwaysassumes the lower levels are also covered. SOA-TAsupports 0 − switch coverage (or Chow − 0) and1− switch coverage (or Chow− 1) test generation.

“0-switch coverage” or “Chow-0” (N=1)

This is the lowest level of coverage used in our sys-tem. It covers only transitions of length 1. It simplytests one service calling another and that is it, for alltransitions in the graph. In the example on Fig. 1,all transitions (edges) of the graph would be testedonce.

From our perspective however, since vertices/n-odes represent steps of an whole joint process, it isonly acceptable to test them in an end-to-end mode,from the first step to the last. So, to achieve a cov-erage of Chow − 0 as mentioned above, we wouldhave to generate two different test cases. One exer-cising the path A→1 B →3 C →4 E, and the otherA→2 C →5 D →6 E.

Astart

B

C

ED

1

23

4

56

Figure 1: Example graph - 1 – Directed Graph

“1-switch coverage” or “Chow-1” (N=2).

This is the higher level of coverage used in our sys-tem. It covers transitions of length 2. It tests alltwo consecutive service calls. On Fig. 1, there are6 transitions of length 2. To achieve Chow − 1 wewould have to tests all pairs of consecutive edges,edges 1 and 3, 3 and 4, 3 and 5, 5 and 6, 2 and 5,2 and 4. So, to achieve a coverage of Chow − 1,we would have to generate at least 4 different testcases, like:

• A→1 B →3 C →4 E

• A→2 C →4 E

• A→1 B →3 C →5 D →6 E

• A→2 C →5 D →6 E

4 Solution Description

In this section, we are going to provide a full de-scription of our system. As mentioned before,sections following 4.1 describe consecutive runningstages.

Figure 2: General System Description

4.1 Modelling the process

We decided to use BPMN to do the modelling pro-cess with the help of Bizagi Modeler, a freewaretool.

Figure 3 shows an example of a possible businessprocess involving five different services, each with

4

4.2 Graph Generation 4.2 Graph Generation

one or more operations executed. As Fig. 3 de-

Figure 3: Process Example 1 - Bizagi Modeler

picts, “Process 1” starts when “Service A OperationA 1” is called and ends when “Service B OperationB 2”, “Service D Operation D 1” or “Service E Op-eration E 1” return. The information seen on theedges might represent one of two different thingsdepending on its source. If it is a task, it representsits output data, on the other hand if it is a gateway,it contains the condition that makes the executionfollow that direction. We will see on the input datageneration section, Section 4.4, how decisive thisedge notes will be. Once the process is created, weexport the model to BPMN file, a XML-like file withall the data required to rebuild the model outsidethe tool.

4.1.1 Tibco Logs Exception Case

Besides reading BPMN files, WinTrust requestedthat we could also build the process descriptiongraph from a log file produced by the Tibco plat-form [36]. Tibco platform is a middleware responsi-ble for managing SOA’s execution and performance.Including this option would be an exception caseand that is why we are not focusing our attentionon it, nonetheless all the shown procedures are com-mon to the both ways of modelling processes andconstructing graphs.

4.2 Graph Generation

The graph generation is the following stage of SOA-TA. Using a XML parser, we will store the nodesand edges information from the BPMN file in away suited to preform graph searches. Upon graphgeneration some alterations are made, two dummynodes are added, the process start and end, andan edge connecting them. The necessity for suchmodifications which will explained in the followingsections.

4.3 Test Path Generation

The goal of the stage of the process is to producea test case for each business process scenario. Inthe graph terms, this means coming up with a pathcovering all edges at least once. This is analogue toprobably the best-known combinatorial optimiza-tion problem [37], called the “Chinese Postman

Problem” or the “Route Inspection Problem”. Inthis classical problem, we assume there is a post-man who needs to deliver mail to a certain neigh-bourhood. The postman is unwilling to walk unnec-essary miles, so he wants to find the shortest routethrough all the neighbourhood.

Solving this problem within our graph we wouldhave a path covering all transitions, all service op-eration calls. To solve this problem, we use thestrategy documented on Costa [38]. For this strat-egy to work, based on finding Eulerian circuits, itrequires that the graph must be strongly connected,i.e. all nodes of the graph must accessible from anyone of them.

In the 1 − switch coverage case, this approachwill not work only for itself. Here, the system mustgenerate a test case in which all pairs of consecutivetransitions (length 2 sequences) should be tested atleast one time.

To solve this, we came up with a solution basedon what is known in graph theory as path contrac-tion [39]. Path contraction is the process of takingall the edges of a path and contracting them to forma single edge between the two endpoints of the path.If we take all paths of length two of a graph, in otherterms, all pairs of consecutive transitions, and forma new graph with its contracted paths, we wouldbe able to use the Chinese postman problem’s so-lution again. The idea is to turn each pair of twoconsecutive edges of the graph into one edge andstill maintain graph consistency. The pseudo-codedescription of the algorithm we propose to do thisis described in the listing Algorithm. 1.

Input: original graphResult: new graphforeach edge e1 in original graph do

if new graph not containsNode (id equalse1.id) then

create node(id=”e1”) in new graph;endforeach edge e2 consecutive to e1 do

if new graph not containsNode (idequals e2.id) then

create node(id=”e2”) in new graph;endcreate edge(from e1 to e2)in new graph;

endreturn new graph

endAlgorithm 1: Path Contraction Algorithm

This algorithm essentially relies on two simplesteps. First, we take every edge of the original graphand transform it to a node in the new graph. Then,connect only the nodes which, in the original graph,were consecutive edges. We will end up having inthe new graph, edges which represent two consecu-tive edges of the original one. Solving the Chinesepostman problem in this new graph, will provide

5

4.4 Input Data 4.4 Input Data

the paths in which all pairs of consecutive edges aretested. Figures 4 and 5 depict an example of whatthis algorithm accomplishes.

A B C D1 2 3

Figure 4: Example graph - 4 Before

1 2 3A− > B− > C B− > C− > D

Figure 5: Example graph - 4 After

4.4 Input Data

Once the execution graph paths are generated bythe methods described in the last sections we needto execute them. To test a specific path, a numberof service operations need to be executed. In regularcases, most operations require some type of inputdata.

Our goal in this stage is to provide semi-automatically input values which will exercise thepaths previously created. If for example, we hadthree service operations in a row and if we alreadyknew that the only way of the task execution to gothrough all of them was if the first had the input“false” and the second and third, the input ”true”,we would automatically feed them with this combi-nation. That is the reason why we need the edgeconditions mentioned on Section 4.1. When there isa fork in the process model, we need to know whatis the condition the last operation output needs tomeet to go one way or another, as shown on Fig. 6.

A

B

C

Output > x

Output ≤ x

Figure 6: Example graph - 5

What we are going to do is prompting the serviceowners to build the BPMN with the conditions ofeach decisive edge and then solve the whole pathwith the restrictions gathered. When we have allthe path operations’s output restrictions it is timeto discover what inputs will fit, i.e, what input datawill produce results matching those restrictions. Todo this we do an auxiliary task of calling each stepoperation exhaustively (the number of executionscan be restricted) from whithin our code to get a

collection of input and output mappings. Then, weneed to solve the problem of assigning a value toeach input parameter in the whole path withoutbraking the restrictions on the edges and withoutmaking decisions that would invalidate the path insteps to come.

This is what in computer science is known as aSatisfiability Modulo Theories problem [40]. It is adecision problem for logical formulas with respect tocombinations expressed in classical first-order logic[41].

There are a number of works done in solving thiskind of problems but we used a tool named Z3, arelatively new SMT solver from Microsoft Research[41].

When given the correct data, the solver checks tosee if the formulas are satisfiable of not, and if so,it provides with a model which satisfies it [40].

4.5 Test Script Generation

SOA – Test Accelerator generated scripts to run onApache JMeter tool. Our goal is to combine datagathered in previous stages and create test plansto exercise the test cases. JMeter has a samplerspecifically designed to work with web services.

4.6 Test Script Execution and Read-ing Results

The .NET framework supports web service calls us-ing standard HTTP requests with a manually builtsoap envelope inside, however, since the JMeterscript was already created, it was simpler to juststart a JMeter process and pass the script as anargument. This is what we end up doing, start athread to execute the scripts and wait for it to bedone. There are several paramenters one can mea-sure when executing tests on webservices, such asresponse data and headers success/failure boolean,timestamp, hostname, latency, number of threads,elapsed Time etc. SOA-TA is able to detail on thescript which of this items to save and now it is con-figured to save response data, success, timestampand the elapsed time.

5 Implementation

In this section, we are going to go through the im-plementation of SOA-TA. As previously stated, itis based on a on-line platform to be accessed viaweb browser and here we are going to detail devel-opment decisions and all the components needed toits functioning. As any web application, function-ality is supported through a set of dynamic webpages, which in the end run a couple of scripts andretrieve information from the server.

Diagram on Fig.7 shows the flow of communica-tions in SOA-TA. Here, as in any other web ap-

6

5.1 Business Logic 5.1 Business Logic

plication, different functionalities are separated inmodules.

Figure 7: SOA-TA Communications Diagram

5.1 Business Logic

In software engineering, business logic or domainlogic refers to the part of the program that encodesthe real-world business rules that determine howdata can be created, displayed, stored, and changed,essentially the business core. To interact with it,users would have to use the operations it provided.In our case, this is where we implemented path gen-eration, which as been extensively described in sec-tion 4.3, algorithms to route inspection problem,data discovery, the parsers, script generation andScript Execution

5.2 Database and Data Access

All the SOA-TA produced data is persisted in a re-lational database in Microsoft SQL Server. Dataaccess is done via an object-relational mapper, Mi-crosoft’s Entity Framework, that allows program-mers to work with relational data using the domain-specific objects.

6 Results

In this section we are going to describe the evalua-tion preformed on SOA-TA. As it was already dis-cussed in previous sections, its operation dependson several components worthy of a independent as-sessment. Having this matter in consideration, wehave chosen to present the following results groupedin two sections, one with the coverage-based testpath generation results and the other with system’send-to-end testing.

6.1 Path Generation Results

Regarding the path generation stage, we are goingto show some results on process examples. Thereare, as it was mentioned on section 4.1.1 two typesof sources SOA-TA can use to build the directedgraph from where paths are extracted, BPMN filesand Tibco logs [36]. In the following examples wehave grouped a small set of each to demonstrate ouroutcomes.

Fig. 8 shows a first example of a small, yetreal-life process description. This graph was ex-tracted from a Tibco log from the TAP - Trans-

Figure 8: Example graph 6

portes Aereos Portugueses, a Wintrust client. Itportraits a simple flow with just one fork afterthe addMultiElementsPNR task. All graph fig-ures presented here originating from Tibco logswere automatically generated by a Quickgraph forC# [45]. For this example, SOA-TA generates thetwo distinct paths. The first: retrieveSbrFeed, in-sertSKFast, retrieveFrequentFlyerCluster, modify-Booking, signInAmadeus, retrievePNR, addMulti-ElementsPNR, signOutAmadeus. With the secondchanging only the last step, ending in the eighthstep, addMultiElementsPNR. In a small and simpleexample like this it is trivial to see that are twopaths exactly are required for us to test all transi-tions of size 1, 0 − Switch coverage. Fig. 9 showsa much more complex and also real-life process de-scription. This graph represents the operations pre-formed by TAP’s web services to make a ticketreservation for a staff member. It presents a struc-ture flow with so many transitions between opera-tions that makes human coverage analysis almostimpossible. For all these transitions, 0 − switchcoverage generated the paths contained on table 1.

Figure 9: Example graph 7 - TAP Staff Booking

It is difficult to manually check, in this case, butif we take a closer look we will see that no edge wasleft untested. All transitions were tested at leastonce.

7

6.2 End-To-End 6.2 End-To-End

Regarding the time consumption, we have reg-istered that for SOA-TA to generate all 7 paths,the one shown on 1 and six more to achieve the1−switch level, it took about 14 milliseconds, 4 onthe first path and 10 on the remaining.

Step Operation Name Step Operation Name

1 EmployeeTicket. 31 modifyTicket2 EmployeeTicket. 32 displayTST3 createBooking 33 ManualTicket4 signInAmadeus 34 retrievePNR5 retrievePNR 35 displayTST6 addMultiElem 36 Cryptic7 Cryptic 37 addMultiElem8 signInAmadeus 38 signOut9 retrievePNR 39 demandTickets10 demandTicket 40 demandTicket11 signInAmadeus 41 addMultiElem12 modifyTicket 42 ManualTicket13 displayTST 43 retrievePNR14 signInAmadeus 44 conIssTickets15 addMultiElem 45 addMultiElem16 retrievePNR 46 Cryptic17 demandTicket 47 modifyTicket18 signOut 48 conIssTickets19 retrievePNR 49 ManualTicket20 demandTicket 50 modifyTicket21 ManualTicket 51 Cryptic22 retrievePNR 52 signOut23 displayTST 53 retrievePNR24 addMultiElem 54 conIssTickets25 addMultiElem 55 Cryptic26 retrievePNR 56 Cryptic27 displayTST 57 ManualTicket28 signOut 58 retrievePNR29 modifyTicket 59 Cryptic30 addMultiElem

Table 1: TAP Staff Booking Generated Paths -Chow-0

We have also tested SOA-TA with process de-scriptions from Bizagi. Figure 10 shows an exampleof a simple process, however tasks 2 4 and 5 are onpurpose connected in a loop. The generated pathfor this example are shown on table 2

Figure 10: Process Example 2

Chow-0 Chow -1

Order Path 1 Order Path 2 Path 3

1 Task 1 1 Task 1 Task 12 Task 2 2 Task 2 Task 23 Task 4 3 Task 4 Task 44 Task 5 4 Task 5 Task 55 Task 1 5 Task 2 Task 16 Task 2 6 Task 4 Task 27 Task 4 7 Task 5 Task 38 Task 5 8 Task 2 Task 69 Task 2 9 Task 310 Task 3 10 Task 611 Task 6

Table 2: Process Example 2 - Generated Paths

6.2 End-To-End

To test our system from end-to-end we need to startfrom a process or task description, then model inBPMN, upload the file to SOA-TA, ask it to gener-ate test paths, provide static input, and then exe-cute the tests and see if something went wrong. Letus now follow the process from the beginning withan example. First, consider the following user storywhich will be our starting point.

“A few years ago I bought a few stock sharesof Apple computer and ever since its price boomI have considered selling them and buy a housein New York, a long time dream. However, firstI will have to make a few checks to see if this isthe right time to do it. First I will check how Applstocks are doing using getStockV alue service andhow much stocks I would need to have to get to500k dollars. If the values are too far away frommy personally established threshold of 400k dollarsa share, I will wait for some other time. Other-wise I want to use the currency converter web ser-vice, ConvertUSDtoEUR to see how much eurosI would have to put in to get to that budget. IfI have to invest less that 100k euros, I will evencheck for a house at walking distance from centralpark, something like 500meter converted to yards(Americans don’t fancy the metric system) withlength converter service, convertionRate . Eitherway, I want to know how is the weather back there,I will use GetCityWeatherByZIP operation of theweather web service, and... that’s right, they useFahrenheit, I’ll have to get it back to Celsius withtemperature converter service, ConvertTemp.”

This fictional user story would produce a processdescription like the one on Fig. 11. Note that wehave included the correct operations’s name for eachtask, this will simplify the path generation process.The services mentioned on the example are workingservices and can be found at www.webservicex.netunder the names “stockquote”, CurrencyConvertor,length, Weather and in w3schools.com the temp-

8

6.2 End-To-End 6.2 End-To-End

Figure 11: Process Example 5

convert.Now, this part is concluded, we will upload this

file to SOA-TA and make it generate the coveragepaths. Generation phase comes up with 4 differ-ent paths which are shown on tables 3 4. Each oneof these, has respectively paths generated from thelower and the higher exhaustiveness levels, Chow−0and Chow− 1. If we take a closer look we will thatin the first, all transitions of size 1 are covered bythe two paths, however without paths 3 and 4 fromthe table 4, not all transitions of size 2 would betested. The path number 2, highlighted in bold isrequired for both coverage levels. Also note that, incases of loops in the graph, as it is the case of theChangeLengthUnit node, may produce test pathsin which the same node gets tested three times ina row. This is conceptually accurate since the firstexecution comes from testing a previous transitionand the first loop iteration and second and third ex-ecutions come from executing the loop twice (tran-sition with length 2).

Step Path 1 Path 2

1 GetQuote GetQuote2 ConversionRate ConversionRate3 ChangeLengthUnit GetWeatherByZIP4 ChangeLengthUnit ConvertTemp5 GetWeatherByZIP6 ConvertTemp

Table 3: Example 5 - Chow-0 Test Paths

Step Path 3 Path 4

1 GetQuote GetQuote2 ConversionRate ConversionRate3 ChangeLengthUnit ChangeLengthUnit4 ChangeLengthUnit GetWeatherByZIP5 ChangeLengthUnit ConvertTemp6 GeWeatherByZIP7 ConvertTemp

Table 4: Example 5 - Chow-1 Test Paths

Now, with the paths already generated, we aregoing to automatically generate the input data nec-essary for the test to be run. As mentioned on sec-

tion 4.4, we use a SMT solver to aid us in this mat-ter. We are going to follow closely input data gen-eration for Path1 from table 3 nonetheless we couldhave chose any of the other 3 since the process issimilar.

So the path to generate input is as fol-lows: GetStockValue, ConvertUsdToEur, Change-LengthUnit, ChangeLengthUnit, GetCityWeather-ByZIP, ConvertTemp. Adding the restrictionson the edges, we get something like: GetStock-Value (GetStockValueResult>400k) ConvertUsd-ToEur (ConvertUsdToEurResult>400k) Change-LengthUnit, ChangeLengthUnit, GetCityWeather-ByZIP, ConvertTemp.

After assigning outputs of previous tasks toserve as inputs to other tasks and after assigningthe static input value of 500 and 200 meters toChangeLengthUnit steps respectively and the the10023 zip code to GetCityWeatherByZIP we canstart input discovery.

This will internally trigger the SMT solver whichwill take the two restrictions and build them in twofirst-order logic restrictions, like:

(> GetStockV alueResult 400000)(> ConvertUsdToEurResult 400000)Then we would preform a set of unit tests to both

GetStockV alue and ConvertUsdToEur with ran-dom values adding the input output relation as arestriction to the solver. The following values weregenerated when executing these tests.

Input Output Input Output

18 756 2098046 21 951 245543860002 6711898 62 146 695165128 715 3212059 43 518 48679238 684 971392 85 889 960754342284 4729928 61 124 68373301 242 138930 63561 710999969 836 7811854 85331 954513424 801 2774239 32 055 3585672

Table 5: GetStockValue - Unit test results (16 sam-ples)

With the results from table 5 from theGetStockV alue operation, we would take all theoutputs and feed it to ConvertUsdToEur. Table 6shows the obtained outputs. Note, that our inten-tion was to get an output from ConvertUsdToEurof less than 400k, we have clearly succeeded withmost of these result, only one is bellow that thresh-old.

With all these results gathered, SOA-TA will jointhis results in form of first-order logic restrictionsalso. For each entry in the table 5 it will producean implication and a logic OR restriction like thefollowing:

(=> (= GetStockValueInput 18756) (= Get-StockValueOutput 2098046,16))

9

Input Output Input Output

2 098 046 1 846 280 2 455 438 2 160 7866 711 898 5 906 470 6 951 651,56 6 117 4533 212 059 2 826 612 4 867 923 4 283 772

971 392 854 825 9 607 543 8 454 6384 729 928 4 162 336 6 837 330 6 016 850

138 930 122 258 7 109 999 6 256 7997 811 854 6 874 432 9 545 134 8 399 7172 774 239 2 441 331 3 585 672 3 155 391

Table 6: ConvertUsdToEur - Unit test results (In-put from table 5)

(or (= GetStockValueInput 18756))

The same for table 6 and one for the relationbeetween GetStockValueOutput and ConvertUsd-ToEurInput, like the folowing:

(=> (= ConvertUsdToEurInput 2098046,16) (=ConvertUsdToEurOutput 1846280,621))

(or (= ConvertUsdToEurInput 2098046,16))

(= ConvertUsdToEurInput GetStockValueOut-put))

SOA-TA will then use all the restrictions givenand will check the formula for satisfiability. Ifthe formula is satisfiable, it can produce a modelwhich will have possible value for all the unknownvariables we might have. In this case, it producedthe following values: GetStockV alueInput =18756, GetStockV alueOutput =2098046, 16, ConvertUsdToEurInput =GetStockV alueOutput,ConvertUsdToEurInput =2098046, 16, ConvertUsdToEurOutput =1846280, 621. In the end of this phase wehave all the inputs necessary to create the script,from the ones manually set as the distance inthe ChangeLengthUnit step or weather valuesin GetCityWeatherByZIP which are less relevantin this case, to the automatically generated andvalidated by the SOA.

The process of script generation is simple. All wedo is gather all these inputs, outputs, service oper-ations and WSDL descriptions in a XML file withspecific JMeter rules. Each step is an instance ofthe “RPC/SOAP sampler” which carries a regularSOAP envelope. To save an operation result to useit as input to the next step, we make use of a reg-ular expressions extractor and save the result in aJMeter variable.

To execute the script, as it was already mentionedin section 4.6 we start a command line process andprovide the fie path to be run. JMeter then, exe-cutes the script, saves the general results in a textfile and the partial results of each step in a answerfile. This file contains the usual XML contained inone SOAP response message. SOA parses this dataand stores results in the database. For the examplewe were testing JMeter produced the following re-suls file (the values are respectively the timeStamp,

elapsed time, responseCode, responseMessage andsuccess boolean):

1442508850,783,200,OK,true1442509634,827,200,OK,true1442510462,827,200,OK,true1442511290,769,200,OK,true1442512163,873,200,OK,true1442512955,792,200,OK,trueThe GetCityWeatherByZIP step with 10023, a

New York ZIP code, as input, returned 88o Fahren-heit, which after ConvertTemp converted to 31.11o

Celsius. For all other paths built from this taskdescription a similar process would take place.

7 Conclusions

As we have seen, service oriented architecturespresent many challenges on what testing is con-cerned. On the other hand, its major spreading andappeal makes these issues unavoidable. Our workdoes not aim to be the solution to all the problemsfaced when testing this kind of systems but is surelya good option when the intention is to speed up theprocess whilst maintain all test robustness.

We have studied what other professionals havedone and we are confident that the future of SOAtesting will have a big part of automation in it. Wealso hope our work will serve academic research andcommercial world interests. As Isaac Asimov oncesaid, “If knowledge can create problems, it is notthrough ignorance that we can solve them”.

References[1] B. W. Boehm, “Verifying and validating software re-

quirements and design specifications,” IEEE software,vol. 1, no. 1, p. 75, 1984.

[2] T. Erl, Soa: principles of service design. Prentice HallUpper Saddle River, 2008, vol. 1.

[3] S. Oasis, “Reference model tc,” OASIS Reference Modelfor Service Oriented Architecture, vol. 1, 2005.

[4] P. Offermann, M. Hoffmann, and U. Bub, “Benefits ofsoa: Evaluation of an implemented scenario against al-ternative architectures,” in Enterprise Distributed Ob-ject Computing Conference Workshops, 2009. EDOCW2009. 13th. IEEE, 2009, pp. 352–359.

[5] R. Heffner, C. Schwaber, J. Browne, T. Sheedy,G. Leganza, and J. Stone, “Planned soa usage growsfaster than actual soa usage,” Forrester Research,vol. 28, 2007.

[6] M. Hedin and IDC, Worldwide SOA-Based Services2007-2011 Forecast and Analy-sis: A Maturing SOAMarket Fuels New and Different Demands for Profes-sional Services, Apr. 2007.

[7] S. Mulik, S. Ajgaonkar, and K. Sharma, “Where do youwant to go in your soa adoption journey?” IT Profes-sional, vol. 10, no. 3, pp. 36–39, 2008.

[8] L. Ribarov, I. Manova, and S. Ilieva, “Testing in aservice-oriented world,” 2007.

[9] G. Canfora and M. Di Penta, “Soa: Testing and self-checking,” in International Workshop on Web Services–Modeling and Testing (WS-MaTe 2006), 2006, p. 3.

10

REFERENCES REFERENCES

[10] C. Bartolini, A. Bertolino, S. Elbaum, and E. Marchetti,“Whitening soa testing,” in Proceedings of the the 7thjoint meeting of the European software engineering con-ference and the ACM SIGSOFT symposium on Thefoundations of software engineering. ACM, 2009, pp.161–170.

[11] F. A. Authority, “Software considerations in airbornesystems and equipment certification, 1992. documentno,” RTCA/DO-178B, RTCA, Inc, Tech. Rep.

[12] S. Rayadurgam and M. P. Heimdahl, “Coverage basedtest-case generation using model checkers,” in Engineer-ing of Computer Based Systems, 2001. ECBS 2001.Proceedings. Eighth Annual IEEE International Con-ference and Workshop on the. IEEE, 2001, pp. 83–91.

[13] S. A. White, “Introduction to bpmn,” IBM Coopera-tion, vol. 2, no. 0, p. 0, 2004.

[14] E. M. Clarke, O. Grumberg, and D. Peled, Model check-ing. MIT press, 1999.

[15] X. Bai, W. Dong, W.-T. Tsai, and Y. Chen, “Wsdl-based automatic test case generation for web ser-vices testing,” in Service-Oriented System Engineer-ing, 2005. SOSE 2005. IEEE International Workshop.IEEE, 2005, pp. 207–212.

[16] D. Jordan, J. Evdemon, A. Alves, A. Arkin, S. Askary,C. Barreto, B. Bloch, F. Curbera, M. Ford, Y. Golandet al., “Web services business process execution lan-guage version 2.0,” OASIS standard, vol. 11, no. 120,p. 5, 2007.

[17] L. Reitman, J. Ward, and J. Wilber, “Service orientedarchitecture (soa) and specialized messaging patterns,”A technical White Paper published by Adobe Corpora-tion USA, 2007.

[18] P. Kalamegam and Z. Godandapani, “A survey on test-ing soa built using web services,” International Journalof Software Engineering and Its Applications, vol. 6,no. 4, 2012.

[19] B. Beizer, Black-box testing: techniques for functionaltesting of software and systems. John Wiley & Sons,Inc., 1995.

[20] T. Ostrand, “White-box testing,” Encyclopedia of Soft-ware Engineering, 2002.

[21] W.-T. Tsai, R. Paul, W. Song, and Z. Cao, “Coyote: Anxml-based framework for web services testing,” in HighAssurance Systems Engineering, 2002. Proceedings. 7thIEEE International Symposium on. IEEE, 2002, pp.173–174.

[22] Hp uft data sheet. [Online]. Avail-able: http://www8.hp.com/h20195/v2/GetDocument.aspx?docname=4AA4-8360ENW

[23] Parasoft soatest data sheet. [Online]. Available:https://www.parasoft.com/product/soatest/

[24] E. H. Halili, Apache JMeter: A practical beginner’sguide to automated testing and performance measure-ment for your websites. Packt Publishing Ltd, 2008.

[25] A. Gargantini and C. Heitmeyer, “Using model check-ing to generate tests from requirements specifications,”in Software Engineering—ESEC/FSE’99. Springer,1999, pp. 146–162.

[26] R. Black, Advanced Software Testing-Vol. 1: Guideto the ISTQB Advanced Certification as an AdvancedTechnical Test Analyst. Rocky Nook, Inc., 2014.

[27] T. S. Chow, “Testing software design modeled by finite-state machines,” IEEE transactions on software engi-neering, no. 3, pp. 178–187, 1978.

[28] T. L. Booth, Sequential machines and automata theory.Wiley New York, 1967, vol. 3.

[29] N. Christofides, “Worst-case analysis of a new heuristicfor the travelling salesman problem,” DTIC Document,Tech. Rep., 1976.

[30] H. Imai and T. Asano, “Efficient algorithms for geomet-ric graph search problems,” SIAM Journal on Comput-ing, vol. 15, no. 2, pp. 478–494, 1986.

[31] E. R. Gansner, S. C. North, and K.-P. Vo, “Dag—a pro-gram that draws directed graphs,” Software: Practiceand Experience, vol. 18, no. 11, pp. 1047–1062, 1988.

[32] T. Takagi, N. Oyaizu, and Z. Furukawa, “Concurrentn-switch coverage criterion for generating test casesfrom place/transition nets,” in Computer and Infor-mation Science (ICIS), 2010 IEEE/ACIS 9th Interna-tional Conference on. IEEE, 2010, pp. 782–787.

[33] D. Schumm, D. Karastoyanova, F. Leymann, andJ. Nitzsche, “On visualizing and modelling bpel withbpmn,” in Grid and Pervasive Computing Conference,2009. GPC’09. Workshops at the. IEEE, 2009, pp.80–87.

[34] J. C. Recker, “Bpmn modeling–who, where, how andwhy,” BPTrends, vol. 5, no. 3, pp. 1–8, 2008.

[35] M. Owen and J. Raj, “Bpmn and business process man-agement,” Introduction to the New Business ProcessModeling Standard, 2003.

[36] P. E. P. Tibco and I. P. D. U. Guide, “Tibco softwareinc,” 2001.

[37] H. A. Eiselt, M. Gendreau, and G. Laporte, “Arc rout-ing problems, part i: The chinese postman problem,”Operations Research, vol. 43, no. 2, pp. 231–242, 1995.

[38] J. C. C. Costa, “Coverage-directed observability-basedvalidation method for embedded software,” Ph.D. dis-sertation, INSTITUTO SUPERIOR TECNICO, 2010.

[39] R. E. Tarjan, “Applications of path compression on bal-anced trees,” Journal of the ACM (JACM), vol. 26,no. 4, pp. 690–715, 1979.

[40] L. De Moura and N. Bjørner, “Satisfiability modulo the-ories: introduction and applications,” Communicationsof the ACM, vol. 54, no. 9, pp. 69–77, 2011.

[41] L. De Moura and N. Bjorner, “Z3: An efficient smtsolver,” in Tools and Algorithms for the Constructionand Analysis of Systems. Springer, 2008, pp. 337–340.

[42] Jmeter api. [Online]. Available: http://jmeter.apache.org/api/

[43] G. Mulligan and D. Gracanin, “A comparison of soapand rest implementations of a service based interac-tion independence middleware framework,” in Simula-tion Conference (WSC), Proceedings of the 2009 Win-ter. IEEE, 2009, pp. 1423–1432.

[44] K. Wagh and R. Thool, “A comparative study of soapvs rest web services provisioning techniques for mobilehost,” Journal of Information Engineering and Appli-cations, vol. 2, no. 5, pp. 12–16, 2012.

[45] J. de Halleux, “Quickgraph: A 100% c# graph librarywith graphviz support,” 2007.

11