Ann TelecommunDOI 101007s12243-014-0442-7

A methodology for validating cloud models usingmetamorphic testing

Alberto Nunez middot Robert M Hierons

Received 3 November 2013 Accepted 20 June 2014copy Institut Mines-Tlcom and Springer-Verlag France 2014

Abstract Cloud computing is a paradigm that providesaccess to a flexible elastic and on-demand computinginfrastructure allowing users to dynamically request virtualresources However researchers typically cannot experi-ment with critical parts of cloud systems such as theunderlying cloud architecture resource-provisioning poli-cies and the configuration of resource virtualisation Thisproblem can be partially addressed through using simula-tions of cloud systems Unfortunately the problem of testingcloud systems is still challenging due to the many param-eters that such systems typically have and the difficultyin determining whether an observed behaviour is correctIn order to alleviate these issues we propose a method-ology to semi-automatically test and validate cloud mod-els by integrating simulation techniques and metamorphictesting

Keywords Metamorphic testing middot Cloud computing middotSimulation and modelling

1 Introduction

Cloud computing is a paradigm that provides access to aflexible and on-demand computing infrastructure by allo-

A Nunez ()Computer Science Faculty University Complutense de MadridMadrid Spaine-mail albertonunezpdiucmes

R M HieronsSchool of Information Systems Computing and MathematicsBrunel University Brunel UKe-mail robhieronsbrunelacuk

wing the user to rent a number of virtual machines for aspecific time slot Currently this paradigm is being adoptedby several major enterprises in communications such asGoogle IBM Microsoft and Amazon All of these organi-sations have invested billions of dollars in order to providetheir own cloud computing solutions In fact the market isexpected to rise from US$407 billion in 2011 to more thanUS$241 billion in 2020 [15]

The underlying complexity of cloud systems leads to test-ing being expensive and requiring much time and effortThus it is desirable to design and develop formal test-ing methodologies for checking cloud systems [1] Usuallyformal testing approaches involve analysing the outputsgenerated by the system under test If an analysed output isnot the expected one then the system under test has failedthis test The mechanism that reliably decides whether a testis passed or failed is called an oracle Unfortunately in somesituations an oracle is not available or it is computationallytoo expensive to apply the oracle and alternative approachesmust be used [18] This problem arises in cloud systemswhere there rarely is an oracle indicating whether the designof a cloud system is correct

In general we can say that a cloud system ldquoworksrdquo ifthis system is able to perform several pre-defined tasks Forexample let us consider a data center with 10000 physi-cal machines a communication network that interconnectsthem and a collection of services that provide virtualisedresources for users that is a cloud system If different usersare able to rent virtual machines (in short VMs) and exe-cute applications using these VMs we can state that thiscloud system works However typically a system has tofulfil many additional conditions Consider a cloud systemwhere each machine contains a quad-core processor If thecloud manager only uses two out of four CPU cores fromeach machine then the cloud system is wasting half of its

Ann Telecommun

computing power However the users can still use the sys-tem by renting VMs and executing applications because thenumber of physical cores managed by the system is trans-parent to them In this case we can state that the systemdoes not work correctly there is inbuilt inefficiency

In testing [12] validating or optimising a cloud system itis often necessary to customise the management aspects ofthe system like resource provisioning policies hypervisorsand configuration of virtualised resources The process ofadapting such aspects is typically expensive and time con-suming and in some cases the researcher does not haveaccess to a cloud system for which they have appropriatepermissions These factors have led to increasing interest insimulation Usually a cloud system modelled using simula-tion techniques consists of thousand of machines networksswitches users and other management modules It may benecessary to assign values to thousands of parameters whenconfiguring this model and so it usually is not feasible to testevery possible input and check the outputs produced More-over a single change in a module requires regression testingto be applied and this involves the tests being run again andthe test output being checked further increasing the costof testing These factors make testing expensive with thechecking of test output (the oracle) often being manual andso significantly contributing to the cost

In this paper we propose a methodology that integratesa complete simulation platform for modelling cloud com-puting systems with testing methods for checking the cor-rectness of modelled cloud systems The main goal of ourresearch is to provide a mechanism that allows users tomodel both software and hardware parts of cloud systemsdesign new cloud system models and automatically testthese models in a cost-effective manner The main advan-tages of the proposed methodology can be summarised asfollows

ndash Scalability The size of the simulated cloud can varyfrom several computers to thousand of machines

ndash Costless Using our proposed methodology does notrequire specific hardware to be executed Also a cloudis no longer required to perform experiments becausethe simulation platform can be executed in any com-puter

ndash Automatic testing Once users provide the requiredinputs for the testing process the cloud model is auto-matically checked in order to analyse its correctness

The rest of the paper is structured as follows Section 2describes the motivation for integrating a simulation plat-form and metamorphic testing (MT) Section 3 describesin detail our methodology for testing cloud systems usingmetamorphic testing Section 4 presents the results of exper-iments Finally Section 5 presents our conclusions andsome directions for future work

2 Motivation

There are several factors that make analysing or reason-ing about the underlying architecture of a cloud systemparticularly challenging First cloud systems are usuallyvery large and this fact hampers the analysis and study ofthese systems Second virtualisation that is the resourcesof the cloud provided to end users are virtual introducesadditional complications because different VMs can behosted in a single machine sharing the same resourcesamong different users Finally we cannot oversee thevast number of users that are concurrently using a cloudsystem

It is important to emphasise that cloud systems are builtout of tens of thousands of commodity machines wherea simple failure in the system may produce catastrophicconsequences Therefore ensuring the good functioning ofthese systems is a priority As an example in 2011 AmazonEC2 suffered an unexpected crash during network reconfig-uration [8] This crash affected more than 70 organisationsincluding FourSquare the New York Times Quora and Red-dit in some cases causing sites to remain offline for manyhours

In recent years simulation has become a widely adoptedloosely formalised approach for testing cloud systems Basi-cally simulators build a model of the system to be simu-lated such that it imitates the behaviour of the target systemand then different measures like performance and powerconsumption are gathered by observing how the modelworks Researchers have designed cloud models and thenperformed ad hoc testing by manually simulating differentscenarios Among the available simulation tools that canbe used to model and simulate cloud computing environ-ments are CloudSim [3] GreenCloud [11] SimGrid [5] andiCanCloud [14]

There are several research articles in the case ofCloudSim which focus the obtained results on this sim-ulator [2 4 10] This tool was initially based on a gridsimulator [3] (this being GridSim [16]) A new layer wasimplemented on top of GridSim to add the possibility ofsimulating cloud systems Since then CloudSim has beenre-designed from scratch and does not rely on GridSim any-more However CloudSim still has some drawbacks Forinstance there are no models for creating communicationsbetween virtual machines in several data centers Anotherdrawback is the fact that it is not possible to simulateoverload in nodes

GreenCloud is an extension for the NS2 network simu-lator [17] GreenCloud is focused on simulating the com-munications between processes running in a cloud at packetlevel In the same way as NS2 it is written in C++ andOTcl being a disadvantage for this tool since two differentlanguages must be used to implement a single experiment

Ann Telecommun

GreenCloud provides plug-ins that allow the use of physicallayer traces that make experiments more detailed

SimGrid is a simulator that provides core functionalitiesto simulate algorithms and distributed applications in dis-tributed computing platforms which go from workstationsto grid environments [5] The resources are modelled bytheir latency and service rate The topology is configurableIndeed defining the topology is the userrsquos responsibil-ity whereas collections of links may be used to simulatecomplex mechanisms like routers

In this work we use the iCanCloud simulation frame-work to represent the behaviour of cloud systems More-over an additional framework called E-mc2 [6] is used tomodel the energy consumption of each hardware device inthe cloud Both simulation frameworks are currently avail-able under the GPL3 license at httpwwwiCanCloudorg There are three main reasons for using these simula-tion frameworks First these simulators are open sourceThus the source code is available and can be modified ifrequired Second these simulators provide highly detailedmodels of both the hardware part of the cloud systemand the virtualisation schema Third there is a GUI thatallows the environment to be easily configured Moreoverthe specific motivations for using simulation in our workare

ndash Simulation is cheaper than performing experimentsdirectly in a real cloud

ndash The level of flexibility obtained by using simulation ismuch higher than when using real cloud systems

ndash Scalability In particular the number of VMs that asingle user may rent in public clouds is limited Incontrast in a simulated environment cloud systemscan be modelled with a fully customisable number ofmachines

ndash Researchers can share simulation models

This paper also investigates the application of Metamor-phic Testing (in short MT) to the testing of cloud systemssince such systems are typically complex and often have nooracle

MT was developed in order to test systems where thereis no oracle or it is expensive to compute the oracle Theessential idea is that instead of checking the output o1 pro-duced when testing with one input x1 we test with a second(follow-up) input x2 observing output o2 and check that o1

and o2 are related as expected Thus in MT there are tworelations the relation between the original test input x1 andthe follow-up input x2 and the expected relation betweenthe two outputs

Consider for example the problem of checking a pro-gram f that should be an implementation of the trigono-metric sine function While it may be difficult to checkwhether the application of f to an input is correct we know

a number of properties of the sine function and any cor-rect implementation should have these properties One suchproperty is that sin(minusx) = minussin(x) In order to checkthis metamorphic relation having tested f with an inputwe also test f with the negation of the original input andcheck the above property If the property does not hold thenf must be faulty and it must have failed on at least one ofthe two inputs (the original value and its negation) In thispaper we adapt the idea of MT to cloud systems

There are three main reasons for using MT in this workFirst it has been shown in different application scenariosthat MT alleviates the oracle problem providing a betterapproach for testing large and complex scenarios than con-ventional testing techniques Second the effectiveness ofthe proposed methodology can be increased by adding newmetamorphic relations (in short MRs) MRs can be sharedamong different research groups in order to increase theapplicability to a wider range of cloud configurations Thirdit is possible to concentrate the testing effort on a particu-lar feature of the system by using specific MRs targetingeither a specific characteristic or a part of the system Thiscan be easily done if MRs are grouped in categories eachone being responsible for analysing a specific feature of thecloud system

MT has been used in the past to test Web Services [7]focusing efforts on checking the correctness of these appli-cations The approach presented in this paper focuses notonly on testing applications but also the underlying archi-tecture of the cloud system on which the application is beingexecuted In other words our approach aims to model andtest the complete system and not only an application

In MT experienced usersdomain experts are responsi-ble for providing useful MRs directly corresponding to themost relevant properties of the system under test MT pro-vides a simple method for deriving follow-up test casesand most importantly is cost effective because the processof checking the accuracy of the system can be performedautomatically without either an oracle or human interaction

Since the core of the proposed methodology lies in thedefinition of MRs the results provided by the testing pro-cess depend on how appropriate these are Thus appropriaterelations should provide useful information about the cor-rectness of the cloud model while poorly defined relationswould provide very little value It is the responsibility of theuser to provide appropriate MRs

Although simulation provides a lot of advantages thatmakes it a powerful tool for research it also entails somedrawbacks The main and obvious one is that simulationdoes not provide real data since we only simulate the per-formance of a cloud system during a given experiment incontrast to executing the experiment in a cloud system

It is important to note that our work is not intended toreplace experiments in real clouds Instead the proposed

Ann Telecommun

GreenCloud provides plug-ins that allow the use of physicallayer traces that make experiments more detailed

SimGrid is a simulator that provides core functionalitiesto simulate algorithms and distributed applications in dis-tributed computing platforms which go from workstationsto grid environments [5] The resources are modelled bytheir latency and service rate The topology is configurableIndeed defining the topology is the userrsquos responsibil-ity whereas collections of links may be used to simulatecomplex mechanisms like routers

In this work we use the iCanCloud simulation frame-work to represent the behaviour of cloud systems More-over an additional framework called E-mc2 [6] is used tomodel the energy consumption of each hardware device inthe cloud Both simulation frameworks are currently avail-able under the GPL3 license at httpwwwiCanCloudorg There are three main reasons for using these simula-tion frameworks First these simulators are open sourceThus the source code is available and can be modified ifrequired Second these simulators provide highly detailedmodels of both the hardware part of the cloud systemand the virtualisation schema Third there is a GUI thatallows the environment to be easily configured Moreoverthe specific motivations for using simulation in our workare

ndash Simulation is cheaper than performing experimentsdirectly in a real cloud

ndash The level of flexibility obtained by using simulation ismuch higher than when using real cloud systems

ndash Scalability In particular the number of VMs that asingle user may rent in public clouds is limited Incontrast in a simulated environment cloud systemscan be modelled with a fully customisable number ofmachines

ndash Researchers can share simulation models

This paper also investigates the application of Metamor-phic Testing (in short MT) to the testing of cloud systemssince such systems are typically complex and often have nooracle

MT was developed in order to test systems where thereis no oracle or it is expensive to compute the oracle Theessential idea is that instead of checking the output o1 pro-duced when testing with one input x1 we test with a second(follow-up) input x2 observing output o2 and check that o1

and o2 are related as expected Thus in MT there are tworelations the relation between the original test input x1 andthe follow-up input x2 and the expected relation betweenthe two outputs

Consider for example the problem of checking a pro-gram f that should be an implementation of the trigono-metric sine function While it may be difficult to checkwhether the application of f to an input is correct we know

a number of properties of the sine function and any cor-rect implementation should have these properties One suchproperty is that sin(minusx) = minussin(x) In order to checkthis metamorphic relation having tested f with an inputwe also test f with the negation of the original input andcheck the above property If the property does not hold thenf must be faulty and it must have failed on at least one ofthe two inputs (the original value and its negation) In thispaper we adapt the idea of MT to cloud systems

There are three main reasons for using MT in this workFirst it has been shown in different application scenariosthat MT alleviates the oracle problem providing a betterapproach for testing large and complex scenarios than con-ventional testing techniques Second the effectiveness ofthe proposed methodology can be increased by adding newmetamorphic relations (in short MRs) MRs can be sharedamong different research groups in order to increase theapplicability to a wider range of cloud configurations Thirdit is possible to concentrate the testing effort on a particu-lar feature of the system by using specific MRs targetingeither a specific characteristic or a part of the system Thiscan be easily done if MRs are grouped in categories eachone being responsible for analysing a specific feature of thecloud system

MT has been used in the past to test Web Services [7]focusing efforts on checking the correctness of these appli-cations The approach presented in this paper focuses notonly on testing applications but also the underlying archi-tecture of the cloud system on which the application is beingexecuted In other words our approach aims to model andtest the complete system and not only an application

In MT experienced usersdomain experts are responsi-ble for providing useful MRs directly corresponding to themost relevant properties of the system under test MT pro-vides a simple method for deriving follow-up test casesand most importantly is cost effective because the processof checking the accuracy of the system can be performedautomatically without either an oracle or human interaction

Since the core of the proposed methodology lies in thedefinition of MRs the results provided by the testing pro-cess depend on how appropriate these are Thus appropriaterelations should provide useful information about the cor-rectness of the cloud model while poorly defined relationswould provide very little value It is the responsibility of theuser to provide appropriate MRs

Although simulation provides a lot of advantages thatmakes it a powerful tool for research it also entails somedrawbacks The main and obvious one is that simulationdoes not provide real data since we only simulate the per-formance of a cloud system during a given experiment incontrast to executing the experiment in a cloud system

It is important to note that our work is not intended toreplace experiments in real clouds Instead the proposed

Ann Telecommun

Fig 1 Basic schema of theproposed methodology

methodology has been designed to help researchers to findand improve those configurations that obtain better resultsand discard those that are not valid However the final stageof the research must consist in executing the correspondingexperiments in a real cloud system

3 The proposed methodology

This section describes in detail the proposed methodologyfor testing cloud systems Figure 1 shows the basic schemaof this methodology that integrates simulation and testing

In this work a cloud model is defined by a data center thevirtualisation schema and a cloud manager The data centerdefines the underlying architecture of a cloud system thatis the number of physical machines allocated in the cloudsystem the configuration of the basic subsystems of eachphysical machine and the network topology The virtuali-sation schema consists of the configuration of each virtualmachine which is defined by setting the virtualised CPUstorage and memory Finally the cloud manager is an algo-rithm that allocates VMs in the available physical machinesof the cloud

Initially users can create a cloud model by using thecloud repository Each cloud model is represented by a textfile readable by the simulator

From now on the term user denotes a person who usesour proposed methodology while tenant refers to a personwho purchases services of the modelled cloud in a simulatedenvironment Basically a tenant is defined by a set of pur-chased VMs each purchased for a specific time slot and aset of applications that are executed in these VMs

31 Modelling of a cloud system

In order to illustrate the concepts described in this sectionwe present a running example This example also presentsa partial configuration of the model taken as input by thesimulator (see Fig 2) For the sake of clarity only the mostrelevant parameters are showed

Consider the data center of a modelled cloud systemthat consists of 128 physical machines (see lines [8ndash9])

Typically each cloud has two types of machines each onededicated to a specific task computing nodes and storageservers A computing node is a machine used to host one orseveral VMs Generally these VMs are provided to tenantsthat request cloud services In contrast storage servers are incharge of managing remote data access In this example weuse 112 computing nodes and 16 storage servers (see lines[16ndash17]) These nodes are connected through an Ethernet 10Gbps network (see lines [3ndash5])

Each physical machine must be configured by setting upits basic sub-systems CPU memory and storage In thisexample computing nodes use a quad-Core CPU 16 GB of

Fig 2 Example of an input cloud model

Ann Telecommun

memory and 1 disk of 500 GB (see lines [26ndash31]) whilestorage nodes use a dual-Core CPU 8 GB of memory and a5-disk RAID system of 2 TB each (see lines [34ndash39])

The virtualisation schema consists of the configuration ofdifferent VMs Since virtualisation allows the separation ofphysical and logical resources it is one of the major aspectsof a cloud-computing environment Hence the behaviourof virtual resources needs to be appropriately simulated inorder to obtain accurate results A VM is modelled as a por-tion of the resources of a given machine Thus a VM cannotexceed the resources of the physical machine on which it isexecuted

For example a virtual machine (quad-Core 21 GHz 0516 GB of RAM 025 500 GB disk 01) indicates that wewould like to use a machine providing a CPU at least asgood as quad-Core and having at least 50 of its use pro-viding 4 GB of RAM and at least 10 of a storage systemof 500 GB or better

Finally a cloud manager must be provided to completethe cloud system model The main objective of the cloudmanager is to map the VMs requested by tenants to theavailable physical machines in the cloud Thus each cloudmanager must implement its own mapping algorithm In thisexample we have used First In First Out (FIFO) (see line 15in Fig 2)

The test engine module receives as input both a cloudmodel and a set of MRs both previously selected by theuser Basically this module automatically generates a set offollow-up test cases which are built from the original modelprovided by the user Each test is executed using iCan-Cloud and the results collected The tool determines whichinstances of MRs were satisfied and which were not andreports this to the user who is responsible for determiningwhether the results are acceptable

32 Definition of metamorphic relations

We will consider MRs to formally compare the resultsobtained in the testing process Next we formally define thepattern of our relations

Definition 1 Let m be a cloud model and M prime be a set ofcloud models such that m isin M prime Let T be a set of tenants Ametamorphic relation MR for m and T is the set of 5-tuples

MR =

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M primeand

p1(mmprime)dArr

p2(mmprime T (m) T (mprime))

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

where p1 is a relation over cloud models and p2 is a relationover the cloud models and the execution of tenants on thesemodels

The set of MRs is partitioned into three different cate-gories each representing a specific aspect of the system tobe tested Thus users are able to choose a category in orderto test a specific feature of the cloud instead of testing themodel completely

ndash Performance This set contains those relations directlyrelated to the performance of a given system Depend-ing on the system to be tested this performance ismeasured in megabit per second million instructionsper second or execution time

ndash Functional This set contains those relations that checkthe underlying functioning of a given system

ndash Energy aware This set contains those relations thatcheck the restrictions regarding the energy consumedby a given system where this system can be a singledevice like a CPU or a complete cloud system

In order to illustrate our approach we show the fullresults for one MR from each group Next the definition ofeach MR used is presented We use two cloud models andone set of tenants denoted by m mprime and T respectivelywhere m represents the original model provided by the usermprime represents a variant automatically generated by the test-ing engine and T represents the workload executed in eachmodel

MRPER The CPU system of m having better perfor-mance than the CPU system of mprime (and all other aspectsbeing the same) denoted by (mcpu) gt (mprime

cpu) impliesthat the time required to execute T over mprime is greater thanor equal to the time required to execute T over m denotedby time(T(mprime)) ge time(T(m)) Formally

MRPER=

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M

and(mcpu) gt (mprime

cpu)

dArrtime(T(mprime))ge time(T(m))

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

MRFUN Let mP and mprimeP be two sets of physical machines

that represent the physical machines used to model m andmprime respectively If |mP | gt |mprime

P | that is the model m con-tains more physical machines than the model mprime both usingthe same hardware configuration then if T (mprime) is executedsuccessfully denoted as uarr T (mprime) T (m) must also result ina successful execution denoted as uarr T (m) Formally

MRFUN =

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M

and|mP | gt |mprime

P |dArruarr T (mprime) rarruarr T (m)

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

We say that T (m) is a successful execution of T over mdenoted by uarr T (m) if every application of every tenant inT is executed completely that is none of these applications

Ann Telecommun

are aborted by the expiration of any time slot of any VM Wedenote by darr T (m) an unsuccessful execution of T over m

MRENE If the energy required to execute T (m) denotedby (T(m)) is greater than the energy required to executeT (mprime) denoted by (T(mprime)) that is (T(m)) = α times(T(mprime)) (α gt 1) then the time required to executeT (m) must be less than the time required to execute T (mprime)denoted by time(T(m)) lt time(T(mprime)) Formally

MRENE=

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M

and(T (m))(T (mprime)) gt 1

dArrtime(T(mprime))gt time(T(m))

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

33 Description of the testing process

Conventional testing methods check whether the output(s)returned by the system under test are the expected onesor not Schematically let p be a system Let I be theinput domain and S be a test selection strategy Let T =t1 t2 tn sube I be the set of tests generated by usingS When these tests are sequentially applied to the programp we obtain a sequence of outputs p(t1) p(t2) p(tn)Therefore if we have a specification implemented by anoracle called s then we find an error if there exists ti isin Tsuch that p(tn) = s(tn) But in general we will not havean oracle and therefore we look for evidence regardingwhether an output is correct Our proposed methodologyincludes a testing framework whose main purpose is to(semi-)automatically test the suitability of a cloud model byusing an approach inspired by metamorphic testing In thiscase a single test case is represented by a set of tenants T

executed over a single cloud model m denoted by T (m)Actually this part can be performed by users even withoutusing the testing part The idea will be to consider variantsof the original model compute the application of the setof tenants to these variants T (mprime

1) T (mprime2) T (m

primek) and

compare the obtained results Let us note that our consideredvariants are not mutants in the sense of mutation testing [9]our goal is not to kill the variants in order to decide the good-ness of the considered test (in this case the set of tenantsT ) but to compare the different obtained results to detect awrong or suboptimal behaviour of the original model

The test engine module is in charge of automatically gen-erating test cases by following a given strategy S Initiallywe use a basic strategy that consists of sorting all the com-ponents that are used to generate a cloud model The sortcriterion is based in the quality of these components Thenfor each variant to be generated from the original cloudmodel the strategy selects the next preferable component ofthe list Each variant is represented as a text file (see Fig 2)Basically the test engine generates a copy of the original

Table 1 Modelling of two different cloud systems

Device Cloud A Cloud B

Computing nodes 80 96

Storage servers 16 32

CPU 4-Core 21 GHz 4-Core 21 GHz

Memory 8 GB 4 GB

Network Ethernet 1 Gbps Ethernet 1 Gbps

Storage 500 GB 500 GB

Switches bandwidth 10 Gbps 10 Mbps

Cloud Manager First Fit Round Robin

cloud model that is a variant where the correspondingmodification is applied eg using a better disk

Let M be a set of all possible cloud models thetesting engine module takes as input a cloud modelm isin M a set of tenants T and a set of relations MRIf T (m) is a successful test case then the set of variantsM prime = mprime

1 mprime2 m

primek sube M from the original model

m is generated Next M prime is used to automatically generatethe follow-up test cases T (mprime

1) T (mprime2) T (m

primek) Finally

these test cases are sent to the simulator to be executed

4 Performance experiments

This section describes experiments carried out with twomodels that acted as case studies The main hardware fea-tures of these systems are shown in Table 1 In order tosimplify the exposition that both cloud systems are homo-geneous systems that is all physical machines in a cloudsystem have the same hardware configuration In theseexperiments different algorithms implementing the cloudmanager (First Fit and Round Robin) were used

The virtualisation schema used in both cloud models ispresented in Table 2

In order to test a cloud system it is required that sev-eral tenants execute applications in it In this work threeapplications were modelled using the iCanCloud simulationplatform a Web server a CPU-intensive application and ahigh performance computing application (in short HPC)

The first application models the behaviour of a Webserver The second application multiplies two large matri-ces Initially these matrices are stored in the disk of a

Table 2 Modelling of different VM types

Type CPU cores Memory Storage

VMsmall 1 core 1 GB 100 GB

VMmedium 2 cores 2 GB 250 GB

VMlarge 4 cores 4 GB 500 GB

Ann Telecommun

Table 3 Modelling of different types of tenants

Type Instances Matrix Server HPC VMsmall VMmed VMlarge Mapping

tcomp 30 5 0 0 3 0 0 Random

tHPC 25 0 0 3 0 0 8 First Fit

tserver 30 0 10 0 0 5 0 Best Fit

tmix 15 5 5 3 3 3 2 Random

storage server This application reads these matrices per-forms the calculation and finally the result is written to diskFinally the third application models an HPC applicationcalled BIPS3D [13]

Table 3 shows the modelling of four types of tenant Wesay that each type of tenant represents a group of users inthe cloud that have a similar behaviour The configurationof this behaviour can be set by using three parameters therented VMs the applications to be executed and how eachapplication is assigned to a rented VM The first columnrefers to the name that defines the type of a tenant The sec-ond column represents the number of simulated tenants ofa given type The next three columns represent the numberof applications requested by each type of tenant These arefollowed by three columns that represent the VMs rentedby each type of tenant The last column gives the algorithmused to map applications to VMs

We now describe the results of experiments that wereperformed by executing 100 tenants (see Table 3) overcloud A and cloud B (see Table 1) using three MRs(MRPER MRFUN and MRENE) by following the methodol-ogy described in Fig 3 In each testing process a total of100 cloud models were generated

Table 4 shows the number of tests that successfully ful-filled each MR We can observe that using cloud A thepercentage of successful tests is greater than 90 In con-trast there is a noticeable drop in the percentage of teststhat fulfil MRs when cloud B is used This difference is

mainly due to cloud B being poorly configured In this casethe parameter that configures the bandwidth of the switchesin cloud B is set to megabit per second instead of giga-bit per secon This ldquomistakerdquo causes a bottleneck in thesystem However we obtain similar results in both cloudmodels when using the MR focusing on the functionality ofthe cloud This is because the functional behaviour of bothmodels is correct even though the obtained performance isnot that expected While cloud A obtains good results whenanalysing performance and energy consumption cloud Bobtains around 60 of successful tests The main reasonfor this result lies in the saturation of the communicationsnetwork This leads to relatively little parallelism beingobtained when executing different VMs and so there beinglittle difference in the total time of execution of the originalmodel and the variants

Figure 4 shows the results of a subset of the experi-ments performed For the sake of clarity these charts showthe generated models by modifying only two parametersof the cloud CPU and number of storage servers Thesecharts show the tendency of the system when some changesare applied in the original model In this case the perfor-mance of each system is the target to be analysed which ismeasured in seconds (less is better)

Using cloud A the MRPER relation is fulfilled by 92100tests This indicates that cloud A is well configured for per-formance tests In contrast only 65 tests performed usingcloud B fulfilled the same relation This means that some

Fig 3 Testing methodology

Ann Telecommun

Table 4 Number of tests that successfully fulfilled each MR

Cloud Model MR PER MR FUN MR ENE

Cloud A 92100 97100 95100

Cloud B 65100 92100 63100

subsystem in the cloud is not working properly The use of abetter CPU does not lead to improved performance becausethe communication network is saturated

When using MRFUN with cloud A there are a few caseswhere the increase in the number of physical machines inthe model leads to the execution requiring more time Thissituation is mainly caused by the BIPS3D application Whenthis application is executed in different physical machinesthat are not connected to the same switch the time to per-form network operations is higher Similarly 92100 testspassed this relation when cloud B is used This means thatthis model works similarly when the size of the system

16 servers

8 servers

4 servers

2 servers

1 server

6000

6500

7000

7500

8000

8500

9000

9500

10000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud model A

16 servers

8 servers

4 servers

2 servers

1 server

6000

8000

10000

12000

14000

16000

18000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud Model B

Fig 4 Results of simulating cloud models A and B

is increased That is the bottleneck caused by the wrongconfiguration of the network does not affect this MR

Finally the relation MRENE is fulfilled by 97100 testcases when cloud A is used The algorithm used for map-ping applications to VMs is directly responsible for thethree tests that do not fulfill this MR In some cases anapplication that requires significant resources is executedin a VM that contains limited resources and vice versaIn these cases powerful VMs are wasted and applicationsthat require significant resources have long execution timesincreasing also the energy consumption In contrast usingcloud B only 63100 tests fulfilled this MR This is mainlycaused by the wrong configuration of the network sys-tem which results in a system bottleneck The bottleneckincreases the time required to execute a workload in thissystem Communications bottlenecks can lead to processorperformance being almost irrelevant making it more likelythat MRENE fails

The selection of the algorithm for allocating VMs tophysical machines also affects the results Cloud A uses aFIFO algorithm for allocating VMs to physical machinesTherefore when the cloud manager receives a request toexecute a VM the first physical machine that contains suf-ficient available resources is selected Consequently evenat the beginning of the execution a physical machine mayhave several VMs In contrast Round Robin assigns thefirst available physical machine in the system using a ded-icated machine if possible In this case different VMs onlyshare a physical machine when there are no idle machinesin the system While the FIFO algorithm leads to physicalmachines being shared by different VMs where possi-ble Round-Robin leads to a dedicated physical machinebeing assigned to each VM Thus Cloud A typically usesfewer physical machines in executing a workload this beingreflected in the energy consumption This effect is more vis-ible when the system is not well-configured like cloud Bcausing a drop in the number of successful tests focusing onenergy consumption

There do not appear to be configurations that are perfectfor all the applications which means that in some casesone application demands a different configuration to obtainbetter performance This explains why in all cases some ofthe MRs were not satisfied When many tests do not fulfillan MR the user has to check those tests that fail and explorepotential solutions

5 Conclusions and future work

This paper presents a methodology that integrates simu-lation techniques with testing methods for checking thecorrectness of cloud systems In particular the iCanCloudsimulation platform has been used to model and simulate

Ann Telecommun

cloud systems while techniques inspired by metamorphictesting were used to validate these models

The main goal of this work was to facilitate the processof modelling and checking complete cloud systems semi-automatically Thus when the testing process is applied toa cloud model provided by the user the generated resultsshould provide useful information about how this cloudmodel is working

In order to show the usefulness and applicability of ourapproach different cloud systems were modelled and testedusing the proposed methodology While the first cloudmodel fits well with the typical architecture of cloud sys-tems the second cloud model provides a slow network thatdoes not fit well with the typical configuration of cloud sys-tems Moreover we gave different MRs each one focusedon checking a specific aspect of the cloud

The main objective of these experiments was to checkwhether the proposed method can reveal that a cloud modelis not well designed We expect that almost all tests willfulfill each MR when a cloud model is well designedIn contrast when we applied the technique to a cloudmodel defined using inappropriate values for parameters asignificant number of tests did not satisfy the MRs

Given an initial cloud model and different MRs it waspossible to automatically generate follow-up tests and checkwhether the MRs were satisfied The results of the experi-ments showed the potential to determine that a cloud systemmodel was poorly designed In this case the cloud that usesa slow network clearly shows a drop in the number of teststhat fulfil an MR focused on performance This is because aslow network acts as a system bottleneck

It is important to remark that although one of the thecloud models was poorly configured the same workloadcould be executed in both cloud models The main dif-ference lies in the time required to execute it Since bothmodels provided the required functionality almost all testssatisfied the MR focused on functionality Although the pro-cess of identifying mistakes in the design of a cloud systemis not entirely automated the user can focus on those MRsthat contains a high percentage of failures and this shouldhelp them find mistakes in a cloud model

Future work will present a formal definition of a cloudsystem in order to provide more consistent notation forincluding new metamorphic relations Further analysis ofa wide-range of cloud models will be performed Finallyan extended collection of metamorphic relations will bedefined

Acknowledgments This research was partially supported by theSpanish MEC projects TESIS (TIN2009-14312-C02-01) and ESTu-DIo (TIN2012-36812-C02-01)

References

1 Bertolino A Grieskamp W Hierons RM Le Traon Y Legeard BMuccini H Paradkar A Rosenblum D Tretmans J (2010) Model-based testing for the cloud In Practical Software Testing ToolAutomation and Human Factors no 10111 in Dagstuhl SeminarProceedings Schloss Dagstuhl - Leibniz-Zentrum fuer InformatikGermany pp 1ndash11

2 Buyya R Beloglazov A Abawajy J (2010) Energy-efficient man-agement of data center resources for cloud computing a visionarchitectural elements and open challenges In 16th Int Conf onParallel and Distributed Processing Techniques and ApplicationsPDPTArsquo10 CSREA Press pp 1ndash12

3 Buyya R Ranjan R Calheiros RN (2009) Modeling and simula-tion of scalable cloud computing environments and the CloudSimtoolkit challenges and opportunities In 7th High PerformanceComputing and Simulation Conference HPCSrsquo09 IEEE Com-puter Society pp 1ndash11

4 Calheiros RN Buyya R De Rose CAF (2010) Building anautomated and self-configurable emulation test bed for grid appli-cations Softw Pract Experience 40(5)405ndash429

5 Casanova H Legrand A Quinson M (2008) SimGrid a genericframework for large-scale distributed experiments In 10th IntConf on Computer Modeling and Simulation UKSIMrsquo 08

6 Castane G Nunez A Llopis P Carretero J (2013) E-mc2 a for-mal framework for energy modelling in cloud computing SimulModel Pract Theor 3956ndash75

7 Chen TY Sun C Wang G Mu B Liu H Wang ZS (2012) A meta-morphic relation-based approach to testing web services withoutoracles Int J Web Serv Res 9(1)51ndash73

8 Garber L (2011) News briefs IEEE Computer 44(6)18ndash209 Hierons RM Merayo MG Nunez M (2010) Mutation testing In

Laplante PA (ed) Encyclopedia of Software Engineering Tayloramp Francis

10 Kim KH Beloglazov A Buyya R (2009) Power-aware pro-visioning of cloud resources for real-time services In Pro-ceedings of the 7th International Workshop on Middleware forGrids Clouds and e-Science Urbana Champaign Illinois USApp 1ndash6

11 Kliazovich D Bouvry P Audzevich Y Khan S (2012) Green-cloud a packet-level simulator of energy-aware cloud computingdata centers J Supercomput 62(3)1263ndash1283

12 Myers G (2004) The Art of Software Testing 2nd edn Wiley13 Nunez A Fernandez J Filgueira R Garcıa F Carretero J (2012)

SIMCAN a flexible scalable and expandable simulation plat-form for modelling and simulating distributed architectures andapplications Simul Model Pract Theory 20(1)12ndash32

14 Nunez A Vazquez-Poletti JL Caminero AC Castane GG Car-retero J Llorente IM (2012) iCanCloud a flexible and scal-able cloud infrastructure simulator J Grid Comput 10(1)185ndash209

15 Ried S Kisker H Matzke P Bartels A Lisserman M (2011) Sizingthe cloudmdasha BT futures report Understanding and quantifying thefuture of cloud computing Forrester Research Report

16 Sulistio A Cibej U Venugopal S Robic B Buyya R (2008)A toolkit for modelling and simulating Data Grids an exten-sion to GridSim Concurr Comput Pract Experience 20(13)1591ndash1609

17 The Network Simulator NS-2 Web page at Date of last access8th August 2013 httpwwwisiedunsnamns

18 Weyuker EJ (1982) On testing non-testable programs Comput J25(4)465ndash470

Ann Telecommun

memory and 1 disk of 500 GB (see lines [26ndash31]) whilestorage nodes use a dual-Core CPU 8 GB of memory and a5-disk RAID system of 2 TB each (see lines [34ndash39])

The virtualisation schema consists of the configuration ofdifferent VMs Since virtualisation allows the separation ofphysical and logical resources it is one of the major aspectsof a cloud-computing environment Hence the behaviourof virtual resources needs to be appropriately simulated inorder to obtain accurate results A VM is modelled as a por-tion of the resources of a given machine Thus a VM cannotexceed the resources of the physical machine on which it isexecuted

For example a virtual machine (quad-Core 21 GHz 0516 GB of RAM 025 500 GB disk 01) indicates that wewould like to use a machine providing a CPU at least asgood as quad-Core and having at least 50 of its use pro-viding 4 GB of RAM and at least 10 of a storage systemof 500 GB or better

Finally a cloud manager must be provided to completethe cloud system model The main objective of the cloudmanager is to map the VMs requested by tenants to theavailable physical machines in the cloud Thus each cloudmanager must implement its own mapping algorithm In thisexample we have used First In First Out (FIFO) (see line 15in Fig 2)

The test engine module receives as input both a cloudmodel and a set of MRs both previously selected by theuser Basically this module automatically generates a set offollow-up test cases which are built from the original modelprovided by the user Each test is executed using iCan-Cloud and the results collected The tool determines whichinstances of MRs were satisfied and which were not andreports this to the user who is responsible for determiningwhether the results are acceptable

32 Definition of metamorphic relations

We will consider MRs to formally compare the resultsobtained in the testing process Next we formally define thepattern of our relations

Definition 1 Let m be a cloud model and M prime be a set ofcloud models such that m isin M prime Let T be a set of tenants Ametamorphic relation MR for m and T is the set of 5-tuples

MR =

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M primeand

p1(mmprime)dArr

p2(mmprime T (m) T (mprime))

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

where p1 is a relation over cloud models and p2 is a relationover the cloud models and the execution of tenants on thesemodels

The set of MRs is partitioned into three different cate-gories each representing a specific aspect of the system tobe tested Thus users are able to choose a category in orderto test a specific feature of the cloud instead of testing themodel completely

ndash Performance This set contains those relations directlyrelated to the performance of a given system Depend-ing on the system to be tested this performance ismeasured in megabit per second million instructionsper second or execution time

ndash Functional This set contains those relations that checkthe underlying functioning of a given system

ndash Energy aware This set contains those relations thatcheck the restrictions regarding the energy consumedby a given system where this system can be a singledevice like a CPU or a complete cloud system

In order to illustrate our approach we show the fullresults for one MR from each group Next the definition ofeach MR used is presented We use two cloud models andone set of tenants denoted by m mprime and T respectivelywhere m represents the original model provided by the usermprime represents a variant automatically generated by the test-ing engine and T represents the workload executed in eachmodel

MRPER The CPU system of m having better perfor-mance than the CPU system of mprime (and all other aspectsbeing the same) denoted by (mcpu) gt (mprime

cpu) impliesthat the time required to execute T over mprime is greater thanor equal to the time required to execute T over m denotedby time(T(mprime)) ge time(T(m)) Formally

MRPER=

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M

and(mcpu) gt (mprime

cpu)

dArrtime(T(mprime))ge time(T(m))

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

MRFUN Let mP and mprimeP be two sets of physical machines

that represent the physical machines used to model m andmprime respectively If |mP | gt |mprime

P | that is the model m con-tains more physical machines than the model mprime both usingthe same hardware configuration then if T (mprime) is executedsuccessfully denoted as uarr T (mprime) T (m) must also result ina successful execution denoted as uarr T (m) Formally

MRFUN =

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M

and|mP | gt |mprime

P |dArruarr T (mprime) rarruarr T (m)

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

We say that T (m) is a successful execution of T over mdenoted by uarr T (m) if every application of every tenant inT is executed completely that is none of these applications

Ann Telecommun

are aborted by the expiration of any time slot of any VM Wedenote by darr T (m) an unsuccessful execution of T over m

MRENE If the energy required to execute T (m) denotedby (T(m)) is greater than the energy required to executeT (mprime) denoted by (T(mprime)) that is (T(m)) = α times(T(mprime)) (α gt 1) then the time required to executeT (m) must be less than the time required to execute T (mprime)denoted by time(T(m)) lt time(T(mprime)) Formally

MRENE=

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M

and(T (m))(T (mprime)) gt 1

dArrtime(T(mprime))gt time(T(m))

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

33 Description of the testing process

Conventional testing methods check whether the output(s)returned by the system under test are the expected onesor not Schematically let p be a system Let I be theinput domain and S be a test selection strategy Let T =t1 t2 tn sube I be the set of tests generated by usingS When these tests are sequentially applied to the programp we obtain a sequence of outputs p(t1) p(t2) p(tn)Therefore if we have a specification implemented by anoracle called s then we find an error if there exists ti isin Tsuch that p(tn) = s(tn) But in general we will not havean oracle and therefore we look for evidence regardingwhether an output is correct Our proposed methodologyincludes a testing framework whose main purpose is to(semi-)automatically test the suitability of a cloud model byusing an approach inspired by metamorphic testing In thiscase a single test case is represented by a set of tenants T

executed over a single cloud model m denoted by T (m)Actually this part can be performed by users even withoutusing the testing part The idea will be to consider variantsof the original model compute the application of the setof tenants to these variants T (mprime

1) T (mprime2) T (m

primek) and

compare the obtained results Let us note that our consideredvariants are not mutants in the sense of mutation testing [9]our goal is not to kill the variants in order to decide the good-ness of the considered test (in this case the set of tenantsT ) but to compare the different obtained results to detect awrong or suboptimal behaviour of the original model

The test engine module is in charge of automatically gen-erating test cases by following a given strategy S Initiallywe use a basic strategy that consists of sorting all the com-ponents that are used to generate a cloud model The sortcriterion is based in the quality of these components Thenfor each variant to be generated from the original cloudmodel the strategy selects the next preferable component ofthe list Each variant is represented as a text file (see Fig 2)Basically the test engine generates a copy of the original

Table 1 Modelling of two different cloud systems

Device Cloud A Cloud B

Computing nodes 80 96

Storage servers 16 32

CPU 4-Core 21 GHz 4-Core 21 GHz

Memory 8 GB 4 GB

Network Ethernet 1 Gbps Ethernet 1 Gbps

Storage 500 GB 500 GB

Switches bandwidth 10 Gbps 10 Mbps

Cloud Manager First Fit Round Robin

cloud model that is a variant where the correspondingmodification is applied eg using a better disk

Let M be a set of all possible cloud models thetesting engine module takes as input a cloud modelm isin M a set of tenants T and a set of relations MRIf T (m) is a successful test case then the set of variantsM prime = mprime

1 mprime2 m

primek sube M from the original model

m is generated Next M prime is used to automatically generatethe follow-up test cases T (mprime

1) T (mprime2) T (m

primek) Finally

these test cases are sent to the simulator to be executed

4 Performance experiments

This section describes experiments carried out with twomodels that acted as case studies The main hardware fea-tures of these systems are shown in Table 1 In order tosimplify the exposition that both cloud systems are homo-geneous systems that is all physical machines in a cloudsystem have the same hardware configuration In theseexperiments different algorithms implementing the cloudmanager (First Fit and Round Robin) were used

The virtualisation schema used in both cloud models ispresented in Table 2

In order to test a cloud system it is required that sev-eral tenants execute applications in it In this work threeapplications were modelled using the iCanCloud simulationplatform a Web server a CPU-intensive application and ahigh performance computing application (in short HPC)

The first application models the behaviour of a Webserver The second application multiplies two large matri-ces Initially these matrices are stored in the disk of a

Table 2 Modelling of different VM types

Type CPU cores Memory Storage

VMsmall 1 core 1 GB 100 GB

VMmedium 2 cores 2 GB 250 GB

VMlarge 4 cores 4 GB 500 GB

Ann Telecommun

Table 3 Modelling of different types of tenants

Type Instances Matrix Server HPC VMsmall VMmed VMlarge Mapping

tcomp 30 5 0 0 3 0 0 Random

tHPC 25 0 0 3 0 0 8 First Fit

tserver 30 0 10 0 0 5 0 Best Fit

tmix 15 5 5 3 3 3 2 Random

storage server This application reads these matrices per-forms the calculation and finally the result is written to diskFinally the third application models an HPC applicationcalled BIPS3D [13]

Table 3 shows the modelling of four types of tenant Wesay that each type of tenant represents a group of users inthe cloud that have a similar behaviour The configurationof this behaviour can be set by using three parameters therented VMs the applications to be executed and how eachapplication is assigned to a rented VM The first columnrefers to the name that defines the type of a tenant The sec-ond column represents the number of simulated tenants ofa given type The next three columns represent the numberof applications requested by each type of tenant These arefollowed by three columns that represent the VMs rentedby each type of tenant The last column gives the algorithmused to map applications to VMs

We now describe the results of experiments that wereperformed by executing 100 tenants (see Table 3) overcloud A and cloud B (see Table 1) using three MRs(MRPER MRFUN and MRENE) by following the methodol-ogy described in Fig 3 In each testing process a total of100 cloud models were generated

Table 4 shows the number of tests that successfully ful-filled each MR We can observe that using cloud A thepercentage of successful tests is greater than 90 In con-trast there is a noticeable drop in the percentage of teststhat fulfil MRs when cloud B is used This difference is

mainly due to cloud B being poorly configured In this casethe parameter that configures the bandwidth of the switchesin cloud B is set to megabit per second instead of giga-bit per secon This ldquomistakerdquo causes a bottleneck in thesystem However we obtain similar results in both cloudmodels when using the MR focusing on the functionality ofthe cloud This is because the functional behaviour of bothmodels is correct even though the obtained performance isnot that expected While cloud A obtains good results whenanalysing performance and energy consumption cloud Bobtains around 60 of successful tests The main reasonfor this result lies in the saturation of the communicationsnetwork This leads to relatively little parallelism beingobtained when executing different VMs and so there beinglittle difference in the total time of execution of the originalmodel and the variants

Figure 4 shows the results of a subset of the experi-ments performed For the sake of clarity these charts showthe generated models by modifying only two parametersof the cloud CPU and number of storage servers Thesecharts show the tendency of the system when some changesare applied in the original model In this case the perfor-mance of each system is the target to be analysed which ismeasured in seconds (less is better)

Using cloud A the MRPER relation is fulfilled by 92100tests This indicates that cloud A is well configured for per-formance tests In contrast only 65 tests performed usingcloud B fulfilled the same relation This means that some

Fig 3 Testing methodology

Ann Telecommun

Table 4 Number of tests that successfully fulfilled each MR

Cloud Model MR PER MR FUN MR ENE

Cloud A 92100 97100 95100

Cloud B 65100 92100 63100

subsystem in the cloud is not working properly The use of abetter CPU does not lead to improved performance becausethe communication network is saturated

When using MRFUN with cloud A there are a few caseswhere the increase in the number of physical machines inthe model leads to the execution requiring more time Thissituation is mainly caused by the BIPS3D application Whenthis application is executed in different physical machinesthat are not connected to the same switch the time to per-form network operations is higher Similarly 92100 testspassed this relation when cloud B is used This means thatthis model works similarly when the size of the system

16 servers

8 servers

4 servers

2 servers

1 server

6000

6500

7000

7500

8000

8500

9000

9500

10000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud model A

16 servers

8 servers

4 servers

2 servers

1 server

6000

8000

10000

12000

14000

16000

18000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud Model B

Fig 4 Results of simulating cloud models A and B

is increased That is the bottleneck caused by the wrongconfiguration of the network does not affect this MR

Finally the relation MRENE is fulfilled by 97100 testcases when cloud A is used The algorithm used for map-ping applications to VMs is directly responsible for thethree tests that do not fulfill this MR In some cases anapplication that requires significant resources is executedin a VM that contains limited resources and vice versaIn these cases powerful VMs are wasted and applicationsthat require significant resources have long execution timesincreasing also the energy consumption In contrast usingcloud B only 63100 tests fulfilled this MR This is mainlycaused by the wrong configuration of the network sys-tem which results in a system bottleneck The bottleneckincreases the time required to execute a workload in thissystem Communications bottlenecks can lead to processorperformance being almost irrelevant making it more likelythat MRENE fails

The selection of the algorithm for allocating VMs tophysical machines also affects the results Cloud A uses aFIFO algorithm for allocating VMs to physical machinesTherefore when the cloud manager receives a request toexecute a VM the first physical machine that contains suf-ficient available resources is selected Consequently evenat the beginning of the execution a physical machine mayhave several VMs In contrast Round Robin assigns thefirst available physical machine in the system using a ded-icated machine if possible In this case different VMs onlyshare a physical machine when there are no idle machinesin the system While the FIFO algorithm leads to physicalmachines being shared by different VMs where possi-ble Round-Robin leads to a dedicated physical machinebeing assigned to each VM Thus Cloud A typically usesfewer physical machines in executing a workload this beingreflected in the energy consumption This effect is more vis-ible when the system is not well-configured like cloud Bcausing a drop in the number of successful tests focusing onenergy consumption

There do not appear to be configurations that are perfectfor all the applications which means that in some casesone application demands a different configuration to obtainbetter performance This explains why in all cases some ofthe MRs were not satisfied When many tests do not fulfillan MR the user has to check those tests that fail and explorepotential solutions

5 Conclusions and future work

This paper presents a methodology that integrates simu-lation techniques with testing methods for checking thecorrectness of cloud systems In particular the iCanCloudsimulation platform has been used to model and simulate

Ann Telecommun

cloud systems while techniques inspired by metamorphictesting were used to validate these models

The main goal of this work was to facilitate the processof modelling and checking complete cloud systems semi-automatically Thus when the testing process is applied toa cloud model provided by the user the generated resultsshould provide useful information about how this cloudmodel is working

In order to show the usefulness and applicability of ourapproach different cloud systems were modelled and testedusing the proposed methodology While the first cloudmodel fits well with the typical architecture of cloud sys-tems the second cloud model provides a slow network thatdoes not fit well with the typical configuration of cloud sys-tems Moreover we gave different MRs each one focusedon checking a specific aspect of the cloud

The main objective of these experiments was to checkwhether the proposed method can reveal that a cloud modelis not well designed We expect that almost all tests willfulfill each MR when a cloud model is well designedIn contrast when we applied the technique to a cloudmodel defined using inappropriate values for parameters asignificant number of tests did not satisfy the MRs

Given an initial cloud model and different MRs it waspossible to automatically generate follow-up tests and checkwhether the MRs were satisfied The results of the experi-ments showed the potential to determine that a cloud systemmodel was poorly designed In this case the cloud that usesa slow network clearly shows a drop in the number of teststhat fulfil an MR focused on performance This is because aslow network acts as a system bottleneck

It is important to remark that although one of the thecloud models was poorly configured the same workloadcould be executed in both cloud models The main dif-ference lies in the time required to execute it Since bothmodels provided the required functionality almost all testssatisfied the MR focused on functionality Although the pro-cess of identifying mistakes in the design of a cloud systemis not entirely automated the user can focus on those MRsthat contains a high percentage of failures and this shouldhelp them find mistakes in a cloud model

Future work will present a formal definition of a cloudsystem in order to provide more consistent notation forincluding new metamorphic relations Further analysis ofa wide-range of cloud models will be performed Finallyan extended collection of metamorphic relations will bedefined

Acknowledgments This research was partially supported by theSpanish MEC projects TESIS (TIN2009-14312-C02-01) and ESTu-DIo (TIN2012-36812-C02-01)

References

1 Bertolino A Grieskamp W Hierons RM Le Traon Y Legeard BMuccini H Paradkar A Rosenblum D Tretmans J (2010) Model-based testing for the cloud In Practical Software Testing ToolAutomation and Human Factors no 10111 in Dagstuhl SeminarProceedings Schloss Dagstuhl - Leibniz-Zentrum fuer InformatikGermany pp 1ndash11

2 Buyya R Beloglazov A Abawajy J (2010) Energy-efficient man-agement of data center resources for cloud computing a visionarchitectural elements and open challenges In 16th Int Conf onParallel and Distributed Processing Techniques and ApplicationsPDPTArsquo10 CSREA Press pp 1ndash12

3 Buyya R Ranjan R Calheiros RN (2009) Modeling and simula-tion of scalable cloud computing environments and the CloudSimtoolkit challenges and opportunities In 7th High PerformanceComputing and Simulation Conference HPCSrsquo09 IEEE Com-puter Society pp 1ndash11

4 Calheiros RN Buyya R De Rose CAF (2010) Building anautomated and self-configurable emulation test bed for grid appli-cations Softw Pract Experience 40(5)405ndash429

5 Casanova H Legrand A Quinson M (2008) SimGrid a genericframework for large-scale distributed experiments In 10th IntConf on Computer Modeling and Simulation UKSIMrsquo 08

6 Castane G Nunez A Llopis P Carretero J (2013) E-mc2 a for-mal framework for energy modelling in cloud computing SimulModel Pract Theor 3956ndash75

7 Chen TY Sun C Wang G Mu B Liu H Wang ZS (2012) A meta-morphic relation-based approach to testing web services withoutoracles Int J Web Serv Res 9(1)51ndash73

8 Garber L (2011) News briefs IEEE Computer 44(6)18ndash209 Hierons RM Merayo MG Nunez M (2010) Mutation testing In

Laplante PA (ed) Encyclopedia of Software Engineering Tayloramp Francis

10 Kim KH Beloglazov A Buyya R (2009) Power-aware pro-visioning of cloud resources for real-time services In Pro-ceedings of the 7th International Workshop on Middleware forGrids Clouds and e-Science Urbana Champaign Illinois USApp 1ndash6

11 Kliazovich D Bouvry P Audzevich Y Khan S (2012) Green-cloud a packet-level simulator of energy-aware cloud computingdata centers J Supercomput 62(3)1263ndash1283

12 Myers G (2004) The Art of Software Testing 2nd edn Wiley13 Nunez A Fernandez J Filgueira R Garcıa F Carretero J (2012)

SIMCAN a flexible scalable and expandable simulation plat-form for modelling and simulating distributed architectures andapplications Simul Model Pract Theory 20(1)12ndash32

14 Nunez A Vazquez-Poletti JL Caminero AC Castane GG Car-retero J Llorente IM (2012) iCanCloud a flexible and scal-able cloud infrastructure simulator J Grid Comput 10(1)185ndash209

15 Ried S Kisker H Matzke P Bartels A Lisserman M (2011) Sizingthe cloudmdasha BT futures report Understanding and quantifying thefuture of cloud computing Forrester Research Report

16 Sulistio A Cibej U Venugopal S Robic B Buyya R (2008)A toolkit for modelling and simulating Data Grids an exten-sion to GridSim Concurr Comput Pract Experience 20(13)1591ndash1609

17 The Network Simulator NS-2 Web page at Date of last access8th August 2013 httpwwwisiedunsnamns

18 Weyuker EJ (1982) On testing non-testable programs Comput J25(4)465ndash470

Ann Telecommun

are aborted by the expiration of any time slot of any VM Wedenote by darr T (m) an unsuccessful execution of T over m

MRENE If the energy required to execute T (m) denotedby (T(m)) is greater than the energy required to executeT (mprime) denoted by (T(mprime)) that is (T(m)) = α times(T(mprime)) (α gt 1) then the time required to executeT (m) must be less than the time required to execute T (mprime)denoted by time(T(m)) lt time(T(mprime)) Formally

MRENE=

⎧⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎩

(T mmprimeT (m) T (mprime)

)

∣∣∣∣∣∣∣∣∣∣

mprime isin M

and(T (m))(T (mprime)) gt 1

dArrtime(T(mprime))gt time(T(m))

⎫⎪⎪⎪⎪⎬

⎪⎪⎪⎪⎭

33 Description of the testing process

Conventional testing methods check whether the output(s)returned by the system under test are the expected onesor not Schematically let p be a system Let I be theinput domain and S be a test selection strategy Let T =t1 t2 tn sube I be the set of tests generated by usingS When these tests are sequentially applied to the programp we obtain a sequence of outputs p(t1) p(t2) p(tn)Therefore if we have a specification implemented by anoracle called s then we find an error if there exists ti isin Tsuch that p(tn) = s(tn) But in general we will not havean oracle and therefore we look for evidence regardingwhether an output is correct Our proposed methodologyincludes a testing framework whose main purpose is to(semi-)automatically test the suitability of a cloud model byusing an approach inspired by metamorphic testing In thiscase a single test case is represented by a set of tenants T

executed over a single cloud model m denoted by T (m)Actually this part can be performed by users even withoutusing the testing part The idea will be to consider variantsof the original model compute the application of the setof tenants to these variants T (mprime

1) T (mprime2) T (m

primek) and

compare the obtained results Let us note that our consideredvariants are not mutants in the sense of mutation testing [9]our goal is not to kill the variants in order to decide the good-ness of the considered test (in this case the set of tenantsT ) but to compare the different obtained results to detect awrong or suboptimal behaviour of the original model

The test engine module is in charge of automatically gen-erating test cases by following a given strategy S Initiallywe use a basic strategy that consists of sorting all the com-ponents that are used to generate a cloud model The sortcriterion is based in the quality of these components Thenfor each variant to be generated from the original cloudmodel the strategy selects the next preferable component ofthe list Each variant is represented as a text file (see Fig 2)Basically the test engine generates a copy of the original

Table 1 Modelling of two different cloud systems

Device Cloud A Cloud B

Computing nodes 80 96

Storage servers 16 32

CPU 4-Core 21 GHz 4-Core 21 GHz

Memory 8 GB 4 GB

Network Ethernet 1 Gbps Ethernet 1 Gbps

Storage 500 GB 500 GB

Switches bandwidth 10 Gbps 10 Mbps

Cloud Manager First Fit Round Robin

cloud model that is a variant where the correspondingmodification is applied eg using a better disk

Let M be a set of all possible cloud models thetesting engine module takes as input a cloud modelm isin M a set of tenants T and a set of relations MRIf T (m) is a successful test case then the set of variantsM prime = mprime

1 mprime2 m

primek sube M from the original model

m is generated Next M prime is used to automatically generatethe follow-up test cases T (mprime

1) T (mprime2) T (m

primek) Finally

these test cases are sent to the simulator to be executed

4 Performance experiments

This section describes experiments carried out with twomodels that acted as case studies The main hardware fea-tures of these systems are shown in Table 1 In order tosimplify the exposition that both cloud systems are homo-geneous systems that is all physical machines in a cloudsystem have the same hardware configuration In theseexperiments different algorithms implementing the cloudmanager (First Fit and Round Robin) were used

The virtualisation schema used in both cloud models ispresented in Table 2

In order to test a cloud system it is required that sev-eral tenants execute applications in it In this work threeapplications were modelled using the iCanCloud simulationplatform a Web server a CPU-intensive application and ahigh performance computing application (in short HPC)

The first application models the behaviour of a Webserver The second application multiplies two large matri-ces Initially these matrices are stored in the disk of a

Table 2 Modelling of different VM types

Type CPU cores Memory Storage

VMsmall 1 core 1 GB 100 GB

VMmedium 2 cores 2 GB 250 GB

VMlarge 4 cores 4 GB 500 GB

Ann Telecommun

Table 3 Modelling of different types of tenants

Type Instances Matrix Server HPC VMsmall VMmed VMlarge Mapping

tcomp 30 5 0 0 3 0 0 Random

tHPC 25 0 0 3 0 0 8 First Fit

tserver 30 0 10 0 0 5 0 Best Fit

tmix 15 5 5 3 3 3 2 Random

storage server This application reads these matrices per-forms the calculation and finally the result is written to diskFinally the third application models an HPC applicationcalled BIPS3D [13]

Table 3 shows the modelling of four types of tenant Wesay that each type of tenant represents a group of users inthe cloud that have a similar behaviour The configurationof this behaviour can be set by using three parameters therented VMs the applications to be executed and how eachapplication is assigned to a rented VM The first columnrefers to the name that defines the type of a tenant The sec-ond column represents the number of simulated tenants ofa given type The next three columns represent the numberof applications requested by each type of tenant These arefollowed by three columns that represent the VMs rentedby each type of tenant The last column gives the algorithmused to map applications to VMs

We now describe the results of experiments that wereperformed by executing 100 tenants (see Table 3) overcloud A and cloud B (see Table 1) using three MRs(MRPER MRFUN and MRENE) by following the methodol-ogy described in Fig 3 In each testing process a total of100 cloud models were generated

Table 4 shows the number of tests that successfully ful-filled each MR We can observe that using cloud A thepercentage of successful tests is greater than 90 In con-trast there is a noticeable drop in the percentage of teststhat fulfil MRs when cloud B is used This difference is

mainly due to cloud B being poorly configured In this casethe parameter that configures the bandwidth of the switchesin cloud B is set to megabit per second instead of giga-bit per secon This ldquomistakerdquo causes a bottleneck in thesystem However we obtain similar results in both cloudmodels when using the MR focusing on the functionality ofthe cloud This is because the functional behaviour of bothmodels is correct even though the obtained performance isnot that expected While cloud A obtains good results whenanalysing performance and energy consumption cloud Bobtains around 60 of successful tests The main reasonfor this result lies in the saturation of the communicationsnetwork This leads to relatively little parallelism beingobtained when executing different VMs and so there beinglittle difference in the total time of execution of the originalmodel and the variants

Figure 4 shows the results of a subset of the experi-ments performed For the sake of clarity these charts showthe generated models by modifying only two parametersof the cloud CPU and number of storage servers Thesecharts show the tendency of the system when some changesare applied in the original model In this case the perfor-mance of each system is the target to be analysed which ismeasured in seconds (less is better)

Using cloud A the MRPER relation is fulfilled by 92100tests This indicates that cloud A is well configured for per-formance tests In contrast only 65 tests performed usingcloud B fulfilled the same relation This means that some

Fig 3 Testing methodology

Ann Telecommun

Table 4 Number of tests that successfully fulfilled each MR

Cloud Model MR PER MR FUN MR ENE

Cloud A 92100 97100 95100

Cloud B 65100 92100 63100

subsystem in the cloud is not working properly The use of abetter CPU does not lead to improved performance becausethe communication network is saturated

When using MRFUN with cloud A there are a few caseswhere the increase in the number of physical machines inthe model leads to the execution requiring more time Thissituation is mainly caused by the BIPS3D application Whenthis application is executed in different physical machinesthat are not connected to the same switch the time to per-form network operations is higher Similarly 92100 testspassed this relation when cloud B is used This means thatthis model works similarly when the size of the system

16 servers

8 servers

4 servers

2 servers

1 server

6000

6500

7000

7500

8000

8500

9000

9500

10000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud model A

16 servers

8 servers

4 servers

2 servers

1 server

6000

8000

10000

12000

14000

16000

18000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud Model B

Fig 4 Results of simulating cloud models A and B

is increased That is the bottleneck caused by the wrongconfiguration of the network does not affect this MR

Finally the relation MRENE is fulfilled by 97100 testcases when cloud A is used The algorithm used for map-ping applications to VMs is directly responsible for thethree tests that do not fulfill this MR In some cases anapplication that requires significant resources is executedin a VM that contains limited resources and vice versaIn these cases powerful VMs are wasted and applicationsthat require significant resources have long execution timesincreasing also the energy consumption In contrast usingcloud B only 63100 tests fulfilled this MR This is mainlycaused by the wrong configuration of the network sys-tem which results in a system bottleneck The bottleneckincreases the time required to execute a workload in thissystem Communications bottlenecks can lead to processorperformance being almost irrelevant making it more likelythat MRENE fails

The selection of the algorithm for allocating VMs tophysical machines also affects the results Cloud A uses aFIFO algorithm for allocating VMs to physical machinesTherefore when the cloud manager receives a request toexecute a VM the first physical machine that contains suf-ficient available resources is selected Consequently evenat the beginning of the execution a physical machine mayhave several VMs In contrast Round Robin assigns thefirst available physical machine in the system using a ded-icated machine if possible In this case different VMs onlyshare a physical machine when there are no idle machinesin the system While the FIFO algorithm leads to physicalmachines being shared by different VMs where possi-ble Round-Robin leads to a dedicated physical machinebeing assigned to each VM Thus Cloud A typically usesfewer physical machines in executing a workload this beingreflected in the energy consumption This effect is more vis-ible when the system is not well-configured like cloud Bcausing a drop in the number of successful tests focusing onenergy consumption

There do not appear to be configurations that are perfectfor all the applications which means that in some casesone application demands a different configuration to obtainbetter performance This explains why in all cases some ofthe MRs were not satisfied When many tests do not fulfillan MR the user has to check those tests that fail and explorepotential solutions

5 Conclusions and future work

This paper presents a methodology that integrates simu-lation techniques with testing methods for checking thecorrectness of cloud systems In particular the iCanCloudsimulation platform has been used to model and simulate

Ann Telecommun

cloud systems while techniques inspired by metamorphictesting were used to validate these models

The main goal of this work was to facilitate the processof modelling and checking complete cloud systems semi-automatically Thus when the testing process is applied toa cloud model provided by the user the generated resultsshould provide useful information about how this cloudmodel is working

In order to show the usefulness and applicability of ourapproach different cloud systems were modelled and testedusing the proposed methodology While the first cloudmodel fits well with the typical architecture of cloud sys-tems the second cloud model provides a slow network thatdoes not fit well with the typical configuration of cloud sys-tems Moreover we gave different MRs each one focusedon checking a specific aspect of the cloud

The main objective of these experiments was to checkwhether the proposed method can reveal that a cloud modelis not well designed We expect that almost all tests willfulfill each MR when a cloud model is well designedIn contrast when we applied the technique to a cloudmodel defined using inappropriate values for parameters asignificant number of tests did not satisfy the MRs

Given an initial cloud model and different MRs it waspossible to automatically generate follow-up tests and checkwhether the MRs were satisfied The results of the experi-ments showed the potential to determine that a cloud systemmodel was poorly designed In this case the cloud that usesa slow network clearly shows a drop in the number of teststhat fulfil an MR focused on performance This is because aslow network acts as a system bottleneck

It is important to remark that although one of the thecloud models was poorly configured the same workloadcould be executed in both cloud models The main dif-ference lies in the time required to execute it Since bothmodels provided the required functionality almost all testssatisfied the MR focused on functionality Although the pro-cess of identifying mistakes in the design of a cloud systemis not entirely automated the user can focus on those MRsthat contains a high percentage of failures and this shouldhelp them find mistakes in a cloud model

Future work will present a formal definition of a cloudsystem in order to provide more consistent notation forincluding new metamorphic relations Further analysis ofa wide-range of cloud models will be performed Finallyan extended collection of metamorphic relations will bedefined

Acknowledgments This research was partially supported by theSpanish MEC projects TESIS (TIN2009-14312-C02-01) and ESTu-DIo (TIN2012-36812-C02-01)

References

1 Bertolino A Grieskamp W Hierons RM Le Traon Y Legeard BMuccini H Paradkar A Rosenblum D Tretmans J (2010) Model-based testing for the cloud In Practical Software Testing ToolAutomation and Human Factors no 10111 in Dagstuhl SeminarProceedings Schloss Dagstuhl - Leibniz-Zentrum fuer InformatikGermany pp 1ndash11

2 Buyya R Beloglazov A Abawajy J (2010) Energy-efficient man-agement of data center resources for cloud computing a visionarchitectural elements and open challenges In 16th Int Conf onParallel and Distributed Processing Techniques and ApplicationsPDPTArsquo10 CSREA Press pp 1ndash12

3 Buyya R Ranjan R Calheiros RN (2009) Modeling and simula-tion of scalable cloud computing environments and the CloudSimtoolkit challenges and opportunities In 7th High PerformanceComputing and Simulation Conference HPCSrsquo09 IEEE Com-puter Society pp 1ndash11

4 Calheiros RN Buyya R De Rose CAF (2010) Building anautomated and self-configurable emulation test bed for grid appli-cations Softw Pract Experience 40(5)405ndash429

5 Casanova H Legrand A Quinson M (2008) SimGrid a genericframework for large-scale distributed experiments In 10th IntConf on Computer Modeling and Simulation UKSIMrsquo 08

6 Castane G Nunez A Llopis P Carretero J (2013) E-mc2 a for-mal framework for energy modelling in cloud computing SimulModel Pract Theor 3956ndash75

7 Chen TY Sun C Wang G Mu B Liu H Wang ZS (2012) A meta-morphic relation-based approach to testing web services withoutoracles Int J Web Serv Res 9(1)51ndash73

8 Garber L (2011) News briefs IEEE Computer 44(6)18ndash209 Hierons RM Merayo MG Nunez M (2010) Mutation testing In

Laplante PA (ed) Encyclopedia of Software Engineering Tayloramp Francis

10 Kim KH Beloglazov A Buyya R (2009) Power-aware pro-visioning of cloud resources for real-time services In Pro-ceedings of the 7th International Workshop on Middleware forGrids Clouds and e-Science Urbana Champaign Illinois USApp 1ndash6

11 Kliazovich D Bouvry P Audzevich Y Khan S (2012) Green-cloud a packet-level simulator of energy-aware cloud computingdata centers J Supercomput 62(3)1263ndash1283

12 Myers G (2004) The Art of Software Testing 2nd edn Wiley13 Nunez A Fernandez J Filgueira R Garcıa F Carretero J (2012)

SIMCAN a flexible scalable and expandable simulation plat-form for modelling and simulating distributed architectures andapplications Simul Model Pract Theory 20(1)12ndash32

14 Nunez A Vazquez-Poletti JL Caminero AC Castane GG Car-retero J Llorente IM (2012) iCanCloud a flexible and scal-able cloud infrastructure simulator J Grid Comput 10(1)185ndash209

15 Ried S Kisker H Matzke P Bartels A Lisserman M (2011) Sizingthe cloudmdasha BT futures report Understanding and quantifying thefuture of cloud computing Forrester Research Report

16 Sulistio A Cibej U Venugopal S Robic B Buyya R (2008)A toolkit for modelling and simulating Data Grids an exten-sion to GridSim Concurr Comput Pract Experience 20(13)1591ndash1609

17 The Network Simulator NS-2 Web page at Date of last access8th August 2013 httpwwwisiedunsnamns

18 Weyuker EJ (1982) On testing non-testable programs Comput J25(4)465ndash470

Ann Telecommun

Table 3 Modelling of different types of tenants

Type Instances Matrix Server HPC VMsmall VMmed VMlarge Mapping

tcomp 30 5 0 0 3 0 0 Random

tHPC 25 0 0 3 0 0 8 First Fit

tserver 30 0 10 0 0 5 0 Best Fit

tmix 15 5 5 3 3 3 2 Random

storage server This application reads these matrices per-forms the calculation and finally the result is written to diskFinally the third application models an HPC applicationcalled BIPS3D [13]

Table 3 shows the modelling of four types of tenant Wesay that each type of tenant represents a group of users inthe cloud that have a similar behaviour The configurationof this behaviour can be set by using three parameters therented VMs the applications to be executed and how eachapplication is assigned to a rented VM The first columnrefers to the name that defines the type of a tenant The sec-ond column represents the number of simulated tenants ofa given type The next three columns represent the numberof applications requested by each type of tenant These arefollowed by three columns that represent the VMs rentedby each type of tenant The last column gives the algorithmused to map applications to VMs

We now describe the results of experiments that wereperformed by executing 100 tenants (see Table 3) overcloud A and cloud B (see Table 1) using three MRs(MRPER MRFUN and MRENE) by following the methodol-ogy described in Fig 3 In each testing process a total of100 cloud models were generated

Table 4 shows the number of tests that successfully ful-filled each MR We can observe that using cloud A thepercentage of successful tests is greater than 90 In con-trast there is a noticeable drop in the percentage of teststhat fulfil MRs when cloud B is used This difference is

mainly due to cloud B being poorly configured In this casethe parameter that configures the bandwidth of the switchesin cloud B is set to megabit per second instead of giga-bit per secon This ldquomistakerdquo causes a bottleneck in thesystem However we obtain similar results in both cloudmodels when using the MR focusing on the functionality ofthe cloud This is because the functional behaviour of bothmodels is correct even though the obtained performance isnot that expected While cloud A obtains good results whenanalysing performance and energy consumption cloud Bobtains around 60 of successful tests The main reasonfor this result lies in the saturation of the communicationsnetwork This leads to relatively little parallelism beingobtained when executing different VMs and so there beinglittle difference in the total time of execution of the originalmodel and the variants

Figure 4 shows the results of a subset of the experi-ments performed For the sake of clarity these charts showthe generated models by modifying only two parametersof the cloud CPU and number of storage servers Thesecharts show the tendency of the system when some changesare applied in the original model In this case the perfor-mance of each system is the target to be analysed which ismeasured in seconds (less is better)

Using cloud A the MRPER relation is fulfilled by 92100tests This indicates that cloud A is well configured for per-formance tests In contrast only 65 tests performed usingcloud B fulfilled the same relation This means that some

Fig 3 Testing methodology

Ann Telecommun

Table 4 Number of tests that successfully fulfilled each MR

Cloud Model MR PER MR FUN MR ENE

Cloud A 92100 97100 95100

Cloud B 65100 92100 63100

subsystem in the cloud is not working properly The use of abetter CPU does not lead to improved performance becausethe communication network is saturated

When using MRFUN with cloud A there are a few caseswhere the increase in the number of physical machines inthe model leads to the execution requiring more time Thissituation is mainly caused by the BIPS3D application Whenthis application is executed in different physical machinesthat are not connected to the same switch the time to per-form network operations is higher Similarly 92100 testspassed this relation when cloud B is used This means thatthis model works similarly when the size of the system

16 servers

8 servers

4 servers

2 servers

1 server

6000

6500

7000

7500

8000

8500

9000

9500

10000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud model A

16 servers

8 servers

4 servers

2 servers

1 server

6000

8000

10000

12000

14000

16000

18000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud Model B

Fig 4 Results of simulating cloud models A and B

is increased That is the bottleneck caused by the wrongconfiguration of the network does not affect this MR

Finally the relation MRENE is fulfilled by 97100 testcases when cloud A is used The algorithm used for map-ping applications to VMs is directly responsible for thethree tests that do not fulfill this MR In some cases anapplication that requires significant resources is executedin a VM that contains limited resources and vice versaIn these cases powerful VMs are wasted and applicationsthat require significant resources have long execution timesincreasing also the energy consumption In contrast usingcloud B only 63100 tests fulfilled this MR This is mainlycaused by the wrong configuration of the network sys-tem which results in a system bottleneck The bottleneckincreases the time required to execute a workload in thissystem Communications bottlenecks can lead to processorperformance being almost irrelevant making it more likelythat MRENE fails

The selection of the algorithm for allocating VMs tophysical machines also affects the results Cloud A uses aFIFO algorithm for allocating VMs to physical machinesTherefore when the cloud manager receives a request toexecute a VM the first physical machine that contains suf-ficient available resources is selected Consequently evenat the beginning of the execution a physical machine mayhave several VMs In contrast Round Robin assigns thefirst available physical machine in the system using a ded-icated machine if possible In this case different VMs onlyshare a physical machine when there are no idle machinesin the system While the FIFO algorithm leads to physicalmachines being shared by different VMs where possi-ble Round-Robin leads to a dedicated physical machinebeing assigned to each VM Thus Cloud A typically usesfewer physical machines in executing a workload this beingreflected in the energy consumption This effect is more vis-ible when the system is not well-configured like cloud Bcausing a drop in the number of successful tests focusing onenergy consumption

There do not appear to be configurations that are perfectfor all the applications which means that in some casesone application demands a different configuration to obtainbetter performance This explains why in all cases some ofthe MRs were not satisfied When many tests do not fulfillan MR the user has to check those tests that fail and explorepotential solutions

5 Conclusions and future work

This paper presents a methodology that integrates simu-lation techniques with testing methods for checking thecorrectness of cloud systems In particular the iCanCloudsimulation platform has been used to model and simulate

Ann Telecommun

cloud systems while techniques inspired by metamorphictesting were used to validate these models

The main goal of this work was to facilitate the processof modelling and checking complete cloud systems semi-automatically Thus when the testing process is applied toa cloud model provided by the user the generated resultsshould provide useful information about how this cloudmodel is working

In order to show the usefulness and applicability of ourapproach different cloud systems were modelled and testedusing the proposed methodology While the first cloudmodel fits well with the typical architecture of cloud sys-tems the second cloud model provides a slow network thatdoes not fit well with the typical configuration of cloud sys-tems Moreover we gave different MRs each one focusedon checking a specific aspect of the cloud

The main objective of these experiments was to checkwhether the proposed method can reveal that a cloud modelis not well designed We expect that almost all tests willfulfill each MR when a cloud model is well designedIn contrast when we applied the technique to a cloudmodel defined using inappropriate values for parameters asignificant number of tests did not satisfy the MRs

Given an initial cloud model and different MRs it waspossible to automatically generate follow-up tests and checkwhether the MRs were satisfied The results of the experi-ments showed the potential to determine that a cloud systemmodel was poorly designed In this case the cloud that usesa slow network clearly shows a drop in the number of teststhat fulfil an MR focused on performance This is because aslow network acts as a system bottleneck

It is important to remark that although one of the thecloud models was poorly configured the same workloadcould be executed in both cloud models The main dif-ference lies in the time required to execute it Since bothmodels provided the required functionality almost all testssatisfied the MR focused on functionality Although the pro-cess of identifying mistakes in the design of a cloud systemis not entirely automated the user can focus on those MRsthat contains a high percentage of failures and this shouldhelp them find mistakes in a cloud model

Future work will present a formal definition of a cloudsystem in order to provide more consistent notation forincluding new metamorphic relations Further analysis ofa wide-range of cloud models will be performed Finallyan extended collection of metamorphic relations will bedefined

Acknowledgments This research was partially supported by theSpanish MEC projects TESIS (TIN2009-14312-C02-01) and ESTu-DIo (TIN2012-36812-C02-01)

References

1 Bertolino A Grieskamp W Hierons RM Le Traon Y Legeard BMuccini H Paradkar A Rosenblum D Tretmans J (2010) Model-based testing for the cloud In Practical Software Testing ToolAutomation and Human Factors no 10111 in Dagstuhl SeminarProceedings Schloss Dagstuhl - Leibniz-Zentrum fuer InformatikGermany pp 1ndash11

2 Buyya R Beloglazov A Abawajy J (2010) Energy-efficient man-agement of data center resources for cloud computing a visionarchitectural elements and open challenges In 16th Int Conf onParallel and Distributed Processing Techniques and ApplicationsPDPTArsquo10 CSREA Press pp 1ndash12

3 Buyya R Ranjan R Calheiros RN (2009) Modeling and simula-tion of scalable cloud computing environments and the CloudSimtoolkit challenges and opportunities In 7th High PerformanceComputing and Simulation Conference HPCSrsquo09 IEEE Com-puter Society pp 1ndash11

4 Calheiros RN Buyya R De Rose CAF (2010) Building anautomated and self-configurable emulation test bed for grid appli-cations Softw Pract Experience 40(5)405ndash429

5 Casanova H Legrand A Quinson M (2008) SimGrid a genericframework for large-scale distributed experiments In 10th IntConf on Computer Modeling and Simulation UKSIMrsquo 08

6 Castane G Nunez A Llopis P Carretero J (2013) E-mc2 a for-mal framework for energy modelling in cloud computing SimulModel Pract Theor 3956ndash75

7 Chen TY Sun C Wang G Mu B Liu H Wang ZS (2012) A meta-morphic relation-based approach to testing web services withoutoracles Int J Web Serv Res 9(1)51ndash73

8 Garber L (2011) News briefs IEEE Computer 44(6)18ndash209 Hierons RM Merayo MG Nunez M (2010) Mutation testing In

Laplante PA (ed) Encyclopedia of Software Engineering Tayloramp Francis

10 Kim KH Beloglazov A Buyya R (2009) Power-aware pro-visioning of cloud resources for real-time services In Pro-ceedings of the 7th International Workshop on Middleware forGrids Clouds and e-Science Urbana Champaign Illinois USApp 1ndash6

11 Kliazovich D Bouvry P Audzevich Y Khan S (2012) Green-cloud a packet-level simulator of energy-aware cloud computingdata centers J Supercomput 62(3)1263ndash1283

12 Myers G (2004) The Art of Software Testing 2nd edn Wiley13 Nunez A Fernandez J Filgueira R Garcıa F Carretero J (2012)

SIMCAN a flexible scalable and expandable simulation plat-form for modelling and simulating distributed architectures andapplications Simul Model Pract Theory 20(1)12ndash32

14 Nunez A Vazquez-Poletti JL Caminero AC Castane GG Car-retero J Llorente IM (2012) iCanCloud a flexible and scal-able cloud infrastructure simulator J Grid Comput 10(1)185ndash209

15 Ried S Kisker H Matzke P Bartels A Lisserman M (2011) Sizingthe cloudmdasha BT futures report Understanding and quantifying thefuture of cloud computing Forrester Research Report

16 Sulistio A Cibej U Venugopal S Robic B Buyya R (2008)A toolkit for modelling and simulating Data Grids an exten-sion to GridSim Concurr Comput Pract Experience 20(13)1591ndash1609

17 The Network Simulator NS-2 Web page at Date of last access8th August 2013 httpwwwisiedunsnamns

18 Weyuker EJ (1982) On testing non-testable programs Comput J25(4)465ndash470

Ann Telecommun

Table 4 Number of tests that successfully fulfilled each MR

Cloud Model MR PER MR FUN MR ENE

Cloud A 92100 97100 95100

Cloud B 65100 92100 63100

subsystem in the cloud is not working properly The use of abetter CPU does not lead to improved performance becausethe communication network is saturated

When using MRFUN with cloud A there are a few caseswhere the increase in the number of physical machines inthe model leads to the execution requiring more time Thissituation is mainly caused by the BIPS3D application Whenthis application is executed in different physical machinesthat are not connected to the same switch the time to per-form network operations is higher Similarly 92100 testspassed this relation when cloud B is used This means thatthis model works similarly when the size of the system

16 servers

8 servers

4 servers

2 servers

1 server

6000

6500

7000

7500

8000

8500

9000

9500

10000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud model A

16 servers

8 servers

4 servers

2 servers

1 server

6000

8000

10000

12000

14000

16000

18000

CPU 4-Core 17

Ghz CPU 4-

Core 18 Ghz

CPU 4-Core 19

Ghz CPU 4-Core 2

Ghz Original Model

Tim

e (s

ec)

Cloud Model B

Fig 4 Results of simulating cloud models A and B

is increased That is the bottleneck caused by the wrongconfiguration of the network does not affect this MR

Finally the relation MRENE is fulfilled by 97100 testcases when cloud A is used The algorithm used for map-ping applications to VMs is directly responsible for thethree tests that do not fulfill this MR In some cases anapplication that requires significant resources is executedin a VM that contains limited resources and vice versaIn these cases powerful VMs are wasted and applicationsthat require significant resources have long execution timesincreasing also the energy consumption In contrast usingcloud B only 63100 tests fulfilled this MR This is mainlycaused by the wrong configuration of the network sys-tem which results in a system bottleneck The bottleneckincreases the time required to execute a workload in thissystem Communications bottlenecks can lead to processorperformance being almost irrelevant making it more likelythat MRENE fails

The selection of the algorithm for allocating VMs tophysical machines also affects the results Cloud A uses aFIFO algorithm for allocating VMs to physical machinesTherefore when the cloud manager receives a request toexecute a VM the first physical machine that contains suf-ficient available resources is selected Consequently evenat the beginning of the execution a physical machine mayhave several VMs In contrast Round Robin assigns thefirst available physical machine in the system using a ded-icated machine if possible In this case different VMs onlyshare a physical machine when there are no idle machinesin the system While the FIFO algorithm leads to physicalmachines being shared by different VMs where possi-ble Round-Robin leads to a dedicated physical machinebeing assigned to each VM Thus Cloud A typically usesfewer physical machines in executing a workload this beingreflected in the energy consumption This effect is more vis-ible when the system is not well-configured like cloud Bcausing a drop in the number of successful tests focusing onenergy consumption

There do not appear to be configurations that are perfectfor all the applications which means that in some casesone application demands a different configuration to obtainbetter performance This explains why in all cases some ofthe MRs were not satisfied When many tests do not fulfillan MR the user has to check those tests that fail and explorepotential solutions

5 Conclusions and future work

This paper presents a methodology that integrates simu-lation techniques with testing methods for checking thecorrectness of cloud systems In particular the iCanCloudsimulation platform has been used to model and simulate

Ann Telecommun

cloud systems while techniques inspired by metamorphictesting were used to validate these models

The main goal of this work was to facilitate the processof modelling and checking complete cloud systems semi-automatically Thus when the testing process is applied toa cloud model provided by the user the generated resultsshould provide useful information about how this cloudmodel is working

In order to show the usefulness and applicability of ourapproach different cloud systems were modelled and testedusing the proposed methodology While the first cloudmodel fits well with the typical architecture of cloud sys-tems the second cloud model provides a slow network thatdoes not fit well with the typical configuration of cloud sys-tems Moreover we gave different MRs each one focusedon checking a specific aspect of the cloud

The main objective of these experiments was to checkwhether the proposed method can reveal that a cloud modelis not well designed We expect that almost all tests willfulfill each MR when a cloud model is well designedIn contrast when we applied the technique to a cloudmodel defined using inappropriate values for parameters asignificant number of tests did not satisfy the MRs

Given an initial cloud model and different MRs it waspossible to automatically generate follow-up tests and checkwhether the MRs were satisfied The results of the experi-ments showed the potential to determine that a cloud systemmodel was poorly designed In this case the cloud that usesa slow network clearly shows a drop in the number of teststhat fulfil an MR focused on performance This is because aslow network acts as a system bottleneck

It is important to remark that although one of the thecloud models was poorly configured the same workloadcould be executed in both cloud models The main dif-ference lies in the time required to execute it Since bothmodels provided the required functionality almost all testssatisfied the MR focused on functionality Although the pro-cess of identifying mistakes in the design of a cloud systemis not entirely automated the user can focus on those MRsthat contains a high percentage of failures and this shouldhelp them find mistakes in a cloud model

Future work will present a formal definition of a cloudsystem in order to provide more consistent notation forincluding new metamorphic relations Further analysis ofa wide-range of cloud models will be performed Finallyan extended collection of metamorphic relations will bedefined

Acknowledgments This research was partially supported by theSpanish MEC projects TESIS (TIN2009-14312-C02-01) and ESTu-DIo (TIN2012-36812-C02-01)

References

1 Bertolino A Grieskamp W Hierons RM Le Traon Y Legeard BMuccini H Paradkar A Rosenblum D Tretmans J (2010) Model-based testing for the cloud In Practical Software Testing ToolAutomation and Human Factors no 10111 in Dagstuhl SeminarProceedings Schloss Dagstuhl - Leibniz-Zentrum fuer InformatikGermany pp 1ndash11

2 Buyya R Beloglazov A Abawajy J (2010) Energy-efficient man-agement of data center resources for cloud computing a visionarchitectural elements and open challenges In 16th Int Conf onParallel and Distributed Processing Techniques and ApplicationsPDPTArsquo10 CSREA Press pp 1ndash12

3 Buyya R Ranjan R Calheiros RN (2009) Modeling and simula-tion of scalable cloud computing environments and the CloudSimtoolkit challenges and opportunities In 7th High PerformanceComputing and Simulation Conference HPCSrsquo09 IEEE Com-puter Society pp 1ndash11

4 Calheiros RN Buyya R De Rose CAF (2010) Building anautomated and self-configurable emulation test bed for grid appli-cations Softw Pract Experience 40(5)405ndash429

5 Casanova H Legrand A Quinson M (2008) SimGrid a genericframework for large-scale distributed experiments In 10th IntConf on Computer Modeling and Simulation UKSIMrsquo 08

6 Castane G Nunez A Llopis P Carretero J (2013) E-mc2 a for-mal framework for energy modelling in cloud computing SimulModel Pract Theor 3956ndash75

7 Chen TY Sun C Wang G Mu B Liu H Wang ZS (2012) A meta-morphic relation-based approach to testing web services withoutoracles Int J Web Serv Res 9(1)51ndash73

8 Garber L (2011) News briefs IEEE Computer 44(6)18ndash209 Hierons RM Merayo MG Nunez M (2010) Mutation testing In

Laplante PA (ed) Encyclopedia of Software Engineering Tayloramp Francis

10 Kim KH Beloglazov A Buyya R (2009) Power-aware pro-visioning of cloud resources for real-time services In Pro-ceedings of the 7th International Workshop on Middleware forGrids Clouds and e-Science Urbana Champaign Illinois USApp 1ndash6

11 Kliazovich D Bouvry P Audzevich Y Khan S (2012) Green-cloud a packet-level simulator of energy-aware cloud computingdata centers J Supercomput 62(3)1263ndash1283

12 Myers G (2004) The Art of Software Testing 2nd edn Wiley13 Nunez A Fernandez J Filgueira R Garcıa F Carretero J (2012)

SIMCAN a flexible scalable and expandable simulation plat-form for modelling and simulating distributed architectures andapplications Simul Model Pract Theory 20(1)12ndash32

14 Nunez A Vazquez-Poletti JL Caminero AC Castane GG Car-retero J Llorente IM (2012) iCanCloud a flexible and scal-able cloud infrastructure simulator J Grid Comput 10(1)185ndash209

15 Ried S Kisker H Matzke P Bartels A Lisserman M (2011) Sizingthe cloudmdasha BT futures report Understanding and quantifying thefuture of cloud computing Forrester Research Report

16 Sulistio A Cibej U Venugopal S Robic B Buyya R (2008)A toolkit for modelling and simulating Data Grids an exten-sion to GridSim Concurr Comput Pract Experience 20(13)1591ndash1609

17 The Network Simulator NS-2 Web page at Date of last access8th August 2013 httpwwwisiedunsnamns

18 Weyuker EJ (1982) On testing non-testable programs Comput J25(4)465ndash470

Ann Telecommun

cloud systems while techniques inspired by metamorphictesting were used to validate these models

The main goal of this work was to facilitate the processof modelling and checking complete cloud systems semi-automatically Thus when the testing process is applied toa cloud model provided by the user the generated resultsshould provide useful information about how this cloudmodel is working

In order to show the usefulness and applicability of ourapproach different cloud systems were modelled and testedusing the proposed methodology While the first cloudmodel fits well with the typical architecture of cloud sys-tems the second cloud model provides a slow network thatdoes not fit well with the typical configuration of cloud sys-tems Moreover we gave different MRs each one focusedon checking a specific aspect of the cloud

The main objective of these experiments was to checkwhether the proposed method can reveal that a cloud modelis not well designed We expect that almost all tests willfulfill each MR when a cloud model is well designedIn contrast when we applied the technique to a cloudmodel defined using inappropriate values for parameters asignificant number of tests did not satisfy the MRs

Given an initial cloud model and different MRs it waspossible to automatically generate follow-up tests and checkwhether the MRs were satisfied The results of the experi-ments showed the potential to determine that a cloud systemmodel was poorly designed In this case the cloud that usesa slow network clearly shows a drop in the number of teststhat fulfil an MR focused on performance This is because aslow network acts as a system bottleneck

It is important to remark that although one of the thecloud models was poorly configured the same workloadcould be executed in both cloud models The main dif-ference lies in the time required to execute it Since bothmodels provided the required functionality almost all testssatisfied the MR focused on functionality Although the pro-cess of identifying mistakes in the design of a cloud systemis not entirely automated the user can focus on those MRsthat contains a high percentage of failures and this shouldhelp them find mistakes in a cloud model

Future work will present a formal definition of a cloudsystem in order to provide more consistent notation forincluding new metamorphic relations Further analysis ofa wide-range of cloud models will be performed Finallyan extended collection of metamorphic relations will bedefined

Acknowledgments This research was partially supported by theSpanish MEC projects TESIS (TIN2009-14312-C02-01) and ESTu-DIo (TIN2012-36812-C02-01)

References

1 Bertolino A Grieskamp W Hierons RM Le Traon Y Legeard BMuccini H Paradkar A Rosenblum D Tretmans J (2010) Model-based testing for the cloud In Practical Software Testing ToolAutomation and Human Factors no 10111 in Dagstuhl SeminarProceedings Schloss Dagstuhl - Leibniz-Zentrum fuer InformatikGermany pp 1ndash11

2 Buyya R Beloglazov A Abawajy J (2010) Energy-efficient man-agement of data center resources for cloud computing a visionarchitectural elements and open challenges In 16th Int Conf onParallel and Distributed Processing Techniques and ApplicationsPDPTArsquo10 CSREA Press pp 1ndash12

3 Buyya R Ranjan R Calheiros RN (2009) Modeling and simula-tion of scalable cloud computing environments and the CloudSimtoolkit challenges and opportunities In 7th High PerformanceComputing and Simulation Conference HPCSrsquo09 IEEE Com-puter Society pp 1ndash11

4 Calheiros RN Buyya R De Rose CAF (2010) Building anautomated and self-configurable emulation test bed for grid appli-cations Softw Pract Experience 40(5)405ndash429

5 Casanova H Legrand A Quinson M (2008) SimGrid a genericframework for large-scale distributed experiments In 10th IntConf on Computer Modeling and Simulation UKSIMrsquo 08

6 Castane G Nunez A Llopis P Carretero J (2013) E-mc2 a for-mal framework for energy modelling in cloud computing SimulModel Pract Theor 3956ndash75

7 Chen TY Sun C Wang G Mu B Liu H Wang ZS (2012) A meta-morphic relation-based approach to testing web services withoutoracles Int J Web Serv Res 9(1)51ndash73

8 Garber L (2011) News briefs IEEE Computer 44(6)18ndash209 Hierons RM Merayo MG Nunez M (2010) Mutation testing In

Laplante PA (ed) Encyclopedia of Software Engineering Tayloramp Francis

10 Kim KH Beloglazov A Buyya R (2009) Power-aware pro-visioning of cloud resources for real-time services In Pro-ceedings of the 7th International Workshop on Middleware forGrids Clouds and e-Science Urbana Champaign Illinois USApp 1ndash6

11 Kliazovich D Bouvry P Audzevich Y Khan S (2012) Green-cloud a packet-level simulator of energy-aware cloud computingdata centers J Supercomput 62(3)1263ndash1283

12 Myers G (2004) The Art of Software Testing 2nd edn Wiley13 Nunez A Fernandez J Filgueira R Garcıa F Carretero J (2012)

SIMCAN a flexible scalable and expandable simulation plat-form for modelling and simulating distributed architectures andapplications Simul Model Pract Theory 20(1)12ndash32

14 Nunez A Vazquez-Poletti JL Caminero AC Castane GG Car-retero J Llorente IM (2012) iCanCloud a flexible and scal-able cloud infrastructure simulator J Grid Comput 10(1)185ndash209

15 Ried S Kisker H Matzke P Bartels A Lisserman M (2011) Sizingthe cloudmdasha BT futures report Understanding and quantifying thefuture of cloud computing Forrester Research Report

16 Sulistio A Cibej U Venugopal S Robic B Buyya R (2008)A toolkit for modelling and simulating Data Grids an exten-sion to GridSim Concurr Comput Pract Experience 20(13)1591ndash1609

17 The Network Simulator NS-2 Web page at Date of last access8th August 2013 httpwwwisiedunsnamns

18 Weyuker EJ (1982) On testing non-testable programs Comput J25(4)465ndash470

本文献由ldquo学霸图书馆-文献云下载rdquo收集自网络仅供学习交流使用

学霸图书馆（wwwxuebalibcom）是一个ldquo整合众多图书馆数据库资源

提供一站式文献检索和下载服务rdquo的24 小时在线不限IP

图书馆

图书馆致力于便利促进学习与科研提供最强文献下载服务

图书馆导航

图书馆首页 文献云下载 图书馆入口 外文数据库大全 疑难文献辅助工具