“evaluating cloud services using super- efficiency dea and ... · evaluating the efficiency of...
TRANSCRIPT
A Summer Internship Project Report
On
“Evaluating Cloud Services Using super-
efficiency DEA and TOPSIS model” Carriedout at the
Institute of Development and Research in Banking Technology,
Hyderabad
Established by ‘Reserve Bank of India’
Submitted by
Akshay Jaiswal
Roll No. – 12400EN011
Integrated Dual Degree (Computer Science and Engineering), 2012-2017
Indian Institute of Technology (BHU) Varanasi
Under the Guidance of
Dr. G. R. Gangadharan
Assistant Professor
IDRBT, Hyderabad
July, 2015
CERTIFICATE
This is to certify that the summer internship project report entitled “Evaluating
Cloud Services Using super-efficiency DEA and TOPSIS model”submitted to
Institute for Development & Research in Banking Technology [IDRBT],
Hyderabad is a bonafide record of work done by “Akshay Jaiswal”, Roll no. –
12400EN011, IDD (CSE), 2012-17, Indian Institute of Technology (BHU)
Varanasi” under my supervision from “13th May, 2015” to “13th July, 2015”.
Dr. G. R. Gangadharan
Assistant Professor,
IDRBT, Hyderabad
Place: Hyderabad
Date: 13th July, 2015
ACKNOWLEDGEMENT
I would like to express my profound gratitude and deep regards to my guide Dr. G. R.
Gangadharan, Assistant Professor, IDRBT, Hyderabad for his guidance, and constant
encouragement throughout the course of this summer internship project.
I also take this opportunity to express a deep sense of gratitude to Dr A. S. Ramasastri,
Director, IDRBT, Hyderabad for his cordial support by providing excellent facilities like labs
and library.I am also grateful to IDRBT staff for their cooperation during the period of my
assignment.
Lastly, I thank my parents, and friends for their constant encouragement and moral support
without which this assignment would not have been possible.
Akshay Jaiswal
Integrated Dual Degree (2012-2017)
Computer Science and Engineering
IIT (BHU) Varanasi
Evaluating the Efficiency of Cloud Services Using
Super-efficiency DEA and TOPSIS
Abstract – With the growing demand and commercial availability of cloud services, the need
for comparison of their functionality available to customers at different prices and
performance has arisen. It is needed to be said that relevant and fair comparison is still
challenging due to diverse deployment options and dissimilar features of different services.
The aim of this paper is to rank cloud services using super-efficiency DEA and TOPSIS.
Keywords – Cloud services, super-efficiency DEA, AHP, TOPSIS
1 INTRODUCTION
Cloud computing has gained tremendous momentum in past few years as the use of
computers in our day-to-day life has increased effectively. Cloud computing offers
undeniable advantages in terms of cost and reliability compared to the traditional
computing model that uses a dedicated in-house infrastructure. Cloud customers don’t
have to pay large sums of money to register for using cloud services, they only need to pay
for what they actually use.
There is a high growth in number of companies that provide public cloud computing
services, such as Amazon, Google, Microsoft, Rackspace, and GoGrid. They offer various
options in pricing, performance and feature set. There are broadly three delivery models
that are provided:
Software-as-a-Service (SaaS), is a software distribution model in which applications
are hosted by a vendor or service provider and made available to customers over a
network, typically the Internet.
Platform-as-a-Service (PaaS), is a paradigm for delivering operating systems and
associated services over the Internet without downloads or installation.
Infrastructure-as-a-Service (IaaS), involves outsourcing the equipment used to
support operations, including storage, hardware, servers and networking
components.
The presence of these many cloud service providers awakens a question: “How good a cloud service provider performs compared to the others?” Answering this will benefit both the customers and the providers. For potential customers, the answer can help them choose a provider that best fits their performance and cost needs. For instance, they may choose one provider for storage intensive applicationsand another for computation intensive applications. For cloud providers, such answers can point them in the right directionfor improvements.
This paper discusses TOPSIS and fuzzy TOPSIS for comparative study of cloud service
providers like Amazon, HP, Azure, Rackspace, Google Compute Engine, Century Link and
City-Cloud. For each service provider, service levels which differ in Virtual Cores and
Memory are considered for the performance evaluation. The parameters considered for
evaluating the QoS is user specific with a relative preference value. The following
calculated Benchmark values are used for performance evaluation, though it can be any
user specific.
CPU Performance
Disk I/O Consistency
Disk Performance
Memory Performance
Price of a service is dependent on number of Virtual Cores and Memory, so only Price is
considered during the evaluation process. The two service level for each service
provider is selected based on number of virtual cores in which we considered virtual
core-2, virtual core- 4 and virtual core-8 for each service provider which among
themselves differs on price and memory.
2. Literature Review
In this section, we briefly describe some of the existing models to evaluate the relative
performance and ranking of cloud services.
With the increasing popularity of cloud computing,a lot of research has been done to
compare the cloud services for different type of applications such as scientific computing,
web services based on attributes including security, accountability, assurance,
performance,cost etc. However, there are research papers which have done comparison based
on properties of the alternative without the comparison of performance[Buyyaet al., 2008].
Kabir et al. (2012) hasevaluated the major factors for travel agency websites quality from the
viewpoint of users' perception and developed a systematic multiple-attribute evaluation
model using Technique for Order Preference by Similarity to Ideal Solution(TOPSIS) and
Fuzzy TOPSIS, to find out the effective travel agency websites. A comparative analysis of
TOPSIS and Fuzzy TOPSIS methods are illustrated in this paper through a practical
application from the websites of five travel agencies. We have done a similar study for cloud
services.
Aryanezhad et al. (2011) takes the values of alternatives with respect to the criteria or/and the
values of criteria weights as fuzzy numbers. In this paper, fuzzy TOPSIS (for Order
Preference by Similarity to Ideal Solution) method based on left and right scores for fuzzy
MADM problems is proposed, and its applicability is shown using two numerical examples.
3. Modelling Super-efficiency Data Envelopment Analysis for Cloud
Services
Since the early 1980s, Data Envelopment Analysis (DEA) has been used as an alternative
method of classification for evaluating the relative efficiency of independent homogenous
units which use the same inputs to produce the same outputs (Cooper, Seiford and Tone,
2000). However, a serious inconvenience in using DEA as a method of classification is the
room of having units tied with relative efficiency equal to 100 percent. That is, units at the
efficient frontier.
The objective function of the input oriented model used for super-efficiency DEA is shown
below:
Subject to
where DMUE is an efficient DMU and other symbols have their usual meanings.
This will give us a super-efficiency score of greater than one,enabling us to distinguish
between the efficient observations. In particular, the super-efficiency measure examines the
maximal radial change in inputs and/or outputs for an observation to remain efficient, that is,
how much can the inputs be increased, or the outputs decreased, while not becoming
inefficient. The larger the value of the super-efficiency measure, the higher an observation is
ranked among the efficient units. Super-efficiency measures can be calculated for both
inefficient and efficient observations, but in the case of inefficient observations the values of
the efficiency measure do not change, while efficient observations may obtain higher values.
The super-efficiency DEA model being used goes through a modification to find the
preferred efficiency using AHP. The modified objective function is shown below:
Subject to
4. Modelling TOPSIS for Cloud Services
Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) method was
introduced by Hwang and Yoon (1981). It is a Multi Attribute Decision Making (MADM)
technique that sorts the alternatives for a decision problem in order of distance from the
positive and negative ideal solutions. The best solution is the one which has minimum
distance from the positive ideal solution and maximum distance from the negative ideal
solution [Benitez et al., 2007]. The positive ideal solution is the one which maximizes the
benefits and minimizes the costs while the negative ideal solution is the one which minimizes
the benefits and maximizes the costs [Lin et al., 2008].
The procedure for TOPSIS model is described below:
Step 1: Construct the Decision Matrix for the alternatives based on the criteria with
each row representing an alternative and each column representing a criterion. So
element xijrefers to ith
alternative and jth
criterion.
The matrix D is defined as:
C1 C2 C3 . . . Cn
A1x11 x12 x13 . . . x1n
A2x21 x22 x23 . . . x2n
A3x31 x32 x33 . . . x3n
D = .. . . . . . .
. . . . . . . .
. . . . . . . .
Am xm1 xm2 xm3 . . . xmn
Where Ai (A1, A2, A3, ……., Am)represents m alternatives and Cj (C1, C2, C3, …….,
Cn) represents ncriteria.
Step 2: Find the normalized decision matrix D using the following formula:
Step 3: Compute the weighted normalized decision matrix by multiplying weights of
criteria wj for j = 1, ……, n with the associated columns. The weighted normalized
value vij is calculated as:
The weights are calculated by using Analytic hierarchy Process (AHP) which is
explained in section 5 of this paper.
Step 4: Determine the positive ideal solution ( ) i.e. best performance from each
criteria and negative ideal solution ( i.e. least performance from each criteria.
Where J is the set of benefit attributes (larger-the- better type) and J' is the set of cost
attributes (smaller-the-better type).
Step 5: Compute separation of each criterion value for each alternative from both
positive ideal and negative ideal solutions.
The formula for calculating the separation from the positive ideal solution is:
And the formula for calculating the separation from the negative ideal solution is:
Where i = 1, 2, 3, ……., m
Step 7: Determine for each alternative, the ratio which is the relative closeness to the
ideal solution or the similarities to the ideal solution.
Larger value of indicates better performance of the alternatives.
Step 8: Rank the alternatives based on the value of .
5. Analytic Hierarchy Process (AHP)
Analytical Hierarchy process (AHP) was used to modify the objective function of super-
efficiency DEA and to find the weight for the criteria which was further used to find the
weighted decision matrices in case of TOPSIS.The scale we use ranges from 1 to 9 as
presented in Table 1.
Intensity of Importance Definition
1 Equal Importance
3 Moderate Importance of one over another
5 Essential or Strong Importance
7 Very Strong Importance
9 Extreme Importance
2,4,6,8 Intermediate value between the two adjacent judgments
Table 1. Scales for comparison
Table 2 shows the relative importance among QoS attributes. For example, the relative importance of
CPU performance is 7 times as that of Disk I/O consistency and the relative importance of disk
performance is 0.2 (1/5) times as that of memory performance, and its reciprocal is in lower triangular
matrix i.e. memory performance is 5 times as important parameter as disk performance.
CPU
performance
Disk I/O
Consistency
Disk Performance Memory
Performance
CPU
performance
1 7 5 2
Disk I/O
Consistency
0.14 1 0.33 0.2
Disk
Performance
0.2 3 1 0.2
Memory
Performance
0.5 5 5 1
Table 2. Relative importance among QoS attributes
After normalizing the resultant matrix and averaging the value, we get the following weights
(as in Table 3):
QoS attributes Weights
CPU Performance 50.24%
Disc I/O Consistency 5.70%
Disc Performance 11.08%
Memory Performance 32.98%
Table 3. Weights for QoS attributes
To check the consistency of the calculated weights, we obtain consistency ratio (CR) as
0.052108. Consistency ratio tells how inconsistent the matrix is, and the result is acceptable if
CR <= 0.1. So our matrix is consistent and weights are valid.
These weights are then used to evaluatepreferred values of theperformance attributes which is
done by multiplying these fractions to the sum of normalised output parameters.
Normalisation is done to remove the units of different attributes so that all the outputs can be
added to evaluate the desired precise value. This new data set emerging from AHP can be
termed as preferred data set. Super-efficiency DEA is then applied on this preferred data set
to find the preferred efficiency.
The new efficiency score can be used directly to rank the cloud services considering priority
on performance benchmarks.
3. Data Collection Methodology and Data Set Description
We have considered 7 cloud service providers which include Amazon, HP, Azure,
Rackspace, Google, Century Link, and City-Cloud (without any order). For each service
provider, the number of virtual cores are considered for the cloud services offered by them.
The services having 2-virual cores are specified as large cloud services, those having 4-
virtual cores are specified as extra-large cloud services and the ones having 8-virtual cores
are specified as 2x-extra-large cloud service. The Dataset for the analysis is illustrated in
Table 5. The cloud service providers are coded as C1, C2....., and C7 (without any order) and
the services provided by each are further coded as S1, S2, S3, and so on. The specified data
(mentioned in column 3, 4, and 5) for Price/Hour (cents), Virtual core, Memory (GB) for
each service are taken from their respective websites. We considered the benchmark values
(mentioned in column 5 onwards) for CPU performance, Disk I/O consistency, Disk
performance, and Memory performance for evaluating the performance of the cloud services,
obtained from cloudharmony.com (a cloud benchmarking service).
During the data collection, consistency check is performed on data to identify if they are out
of range, logically inconsistent, or have extreme values. Inconsistent data for any service
provider are inadmissible and we either corrected it if possible or we did not consider the
service provider for the analysis. There are few missing values for benchmarks of the cloud
services. There is no trend in the data set for cloud services within a service provider and
among the service providers so as to substitute a neutral or mean value. Hence, we deleted
those cases with incomplete benchmark values.
One may argue that the different quantitative QoS attributes of cloud service providers
considered in this study are rather limited. However, it should be noticed that collecting the
real world data set regarding quantitative QoS attributes of cloud service providers were
extremely challenging.
Providers Service
price/Hr
(cents)
Virtual
core Memory
CPU
Performance Disk IO Consistency
Disk
Performance
Memory
Performance
C1 C1S2 28 4 15 25.86 92.89 110.33 129.03
C1S3 56 8 30 48.23 53.28 67.22 131.79
C2S1 14 2 7.5 13.89 114.44 97.38 144.86
C2 C2S2 28 4 15 23.66 119.63 100.55 131.81
C2S3 56 8 30 51.7 77.46 73.44 125.59
C3 C3S2 16 4 4 7.21 70.29 125.48 54.28
C3S3 32 8 8 15.33 57.11 111.18 55.68
C4S1 18 2 3.5 8.83 67.87 83.73 52.27
C4 C4S2 36 4 7 16.07 67.97 78.49 61.8
C4S3 72 8 14 28.4 78.72 70.91 27.33
C5S1 12 2 4 16.41 23.43 40.23 80.67
C5 C5S2 45 4 15 32.4 29.07 42.47 90.83
C5S3 90 8 30 52.82 35.35 55.07 83.92
C6S1 8 2 4 17.34 43.02 141.23 51.71
C6 C6S2 16 4 8 37.05 36.15 102.74 132.87
C6S3 32 8 16 71.11 39.66 99.15 135.88
C7S1 10.132 2 4 23.43 89.31 173.49 89.84
C7 C7S2 20.8624 4 8 42.05 59.63 174.5 97.16
C7S3 34.6528 8 16 75.89 64.64 174.12 100.14
Table 4. The collected dataset
7. Evaluation of Cloud Services using super-efficiency DEA
Our experimental evaluation is based on the input-oriented super-efficiency DEA modelthat
addresses the problem “By how much can input parameter (Price/Hour) be proportionally be
decreased without changing the output parameter (Performance Benchmark values)”. The
input parameter considered for the analysis is Price/Hour charged by the cloud service
provider for a cloud service. We employed four output parameters; however any number of
parameters could be included for both outputs and inputs. The output parameters include
CPU performance, Disk I/O consistency, Disk performance, and Memory performance.
Figure 1. Relative Efficiency Score of Cloud Services using super-efficiency DEA model
0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
120.00%
140.00%
160.00%
C
4S3
C
3S2
C
4S2
C
5S3
C
5S2
C
1S2
C
2S3
C
4S1
C
1S1
C
3S1
C
2S2
C
5S1
C
7S2
C
7S3
C
6S3
C
6S2
C
6S1
C
2S1
C
7S1
Super-efficiency DEA
Efficiency Efficiency (Preferred)
Table 5. Calculated Slack Value and Total slack for each cloud service.
The results of super-efficiency DEA model is shown in figure 1. The result in blue
colourindicates the efficiency score of DMUs with equal preference on QoS attribute i.e.
while calculating the efficiency score all performance benchmarks are given equal
weightage.Super-efficiency DEA is used to rank DMUs which are on the efficient frontier
line [Adler and Friedman, 2002]. The calculated slack value for each cloud services is
indicated in Table 5.
Apart from super-efficiency DEA, a modified version was also used to evaluate the preferred
efficiency based on priorities of the user. The result is shown in figure 1in red color.
C7S1 is performing relatively better among other cloud services with virtual core 2. The
service C7S1 provides better performance on the preference QoS attribute at a reasonable
service charge, however it has virtual core-2. So if a user wishes for a higher service with
higher virtual core (for example, a service with virtual core-4), then C6S2 is efficient on
relative scale. Similarly, among services with virtual core-8, C6S3 is better than other
services. Overall it can be seen that service provider C6 are relatively performing best among
considered cloud service providers.
8. Evaluation of Cloud Services using TOPSIS
For the analysis of cloud services, we can use benchmark values and also Price per
hour for the usage of cloud services as criteria. So in this case,maximum value of
performance-oriented benchmark for positive ideal solution and minimum value of
price data and time-oriented benchmarks are considered under each criteria. Whereas
in case of negative ideal solution, minimum value of performance-oriented
benchmark and maximum value of price data and time-oriented benchmark are
considered.
We use price per hour, CPU performance, disk I/O consistency, disk performance and
memory performance as criteria and cloud services C1S2, C1S3,……, C7S3 as the
alternatives. For giving the preference to the criteria, we use AHP.
Positive ideal solution is obtained by considering maximum value under each benchmark
attribute and minimum value under price per hour attribute. Similarly, negative ideal solution
is obtained by considering minimum values among benchmark attribute and maximum value
under price per hour attribute among the cloud services.
According to TOPSIS, the preference order of the cloud service providers is given below in
Figure 2.
Figure 2: Cloud services ranked in descending order according to TOPSIS
According to TOPSIS model also, service provider C6 and C7 are performing relatively better.
C6S2 service is relatively has best performance with Relative Ratio of 0.775, which indicates
that the service is 77.5% of the ideal solution. Among the virtual core-8 service also service
provider C6 with service C6S3 is performing the best with the ratio of 73.7%. For the service
among virtual core-2, service provider C7 with service C7S1 is relatively performing better.
Interestingly, it can be seen that Service Provider C4 and C5 are performing well in the
service with virtual-core-2 but the ratio decrease for their service with virtual-core-4 and 8.
This may be because the ratio of providing high quality of service to price charged is very
low which means that they are charging heavily for providing more quality service.
9. Conclusion and Future Work
With the growing demand for the cloud service there are many cloud service
providers are available with many cloud services with different prices and
performance. It has now become a challenge to cloud customers to select the best
cloud service which will satisfy their required QoS attribute. To select the best
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
TOPSIS
service customers need to have a framework to identify and measure key
performance criteria according to their requirement in the applications.
The Study discussed how to select the best cloud service model among various
service providers based on user specific QoS attributes. Although many
frameworks exist to evaluate the performance of cloud service, the study
introduced super-efficiency DEA model for the evaluation of cloud services
which has added advantage that it give projected values for increasing the
performance to a competing level. Other models that were used to evaluate the
performance of cloud services were AHP and TOPSIS. AHP was used to
evaluate the relative preference of QoS attribute. We used this preferences in
analysis with the other models.
In future we will incorporate other performance benchmarks other than
traditional High Performance Computing Benchmarks as these benchmarks focus
primarily on static system specific to performance and cost. A Paper [Binning]
propose metrics for measuring peak load handling, fault tolerance of cloud
computing, scalability and cost.
References
Aryanezhad, B., Tarokh, M. J., Mokhtarian, M. N., &Zaheri, F. (2011). A fuzzy TOPSIS
method based on left and right scores. International Journal of Industrial Engineering &
Production Research, 22(1).
Binning, C., Kossamann. D., Kraska. T., Loesing. S. (2009). How is the weather tomorrow?:
Towards a benchmark for the cloud. In: Proceedings of the Second International Workshop
on Testing Database Systems. ACM.
Cloudharmony.com. May 2014. https://cloudharmony.com/ Fathi, M. R., Matin, H. Z., Zarchi, M. K., &Azizollahi, S. (2011). The application of fuzzy TOPSIS approach to personnel selection for Padir Company, Iran. Journal of management Research, 3(2). Saaty, T., (2000). Fundamental of decision making and priority theory with analytic hierarchy process. RWS publications, USA. Yawe, B. (2010). Hospital Performance Evaluation in Uganda: A Super-Efficiency Data Envelope Analysis Model. Zambia Social Science Journal,1(1), 6. Zheng, Z., Wu, X., Zhang, Y., Lyu, M.R., Wang, J. (2013). QoS Ranking Prediction for Cloud Services. IEEE Transactions on Parallel and Distributed Systems. 24 (6). pp. 1213-1222.
Project Summary
Name of the Student: Akshay Jaiswal
Course and Year of Study: Integrated Dual Degree, 3rd
year
Name of the Institution: IIT (BHU) Varanasi
Name of the Project: Evaluating Cloud Services Using super-efficiency DEA and
TOPSIS model
Name of the Guide: Dr G R Gangadharan
Project Description: – With the growing demand and commercial availability of cloud
services, the need for comparison of their functionality available to customers at different
prices and performance has arisen. It is needed to be said that relevant and fair comparison is
still challenging due to diverse deployment options and dissimilar features of different
services. The aim of this paper is to rank cloud services using super-efficiency DEA and
TOPSIS.
Objectives: Ranking of cloud services using models such as super efficiency DEA and
TOPSIS to help the customers to choose a better service and to encourage cloud service
providers to improve their quality of services.
Deliverables: A highly organised ranking of the cloud services