11cloud-spectator-report_final.pdf
Post on 20-Feb-2016
219 Views
Preview:
TRANSCRIPT
Cloud IaaS Performance & Price-Performance Comparing Linux Compute Performance of 1&1, Amazon AWS, Aruba Cloud,
CloudSigma, and Microsoft Azure
Prepared for 1&1 on Behalf of SolidFire
Commercial Report
Published on 6/2015
Cloud Spectator Performance Report
1
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Contents
1. Introduction 2
Why Performance Matters 2
2. Executive Summary 4
Findings 4
3. Methodology 6
Process 6
Tests Used 7
Provider Data Center/Region Locations 8
VM Configurations and Pricing 8
Understanding Performance Results 10
Understanding The CloudSpecs Score (Price-Performance) 10
Key Considerations 11
4. Detailed Performance Findings 12
Processor & Memory Bandwidth 12
Disk IOPS: Sequential and Random Operations 14
Internal Network 17
5. Detailed Price-Performance Findings 19
Processor & Memory Bandwidth 19
Disk IOPS: Sequential and Random Operations 21
Internal Network 24
6. Conclusion 26
7. About 26
Cloud Spectator Performance Report
2
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Introduction
In an effort to simulate an end-user experience regarding performance of virtual machines across various cloud providers, Cloud Spectator
ran its iterative benchmark suite for 72 hours on each of the following providers: 1&1, AWS, Aruba Cloud, Microsoft Azure, and CloudSigma. SolidFire sponsored this study on behalf of its client, 1&1.
In most cases, much of the work was straightforward regarding provisioning and the setup process. Occasionally, vendor-side issues
occurred during the provisioning process, and the Cloud Spectator team contacted the corresponding vendor’s support team in order to resolve
issues such as VM provisioning errors. Three primary VMs of each size were tested on all providers for 24 hours each (72 hours total). This was done sequentially; once one VM had run the test suite for 24 hours, that VM was terminated and a new VM was created.
This study not only examined the performance of each vendor, but also tracked performance variability for each of the three 24-hour
periods. The methodology allowed Cloud Spectator to capture performance variability both over time on the same VM as well as across different VMs on multiple physical hosts. Some providers, such as 1&1, show strong processor and memory bandwidth performance stability for all of its VMs
throughout the course of the study. Other providers, such as AWS, exhibited controlled periods of burst followed by throttled performance on network
storage depending on the size of the storage volume. Others, such as CloudSigma, exhibited unstable performance across all resources throughout
the study, possibly due to server-side issues at the time of the study, which contributed to provisioning problems as well. Taking performance and stability a step further, price-performance analyses are conducted to help readers understand the value ratio
between the cost of the VM and the performance output. While the performance output is limited to the data points collected in the study, by
comparing the price-performance ratio, readers can gain better insight into the overall user experience seen on these providers.
Why Performance Matters
Performance and pricing are both key considerations in the public cloud industry, together having a substantial impact on a company’s annual
operating costs. Cloud users may need fewer resources on better performing services, which can lower costs. Since many users only consider price and not price-performance, these users may be paying more because they require additional resources to achieve a desired level of performance.
While some providers try to differentiate their offerings by cutting prices, others try to differentiate by focusing on improved performance and user
experience.
Differences in performance outputs of VMs across IaaS providers
can greatly impact quality of service as well as annual operating
costs. The graph on the right illustrates an example of the average processor performance from a sample of six Cloud
Service Providers (CSPs) as studied by Cloud Spectator. CSP 1
has a processor performance three times as high as CSP 6
(names removed), which gives CSP 1 a notable advantage in many processor-intensive workloads. CSPs 2-5 exhibit a closer
resemblance in processor performance, but do not offer nearly as
much processing power as CSP 1 does.
Cloud Spectator Performance Report
3
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
The table below lists the 3 hardware components studied in this project, and each purpose as a function in the server.
CPU & MEMORY PERFORMANCE
STORAGE PERFORMANCE
NETWORK PERFORMANCE
The performance of all applications is highly
dependent on the CPU. The CPU is responsible for
the processing and orchestration of all applications.
The relationship between CPU performance and RAM
is also observed by examining RAM bandwidth. While
memory performance is not considered one of the key
bottlenecks in performance for many applications, a
subset of applications—particularly HPC and in-
memory databases—is highly dependent on large
sustained memory bandwidth.
Because most applications and all data reside
on the disk, having fast disk performance is a
key consideration for best application
performance in many cases.
In a cloud environment, network performance
is a critical piece. Scalability, in many cases,
is dependent on the availability of additional
VMs that must maintain a strong network
backbone.
Cloud Spectator Performance Report
4
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Executive Summary
On behalf of 1&1, SolidFire commissioned Cloud Spectator to gauge the performance of VMs on five different cloud providers’ European data
centers: 1&1, Amazon AWS, Aruba Cloud, CloudSigma, and Microsoft Azure. Both performance and price-performance were examined to evaluate the value of each provider’s VMs. The purpose of the study was to understand, from an end-user perspective, the disparity of performance and value
(defined as price-performance) among cloud providers with similarly sized VMs. Overall, 1&1 exceled in performance and price-performance for all
component resources of the VMs tested. Its high performance rankings, combined with hourly pricing, introduces powerful, scalable cloud
infrastructure at low cost to its users.
Findings
vCPU & Memory Performance Findings
For this study, Cloud Spectator evaluated vCPU and memory
bandwidth performance by benchmarking the VMs using Geekench
3, a suite of benchmark tests that simulate tasks such as cryptographic encoding and image processing. Testing occurred over
the course of a 72-hour testing period. Pricing was examined in
conjunction with the performance tests.
vCPU & Memory Performance Key Findings:
• 1&1’s VMs achieved the highest performance across processor and memory performance in the study.
• 1&1’s VMs achieved the highest CloudSpecs ScoreTM in the test
group, indicating the strongest price-performance value for
processor and memory bandwidth.
• CloudSigma’s processor performance varied the most. Its
coefficients of variation (CV), which is a percentage expressing the
relationship between the average and standard deviation (useful for determining variability in performance), ranges up to 43.7%.
• The virtual processors on 1&1, AWS, and Azure exhibited the most
stability throughout the study, resulting in less than 3% coefficients of variation.
Cloud Spectator Performance Report
5
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Storage Performance Findings
Storage was evaluated using the FIO disk benchmark, which tests
the sequential read/write and random read/write operations of storage. In this study an 8KB block size was used. Testing occurred
over a 72-hour test period. Persistent storage (offered as “block
storage” or “redundant storage”) was used in all storage tests. Pricing
was examined in conjunction with the performance tests.
Storage Performance Key Findings:
• 1&1 VMs offered the highest disk performance among all included in this study.
• Although AWS offers SSD technology on its block storage offering,
EBS, the performance of that offering is more dependent on the size of the block storage volume provisioned.
• Despite being one of the lower-tier performers in disk IOPS, Azure
displayed the most stable disk performance throughout the study.
• Out of all providers examined in the study, only AWS appeared to
provide a period of burst performance for its block storage. On the
2vCPU VMs, which have 100GB of block storage, AWS volumes
displayed a burst behavior. After the period of burst, sequential read/write operations and random read/write operations dropped to
become 10% and 20%, respectively, of the initial IOPS achieved
during burst. This burst behavior was not seen on VMs with 400GB and 800GB of block storage, due to the larger number of IOPS.
• 1&1 displayed the best price-performance value for disk IOPS. Its
high-performance SAN disk offering is designed with SolidFire technology.
Internal Network Performance Findings
Internal network performance was measured as the throughput between VMs within the same data center of the cloud provider
(measured using iperf and ping respectively) over the course of a 72-
hour test period. Pricing was examined in conjunction with the performance tests.
Internal Network Performance Key Findings:
• CloudSigma VMs achieved the highest internal network
throughput, although the high throughput is unstable and
fluctuates between less than 500Mbits to over 10Gbits.
• CloudSigma displayed the best price-performance value for
internal network. Despite its large fluctuation in internal network
throughput, its median score range significantly outperformed all other providers.
• 1&1 achieved the second-highest internal network CloudSpecs
ScoreTM.
• Other than CloudSigma, only certain Azure VMs exceeded 1 GB/s
throughput.
Cloud Spectator Performance Report
6
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Methodology
Cloud Spectator strives to create a transparent and detailed methodology to allow readers to understand the testing process and recreate any
studies. If any information is unclear or if you have any questions, please email the team at contact@cloudspectator.com or call +1 (617) 300-0711.
Process
1. Three iterations of 24-hour test cycles were run for each VM on each provider for a total of 72 hours of testing per VM size. After each 24-
hour block, VMs were terminated before beginning another cycle of tests on newly provisioned machines. 2. Each VM was provisioned with a Linux Ubuntu 14.04 OS by default, available from all providers. For AWS, the HVM image was used.
3. Before each 24-hour test period, and after provisioning the VMs, system updates and upgrades were conducted via apt-get.
4. The following dependencies were installed for testing: a. Git. Git was used to clone the test repository on the VM.
b. MySQL. For automation purposes, mysql-server was installed to automate data uploads.
c. Pip. Used to download the appropriate libraries for Python in order to run the testing. SQL Alchemy was downloaded to interact
with MySQL and upload data. d. Libmysqlclient-dev. MySQL database’s development files, which are necessary for the SQL Alchemy and MySQL interaction.
5. Each test cycled through in the following sequence: Geekbench 3 (process & memory), fio sequential operations, fio random operations,
Iperf internal network throughput (for more information on testing, see Tests Used). a. For fio testing (to measure disk IOPS), sequential operations ran first. Files from the sequential tests were deleted, and fio
recreated files before running random operations. Once random operations completed, the files were also deleted. Thus, before
each disk IOPS test, the files associated with the tests were deleted and recreated.
6. Internal network testing was conducted in one of the following manners: a. On AWS and Azure, where VMs demonstrated varying internal network throughput depending on size and/or instance type, a
clone of that VM was created in the same region/availability zone. The cloned server listened for a TCP connection via Iperf.
E.g., two c4.large instances were created in Amazon AWS’s EU West 1 region to test throughput. The cloned VM was terminated alongside the tested VMs at the conclusion of each 24-hour test cycle.
b. On 1&1, Aruba Cloud, and CloudSigma, where VMs did not demonstrate varying internal network throughput depending on size
and/or instance type, a screen session was created on each VM to listen for a TCP connection. Each category of VMs, which
contains 2 VMs, connected with each other to perform throughput testing; i.e., 1&1’s 2 vCPU 4GB RAM and 2 vCPU 8GB RAM virtual machines conducted network throughput tests across one another.
7. A total of approximately 1.4 million data points were collected throughout the period of the study.
8. At the end of each test iteration, results were uploaded into Cloud Spectator’s database through use of SQL Alchemy (Python—see 4c in Process).
Cloud Spectator Performance Report
7
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Tests Used
Processor & Memory Bandwidth: Geekbench 3
Geekbench 3, a licensable product created by Primate Labs, is a cross-platform processor benchmark that can measure single-core and multi-core performance by simulating real-world workloads. The Geekbench 3 test suite is comprised of 27 individual tasks/workloads: 13 integer workloads, 10
floating point workloads, and 4 memory-bandwidth tasks. While processor and memory bandwidth are both performance factors that contribute to the
final score provided by Geekbench 3, the test suite weighs processing performance much more heavily than memory bandwidth. Also, memory
bandwidth is not necessarily affected by the amount of memory available for the VM, so VMs with larger amounts of memory may not exhibit larger bandwidth. For more information on Geekbench 3 and to see its individual workloads, please see http://www.primatelabs.com/geekbench/.
Geekbench 3 Tasks (Figure 3.1)
TEST TOOL TASK Separate CPU tests that are all aggregated into a final
score.
• Subtests include: Integer Math, Floating Point Math
DESCRIPTION
Integer Geekbench 3 AES, Twofish, SHA1, SHA2, BZip2 Compression, BZip2 Decompression, JPEG Compression, JPEG Decompression,
PNG Compression, PNG Decompression, Sobel, Lua, Dijkstra
Integer and Floating Point tasks together represent vCPU performance. The performance of all applications
is highly dependent on the vCPU since the vCPU is responsible for the processing and orchestration of all
applications. Floating Point Geekbench 3 Black Scholes, Mandelbrot, Sharpen Filter, Blur Filter,
SGEMM, DGEMM, SFFT, DFFT, N-Body, Ray Trace
Memory Geekbench 3 STREAM Copy, STREAM Scale, STREAM Add, STREAM
Triad
While memory performance is not considered one of
the key bottlenecks in performance for many common applications, a subset of applications—particularly HPC
and in-memory databases—is highly dependent on
large sustained memory bandwidth.
Sequential and Random Disk IOPS: fio
Fio is an open source I/O generator that spawns a number of threads and processes to conduct a particular type of I/O action specified. For the purpose of this study, fio was used to measure disk IOPS by tracking direct I/O to the VM’s network storage. 5 x 200mb files were created for
sequential operations testing, and 5 x 200 mb files were created for random operations testing. All operations were 50% read and 50% write. Each
test iteration used an 8kb block size. Each test iteration lasted 60 seconds.
Internal Network Throughput: Iperf
Iperf is an open source tool used to measure TCP or UDP network bandwidth performance. In this study, Cloud Spectator used Iperf to measure the network throughput between VMs residing in the same region/availability zone. A screen session was created for Iperf as a server machine on each
appropriate VM (see Process 6). Each Iperf test iteration lasted 60 seconds, and data was transferred in one direction, from the test VM to the Iperf
server VM.
Cloud Spectator Performance Report
8
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
VM Configurations & Pricing
VM configurations were matched to standardize by virtual processors. 2 vCPUs, 4 vCPUs, and 8 vCPUs machines from each provider were used in
this study that most closely matched in size. While some providers may offer the option for local storage, none was used and all disk testing was
conducted on persistent SAN storage. The storage columns in Figures 3.3 A through C are reflective of the SAN storage provisioned for each VM. Azure uses Blob Storage, which automatically provides the user with as-needed storage. Thus, only the space on the volume that is needed
(depending on the OS and installed applications) is given to the user. With Blob Storage, users cannot deploy volumes with a pre-defined amount of
storage, although the user can specify how large he or she expects the blob to grow. For more information, see this article: https://msdn.microsoft.com/en-us/library/azure/ee691964.aspx.
2 vCPU Virtual Machines (Figure 3.3 A)
Provider Instance vCPU RAM (GB) Storage (GB) Monthly (€)
1&1 4GB 2 4 100 €29.99
1&1 8GB 2 8 100 €79.20
AWS C4.large 2 3.75 100 (EBS Optimized) € 95.04
AWS M3.large 2 7.5 100 (EBS Optimized) € 108.72
Aruba Cloud 4GB 2 4 100 € 64.80
Aruba Cloud 8GB 2 8 100 € 79.20
Azure A2 2 3.5 Blob Storage € 68.10
Azure D2 2 7 Blob Storage € 104.60
CloudSigma 4GB 2 4 100 € 49.33
CloudSigma 8GB 2 8 100 € 73.73
Cloud Spectator Performance Report
9
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
4 vCPU Virtual Machines (Figure 3.3 B)
Provider Instance vCPU RAM (GB) Storage (GB) Monthly (€)
1&1 8GB 4 8 400 €93.60
1&1 15GB 4 15 400 €144.00
AWS C4.xlarge 4 7.5 400 (EBS Optimized) € 208.08
AWS M3.xlarge 4 15 400 (EBS Optimized) € 236.16
Aruba Cloud 8GB 4 8 400 € 158.40
Aruba Cloud 15GB 4 15 400 € 183.60
Azure A3 4 7 Blob Storage € 143.65
Azure D3 4 14 Blob Storage € 216.59
CloudSigma 8GB 4 8 400 € 135.27
CloudSigma 15GB 4 15 400 € 177.97
8 vCPU Virtual Machines (Figure 3.3 C)
Provider Instance vCPU RAM (GB) Storage (GB) Monthly (€)
1&1 15GB 8 15 800 (2 x 400) € 172.80
1&1 30GB 8 30 800 (2 x 400) € 280.80
AWS C4.2xlarge 8 15 800 (EBS Optimized) € 416.88
AWS M3.2xlarge 8 30 800 (EBS Optimized) € 473.76
Aruba Cloud 15GB 8 15 800 € 313.20
Aruba Cloud 30GB 8 30 800 € 367.20
Azure A4 8 14 Blob Storage € 287.24
Azure D4 8 28 Blob Storage € 433.11
CloudSigma 15GB 8 15 800 € 276.63
CloudSigma 30GB 8 30 800 € 368.15 Conversion Rates: £1.00 = €1.40 $1.00 = €0.89
Cloud Spectator Performance Report
10
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Understanding Performance Results
The virtual machines’ performance information was depicted using the minimum, 5th percentile, median, 95th percentile, and maximum scores
retrieved from all data points collected for each of the tests mentioned above during the testing period. The information was integrated into percentile
graphs and value tables designed to visualize performance variation captured while testing over time. An example of a performance percentile graph is displayed below:
Understanding The CloudSpecs Score (Price-Performance)
Cloud Spectator uses the CloudSpecs ScoreTM as an indication of price-performance value for each resource of the VM, separated by 1) processor & memory bandwidth, 2) disk IOPS, and 3) internal network throughput. By definition, the CloudSpecs ScoreTM provides information on how much
performance the user receives for each unit of cost. The CloudSpecs ScoreTM is an indexed, comparable score ranging from 0-100 indicative of
value based on a combination of cost and performance. The calculation of the CloudSpecs ScoreTM is:
price-performance_value = [VM performance score] / [VM cost]
best_VM_value = max{price-performance_values}
CloudSpecs ScoreTM = 100*price-performance_value / best_VM_value
In this report, Cloud Spectator uses the aggregated performance scores as
the [provider performance score] to calculate each machine’s CloudSpecs
ScoreTM.
The graph on the left is an example of how Cloud Spectator’s price-
performance analysis is visualized. The closer the score is to 100, the
higher price-performance value it indicates. The score 100 represents the
best-value VM among all in the comparison. The value is scaled; e.g., the
100
44 41 41 25
0
20
40
60
80
100
CSP1 CSP2 CSP3 CSP4 CSP5
Clou
dSpe
cs S
core
TM
Cloud Service Provider (CSP)
Legend
Maximum: highest score achieved on this VM over the duration of the testing.
95TH Percentile (High-Score Category): 95% of all scores on this VM achieved this score or lower.
Median (Median-Score Category): The number separating the higher half of the scores of that VM from the lower half. If the median is closer to the
95th percentile, then more high scores were observed than low scores; vice versa.
5TH Percentile (Low-Score Category): 5% of all scores on this provider achieved this score or lower.
Minimum: lowest score achieved on this VM over the duration of the testing.
0
500
1000
1500
2000
2500
3000
3500
4000
AWS Azure Google Rackspace Softlayer
Mpixe
ls/se
c
Sample Performance Graph
Cloud Spectator Performance Report
11
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
VM from Cloud Service Provider 1 (CSP1) with a score of 100 gives 4x the value of the VM from CSP5 with a score of 25.
The CloudSpecs ScoresTM of any VM can change depending on the participants in the comparison. For example, if the highest score in a comparison changes, the price-performance value represented by score 100 will change accordingly, and so will the other CloudSpecs ScoreTM
values.
If you have questions regarding Cloud Spectator’s price-performance calculation, please contact us at contact@cloudspectator.com.
Key Considerations
• Pricing used in this study (for price-performance comparisons) is up to date effective June 10, 2015. Pricing may change for the specified
VMs after the release of this report.
• Testing was conducted on specific VM types for each provider. Different VM configurations may yield different comparative results between the providers. AWS and Azure offered fixed VM configurations, while 1&1, Aruba Cloud, and CloudSigma offered independently
customizable resource configurations.
• Users may experience different performance across different physical hosts. Factors such as user contention or malfunctions of the physical hardware can cause suboptimal performance. Cloud Spectator terminated and created new VMs for each test iteration to increase
the likelihood of testing on different physical hosts.
• VMs selected were the base offerings across providers; greater performance may be obtained on certain providers by paying for additional features/services.
Cloud Spectator Performance Report
12
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Detailed Performance Findings
Processor and Memory Bandwidth
Below are the results of processor and memory bandwidth testing on all providers. Because memory bandwidth is not affected by the amount of provisioned RAM, the VMs with larger amounts of RAM do not necessarily have higher Indexed Scores. The tables on the right specify the scores
achieved by each provider’s VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in
each category is highlighted and bolded in green in the corresponding tables.
0 500
1000 1500 2000 2500 3000 3500 4000 4500 5000
4GB 8GB c4.large m3.large 4GB 8GB A2 D2 4GB 8GB
1&1 AWS ArubaCloud Azure CloudSigma
Inde
xed
Scor
e
Processor & Memory PERFORMANCE: 2vCPUs Figure 4.1 A
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
10000
8GB 15GB c4.xlarge m3.xlarge 8GB 15GB A3 D3 8GB 15GB
1&1 AWS ArubaCloud Azure CloudSigma
Inde
xed
Scor
e
Processor & Memory PERFORMANCE: 4vCPUs Figure 4.1 B
0 2000 4000 6000 8000
10000 12000 14000 16000 18000
15GB 30GB c4.2xlarge m3.2xlarge 15GB 30GB A4 D4 15GB 30GB
1&1 AWS ArubaCloud Azure CloudSigma
Inde
xed
Scor
e
Processor & Memory PERFORMANCE: 8vCPUs Figure 4.1 C
Provider VM Min 5TH Median 95TH Max
2 vCP
Us
1&1 4GB 4575 4619 4711 4731 4746 8GB 4059 4663 4708 4732 4748
AWS c4.large 3901 3934 3952 3973 4002 m3.large 3125 3139 3155 3173 3192
Aruba Cloud
4GB 3731 4017 4085 4165 4203 8GB 2973 3883 4084 4167 4197
Azure A2 1781 2132 2166 2185 2304 D2 2176 3557 3621 3693 3746
CloudSigma 4GB 2457 2599 2896 2965 2986 8GB 1150 1374 2826 2981 3000
Provider VM Min 5TH Median 95TH Max
4 vCP
Us
1&1 8GB 7993 8493 8764 9088 9132 15GB 8304 8688 8852 9101 9144
AWS c4.xlarge 6932 7672 7736 7775 7808 m3.xlarge 5699 6105 6261 6286 6308
Aruba Cloud
8GB 6872 7459 7693 7842 7868 15GB 5660 7388 7663 7840 7881
Azure A3 3648 4096 4118 4136 4161 D3 4978 6650 6677 6700 6716
CloudSigma 8GB 1149 1454 3698 5782 5888 15GB 1192 1474 4538 4927 5387
Provider VM Min 5TH Median 95TH Max
8 vCP
Us
1&1 15GB 14218 15648 16138 16281 16349 30GB 14858 15768 16097 16767 16910
AWS c4.2xlarge 12718 13257 13705 14056 14140 m3.2xlarge 10580 11228 11519 11588 11659
Aruba Cloud
15GB 10739 11848 13775 14119 14337 30GB 10765 11692 13874 14206 14400
Azure A4 5541 7418 7839 7917 7963 D4 8687 12646 12739 12793 12866
CloudSigma 15GB 4019 5076 7281 7848 8056 30GB 1336 1550 5900 6803 7640
Cloud Spectator Performance Report
13
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Observations: Processor & Memory Bandwidth Performance
The scores in Figures 4.1 A – C are indexed through a combination of floating point and integer performance on processors and memory bandwidth
on RAM. CloudSigma’s performance varied the most across processors. Its coefficient of variation (CV), which is a percentage expressing the
relationship between the average and standard deviation (useful for determining variability in performance), ranges up to 43.7% (on the 8 vCPU, 30GB RAM VM). By contrast, the CVs on 1&1’s tested VMs ranged up to 2.6% (on the 4 vCPU, 15GB RAM VM).
Additional Observations
• 1&1’s VMs achieved the highest performance across processor and memory performance in the study.
• The Indexed Score, which weighs heavily on processor performance, is similar between VMs of the same vCPU count for 1&1, Aruba
Cloud, and CloudSigma. All three providers offer independently customizable VMs, which are backed with the same hardware. AWS and Azure exhibit differences in performance results. AWS’s C4 Family and M3 Family are provisioned with different processors; Azure’s A
Series and D Series are provisioned with different processors as well.
• Although AWS and Azure VMs’ processor and memory performance scored lower than 1&1 VMs, both providers also achieved similar levels of performance stability in the 72-hour test period, with CVs lower than 3% on all VMs. Aruba Cloud VMs’ CVs ranged up to 9.3%.
• CloudSigma VMs’ large variation in performance, seen in the processor and memory performance results, are present in disk IOPS and
network performance as well.
• CloudSigma and Azure VMs continue to display the lowest processor and memory bandwidth performance in each category (see tables in Figure 4.1 A through C).
Cloud Spectator Performance Report
14
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
SAN Disk IOPS: Sequential Operations Below are the results of disk IOPS testing on all providers, specifically for sequential operations (50% read, 50% write). Disk IOPS was tested with
direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the
scores achieved by each provider’s VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables.
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
10000
4GB 8GB c4.large m3.large 4GB 8GB A2 D2 4GB 8GB
1&1 AWS ArubaCloud Azure CloudSigma
IOPS
Sequential Read/Write- 2vCPU (100GB) Figure 4.2 A
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
10000
8GB 15GB c4.xlarge m3.xlarge 8GB 15GB A3 D3 8GB 15GB
1&1 AWS ArubaCloud Azure CloudSigma
IOPS
Sequential Read/Write- 4vCPU (400GB) Figure 4.2 B
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
10000
15GB 30GB c4.2xlarge m3.2xlarge 15GB 30GB A4 D4 15GB 30GB
1&1 AWS ArubaCloud Azure CloudSigma
IOPS
Sequential Read/Write - 8vCPU (800GB) Figure 4.2 C
Provider VM Min 5TH Median 95TH Max
2 vCP
Us
1&1 4GB 4477 6138 7143 7947 8341 8GB 4622 6082 6996 7972 8358
AWS c4.large 235 299 299 3064 3064 m3.large 227 299 299 3064 3064
Aruba Cloud
4GB 379 713 1493 2494 4060 8GB 267 597 1372 4414 6204
Azure A2 586 1056 1380 1384 1399 D2 480 1220 1382 1412 1436
CloudSigma 4GB 214 349 753 2170 4538 8GB 208 328 749 2011 3621
Provider VM Min 5TH Median 95TH Max
4 vCP
Us
1&1 8GB 3052 5437 7675 9133 9437 15GB 2845 6570 7933 8804 9138
AWS c4.xlarge 2093 2099 3015 3017 3017 m3.xlarge 2452 3013 3015 3015 3015
Aruba Cloud
8GB 241 618 1186 2483 4496 15GB 278 655 1200 2228 3892
Azure A3 580 1069 1380 1384 1421 D3 427 1166 1381 1385 1404
CloudSigma 8GB 219 311 603 1768 4191 15GB 177 306 633 2155 3909
Provider VM Min 5TH Median 95TH Max
8 vCP
Us
1&1 15GB 3198 4821 6755 8703 9280 30GB 3265 5709 7650 9214 9577
AWS c4.2xlarge 2541 3003 3016 3017 3017 m3.2xlarge 2907 3013 3015 3015 3015
Aruba Cloud
15GB 342 617 1221 2352 4345 30GB 367 599 1300 3000 5843
Azure A4 396 620 663 687 712 D4 531 1195 1385 1414 1435
CloudSigma 15GB 202 316 646 1852 4724 30GB 195 306 596 2074 4012
Cloud Spectator Performance Report
15
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
SAN Disk IOPS: Random Operations Below are the results of disk IOPS testing on all providers, specifically for random operations (50% read, 50% write). Disk IOPS was tested with
direct I/O, so results are not reflective of cached performance, which may sustain higher IOPS on each provider. The tables on the right specify the
scores achieved by each provider’s VMs. The lowest score in each category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables.
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
10000
4GB 8GB c4.large m3.large 4GB 8GB A2 D2 4GB 8GB
1&1 AWS ArubaCloud Azure CloudSigma
IOPS
Random Read/Write - 2vCPU (100GB) Figure 4.3 A
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
10000
8GB 15GB c4.xlarge m3.xlarge 8GB 15GB A3 D3 8GB 15GB
1&1 AWS ArubaCloud Azure CloudSigma
IOPS
Random Read/Write - 4vCPU (400GB) Figure 4.3 B
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
10000
15GB 30GB c4.2xlarge m3.2xlarge 15GB 30GB A4 D4 15GB 30GB
1&1 AWS ArubaCloud Azure CloudSigma
IOPS
Random Read/Write - 8vCPU (800GB) Figure 4.3 C
Provider VM Min 5TH Median 95TH Max
2 vCP
Us
1&1 4GB 4710 6002 6620 7346 8234 8GB 4772 5943 6621 7314 7695
AWS c4.large 576 757 760 3064 3065 m3.large 630 819 822 3064 3064
Aruba Cloud
4GB 165 291 612 952 1181 8GB 115 307 552 1064 1709
Azure A2 163 1021 1383 1430 1438 D2 149 1127 1386 1428 1436
CloudSigma 4GB 243 325 661 1741 3224 8GB 210 351 652 1435 2997
Provider VM Min 5TH Median 95TH Max
4 vCP
Us
1&1 8GB 2830 5110 7272 8605 8896 15GB 3868 6232 7505 8359 8667
AWS c4.xlarge 2759 2985 2987 2988 2988 m3.xlarge 2834 2984 2985 2985 2986
Aruba Cloud
8GB 117 296 532 983 1167 15GB 150 305 537 951 1173
Azure A3 167 963 1376 1424 1436 D3 113 1116 1376 1380 1436
CloudSigma 8GB 232 308 561 1674 3049 15GB 115 302 558 1557 2778
Provider VM Min 5TH Median 95TH Max
8 vCP
Us
1&1 15GB 2814 4557 6463 8042 8911 30GB 3087 5490 7364 8751 9109
AWS c4.2xlarge 2816 2975 2987 2988 2988 m3.2xlarge 2797 2984 2985 2985 2986
Aruba Cloud
15GB 0 295 534 992 1629 30GB 155 296 518 919 1175
Azure A4 164 341 369 382 392 D4 83 1085 1385 1401 1432
CloudSigma 15GB 114 314 651 2156 3897 30GB 168 319 611 2151 3474
Cloud Spectator Performance Report
16
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Observations: Sequential and Random Disk IOPS Performance
Providers offer unique approaches to SAN disk, from hardware components (SSD vs. traditional magnetic drives) to performance throttling. Even the
similarities can be quite different; for example, while 1&1 and CloudSigma both offer SSD-backed storage volumes, the performance difference
between the two providers is noticeable (see Figure 4.2-4.3), with 1&1 producing more than 6,500 IOPS, while CloudSigma produces less than 700 in the median range.
Additional Observations
• Providers with SSD offerings exhibit little performance difference between random and sequential IOPS, which is expected for SSDs.
These providers include 1&1, AWS and CloudSigma. One exception is on the small VMs for AWS, which exhibited different IOPS
performance results due to throttling.
• Although 1&1 and CloudSigma both offer SSDs, 1&1’s volumes achieved 4.4 – 4.7x more IOPS than CloudSigma’s SSDs when examining
95TH percentile figures.
• 1&1, Aruba Cloud, and CloudSigma express large performance variability across the 24 hours of testing. Despite the variability, the low points of 1&1’s disk IOPS in both sequential and random operations still surpass other providers for the majority of tests. Aruba Cloud and
CloudSigma’s variability results in some of the lowest performance observed during the study.
• Although AWS offers SSD technology on its block storage offering, EBS, the performance of that offering is more dependent on the size of
the block storage volume provisioned. Although the actual IOPS performance varies between sequential and random operations, the pattern of performance remains similar based on the size of the provisioned storage (in this test scenario, 100GB, 400GB, and 800GB
sizes were provisioned on 2 vCPU, 4 vCPU, and 8 vCPU machines, respectively). For its General Purpose SSD volumes, AWS offers 3
IOPS per GB with burst up to 3000 IOPS. Figures 4.2 A and 4.3 A illustrate AWS’s burst for a 100GB volume. While burst for the 100GB volume never exceeded 3065 IOPS (see Max in the corresponding tables), the non-burst range, expressed by the median, stays at 299
IOPS for sequential operations, but is higher for random operations. The minimum guaranteed IOPS, 300 (100GB * 3 IOPS per GB), is
sustained for the most part, although minimum values showed lower IOPS, with dips to 227 IOPS. At 800 GB, AWS sustains a fairly stable
rate of approximately 2750 – 3000 IOPS. Burst credits are assigned to volumes, and as long as the volume still has credits, it can burst to 3000 IOPS. In the period of the study, for 4 vCPU and 8 vCPU machines on AWS, the 400GB and 800GB volumes sustained enough
credits for a continuous 24-hours of high performance; interestingly, the c4.xlarge VMs dropped in performance for IOPS on sequential
operations in all three 24-hour iterations, which suggests that credits for IOPS are being used up on the c4.xlarge VMs much faster than on others.
• Azure and Aruba Cloud do not offer SSD technology-backed storage. Therefore, both providers offer higher IOPS performance on
sequential operations when compared with random operations, which is common behavior for magnetic drives. Figures 4.2 and 4.3 illustrate this behavior for both providers. In one of the sequences for the 8 vCPU with 15GB RAM VM, Aruba Cloud did not successfully
complete a cycle of random read/write operations, and therefore achieved a minimum of 0 in the total 72-hour study.
• 1&1 achieved much higher IOPS than all other providers tested, with maximums exceeding 9500 IOPS for sequential operations. Even in
the median, 1&1’s SolidFire-backed SSD volumes obtained a minimum of 6463 IOPS, which is 2.15x more IOPS than can be achieved per 800GB volume on AWS, unless a user decides to purchase provisioned IOPS at an additional charge.
• For random IOPS, Azure and Aruba Cloud displayed the lowest results in most scenarios. Azure and Aruba Cloud, as mentioned before,
are the only providers in this study that use magnetic storage for their storage volumes.
Cloud Spectator Performance Report
17
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Internal Network Throughput Below are the results of internal network testing on all providers. Internal network was examined by running TCP connections between two servers
within the same data center/region/zone. The tables on the right specify the scores achieved by each provider’s VMs. The lowest score in each
category (Min, 5th, Median, 95TH, and Max) is highlighted in red. The highest score in each category is highlighted and bolded in green in the corresponding tables.
0 500
1000 1500 2000 2500 3000 3500 4000 4500 5000
4GB 8GB c4.large m3.large 4GB 8GB A2 D2 4GB 8GB
1&1 AWS ArubaCloud Azure CloudSigma
Thro
ughp
ut (M
bit/s
)
Internal Network Throughput PERFORMANCE: 2 vCPUs Figure 4.4 A
0 2000 4000 6000 8000
10000 12000 14000 16000 18000
8GB 15GB c4.xlarge m3.xlarge 8GB 15GB A3 D3 8GB 15GB
1&1 AWS ArubaCloud Azure CloudSigma
Thro
ughp
ut (M
bit/s
)
Internal Network Throughput PERFORMANCE: 4 vCPUs Figure 4.4 B
0
1000
2000
3000
4000
5000
6000
15GB 30GB c4.2xlarge m3.2xlarge 15GB 30GB A4 D4 15GB 30GB
1&1 AWS ArubaCloud Azure CloudSigma
Thro
ughp
ut (M
bit/s
)
Internal Network Throughput PERFORMANCE: 8 vCPUs Figure 4.4 C
Provider VM Min 5TH Median 95TH Max
2 vCP
Us
1&1 4GB 700 929 955 955 956 8GB 774 928 955 955 956
AWS c4.large 473 473 473 492 492 m3.large 583 628 628 663 664
Aruba Cloud
4GB 792 834 875 895 895 8GB 794 838 894 895 895
Azure A2 193 302 708 734 746 D2 640 863 880 895 909
CloudSigma 4GB 914 1466 2760 4359 4832 8GB 968 1979 2900 3660 4361
Provider VM Min 5TH Median 95TH Max
4 vCP
Us
1&1 8GB 619 885 956 956 956 15GB 661 896 956 956 956
AWS c4.xlarge 723 725 725 754 754 m3.xlarge 684 899 906 957 957
Aruba Cloud
8GB 779 826 857 882 895 15GB 297 743 853 881 895
Azure A3 570 863 888 910 930 D3 1451 1566 1624 1683 1735
CloudSigma 8GB 1123 1942 6526 13265 16152 15GB 1423 2517 6970 11396 12821
Provider VM Min 5TH Median 95TH Max
8 vCP
Us
1&1 15GB 696 841 953 956 956 30GB 709 891 956 956 956
AWS c4.2xlarge 902 921 925 962 962 m3.2xlarge 886 903 906 957 957
Aruba Cloud
15GB 724 801 852 895 895 30GB 658 820 861 893 895
Azure A4 123 890 910 925 936 D4 1432 1529 1590 1649 1684
CloudSigma 15GB 1344 2001 3208 4657 5983 30GB 1005 2191 3493 5105 5792
Cloud Spectator Performance Report
18
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Observations: Internal Network Throughput Performance
With regards to internal network throughput, three out of five providers exhibited fairly stable performance of less than 10% CV. Azure’s A2 and A4
VMs expressed higher variability at 33.3% and 12.1%, respectively. CloudSigma’s variability ranged much higher, up to 45.9% on the 4vCPU 8GB
RAM VM. When examining the median value of the same VM size, CloudSigma achieves much higher throughput (up to almost 7 Gbits) than the other VMs. Comparing with the same VM’s minimum value, the almost 7 Gbits drops to 1.5 Gbits. Despite the sharp decrease in performance,
CloudSigma’s unstable performance still achieved higher throughput, for the most part, than all other providers.
Additional Observations
• AWS’s VMs deliver a specific network throughput depending on the size and family of VM. As the size of the VMs increase, the network
throughput increases as well. The difference in network performance between the C4 Family and M3 Family are comparable. By contrast, Azure’s A Series provides less throughput than its D Series.
• Azure’s VMs, which scale depending on size and Series, exceeds 1GB/s network throughput for D3 and D4s (see Figures 4.4B and C). All
other providers, with the exception of CloudSigma, never exceed 1GB/s, although 1&1 VMs and AWS’s larger VMs come close.
• 1&1 provides a continuous and fairly steady throughput of slightly less than 1GB/s (956 GB/s max) regardless of VM size. Similarly, Aruba Cloud provides a lower, continuous throughput at a little less than 900 MB/s, regardless of VM size as well.
• While AWS displayed some of the lowest internal network throughput numbers for 2 vCPU and 4 vCPU VMs, Aruba Cloud, which does not
scale internal network throughput with increased VM size, displayed the lowest throughput numbers for 8 vCPU VMs.
Cloud Spectator Performance Report
19
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Detailed Price-Performance Findings
Processor & Memory
Figures 5.1 A through F illustrate the price-performance results for processor and memory bandwidth performance across all tested providers. Rows
are organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
100
40
26 20
37
0 10 20 30 40 50 60 70 80 90
100
4GB
c4.la
rge
4GB
A2
4GB
1&1 AWS ArubaCloud Azure CloudSigma
Clo
udSp
ecs
Scor
e
Processor & Memory Price-Performance: 2 vCPUs Figure 5.1 A
100
87
49 58
64
0 10 20 30 40 50 60 70 80 90
100
8GB
m3.
larg
e
8GB
D2
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clo
udSp
ecs
Scor
e
Processor and Memory Price-Performance: 2 vCPUs Figure 5.1 B
100
52
40 31 29
0 10 20 30 40 50 60 70 80 90
100
8GB
c4.x
larg
e
8GB
A3
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clo
udSp
ecs
Scor
e
Processor & Memory Price-Performance: 4 vCPUs Figure 5.1 C
100
68
43 50
41
0 10 20 30 40 50 60 70 80 90
100
15G
B
m3.
xlar
ge
15G
B
D3
15G
B
1&1 AWS ArubaCloud Azure CloudSigma
Clo
udSp
ecs
Scor
e
Processor and Memory Price-Performance: 4 vCPUs Figure 5.1 D
100
47
35 29 28
0 10 20 30 40 50 60 70 80 90
100
15G
B
c4.2
xlar
ge
15G
B
A4
15G
B
1&1 AWS ArubaCloud Azure CloudSigma
Clo
udSp
ecs
Scor
e
Processor & Memory Price-Performance: 8 vCPUs Figure 5.1 E
100
66
42 51
28
0 10 20 30 40 50 60 70 80 90
100
30G
B
m3.
2xla
rge
30G
B
D4
30G
B
1&1 AWS ArubaCloud Azure CloudSigma
Clo
udSp
ecs
Scor
e
Processor and Memory Price-Performance: 8 vCPUs Figure 5.1 F
Cloud Spectator Performance Report
20
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Observations: Processor & Memory Price-Performance
In all cases illustrated from Figure 5.1 A through F, 1&1’s VMs achieved the highest CloudSpecs ScoreTM in the test group, indicating the strongest
price-performance value for processor and memory bandwidth. AWS’s m3.large VM comes closest to matching in value with a 1&1 VM counterpart,
at a CloudSpecs ScoreTM of 87 (see Figure 5.1 B). Additional Observations
• For VMs with 2 vCPUs and approximately 8GB RAM (see Figure 5.1 B), all values are most evenly matched, compared with other VM sizes.
• As VM sizes scale up in processors and RAM, CloudSigma’s CloudSpecs ScoreTM drops, and its rank falls in relativity to other tested
providers’ VMs, due largely to performance (see Figure 4.1).
• Relatively, Azure’s D Series, which has an average CloudSpecs ScoreTM of 53 across all VM sizes, provides more relative price-
performance value than its A Series, which has an average CloudSpecs ScoreTM of 27. Because charts cannot be relatively compared,
though, the difference between the CloudSpecs ScoresTM does not equate to an almost 2x value on Azure’s D Series.
Cloud Spectator Performance Report
21
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
SAN Disk IOPS: Sequential Operations Figures 5.2 A through F illustrate the price-performance results for sequential disk IOPS performance across all tested providers. Rows are
organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
100
2 7 8 6
0
10
20
30
40
50
60
70
80
90
100
4GB
c4.large
4GB A2
4GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Sequen9al Read/Write: 2 vCPUs (100GB) Figure 5.2 A
100
4 14 15 11
0 10 20 30 40 50 60 70 80 90 100
8GB
m3.large
8GB
D2
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Sequen9al Read/Write: 2 vCPUs (100GB) Figure 5.2 B
100
23
7 12
5
0 10 20 30 40 50 60 70 80 90 100
8GB
c4.xlarge
8GB A3
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Sequen9al Read/Write: 4 vCPUs (400GB) Figure 5.2 C
100
30
9 12 6
0 10 20 30 40 50 60 70 80 90 100
15GB
m3.xlarge
15GB
D3
15GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Sequen9al Read/Write: 4 vCPUs (400GB) Figure 5.2 D
100
25
8 6 6
0 10 20 30 40 50 60 70 80 90
100
15GB
c4.2xlarge
15GB
A4
15GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Sequen9al Read/Write: 8 vCPUs (800GB) Figure 5.2 E
100
30
10 12 6
0 10 20 30 40 50 60 70 80 90 100
30GB
m3.2xlarge
30GB
D4
30GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Sequen9al Read/Write: 8 vCPUs (800GB) Figure 5.2 F
Cloud Spectator Performance Report
22
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
SAN Disk IOPS: Random Operations Figures 5.3 A through F illustrate the price-performance results for random disk IOPS performance across all tested providers. Rows are organized
by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
100
5 3 9 6
0
10
20
30
40
50
60
70
80
90
100
4GB
c4.large
4GB A2
4GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Random Read/Write: 2 vCPUs (100GB) Figure 5.3 A
100
12 6
16 11
0 10 20 30 40 50 60 70 80 90 100
8GB
m3.large
8GB
D2
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Random Read/Write: 2 vCPUs (100GB) Figure 5.3 B
100
24
3 12
5
0 10 20 30 40 50 60 70 80 90 100
8GB
c4.xlarge
8GB A3
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Random Read/Write: 4 vCPUs (400GB) Figure 5.3 C
100
31
4 12
6
0 10 20 30 40 50 60 70 80 90 100
15GB
m3.xlarge
15GB
D3
15GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Random Read/Write: 4 vCPUs (400GB) Figure 5.3 D
100
25
3 4 6
0 10 20 30 40 50 60 70 80 90
100
15GB
c4.2xlarge
15GB
A4
15GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Random Read/Write: 8 vCPUs (800GB) Figure 5.3 E
100
31
4 12
6
0 10 20 30 40 50 60 70 80 90 100
30GB
m3.2xlarge
30GB
D4
30GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Random Read/Write: 8 vCPUs (800GB) Figure 5.3 F
Cloud Spectator Performance Report
23
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Observations: Sequential and Random Disk IOPS Price-Performance
For sequential and random operations across all VMs, 1&1 offers the highest value due to a combination of high-performance SSDs and low overall
cost of the virtual machine (see Figures 5.2 and 5.3). While CloudSigma also offers SSDs, the disk IOPS achieved on CloudSigma VMs display
much lower performance (see Figures 4.2 and 4.3). AWS, which also offers SSD-based storage (General Purpose) offering was used in this study, throttles performance at a maximum of 3,000 IOPS unless users purchase additional provisioned IOPS; therefore, performance on AWS’s SSDs did
not exceed that of 1&1 either, which had a median range of 6,500-7,500 IOPS.
Additional Observations
• 1&1 achieved the highest CloudSpecs ScoresTM as well as most IOPS for each VM tested.
• Aruba Cloud offered the lowest value for random IOPS on SAN disk on all tested VMs, due to its magnetic drives and cost of the VMs.
• AWS’s c4.large and m4.large (see Figure 5.2 A and B) offered lower price-performance value due to throttled IOPS after burst limits were
exceeded for the 100GB storage volume.
• Although CloudSigma used SSD technology for its storage volume, the price-performance value of those storage volumes tested are similar to Aruba Cloud and Azure volumes, which use magnetic disks.
Cloud Spectator Performance Report
24
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Internal Network Throughput Figures 5.4 A through F illustrate the price-performance results for internal network throughput performance across all tested providers. Rows are
organized by vCPU count from lowest to highest. Charts on the left columns compare VMs with less RAM.
Observations: Internal Network Throughput Price-Performance
CloudSigma VMs deliver the highest available throughput out of all providers tested. Despite having higher prices than 1&1, the high network throughput performance resulted in a better CloudSpecs ScoreTM for CloudSigma. Similar with Processor & Memory price-performance,
57
13 16 19
100
0
10
20
30
40
50
60
70
80
90
100
4GB
c4.large
4GB A2
4GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Internal Network Price-‐Performance: 2 vCPUs Figure 5.4 A
31
20 21 21
100
0 10 20 30 40 50 60 70 80 90 100
8GB
m3.large
8GB
D2
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Internal Network Price-‐Performance: 2 vCPUs Figure 5.4 B
21
9 9 13
100
0 10 20 30 40 50 60 70 80 90 100
8GB
c4.xlarge
8GB A3
8GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Internal Network Price-‐Performance: 4 vCPUs Figure 5.4 C
17 13 9
19
100
0 10 20 30 40 50 60 70 80 90 100
15GB
m3.xlarge
15GB
D3
15GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Internal Network Price-‐Performance: 4 vCPUs Figure 5.4 D
48
25 18
27
100
0 10 20 30 40 50 60 70 80 90
100
15GB
c4.2xlarge
15GB
A4
15GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Internal Network Price-‐Performance: 8 vCPUs Figure 5.4 E
36 26
19
39
100
0 10 20 30 40 50 60 70 80 90 100
30GB
m3.2xlarge
30GB
D4
30GB
1&1 AWS ArubaCloud Azure CloudSigma
Clou
dSpe
cs Score
Internal Network Price-‐Performance: 8 vCPUs Figure 5.4 F
Cloud Spectator Performance Report
25
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Azure also achieved a better CloudSpecs ScoreTM for its D Series in regards to internal network throughput. The D series offered approximately 1.7x
more network throughput on VMs tested in the study.
Although AWS’s virtual machines increased throughput based on size and family, that increase did not surpass the price-performance ratio of 1&1 VMs’ throughput and cost; therefore, 1&1 achieved a higher CloudSpecs ScoreTM on network throughput than AWS.
Aruba Cloud, which offered higher throughput than AWS on smaller VMs (see Figure 4.4 A), displayed higher internal network throughput
price-performance than AWS until VM sizes scale up. As machine sizes increased, AWS’s throughput scaled while Aruba Cloud’s did not, which is
the reason AWS has higher CloudSpecs ScoresTM for the larger VMs. For these VMs, (see Figure 5.4 B and C), Aruba Cloud fell short of AWS, as well as all other competitors in the study.
Additional Observations
• Azure’s D3 and D4 VMs exceeded a throughput of 1 GB/s (see Figures 4.4 B and C), giving it better price-performance value in internal
network than all other providers for those VM sizes with the exception of CloudSigma (see Figures 5.4 D and F).
• In all other VM sets, 1&1 achieves the highest CloudSpecs ScoreTM with the exception of CloudSigma for internal network.
• Aruba Cloud, which does not scale internal network throughput with the size of the VM (see Figure 4.4), drops in value ranking as VMs are
scaled up in size; other providers, such as AWS and Azure, which scale internal network throughput with the corresponding VM size and
family/series, surpass Aruba Cloud in value for internal network as the VMs are scaled up.
Cloud Spectator Performance Report
26
Copyright 2015 Cloud Spectator, Inc. | All Rights Reserved. For non-commercial use only; do not distribute without permission from Cloud Spectator.
Conclusion
Selecting the right provider and virtual machines requires a thorough and accurate performance comparison. The study conducted for this
report offers a general understanding of performance and price-performance strengths and weaknesses across each included vendor’s VMs, and
should be considered a source of information to help guide readers in their own testing and analyses. The processor, memory bandwidth, network storage, and internal network were all examined and results clearly show that no single provider can offer everything to fit everyone’s needs.
Therefore, in order to accurately select the right provider for a business or an application, performance and price-performance analysis is crucial.
Results from this study show that, 1&1’s VMs displayed strong performance and price-performance. 1&1’s VMs demonstrated high performance and stability for processing and memory bandwidth both on the same VM, and also across VMs of the same size. Its network-attached
storage produced the largest amount of IOPS seen in the study as well, regardless of VM size and without the need for purchasing additional
volumes, sizing up, or provisioning IOPS. These high-performance results, combined with the low cost of the VMs, reflect on the value that 1&1’s
VMs can deliver to potential users, as seen with the price-performance results. 1&1’s performance and price-performance offers an excellent alternative for high-performance environments such as distributed file systems and data analytics processing.
Performance in the industry cannot be assumed to be equal or even similar, as illustrated in this report. When it comes to processor and
memory bandwidth performance, tiered providers such as AWS and Azure may offer varying performance depending on the family/series of the VM, despite having equivalent amounts of vCPUs and similar amounts of memory. For disk IOPS, although both 1&1 and CloudSigma advertise SSD
volumes, 1&1’s SSDs achieved 4.4 – 4.7x more IOPS than their CloudSigma counterparts. Internal network performance on CloudSigma, though,
exceeds all providers examined in the study.
While this study was conducted in the manner of understanding a typical end user experience, it should not be assumed to be accurate for all use cases. Stress testing was conducted to better understand fluctuation and theoretically sustained performance, and should be seen as a
general indication of provider performance. For more detailed analysis on any specific use case, please contact Cloud Spectator at
contact@cloudspectator.com or by phone at +1 (617) 300 0711.
About
About Cloud Spectator
Cloud Spectator is a cloud analyst agency focused on cloud Infrastructure-as-a-Service (IaaS) performance. The company actively monitors several
of the largest IaaS providers in the world, comparing VM performance (i.e., CPU, RAM, disk, internal network, and workloads) and pricing to achieve transparency in the cloud market. The company helps cloud providers understand their market position and helps business make intelligent
decisions in selecting cloud providers and lowering total cost of ownership. The firm was founded in early 2011 and is located in Boston, MA.
For questions about this report, to request a custom report, or if you have general inquiries about our products and services, please contact Cloud
Spectator at +1 (617) 300-0711 or contact@cloudspectator.com.
For press/media related inquiries, please contact: Ken Balazs
VP Sales & Marketing
kbalazs@cloudspectator.com
top related