Alexa and Fortune 500 Compute Marketshare - 2 Sep 2014

This post provides marketshare analysis for compute services (virtual machines) and top Alexa and US Fortune 500 websites. We compile this marketshare using a variety of techniques including ASN and IP/CIDR matching, multi-level site crawling, and related domain associations. We attempt to filter out known white labeled SaaS sites using a filter list. The network identifiers we use for marketshare correlations (ASN, and IP/CIDR) are available via our Get Network Identifiers API method.

This is the first such marketshare post and our techniques are still evolving. We do not guarantee this analysis to be 100% accurate. Rackspace and SoftLayer marketshare includes both cloud and dedicated servers.

Alexa Top 1,000 Compute Marketshare - 1 Sep 2014

Historical Alexa 1,000 Compute Marketshare - 31 Jul 14 to 1 Sep 2014

Alexa Top 1,000 Compute Marketshare - 1 Sep 2014

This table provides marketshare for Compute services amongst top 1,000 Alexa websites. Change metrics are derived from the difference in marketshare from Jul 2014 to Sep 2014.
ProviderRankWebsites (out of 1000)Marketshare
Amazon EC2114414.4%
Rackspace Cloud Servers2525.2%
SoftLayer Cloud Servers3373.7%
Savvis VPDC4171.7%
Linode Cloud Hosting5111.1%
Storm on Demand690.9%
Microsoft Azure Virtual Machines750.5%
DigitalOcean840.4%
Aruba Cloud Compute910.1%
Terremark vCloud Express1010.1%

Alexa Top 1,000 Compute Switchers - 1 Sep 2014

We did not detect any changes for Alexa 1,000 websites in this time period.

Alexa Top 10,000 Compute Marketshare - 1 Sep 2014

Historical Alexa 10,000 Compute Marketshare - 31 Jul 14 to 1 Sep 2014

Alexa Top 10,000 Compute Marketshare - 1 Sep 2014

This table provides marketshare for Compute services amongst top 10,000 Alexa websites. Change metrics are derived from the difference in marketshare from 31 Jul 2014 to 1 Sep 2014.
ProviderRankWebsites (out of 10000)Marketshare
Amazon EC21125312.53%
Rackspace Cloud Servers25035.03%
SoftLayer Cloud Servers34544.54%
Linode Cloud Hosting41591.59%
Savvis VPDC51231.23%
Storm on Demand6890.89%
Microsoft Azure Virtual Machines7630.63%
DigitalOcean8570.57%
Aruba Cloud Compute9130.13%
Google Compute Engine10130.13%
Blue Box VPS11120.12%
Terremark vCloud Express12120.12%
Webair Cloud13100.1%
Joyent Cloud14100.1%
FireHost15100.1%
Gandi Cloud VPS1680.08%
VPS.NET1760.06%
GMO Cloud - US1840.04%
vnCloud1930.03%
Colosseum Cloud2020.02%
CenturyLink Cloud Servers2120.02%
StratoGen VMware Cloud2220.02%
City Cloud2320.02%
GoGrid2410.01%
Crucial Cloud2510.01%
AgileCLOUD2610.01%
Exoscale Compute2710.01%
CloudSigma2810.01%
Logicworks infiniCloud2910.01%

Alexa Top 10,000 Compute Switchers - 1 Sep 2014

This table lists Alexa 10,000 websites that have switched Compute services since prior marketshare analysis on 31 Jul 2014.
HostnameRankNew Compute ProviderPrevious Compute Provider
howtogeek.com1926Linode Cloud HostingSoftLayer Cloud Servers
indeed.co.in2243SoftLayer Cloud ServersRackspace Cloud Servers
indeed.co.uk2562SoftLayer Cloud ServersRackspace Cloud Servers
empireavenue.com3104Google Compute EngineRackspace Cloud Servers
jobdiagnosis.com3151Joyent CloudLinode Cloud Hosting
indeed.fr4164SoftLayer Cloud ServersRackspace Cloud Servers
tv2media.dk5444DigitalOceanRackspace Cloud Servers
corporate.ancestry.com5677Amazon EC2Rackspace Cloud Servers
protradingindicators.com7076DigitalOceanAmazon EC2
twitlonger.com7081Linode Cloud HostingRackspace Cloud Servers
mashpedia.com7822SoftLayer Cloud ServersAmazon EC2
fish4jobs.co.uk8958Rackspace Cloud ServersAmazon EC2
linio.com9774Amazon EC2Rackspace Cloud Servers

Fortune 500 Compute Marketshare - 1 Sep 2014

Historical Fortune 500 Compute Marketshare - 31 Jul 14 to 1 Sep 2014

Fortune 500 2013 Compute Marketshare - 1 Sep 2014

This table provides marketshare for Compute services amongst 2014 US Fortune 500 companies. Change metrics are derived from the difference in marketshare from 31 Jul 14 to 1 Sep 2014.
ProviderRankWebsites (out of 500)Marketshare
Rackspace Cloud Servers110020%
Amazon EC22489.6%
Savvis VPDC3295.8%
Microsoft Azure Virtual Machines4102%
SoftLayer Cloud Servers591.8%
Storm on Demand661.2%
Linode Cloud Hosting751%
FireHost840.8%
GoGrid920.4%
Google Compute Engine1010.2%
Joyent Cloud1110.2%
Blue Box VPS1210.2%
City Cloud1310.2%
Logicworks infiniCloud1410.2%
Crucial Cloud1510.2%

Fortune 500 Compute Switchers - 1 Sep 2014

We did not detect any changes for Fortune 500 websites in this time period.

TAGS:
IaaS Marketshare;
Alexa 10,000;
Alexa 1,000;
Fortune 500
Read More

Alexa and Fortune 500 CDN Marketshare - 2 Sep 2014

This post provides marketshare analysis for CDN services and top Alexa and US Fortune 500 websites. We compile this marketshare using a variety of techniques including CNAME chain analysis, ASN and IP/CIDR matching, multi-level site crawling, and related domain associations. We attempt to filter out known white labeled SaaS sites using a filter list. The network identifiers we use for marketshare correlations (CNAME regex, ASN, and IP/CIDR) are available via our Get Network Identifiers API method.

Alexa Top 1,000 CDN Marketshare - 1 Sep 2014

Historical Alexa 1,000 CDN Marketshare - 31 Jul 14 to 1 Sep 2014

Alexa Top 1,000 CDN Marketshare - 1 Sep 2014

This table provides marketshare for CDN services amongst top 1,000 Alexa websites. Change metrics are derived from the difference in marketshare from Jul 2014 to Sep 2014.
ProviderRankWebsites (out of 1000)MarketshareMarketshare Change
Akamai CDN123023%-2
CloudFlare2585.8%-2
EdgeCast CDN3282.8%0
Amazon CloudFront4242.4%-1
Fastly CDN5242.4%+1
ChinaCache CDN6151.5%+2
CDNetworks CDN7111.1%0
MaxCDN8101%0
Level 3 CDN990.9%0
Incapsula1070.7%-2
Limelight CDN1170.7%-2
Cedexis1250.5%-1
SoftLayer CDN1310.1%0
CacheFly CDN1410.1%-1

Alexa Top 1,000 CDN Switchers - 1 Sep 2014

This table lists Alexa 1,000 websites that have switched CDN services since the prior marketshare analysis in Jul 2014.
HostnameRankNew CDN ProviderPrevious CDN Provider
abs.twimg.com55EdgeCast CDNFastly CDN

Alexa Top 10,000 CDN Marketshare - 1 Sep 2014

Historical Alexa 10,000 CDN Marketshare - 31 Jul 14 to 1 Sep 2014

Alexa Top 10,000 CDN Marketshare - 1 Sep 2014

This table provides marketshare for CDN services amongst top 10,000 Alexa websites. Change metrics are derived from the difference in marketshare from 31 Jul 2014 to 1 Sep 2014.
ProviderRankWebsites (out of 10000)MarketshareMarketshare Change
Akamai CDN1158515.85%-26
CloudFlare27927.92%+21
EdgeCast CDN32012.01%-1
Amazon CloudFront41271.27%-9
Fastly CDN51011.01%+6
CDNetworks CDN6750.75%0
Level 3 CDN7710.71%-4
Incapsula8710.71%-13
MaxCDN9610.61%-9
Limelight CDN10560.56%-12
ChinaCache CDN11390.39%-4
Cedexis12270.27%-3
Rackspace Cloud CDN13210.21%+1
Internap CDN14160.16%+2
CacheFly CDN15120.12%-2
SoftLayer CDN1640.04%0
CDN771730.03%0
Turbobytes1820.02%-2
KeyCDN1910.01%
CDN.net2010.01%0

Alexa Top 10,000 CDN Switchers - 1 Sep 2014

This table lists Alexa 10,000 websites that have switched CDN services since prior marketshare analysis on 31 Jul 2014.
HostnameRankNew CDN ProviderPrevious CDN Provider
abs.twimg.com55EdgeCast CDNFastly CDN
www.nhk.or.jp1196Akamai CDNLimelight CDN
www.leparisien.fr1484EdgeCast CDNLevel 3 CDN
www.elleuk.com2551Fastly CDNCDNetworks CDN
www.euronews.com2615EdgeCast CDNCDNetworks CDN
www.matchesfashion.com4959Fastly CDNAkamai CDN
www.channelnewsasia.com6356CedexisAkamai CDN
bostonherald.com8721Fastly CDNCloudFlare
www.ibis.com9210EdgeCast CDN

Fortune 500 CDN Marketshare - 1 Sep 2014

Historical Fortune 500 CDN Marketshare - 31 Jul 14 to 1 Sep 2014

Fortune 500 2013 CDN Marketshare - 1 Sep 2014

This table provides marketshare for CDN services amongst 2014 US Fortune 500 companies. Change metrics are derived from the difference in marketshare from 31 Jul 14 to 1 Sep 2014.
ProviderRankWebsites (out of 500)MarketshareMarketshare Change
Akamai CDN116332.6%-6
Limelight CDN281.6%+1
Amazon CloudFront361.2%0
Level 3 CDN440.8%0
Incapsula530.6%0
CDNetworks CDN630.6%0
EdgeCast CDN730.6%0
Internap CDN820.4%0
CloudFlare920.4%0
CacheFly CDN1010.2%0
Cedexis1110.2%0
MaxCDN1210.2%0
Rackspace Cloud CDN1310.2%0

Fortune 500 CDN Switchers - 1 Sep 2014

There were no changes in CDN providers amongst Fortune 500 websites within this time period.

TAGS:
CDN Marketshare;
Alexa 10,000;
Alexa 1,000;
Fortune 500
Read More

Alexa and Fortune 500 DNS Marketshare - 1 Sep 2014

This post provides current marketshare statistics for managed DNS services amongst top Alexa websites and US Fortune 500 companies. The methodology used to collect this data is explained in a prior post.

Alexa Top 1,000 DNS Marketshare - 1 Sep 2014

Historical Alexa 1,000 DNS Marketshare - 4 Jul 14 to 1 Sep 2014

Alexa Top 1,000 DNS Marketshare - 1 Sep 2014

This table provides marketshare for managed DNS services amongst top 1,000 Alexa websites. Change metrics are derived from the difference in marketshare from Jul 2014 to Sep 2014.
ProviderRankWebsites (out of 1000)MarketshareMarketshare Change
Dyn DNS113213.2%+7
Amazon Route 53212412.4%+17
UltraDNS3787.8%-2
Akamai DNS4777.7%+6
CloudFlare DNS5747.4%+2
DNS Made Easy6333.3%+3
GoDaddy DNS7323.2%+3
Versign DNS8303%+7
Rackspace Cloud DNS9181.8%+1
DNSPod10171.7%+4
Internap DNS1160.6%-1
Easy DNS1240.4%0
SoftLayer DNS1340.4%+2
EdgeCast DNS1420.2%-2
CDNetworks DNS1510.1%0
NSONE DNS1610.1%0

Alexa Top 1,000 DNS Switchers - 1 Sep 2014

This table lists Alexa 1,000 websites that have switched DNS services since the prior marketshare analysis in Jul 2014.
HostnameRankNew DNS ProviderPrevious DNS Provider
reddit.com50CloudFlare DNSAkamai DNS
odesk.com311CloudFlare DNSAmazon Route 53
eventbrite.com671Dyn DNSRackspace Cloud DNS
pixlr.com754Amazon Route 53Linode DNS
smh.com.au804Amazon Route 53Akamai DNS
ad4game.com950Akamai DNSDyn DNS
zulily.com957Akamai DNSDyn DNS

Alexa Top 10,000 DNS Marketshare - 1 Sep 2014

Historical Alexa 10,000 DNS Marketshare - 4 Jul 14 to 1 Sep 2014

Alexa Top 10,000 DNS Marketshare - 1 Sep 2014

This table provides marketshare for managed DNS services amongst top 10,000 Alexa websites. Change metrics are derived from the difference in marketshare from 4 Jul 2014 to 1 Sep 2014.
ProviderRankWebsites (out of 10000)MarketshareMarketshare Change
CloudFlare DNS1107310.73%+74
Amazon Route 532106110.61%+106
Dyn DNS36906.9%+50
GoDaddy DNS45275.27%+59
UltraDNS54774.77%+12
Akamai DNS64214.21%+26
DNS Made Easy73593.59%+20
Rackspace Cloud DNS82032.03%+18
Versign DNS91941.94%+11
DNSPod101551.55%+17
Easy DNS11820.82%+5
SoftLayer DNS12800.8%+5
Linode DNS13430.43%+10
Internap DNS14290.29%+1
CDNetworks DNS15160.16%+4
EdgeCast DNS16100.1%-13
NSONE DNS1740.04%0
Google Cloud DNS1830.03%
Azure DNS1920.02%0
Rage4 Networks DNS2010.01%0

Alexa Top 10,000 DNS Switchers - 1 Sep 2014

This table lists Alexa 10,000 websites that have switched DNS services since prior marketshare analysis on 4 Jul 2014.
HostnameRankNew DNS ProviderPrevious DNS Provider
reddit.com50CloudFlare DNSAkamai DNS
odesk.com311CloudFlare DNSAmazon Route 53
eventbrite.com671Dyn DNSRackspace Cloud DNS
pixlr.com754Amazon Route 53Linode DNS
smh.com.au804Amazon Route 53Akamai DNS
ad4game.com950Akamai DNSDyn DNS
zulily.com957Akamai DNSDyn DNS
wayfair.com1347CloudFlare DNSDyn DNS
tomoson.com1716Amazon Route 53GoDaddy DNS
theage.com.au2498Amazon Route 53Akamai DNS
theepochtimes.com3215Google Cloud DNSAmazon Route 53
1mpics.com3432CloudFlare DNSGoDaddy DNS
800notes.com3951CloudFlare DNSGoDaddy DNS
domain.com.au4108Amazon Route 53Akamai DNS
buzzsumo.com4120CloudFlare DNSAmazon Route 53
economicos.cl5131CloudFlare DNSRackspace Cloud DNS
www.erepublik.net5749CloudFlare DNSAmazon Route 53
erepublik.com5749CloudFlare DNSAmazon Route 53
sheinside.com5850Dyn DNSGoDaddy DNS
parimatchru.com5879CloudFlare DNSGoDaddy DNS
eventbrite.co.uk6094Dyn DNSRackspace Cloud DNS
marunadanmalayali.com6235CloudFlare DNSGoDaddy DNS
bcash.com.br6552Amazon Route 53UltraDNS
classiccars.com6601Dyn DNSGoDaddy DNS
kyofun.com6786DNSPodCloudFlare DNS
t24.com.tr6849CloudFlare DNSAmazon Route 53
thescore.com7352Amazon Route 53Dyn DNS
somuch.com7681CloudFlare DNSDNS Made Easy
gumroad.com7930CloudFlare DNSAmazon Route 53
andhrajyothy.com8251DNS Made EasyAmazon Route 53
thepioneerwoman.com8287CloudFlare DNSDyn DNS
sugarcrm.com8298Amazon Route 53Dyn DNS
www.myspicetrip.com8445Amazon Route 53GoDaddy DNS
alaan.tv8457Amazon Route 53Rackspace Cloud DNS
nerdwallet.com8459Amazon Route 53Rackspace Cloud DNS
ads.ad4game.com8527Akamai DNSDyn DNS
thebump.com8623Dyn DNSCloudFlare DNS
sporcle.com8738Amazon Route 53GoDaddy DNS
sharecare.to9514Amazon Route 53Versign DNS
mb-view.com9895GoDaddy DNSCloudFlare DNS

Fortune 500 DNS Marketshare - 1 Sep 2014

Historical Fortune 500 DNS Marketshare - 4 Jul 14 to 1 Sep 2014

Fortune 500 2013 DNS Marketshare - 1 Sep 2014

This table provides marketshare for managed DNS services amongst 2014 US Fortune 500 companies. Change metrics are derived from the difference in marketshare from 4 Jul 14 to 1 Sep 2014.
ProviderRankWebsites (out of 500)MarketshareMarketshare Change
Versign DNS15711.4%+10
UltraDNS25410.8%+3
Akamai DNS3346.8%+1
GoDaddy DNS4306%+9
Dyn DNS5224.4%+3
Amazon Route 536193.8%+5
Rackspace Cloud DNS7193.8%+4
DNS Made Easy8183.6%+2
CloudFlare DNS981.6%0
Internap DNS1061.2%0
Easy DNS1151%0
Linode DNS1220.4%+1
DNSPod1320.4%0
SoftLayer DNS1420.4%0
EdgeCast DNS1510.2%0

Fortune 500 DNS Switchers - 1 Sep 2014

There were no changes in managed DNS providers amongst Fortune 500 websites within this time period.

TAGS:
DNS Marketshare;
Alexa 10,000;
Alexa 1,000;
Fortune 500;
Dyn;
AWS Route 53;
UltraDNS;
DNSPod
Read More

Comparing Cloud Compute Services

Over the years we've spent a good amount of time testing and thinking about how to compare cloud services. Some services, like content delivery networks (CDN), managed DNS and object storage are relatively easy because they have few deployment options and similar features between providers.

Comparing cloud compute or servers is a different story entirely. Because of the diverse deployment options and dissimilar features of different services, formulating relevant and fair comparisons is challenging to say the least. In fact, we've come to the conclusion that there is no perfect way to do it. This isn't to say that you can't - but if you do, or if you are handed a third party comparison to look over, there are some things you should keep in mind - and watch out for (we've seen some poorly constructed comparisons).

The purpose of this post is to highlight some of these considerations. To do so, I'll present actual comparisons from testing we've done recently on Amazon EC2, DigitalOcean, Google Cloud Platform, Microsoft Azure and SoftLayer. I won't declare any grand triumphs of one service over another (this is something we try to avoid anyway). Instead, I'll just present the numbers as we observed them with some cursory commentary, and let you draw your own conclusions. I'll also discuss value and some non-performance related factors you might want to consider.

This post is essentially a summary of a report we've published in tandem. The report covers the same topics, but in more detail. You can download the report for free.

Apples and Oranges

Before you start running tests, you first need to pick some compute instances from different services you are considering. Ideally your choices will be based on your actual requirements and workloads.

Some comparisons I've seen get off to a bad start here - picking instances that are essentially apples and oranges. For example, I still see studies that compare older Amazon EC2 m1 instances (which have been replaced with newer classes) to the latest and greatest of other services. I've also seen comparisons where instances have dissimilar CPU cores (CPU benchmark metrics are often proportional to cores). Watch out for these types of studies, because often the conclusions they come to are inaccurate and irrelevant.

Test Workloads

Before comparing compute services, you should have an idea about the type of workloads you'd like to compare. Different workloads have different performance characteristics. For our comparisons, we based testing on two workloads - web and database servers.

For the web server workload, our focus was CPU, disk read and external network performance. Because web servers usually don't store mission critical data, we picked instances with generally faster local drives (SSD where possible). We looked for instances with a ratio of 1-2X CPU cores-to-memory (e.g. 1 core/2GB memory) - sufficient for many web server workloads.

For the database server workload, our primary focus was CPU, disk read and write, memory and internal network performance. Because database servers are typically a critical component in an application stack, we chose off instance storage instead of local because of its better resilience. If a host system that fails, off instance storage volumes can usually be quickly restored on a new compute instance. We looked for instances with a ratio of 2-4X CPU cores-to-memory for this workload.

We picked three instance sizes for each workload - small, medium and large. In order to provide an apples-to-apples comparison, we chose from current instance classes with each service and matched the number of CPU cores precisely.

Our Comparisons

Based on the criteria above - the tables below show the instances we picked for each service and workload. More details on our reasoning for these selections are covered in the report.

CPU cores-to-memory ratios are often where it is nearly impossible to match services exactly. Our primary consideration was to match CPU cores - since this is what affects CPU benchmark metrics the most.

Web Server Comparisons

On July 1 2014, Amazon announced T2 instances with burstable CPU capacity. This instance class offers 1 to 2 CPU cores and provides bursting using on a predictable, credit based method. CPU bursting is nothing new, but the T2 implementation with a predictable, credit based burst model is, and offers good value for workloads that fall within its 10-20% bursting allowance. Because workloads with temporary bursting are common, we have included the t2.medium instance in the small web server CPU performance and value analysis (in addition to the c3.large instance type).

Compute Service Small Web Server Medium Web Server Large Web Server
Amazon EC2 c3.large + t2.medium c3.xlarge c3.2xlarge
DigitalOcean 4 GB / 2 Cores 8 GB / 4 Cores 16 GB / 8 Cores
Google Compute Engine n1-highcpu-2 n1-highcpu-4 n1-highcpu-8
Microsoft Azure Medium (A2) Large (A3) Extra Large (A4)
Rackspace Performance 1 2GB Performance 1 4GB Performance 1 8GB
SoftLayer 2 GB / 2 Cores 4 GB / 4 Cores 8 GB / 8 Cores

Database Server Comparisons

Compute Service Small Web Server Medium Web Server Large Web Server
Amazon EC2 c3.xlarge c3.2xlarge c3.4xlarge
DigitalOcean 8 GB / 4 Cores 16 GB / 8 Cores 48 GB / 16 Cores
Google Compute Engine n1-standard-4 n1-standard-8 n1-standard-16
Microsoft Azure Large (A3) Extra Large (A4) A91
Rackspace Performance 2 15GB Performance 2 30GB Performance 2 60GB
SoftLayer 8 GB / 4 Cores 16 GB / 8 Cores 32 GB / 16 Cores

1 The A9 instance is part of Azure's new compute intensive instance class. It is based on new hardware and a higher CPU cores-to-memory ratio (7X) than the other Azure instances included in the comparisons.

Benchmark Relevance

Once you've picked services and compute instances to compare, and workload to focus on - the next step is to choose benchmarks that are relevant to those workloads. This is another area where I've seen some bad comparisons - often arriving at broad conclusions based on simplistic or irrelevant benchmarks.

The benchmarks we chose are SPEC CPU 2006 for CPU performance, fio for disk IO, STREAM for memory, and our own benchmarks (based on standard Linux tools) for internal and external network testing. More details about these benchmarks, and the runtime configurations we used are provided in the report.

Performance Comparisons

The graphs below show some of the results from our testing. Complete results are available in the full report.

CPU Performance

With the exception of Microsoft Azure (and occasionally SoftLayer), all of the services we tested use current generation Sandy or Ivy bridge processors resulting in similar CPU metrics.

SPEC CPU 2006 is an industry standard benchmark for measuring CPU performance using 29 different CPU test workloads. A higher number in these graphs represents better CPU performance. SPEC CPU 2006 has two components - integer and floating point. Only integer results are included because floating point results were similar. Complete results are available in the report.

Due to SPEC rules governing test repeatability (which cannot be guaranteed in virtualized, multi-tenant cloud environments), the results below should be taken as estimates.

In the web server tests Amazon EC2 edged out other services - likely due to slightly faster hardware - 2.8 GHz Ivy Bridge. This was particularly apparent for the t2.medium instance in peak/burst operating mode in the small web server comparisons. The report lists all CPU architectures we observed for each service. Azure predictably performed poorer due to its use of older AMD 4171 processors.

In the database server tests Rackspace Performance 2 instances were slightly faster except for the Large Database Server where Azure's new A9 compute intensive instance performed best.

CPU Variability

One factor many comparisons overlook is performance variability. Cloud services are usually virtualized and multi-tenant and these factors can introduce performance variability due to sharing of resources.

To capture CPU performance variability, we ran SPEC CPU multiple times on multiple instances from each service and measured the relative standard deviation from the resulting metrics. In these graphs a higher value represents greater variability.

For web server tests CPU variability was highest for SoftLayer and Rackspace. For SoftLayer the cause of this was their use different CPU models, some older and some newer (we observed 6 different architectures on SoftLayer from 2009 X3470 to 2013 Ivy Bridge). For Rackspace and DigitalOcean the CPU architecture was consistent, so the reason may be resource contention and/or use of hyperthreading and floating cores.

For database server tests SoftLayer variability was again the highest due to changing CPU types across instances. Rackspace Performance 2 instances were less variable than Performance 1 instances.

Disk Performance

Our disk performance tests were conducted using 6 block sizes (4k, 16k, 32k, 64k, 128k and 1024k) and asynchronous IO using an optimal queue depth for each service, workload and block size. For the web server comparisons our testing used 100% read workloads. For database server comparisons we used mixed read + write workloads (80% write and 20% read). More details about the fio test settings are provided in the report.

At the time of testing, Google, Microsoft and SoftLayer did not offer a local SSD storage (Google recently announced external SSD storage). Because of this, web server disk performance for these services was notably slower compared to that of services with a local SSD storage option.

In these graphs, a higher value signifies faster performance. Our fio test results include 12 sets of test per instance (6 block sizes with both random and sequential IO). For brevity purposes, I've provided a sampling of these results in this post. The complete results are available in the report.

For the large web server random read tests, Amazon EC2 and Rackspace performed best. Although DigitalOcean also uses local SSD storage, its performance was consistently slower that other local SSD based services.

For the database server disk tests we used Amazon EC2 standard and EBS optimized instances with General Purpose (SSD) and provisioned IOPS (PIOPS) EBS volumes. For Rackspace we used both SSD and SATA off instance storage. DigitalOcean was excluded from the database server tests because they do not have an off instance storage option. For the small database server the Amazon EBS PIOPS volumes were provisioned for 1500 IOPS, and the Rackspace database servers used SATA storage volumes.

For the medium database server the Amazon EBS PIOPS volumes were provisioned for 3000 IOPS. The Rackspace database servers used SSD storage volumes.

For the large database server the Amazon EBS PIOPS volumes were provisioned for 4000 IOPS. The Rackspace database servers used SATA storage volumes.

Disk Consistency

Disk IO is a performance characteristic where variability can be very high for cloud environments. This is often due to the same physical disk drives being shared by multiple users. For off instance storage variability can be magnified due to network effects.

To measure disk performance consistency, we captured relative standard deviation of IOPS from many test iterations on multiple instances. In these graphs a higher value signifies greater variability. These graphs provide a subset of the analysis included in the report.

For the web server tests, DigitalOcean consistently demonstrated the highest variability of all services.

For all three database server test iterations Amazon EBS PIOPS volumes were the least variable, and Rackspace SATA volumes had the highest variability.

The medium Rackspace database servers used SSD external volumes which was less variable than SATA volumes. IO for Amazon EBS PIOPS volumes was very consistent. Microsoft Azure consistency improved with this instance type - perhaps because it is a larger type.

Rackspace SATA external volumes demonstrated high variability in the large database server tests.

Memory Performance

Memory performance is usually dependent on CPU architecture, with newer CPU models usually having faster memory. Unlike CPU and disk, memory is often not shared and thus provides consistent performance.

We ran memory benchmarks using the STREAM benchmark. STREAM uses multiple memory tests to measure memory bandwidth. In these graphs a higher value signifies faster memory bandwidth.

Memory performance results were similar for all database sizes with the exception of SoftLayer (due to use of multiple CPU types), so I've only included results for the medium and large database instances. Amazon EC2 and Google were consistently the fastest for memory tests.

Microsoft Azure A9 instance performed well in the large database server group due to its use of newer Intel E5-2670 processors (as opposed to old AMD 4171 processors used on other Azure instances).

External Network Performance

Web servers typically respond to requests from users over the Internet. The speed at which web servers respond is dependent on the distance between the user and web server, and the Internet connectivity provided by the service.

Measuring external network performance is complex due to a large number of variables (there are tens of thousands of ISPs and infinite routing possibilities). We host a cloud speedtest that lets users run network tests to measure their connectivity to different cloud services. Using a Geo IP database, we summarize the results of these tests regionally. These summaries are the basis for the data provided in the graphs below. In these graphs we provide mean latency between cloud services and users located in different geographical regions. A lower value signifies better latency - a shorter logical path between users and the service.

Compute services often allow users to provision instances in data centers located in different regions. These results are based on the most optimal service region for each and continent.

Region Amazon EC2 DigitalOcean Google Compute Engine Microsoft Azure Rackspace
North America us-east-1 NY2 us-central1 South Central US ORD
Europe eu-west-1 AMS1 europe-west1 West Europe LON
Asia ap-southeast-1 SG1 asia-east1 Southeast Asia HKG
Oceania ap-southeast-1 SFO1 asia-east1 Southeast Asia SYD
South America sa-east-1 NY2 us-central1 South Central US DFW

SoftLayer is excluded from these tests because at the time of writing, we did not have it available on the cloud speedtest.

Network connectivity in Asia and South America is generally slower than North America and Europe. Because of this, every service performed worse in those regions.

Internal Network Performance

Database servers often interact with other servers located in the same data center over the internal network. For example, a web server might query a database server to display dynamic content on a website. Because of this, we chose to measure internal network performance for the database workload. To do so, we used a custom test that uses ping and http to measure latency and throughput performance within the internal network of each service.

The data below is based on interaction between each of the three types of web and database server instances - small, medium and large. To maximize performance, we provisioned instances using the following optimal network settings for each service:

For latency tests a lower value signifies better performance, while for throughput tests a higher value is better.

Use of Amazon EC2 placement groups and VPC provided significantly higher throughput compared to other services - nearly 9 Gb/s. SoftLayer and DigitalOcean networks are likely capped at 1 Gb/s.

For some services, uplink and downlink network throughput performance was asymmetrical. Rackspace for example appears to cap the uplink (outbound traffic) but not the downlink (inbound traffic) - even on the internal network interface. The caps listed on the Rackspace website range between 200 Mb/s to 1,600 Mb/s for Performance 1 compute instances.

Network latency for all services was less than 0.7ms. Amazon EC2 and SoftLayer had the best latency at between 0.10 and 0.15ms.

Value

When picking a service - you may want to look for the best combination of performance and price - or in other words, the best value.

In this section, I estimate value for each service using CPU performance and hourly costs for each service and instance type. Because services sometimes offer different prices based term commitments, I included the following term options (where applicable):

Value was carried over for services not offering a particular term option. All of the prices are hourly normalized. If a term option has a setup fee, that fee was added to the hourly cost by dividing it by the total number of hours in the term.

The benchmark metric used for this value analysis is SPEC CPU 2006. The value metric provided is a ratio of SPEC CPU 2006 to the hourly cost for each service and instance type. More detail and data is available in the report.

In peak/burst operating mode the t2.medium provides the best value by a significant margin in the small web server category. Keep in mind that this level of performance is achievable for between 10-20% of the total operational time. The the baseline (non-burst) value is substantially lower (about 5X).

Google's new monthly discounts offer a good value without requiring any upfront setup fees (if you keep the instance live for a month you automatically get a discount). DigitalOcean provides good value for on-demand instances. Amazon EC2 is generally the best value for 1 or 3 year terms. Microsoft Azure value is low due to use of older CPU hardware resulting in poor performance on CPU benchmarks.

DigitalOcean value for database server instances was best for on-demand pricing, while Amazon EC2 was best with for 1 and 3 year terms.

Although Microsoft Azure performed well in the large database server comparisons, its value was low due to its higher price tag (partially attributable to its 7X CPU cores-to-memory ratio).

Other Considerations

When choosing a compute service, there are other things you should consider unrelated to performance or price. Depending on your workloads, these may be of greater or lesser importance.

Service Reliability

Service reliability is likely important to any organization. Quantifying reliability is difficult because there are many possible points of failure. We maintain compute instances with most compute services and use them to track outages and measure availability. Although maintaining a single compute instance with each service isn't a full proof method for measuring availability, we do manage to capture many service related outages this way. The table below shows our availability data for each service over the past 30 days. These metrics are based on mean availability for each service across all service regions. Our cloud status page provides realtime status and lets you view availability and outages over different periods.

Service 30 Day Availability Outages Downtime
Amazon EC2 100% 0
Rackspace 99.9971% 3 7.45 mins
Google Compute Engine 99.9862% 10 17.85 mins
Microsoft Azure 99.9922% 23 25.5 mins
DigitalOcean 99.9756% 16 52.67 mins

Service Quotas

Most compute services impose limits on the number of instances you can have active at a given time. These limits may affect you if you experience rapid growth, or have elastic workloads with high usage requirements during peak times. Although you can typically request an increase to quotas, the default limits can convey the scale at which a service operates. Services operating at a larger scale may be better able to support rapid growth and elastic workloads.

Each service typically has a different procedure for obtaining quote increases. Our experience with these has been mixed across services. With Amazon and Google, for example, we have often obtained increases within hours, while with some others responses have slower and capacity more limited.

These tables list both the quota policies and how they would affect provisioning of compute instances of different sizes.

Quota Policies

Service Policy
Amazon EC2 20 instances per region for most compute instance types
DigitalOcean 10 compute instances (Droplets)
Google Compute Engine 24 CPU cores per region
Microsoft Azure 20 CPU cores
Rackspace 128 GB memory
SoftLayer 20 compute instances per day, additional instances reviewed manually for approval

Instance Quotas

Instance Size Amazon EC2 DigitalOcean Google Compute Engine Microsoft Azure Rackspace SoftLayer
2 GB / 1 Core 160 (20 per region) 10 72 (24 per region) 20 64 20 daily1
4 GB / 2 Core 160 (20 per region) 10 32 (12 per region) 10 32 20 daily
8 GB / 4 Core 160 (20 per region) 10 18 (6 per region) 5 16 20 daily
16 GB / 8 Core 160 (20 per region) 10 9 (3 per region) 2 8 20 daily
32 GB / 16 Core 160 (20 per region) 10 3 (1 per region) 1 4 20 daily

1Certain SoftLayer regions have more cloud capacity than others. We have experienced provisioning requests in some regions being cancelled due to lack of capacity, while in others they are successful.

Storage Offerings

Compute services often offer different storage options. These options may include:

This table shows support of each of these storage options by each service in this post:

Compute Service Local External Drive Types Multiple Volumes Snapshots Provisioned IO
Amazon EC2 Yes Yes SATA; SSD Yes Yes Yes
DigitalOcean Yes No SSD No Yes No
Google Compute Engine Yes (beta) Yes Unknown1 Yes Yes No
Microsoft Azure Yes Yes Unknown Yes Yes No
Rackspace Yes Yes SATA; SSD Yes Yes No
SoftLayer Yes Yes SATA Yes Yes No

1Google recently announced preview of SSD based off instance storage

Networking Capabilities

Another consideration for compute services is the networking capabilities. The following are a few such considerations:

This table shows support of each of these networking capabilities by each service in this post:

Compute Service IPv6 Support Multiple IPv4 Private IP Load Balancing Health Checks
Amazon EC2 Yes1 Yes Yes Yes Yes
DigitalOcean No No Partial (3 of 6 regions) No No
Google Compute Engine No No Yes Yes Yes
Microsoft Azure No Yes Yes Yes Yes
Rackspace Yes Yes2 Yes Yes Yes
SoftLayer Yes Yes Yes Yes Yes

1 IPv6 supported when used with elastic load balancer (ELB) only

2 Must request with support ticket with Rackspace and provide valid justification

Data Center Locations

If your users are dispersed globally - or located primarily in a specific geographical region, you may want to consider where a services' data centers are located. The table below lists the number of data centers for each service in 6 continents.

Compute Service North America South America Europe Asia Australia
Amazon EC2 4 1 1 3 1
DigitalOcean 3 0 1 1 0
Google Compute Engine 1 0 1 1 0
Microsoft Azure 6 1 2 4 0
Rackspace 4 0 1 1 1
SoftLayer 4 0 1 1 0

Security

Security is another concern many have with the cloud. A services' security capabilities may be an important consideration. Although operating systems usually support security features and software based firewalls, it is often better to deal with security separately, outside of the operating system entirely. Some common security features include:

This table lists support for these security features by each compute service in this post:

Compute Service Firewall VPN VPC PCI DSS
Amazon EC2 Yes Yes Yes Yes
DigitalOcean No No No No
Google Compute Engine Yes No Yes No
Microsoft Azure Yes Yes Yes Yes
Rackspace $160/mo1 $160/mo1 Yes Yes
SoftLayer Yes Yes Yes Yes

1 Requires additional subscription to Brocade Vyatta vRouter starting at $160 per month for each compute instance

Service Ecosystem

If you are using a cloud compute service, you likely use other types of cloud services. When evaluating compute services you should consider the provider's ability to fulfill other hosting requirements you may have. Some other types of cloud services include:

This table lists support for each of these other cloud service types by each provider included in this post:

Compute Service Object Storage CDN DNS DbaaS PaaS
Amazon Yes Yes Yes Yes Yes
DigitalOcean No No No No No
Google Yes No Yes Yes Yes
Microsoft Azure Yes Yes No Yes Yes
Rackspace Yes Yes1 Yes Yes No
SoftLayer Yes Yes2 Yes No No

1 Rackspace resells Akamai CDN using a subset of 219 Akamai POPs

2 SoftLayer resells Edgecast CDN

Conclusion

Comparing compute services is a challenging task. I've covered a lot of ground in this post including how to properly choose instances from different services, picking relevant benchmarks, some actual comparisons of services, estimating value, and other considerations. The biggest take away I'd hope for is a better understanding about how to compare compute services accurately, and identify comparisons that are of questionable quality.

It should also be noted that because compute services are frequently updated, the validity of the benchmark metrics in this post are time limited.

If you'd to know more, the full report download contains 120 pages of graphs, tables and additional commentary.

Correction 7/16/2014: Post and report have been updated to reflect availability of local storage, reduced pricing, and a compute region in South America for Microsoft Azure.