Cloud Server Benchmarking Part 3 - Java, Ruby, Python and PHP

This is the third post in our series on cloud server performance benchmarking. The previous blog posts, What is an ECU? CPU Benchmarking in the Cloud and Disk IO Benchmarking in the Cloud focused on more synthetic (i.e. raw performance numbers with no real-world application) CPU and Disk IO performance. In this post, we'll look at performance using 4 common interpreted (full or byte-code) programming languages: Java, Ruby, Python and PHP.

This post is by no means intended to provide exhaustive and exact metrics for cloud server performance with these languages. The purpose of this and the other benchmarking posts is to provide a quick reference and starting point for further research and evaluation of different cloud providers. There is much variation in performance, features, reliability, support, pricing and other factors of different cloud providers, yet if you were to evaluate those providers based solely on marketing literature, you'd be hard pressed to distinguish one from another. Our goal is to reduce this ambiguity by providing objective, quantifiable measurements for comparing cloud providers.

Benchmark Setup

All benchmarked cloud servers were configured almost identically in terms of OS and software:

Benchmark Methodology

We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is Storm on Demand's bare-metal instance on E5506 2.13 hardware (dual processors - 8 cores total), with 4 x 15K RPM SAS drives (Raid 10) and 8GB ram. This is a fairly high-end server with many cores and very fast IO. We assigned this server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 4 language benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Example Score Calculation:

Baseline aggregate score = SPECjvm2008: 100; ruby-benchmark-suite: 100; PyBench: 100; PHPBench 100

Total baseline score = 400

Server X aggregate score = SPECjvm2008: 80/100; ruby-benchmark-suite: 90/100; PyBench: 70/100; PHPBench 95/100

Total baseline score = 335

Server X Score = (335/400) * 100 = 83.75

Benchmarks

The following benchmarks were used in this post:

Results

The results are divided into tables separated by provider. If the provider has more than one data center locations, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), PHPBench score, PyBench score, ruby-benchmark-suite score, SPECjvm2008 score, and the aggregate baseline relative score (as described above). Each of the 4 individual scores are linked to the complete results for those specific tests (a single small text file for PHPBench, PyBench, ruby-benchmark-suite and a 3-4MB zip file for SPECjvm2008).

Note: Some smaller server configurations (e.g. EC2 m1.small and c1.medium) are not included in these results due to their inability to run one more more of the individual benchmarks.

Amazon Web Services (AWS) [US East]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50328 3725 4.12 197.42 138.58
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50253 3737 4.08 108.09 115.45
c1.xlarge Xeon E5410 2.33 8 cores 7 44460 4586 5.46 131.12 105.76
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50396 3756 4.1 58.15 102.27
m1.xlarge Xeon E5430 2.66 4 cores 15 38737 5279 6.45 68.09 79.62
m1.large Xeon E5430 2.66 2 cores 7.5 38625 5324 6.58 38.21 71.29

Amazon Web Services (AWS) [US West]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50553 3723 4.09 197.83 139.02
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50062 3797 4.1 108.53 114.82
c1.xlarge Xeon E5410 2.33 8 cores 7 44733 4566 5.48 130.26 105.75
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50832 3724 4.12 58.59 102.81
m1.xlarge Xeon E5430 2.67 4 cores 15 39001 5323 6.36 67.98 79.79
m1.large Xeon E5430 2.66 2 cores 7.5 38716 5382 6.46 38.17 71.38

Amazon Web Services (AWS) [EU West]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50775 3739 4.1 200.09 139.55
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50207 3723 4.1 107.32 115.22
c1.xlarge Xeon E5410 2.33 8 cores 7 44656 4563 5.51 131.52 105.94
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50275 3732 4.11 58.92 102.55
m1.xlarge Xeon E5430 2.67 4 cores 15 38387 5259 6.37 69.06 79.94
m1.large Xeon E5430 2.67 2 cores 7.5 38327 5299 6.39 39.15 71.91

Amazon Web Services (AWS) [APAC]
ID CPU Memory PHP Python Ruby SPECjvm Score
m2.4xlarge Xeon X5550 2.67 8 cores 68.4 50336 3743 4.42 131.72 119.05
m2.2xlarge Xeon X5550 2.67 4 cores 34.2 50095 3762 4.42 89.86 108.51
m2.xlarge Xeon X5550 2.67 2 cores 17.1 50042 3799 4.43 50.63 97.92
c1.xlarge Xeon E5410 2.33 8 cores 7 44992 4608 5.57 63.93 88.13
m1.xlarge Xeon E5430 2.67 4 cores 15 38793 5314 6.56 54.15 75.62
m1.large Xeon E5430 2.67 2 cores 7.5 38214 5299 6.49 34.1 70.28

Rackspace Cloud [Dallas]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 2374 HE 2.20 4 cores 4 39210 5465 5.74 63.07 79.76
16gb Opteron 2374 HE 2.20 4 cores 16 37914 5517 5.75 62.77 78.66
1gb Opteron 2374 HE 2.20 4 cores 1 38679 5396 5.81 56.22 77.73
2gb Opteron 2374 HE 2.20 4 cores 2 38868 5454 5.87 41.89 73.71

Rackspace Cloud [Chicago]
ID CPU Memory PHP Python Ruby SPECjvm Score
16gb-il Opteron 2374 HE 2.20 4 cores 16 38729 5397 5.7 63.86 80.08
4gb-il Opteron 2374 HE 2.20 4 cores 4 39065 5424 5.8 62.14 79.42
8gb-il Opteron 2374 HE 2.20 4 cores 8 38958 5455 5.66 63.88 78.31
1gb-il Opteron 2374 HE 2.20 4 cores 1 38443 5434 5.68 56.88 77.99

Storm on Demand [MI, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
48gb Xeon X5650 2.67 12 cores 45.9 50602 3810 6.17 174.81 124.05
i5-750-4gb-2disks Intel i5 750 2.67 4 cores 4 45560 4326 3.17 89.03 110.35
i5-750-2gb-1disk Intel i5 750 2.67 4 cores 2 45620 4352 3.16 82.16 108.53
x3440-8gb-2disks Xeon X3440 2.53 8 cores 8 43428 4599 3.38 97.18 107.55
8gb Xeon X3440 2.53 4 cores 7 43861 4565 3.35 86.55 105.51
e5506x2-4gb-1disk Xeon E5506 2.13 8 cores 8 40577 4774 3.93 108.06 103.52
e5506x2-4gb-2disks Xeon E5506 2.13 8 cores 8 40136 4766 3.94 105.9 102.67
32gb Opteron 2378 2.40 8 cores 30.4 40954 5104 3.8 104.01 101.96
4gb Core i5 750 2.67 2 cores 3.5 45983 4331 3.17 49.71 100.35
e5506x2-8gb-4disks Xeon E5506 2.13 8 cores 8 40731 4765 4.02 96.18 100
e5506x2-8gb-4disks Xeon E5506 2.13 8 cores 8 40306 4780 4.01 95.13 99.45
2gb Core 2 Quad Q9400 2.66 1 core 1.7 49905 4142 3.59 23.91 93.6
amd2350x2-32gb-4disks Opteron 2350 2.00 8 cores 32 32182 6271 4.96 86.73 81.55
16gb Opteron 2350 2.00 4 cores 15.2 32240 6187 4.87 53.1 73.48

GoGrid [CA, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5450 2.99 6 cores 8 55970 3616 3.53 101.76 121.46
4gb Xeon E5520 2.27 4 cores 4 40513 4806 3.8 74.34 95.42
2gb Xeon E5520 2.27 2 cores 2 38206 5090 4.09 36.13 80.82
1gb Xeon E5520 2.27 1 core 1 38215 4945 4.02 20.56 77.89

Voxel [New York]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-ny Xeon L5520 2.26 7 cores 14 44463 4214 4.98 119.82 106.89
8gb-ny Xeon L5520 2.26 4 cores 8 44043 4237 4.91 81.23 96.73
4gb-ny Xeon L5520 2.26 2 cores 4 40921 4446 4.95 45.71 84.1
2gb-ny Xeon L5520 2.26 1 core 2 44022 4210 4.94 24.47 82.02

Voxel [Amsterdam]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-nl Xeon L5520 2.26 7 cores 14 45069 4221 4.85 124.22 108.9
8gb-nl Xeon L5520 2.26 4 cores 8 45043 4219 4.9 82.02 97.71
4gb-nl Xeon L5520 2.26 2 cores 4 44880 4217 4.83 45.52 88.44
2gb-nl Xeon L5520 2.26 1 core 2 44436 4229 4.86 24.57 82.51

Voxel [Singapore]
ID CPU Memory PHP Python Ruby SPECjvm Score
14gb-sg Xeon L5520 2.26 7 cores 14 42488 4468 5.14 111.08 101.17
8gb-sg Xeon L5520 2.26 4 cores 8 42355 4473 5.15 75.75 91.84
4gb-sg Xeon L5520 2.26 2 cores 4 42433 4480 5.15 42.15 83.11
2gb-sg Xeon L5520 2.26 1 core 2 42153 4482 5.16 23.08 77.93

NewServers [FL, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
fast Xeon E5450 2.99 4 cores 4 57795 3600 3.24 81.3 120.71
jumbo Xeon E5504 2.00 8 cores 48 37593 5087 4.2 110.03 99.02
large Xeon E5405 2.00 4 cores 4 38161 5462 4.59 59.75 82.66
med Xeon 3.20 2 cores 2 29381 7383 5.89 25.74 57.92
small Xeon 2.80 1 core 1 25223 8183 6.9 12.37 47.82

Linode VPS Hosting [Atlanta]
ID CPU Memory PHP Python Ruby SPECjvm Score
5760 Xeon L5520 2.27 4 cores 5.54 42841 4418 4.67 78.51 95.19
14400 Xeon L5520 2.27 4 cores 14.06 42644 4410 4.65 78.45 95.19
2880 Xeon L5520 2.27 4 cores 2.81 39050 4503 5.02 66.78 87.8
1080 Xeon L5520 2.27 4 cores 1.05 40442 4579 5.12 59.57 85.95
8640 Xeon L5520 2.27 4 cores 8.45 32904 5632 5.28 67.85 78.02

SoftLayer [Dallas]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42414 4439 5.5 123.82 103.33
4gb Xeon E5520 2.27 4 cores 4 42471 4459 5.51 78.74 91.49
2gb Xeon X3460 2.80 2 cores 2 47816 4130 5.53 49.59 89.26
1gb Xeon X3460 2.80 1 core 1 47634 4118 4.61 27.86 87.21

SoftLayer [WDC]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42981 4441 5.48 127.55 104.7
4gb Xeon X3460 2.80 4 cores 4 47425 4101 4.62 91.07 103.58
2gb Xeon X3460 2.80 2 cores 2 47478 4108 4.4 48.95 93.7
1gb Xeon X3460 2.80 1 core 1 47614 4113 4.47 27.61 87.85

SoftLayer [Seattle]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb Xeon E5520 2.27 8 cores 8 42642 4451 5.54 125.75 103.76
2gb Xeon X3460 2.80 2 cores 2 48095 4138 4.63 48.7 92.67
1gb Xeon X3460 2.80 1 core 1 47472 4113 4.61 26.49 86.79

Terremark vCloud Express [FL, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
16gb-8vpu Opteron 8389 2.91 8 cores 16 48625 4623 3.99 100.42 106.9
1gb Opteron 8389 2.91 1 core 1 48997 4254 3.18 22 95.4
2gb Opteron 8389 2.91 1 core 2 48651 4321 3.17 21.87 94.82
4gb-2vpu Opteron 8389 2.91 2 cores 4 39078 5019 3.72 38.78 84.82
8gb-4vpu Opteron 8389 2.91 4 cores 8 32959 5388 4.19 70.56 84.67

VPS.NET [Atlanta]
ID CPU Memory PHP Python Ruby SPECjvm Score
18node Xeon E5520 2.27 7 cores 6.6 39447 4762 5.85 96.35 91.45
12node Xeon E5520 2.27 [5 processors, 5 cores 4.4 42843 5035 5.39 77.12 88.65
6node Xeon E5520 2.27 [3 processors, 3 cores 2.2 33213 5730 6.8 45.81 67.86
3node Xeon E5520 2.27 2 cores 1.1 34188 5452 6.54 35.72 67.49

OpSource Cloud [VA, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
2gb Xeon X7460 2.66 1 core 2 28083 7721 7.93 11.13 48.23
8gb-4cpu Xeon X7460 2.66 4 cores 8 24639 8758 9.08 9.03 42.14
32gb-4cpu Xeon X7460 2.66 4 cores 32 23604 8790 8.87 9.04 41.72
16gb-4cpu Xeon X7460 2.66 4 cores 16 24636 8738 9.43 8.77 41.69

Speedyrails [Quebec]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Xeon E5520 2.27 8 cores 4 41819 4735 5.26 100.74 96.12
1gb Xeon E5520 2.27 8 cores 1 41541 4975 5.28 56.54 83.18

Zerigo [CO, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 2374 HE 2.20 4 cores 4 39030 5400 5.58 55.54 78.46

ReliaCloud [MN, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Xeon E5504 2.00 4 cores 4 13287 13209 10.7 3.85 27.56
8gb 8 14159 15567 10.79 3.69 26.62
2gb Xeon E5504 4.40 2 cores 2 12131 13819 11.36 5.36 26.31

CloudSigma [Switzerland]
ID CPU Memory PHP Python Ruby SPECjvm Score
32gb-20gh Six-Core Opteron 2427 2.21 8 cores 32 37912 5646 4.27 97.2 93.17
16gb-16gh Six-Core Opteron 2427 2.21 8 cores 16 37599 5654 4.3 98.23 93.05
8gb-8gh Six-Core Opteron 2427 2.21 4 cores 8 37473 5619 4.2 59.99 83.72
4gb-4gh Six-Core Opteron 2427 2.21 2 cores 4 37409 5582 4.26 32.54 76.35
1gb-2gh Six-Core Opteron 2427 2.21 1 core 1 37326 5609 4.27 17.2 72.16
2gb-2gh Six-Core Opteron 2427 2.21 1 core 2 36561 5789 4.25 17.82 71.3

IBM Development & Test Cloud [NY, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
large Xeon X5570 2.93 8 cores 3.5 45260 4711 3.6 140.6 117.53

BlueLock [IN, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb-4cpu Xeon X5550 2.67 4 cores 8 54399 3761 3 94.08 123.02
16gb-8cpu Xeon X5550 2.67 8 cores 16 48314 3859 3.36 96.07 115.41
4gb-2cpu Xeon X5550 2.67 2 cores 4 45071 3894 3.21 51.9 103.05
2gb Xeon X5550 2.67 1 core 2 50021 3739 3.16 26.72 101.31
1gb Xeon X5550 2.67 1 core 1 49499 3745 3.06 16.01 99.2

Cloud Central [Australia]
ID CPU Memory PHP Python Ruby SPECjvm Score
reg Opteron 2374 HE 2.20 4 cores 4 38725 5469 5.55 60.59 79.41
large Opteron 2374 HE 2.20 4 cores 8 38921 5439 5.64 60.58 79.36
huge Opteron 2374 HE 2.20 4 cores 16 37989 5573 6.43 61.86 76.4
small Opteron 2374 HE 2.20 4 cores 1 37356 5597 6.18 49.6 73.36

RimuHosting [TX, US]
ID CPU Memory PHP Python Ruby SPECjvm Score
tx-2gb Xeon E5506 2.13 2 cores 2 39096 4883 4.89 39.28 79.16

RimuHosting [New Zealand]
ID CPU Memory PHP Python Ruby SPECjvm Score
nz-2gb Xeon E5506 2.13 2 cores 2 39529 4854 5.08 39.22 78.78

ElasticHosts [UK]
ID CPU Memory PHP Python Ruby SPECjvm Score
8gb-20gh Xeon E5420 2.50 6 cores 8 47168 4431 4.1 92.82 104.47
2gb-4gh Xeon E5420 2.50 2 cores 2 43723 4508 4.11 34.92 86.79
1gb-2gh Xeon E5420 2.50 1 core 1 44943 4470 4.1 20.94 84.19

Flexiscale [UK]
ID CPU Memory PHP Python Ruby SPECjvm Score
4gb Opteron 8220 2.80 2 cores 4 36319 6033 4.03 34.1 69.57
8gb Opteron 8218 2.62 4 cores 8 33878 6570 4.35 53.46 67.87
2gb Opteron 8218 2.60 1 core 2 35141 6505 4.16 17.19 63.09
1gb Opteron 8218 2.60 1 core 1 34497 6590 4.39 16.16 63.05

Summary

We think it is important for these and other cloud performance benchmarks to be made available publicly. With all the hype surrounding cloud computing today, new cloud users may not understand what exactly they are getting in terms of performance and scalability. As these benchmarks show, not all clouds are created equal, and there can be very pronounced performance variations from one provider to another. One of the greatest benefits of the cloud is simplicity and a perceived ability to infinitely scale. However, it should be understood that with that simplicity and abstraction, the cloud is still powered by physical hardware, CPUs, disks and memory, and subject to the same performance limitations that same hardware exhibits in a non-cloud environment.