Comments on CloudHarmony Blog: What is an ECU? CPU Benchmarking in the Cloud

Welcome to RSA performance.com your source for the...

2011-10-08T10:56:31.850-07:00

Welcome to RSA performance.com your source for the lowest price on performance parts. We authorize dealer for Tein, HKS, Greddy, Skunk2, Ark performance, Tanabe, Ksport, Weapon R, Drag wheels and more. We offer great service and gaurantee the lowest price.

RSAperformance

Where did you find the hardware being run behind t...

2011-01-25T08:39:32.839-08:00

Where did you find the hardware being run behind the various instance types on Amazon EC2?

The hardware for the High Memory Instances is given on Amazon's Instance Types page, but I don't see anywhere that the other instances are actually given an actual hardware correlation (like AMD Opteron 2218 2.6 GHz or Xeon E5430 "Harpertown" 2.66 GHz).

Thanks for your help.

Thanks for this benchmark. It's very useful fo...

2010-06-15T01:21:52.714-07:00

Thanks for this benchmark. It's very useful for us.
Are you planning to get a benchmark of "CloudShare"?

would like to add Joyent, but have not been able t...

2010-06-07T14:46:42.931-07:00

would like to add Joyent, but have not been able to get a response

Did you look at Joyent?

2010-06-07T14:27:52.601-07:00

Did you look at Joyent?

GigeNET has been added to our site and will be inc...

2010-06-03T23:49:27.301-07:00

GigeNET has been added to our site and will be included in the next round of benchmarks

2010-06-03T12:30:26.621-07:00

This comment has been removed by the author.

You should benchmark gigenetcloud too. its one of ...

2010-06-03T11:31:28.467-07:00

You should benchmark gigenetcloud too.
its one of the fastest clouds there is.

Thanks for the comments. > Cost to performance...

2010-06-01T18:28:09.361-07:00

Thanks for the comments.

> Cost to performance ratios: this is a great suggestion. We have recorded all of the different pricing options (i.e. hourly, reserve, spot, pre-pay) for every server we've benchmarked. Once we finish the series of posts on performance, we'll write a post dedicated to pricing and performance to cost ratios based on each payment method

> Multi-threaded focused metric: I updated the post to highlight the benchmarks that are multi-thread aware versus those that are not. 900 out of 1700 points (52%) for the CCU calculation are multi-thread aware (c-ray, crafty, geekbench, graphics-magick, hmmer, openssl and unixbench). I created a sample mtCCU metric using only those benchmarks with the resulting mtCCU calculation table (900 pts max):

26 ECUs (m2.4xlarge) = 851
13 ECUs (m2.2xlarge) = 707
6.5 ECUs (m2.xlarge) = 529
4 ECUs (m1.large) = 385
1 ECU (m1.small) = 110

We'll make the mtCCU metric available soon.

> Graphs: we'll be adding graphing functionality to the website soon

> Sharing data: We'll be making all of the data (raw benchmark scores and aggregate metrics) available via web services and on our website soon. The web services will enable you to create your own CCU metric using whatever combination of benchmarks and baseline calculation you desire

> Rimu Hosting: sorry about that, it has been corrected

Point of national pride: RimuHosting are a New Zea...

2010-06-01T13:10:54.106-07:00

Point of national pride: RimuHosting are a New Zealand company, not an Australian company. (The rimu is a tree native to New Zealand.)

This is impressive work, if for nothing else than ...

2010-06-01T11:05:05.751-07:00

This is impressive work, if for nothing else than noting the breadth of the cloud providers surveyed. Would you be willing to share the data used to build the charts above to facilitate further analysis?

where are the graphs?

2010-06-01T08:00:28.905-07:00

where are the graphs?

This is a great initiative! I've always thoug...

2010-06-01T02:20:16.810-07:00

This is a great initiative!

I've always thought that AWS availability zones didn't provide the same horse power for he same instances and now I have proof ...

Establishing $/CCU/h as the normalized pricing comparison metric (abstracting SLAs, platform capabilities and 3rd party support) would be a great achievement. I would love to that added to the tables above.

I would think that displaying that metric ($/CCU/h) as a function of the CCU offered would a downward trend up until a point where one starts paying a premium for every extra CCU that an instance offers. This matches the costs of CPUs and infrastructure in general. Would love to see that chart to confirm this hunch ...

Thanks for this benchmarking effort, this is reall...

2010-05-31T23:29:40.095-07:00

Thanks for this benchmarking effort, this is really good.

One question though, what specific benchmark test would really measure the power of a heavily multi-threaded server application (such as a database or an application server)? I get the feeling that quite a few of the benchmarks you've used are not heavily multi-threaded and might not be able to use the full power of those high-end machines.

Consequently, wouldn't it make sense to have a mtCCU index as well in order to provide a bench for those heavily multi-threaded environments - which will be the most probable to run on those high-end machines and be able to fully leverage their multi-CPU/Core architecture?

That would also probably give you a much more linear scale for your mapping table. Currently, the pretty wide variations in your mapping table make me feel pretty un-easy and, IMO, cannot be simply explained by their CPU-level architecture.

Cheeers,

Sacha Labourey
CloudBees, Inc.

Thank you for your thoughtful efforts to benchmark...

2010-05-31T11:28:59.698-07:00

Thank you for your thoughtful efforts to benchmark cloud server CPU performance. Your efforts make us here at GoGrid feel like the industry is finally maturing.

The "CCU" CloudHarmony Compute Unit seems like a reasonable approach to a benchmark. We hope that your benchmarking will progress to establish relative costs per CCU per hour.

As an example, our standard GoGrid 8 GB VM produces 23.2 CCUs. The cost of this server is actually only $0.40 / hour with GoGrid's volume pricing plan, which makes it only 1.7 cents per CCU / Hour. ($0.40 / 23.2 = $0.017) And it runs both Windows and Linux at that price.

By comparison, it appears the comparable AWS server from your testing in terms of CCUs is the "High-Memory Quadruple Extra Large/m2.4xlarge = 26 ECUs" which produced 27.45 CCUs. This server costs $2.40 / hour so $2.40 / 27.45 CCUs = 8.7 cents per CCU hour.

Naturally we're pleased that the GoGrid server delivers approximately 5 times the CCUs per dollar in this test. However, a more interesting point to your readership is that there are huge variations in pricing, pricing models and feature-sets. It is quadruple-extra-large important that folks like CloudHarmony start to make sense of it all.

Thank you for helping our industry,

John Keagy
CEO
www.GoGrid.com