Evaluating Public Cloud Computing Performance with CloudHarmony

Pacific-Tier Communications met up with Jason Read, owner of CloudHarmony, a company specializing in benchmarking the cloud, at Studio City, California, on 25 October.  Read understands how confusing and difficult it is to evaluate different service providers without an industry-standard benchmark.  In fact, Read started CloudHarmony based on his own frustrations as a consultant helping a client choose a public cloud service provider, while attempting to sort through vague cloud resource and service terms used by industry vendors.

“Cloud is so different. Vendors describe resources using vague terminology like 1 virtual CPU, 50 GB storage. I think cloud makes it much easier for providers to mislead. Not all virtual CPUs and 50 GB storage volumes are equal, not by a long shot, but providers often talk and compare as if they are. It was this frustration that led me to create CloudHarmony” explained Read.

So, Read went to work creating a platform for not only his client, but also other consultants and IT managers that would give a single point of testing public cloud services not only within the US, but around the world.    Input to the testing platform came from aggregating more than 100 testing benchmarks and methodologies available to the public.  However CloudHarmony standardized on CentOS/RHEL Linux as an operating system  which all cloud vendors support, “to provide as close to an apples to apples comparison as possible” said Read.

Customizing a CloudHarmony Benchmark Test

Setting up a test is simple.  You go to the CloudHarmony Benchmarks page, select the benchmarks you would like to run, the service providers you would like to test, configurations of virtual options within those service providers, geographic location, and the format of your report.

Figure 1.  Benchmark Configuration shows a sample report setup.

“CloudHarmony is a starting point for narrowing the search for a public cloud provider” advised Read.  “We provide data that can facilitate and narrow the selection process. We don’t have all of the data necessary to make a decision related to vendor selection, but I think it is a really good starting point.

Read continued “for example, if a company is considering cloud for a very CPU intensive application, using the CPU performance metrics we provide, they’d quickly be able to eliminate vendors that utilize homogenous infrastructure with very little CPU scaling capabilities from small to larger sized instance.”

Cloud vendors listed in the benchmark directory are surprisingly open to CoudHarmony testing.  “We don’t require or accept payment from vendors to be listed on the site and included in the performance analysis” mentioned Read.  “We do, however, ask that vendors provide resources to allow us to conduct periodic compute benchmarking, continual uptime monitoring, and network testing.”

When asked if cloud service providers contest or object to CloudHarmony’s methodology or reports, Read replied “not frequently. We try to be open and fair about the performance analysis. We don’t recommend one vendor over another. I’d like CloudHarmony to simply be a source of reliable, objective data. The CloudHarmony performance analysis is just a piece of the puzzle, users should also consider other factors such as pricing, support, scalability, etc.”

During an independent trial of CloudHarmony’s testing tool, Pacific-Tier Communications selected the following parameters to complete a sample CPU benchmark:

• CPU Benchmark (Single Threaded CPU)
• GMPbench math library
• Cloud Vendor – AirVM (MO/USA)
• Cloud Vendor – Amazon EC2 (CA/USA)
• Cloud Vendor – Bit Refinery Cloud Hosting (CO/USA)
• 1/2/4 CPUs
• Small/Medium/Large configs
• Bar Chart and Sortable Table report

The result, shown above in Figure 2., shows a test result including performance measured against each of the above parameters.  Individual tests for each parameter are available, allowing a deeper look into the resources used and test results based on those resources.

In addition, as shown in Figure 3., CloudHarmony provides a view providing uptime statistics of dozens of cloud service providers over a period of one year.  Uptime statistics showed a range (at the time of this article) between 98.678% availability to 100% availability, with 100% current uptime (27 October).

Who Uses CloudHarmony Benchmark Testing?

While the average user today may be in the cloud computing industry, likely vendors eager to see how their product compares against competitors, Read targets CloudHarmony’s product to “persons responsible for making decisions related to cloud adoption.”  Although he admits that today most users of the site lean towards the technical side of the cloud service provider industry.

Running test reports on cloud harmony is based on a system of purchasing credits.  Read explained “we have a system in place now where the data we provide is accessible via the website or web services – both of which rely on web service credits to provide the data. Currently, the system is set up to allow 5 free requests daily. For additional requests, we sell web service credits where we provide a token that authorizes you to access the data in addition to the 5 free daily requests.”

The Bottom Line

“Cloud is in many ways a black box” noted Read.  “Vendors describe the resources they sell using sometimes similar and sometimes very different terminology. It is very difficult to compare providers and to determine performance expectations. Virtualization and multi-tenancy further complicates this issue by introducing performance variability. I decided to build CloudHarmony to provide greater transparency to the cloud.”

And to both vendors and potential cloud service customers, provide an objective, honest, transparent analysis of commercially available public cloud services.

Check out CloudHarmony and their directory of services at cloudharmony.com.