potCloud Aims to Change Moore’s Law and Cloud Dynamics

Enomaly has managed to cobble together between 10,000 and 25,000 servers available on any given day for its SpotCloud cloud computing market, according to CTO and Founder Ruven Cohen. Two and half months ago, Enomaly, a provider of software to create compute clouds, announced SpotCloud, a brokerage service that allows Infrastructure-as-a-Service providers a way to sell their excess compute capacity and buyers a way to find smaller regional cloud providers for batch jobs. I chatted with Cohen this morning ahead of the company opening its beta to more potential sellers wanting to list their unused compute capacity on the SpotCloud market.

Cohen said the marketplace can now offer access to servers in more than 40 countries and 100-plus cities, with the total capacity fluctuating between 10,000 and 25,000 servers with about four to six cores each. Cohen says the focus has been on establishing a solid base of sellers before opening the market to a wider audience next month.

So far, the market concept has led to several new realizations for Cohen, with the first being that by having access to so much compute capacity in one place and available on-demand, he thinks such an aggregate market can accelerate Moore’s Law. Instead of thinking of Moore’s Law narrowly (it says that the number of transistors on a chip doubles every 18 months), Cohen is applying it broadly, saying that if one thinks of the law as a mean of measuring compute performance, SpotCloud can double the performance in a few months by adding more sellers. One could argue that Amazon or Rackspace could also double in performance in a few months, but it would cost a lot in terms of buying servers and building out data centers. One could also argue that SpotCloud can also shrink in a few months depending on the whims of sellers and their workloads.

Taking on unused capacity comes with risks and caveats. The first caveat is that all of those cores on offer from sellers aren’t a monolithic group. They’re aggregated across different data centers from different providers, which means the available slices are limited as a practical matter, because one can’t run a giant job across multiple clouds. A corollary to this is that the compute capacity is also located in different data centers with different geographies and quality.

It’s possible that if you need access to 200 servers with a set performance in Europe, SpotCloud may not be able to help if the sellers aren’t there. And if it is able to provide what you need at a price that works, it can’t guarantee those servers will be around to finish the job. Cohen is up front about this, but it’s clearly a market that isn’t designed for all jobs. However, it does present an opportunity for data center operators and for certain companies or researchers in need of computing for a lower rate, or that’s in a certain geographic region.

Cohen explains that one seller is a data center operator that has several oil and gas and entertainment customers. One customer’s contract requires 4,000 servers (of 16-cores each) to be up and running at all times for the customer application, with an additional 4,000 on standby. The seller has to keep those 4,000 machines available for his customer, but now he can also offer those machines on the SpotCloud market and pull them whenever the original customer needs them. Cohen says that most of the sellers on SpotCloud have no interest in running a private or public cloud, they just see an opportunity to make money using the market.

For Cohen, the second realization gleaned from the last two and half months is that there’s a market for pricing information that SpotCloud can provide. He’s working on an API that can show the current SpotCloud prices for cloud computing in various areas of the world on a fairly real-time basis. “It turns out that competition and where you are dictates the price you will pay for service, with downward pricing pressure in the Valley and higher prices in places like Sao Paulo, Brazil, where there’s a growing demand and no competition,” Cohen said. Sure that’s a general function of economics, but having the ability to see those economics at play for a unit of computing is pretty powerful.

That brings us to the third realization. There is no defined unit of computing yet for the market. Customers choose their region and lay out their system needs and price and hope for a match. But it’s possible that one day they can bid on a more universal compute unit (Cohen suggests compute cores per hour as a good metric). As cloud computing evolves, it’s clear that customers and providers are adapting to it in different ways and for their own reasons. So far, SpotCloud seems to be finding a market for those trying to sell compute capacity. Next month, we’ll see how many buyers are interested.