Cloud challenge: right data in the right place

Remember grid computing? An important part of that idea was to use processing power where it was available, and one of the challenges was figuring out ways of getting often large amounts of data to where it was to be processed.

Fast-forward a few years, and we’re all talking about cloud computing. Whether that’s public, private or hybrid, that same issue crops up: if you’re not doing all the processing in one place, how do you get the data where it’s needed, and preferably before it’s needed?

After all, one of the promises of cloud is the idea of responding to peak loads by ‘bursting’ out of your own data centre into one operated by a service provider.

"For the first time, we can see the potential for load balancing between data centres," Chuck Hollis, global VP of storage vendor EMC, told iTWire.

Typically, 10% of a given set of data gets used and the other 90% doesn’t. "The trick is to make sure that the vast majority of the data is there [before it is required]," he said. Blindly copying data between data centres is clearly inefficient.

It turns out that the problem has a lot in common with managing tiered storage, where the idea is to ensure that the data most likely to be used is resident on the fastest drives (which these days usually means flash storage), with slower and cheaper drives used for the rest – and EMC has 15 years of experience with the algorithms needed to make that work.

Those algorithms can also be used to determine which pieces of data should be moved to a remote location first. The main difference is that while EMC’s storage products typically internally promote or relegate data in chunks of around 7.5MB, the latency and restricted bandwidth of long-haul links makes a 4KB chunk more appropriate. As latency drops, the chunk size can be increased.

Mr Hollis said the technology was originally developed by governments, as the large amounts of data collected by satellites needed to be made available in multiple locations simultaneously.

While it is not yet in commercial use, EMC customers – notably financial services firms and service providers such as telcos – are conducting pilot tests.