Four trends that shaped cloud computing in 2011

Amazon’s cloud outages in April and August were certainly disruptive to many and raised doubts about the stability and scalability of a service possibly expanding too quickly. But the lesson to be drawn was not to rewind the clock and return to hosting your own infrastructure. Rather, it was to distribute your infrastructure across multiple locations and plan for failures. Enterprise IT departments make considerable investments outfitting secondary datacenters and maintaining disaster recovery plans that rarely work as planned. In contrast, many of the startups that diversified geographically with AWS availability zones recovered quickly or avoided service interruptions altogether because they had become accustomed to spinning up new instances of their applications quickly, or because they built their applications with robust levels of redundancy that would have been a stretch to achieve on their own.

2. Private clouds grew and alternatives to VMware matured

Server virtualization remains a core building block for clouds. According to industry analysts, some 40-50% of all workloads running on x86 servers will be virtualized by the end of 2011. For enterprises building and maintaining their own private clouds, VMware delivered a number of compelling enhancements to the vSphere 5 platform, such as “monster” virtual machines (up to 32 virtual CPUs & 1TB of RAM), storage DRS, and a virtual storage appliance. The VMware vCloud Director ecosystem also received a boost from new features and partners, with 25 different datacenters across 13 countries slated to be powered by vCloud by the end of the year.

However, smaller players and new open source “cloud operating system” offerings are finding open doors amongst enterprises considering alternatives to VMware’s premium features and expanding the overall market. For instance, OpenStack (the NASA and Rackspace-backed open source cloud platform), gained significant traction, celebrating its first birthday in July and now claiming more than 130 participants and contributors. Rackspace has done a nice job building an ecosystem around OpenStack and is driving a wedge into the market with its Cloudbuilders and RackConnect initiatives, which enable enterprises to get private instances of an OpenStack cloud and augment them with resources from Rackspace’s public cloud. Citrix acquired Cloud.com to boost its product offerings for service providers and is surely gaining valuable insights from marquee customers creating their own clouds such as Zynga. Startups like Eucalyptus and Nimbula also continued to win customers by addressing hybrid use cases that combine some of the benefits of private and public clouds.

3. Cloud productivity apps became good enough

Cloud-based productivity applications like Google Docs have been around for a few years, but in 2011 they went mainstream and became credible replacements to Microsoft Office for most people. Google rounded out its Docs suite with an updated Presentations program, and the commenting, revision history, and sharing features in Docs are cleverly implemented to make emailing documents back and forth seem archaic. Microsoft responded with Office365, and its licensing plan for small businesses has even replaced the flagship Office desktop software with the new web-based offerings. Other startups such as SlideRocket and Prezi delighted their users by demonstrating innovations of their own.

Of course, many companies still use Office, and it’s not going away anytime soon. But most startups don’t install any software at all, and larger businesses are starting to consider foregoing desktop software upgrades. Many of us are rarely without access to the Internet or a 3G connection, and we’re spending more and more of our day in a browser (or across browsers on different devices). Many common, everyday tasks are now easier and better done on the web, like working on a budget or collaborating on a memo. The powerful combination of utility computing, ‘good enough’ HTML5-based apps, and the proliferation of new devices suggests the demand for cloud applications is set to match the hype and truly takeoff in 2012.

4. Tools for wrangling big data showed that volume of data is only part of the equation

Consumers, enterprises, mobile devices, and sensors in the world today generate more data than ever — so much that an estimated 90% of the world’s data has been created in the last 2 years, and much of it is landing in the cloud. Volume of data is certainly not slowing down, but in the past year we’ve seen a shift in the variety of data and an increase in how fast its generated. Unstructured or loosely structured data is adding to the variety and will likely alter the budgets being allocated to databases and storage systems. Time sensitivity is also increasing, and data must be streamed through applications and enterprises at higher speeds than before to maximize its value.

Given these trends, MapReduce, Hadoop, and “big data” analytics are crossing the chasm from early adopters to more mainstream use cases. Companies across a variety of verticals such as e-commerce, financial services, life sciences, and telecommunications are deploying new tools to pipeline data analysis through their organizations more efficiently. Management tools and user interfaces for the new data processing engines are also making complex queries and analyses accessible to audiences beyond the “data scientists”. The Hadoop World conference last month showcased a thriving ecosystem of vendors and household names such as eBay and JP Morgan Chase using products from those vendors to crunch their data. There’s similar momentum behind MongoDB and other open source NoSQL initiatives, with developers using new tools to build new applications.

In aggregate, developers are downloading map-reduce frameworks and non-relational databases more than 200,000 times per month. The business intelligence companies are deriving from data is becoming an increasingly important competitive advantage, and companies are creating new jobs and paying significant salary premiums to attract and retain the best data scientists — inside and outside the San Francisco Bay Area.

These are just a few of the highlights from a remarkable year in cloud computing. What other trends would you consider to be truly disruptive? As customers and early adopters, what feature updates or new product offerings would you like to see from some of these technology vendors? I’d love to hear your thoughts in the comments below and look forward to meeting many of you at VentureBeat’s CloudBeat 2011 conference this week.