Netflix exec: The master copy of our data is in the cloud

Grazed from GigaOM. Author: Janko Roettgers.

If there’s one key lesson startups in the consumer web space can learn from the big guys, it’s this: Don’t invest in in-house servers. That was one of the key take-aways from the guru panel at GigaOM’s Structure conference in San Francisco, where Netflix’s director of architecture Adrian Cockcroft confided: “The master copy of our data is now stored in the cloud.”

Dropbox’s VP of engineering Aditya Agarwal agreed, and gave founders this bit of advice: “If you are a startup, you’d be crazy if you built your own infrastructure.” He added that using the cloud instead of homegrown infrastructure wasn’t just about costs, but also about giving your own engineers the flexibility they need. “If you build it in-house, it should be easier to use than AWS,” he said…

Cockcroft said that Amazon’s web services have given Netflix much more flexibility than in-house infrastructure ever could. “Any develper can click a button and make hundreds of machines appear,” he said. But he added that money is of course an issue as well, especially for a company with Netflix’s scale. “I could spend $100 million on a data center, or I could buy an awful lot of movies with that.”

Cockcroft also shared some more insights in how Netflix is using AWS to rapidly expand internationally. For its launch in Europe, it made use of Amazon’s servers in Ireland, which only took the company a few weeks. “And then you start paying after a month,” he said with a smirk. Building its own data center on the continent would have taken nine months and reqired huge upfront investments, he argued.

The lone dissenter on Thursday’s panel was Alexei Rodriguez, VP of operations for Evernote, which is heavily relying on in-house infrastructure for its own operations. Rodriguez said that the main reason for this was data security and privacy. AWS is a little bit of a black box for data, he argued, because there is no way to control whether Amazon is following its own policies. Still, even Rodriguez agreed that a startup in the consumer web space should start with AWS these days, but then move data in-house eventualy “There is a certain tipping point,” he said.

Cockroft agreed in theory, but said that this tipping point was very high. Zynga with its hundreds of thousands of machines would be too big even for AWS, he argued, but a company like Netflix that uses just tens of thousands of machines doesn’t have to worry. Addressing privacy and security concerns, he said that Netflix uses multiple levels of encryption and a number of backup copies on separate S3 accounts as well as outside of the realm of Amazon.