New Survey Shows Enterprises Adopting Kubernetes for Big Data Cloud Migration, Cost Reduction

Pepperdata announced the findings from a survey of enterprise IT professionals to understand how companies are meeting their big data applications needs through the use of Kubernetes. The survey was conducted in March 2021, among 800 participants from a range of industries, 72 percent of which worked at companies with between 500 and 5000 employees. 

Kubernetes is rapidly becoming the standard for cloud and on-premises clusters. It is a complex technology, and companies are struggling to properly and effectively implement and manage these environments. The complexity of big data applications makes resource optimization a real challenge. Unsurprisingly, when IT doesn’t have granular visibility into big data Kubernetes performance, optimized performance and spend are hard to achieve.  

“Kubernetes is increasingly being adopted by our customers for big data applications. As a result, we see customers experiencing performance challenges,” said Ash Munshi, CEO, Pepperdata. “This survey clearly indicates that these problems are universal and there is a need to better optimize these big data workloads.”  

Survey data collected reveals a number of insights into how businesses are adopting Kubernetes for big data applications:

• When asked what their goals were for adopting Kubernetes for big data workloads, 30 percent said to “improve resource utilization for reduced cloud costs.” 23 percent want to enable their migration to the cloud; 18 percent said to shorten deployment cycles; 15 percent wanted to make their platforms and applications cloud-agnostic; and 14 percent said to containerize monolithic apps.
• Porting hundreds or thousands of apps over to Kubernetes can be challenging, and the biggest hurdles for survey respondents included initial deployment, followed by migration, monitoring and alerting, complexity and increased cost, and reliability, in that order.
• The kinds of applications and workloads respondents are running, in order of most to least, include Spark, 30 percent; Kafka, 25 percent; Presto 23 percent; AI/deep learning workloads using PyTorch or Tensorflow at 18 percent; and “other” at five percent.
• Surprisingly, and despite how much the media writes about the move to public cloud, this survey found that 47 percent of respondents are using Kubernetes in private cloud environments. On-premises use made up 35 percent, and just 18 percent of respondents said they were using Kubernetes containers in public cloud environments.
• 45% of Kubernetes workloads are in development and testing environments, as users move production workloads into a new resource management framework. 30 percent are doing proof-of-concept work.
• 66 percent of respondents said 75-100 percent of their big data workloads will be on Kubernetes by the end of 2021.
• IT operations was the clear leader-at 80 percent-in deploying Spark and other big data apps built on Kubernetes; Engineering followed with 11 percent; with business unit developers at just nine percent.

View the full Pepperdata Big Data and Kubernetes report