Alluxio Accelerates Scale, Introduces Data Orchestration Console to Advance Hybrid and Multi-cloud Data Orchestration

Alluxio, the developer of open source cloud data orchestration software, announced the immediate availability of the next major release of its Data Orchestration platform featuring an expanded metadata service, a new management console for hybrid and multi-cloud deployments, and more cloud native deployments. Data platform teams can now save on infrastructure and operational costs, while easily managing data access across multiple environments.

Enterprises leverage Alluxio at enormous scale, both in data size and number of files.  Data orchestration decouples compute from the location of data to optimize for which data resides where and for how long without the management overhead. With this announcement, users will be able to manage namespaces with billions of files without relying on third party systems, greatly reducing the overall deployment footprint of the solution. The new management console will make it easy to connect an analytics cluster, with engines such as Presto and Spark, with data sources across multiple clouds, single cloud or on-premises using Alluxio.

“Organizations have adopted an infrastructure with compute engines and data sets spread across private data centers and public clouds for business agility and cost effectiveness. Our customers have turned to Alluxio to bridge the gap between applications and the storage systems spread across regions and cloud providers,” said Haoyuan Li, founder and CEO, Alluxio.

Alluxio 2.4 Community and Enterprise Edition features new capabilities, including:

Expanded Metadata Service – At the core of the Alluxio Data Orchestration Platform is a metadata service, a scalable, distributed data service for management across multiple sources like traditional Hadoop-based data lakes on-premises or modern cloud-based data lakes. Leveraged to unify data lakes at enormous scale, both in data size and number of files, Alluxio has expanded this service to provide support for billions of files while removing third-party system dependencies. Breaking away from dependencies on traditional Hadoop components, Alluxio has bolstered support for cloud native and container-based deployments. The lifecycle management of the Alluxio’s metadata service now also supports automatic backups without impacting the live system to further reduce the platform management overhead.

New management console – The Alluxio Data Orchestration Hub is a new web-based management console that makes it easy to connect an analytics or machine learning clusters with multiple data sources to unify data lakes. The new service provides an easy-to-use unified management view for configuration and monitoring.

Cloud native deployment – Spawning analytics clusters in AWS and GCP is now easier than ever. Based on Terraform, Alluxio now makes it easy to launch pre-configured clusters programmatically using a single command. Alluxio has been featured as a recommended data lake partner for data lake modernization solution with Google Cloud, including the ability to launch an Alluxio-enabled cluster using the Dataproc component exchange console.

Sensitive data management – Alluxio now integrates with Vault for secure management of sensitive information for data access with dynamic infrastructure across multiple clouds and on-premises environments. With the shift from on-premises infrastructure to multiple cloud-providers, protection of data access tokens and credentials is more important than ever.

Simplified DevOps and system monitoring – Alluxio 2.4 adds several system enhancements to simplify and improve cluster management and maintenance. The system provides an aggregated cluster view of key performance metrics like I/O throughput and metadata request rate through the UI and programmatic monitoring endpoints. Internal monitoring for failures and system slow-downs has been added, further improving the operator view of the health and performance of the system.

Support for Java 11 – Java 11 is the latest long term support version of Java. Alluxio 2.4 provides compatibility with Java 11 while maintaining support for Java 8. Users looking to move their compute engines or Alluxio systems to Java 11 can now do so without any concerns.Availability

Alluxio 2.4 Community and Enterprise Edition are generally available for download here: https://www.alluxio.io/download/