iXsystems and Klara Celebrate Valentine’s Day with a Heartfelt Donation of Fast Dedup to OpenZFS and TrueNAS

iXsystems and Klara reasserted their joint commitment to open source with the contribution of the Fast Dedup feature to the OpenZFS and TrueNAS communities. In a move sponsored and supported by engineers from iXsystems and the development efforts of Klara, the contribution of Fast Dedup significantly enhances sustained deduplication performance for ZFS and TrueNAS storage systems, marking a new milestone in storage efficiency and performance.

Deduplication performance has been limited in ZFS and OpenZFS since the inception of the file system due to its metadata handling. Fast Dedup ameliorates this with the introduction of a re-engineered metadata structure dynamically sized to operate in RAM or on dedicated flash devices for expedited processing. These advancements collectively enable up to 20 times faster performance and 5 times more usable capacity for certain workloads while greatly improving reliability for systems with deduplication enabled. The technology outperforms legacy dedup by employing algorithms and strategies that scan, identify, and eliminate duplicates, while keeping metadata in RAM and SSDs.

“The OpenZFS Community is very excited to receive this major contribution of Fast Dedup software,” said Matt Ahrens, co-creator of ZFS and the OpenZFS Project. “Fast Dedup has been a major user desire for many years and we are pleased to see that Klara and iXsystems have developed the feature in-line with our general requirements. We look forward to testing and integrating the new software.”

Key Fast Dedup Advancements Include:

• Metadata size is now dynamically sized to fit on either RAM or dedicated flash devices to avoid hitting the performance penalty wall.
• Metadata structure completely re-engineered to enable efficient updates using a log append process, greatly improving performance for large updates such as deletions.
• Dedup table now favors dedup-able data and prune blocks that show no dedup potential.

 Metadata improvements combined with properly configured storage, including dedicated metadata flash devices designed to improve sustained dedup performance by an order of magnitude or greater for larger systems.

“Klara is very proud of the speed and functional quality of the Fast Dedup development,” said Allan Jude, FreeBSD Engineering Manager at Klara “We look forward to completing the integration and testing work needed for this to be a standard OpenZFS capability, with no extra cost.”

“Fast Dedup has been a longstanding desired feature of ZFS and TrueNAS, and can deliver 5X the usable capacity and 20X the performance.” said Kris Moore, SVP of iXsystems Engineering. “These attributes will significantly improve the economics of OpenZFS storage relative to Cloud storage and proprietary storage, and our team could not be more ecstatic to see this come to fruition.”

The donation of Fast Dedup technology is a testament to the shared vision and collaboration between iXsystems and Klara, underscoring their dedication to the advancement of open-source storage solutions. Expected in Q2 for TrueNAS SCALE, Fast Dedup will be available as test software, not to be used for production. This will allow validation of functionality and performance prior to a standard product release later in 2024.