Microsoft open-sources Project Zipline, its data compression algorithm and hardware for the cloud

Microsoft announced that it is open-sourcing its new cutting-edge compression technology, called Project Zipline, yesterday. As a part of this open-source release, Project Zipline compression algorithms, hardware design specifications, and Verilog source code for register transfer language (RTL) has been made available.

Apart from the announcement of Project Zipline, the Open Compute Project (OCP) Global Summit 2019 also started yesterday in San Jose. In the summit, the latest innovations that can make hardware more efficient, flexible, and scalable are shared. Microsoft states that its journey with OCP began in 2014 when it joined the foundation and contributed the server and data center designs that power its global Azure Cloud. Moreover, Microsoft contributes innovations to OCP every year at the summit. Microsoft has again decided to contribute Project Zipline this year.

“This contribution will provide collateral for integration into a variety of silicon components across the industry for this new high-performance compression standard. Contributing RTL at this level of detail as open source to OCP is industry leading”, states Microsoft team.

Project Zipline is aimed to optimize the hardware implementation for different types of data on cloud storage workloads. Microsoft has been able to achieve higher compression ratios, higher throughput, and lower latency than the other algorithms currently available. This allows for compression without compromise as well as data processing for different industry usage models (from cloud to edge).

Microsoft’s Project Zipline compression algorithm produces great results with up to 2X high compression ratios as compared to the commonly used Zlib-L4 64KB model. These enhancements, in turn, produce direct customer benefits for cost savings and allow access to petabytes or exabytes of capacity in a cost-effective way for the customers.

Project Zipline has also been optimized for a large variety of datasets, and Microsoft’s release of RTL allows hardware vendors to use the reference design that offers the highest compression, lowest cost, and lowest power in an algorithm. Project Zipline is available to the OCP ecosystem, so vendors can contribute further to benefit Azure and other customers.

Microsoft team states that this contribution towards open source will set a “new precedent for driving frictionless collaboration in the OCP ecosystem for new technologies and opening the doors for hardware innovation at the silicon level”.

In the future, Microsoft expects Project Zipline compression technology to enter different market segments and usage models such as network data processing, smart SSDs, archival systems, cloud appliances, general purpose microprocessor, IoT, and edge devices.

For more information, check out the official Microsoft announcement.