Debian maintainer points out difficulties in Deep Learning Framework Packaging

A Debian maintainer, named, Mo Zhou, wrote a mail to Development of Debian or Debian Devel team (responsible for discussion over tech development topics) on Debian mailing list, stating difficulties in deep learning framework Packaging.

Zhou re-evaluated the status of TensorFlow’s latest build systems and shared point related to deep learning framework packaging. “My thoughts are concluded from failures instead of success. That said, they should be helpful to future maintainers who'd like to maintain similar packages”, writes Zhou.

Zhou elaborates on three obstacles faced by maintainers in Debian's context in case of License, ISA Baseline, and Build System.

License

Zhou states that although the de facto dominating performance library is cuDNN, no user would prefer using a D-L framework without cuDNN or TPU acceleration. He states that packaging for cuDNN is available under Salsa:nvidia-team, however, the plan to upload it had been aborted since its license looks “too scary”.

ISA Baseline

Zhou writes that the absence of SIMD code affects critical computational performance. There have been certain helpful suggestions made by other volunteers including ld.so tricks and some gcc features that enables run-time code selection as per the CPU capability. The ld.so tricks help to bloat the resulting .deb packages but it's the most applicable solution. On the other hand, patching a million lines of Tensorflow code that would enable the "function attributes" feature is very difficult and “impossible” to a volunteer.

Build System

Zhou states that the build systems of TensorFlow and PyTorch are volatile due to the fast pace of development, especially TensorFlow's build system "bazel" is very hard to package for Debian. Also, a good amount of patching work is required to prevent bazel from downloading ~3.0GiB of before building TensorFlow. Additionally, PyTorch's setup.py+cmake+shell build system also requires some patching work.

Zhou writes that any future contributor who is about to deal with any deep learning packages to carefully assess the three factors listed above. Apart from that, Zhou has also filed Orphan bugs against tensorflow and several of its dependencies, except src:nsync that contains cmake files. Zhou also mentions that DUPR is the best choice for him in case of .deb packages.

For more information, check out the official Debian mailing list.

Debian project leader elections goes without nominations. What now?

It is supposedly possible to increase reproducibility from 54% to 90% in Debian Buster!

Debian 9.7 released with fix for RCE flaw