Precomputing and caching

While processing data, we usually come across instances where few common computation steps precede several kinds of subsequent steps. That is to say, some amount of computation is common and the remaining is different. For high-latency common computations (I/O to access the data and memory/CPU to process it), it makes a lot of sense to compute them once and store in digest form, such that the subsequent steps can simply use the digest data and proceed from that point onward, thus resulting in reduced overall latency. This is also known as staging of semi-computed data and is a common technique to optimize processing of non-trivial data.

Clojure has decent support for caching. The built-in clojure.core/memoize function performs basic caching of computed results with no flexibility in using specific caching strategies and pluggable backends. The Clojure contrib library core.memoize offsets the lack of flexibility in memoize by providing several configuration options...