Processing data with reducers
We will now study a simple example that depicts the use of reducers in efficiently processing large collections. For this example, we will use the iota
library (https://github.com/thebusby/iota) to handle large memory-mapped files. The usage of the iota
library with large files is encouraged as an efficient alternative to using concrete collections. For example, loading the records in a 1 GB TSV file as strings into a Clojure vector would consume over 10 GB of memory due to the inefficient storage of Java strings. The iota
library avoids this by efficiently indexing and caching the contents of a large file, and this is done with much lower amount of memory overhead when compared to using concrete collections.
Note
The following library dependencies are required for the upcoming examples:
[iota "1.1.2"]
Also, the following namespaces must be included in your namespace declaration:
(ns my-namespace (:require [iota :as i] [clojure.string :as...