Extracting the data

After you run the preceding script, the articles will be unzipped to the directory data/reuters-sgml. Each .sgm file in the extract contains around 1,000 short articles that have been wrapped in XML-style tags using Standard Generalized Markup Language (SGML). Rather than write our own parser for the format, we can make use of the one already written in the Lucene text indexer.

(:import [org.apache.lucene.benchmark.utils ExtractReuters])

(defn sgml->txt [in-path out-path]
  (let [in-file  (clojure.java.io/file in-path)
        out-file (clojure.java.io/file out-path)]
    (.extract (ExtractReuters. in-file out-file))))

Here we're making use of Clojure's Java interop to simply call the extract method on Lucene's ExtractReuters class. Each article is extracted as its own text file.

This code can be run by executing:

lein extract-reuters

on the command line within the project directory. The output will be a new directory, data/reuters-text, containing over 20,000 individual...