Training a neural network tokenizer for specialized text

Sometimes, we need to work with specialized text, such as an uncommon language or text that is unique to a problem domain. In such cases, the standard tokenizers are not always sufficient. This necessitates the creation of a unique model that will work better with the specialized text. In this recipe, we will demonstrate how to train a model using OpenNLP.

Getting ready

To prepare, we need to do the following:

1. Create a new Maven project
2. Add the following dependency to the POM file:

<dependency>
    <groupId>org.apache.opennlp</groupId>
    <artifactId>opennlp-tools</artifactId>
    <version>1.9.0</version>
</dependency>

How to do it...

Let's go through the following steps:

1. Create a file called training-data.train. Add the following to the file:

The first sentence is terminated by a period<SPLIT>. We will want to be able to identify tokens that are separated by something other than whitespace<SPLIT>. This can include commas<SPLIT>, numbers such as 100.204<SPLIT>, and other punctuation characters including colons:<SPLIT>.

2. Next, add the following imports to the program:

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import opennlp.tools.tokenize.TokenSample;
import opennlp.tools.tokenize.TokenSampleStream;
import opennlp.tools.tokenize.Tokenizer;
import opennlp.tools.tokenize.TokenizerFactory;
import opennlp.tools.tokenize.TokenizerME;
import opennlp.tools.tokenize.TokenizerModel;
import opennlp.tools.util.InputStreamFactory;
import opennlp.tools.util.ObjectStream;
import opennlp.tools.util.PlainTextByLineStream;
import opennlp.tools.util.TrainingParameters;

3. Next, add the following try block to the project's main method that contains the code needed to obtain the training data:

InputStreamFactory inputStreamFactory = new InputStreamFactory() {
    public InputStream createInputStream() 
            throws FileNotFoundException {
        return new FileInputStream(
            "C:/NLP Cookbook/Code/chapter2a/training-data.train");
    }
};

4. Insert the following code segment into the try block that will train the model and save it:

try (
    ObjectStream<String> stringObjectStream = 
        new PlainTextByLineStream(inputStreamFactory, "UTF-8");
    ObjectStream<TokenSample> tokenSampleStream = 
        new TokenSampleStream(stringObjectStream);) {
  
    TokenizerModel tokenizerModel = TokenizerME.train(
        tokenSampleStream, new TokenizerFactory(
            "en", null, true, null), 
            TrainingParameters.defaultParams());
    BufferedOutputStream modelOutputStream = 
        new BufferedOutputStream(new FileOutputStream(
            new File(
                "C:/NLP Cookbook/Code/chapter2a/mymodel.bin")));
    tokenizerModel.serialize(modelOutputStream);
} catch (IOException ex) {
    // Handle exception
}

5. To test the new model, we will reuse the code found in the Tokenization using OpenNLP recipe. Add the following code after the preceding try block:

String sampleText = "In addition, the rook was moved too far to be effective.";
try (InputStream modelInputStream = new FileInputStream(
        new File("C:/Downloads/OpenNLP/Models", "mymodel.bin"));) {
    TokenizerModel tokenizerModel = 
        new TokenizerModel(modelInputStream);
    Tokenizer tokenizer = new TokenizerME(tokenizerModel);
    String tokenList[] = tokenizer.tokenize(sampleText);
    for (String token : tokenList) {
        System.out.println(token);
    }
} catch (FileNotFoundException e) {
    // Handle exception
} catch (IOException e) {
    // Handle exception
}

6. When executing the program, you will get an output similar to the following. Some of the training model output has been removed to save space:

Indexing events with TwoPass using cutoff of 5

Computing event counts... done. 36 events
 Indexing... done.
Sorting and merging events... done. Reduced 36 events to 12.
Done indexing in 0.21 s.
Incorporating indexed data for training... 
done.
 Number of Event Tokens: 12
 Number of Outcomes: 2
 Number of Predicates: 9
...done.
Computing model parameters ...
Performing 100 iterations.
 1: ... loglikelihood=-24.95329850015802 0.8611111111111112
 2: ... loglikelihood=-14.200654164477221 0.8611111111111112
 3: ... loglikelihood=-11.526745527757855 0.8611111111111112
 4: ... loglikelihood=-9.984657035211438 0.8888888888888888
...
 97: ... loglikelihood=-0.7805227945549726 1.0
 98: ... loglikelihood=-0.7730211829010772 1.0
 99: ... loglikelihood=-0.765664507836384 1.0
100: ... loglikelihood=-0.7584485899716518 1.0
In
addition
,
the
rook
was
moved
too
far
to
be
effective
.

How it works...

To understand how this all works, we will explain the training code, the testing code, and the output. We will start with the training code.

To create a model, we need test data that was saved in the training-data.train file. Its contents are as follows:

These fields are used to provide further information about how tokens should be identified<SPLIT>.  They can help identify breaks between numbers<SPLIT>, such as 23.6<SPLIT>, punctuation characters such as commas<SPLIT>.

The <SPLIT> markup has been added just before the places where the tokenizer should split code, in locations rather than white spaces. Normally, we would use a larger set of data to obtain a better model. For our purposes, this file will work.

We created an instance of the InputStreamFactory to represent the training data file, as shown in the following code:

InputStreamFactory inputStreamFactory = new InputStreamFactory() {
    public InputStream createInputStream() 
            throws FileNotFoundException {
        return new FileInputStream("training-data.train");
    }
};

An object stream is created in the try block that read from the file. The PlainTextByLineStream class processes plain text line by line. This stream was then used to create another input stream of TokenSample objects, providing a usable form for training the model, as shown in the following code:

try (
    ObjectStream<String> stringObjectStream = 
        new PlainTextByLineStream(inputStreamFactory, "UTF-8");
    ObjectStream<TokenSample> tokenSampleStream = 
        new TokenSampleStream(stringObjectStream);) {
    ...
} catch (IOException ex) {
    // Handle exception
}

The train method performed the training. It takes the token stream, a TokenizerFactory instance, and a set of training parameters. The TokenizerFactory instance provides the basic tokenizer. Its arguments include the language used and other factors, such as an abbreviation dictionary. In this example, English is the language, and the other arguments are not used. We used the default set of training parameters, as shown in the following code:

TokenizerModel tokenizerModel = TokenizerME.train(
    tokenSampleStream, new TokenizerFactory("en", null, true, null), 
    TrainingParameters.defaultParams());

Once the model was trained, we saved it to the mymodel.bin file using the serialize method:

BufferedOutputStream modelOutputStream = new BufferedOutputStream(
    new FileOutputStream(new File("mymodel.bin")));
tokenizerModel.serialize(modelOutputStream);

To test the model, we reused the tokenization code found in the Tokenization using the OpenNLP recipe. You can refer to that recipe for an explanation of the code.

The output of the preceding code displays various statistics, such as the number of passes and iterations performed. One token was displayed per line, as shown in the following code. Note that the comma and period are treated as separate tokens using this model:

In
addition
,
the
rook
was
moved
too
far
to
be
effective
.

There's more...

The training process can be tailored using training parameters. Details of how to use these parameters are hard to find; however, cut-off and iteration are described at: https://stackoverflow.com/questions/30238014/what-is-the-meaning-of-cut-off-and-iteration-for-trainings-in-opennlp.

See also

• The OpenNLP API can be found at: https://opennlp.apache.org/docs/1.9.0/apidocs/opennlp-tools/index.html
• See the Tokenization using OpenNLP recipe for an explanation of how the model is tested