Determining the lexical meaning of a word using OpenNLP

In this recipe, we will use the model we created in the previous recipe to perform lemmatization. We will perform lemmatization on the following sentence:

The girls were leaving the clubhouse for another adventurous afternoon.

In the example, the lemmas for each word in the sentence will be displayed.

Getting ready

To prepare, we need to do the following:

1. Create a new Maven project
2. Add the following dependency to the POM file:

<dependency>
    <groupId>org.apache.opennlp</groupId>
    <artifactId>opennlp-tools</artifactId>
    <version>1.9.0</version>
</dependency>

How to do it...

Let's go through the following steps:

1. Add the following imports to the project:

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import opennlp.tools.lemmatizer.LemmatizerME;
import opennlp.tools.lemmatizer.LemmatizerModel;

2. Add the following try block to the main method. An input stream and model are created, followed by the instantiation of the lemmatization model:

LemmatizerModel lemmatizerModel = null;
try (InputStream modelInputStream = new FileInputStream(
        "C:\\Downloads\\OpenNLP\\en-lemmatizer.bin")) {
    lemmatizerModel = new LemmatizerModel(modelInputStream);
    LemmatizerME lemmatizer = new LemmatizerME(lemmatizerModel);
    …
} catch (FileNotFoundException e) {
    // Handle exception
} catch (IOException e) {
    // Handle exception
}

3. Add the following code to the end of the try block. It sets up arrays holding the words of the sample text and their POS tags. It then performs the lemmatization and displays the results:

String[] tokens = new String[] { 
    "The", "girls", "were", "leaving", "the", 
    "clubhouse", "for", "another", "adventurous", 
    "afternoon", "." };
String[] posTags = new String[] { "DT", "NNS", "VBD", 
    "VBG", "DT", "NN", "IN", "DT", "JJ", "NN", "." };
String[] lemmas = lemmatizer.lemmatize(tokens, posTags);
for (int i = 0; i < tokens.length; i++) {
    System.out.println(tokens[i] + " - " + lemmas[i]);
}

4. Upon executing the program, you will get the following output that displays each word and then its lemma:

The - the
girls - girl
were - be
leaving - leave
the - the
clubhouse - clubhouse
for - for
another - another
adventurous - adventurous
afternoon - afternoon
. - .

How it works...

We performed lemmatization on the sentence The girls were leaving the clubhouse for another adventurous afternoon. A LemmatizerModel was declared and instantiated from the en-lemmatizer.bin file. A try-with-resources block was used to obtained an input stream for the file, as shown in the following code:

LemmatizerModel lemmatizerModel = null;
try (InputStream modelInputStream = new FileInputStream(
        "C:\\Downloads\\OpenNLP\\en-lemmatizer.bin")) {
    lemmatizerModel = new LemmatizerModel(modelInputStream);

Next, the lemmatizer was created using the LemmatizerME class, as shown in the following code:

LemmatizerME lemmatizer = new LemmatizerME(lemmatizerModel);

The following sentence was processed, and is represented as an array of strings. We also need an array of POS tags for the lemmatization process to work. This array was defined in parallel with the sentence array. As we will see in Chapter 4, Detecting POS Using Neural Networks, there are often alternative tags that are possible for a sentence. For this example, we used tags generated by the Cognitive Computation Group's online tool at http://cogcomp.org/page/demo_view/pos:

String[] tokens = new String[] { 
    "The", "girls", "were", "leaving", "the", 
    "clubhouse", "for", "another", "adventurous", 
    "afternoon", "." };
String[] posTags = new String[] { "DT", "NNS", "VBD", 
    "VBG", "DT", "NN", "IN", "DT", "JJ", "NN", "." };

The lemmatization then occurred, where the lemmatize method uses the two arrays to build an array of lemmas for each word in the sentence, as shown in the following code:

String[] lemmas = lemmatizer.lemmatize(tokens, posTags);

The lemmas are then displayed, as shown in the following code:

for (int i = 0; i < tokens.length; i++) {
    System.out.println(tokens[i] + " - " + lemmas[i]);
}

See also

• The OpenNLP API can be found at https://opennlp.apache.org/docs/1.9.0/apidocs/opennlp-tools/index.html
• The Training an OpenNLP lemmatization model recipe shows how the model was trained