Detecting tokens (words) and sentences using OpenNLP
The previous two recipes in this chapter detected tokens (words) and sentences using legacy Java classes and methods in them. In this recipe, we will combine the two tasks of detecting tokens and sentences with an open-source library of Apache named OpenNLP. The reason for introducing OpenNLP with these two tasks that can be accomplished well with the legacy methods is to introduce data scientists to a tool that is really handy and has very high accuracy in several information retrieval tasks on standard and classic corpora. The homepage for OpenNLP can be found at https://opennlp.apache.org/. One strong argument of using this library for tokenization, sentence segmentation, part-of-speech tagging, named entity recognition, chunking, parsing, and co-reference resolution is that you can have your own classifier trained on your corpora of articles or documents.
Getting ready
At the time of writing this book, the 1.6.0 version was the latest...
