Understanding SBD rules of LingPipe's HeuristicSentenceModel class
There are other rules that can be used to perform SBD. LingPipe's HeuristicSentenceModel class uses a series of token rules to perform SBD. We will present them here, as they provide insight into what rules can be useful.
This class uses three sets of tokens and two flags to assist in the process:
- Possible Stops: This is a set of tokens that can be the last token of a sentence
- Impossible Penultimates: These tokens cannot be the second to last token in a sentence
- Impossible Starts: This is a set of tokens that cannot be used to start a sentence
- Balance Parentheses: This flag indicates that a sentence should not be terminated until all matching parentheses are matched in that sentence
- Force Final Boundary: This specifies that the final token in an input stream should be treated as a sentence terminator even if it is not a possible stop
Balance parentheses include () and []. However, this rule will fail if the text is malformed...
