Does anyone have recommendations for doing sentence boundary detection for Chinese and other languages that don’t use periods?
Most of the sentence boundary detection systems I’ve found have used a strategy similar to “Unsupervised Multilingual Sentence Boundary Detection”. This involves attempting to distinguish between the periods that separate sentences and the periods in acronyms and abbreviations. DeepSegment or something like it seems like it may work even though all of the pretrained models are for European languages with periods. Is sentence boundary something that can be done in a general way for all languages or does it require customization for different punctuation systems.
The thing about most NLP libraries that support sentence tokenization, including NLTK, spaCy, and Stanza, is that they come with gigantic models.
Recently, I came across pySBD, which works out of the box for 22 languages, including several non-Latin languages. It can be combined with Sentence-Splitter to expand support to about 36 languages. I guess verything else can be defaulted to English.
pySBD looks like a good option for a lot of use cases.
Argos Translate currently uses Stanza, which generally works well but is a large dependency. The issues I’ve had with Stanza aren’t as much that the model is large but it requires Pytorch which is ~1GB.
I’ve also experimented with using a CTranslate2 seq2seq model to do sentence boundary detection. For example:
<detect-sentence-boundary> Max walked down to the pond. He came back with
Max walked down to the pond.
I have this working with a separate 100MB CTranslate2 model but to be practical I think I want to use the same CTranslate2 that is being used for translation. If the model is able to recognize the <detect-sentence-boundary> token and change function then it can use all of the understanding it has of the source language to do its own sentence boundary detection.