I have tried several ways to tokenize Chinese content during preprocess. However, because I used space to tokenize Chinese content, my translated file has spaces. I’d like to remove those inserted space during tokenization.
I didn’t find sufficient information on how to use detokenize.lua.
Does anyone have any solution?