The sentencepiece has a byte_fallback parameter for byte representation of tokens outside the vocabulary, giving translation stability for these tokens. I would like to know if it is possible to use this parameter with this tokeniser:
If not, perhaps you can tell me how it would be most correctly implemented through your library?
The way sentencepiece does this is they just add every byte token as a token to the tokenizer’s vocabulary, and if there is no other suitable token that fits a character, itll be split up into the individual byte tokens.
You can’t enable it like an option on a tokenizer that hasn’t been trained with it on, you need to train it using sentencepiece yourself.