When converting my trained checkpoint into a SavedModel, I am getting an error message that
SentencePieceTokenizer is not an acceptable tokenizer. However, I am pretty confident that in the past I did manage to make SentencePiece part of the model’s graph.
Current version: OpenNMT-tf 2.27.1
ValueError: SentencePieceTokenizer is not in list of accepted tokenizers: CharacterTokenizer, OpenNMTTokenizer, SpaceTokenizer
Here is the tokenizers’ parameters in the data section of the config file:
data: source_tokenization: type: SentencePieceTokenizer params: model: /src/xp/model_1/vocabs/src-bpe.sp.40k.model target_tokenization: type: SentencePieceTokenizer params: model: /src/xp/model_1/vocabs/tgt-bpe.sp.40k.model