OpenNMT Forum

How to define the tokenization technique in opennmt-tf

How to define tokenizer in the configuration of opennmt-tf. As before training the model i am using the bytepair encoding scripts for creating the vocab and the training data . Do i have to explicitly define the bpe technique in the configuration setting of opennmt-tf .

If you already tokenized the data, there is no need to configure any tokenization settings in OpenNMT-tf.