How to define tokenizer in the configuration of opennmt-tf. As before training the model i am using the bytepair encoding scripts for creating the vocab and the training data . Do i have to explicitly define the bpe technique in the configuration setting of opennmt-tf .
If you already tokenized the data, there is no need to configure any tokenization settings in OpenNMT-tf.
1 Like