Convert SentencePiece .vocab format to OpenNMT-py .onmt_vocab format

[HURIMOZ (Huri Translations)]

Hi, Iʻm looking for a Python script to convert SentencePiece .vocab files to OpenNMT-py .onmt_vocab format. SentencePiece vocab files have negative values and we need positive values for the frequency of the words when using OpenNMT-py.

[ymoslem (Yasmin Moslem)]

Hello!

Does this work?

Steps to convert SentencePiece vocab to OpenNMT-py vocab

If we are going to build vocabulary from scratch for OpenNMT-py, we should use something like this: # -config: path to your config.yaml file # -n_sample: use -1 to build vocabulary on all the segment in the training dataset # -num_threads: change it to match the number of CPUs to run it faster onmt_build_vocab -config config.yaml -n_sample -1 -num_threads 2 However, many of us use SentencePiece for sub-wording, which generates both a sub-wording model and a vocabulary list. We cannot use this…

[HURIMOZ (Huri Translations)]

Yes that works! Thank you so much!