"Import pyonmttok" failed & "pip install pyonmttok" faileld - Windows 10

My question is “Could you help me to make “pip install pyonmttok” work on Windows 10, please…”. Below is the situation:

I am trying to run opennmt-tf on Windows 10, and I want to use tokenization on the fly.
However, import pyonmttok in opennmt/tokenizers/tokenizer.py and opennmt/tokenizers/opennmt_tokenizer.py failed.

I tried “pip install pyonmttok”, and it failed. So I found these include files, but eventually it still faied, because there is no opennmttokenizer_export.h for the line #include “onmt/opennmttokenizer_export.h” in those files like “SubwordEncoder.h”.

After using CMake, I can obtain onmt/opennmttokenizer_export.h. But when I tried “pip install pyonmttok” again, it showed:

LINK : fatal error LNK1181: cannot open input file ‘boost_python36.lib’.

I can’t find boost_python36.lib, so I tried to copy a python36.lib and rename it as boost_python36.lib (I know it is not working this way, but I just wonder what happens), it shows

LINK : fatal error LNK1181: cannot open input file ‘OpenNMTTokenizer.lib’

Sorry, we don’t publish the pyonmttok package for Windows and there are currently no plans to do that.

I recommend that you tokenize your data separately. If you still want to apply tokenization on the fly, you should be able to build the Python wheel yourself using this script as a reference:

(or move to a Linux system :slight_smile:)