"Import pyonmttok" failed & "pip install pyonmttok" faileld - Windows 10

chiapas · December 8, 2018, 2:36am

My question is “Could you help me to make “pip install pyonmttok” work on Windows 10, please…”. Below is the situation:

I am trying to run opennmt-tf on Windows 10, and I want to use tokenization on the fly.
However, import pyonmttok in opennmt/tokenizers/tokenizer.py and opennmt/tokenizers/opennmt_tokenizer.py failed.

I tried “pip install pyonmttok”, and it failed. So I found these include files, but eventually it still faied, because there is no opennmttokenizer_export.h for the line #include “onmt/opennmttokenizer_export.h” in those files like “SubwordEncoder.h”.

After using CMake, I can obtain onmt/opennmttokenizer_export.h. But when I tried “pip install pyonmttok” again, it showed:

LINK : fatal error LNK1181: cannot open input file ‘boost_python36.lib’.

I can’t find boost_python36.lib, so I tried to copy a python36.lib and rename it as boost_python36.lib (I know it is not working this way, but I just wonder what happens), it shows

LINK : fatal error LNK1181: cannot open input file ‘OpenNMTTokenizer.lib’

guillaumekln · December 8, 2018, 9:05am

Sorry, we don’t publish the pyonmttok package for Windows and there are currently no plans to do that.

I recommend that you tokenize your data separately. If you still want to apply tokenization on the fly, you should be able to build the Python wheel yourself using this script as a reference:

github.com

OpenNMT/Tokenizer/blob/master/bindings/python/tools/build_wheel.sh

#! /bin/bash

# Script to build Python wheels in the manylinux1 Docker environment.

# docker pull quay.io/pypa/manylinux1_x86_64
# mkdir docker
# docker run --rm -ti -v $PWD/docker:/docker -w /docker quay.io/pypa/manylinux1_x86_64 bash
# curl -L -O https://raw.githubusercontent.com/OpenNMT/Tokenizer/master/bindings/python/tools/build_wheel.sh
# bash build_wheel.sh v1.3.0
# twine upload wheelhouse/*.whl

set -e
set -x

ROOT_DIR=$PWD

TOKENIZER_BRANCH=${1:-master}
TOKENIZER_REMOTE=https://github.com/OpenNMT/Tokenizer.git

BOOST_VERSION=1.67.0

This file has been truncated. show original

(or move to a Linux system )