OpenNMT on Python in Windows: Not working

tensorflow

(Loek van Kooten) #1

Hi all, I managed to get OpenNMT working on a virtual Ubuntu machine, but now I’d like to leverage my GPU’s (SLI) as 40 hours of training time is stretching things a bit.

I’ve got Python working on Windows and even got Tensorflow installed and working. My last obstacle is installing OpenNMT on Python.

pip install URL
or
pip install opennmt-tf

does not work. Please see the attached screenshot.

How exactly do I go about this? I need to make sure the package is installed in Python 3.6. Please be precise: I’m a total noob when it comes to non-Windows stuff and the path names and directory chaos are confusing me.

Thank you very much in advance!


(Guillaume Klein) #2

Hello,

You should install Git for Windows in order to retrieve OpenNMT-tf. It is not available via pip for now.


(Loek van Kooten) #3

I tried that before. It does seem to install opennmt, but all I see in the Python27/Lib/site-packages/opennmt directory is six files (__init, config and constants.py/pyc). There’s no bin/main or anything similar. One thing that worries me: is this the right location, as this needs to run on Python 36 and not 27? And where have all the files gone?

Sorry for being a noob. I’m trying really hard.


(Guillaume Klein) #4

Could you try the following:

  1. Open Git Bash
  2. Type:
git clone https://github.com/OpenNMT/OpenNMT-tf.git
cd OpenNMT-tf
python -m bin.main -h

Right now, OpenNMT-tf expects the user to use it from its own directory.


(Loek van Kooten) #5

One step closer! But it seems I need to choose between running at the Python prompt or the Windows prompt and can’t have my cake and eat it. Please see the attached screenshot. As you can see, Python is there.


(Loek van Kooten) #6

P.S. Tensorflow is there when I use import tensorflow as tf in the Python prompt.


(Loek van Kooten) #7

So my guess is that I need to use import opennmt as nmt or something like that in the Python prompt. From there, how do I get access to commands like train?


(Guillaume Klein) #8

So python is 2.7 but py is 3.6. This is confusing.

As you installed TensorFlow for Python 3, you should:

  1. Install pyyaml for Python 3 as well
  2. Invoke OpenNMT-tf scripts with py

(Loek van Kooten) #9

So Python finds main.py, but main.py has issues finding other stuff.


(Guillaume Klein) #10

Use:

py -m bin.main -h

This syntax is important as it adds the current directory to the Python paths.


(Loek van Kooten) #11

Thank you. But then it has issues finding yaml, which is installed.

(I feel for the poor translator - who mostly has not background in programming or whatsoever) who ever needs to get this running! :wink:


(Guillaume Klein) #12

You have conflicting Python versions.

You installed TensorFlow for Python 3 and PyYAML for Python 2. You should select one version and install all required packages for that version.


(Loek van Kooten) #13

Ah, fantastic. Python 3.6 for Windows installs 2.7 with it by default, which is indeed very confusing. I just killed the entire 2.7 directory, then installed YAML via Git for Windows and voila, main is responding! I’ll keep on playing with this. Thank you very much for your help so far!


(Loek van Kooten) #14

It seems the sample files have codes in them that are not allowed?


(Loek van Kooten) #15

Yes, no matter what file I feed, I always get UnicodeDecodeError: ‘cp932’ codec can’t decode byte 0xef in position xxxx: illegal multibyte sequence during the train phase.


(Loek van Kooten) #16

I can add sitecustomize.py to Python36/Lib stating:

import sys
import codecs
sys.stdout = codecs.getwriter(‘utf8’)(sys.stdout)
sys.stderr = codecs.getwriter(‘utf8’)(sys.stderr)

Now the program does run, but it quits without result, error or whatsoever. It starts, and stops. All I see is my prompt again.


(Guillaume Klein) #17

What command did you run?


(Loek2) #18

Answering from another account as my first has reached its posting limit for the first day after registration.

python -m bin.main train --model config/models/nmt_small.py --config config/opennmt-defaults.yml config/data/xxx.yml

Whereby xxx is the name of the yaml file in question (it doesn’t work for the test files or my own files): both show the same symptom.


(Loek2) #19

Should I conclude that OpenNMT does not run on Windows?


(Guillaume Klein) #20

To be clear, Windows support is not a priority as people are massively using Linux systems for this kind of application. But if TensorFlow is correctly working on Windows, it would be nice that OpenNMT-tf works there as well.

Could you open an issue on GitHub? I will try to debug that when I have my hands on a Windows system. Thanks.