OpenNMT on Python in Windows: Not working

Support
1

Hi all, I managed to get OpenNMT working on a virtual Ubuntu machine, but now I’d like to leverage my GPU’s (SLI) as 40 hours of training time is stretching things a bit.

I’ve got Python working on Windows and even got Tensorflow installed and working. My last obstacle is installing OpenNMT on Python.

pip install URL
or
pip install opennmt-tf

does not work. Please see the attached screenshot.

Clipboard01
Clipboard01.jpg1920×1080 481 KB

How exactly do I go about this? I need to make sure the package is installed in Python 3.6. Please be precise: I’m a total noob when it comes to non-Windows stuff and the path names and directory chaos are confusing me.

Thank you very much in advance!

2

Hello,

You should install Git for Windows in order to retrieve OpenNMT-tf. It is not available via pip for now.

3

I tried that before. It does seem to install opennmt, but all I see in the Python27/Lib/site-packages/opennmt directory is six files (__init, config and constants.py/pyc). There’s no bin/main or anything similar. One thing that worries me: is this the right location, as this needs to run on Python 36 and not 27? And where have all the files gone?

Sorry for being a noob. I’m trying really hard.

4

Could you try the following:

  1. Open Git Bash
  2. Type:
git clone https://github.com/OpenNMT/OpenNMT-tf.git
cd OpenNMT-tf
python -m bin.main -h

Right now, OpenNMT-tf expects the user to use it from its own directory.

5

One step closer! But it seems I need to choose between running at the Python prompt or the Windows prompt and can’t have my cake and eat it. Please see the attached screenshot. As you can see, Python is there.

Clipboard01
Clipboard01.jpg1920×1080 412 KB

6

P.S. Tensorflow is there when I use import tensorflow as tf in the Python prompt.

7

So my guess is that I need to use import opennmt as nmt or something like that in the Python prompt. From there, how do I get access to commands like train?

8

So python is 2.7 but py is 3.6. This is confusing.

As you installed TensorFlow for Python 3, you should:

  1. Install pyyaml for Python 3 as well
  2. Invoke OpenNMT-tf scripts with py
9

So Python finds main.py, but main.py has issues finding other stuff.

Clipboard01
Clipboard01.jpg1920×1080 345 KB

10

Use:

py -m bin.main -h

This syntax is important as it adds the current directory to the Python paths.

11

Thank you. But then it has issues finding yaml, which is installed.

Clipboard01
Clipboard01.jpg1920×1080 343 KB
(I feel for the poor translator - who mostly has not background in programming or whatsoever) who ever needs to get this running! :wink:

12

You have conflicting Python versions.

You installed TensorFlow for Python 3 and PyYAML for Python 2. You should select one version and install all required packages for that version.

13

Ah, fantastic. Python 3.6 for Windows installs 2.7 with it by default, which is indeed very confusing. I just killed the entire 2.7 directory, then installed YAML via Git for Windows and voila, main is responding! I’ll keep on playing with this. Thank you very much for your help so far!

14

It seems the sample files have codes in them that are not allowed?

Clipboard01
Clipboard01.jpg1920×1080 422 KB

15

Yes, no matter what file I feed, I always get UnicodeDecodeError: ‘cp932’ codec can’t decode byte 0xef in position xxxx: illegal multibyte sequence during the train phase.

16

I can add sitecustomize.py to Python36/Lib stating:

import sys
import codecs
sys.stdout = codecs.getwriter(‘utf8’)(sys.stdout)
sys.stderr = codecs.getwriter(‘utf8’)(sys.stderr)

Now the program does run, but it quits without result, error or whatsoever. It starts, and stops. All I see is my prompt again.

1 Like
17

What command did you run?

18

Answering from another account as my first has reached its posting limit for the first day after registration.

python -m bin.main train --model config/models/nmt_small.py --config config/opennmt-defaults.yml config/data/xxx.yml

Whereby xxx is the name of the yaml file in question (it doesn’t work for the test files or my own files): both show the same symptom.

19

Should I conclude that OpenNMT does not run on Windows?

20

To be clear, Windows support is not a priority as people are massively using Linux systems for this kind of application. But if TensorFlow is correctly working on Windows, it would be nice that OpenNMT-tf works there as well.

Could you open an issue on GitHub? I will try to debug that when I have my hands on a Windows system. Thanks.

1 Like