Tokenize.lua results in


(Wiktor Stribiżew) #1

When I run

> cd ~/tds/OpenNMT/tools
> th tokenize.lua ende_srcs.txt ende_srcs.rawtok.txt

I get an error:

/home/wiktorstribizew/torch/install/bin/luajit: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: module 'onmt.init' not found:No LuaRocks module found for onmt.init
	no field package.preload['onmt.init']
	no file '/home/wiktorstribizew/.luarocks/share/lua/5.1/onmt/init.lua'
	no file '/home/wiktorstribizew/.luarocks/share/lua/5.1/onmt/init/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.1/onmt/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.1/onmt/init/init.lua'
	no file './onmt/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/luajit-2.1.0-beta1/onmt/init.lua'
	no file '/usr/local/share/lua/5.1/onmt/init.lua'
	no file '/usr/local/share/lua/5.1/onmt/init/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.2/onmt/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.2/onmt/init/init.lua'
	no file '/home/wiktorstribizew/.luarocks/lib/lua/5.1/onmt/init.so'
	no file '/home/wiktorstribizew/torch/install/lib/lua/5.1/onmt/init.so'
	no file '/home/wiktorstribizew/torch/install/lib/onmt/init.so'
	no file './onmt/init.so'
	no file '/usr/local/lib/lua/5.1/onmt/init.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
	no file '/home/wiktorstribizew/torch/install/lib/lua/5.2/onmt/init.so'
	no file '/home/wiktorstribizew/.luarocks/lib/lua/5.1/onmt.so'
	no file '/home/wiktorstribizew/torch/install/lib/lua/5.1/onmt.so'
	no file '/home/wiktorstribizew/torch/install/lib/onmt.so'
	no file './onmt.so'
	no file '/usr/local/lib/lua/5.1/onmt.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
	no file '/home/wiktorstribizew/torch/install/lib/lua/5.2/onmt.so'
stack traceback:
	[C]: in function 'error'
	...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: in function 'require'
	tokenize.lua:2: in main chunk
	[C]: in function 'dofile'
	...izew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x00406240

Note that I am using Lua 5.2:

lua -v
Lua 5.2.3  Copyright (C) 1994-2013 Lua.org, PUC-Rio

While following the Installation instructions, I could not make

luarocks install tds

work. It returned

Warning: Failed searching manifest: Failed fetching manifest for https://raw.githubusercontent.com/torch/rocks/master - Failed downloading https://raw.githubusercontent.com/torch/rocks/master/manifest - /home/wiktorstribizew/.cache/luarocks/https___raw.githubusercontent.com_torch_rocks_master/manifest
Warning: Failed searching manifest: Failed fetching manifest for https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master - Failed downloading https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/manifest - /home/wiktorstribizew/.cache/luarocks/https___raw.githubusercontent.com_rocks-moonscript-org_moonrocks-mirror_master/manifest

Error: No results matching query were found.

I could not work around it - I tried all suggestions there are on this Web site, StackOverflow, and other sites (like in Tds not found when running preprocess.lua). I could not make it work the way described at Installing Torch7 and OpenNMT On Ubuntu 14.04. The solution here worked:

git clone https://github.com/torch/tds.git
cd tds
luarocks make rocks/tds-scm-1.rockspec

Now, running luarocks command, I get

CONFIGURATION
	Lua version: 5.2
	Configuration files:
		System: /home/wiktorstribizew/torch/install/etc/luarocks/config.lua (ok)
		User  : /home/wiktorstribizew/.luarocks/config-5.2.lua (not found)

	Rocks trees in use: 
		/home/wiktorstribizew/.luarocks
		/home/wiktorstribizew/torch/install

No idea how to proceed. I can run

  • preprocess.lua script
  • train.lua script

Basically, I tried the demo training for ENDE and was going to try in-domain training using the OpenNMT ENDE model and some in-domain corpus, and that is where I am stuck right now.

P.S. In addition to the answer marked as solved:

  • Had to uninstall Torch (cd /; sudo rm -rf ~/torch) and reinstall (cd ~/torch; git clone --recursive https://github.com/torch/distro.git ~/torch)
  • Since after I ran ./install.sh I saw many errors, I had to install Moses manually: cd ~/torch; git clone git://github.com/Yonaba/Moses.git; cd ~/torch/Moses; luarocks make rockspec/moses-1.6.1-1.rockspec; cd ~/torch; ./install.sh but there were errors again: luarocks install bit32 output errors
  • Then I had to install bits32 manually as explained in the accepted answer, but before, I ran export {http,https,ftp}_proxy='http://LOGIN:PASSWORD@PROXY_IP:PORT and replaced ; in my password with %3B
  • To run tools/tokenize.lua, I needed to run it from the OpenNMT folder using th tools/tokenize.lua < input.txt > output.txt

(Guillaume Klein) #2

It is a bit hidden in the documentation but this is an important point:

By default, OpenNMT’s scripts can only be called from the root of OpenNMT’s directory.

See http://opennmt.net/OpenNMT/options/usage/.


For your issues when installing tds, did you follow the standard Torch installation?


(Wiktor Stribiżew) #3

I think I have. Here are my installation notes:

Looks like I ran the same scripts twice, but it should not be a problem, I believe. I just had problems with getting through the company’s gateway, but managed to do it using AskUbuntu’s help.

As for running the scripts from the OpenNMT folder, I first tried with

cd ~/tds/OpenNMT/
th tools/tokenize.lua ...

with the same result.

To summarize, I have Home>tds>OpenNMT path. And … just now, I found another OpenNMT folder inside that OpenNMT folder. The Home>tds>OpenNMT has a CHANGELOG of version 0.6, and the other one (that I have never used) has a change log saying it is 0.7.

Should I remove the nested OpenNMT folder? Maybe I should remove everything in ~/tds? Should I try to uninstall Lua 5.2 and downgrade to Lua 5.1? I am working on a VM for the time being with Ubuntu 14.04 installed.


(jean.senellart) #4

the initial error log you have is clearly related to the path issue - could you share the error message you have when running from the OpenNMT directory?

all of this is a bit strange but should not be a problem, however I would myself cleanup everything and do a new clone outside and remove these directories.

I don’t think it is related to lua version.


(Wiktor Stribiżew) #5

Yes, @jean.senellart, it seems you are right, sorry for not noticing the difference. The error log is a bit different. It seems the script requires lua 5.1.

/home/wiktorstribizew/torch/install/bin/luajit: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: module 'bit32' not found:No LuaRocks module found for bit32
	no field package.preload['bit32']
	no file '/home/wiktorstribizew/.luarocks/share/lua/5.1/bit32.lua'
	no file '/home/wiktorstribizew/.luarocks/share/lua/5.1/bit32/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.1/bit32.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.1/bit32/init.lua'
	no file './bit32.lua'
	no file '/home/wiktorstribizew/torch/install/share/luajit-2.1.0-beta1/bit32.lua'
	no file '/usr/local/share/lua/5.1/bit32.lua'
	no file '/usr/local/share/lua/5.1/bit32/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.2/bit32.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.2/bit32/init.lua'
	no file '/home/wiktorstribizew/.luarocks/lib/lua/5.1/bit32.so'
	no file '/home/wiktorstribizew/torch/install/lib/lua/5.1/bit32.so'
	no file '/home/wiktorstribizew/torch/install/lib/bit32.so'
	no file './bit32.so'
	no file '/usr/local/lib/lua/5.1/bit32.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
	no file '/home/wiktorstribizew/torch/install/lib/lua/5.2/bit32.so'
stack traceback:
	[C]: in function 'error'
	...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: in function 'require'
	tools/tokenize.lua:5: in main chunk
	[C]: in function 'dofile'
	...izew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x00406240

@guillaumekln: If you can, please also have a look at this error stack.


(Guillaume Klein) #6

Run:

luarocks install bit32

cf. http://opennmt.net/OpenNMT/tools/tokenization/.


(Wiktor Stribiżew) #7

@guillaumekln I have a generic issue with luarocks ... commands :frowning:

wiktorstribizew@wiktorstribizew-VirtualBox:~/tds/OpenNMT$ luarocks install bit32Warning: Failed searching manifest: Failed fetching manifest for https://raw.githubusercontent.com/torch/rocks/master - Failed downloading https://raw.githubusercontent.com/torch/rocks/master/manifest - /home/wiktorstribizew/.cache/luarocks/https___raw.githubusercontent.com_torch_rocks_master/manifest
Warning: Failed searching manifest: Failed fetching manifest for https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master - Failed downloading https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/manifest - /home/wiktorstribizew/.cache/luarocks/https___raw.githubusercontent.com_rocks-moonscript-org_moonrocks-mirror_master/manifest

Error: Rock bit32 5.2-1 was found but it is provided by VM or 'rocks_provided' in the config file.

(Guillaume Klein) #8

Could you run:

rm -rf ~/.cache/luarocks

and try again?

If that does not help, could you share the output of:

which luarocks

(Wiktor Stribiżew) #9

@guillaumekln: I mentioned in my original post that I tried that solution (mentione here). It makes no difference. which luarocks results in /home/wiktorstribizew/torch/install/bin/luarocks.


(Guillaume Klein) #10

There is something wrong with your installation I think as you said you installed Lua 5.2 but have LuaJIT issues and search paths.

I would reinstall Torch from scratch:

rm -rf ~/torch
git clone --recursive  https://github.com/torch/distro.git ~/torch
cd ~/torch
./install.sh
source ~/torch/install/bin/torch-activate

(Wiktor Stribiżew) #11

@guillaumekln I am sorry, it looks like I have something really wrong. I uninstalled torch using sudo rm -rf ~/torch as rm -rf ~/torch did not work. Then, git clone --recursive https://github.com/torch/distro.git ~/torch installed torch, but after running the ./install.sh, I got

...
-- Set runtime path of "/home/wiktorstribizew/torch/install/lib/luarocks/rocks/sys/1.1-0/lib/libsys.so" to "$ORIGIN/../lib:/home/wiktorstribizew/torch/install/lib:/opt/OpenBLAS/lib"
-- Installing: /home/wiktorstribizew/torch/install/lib/luarocks/rocks/sys/1.1-0/lua/sys/init.lua
-- Installing: /home/wiktorstribizew/torch/install/lib/luarocks/rocks/sys/1.1-0/lua/sys/colors.lua
-- Installing: /home/wiktorstribizew/torch/install/lib/luarocks/rocks/sys/1.1-0/lua/sys/fpath.lua
Updating manifest for /home/wiktorstribizew/torch/install/lib/luarocks/rocks
sys 1.1-0 is now built and installed in /home/wiktorstribizew/torch/install/ (license: BSD)

Updating manifest for /home/wiktorstribizew/torch/install/lib/luarocks/rocks
xlua 1.0-0 is now built and installed in /home/wiktorstribizew/torch/install/ (license: BSD)


Missing dependencies for nn:
moses >= 1.0

Warning: Failed searching manifest: Failed fetching manifest for https://raw.githubusercontent.com/torch/rocks/master - Failed downloading https://raw.githubusercontent.com/torch/rocks/master/manifest - /home/wiktorstribizew/.cache/luarocks/https___raw.githubusercontent.com_torch_rocks_master/manifest
Warning: Failed searching manifest: Failed fetching manifest for https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master - Failed downloading https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror/master/manifest - /home/wiktorstribizew/.cache/luarocks/https___raw.githubusercontent.com_rocks-moonscript-org_moonrocks-mirror_master/manifest

Error: Could not satisfy dependency: moses >= 1.0

And when I try tokenizing now, I get

/home/wiktorstribizew/torch/install/bin/luajit: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: ...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: module 'nngraph' not found:No LuaRocks module found for nngraph
	no field package.preload['nngraph']
	no file '/home/wiktorstribizew/.luarocks/share/lua/5.1/nngraph.lua'
	no file '/home/wiktorstribizew/.luarocks/share/lua/5.1/nngraph/init.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.1/nngraph.lua'
	no file '/home/wiktorstribizew/torch/install/share/lua/5.1/nngraph/init.lua'
	no file './nngraph.lua'
	no file '/home/wiktorstribizew/torch/install/share/luajit-2.1.0-beta1/nngraph.lua'
	no file '/usr/local/share/lua/5.1/nngraph.lua'
	no file '/usr/local/share/lua/5.1/nngraph/init.lua'
	no file '/home/wiktorstribizew/.luarocks/lib/lua/5.1/nngraph.so'
	no file '/home/wiktorstribizew/torch/install/lib/lua/5.1/nngraph.so'
	no file '/home/wiktorstribizew/torch/install/lib/nngraph.so'
	no file './nngraph.so'
	no file '/usr/local/lib/lua/5.1/nngraph.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'error'
	...ktorstribizew/torch/install/share/lua/5.1/trepl/init.lua:389: in function 'require'
	tools/tokenize.lua:2: in main chunk
	[C]: in function 'dofile'
	...izew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
	[C]: at 0x00406240

(Guillaume Klein) #12

It seems related to your network issues.

Try installing moses manually and then re-try install.sh.


(Wiktor Stribiżew) #13

@guillaumekln I do not know if I am doing all that correctly. Here is what I did:

cd ~/torch
git clone git://github.com/Yonaba/Moses.git 
cd ~/torch/Moses
luarocks make rockspec/moses-1.6.1-1.rockspec
cd ~/torch
./install.sh

And it looks like the scripts did a good job:

Do you want to automatically prepend the Torch install location
to PATH and LD_LIBRARY_PATH in your /home/wiktorstribizew/.bashrc? (yes/no)
[yes] >>> 
yes

But trying to install bits32

luarocks install bit32

Resulted in the same errors - no manifest could be found. I also tried adding sudo at the start and got another issue:

wiktorstribizew@wiktorstribizew-VirtualBox:~/torch$ sudo luarocks install bit32
[sudo] password for wiktorstribizew: 
Warning: Failed searching manifest: Failed loading manifest: Failed fetching manifest for http://luarocks.org/repositories/rocks

Error: No results matching query were found.

Is there a special proxy I should set for luarocks? I feel that is the culprit.


(Guillaume Klein) #14

Yes it seems network related.

Maybe you could also install bit32 manually. Try this:

wget https://luarocks.org/manifests/siffiejoe/bit32-5.3.0-1.src.rock
luarocks unpack bit32-5.3.0-1.src.rock
cd bit32-5.3.0-1/lua-compat-5.2
luarocks make bit32-5.3.0-1.rockspec

(Wiktor Stribiżew) #15

@guillaumekln Great! That seems to have fixed the issue that bugged me so long. Now, when trying to tokenize the corpus like

cd ~/tds/OpenNMT
th tools/tokenize.lua ~/share/MT/_Evaluations/__test/ende/crp/ende_srcs.txt ~/share/MT/_Evaluations/__test/ende/crp/ende_srcs.rawtok.txt

I get:

tokenize.lua: invalid argument: /home/wiktorstribizew/share/MT/_Evaluations/__test/ende/crp/ende_srcs.txt
Try 'tokenize.lua -h' for more information, or visit the online documentation at http://opennmt.net/OpenNMT/.

How should I use the tokenize.lua script to pass the input/output files?


(jean.senellart) #16

th tools/tokenize.lua < ~/share/MT/_Evaluations/__test/ende/crp/ende_srcs.txt > ~/share/MT/_Evaluations/__test/ende/crp/ende_srcs.rawtok.txt


(Wiktor Stribiżew) #17

@jean.senellart, @guillaumekln

Many a thank, all works well now. I did not realize the < ... > are not any syntactic sugar when I saw $ for f in ../wmt15-de-en/*.?? ; do th tools/tokenize.lua < $f > $f.tok ; done.

I see now that my problem was close to Tokenize.lua needs tools.utils.unicode, can’t find it question, just I had to install bits32 and moses manually.

Thank you a lot.