Crash in preprocessing: bad argument #1 to 'ipairs' (table expected, got userdata)

Support
1

I get a crash when preprocessing my data. Don’t know what I am doing wrong, as it seemed to work before on different data


Preparing vocabulary...
[09/01/17 11:06:45 INFO]  * Building source vocabularies...
[09/01/17 11:08:24 INFO]  * Created word dictionary of size 50004 (pruned from 254504)
[09/01/17 11:08:24 INFO] 
[09/01/17 11:08:24 INFO]  * Building target vocabularies...
[09/01/17 11:10:03 INFO]  * Created word dictionary of size 50004 (pruned from 111795)
[09/01/17 11:10:03 INFO] 
[09/01/17 11:10:03 INFO] Preparing training data...
[09/01/17 11:10:16 INFO] ... 100000 sentences prepared
...
[09/01/17 11:14:24 INFO] ... shuffling sentences
[09/01/17 11:15:00 INFO] ... sorting sentences by size
[09/01/17 11:15:35 INFO] Prepared 1785667 sentences:
[09/01/17 11:15:35 INFO]  * 191780 sequences not validated (length, other)
[09/01/17 11:15:35 INFO]  * average sequence length: source = 22.3, target = 24.0
[09/01/17 11:15:35 INFO]  * % of unknown words: source = 1.0%, target = 0.2%
[09/01/17 11:15:35 INFO]  * source sentence length (range of 10): [ 11% ; 28% ; 27% ; 17% ; 8% ; 3% ; 1% ; 0% ; 0% ; 0% ]
[09/01/17 11:15:35 INFO]  * target sentence length (range of 10): [ 8% ; 26% ; 27% ; 18% ; 9% ; 4% ; 2% ; 1% ; 0% ; 0% ]
[09/01/17 11:15:35 INFO] 
[09/01/17 11:15:35 INFO] Preparing validation data...
[09/01/17 11:15:35 INFO] ... shuffling sentences
[09/01/17 11:15:35 INFO] ... sorting sentences by size
[09/01/17 11:15:35 INFO] Prepared 571 sentences:
[09/01/17 11:15:35 INFO]  * 3 sequences not validated (length, other)
[09/01/17 11:15:35 INFO]  * average sequence length: source = 14.8, target = 16.7
[09/01/17 11:15:35 INFO]  * % of unknown words: source = 7.2%, target = 1.2%
[09/01/17 11:15:35 INFO]  * source sentence length (range of 10): [ 36% ; 35% ; 19% ; 7% ; 1% ; 0% ; 0% ; 0% ; 0% ; 0% ]
[09/01/17 11:15:35 INFO]  * target sentence length (range of 10): [ 25% ; 39% ; 23% ; 9% ; 1% ; 0% ; 0% ; 0% ; 0% ; 0% ]
[09/01/17 11:15:35 INFO] 
[09/01/17 11:15:35 INFO] Saving source vocabulary to '/home/suske/Scate/Data/WP4/np2en/opennmt/preproc.src.dict'...
[09/01/17 11:15:35 INFO] Saving target vocabulary to '/home/suske/Scate/Data/WP4/np2en/opennmt/preproc.tgt.dict'...
[09/01/17 11:15:35 INFO] Saving data to '/home/suske/Scate/Data/WP4/np2en/opennmt/preproc-train.t7'...
lua: /home/pricie/vincent/.luarocks/share/lua/5.1/tds/vec.lua:202: bad argument #1 to 'ipairs' (table expected, got userdata)

Any ideas on what I am doing wrong? Thanks

2

Hi @vincent, could you please copy the error lines following that shows where exactly the problem happens.
Did you checkout new version of the code?

3

stack traceback:
[C]: in function ‘ipairs’
/home/pricie/vincent/.luarocks/share/lua/5.1/tds/vec.lua:202: in function ‘write’
/home/pricie/vincent/torch/install/share/lua/5.1/torch/File.lua:210: in function ‘writeObject’
/home/pricie/vincent/torch/install/share/lua/5.1/torch/File.lua:235: in function ‘writeObject’
/home/pricie/vincent/torch/install/share/lua/5.1/torch/File.lua:235: in function ‘writeObject’
/home/pricie/vincent/torch/install/share/lua/5.1/torch/File.lua:235: in function ‘writeObject’
/home/pricie/vincent/torch/install/share/lua/5.1/torch/File.lua:388: in function 'save’
preprocess.lua:181: in function 'main’
preprocess.lua:185: in main chunk
[C]: ?

4

how do I check which version I have? --version or -v does not work.
I think I used the latest version. I’ve downloaded a new version from github and did a diff with the version I used, and there were no differences

5

just do:

git log -1
6

My version:

commit 2aedf53de060a4e1ecd7590d69a9f1b8dc6de1bd
Merge: 8aa50fb 3d01e63
Author: Guillaume Klein guillaumekln@users.noreply.github.com
Date: Thu Jul 13 10:16:09 2017 +0200

Merge pull request #342 from guillaumekln/save-validation-translation

Add option to save the validation translation to a file