OpenNMT Forum

Having issue while run preprocessing.py

In preprocess.py
issue about "torch.save(fields, vocab_path)

Please let me know the command

python preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/data -src_vocab_size 1000 -tgt_vocab_size 1000

please

pip3 install torch
pip3 install -r requirements.txt

Also check the python version

Try

python3 preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/data -src_vocab_size 1000 -tgt_vocab_size 1000

my python -V is Python 3.7.0
I have installed requirements.txt and torch, but the problem does still exists.

Traceback (most recent call last):
File “preprocess.py”, line 215, in
main(opt)
File “preprocess.py”, line 196, in main
‘train’, fields, src_reader, tgt_reader, opt)
File “preprocess.py”, line 139, in build_save_dataset
torch.save(fields, vocab_path)
File “C:\Users\Administrator\Anaconda3\lib\site-packages\torch\serialization.py”, line 209, in save
return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))
File “C:\Users\Administrator\Anaconda3\lib\site-packages\torch\serialization.py”, line 134, in _with_file_like
return body(f)
File “C:\Users\Administrator\Anaconda3\lib\site-packages\torch\serialization.py”, line 209, in
return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))
File “C:\Users\Administrator\Anaconda3\lib\site-packages\torch\serialization.py”, line 282, in _save
pickler.dump(obj)
_pickle.PicklingError: Can’t pickle <function Field. at 0x0000013D8D5D6510>: attribute lookup Field. on torchtext.data.field failed

please command like below

python3 setup.py build
python3 setup.py install

it doesn’t work, but thank you for your help

I see you are using Anaconda on Windows. What is the Torch version installed for Anaconda?

You need to update to the latest version of PyTorch. To do so, open the PyTorch website, scroll down, and click on the relevant options: Stable, Windows, Conda, Python 3.7, and CUDA version if you are on a GPU. This will change the field “Run this Command” that you have to use.

Please note also that PyTorch doesn’t work on 32-bit system. Please use Windows and Python 64-bit version.

Kind regards,
Yasmin