Problem retraining from a checkpoint

Support
1

Hi all,

I have trained a model two months ago. Now, I want to start retraining it for a few more epochs. But it comes up with the following error.
Here is what I get in the output:

Loading train data from ‘/home/preprocesses_new’

  • number of train sentences: 4544648
    Loading valid data from ‘/home/preprocesses_new’
  • number of valid sentences: 2732
  • maximum batch size: 64
    Loading checkpoint from /home/models/de-en_BPE_param_exp_acc_59.97_ppl_11.00_e9.pt
    Traceback (most recent call last):
    File “/home/projects/opennmt/OpenNMT-py/train.py”, line 349, in
    main()
    File “/home/projects/opennmt/OpenNMT-py/train.py”, line 330, in main
    fields = load_fields(train_dataset, valid_dataset, checkpoint)
    File “/home/projects/opennmt/OpenNMT-py/train.py”, line 241, in load_fields
    torch.load(opt.data + ‘.vocab.pt’), data_type)
    File “/home/projects/opennmt/OpenNMT-py/onmt/io/IO.py”, line 61, in load_fields_from_vocab
    fields[k].vocab = v
    TypeError: ‘NoneType’ object has no attribute ‘getitem’

Note that I have pulled the latest version just yesterday. I think it is because of using two different versions for the original training and retraining. However, the question is how I can fix it.

Best,
Hamid

2

Since it seems to be data related I would suggest you to run the preprocessing step again (in another folder maybe) then try again.

3

I actually did it. As the result, it is now able to start loading the checkpoint. However, it stops with the following error.

Traceback (most recent call last):
File “/home/hghader1/projects/opennmt/OpenNMT-py/train.py”, line 349, in
main()
File “/home/hghader1/projects/opennmt/OpenNMT-py/train.py”, line 345, in main
train_model(model, train_dataset, valid_dataset, fields, optim, model_opt)
File “/home/hghader1/projects/opennmt/OpenNMT-py/train.py”, line 168, in train_model
train_stats = trainer.train(epoch, report_func)
File “/home/hghader1/projects/opennmt/OpenNMT-py/onmt/Trainer.py”, line 170, in train
self.optim.step()
File “/home/hghader1/projects/opennmt/OpenNMT-py/onmt/Optim.py”, line 87, in step
if self.decay_method == “noam”:
AttributeError: ‘Optim’ object has no attribute ‘decay_method’