Problem retraining from a checkpoint


(Hamidreza Ghader) #1

Hi all,

I have trained a model two months ago. Now, I want to start retraining it for a few more epochs. But it comes up with the following error.
Here is what I get in the output:

Loading train data from ‘/home/preprocesses_new’

  • number of train sentences: 4544648
    Loading valid data from ‘/home/preprocesses_new’
  • number of valid sentences: 2732
  • maximum batch size: 64
    Loading checkpoint from /home/models/
    Traceback (most recent call last):
    File “/home/projects/opennmt/OpenNMT-py/”, line 349, in
    File “/home/projects/opennmt/OpenNMT-py/”, line 330, in main
    fields = load_fields(train_dataset, valid_dataset, checkpoint)
    File “/home/projects/opennmt/OpenNMT-py/”, line 241, in load_fields
    torch.load( + ‘’), data_type)
    File “/home/projects/opennmt/OpenNMT-py/onmt/io/”, line 61, in load_fields_from_vocab
    fields[k].vocab = v
    TypeError: ‘NoneType’ object has no attribute ‘getitem

Note that I have pulled the latest version just yesterday. I think it is because of using two different versions for the original training and retraining. However, the question is how I can fix it.


(Pltrdy) #2

Since it seems to be data related I would suggest you to run the preprocessing step again (in another folder maybe) then try again.

(Hamidreza Ghader) #3

I actually did it. As the result, it is now able to start loading the checkpoint. However, it stops with the following error.

Traceback (most recent call last):
File “/home/hghader1/projects/opennmt/OpenNMT-py/”, line 349, in
File “/home/hghader1/projects/opennmt/OpenNMT-py/”, line 345, in main
train_model(model, train_dataset, valid_dataset, fields, optim, model_opt)
File “/home/hghader1/projects/opennmt/OpenNMT-py/”, line 168, in train_model
train_stats = trainer.train(epoch, report_func)
File “/home/hghader1/projects/opennmt/OpenNMT-py/onmt/”, line 170, in train
File “/home/hghader1/projects/opennmt/OpenNMT-py/onmt/”, line 87, in step
if self.decay_method == “noam”:
AttributeError: ‘Optim’ object has no attribute ‘decay_method’