How to resume training from last interupted state

Support
1

I was training a translation model, but due to a power failure system eventually restarted. The last checkpoint I have saved is on 90K training step. Is there any way to resume training from that same state??

1 Like
2

If you train by OpenNMT-py

–train_from, -train_from

If training from a checkpoint then this is the path to the pretrained model’s state_dict.

Default: “”

3

Dear @park, Thanks for your reply. But I am getting an error “[Errno 2] No such file or directory: ‘sum_eng-model_90000.pt’”
My saved model is in the main directory, with other files like preprocessing.py, train.py are.(Default location where OpenNMT saves them).
I just added -train_from sum_eng-model_90000.pt to my command.
What correction should I made to make this work??
Many thanks in advance.

4

You need to specify your model PATH

-train_from YOUR/MODEL/PATH/ sum_eng-model_90000.pt

Please add the PATH

5

@park Sorry, but I am still not able to resolve it. I shifted sum_eng-model_90000.pt file in the data folder, with other trains, test, valid data files. Now I added -train_from data/sum_eng-model_90000.pt to my command, But I am still getting error file not found. Can you please help a bit.

sum_eng-model_90000.pt this is the name of file and available in the data folder.
What should I add to my command??

6

Please give me the full command

7

@park Here is the full command I am using

CUDA_VISIBLE_DEVICES=0 python train.py -src_word_vec_size 200 -tgt_word_vec_size 200 -data data/model -save_model sum_eng-model -save_checkpoint_steps 100 -world_size 1 -gpu_ranks 0 -batch_size 64 -valid_steps 10000 -train_steps 100000 -report_every 50 -train_from data/sum_eng-model_90000.pt

8

Try

CUDA_VISIBLE_DEVICES=0 python3 train.py -src_word_vec_size 200 -tgt_word_vec_size 200 -data data/model -save_model sum_eng-model -save_checkpoint_steps 100 -world_size 1 -gpu_ranks 0 -batch_size 64 -valid_steps 10000 -train_steps 100000 -report_every 50 --train_from data/sum_eng-model_90000.pt

9

@park still same error [Errno 2] No such file or directory: ‘data/sum_eng-model_90000.pt’.
File is there and file name is correct too.

10

try

python3 setup.py build
python3 setup.py install

12

@perk Thanks for your effort. This was a dependencies Issue.

13

Okay, Nice work