Is Data Checkpointing possible?

I use opennmt-py a lot and I understand that checkpointing will let you save model weights and begin training a new model starting with those same weights. Is it possible to checkpoint where in the data you are training? When a training job is canceled at a specific point in the data, I would like to be able to start from the same location in the data when beginning to train on the checkpoint again.

this has not been implemented yet

1 Like