Averaging nonconsecutive checkpoints

Support
1

Hello!

It is possible in OpenNMT-py to average nonconsecutive model checkpoints. Is there any way to achieve the same with OpenNMT-tf? I can only see this command which assumes checkpoints to be conseutive.

onmt-main --config config.yml --auto_config average_checkpoints --output_dir avg_model_dir --max_count 5

Thanks!
Yasmin

2

Hi,

There is no direct way to do this from the command line, but you could manually and temporarily change the checkpoint file in the model directory.

The file looks like this:

model_checkpoint_path: "ckpt-38000"
all_model_checkpoint_paths: "ckpt-34000"
all_model_checkpoint_paths: "ckpt-35000"
all_model_checkpoint_paths: "ckpt-36000"
all_model_checkpoint_paths: "ckpt-37000"
all_model_checkpoint_paths: "ckpt-38000"
all_model_checkpoint_timestamps: 1646876022.1235282
all_model_checkpoint_timestamps: 1646876991.0456057
all_model_checkpoint_timestamps: 1646877960.1573496
all_model_checkpoint_timestamps: 1646878926.6658986
all_model_checkpoint_timestamps: 1646879891.9810448
last_preserved_timestamp: 1646841794.5741713

If you want to average the checkpoints at 34000, 36000, and 37000, you could change it like this:

model_checkpoint_path: "ckpt-37000"
all_model_checkpoint_paths: "ckpt-34000"
all_model_checkpoint_paths: "ckpt-36000"
all_model_checkpoint_paths: "ckpt-37000"
2 Likes