anderleich
August 25, 2021, 11:59am
1
Hi,
I need to change the dropout when finetuning a base Transformer model. When training from a checkpoint, it uses the dropout from the checkpoint. Is there a way I can force the dropout to be changed?
Thanks
anderleich
September 3, 2021, 8:25am
2
I considered adding the following lines in train_single.py:
# Override checkpoint's droupout
model_opt.dropout = opt.dropout
model_opt.attention_dropout = opt.attention_dropout
if checkpoint is not None:
model_opt = ArgumentParser.ckpt_model_opts(checkpoint["opt"])
ArgumentParser.update_model_opts(model_opt)
ArgumentParser.validate_model_opts(model_opt)
if (opt.tensorboard_log_dir == model_opt.tensorboard_log_dir and
hasattr(model_opt, 'tensorboard_log_dir_dated')):
# ensure tensorboard output is written in the directory
# of previous checkpoints
opt.tensorboard_log_dir_dated = model_opt.tensorboard_log_dir_dated
# Override checkpoint's update_embeddings as it defaults to false
model_opt.update_vocab = opt.update_vocab
else:
model_opt = opt
return model_opt
def _build_valid_iter(opt, fields, transforms_cls):
"""Build iterator used for validation."""
valid_iter = build_dynamic_dataset_iter(
fields, transforms_cls, opt, is_train=False)
return valid_iter
This seems like a good idea, but I wonder if we should add as well some sort of flag e.g. --override-opts to avoid changing some opts inadvertently.
Does OpenNMT-py support fine tuning?