While training with the following command:
CUDA_VISIBLE_DEVICES=0 nohup onmt-main train_and_eval --model_type TransformerFP16 --config config/main_v2.yml --auto_config &> foobar.log &
after evaluation at 1000 steps (although in the
main_v2.yml it is defined that
eval_delay: 3600 and at this point:
INFO:tensorflow:Loading best metric from event files.
I get a traceback with the following error:
ValueError: best_eval_result cannot be empty or no loss is found in it.
Any help will be greatly appreciated.