OpenNMT Forum

Early stopping not working

Hi,
I am on the training process and I think that early stopping is not working well.
I defined early stopping as following:
eval:
batch_size: 32
steps: 3000
exporters: last
external_evaluators: BLEU
early_stopping:
metric: bleu
min_improvement: 0.01
steps: 4

Is it correct?
My evaluation score is the following:

INFO:tensorflow:Evaluation result for step 162000: loss = 0.982170 ; bleu = 50.186870
INFO:tensorflow:Evaluation result for step 165000: loss = 0.982629 ; bleu = 50.013698
INFO:tensorflow:Evaluation result for step 168000: loss = 0.981667 ; bleu = 50.349343
INFO:tensorflow:Evaluation result for step 171000: loss = 0.979857 ; bleu = 50.553991
INFO:tensorflow:Evaluation result for step 174000: loss = 0.981263 ; bleu = 50.436176
INFO:tensorflow:Evaluation result for step 177000: loss = 0.979642 ; bleu = 50.419565
INFO:tensorflow:Evaluation result for step 180000: loss = 0.979617 ; bleu = 50.219028
INFO:tensorflow:Evaluation result for step 183000: loss = 0.980253 ; bleu = 50.375434
INFO:tensorflow:Evaluation result for step 186000: loss = 0.980024 ; bleu = 50.331372
INFO:tensorflow:Evaluation result for step 189000: loss = 0.976821 ; bleu = 50.497546
INFO:tensorflow:Evaluation result for step 192000: loss = 0.978794 ; bleu = 50.345844
INFO:tensorflow:Evaluation result for step 195000: loss = 0.977428 ; bleu = 50.468922

The best score was in 171000 steps (50.55) and it does not improve in the next 24000 steps. Should not it stop already?

Regards

Hi,

It seems that the training should stop.

Do you confirm it is a training you started from scratch with OpenNMT-tf 2.0?

Yes but on 162000 step I had to stop it. Later I continued the training. In addition there are any early stopping log during the training. Is it ok?

Thanks for testing and reporting. I did find an incorrect logic:

Regarding logs, early stopping only produces a log message when the stopping condition is met.