Early stopping not working

Support
1

Hi,
I am on the training process and I think that early stopping is not working well.
I defined early stopping as following:
eval:
batch_size: 32
steps: 3000
exporters: last
external_evaluators: BLEU
early_stopping:
metric: bleu
min_improvement: 0.01
steps: 4

Is it correct?
My evaluation score is the following:

INFO:tensorflow:Evaluation result for step 162000: loss = 0.982170 ; bleu = 50.186870
INFO:tensorflow:Evaluation result for step 165000: loss = 0.982629 ; bleu = 50.013698
INFO:tensorflow:Evaluation result for step 168000: loss = 0.981667 ; bleu = 50.349343
INFO:tensorflow:Evaluation result for step 171000: loss = 0.979857 ; bleu = 50.553991
INFO:tensorflow:Evaluation result for step 174000: loss = 0.981263 ; bleu = 50.436176
INFO:tensorflow:Evaluation result for step 177000: loss = 0.979642 ; bleu = 50.419565
INFO:tensorflow:Evaluation result for step 180000: loss = 0.979617 ; bleu = 50.219028
INFO:tensorflow:Evaluation result for step 183000: loss = 0.980253 ; bleu = 50.375434
INFO:tensorflow:Evaluation result for step 186000: loss = 0.980024 ; bleu = 50.331372
INFO:tensorflow:Evaluation result for step 189000: loss = 0.976821 ; bleu = 50.497546
INFO:tensorflow:Evaluation result for step 192000: loss = 0.978794 ; bleu = 50.345844
INFO:tensorflow:Evaluation result for step 195000: loss = 0.977428 ; bleu = 50.468922

The best score was in 171000 steps (50.55) and it does not improve in the next 24000 steps. Should not it stop already?

Regards

2

Hi,

It seems that the training should stop.

Do you confirm it is a training you started from scratch with OpenNMT-tf 2.0?

3

Yes but on 162000 step I had to stop it. Later I continued the training. In addition there are any early stopping log during the training. Is it ok?

4

Thanks for testing and reporting. I did find an incorrect logic:

Regarding logs, early stopping only produces a log message when the stopping condition is met.