About onmt.utils.loss

[guz18 (Göksu g)]

Hi everyone,
I am trying to see the loss graph and epochs by using onmt.utils.loss, however I am new to Deep Learning and there is no guide or tutorial on how to use it. Can you help me regarding this issue? (My research topic is on Grammar Error Correction in Turkish language.)

[ymoslem (Yasmin Moslem)]

Dear Göksu,

OpenNMT reports Perplexity. See this nice explanation by Paul Tardy:

In NLP, why do we use perplexity instead of the loss?

As for how to calculate epochs, see this answer:

Epochs Determination

You also need to take into account the number of GPU you use and eventual gradient accumulation (-accum_count) to have your ‘true’ batch size. E.g. with -batch_size 512, 2 GPUs, -accum_count 3, one step will be 512 * 2 * 3 examples. You can check it in the log, when it loads the data again, hence starting an epoch, it will log ‘Loading dataset …’. (This may be a bit off because examples are loaded in advance to improve training speed and GPU utilization.)

See also this item in FAQs; although it is for OpenNMT-tf, it applies to OpenNMT-py, too:
https://opennmt.net/OpenNMT-tf/faq.html#how-to-count-the-number-of-epochs

All the best,
Yasmin