How to calculate the number of epochs?

[azLe]

I am new to ONMT, and am a bit confused about the number of epochs and training steps. I know the number of epochs is the number of times that we iterate through the training dataset, but how can I calculate the number of epochs that I trained my model with? e.g., let’s say I have 20000 training samples and my batch_size is 64 and had 3200 training steps. If this is the case, then what is the number of epochs?

[ymoslem (Yasmin Moslem)]

Hello!

Your calculation is relatively correct. You will have also to take into consideration “gradient accumulation”. In OpenNMT-py, this is determined by the accum_count option. In OpenNMT-tf, this is determined by the effective_batch_size option.

Note also that if your batch_type is examples, this is easier to determine than when it is tokens. In the case of tokens, it can only be a rough estimation.

There are a couple of good answers on the forum explaining the topic, like this one.

Epochs Determination

You also need to take into account the number of GPU you use and eventual gradient accumulation (-accum_count) to have your ‘true’ batch size. E.g. with -batch_size 512, 2 GPUs, -accum_count 3, one step will be 512 * 2 * 3 examples. You can check it in the log, when it loads the data again, hence starting an epoch, it will log ‘Loading dataset …’. (This may be a bit off because examples are loaded in advance to improve training speed and GPU utilization.)

To force training on epochs rather than steps, you can use the option single_pass, supported by both OpenNMT-tf and OpenNMT-py.

https://opennmt.net/OpenNMT-tf/faq.html#how-to-count-the-number-of-epochs

Kind regards,
Yasmin