Number of samples in one training step together with accumulation

[jechoi]

Hi,

how many samples would 1 training step cover together with accumulation?

For example if you see from log output 1/1000 training step and you have 2 gpus, 6 batchsize, 2 accum_count… then does that 1 training step include 2 x 6 x 2 samples? or does it only include 2 x 6 samples and you do not see that actual the optimization happening?

[francoishernandez (François Hernandez)]

Epochs Determination

You also need to take into account the number of GPU you use and eventual gradient accumulation (-accum_count) to have your ‘true’ batch size. E.g. with -batch_size 512, 2 GPUs, -accum_count 3, one step will be 512 * 2 * 3 examples. You can check it in the log, when it loads the data again, hence starting an epoch, it will log ‘Loading dataset …’. (This may be a bit off because examples are loaded in advance to improve training speed and GPU utilization.)

[jechoi]

ah alright that was also one question that the print out about loading data seemed off. Thanks!!!

[anderleich]

Hi,
I found this topic really interesting.
If a dataset has 4500 examples and we have a batch_size of 4096, 1 GPU and accum_count is 4. I guess in each steps the dataset is loaded 4 times. Is it correct?

[francoishernandez (François Hernandez)]

batch_size of 4096 is probably tokens, not examples?

But if batch_type were to be examples, yes that would be the idea.

[anderleich]

I see. Are examples taken until 4096 tokens are filled?

[francoishernandez (François Hernandez)]

Yes.

OpenNMT/OpenNMT-py/blob/8b073fb2a047509ff590839b1194a155ec1a50bf/onmt/inputters/iterator.py#L135-L153