Efficient implementation for training with uneven batch

Development
1

Although OpenNMT use ‘even source batch’ (i.e. source minibatch have same length) as default batch configuration, I need ‘uneven source batch’ (i.e. sourec minibatch have different length) for specific purpose.
In case training with uneven source batch, OpenNMT provide :maskPadding()

i.e. ) Seq2Seq.lua line 159
if batch.uneven then
self.models.decoder:maskPadding(batch.sourceSize, batch.sourceLength)

However, this function make nn.MaskedSoftmax() whenever new batch is coming.
And making this module include making submodules (nn.Narrow(), nn.Padding()) for each sentence in minibatch.

My question is
How much overhead is produced when training with ‘uneven batch’ mode?
(I wonder whether overhead is negligible or not)

2

we have not quantified precisely the overhead - but think it is not very important. I will run a specific profile to double check and will report soon.

In any case, we are aware of the complicated implementation using nngraph and we are also planning a new lighter implementation of this module.

3

Thanks for the reply.

I appreciate your hard works of implementation.