Hi, could you please explain why there is a ‘’ in the dictionary after preprocessing? Why do we need this apart from and ?
It is reserved for the padding which has its own entry in the embedding lookup table.
Could you please explain in more detail?
The lookup table is used as input for the encoder and decoder.
By default, there is no padding on the source side (unless the -uneven_batches flag is enabled). The target side usually contains padding.