In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

[paulkp (Paul Pallaghy)]

This question comes from here , but is unanswered.

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

Can someone here answer it HERE please?

It seems kind of crazy to ADD positional encoding to the attention and thus - seemingly irreversibly - confound them. Do we just expect NNs to be so good at de-convoluting things? But why give it unnecessary work which would hit the LR. Or am I missing something? Does add mean concatenate?

Concatenation would make more sense.

So why does the standard Transformer imp work?

[paulkp (Paul Pallaghy)]

Anyone . . ?

[guillaumekln (Guillaume Klein)]

Adding positional encodings is no different than adding bias in fully connected layers. Maybe that helps answer your question?

@paulkp

Concatenation would make more sense.

There is nothing that tells the next layer which part comes from the word embedding and which part come from positional encoding. So the NN would also need to figure this out.