Word embeddings sizes

Support
1

Hello, I have some doubts about the generated word embedding for NMTSmall architecture. As far as I know about word embeddings and neural architectures, word embedding matrix is supposed to be in vocab_source_sizeXembedding_size space. It happens quite the same with target words embeddings. However, when I was playing around with the generated embeddings, I found out that the dimensions does not corresponds to the source/target vocabulary. You can see the picture below…

capture_word_embeddings_all
capture_word_embeddings_all.png1140×724 66.8 KB

I thought about the special tokens (, and so on), but vocabulary files contains special tokens too, so I can’t find out why the embedding matrix is in (vocab_size+2)Xsize_embedding space. If you can explain to me the reason, I will appreciate it.

Thank you in advance.

2

Hi,

There is an additional index for tokens that are not in your vocabulary file. So the final embedding shape is [vocab_size + 1, embedding_size].

3

Thank you a lot for the quickly response. However, since the index begin in 0, it supposed to be [(vocab_size-1) + 1, embedding_size]. Vocab_size-1 is because of indexation and the +1 because of the OOV token. In my case it is still one more dimension.

4

No, it’s not because of the indexation.

[vocab_size + 1, embedding_size]

here vocab_size simply refers to the number of lines in the vocabulary file, e.g. for you source file: [2151 + 1, 512]. No need to look further.

5

It’s true. Thank you a lot!