How to change model dimension

Support
1

Hi,
I’m new in tensorflow and I have little problem during in-domain adaptation.
I trained my general model with source_embedding:de.wiki.bpe.d300, so my general model has 300 dimension.
Now when I want to retrain my model with new vocab, I don’t set embedding anymore (I supposed it’s not necessary anymore) I get error which there is dimension diferences between these two model. So if I set previouse embedding, everything works correctly.
How can I set the dimension of my new model to 300? or is it necessary to use previus embedding while i want to adapt my indomain model?

thenks
Hadis

2

Hi,

When an embedding file is configured, the embedding size is inferred from this file, otherwise the model defines a default size. For example, the default embedding size for a TransformerBase model is 512.

If you don’t change the vocabulary size, you can just keep the same embedding configuration for a retraining. The pretrained embeddings are only used for the weight initialization and will be overriden when loading a checkpoint.

However, if the vocabulary size is different, you may need to define a custom model with the correct embedding size. For example, you can create a file my_transformer.py with this content:

import opennmt

class MyTransformer(opennmt.models.Transformer):
    def __init__(self):
        super().__init__(
            source_inputter=opennmt.inputters.WordEmbedder(embedding_size=300),
            target_inputter=opennmt.inputters.WordEmbedder(embedding_size=300),
            num_layers=6,
            num_units=512,
            num_heads=8,
            ffn_inner_dim=2048,
            dropout=0.1,
            attention_dropout=0.1,
            ffn_dropout=0.1)

model = MyTransformer

and then use onmt-main --model my_transformer.py [...]

2 Likes