How to add extra information to the Transformer

I was wondering how extra information can be passed to the Transformer during the training phase. I guess there were some features in the previous version of OpenNMT-py. However, as far as I know this functionality was removed in v2.0. Is there any way I can add information, like positional encoding is added to input embeddings?

What did the former features do (concat/add…) with the input embeddings? Which do you think it is the best approach to add extra information about tokens (POS tagging, classification labels…)?


Hi! Currently, features can be implemented in OpenNMT-tf only (example for multi-feature Transformer) This discussion might help.

All the best,

At the end of the day, are the features files added in the original text as tags?

Or they are dealt with in a completely different way by opennmt?

No, tags is a different approach. You can check this paper under Section: 4.2 Word Feature for more details. The paper also explores both approaches; so the difference should be clear.

Kind regards,

1 Like