How to add extra information to the Transformer

[anderleich]

Hi,
I was wondering how extra information can be passed to the Transformer during the training phase. I guess there were some features in the previous version of OpenNMT-py. However, as far as I know this functionality was removed in v2.0. Is there any way I can add information, like positional encoding is added to input embeddings?

What did the former features do (concat/add…) with the input embeddings? Which do you think it is the best approach to add extra information about tokens (POS tagging, classification labels…)?

Thanks

[ymoslem (Yasmin Moslem)]

Hi! Currently, features can be implemented in OpenNMT-tf only (example for multi-feature Transformer) This discussion might help.

POS tagging/Word Features and BPE encoding tutorial needed for opennnmt python

Hi, The POS tagging tutorial is indeed outdated. It was written for OpenNMT-lua and RNN models, which allowed to embed the factors directly in the words of the training corpus. Also, it assumed no BPE was used. For this to work in the current OpenNMT-tf/py versions, the factors (POS tags in this case) need to be in a separate file, and this is straightforward: the tagger actually adds the POS tags in a separate file anyway, so you just need to omit the merging step (combine factors) in the tut…

All the best,
Yasmin

[SamuelLacombe]

At the end of the day, are the features files added in the original text as tags?

Or they are dealt with in a completely different way by opennmt?

[ymoslem (Yasmin Moslem)]

@SamuelLacombe

At the end of the day, are the features files added in the original text as tags?

No, tags is a different approach. You can check this paper under Section: 4.2 Word Feature for more details. The paper also explores both approaches; so the difference should be clear.

Domain Control for Neural Machine Translation

Kind regards,
Yasmin