OpenNMT Forum

How to add image features in onmt-py

How can I use onmt to cancat the output of the encoder with the processed image features as the input of the decoder?The image is a description of the src,in order to obtain better translation performance.

You’ll probably have to add a custom field to your examples with these additional features – during preprocessing.
And then you’d need to adapt some things in the inputters as well as the decoder code.
You can take a look at this PR for a relatively straightforward example on how to add a field and use it down the line.

In onmt tf there is a multi features transformer example.
However, the example is based on text, so you have to change the code appropriately for the image.