How to add image features in onmt-py

TellMeWhy · March 25, 2020, 7:11am

How can I use onmt to cancat the output of the encoder with the processed image features as the input of the decoder?The image is a description of the src,in order to obtain better translation performance.

francoishernandez · March 26, 2020, 1:34pm

You’ll probably have to add a custom field to your examples with these additional features – during preprocessing.
And then you’d need to adapt some things in the inputters as well as the decoder code.
You can take a look at this PR for a relatively straightforward example on how to add a field and use it down the line.

park · March 29, 2020, 3:59pm

github.com

OpenNMT/OpenNMT-tf/blob/master/config/models/multi_features_transformer.py

"""Defines a Transformer model with multiple input features. For example, these
could be words, parts of speech, and lemmas that are embedded in parallel and
concatenated into a single input embedding.

The features are separate data files with separate vocabularies. The YAML
configuration file should look like this:

data:
  train_features_file:
    - features_1.txt
    - features_2.txt
    - features_3.txt
  train_labels_file: target.txt
  source_1_vocabulary: feature_1_vocab.txt
  source_2_vocabulary: feature_2_vocab.txt
  source_3_vocabulary: feature_3_vocab.txt
  target_vocabulary: target_vocab.txt
"""

import tensorflow as tf

This file has been truncated. show original

In onmt tf there is a multi features transformer example.
However, the example is based on text, so you have to change the code appropriately for the image.