Encoder Output (Embedding) from Trained OPENNMT-TF Model

rl19 · March 26, 2021, 6:31pm

Hi there,

I have a trained opennmt-tf model and would like to get the embedding of the input sequence, in other words, the outcome of the encoder. Is there an easy way to do this.

Thank you in advance.
-RL

guillaumekln · March 29, 2021, 7:55am

Hi,

You would need to edit the code. The encoder output is returned here:

github.com

OpenNMT/OpenNMT-tf/blob/v2.17.1/opennmt/models/sequence_to_sequence.py#L168



def build(self, input_shape):
    super().build(input_shape)
    if EmbeddingsSharingLevel.share_target_embeddings(self.share_embeddings):
        self.decoder.reuse_embeddings(self.labels_inputter.embedding)

def call(self, features, labels=None, training=None, step=None):
    # Encode the source.
    source_length = self.features_inputter.get_length(features)
    source_inputs = self.features_inputter(features, training=training)
    encoder_outputs, encoder_state, encoder_sequence_length = self.encoder(
        source_inputs, sequence_length=source_length, training=training
    )

    outputs = None
    predictions = None

    # When a target is provided, compute the decoder outputs for it.
    if labels is not None:
        outputs = self._decode_target(
            labels,

If you want to write the embedding somewhere, you could add the encoder output in the predictions dictionary, and then update the method print_prediction.