TL;DR I would like to append an extra feature to each word in the source document, which is the embedding of the entire document given by a specific pre-trained model. Any advice on where to make code changes?
I was trying to replicate Harrison et al. (2018), https://arxiv.org/abs/1809.02637, which is a question generation paper (input: a wikipedia passage from SQuAD; output: a question regarding that passage). In the paper, they train a specific model for embedding the passage, which is supposed to give a question-focused sentence embedding. Each word token in the full encoder-decoder network is the concatenation of its word embedding, extra features (NER, case, …) and the full passage embedding that is output of the aforementioned model.
In the paper they mention they implemented this architecture with OpenNMT-py and PyTorch. I see how to concatenate categorical features to words (http://opennmt.net/OpenNMT/data/word_features/) but what I need here is to run each source sentence through a pre-trained model, and concatenate the output to each of its words.
I have worked before with PyTorch, but never with OpenNMT; I was starting to dive into its code: do you have any advice on how to approach the problem?
Thank you very much!