Changing training criteria of Seq2Seq model

gmkim90 · March 6, 2017, 7:09am

I am planning to change training criteria for Seq2Seq model.
For example, GAN would be one example. In this scenario, Seq2Seq (=Generator) model is trained towards maximizing Discriminator’s output.

Is there any example tutorial/recipes related to changing model’s training criteria?
If not, could you tell me which files are related source files?

I really appreciate any comments

guillaumekln · March 6, 2017, 8:42am

The current training criterion is defined in this file:

github.com

OpenNMT/OpenNMT/blob/master/onmt/modules/ParallelClassNLLCriterion.lua

--[[
  Define parallel ClassNLLCriterion.
--]]
local ParallelClassNLLCriterion, parent = torch.class('onmt.ParallelClassNLLCriterion', 'nn.ParallelCriterion')

function ParallelClassNLLCriterion:__init(outputSizes)
  parent.__init(self, false)

  for i = 1, #outputSizes do
    local nll = self:_addCriterion(outputSizes[i])
    if i == 1 then self.mainCriterion = nll end
  end
end

function ParallelClassNLLCriterion:_addCriterion(size)
  -- Ignores padding value.
  local w = torch.ones(size)
  w[onmt.Constants.PAD] = 0

  local nll = nn.ClassNLLCriterion(w)

This file has been truncated. show original

and used in the Seq2Seq model here:

github.com

OpenNMT/OpenNMT/blob/master/onmt/Seq2Seq.lua#L50


    structural = 0
  }
},
{
  '-pre_word_vecs_enc', '',
  [[Path to pretrained word embeddings on the encoder side serialized as a Torch tensor.]],
  {
    valid = onmt.utils.ExtendedCmdLine.fileNullOrExists,
    init_only = true
  }
},
{
  '-pre_word_vecs_dec', '',
  [[Path to pretrained word embeddings on the decoder side serialized as a Torch tensor.]],
  {
    valid = onmt.utils.ExtendedCmdLine.fileNullOrExists,
    init_only = true
  }
},
{
  '-fix_word_vecs_enc', false,