What's the use of `coverage` in the forward pass for GlobalAttention?

nikhilweee · May 31, 2018, 7:03am

OpenNMT/OpenNMT-py/blob/0ecec8b4c16fdec7d8ce2646a0ea47ab6535d308/onmt/modules/GlobalAttention.py#L128-L143


def forward(self, input, memory_bank, memory_lengths=None, coverage=None):
    """


    Args:
      input (`FloatTensor`): query vectors `[batch x tgt_len x dim]`
      memory_bank (`FloatTensor`): source vectors `[batch x src_len x dim]`
      memory_lengths (`LongTensor`): the source context lengths `[batch]`
      coverage (`FloatTensor`): None (not supported yet)


    Returns:
      (`FloatTensor`, `FloatTensor`):


      * Computed vector `[tgt_len x batch x dim]`
      * Attention distribtutions for each query
         `[tgt_len x batch x src_len]`
    """

The docstring says that coverage is not supported yet. I wonder what was the intention originally? I also wonder what’s the use of the code block below. Can someone provide a reference?

github.com

OpenNMT/OpenNMT-py/blob/0ecec8b4c16fdec7d8ce2646a0ea47ab6535d308/onmt/modules/GlobalAttention.py#L162-L165


if coverage is not None:
    cover = coverage.view(-1).unsqueeze(1)
    memory_bank += self.linear_cover(cover).view_as(memory_bank)
    memory_bank = self.tanh(memory_bank)