OpenNMT-py 3 Better Transformer

martin_bombin · November 22, 2022, 5:36pm

Does the new version of OpenNMT-py (3) includes new PyTorch 1.12 Better Transformer (A BetterTransformer for Fast Transformer Inference | PyTorch) that accelerates inference time?

guillaumekln · November 24, 2022, 1:24pm

As far as I know the new version does not include these layers.

@vince62s Any plans to integrate the nn.modules listed in this blog post?

Note that OpenNMT-py models can be converted to CTranslate2 which is our take on accelerated inference. CTranslate2 already implements these optimizations, and more!

vince62s · November 24, 2022, 1:59pm

While working on v3.0 I tested extensively with the implementation of this function

github.com

pytorch/pytorch/blob/master/torch/nn/modules/activation.py#L1153


      
              if key is value:
                  if query is key:
                      query = key = value = query.transpose(1, 0)
                  else:
                      query, key = [x.transpose(1, 0) for x in (query, key)]
                      value = key
              else:
                  query, key, value = [x.transpose(1, 0) for x in (query, key, value)]
          
          
if not self._qkv_same_embed_dim:
              attn_output, attn_output_weights = F.multi_head_attention_forward(
                  query, key, value, self.embed_dim, self.num_heads,
                  self.in_proj_weight, self.in_proj_bias,
                  self.bias_k, self.bias_v, self.add_zero_attn,
                  self.dropout, self.out_proj.weight, self.out_proj.bias,
                  training=self.training,
                  key_padding_mask=key_padding_mask, need_weights=need_weights,
                  attn_mask=attn_mask, use_separate_proj_weight=True,
                  q_proj_weight=self.q_proj_weight, k_proj_weight=self.k_proj_weight,
                  v_proj_weight=self.v_proj_weight, average_attn_weights=average_attn_weights)
          else:

but it did not bring any improvement compared to our implementation which already optimized.

However, the rigth anwser is: CTranslate2 is the best inference solution at this time for any kind of transformer model.