How to caculated the PRED AVG SCORE and the PRED SCORE

baoloc97 · September 27, 2021, 2:58am

when i finish a shard in translate it shows that pred avg score is -0.5 but when I look into each sentence its show from -5 → -14 in PRED SCORE. I think Pred avg score = sum(pred score) / count(pred score) but its seem don’t like that . Can some one give me how to calculated this number ?

ymoslem · September 27, 2021, 5:26pm

Looking up it in the GitHub repository, I found this line:

github.com

OpenNMT/OpenNMT-py/blob/4ccf5821b86bd67644c446fb953ae03dd0c463b1/onmt/translate/translator.py#L559

    
      
              full_tgt = torch.cat((bos_tensor, paded_tgt), dim=-1)
              batched_nbest_predict = full_tgt.view(
                  len(predictions), -1, full_tgt.size(-1)
              )  # (batch, n_best, tgt_l)
              return batched_nbest_predict
          
          
def _report_score(self, name, score_total, words_total):
              if words_total == 0:
                  msg = "%s No words predicted" % (name,)
              else:
                  avg_score = score_total / words_total
                  ppl = np.exp(-score_total.item() / words_total)
                  msg = "%s AVG SCORE: %.4f, %s PPL: %.4f" % (
                      name,
                      avg_score,
                      name,
                      ppl,
                  )
              return msg
          
          
def _decode_and_generate(

Still, I believe BLEU and/or COMET would make more sense for translation evaluation.