Adding perplexity or negative log-likelihood to CTranslate

devinbostIL · June 22, 2017, 9:54pm

What would be required to add to CTranslate the prediction score that’s typically output by OpenNMT when run from the translate.lua script?

guillaumekln · June 23, 2017, 8:21am

Not much.

Would it be just part of the API or also the standard output? If the latter, what should be the format?

devinbostIL · June 23, 2017, 9:17pm

It would be helpful if there could be a parameter that we could pass that would give the output in the format provided by translate.lua, which is like:

[-4.37] How can I help you today?

where the score is in the square brackets.
I’d be happy to make the change on a fork and submit a pull request if someone can point me in the right direction.

guillaumekln · June 26, 2017, 12:58pm

It’s about passing the highest score from include/onmt/Translator.hxx to cli/translate.cc. It is either end_score[b] or max_score[b]:

github.com

OpenNMT/CTranslate/blob/master/include/onmt/Translator.hxx#L748


    if (with_input_feeding)
      input_feed = new_input_feed;
  }


  remaining_sents = new_remaining_sents;
}


// Build final translation by following the beam path for each batch.
std::vector<std::vector<std::string> > batch_tgt_tokens;
std::vector<std::vector<std::vector<std::string> > > batch_tgt_features;
std::vector<std::vector<std::vector<float> > > batch_attention;


for (size_t b = 0; b < batch_size; ++b)
{
  size_t start_k = best_k[b];
  size_t len = best_finished_at[b] + 1;
  if (end_score[b] > max_score[b]) // End score is the score of the top beam.
  {
    start_k = 0;
    len = end_finished_at[b] + 1;
  }

It could be done by extending the TranslationResult class.

Then you may need to call another Translator's method from translate.cc to access a TranslationResult object.