After some testing, I have the feeling that Bleu is not the best metric for NMT.
Indeed, that could be just an impression, (or a wish ) but when comparing some SMT and NMT results, we get comparable Bleu score, however NMT phrase seem better constructed.
For clarification, I would like to discuss here how PPL and Prediction are computed in onmt.
My understanding is that at training time:
- we have a batch size of n_s sentences made of n_w words in total. (n_s is the same for training batch size and valid data batch size)
- at each “report iteration” we compute the training perplexity for N batches, ie N x n_s words.
This per-word perplexity is computed according to the model weights as the log likelihood sum of each predicted word.
- at the end of each “epoch” we compute the validation perplexity on the total words of the valid data set.
At inference time:
- we compute a “PRED SCORE” : log likelihood of the target words according to the model (what scale ?)
for instance when I see PRED AVG SCORE: -0.67, PRED PPL: 1.96 - I can’t do the math.
EDITED: Now I can exp(0.67) = 1.96
- optionally if Gold Data is provided, we compute Gold score and Gold PPL (“according to the model”): how are they computed ?
To get back to the main question (Bleu, PPL, …) I was wondering if we could try to simultaneously have a target side LM that could be used to measure the PPL_lm of the model output which could give a metric on how relevant the sentence is based on the language only.
My point is that a n-gram based only metric is far from being accurate and I have the feeling that it is not bridging the same gap that we have between a n-gram LM and a lstm-LM.