Validation and Translation Perplexity


(Dimitar Shterionov) #1

Hello,

I am running some tests and trying to figure out something related to perplexity. During training, the validation set is used to validate each model and a perplexity score is generated.

During translation, also a score is generated (that’s the confidence score, according to the comments in the code). If I use the same model and the same set the validation perplexity score and the translation perplexity are different. Can someone say what perplexity refers to in both cases?

Thanks,
Dimitar


(Guillaume Klein) #2

Hi,

The validation perplexity is the perplexity of the validation’s true target data.
The translation perplexity is the perplexity of the model’s own prediction.

See also this topic:


(Dimitar Shterionov) #3

@guillaumekln,
thanks a lot for the prompt response.
Is there a way to compute the validation perplexity after the model is trained?

Thank you in advance.
Dimitar.


(Guillaume Klein) #4

Yes, you can provide the target data during translation with the option -tgt. Then the GOLD perplexity should be comparable with the validation perplexity.


(Dimitar Shterionov) #5

Yes, I checked it in the documentation meanwhile… sorry about the question :).

Cheers,
Dimitar.


(Dimitar Shterionov) #6

When computing the GOLD perplexity I get a score closer to the validation perplexity computed during training.

However, they are not the same and I would say they are rather different actually - the one is 7.27, the other is 10.19. I would expect some variation, but that is a bit too much I think. Is that normal? (The prediction perplexity is 2.14).

Thank you,
Kind regards,
Dimitar


(Guillaume Klein) #7

The preprocessing certainly dropped some validation sentences during training (due to length constraints) which are not filtered during translation. That’s the only additional difference I can think of right now.


(Dimitar Shterionov) #8

OK, Interesting,

Thank you.
Cheers,
Dimitar.