Language model inference

[vince62s (Vincent Nguyen)]

Just a reminder

It would be great to support LM inference.

Output may be:

1. next word prediction
2. text generation
3. log likelihood for rescoring

Cheers.

[jean.senellart]

for reference - tool documented there: http://opennmt.net/OpenNMT/options/lm/ - and see also this thread:

Language Model scorer and sampler

Hello all, I have commited tools for language models - you can now use lm.lua to score and sample corpus with language models. It supports text with features. To try it: train a language model: th preprocess.lua -data_type monotext -train train.tok -valid valid.tok -save_data datalm if you use features, activate time_shift_feature false. train with -model_type lm score with a language model: th lm.lua score -model model.t7 -src t will generate output.txt with perplexity by sen…

[vince62s (Vincent Nguyen)]

Jean,
Did you run some benchmark for this on your end ?
PTB ?