Hello, i am a beginner in OpenNMT and after i finished te training model, i want to compute the score BLEU but i have some difficulties for this .
So i want to have an example of computing score Bleu for the scoring?
Best regards
Thanks
Hi,
You can use the multi-bleu.perl script that is shipped with OpenNMT:
perl benchmark/3rdParty/multi-bleu.perl reference.txt < predictions.txt
Ok, I used this command but i receive this message
perl benchmark/3rdParty/multi-bleu.perl data/src-test.txt < pred.txt
Use of uninitialized value in division (/) at benchmark/3rdParty/multi-bleu.perl line 129, line 2737.
Use of uninitialized value in division (/) at benchmark/3rdParty/multi-bleu.perl line 129, line 2737.
Use of uninitialized value $CORRECT[3] in multiplication () at benchmark/3rdParty/multi-bleu.perl line 134, line 2737.
Use of uninitialized value $CORRECT[4] in multiplication () at benchmark/3rdParty/multi-bleu.perl line 134, line 2737.
BLEU = 0.00, 9.5/0.0/0.0/0.0 (BP=0.991, ration=0.991)
What is the problem ??
Hello - it is simpler to use the lua version of BLEU with the same syntax:
th tools/score.lua data/src-test.txt -scorer bleu < pred.txt
However the message your problem seems to be related to the fact that there is no 3-gram and 4-gram matching in your pred.txt file.
Yes the problem still persists because the command th. But I do not understand why there is no 3 or 4-gram in the pred.txt file because it was during the translation that this file was created.
th tools/score.lua data/src-test.txt -scorer bleu < pred.txt
0.00 BLEU = 0.00, 9.5/0.0/0.0/0.0 (BP=0.991, ratio=0.991, hyp_len=60849, ref_len=61376)
Not sure if this will solve your problem, but it looks like you’re comparing the predicted text with the source. I think you want target in place of source.
Are you using features ? Did you use files where they were removed ?
th tools/score.lua data/src-swahili.txt -scorer bleu < pred.txt
/usr/bin/luajit: cannot open tools/score.lua: No such file or directory
stack traceback:
[C]: in function ‘dofile’
/usr/lib/torch-trepl/th:149: in main chunk
[C]: at 0x564d519731d0
I later ran the cmd
perl tools/multi-bleu.perl data/src-swahili.txt < pred.txt
and got the the follwing output:
Use of uninitialized value in division (/) at tools/multi-bleu.perl line 139, line 5.
Use of uninitialized value in division (/) at tools/multi-bleu.perl line 139, line 5.
Use of uninitialized value in division (/) at tools/multi-bleu.perl line 139, line 5.
Use of uninitialized value in division (/) at tools/multi-bleu.perl line 139, line 5.
BLEU = 0.00, 0.0/0.0/0.0/0.0 (BP=1.000, ratio=1.261, hyp_len=29, ref_len=23)
