Bleu Script Throws OSError: [Errno 7] Argument list too long: '/bin/sh'

Hi everyone,
While doing inference and trying to compute the bleu score for a dataset that I’m using, I get the following error:

Traceback (most recent call last):
File “translate.py”, line 63, in
main(opt)
File “translate.py”, line 32, in main
attn_debug=opt.attn_debug
File “/workspace/model_lib/OpenNMTpy/onmt/translate/translator.py”, line 391, in translate
msg = self._report_bleu(tgt)
File “/workspace/model_lib/OpenNMTpy/onmt/translate/translator.py”, line 833, in _report_bleu
stdin=self.out_file, shell=True
File “/opt/conda/lib/python3.6/subprocess.py”, line 356, in check_output
**kwargs).stdout
File “/opt/conda/lib/python3.6/subprocess.py”, line 423, in run
with Popen(*popenargs, **kwargs) as process:
File “/opt/conda/lib/python3.6/subprocess.py”, line 729, in init
restore_signals, start_new_session)
File “/opt/conda/lib/python3.6/subprocess.py”, line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: ‘/bin/sh’

The command that I’m using to run inference is:
python translate.py -model models/zh-en/zh-en_step_200000.pt -src data/src-test.txt -tgt data/tgt-test.txt -output pred.txt --report_bleu -replace_unk -verbose

Any ideas on what is causing this? That location is at the place where it calls the multi_bleu.pl script and it works fine when I run translate.py without the bleu argument.

Not sure what the issue was with translate.py, but I found that you can run the multi_bleu.pl script manually with perl tools/multi-bleu.perl data/tgt-test.txt < predPost.txt which works fine.

Yes this is a current issue (#1429 & #1530). There is no consensus yet on the best way to handle this.
You can also use sacrebleu instead of the perl script.