Output from CTranslate and OpenNMT translate differ

Support
1

Hi,

I am testing out OpenNMT and CTranslate with a trained model posted on the OpenNMT website (the English summary one trained on CPU). Curiously I got different outputs from using OpenNMT vs CTranslate, and I am wondering why that is and if there is some model parameters that I needed to set explicitly.

I ran OpenNMT as follows:

th translate.lua -model model/textsum_epoch7_14.69_release.t7 -src data/short-src-test.txt -output short-sum.txt

I ran CTranslate as follows:

./build/cli/translate --model …/model/textsum_epoch7_14.69_release.t7 --src …/data/short-src-test.txt --tgt …/short-sum.txt

The short-src-test.txt contains the first 115 lines of test inputs from data/src-test.txt that came with OpenNMT. The outputs short-sum.txt produced by both systems are quite different.

Based on the reading of document / code, I believe that the beam search width used by both systems is 6 by default. Is there other configuration parameters that I might be missing? Or, is there an explanation why I am seeing different results?

Any help / pointers are greatly appreciated!

2

Hi,

The beam search implementation is a bit different in OpenNMT and CTranslate. It is likely the cause and can be more or less visible depending on the model.

I had on my list for quite some time to rewrite CTranslate’s beam search to make it behave like OpenNMT. I should get it done.

3

Hi,

Thanks for the quick response!
I see. Upon trying with beam_size 1, it does seem to output mostly consistently results (a couple of queries differed slightly, but not by much).

I would love to know if / when CTranslate is updated to output the same thing as OpenNMT, so please keep me posted; thanks!