OpenNMT Forum

OpenNMT torch vs OpenNMT-tf

(Zuzanna Parcheta) #1

Hi, everyone. I wonder if someone did some comparison of both frameworks, OpenNMT torch and OpenNMT-tf. I did some experiments and results are surprising, because I get about 20 points of difference between them. The task is about grammatical error corrections so I basically check if corrected sentence (hypothesis) is the same as target sentence. My metric is the % of identic sentences over a total number of sentences. I get a much better score in torch using BPE, and the following configuration:
-rnn_size 128
-encoder_type rnn
-rnn_type LSTM
-end_epoch 60
-max_batch_size 50
-save_model ${folder}/models/
-layers 2
-dropout 0.3
-optim adam
-learning_rate 0.0002
-learning_rate_decay 1.0
-src_word_vec_size 128
-tgt_word_vec_size 128 \

than using OpenNMT-tf with BPE (I tried different models as NMT-medium, transformer with their default parameters, only varying updates number).

Has anyone done similar experiments comparing both backends?

0 Likes

(Terence Lewis) #2

Hi Suzanna, I’m planning to do “scientific” comparisons like you but at present am doing human evaluations as I know the two languages involved. What I can see is that OpenNMT-tf is producing much more natural-sounding, fluent translations than the Torch (Lua) based toolkit which I have been using for production. When I do a scientific comparison I will post the results as this topic interests me.

1 Like