Small differences in scores between architectures / seed -1

Support
1

Hi everyone,

Some time ago the seed was set by default to 3435. But a couple of days ago I ran some experiments with the same data and version of OpenNMT on another machine and noticed a slide difference in the metrics at the end (both RNN and Transformer). Then I saw that the default seed is now -1; does that imply random? Can version of CUDA and the nvidia driver have an impact?

Thanks,
Dimitar

2

Not sure why this was changed, but no it’s not random, it’s just -1 instead of 3435. (@Zenglinxiao it was one of your commits it seams, any rationale for the change?)

IIRC, the seed is intended to allow reproducibility through time on a same machine, but the same seed on different machine does not.

3

Thanks @francoishernandez.
That’s good to know. If that is indeed the case then I’d need to dig deeper to understand why the differences in results. Maybe it has to do with the fact that in the one case I use 4 GPUs and in the other 3 GPUs in parallel.Any way, thanks a lot.

4

Before version 2.0, we actually have three different seed value for preprocess (3435), train(-1) and translate(829). I kept the training one as the new default since we removed preprocess and deterministic operations are often slower.

5

Maybe it has to do with the fact that in the one case I use 4 GPUs and in the other 3 GPUs in parallel.

Yes, if you don’t use the exact same configuration the results will differ.

6

Thank you.