OpenNMT Forum

Problems with learning rate decay


(Fermat97) #1

When I fix the -start_decay_steps 6084888 and -decay_steps 3042444 with -decay_method noam then I get this error:

RuntimeError: value cannot be converted to type float without overflow: (-7.65404e-27,1.25e-10)


/OpenNMT-py/onmt/utils/", line 281, in step
python3.7/site-packages/torch/optim/", line 107, in step, exp_avg, denom)

I use Pytorch 1.0 and adam optimizer with learning rate 0.0001. Any idea?

(Guillaume Klein) #2

First, these settings -start_decay_steps 6084888 -decay_steps 3042444 are not used in the Noam decay schedule.

What is your current training step?

(Fermat97) #3

It is -train_steps 3100000000.

(Guillaume Klein) #4

The learning rate becomes very small when going that far in the training. Maybe you want to switch to a constant learning rate at some point?