Gradient checking

(Mehdi) #1

Is there any predefined function in OpenNMT for gradient checking?
I’ve developed a new encoder inside OpenNMT and want to be sure that it works well. If there is no gradient check function in the tool how can I write one?
I tried to write it by getting all parameters from my decoder and the loss value from the decoder, but I couldn’t, because if I’m not wrong, there is some randomness in decoder that produces different output and hence very different loss value each time. I appreciate your help.

(Guillaume Klein) #2


We don’t provide such function because we usually do not compute gradients by hand but rely on existing nn modules.

If you have custom gradient computations, you should take a look at nn.Jacobian available in torch/nn.

(Mehdi) #3

Thanks for the response.
In fact, I want to be sure that the whole process is correct. For example in the default encoder the network is cloned for each timestep. I do the same based on the structure of my input data, but, as the structure of data is not linear and somehow complicated I’m not sure if I call the right cloned network with the right parameters each time or not. My expectation doesn’t match the output of the model and I couldn’t find any bug in the code. Do you have any suggestion how I can check the whole process works well or not?