Hi,
Is there any predefined function in OpenNMT for gradient checking?
I’ve developed a new encoder inside OpenNMT and want to be sure that it works well. If there is no gradient check function in the tool how can I write one?
I tried to write it by getting all parameters from my decoder and the loss value from the decoder, but I couldn’t, because if I’m not wrong, there is some randomness in decoder that produces different output and hence very different loss value each time. I appreciate your help.
Hi,
We don’t provide such function because we usually do not compute gradients by hand but rely on existing nn modules.
If you have custom gradient computations, you should take a look at nn.Jacobian available in torch/nn.
Thanks for the response.
In fact, I want to be sure that the whole process is correct. For example in the default encoder the network is cloned for each timestep. I do the same based on the structure of my input data, but, as the structure of data is not linear and somehow complicated I’m not sure if I call the right cloned network with the right parameters each time or not. My expectation doesn’t match the output of the model and I couldn’t find any bug in the code. Do you have any suggestion how I can check the whole process works well or not?