Setting a few custom hyperparameters and rest as defaults

Support
1

Hi Guillaume,

I am trying to use some custom optimizer hyperparameters and leave the rest as defaults for a predefined model. Is there a list of default values somewhere?

2

Hello,

Are you referring to the configuration field optimizer_params? Can you specify the optimizer you are using and which hyperparameters you want to set?

3

Yes I wish to use custome values under optimizer_params and sometime other hyperparameters under params. It is not necessary that all values are set by user everytime and in those cases i will use default. So I wanted to know if there’s a default-config.yml present for different predefined models?

For example:
Case1 : I wish to use Adam with default parameters but change learning_rate under params only
Case2 : I wish to use Adam with custom values for beta_2
Case3 : Use different optimizer altogether maybe SGD
etc… and so on and so forth

4

If you are using --auto_config, the default values depend on the model. For Transformer models, the automatic defaults are here:

5

Thanks! These are the defaults for ALL the predefined Trasnformer models, right? (Just confirming)

6

Yes, exactly.