Model parameters for back translation and Examples

Hi @guillaumekln

I see the OpenNMT-tf supports back translation and lot many users are interested in this.
I’m slightly confused with the latest version what new commands and parameters I need to change to make it work, I did not see any clear example of how to go about doing this therefore the confusion.

Do I separately pass the monolingual data file, if not, what pre-processing I need to do append it to training data.

Any additional model parameters I need to set - I do see some config parameters like “freeze_layers”, “sampling_topk” , “decoding_noise” etc. available ?

Thanks !

1 Like

Hi,

All NMT systems support back translation as it just a comibination of training and translating: training a model on the reverse direction and translating a monolingual corpus to generate new corpus. Are you looking for a script that does this workflow for you? There is no such thing in OpenNMT-tf.

The parameters “sampling_topk” and “decoding_noise” are some translation parameters that were proposed in the following paper:

They can be used to improve back translation results but are not required.

No problem. I can run those steps, how do i get started. Below is my thinking -
Let’s say I’m building for English-Spanish. I have a lot of Spanish monolingual corpus. Do I just run inference on this monolingual corpus using a pre-trained Spanish-English model to get synthetic sentences. ? At what point in the process will I need to set the mentioned back-translation parameters. ?

Thanks !

Yes.

When running the inference on the monolingual corpus.

Great. This is will take a long time to run if we have millions of sentences. Apart from running on GPU and batching, any thoughts on fast forwarding this process.

Nice. Any suggestions on what best parameter values to set , below is my guess -

freeze_layers:
- "encoder/layers/0"
- "decoder/output_layer"

beam_width: 1

decoding_noise:
- dropout: 0.1
- replacement: [0.1, ⦅unk⦆]
- permutation: 3

replace_unknown_target: false

Any other config parameters in relation to BT I need to consider. ?

Thanks !

Not a lot of magic tricks but beam_width: 1 is usually enough for back translation.

Not sure where you found that freeze_layers was related to BT. To get started, I’d say to only set beam_width: 1 for now.

:slight_smile: Thanks. I will run and let you know how it goes.