Model parameters for back translation and Examples

[mayub (Mohammed Ayub)]

Hi @guillaumekln

I see the OpenNMT-tf supports back translation and lot many users are interested in this.
I’m slightly confused with the latest version what new commands and parameters I need to change to make it work, I did not see any clear example of how to go about doing this therefore the confusion.

Do I separately pass the monolingual data file, if not, what pre-processing I need to do append it to training data.

Any additional model parameters I need to set - I do see some config parameters like “freeze_layers”, “sampling_topk” , “decoding_noise” etc. available ?

Thanks !

[guillaumekln (Guillaume Klein)]

Hi,

All NMT systems support back translation as it just a comibination of training and translating: training a model on the reverse direction and translating a monolingual corpus to generate new corpus. Are you looking for a script that does this workflow for you? There is no such thing in OpenNMT-tf.

The parameters “sampling_topk” and “decoding_noise” are some translation parameters that were proposed in the following paper:

Understanding Back-Translation at Scale

They can be used to improve back translation results but are not required.

[mayub (Mohammed Ayub)]

Are you looking for a script that does this workflow for you? There is no such thing in OpenNMT-tf.

No problem. I can run those steps, how do i get started. Below is my thinking -
Let’s say I’m building for English-Spanish. I have a lot of Spanish monolingual corpus. Do I just run inference on this monolingual corpus using a pre-trained Spanish-English model to get synthetic sentences. ? At what point in the process will I need to set the mentioned back-translation parameters. ?

Thanks !

[guillaumekln (Guillaume Klein)]

Do I just run inference on this monolingual corpus using a pre-trained Spanish-English model to get synthetic sentences. ?

Yes.

At what point in the process will I need to set the mentioned back-translation parameters. ?

When running the inference on the monolingual corpus.

[mayub (Mohammed Ayub)]

Do I just run inference on this monolingual corpus using a pre-trained Spanish-English model to get synthetic sentences. ?

Yes.

Great. This is will take a long time to run if we have millions of sentences. Apart from running on GPU and batching, any thoughts on fast forwarding this process.

At what point in the process will I need to set the mentioned back-translation parameters. ?

When running the inference on the monolingual corpus.

Nice. Any suggestions on what best parameter values to set , below is my guess -

freeze_layers:
- "encoder/layers/0"
- "decoder/output_layer"

beam_width: 1

decoding_noise:
- dropout: 0.1
- replacement: [0.1, ｟unk｠]
- permutation: 3

replace_unknown_target: false

Any other config parameters in relation to BT I need to consider. ?

Thanks !

[guillaumekln (Guillaume Klein)]

Apart from running on GPU and batching, any thoughts on fast forwarding this process.

Not a lot of magic tricks but beam_width: 1 is usually enough for back translation.

Any suggestions on what best parameter values to set

Not sure where you found that freeze_layers was related to BT. To get started, I’d say to only set beam_width: 1 for now.

[mayub (Mohammed Ayub)]

Not sure where you found that freeze_layers was related to BT. To get started, I’d say to only set beam_width: 1 for now.

Thanks. I will run and let you know how it goes.