Fine-tune OpenNMT model on domain-specific data

Support
1

I am trying to fine-tune the OpenNMT model on domain-specific data. I have used sentence piece to tokenize data with a subword segmentation model.

params:
freeze_layers:
- “encoder/layers/0”
- “decoder/output_layer”

There are 260 trained layers, and the untrained layer is 0. I want to freeze 5 to 10 layers to fine-tune the model. How can I freeze a few layers of both the embeddings?

2

You should be able to freeze the embeddings with the following configuration:

params:
  freeze_layers:
    - examples_inputter
3

Thanks for the reply.

I have a question related to this. As I have gone through the model fine-tuning, I noticed the missing layer’s name.

model consist of 260 trained layer.
total encoder layer = 16 * 6 = 96
total decoder layer = 26 * 6 = 156
examples_inputter = 2
decoder/output_layer = 2
total = 256 layers

How can I retrieve the 4 missing layers using the config file?
What are the 4 missing layer’s names?

params:
freeze_layers:
- “examples_inputter”
- “encoder/layers/0”
- “encoder/layers/1”
- “encoder/layers/2”
- “encoder/layers/3”
- “encoder/layers/4”
- “encoder/layers/5”
- “decoder/layers/0”
- “decoder/layers/1”
- “decoder/layers/2”
- “decoder/layers/3”
- “decoder/layers/4”
- “decoder/layers/5”
- “decoder/output_layer”

4

I think it’s “encoder/layer_norm” and “decoder/layer_norm”.

But why do you want to freeze all layers?

5

Hi,
Thank you for your prompt reply. I don’t want to freeze all the layers. I am trying to fine-tune the model for domain adaption, and I was checking the different layer’s names to fine-tune the model.

Could you please suggest what layer’s can I freeze to fine-tune the model for domain adaption?

6

I don’t have an answer but you can check out this related paper:

1 Like