Domain adaptation: Unable to converge

portia · April 19, 2019, 12:10am

Hi,

I am using opennmt-tf to implement domain adaptation. When I train the out-of-domain model everything seems fine and the test scores are reasonably good (about 55 BLEU), but when I start fine tuning (after updating the vocabulary by replacement, not merging), there’s a point in the training when the loss value switches from decreasing to increasing. After evaluating my in-domain test data, the scores (BLEU, NIST, TER, CHARCUT) are absolutely terrible.

I first thought it could be a problem of overfitting, but I’m providing about 9000 validation pairs. Then I thought the problem could be using a too big learning rate, but I am using noam_decay, so I don’t understand why, after some training, the loss is growing.

I am using a TransformerBigFP16 using a BPE vocabulary of 16000 tokens (by means of SentencePiece). The out of domain dataset is about 3.5M and the in-domain dataset is about 350K pairs.

I ran out of ideas, so any suggestion would be much appreciated.
Thanks for your help

I paste below my configuration in case it could be of any help
data:
eval_features_file: gen_bpe16_enfr_en_training_set_val.txt
eval_labels_file: gen_bpe16_enfr_fr_training_set_val.txt
source_words_vocabulary: gen_bpe16_enfr_en_vocab.txt
target_words_vocabulary: gen_bpe16_enfr_fr_vocab.txt
train_features_file: gen_bpe16_enfr_en_training_set_train.txt
train_labels_file: gen_bpe16_enfr_fr_training_set_train.txt
eval:
batch_size: 32
eval_delay: 18000
exporters: last
infer:
batch_size: 32
bucket_width: 5
model_dir: run_gen_bpe16_enfr/
params:
average_loss_in_time: true
beam_width: 4
decay_params:
model_dim: 512
warmup_steps: 4000
decay_type: noam_decay_v2
label_smoothing: 0.1
learning_rate: 2.0
length_penalty: 0.6
optimizer: LazyAdamOptimizer
optimizer_params:
beta1: 0.9
beta2: 0.998
score:
batch_size: 64
train:
average_last_checkpoints: 5
batch_size: 3072
batch_type: tokens
bucket_width: 1
effective_batch_size: 25000
keep_checkpoint_max: 20
maximum_features_length: 100
maximum_labels_length: 100
sample_buffer_size: -1
save_summary_steps: 100
train_steps: 500000

guillaumekln · April 19, 2019, 10:05am

Hi,

Can you post the log containing the training loss?

portia · April 19, 2019, 12:25pm

Do you mean the loss progress I sent below or the whole log? It’s a pretty long one…

portia · April 19, 2019, 1:03pm

Find here the complete log. Thanks!

guillaumekln · April 19, 2019, 5:35pm

The loss is too low. I suspect a vocabulary issue. How did you generate the vocabulary that you then used for replacement?

portia · April 19, 2019, 7:08pm

Hi Guillaume,

1st of all: Thanks very much for your help

I used SentencePiece as you suggested to other people in the forum
I created the subword model by doing:

spm_train --input=my_generic_normalized_file --model_type=bpe --model_prefix=gen --vocab_size=16000 --character_coverage=1.0 --user_defined_symbols=<list_of_normalized_tokens>

where <list_of_normalized_tokens> are tokens looking like ｟CODE0｠｟CODE1｠…｟NUMBER0｠｟NUMBER1｠…｟URL0｠｟URL1｠…
These tokens come from a previous normalization stage where the original text is preprocessed to replace URL’s, emails, codes, numbers, etc… by these tokens. All these things are never translated into something different, so I think considering them atomic symbols make sense to “alleviate” the neural learning process.
Once I have both BPE models (source and target language), I encode the normalized files into BPE-encoded files ready for training.

I follow the exact same process to create the in-domain BPE model, to obtain the BPE-encoded files ready for fine-tuning training. The only difference is the source data, which this time contains just in-domain normalized data.
The generic dataset does not contain in-domain data at all, I don’t know if this approach is correct.

The initial in-domain model is obtained by updating the vocabulary with replacement (not merge) like this:

onmt-update-vocab --model_dir run_generic_bpebig16_deen/ --output_dir run_indomain_bpebig16_deen/ --src_vocab generic_bpebig16_vocab_deen_de.txt --tgt_vocab generic_bpebig16_vocab_deen_en.txt --new_src_vocab indomain_bpebig16_vocab_deen_de.txt --new_tgt_vocab indomain_bpebig16_vocab_deen_en.txt --mode replace

And then using the in-domain data to follow the training. I am trying as much as I can to follow the canonical approach to domain adaptation.

guillaumekln · April 20, 2019, 9:06am

I would suggest:

Using the initial BPE models to encode the in-domain data
Using the “merge” vocabulary replacement mode (don’t forget to update the configuration, the new vocabularies are generated in the output_dir)

Can you give it a try?

portia · April 20, 2019, 7:13pm

Hi Guillaume,

Indeed I’ll try both (In fact it’s WIP)
Would you say the dataset for the fine-tuning stage should contain just in-domain data, or Would you recommend having some amount of generic data too?