Problem with incremental/in-domain training

(Vincent Nguyen) #21


(Terence Lewis) #22

I thought I would report -in case it’s useful to others - that I’ve been specialising some models over the last few days. I have been launching using -train_from from Epoch 13 in my general model, using -update_vocab. It has generally taken me a further 13 epochs to get my model to change its beliefs and translate with the “new terminology” contained in my additional training material. An example would be “aanbestedingsdocument” which was previously translated (not wholly incorrectly) as “tender document” and with the “retrained” model is now translated with the preferred translation of “procurement document”. The successful retraining session which took 52000 “new” sentences lasted just under 2 hours on a machine with a GTX 1080Ti GPU.

(Vincent Nguyen) #23

Hi Terence,
this sound great. Out of curiosity what learning rate did you use during the additional 13 epochs ?

(Terence Lewis) #24

Hi Vincent,
Been away from my machines! Just got back and looked at the training log and see.
epoch 14 LR = 0.08751 decaying to
epoch 26 LR = 0.00050
Retraining had started from Epoch 1 with (LR = 1.0) only submitting the new material
and decay of 0.65 had started at Epoch 4. I used -train_from without -continue but was surprised to see training start from Epoch 1. However, I am pleased with the result as I now have an effective way of getting post-edited translations in memoQ into the model in a reasonably short time.