Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task.
My application scenario is to train a generic model where users can upload a small amount of data and can customize a model for each user individually with a small number of parameters updated.
What’s the best way to achieve this?
Hello,
I believe the approach ModerMT is using would fit your needs.
First watch this video:
ModernMT approach explained you can jump at 7 mins… That’s where it’s start to be interesting.
Then here the link to the source code:
ModernMT open source code
If you want to see my thoughts about my test:
My thoughts about it
Hope this help. Cheer.
Note: I have not used their code, just tried their paid service.
Hi, If you search for fine tuning in this forum you will find discussions on fine tuning generic models. Outside the OpenNMT ecosystem you could also look at the Hugging Face pretrained models for which there are published fine tuning scripts. Success in fine tuning is very much a matter of trial & error. I am not convinced you get decent results with just a few epochs fine tuning.
Hello,
I didn’t think about it initially, but you could also train one model but use the same approach as when you do many languages to one. Instead of being language it would be each different individual or what ever “category” you needed.
The only thing, is having just one model means having the same parameters for all of them. But in the other hand you should save lots of training.
Best regards,
Samuel
Interesting discussion. I would like to refer you to this nice thesis by Rebecca Knowles, including a lot of information and references.
The main paper of ModernMT is Farajian et al. (2017)
Some time ago, I wrote a brief description about this paper:
Kind regards,
Yasmin