Transformer with really extrasmall dataset

[SamuelLacombe]

Hello,

I’ve been playing around with both opennmt-py and opennmt-tf. I have been having really good results with SGD in opennmt-py with extremely small dataset… below 200k. But with transformer in opennmt-tf the results are extremely bad.

Is it expected from transformer to behave pretty bad on externally small dataset? Or is there any parameter i could fine tune to help. I’m currently using the default one.

I have an average of -6 predict score with SGD and an average -20 predict score with transformer

Thank you!

[guillaumekln (Guillaume Klein)]

SGD is an optimizer. Do you mean you have good results with a small LSTM compared to a Transformer model?

If yes, then the result is expected. The Transformer with its default configuration requires large datasets.

[SamuelLacombe]

Sorry, yes your right. I meant LSTM. I will probably have to change the optimizer for small dataset…

If anyone has suggestion for a starting point parameter for small dataset let me know. It will surely help out👌

[SamuelLacombe]

If anyone face the same issue… Here some documentation I found about it.

https://datascience.stackexchange.com/questions/80483/based-on-transformer-how-to-improve-the-text-generation-results

[ymoslem (Yasmin Moslem)]

Dear Samuel,

@SamuelLacombe

extremely small dataset… below 200k

First, by definition, a 100-200k dataset is not “extremely” small. It is just small, and there are approaches to make it work. You might want to check the following resources:

An Empirical Comparison of Domain Adaptation Methods for Neural Machine...

Fast Domain Adaptation for Neural Machine Translation

Successful Domain Adaptation with OpenNMT-py

Nice post! This may be a case of “catastrophic forgetting”. Usually the retraining should be a combination of in-domain and generic data.

Domain Adaptation Techniques

Domain Adaptation can be useful when you want to train a Machine Translation model, but you have only too limited data. There are several approaches of Domain Adaptation including: Incremental Training / Re-training: So you have a big pre-trained model trained on a big corpus, and you continue training it with the new data from the small corpus. Ensemble Decoding (of two models): You have two models and you use both models during translation. Combining Training Data: You merge the two c…

Terminology-Aware_Sentence_Mining_for_NMT_Domain_Adaptation.pdf

Kind regards,
Yasmin

[SamuelLacombe]

Thank you, Yasmin.

I will have a look into that, but the reason I said “extremely small” is that I have some languages which have absolutely nothing on the internet and I have only about 15k to 50k sentences (before data augmentation). So using an existing model is not an option. Yet, I was able to generate decent model that would give me fair to good translation 30% of the time with LSTM.

I might not be able to accomplish that level of accuracy with Transformer for these languages. I will go through the documentation you posted and see what I can come up with!

[SamuelLacombe]

If anyone happens to come upon this post… here is some additional documentation I found that helped me out:

https://aclanthology.org/2020.coling-main.304.pdf

[SamuelLacombe]

I noticed that there is a TransformerTiny in the catalogue of transformer… is there something missing to use it?

[guillaumekln (Guillaume Klein)]

This model is not included in a released version yet.