LSTM or Transformer?

[tuankstn (Phan Anh Tuan)]

I am building a machine translator. My dataset has 90,000 sentence pairs. Should I use Transformer to train the machine translator with such a small dataset? Will it be better than using LSTM?
Thank you.

[ymoslem (Yasmin Moslem)]

If this is a research work, I assume you can try both. Consider also the possibility of fine-tuning a (pretrained) multilingual model, especially if your language pair is similar to some high-resource languages.

Check also some posts on the forum about low-resource languages, including this one.

Transformer Parameters for Low-Resource Machine Translation

I have been conducting an experiment on a small dataset of 30k segments when I noticed that a 3-layer Transformer starts to give meaningful translations faster than a 6-layer Transformer. This made me remember discussions about how Transformer parameters might differ for low resource NMT. Here are a few interesting papers I found on the topic: Obviously, sub-wording helps, and common approaches like Transfer Learning between similar languages, and data augmentation with monolingual Back-…

Kind regards,
Yasmin

[tuankstn (Phan Anh Tuan)]

Thank you so much for your help.

[argosopentech]

I would also recommend fine tuning a model for a larger similar language if possible. I trained an Azerbaijani model by fine tuning from Turkish with good results.