Chinese to english translation many unknown words

Support
1

I’m trying to train a model for unsupervised translation from Chinese(simplified) to English. I collected a corpus of 28M sentences and tokenized the Chinese side by inserting spaces between words. But after training this model it seems to not ‘remember’ many words. Translating a random sample from a Chinese news site often works pretty well, but when I try to translate more random text my model often returns a high percentage of unks or it starts to repeat one or 2 words on the output.

Am I doing something massively wrong or is there a way to increase the translation memory of a model? I would greatly appreciate any pointers.

2

Do you tokenize the test data the same way you tokenized the training data?

3

Yes, and i use the same tokenizing code to tokenize before translation.

4

Usually this means:

  • the model is not trained enough (not enough training data, too few iterations, too small model, etc.)
  • or the test data contains a lot of out of vocabulary words (out of domain data, different tokenization, etc.)
1 Like
5

Thank you, I think it might be a problem with my validation data. Before I start another 2 week training process, do you have any advice on other knobs to tweak like extra layers etcetera?

6

People usually use as baselines models with 4 layers, 1000 as RNN size and a bidirectional encoder.

7

Hello
I am doing the research about NMT recently. Could you tell me where are you find the ZH-EN training dataset?

1 Like
8

Same here, also interested in training dataset

9

You could find UN parallel corpus, or go to WMT to see what kind of data set you need