Issues while train the data using pytorch on 180MB dataset

Support
1

Hi,

Presently I am training the model for ENG to GER language translation.

The dataset has following number of lines :
SOURCE DATA
(base) fujiadmin@offlinetranslation:~/pytorch/OpenNMT-py/data$ wc -l europarl-v7.de-en.en
1920209 europarl-v7.de-en.en

DESTINATION DATA
(base) fujiadmin@offlinetranslation:~/pytorch/OpenNMT-py/data$ wc -l europarl-v7.de-en.de
1920209 europarl-v7.de-en.de

Out of the above 1920209 lines I have used 5000 lines as validation text for both source and destination.

I am getting the following error message after the training 10000 epochs of the total of 100000 epochs.

The error messages are as follows:

error_1
error_1.jpg1600×900 222 KB

number of examples: 924406 as marked in RED
error_2
error_2.jpg1600×900 304 KB
error_3
error_3.jpg1600×900 253 KB
error_4
error_4.jpg1600×900 548 KB
error_5
error_5.jpg1600×900 474 KB

** number of examples: 5000** as marked in RED

Could someone please assist me in resolving this issue ?

Thank You,
Kishor.

2

Hi,
Are there any empty lines in your validation set?

3

Hi,

Yeah there are many empty lines in validation set. How do we get rid of them ?

Thank you,
Kishor.

4

Few python lines that should do the trick:

with open("your_valid_source") as vs, open("your_valid_target") as vt,\
    open("clean_valid_source") as cvs, open("clean_valid_target") as cvt:
    for s, t in zip(vs, vt):
        if s.strip() == "" or t.strip() == "":
            cvs.write(s)
            cvt.write(t)

You could also achieve this with some combination of paste and awk in unix shell.