Nothing in after preprocessing


I remember that I use the same data to preprocess without problems, but these times something strange happened.

I have both * and *, however, the * is empty. The size of and look normally, but I cannot open them to make sure. I opened the and found there’re only 4 default tokens inside.

This didn’t happen to me before, and I cannot find out the reason. I’m processing Chinese data, and it work normally when the data have been segmented. It didn’t work while I split the data to single chars.

Is there any idea could help me fix it or debug it?

I solved it by setting src_seq_length and tgt_seq_length.

More information: