Preprocess with -dynamic_dict does not create a valid data file?


(Zewei Chu) #1

I am trying to train a model with the following flags:

-copy_attn -copy_attn_force -copy_loss_by_seqlength -coverage_attn

My understanding is that I need to preprocess the data files with the flag


However, if I got a tiny preprocessed file of only 137 bits, ending with There are no or files. I wonder why this happens.

If I do not provide the flag of > -dynamic_dict, everything works perfectly but I cannot train with the coverage loss and copy mechanism.

(Zewei Chu) #2

I found it’s because I ran out of memory. Adding “max_shard_size” solved my problem.