Hi, has anyone used OpenNMT-py to train a speech to text model on a relatively big data set (one that is bigger than AN4, used in the tutorial: e.g. librispeech)? I am trying to do so and I keep having memory errors.
I am using shard size 60, batch size 2, max audio length = 15s.
*** Error in `python’: double free or corruption (fasttop): 0xaf9031c0 ... ... ... (core dumped)
I am also printing
nvidia-smi output each 3s and at some point the GPU memory jumped from 2G to 10G; which I found unreasonable.