The training process is hanging


(Andrew) #1

Hello, there.
When I start training, everything stops on the line “Preallocating memory”.
Could you tell me, please, what’s the matter?
Thank you.


(David Landan) #2

Is it stopping or just “hanging”? I see you specified -gpuid 0 which is CPU, so it could just be taking a long time to set up given that you’re using CPU & system RAM.

It’s a little confusing and takes some getting used to, but NVIDIA uses 0-indexing for GPU numbering, and OpenNMT uses 1-indexing, so if you want to use GPU0, you need to specify -gpuid 1.


(Andrew) #3

Yes, it’s hanging. But I use CPU, it’s right.


(David Landan) #4

Then I’m afraid you’ll probably just need to be patient. :wink: Even with a small-ish model, and just shy of a million training pairs, this will take a long time on a CPU. It is strongly recommended that you train on GPU if at all possible. (There are cloud options available if you don’t have access to local GPUs.)