Hi Jean, thanks for looking into this. What do you mean by problem is happening quite fast? I have tried running on 10% and 50% of my data and it runs fine. How can I see which utterances ids in any particular minibatch are giving problem?
My hardware is a Titan X gpu, hence this slow speeds are very unexpected. On similar size data, training did not take this long. Does parameter tuning also affect training speed?
In the log that I have shared, could you please take a look at the time after the 5th minibatch. The model took too long to complete minibatch number 6. I have tried the system again, and it gives exactly the same problems as I reported above.