Do your embeddings be trained on the training data vocabulary ?
Do you extend your embeddings using the embeddings.lua script according to the training data ?
Do your test data share domain/vocabulary with your training data?
Do you use bpe segmentation?
It seems like you are having a representation problem, because, as I see it, you are using enough data and a big vocabulary but you are not having a good coverage.
I will suggest you to try the bpe segmentation. This will help the system to handle the less frequent or unknown words.
Also, you can introduce word-level aligned pairs or even segment/phrase aligned pairs in your training data, but be careful to introduce enough examples to let the system see this particular cases enough times to learn them. Take into account that this will let the system learn how to deal with parts of a sentence, or even single words, more than learning more or better vocabulary inferences.