Blas GEMM launch failed when running on char tokanized input

File “/home/mantili/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py”, line 971, in run
return self._sess.run(*args, **kwargs)
File “/home/mantili/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 900, in run
run_metadata_ptr)
File “/home/mantili/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1135, in _run
feed_dict_tensor, options, run_metadata)
File “/home/mantili/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1316, in _do_run
run_metadata)
File “/home/mantili/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(1, 1024), b.shape=(1024, 2048), m=1, n=2048, k=1024
[[Node: seq2seq/parallel_0/seq2seq/encoder/rnn/while/rnn/multi_rnn_cell/cell_0/lstm_cell/MatMul = MatMul[T=DT_FLOAT, _class=[“loc:@optim…rayWriteV3”], transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](seq2seq/parallel_0/seq2seq/encoder/rnn/while/rnn/multi_rnn_cell/cell_0/lstm_cell/concat, seq2seq/parallel_0/seq2seq/encoder/rnn/while/rnn/multi_rnn_cell/cell_0/lstm_cell/MatMul/Enter)]]
[[Node: seq2seq/parallel_0/seq2seq/decoder/LuongAttention/assert_equal/Equal/_227 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name=“edge_1800_seq2seq/parallel_0/seq2seq/decoder/LuongAttention/assert_equal/Equal”, tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op ‘seq2seq/parallel_0/seq2seq/encoder/rnn/while/rnn/multi_rnn_cell/cell_0/lstm_cell/MatMul’,

My vocabulary is only a list of characters + special symbols.

Looks like you are running out of memory? Make sure no other processes are using the GPU and then you may need to reduce some settings (batch size, vocabulary size, model size, sequence length).

1 Like