Indices are not valid (out of bounds)

Support
1

I’m not sure what is causing this error!? I re-checked the data and found no empty rows in either source, target or the alignment files. Any thoughts?

INFO:tensorflow:loss = 3.9174447, step = 6100 (49.970 sec)
INFO:tensorflow:source_words/sec: 1976
INFO:tensorflow:target_words/sec: 2322
INFO:tensorflow:global_step/sec: 1.98664
INFO:tensorflow:loss = 3.3448534, step = 6200 (50.336 sec)
INFO:tensorflow:source_words/sec: 1987
INFO:tensorflow:target_words/sec: 2334
INFO:tensorflow:global_step/sec: 2.05405
INFO:tensorflow:loss = 2.5677798, step = 6300 (48.684 sec)
INFO:tensorflow:source_words/sec: 1948
INFO:tensorflow:target_words/sec: 2297
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Indices are not valid (out of bounds).  Shape: [34,4]
         [[{{node SparseToDense}}]]
         [[{{node IteratorGetNext}}]]

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/root/anaconda3/bin/onmt-main", line 11, in <module>
    sys.exit(main())
  File "/root/anaconda3/lib/python3.6/site-packages/opennmt/bin/main.py", line 172, in main
    runner.train_and_evaluate(checkpoint_path=args.checkpoint_path)
  File "/root/anaconda3/lib/python3.6/site-packages/opennmt/runner.py", line 297, in train_and_evaluate
    result = tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 471, in train_and_evaluate
    return executor.run()
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 611, in run
    return self.run_local()
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 712, in run_local
    saving_listeners=saving_listeners)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model_default
    saving_listeners)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1407, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
    run_metadata=run_metadata)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
    run_metadata=run_metadata)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1270, in run
    raise six.reraise(*original_exc_info)
  File "/root/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(*args, **kwargs)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1327, in run
    run_metadata=run_metadata)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1091, in run
    return self._sess.run(*args, **kwargs)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Indices are not valid (out of bounds).  Shape: [34,4]
         [[{{node SparseToDense}}]]
         [[node IteratorGetNext (defined at /root/anaconda3/lib/python3.6/site-packages/opennmt/estimator.py:132) ]]
2

ok, looks like there was some issue with few corrupted rows in source, target and alignment which was causing that error. Instead of checking them separately, I re-checked them by combine together and found few rows which had (a) either corrupted alignments, (b) corrupted target text or © corrupted source text. Removing those lines solved the issue.

1 Like