I tried with both 40k and 50k vocabular size (created via opennmt-vocab).
Here is error when I try to train_and_eval with 50k vocab.
I think this is because my corpus is not big enough, but wanted to see if there could be other errors too.
2018-06-03 21:24:01.846172: I tensorflow/core/common_runtime/placer.cc:886] string_to_index/hash_table: (HashTableV2)/job:localhost/replica:0/task:0/device:CPU:0
global_step/Initializer/zeros: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-06-03 21:24:01.846187: I tensorflow/core/common_runtime/placer.cc:886] global_step/Initializer/zeros: (Const)/job:localhost/replica:0/task:0/device:GPU:0
Traceback (most recent call last):
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1322, in _do_call
return fn(*args)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [50001,512] rhs shape= [49656,512]
[[Node: save/Assign_9 = Assign[T=DT_FLOAT, _class=[“loc:@seq2seq/decoder/w_embs”], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](seq2seq/decoder/w_embs, save/RestoreV2/_17)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “trunk/main.py”, line 3, in
main.main()
File “/home/testing/open_nmt/OpenNMT-tf/opennmt/bin/main.py”, line 120, in main
runner.train_and_evaluate()
File “/home/testing/open_nmt/OpenNMT-tf/opennmt/runner.py”, line 144, in train_and_evaluate
tf.estimator.train_and_evaluate(self._estimator, train_spec, eval_spec)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/estimator/training.py”, line 439, in train_and_evaluate
executor.run()
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/estimator/training.py”, line 518, in run
self.run_local()
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/estimator/training.py”, line 657, in run_local
eval_result = evaluator.evaluate_and_export()
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/estimator/training.py”, line 847, in evaluate_and_export
hooks=self._eval_spec.hooks)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py”, line 425, in evaluate
name=name)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/estimator/estimator.py”, line 1117, in _evaluate_model
config=self._session_config)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/evaluation.py”, line 209, in _evaluate_once
session_creator=session_creator, hooks=hooks) as session:
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py”, line 816, in init
stop_grace_period_secs=stop_grace_period_secs)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py”, line 539, in init
self._sess = _RecoverableSession(self._coordinated_creator)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py”, line 1002, in init
_WrappedSession.init(self, self._create_session())
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py”, line 1007, in _create_session
return self._sess_creator.create_session()
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py”, line 696, in create_session
self.tf_sess = self._session_creator.create_session()
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py”, line 467, in create_session
init_fn=self._scaffold.init_fn)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/session_manager.py”, line 279, in prepare_session
config=config)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/session_manager.py”, line 191, in _restore_checkpoint
saver.restore(sess, checkpoint_filename_with_path)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/saver.py”, line 1802, in restore
{self.saver_def.filename_tensor_name: save_path})
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 900, in run
run_metadata_ptr)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1135, in _run
feed_dict_tensor, options, run_metadata)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1316, in _do_run
run_metadata)
File “/home/testing/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [50001,512] rhs shape= [49656,512]
[[Node: save/Assign_9 = Assign[T=DT_FLOAT, _class=[“loc:@seq2seq/decoder/w_embs”], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](seq2seq/decoder/w_embs, save/RestoreV2/_17)]]