Greetings.
I’m running into problems using a multisource model with one source being a TFRecord (each line of text corresponds to a sequence of floating-point vectors) and the other source being text. I was able to do this prior to OpenNMT 2. The TFRecord files have been created with OpenNMT 2.
I am getting the error “tensorflow.python.eager.core._FallbackException: This function does not handle the case of the path where all inputs are not already EagerTensors.”
I’m using OpenNMT-tf_v1.21.0, python 3.6
Files are below. Thanks in advance for your help!
command line:
onmt-main
–config 2src_sp32k_200_6L_fulltrain_nogrowth_tf2_onmt240/config.yml
–model 2src_sp32k_200_6L_fulltrain_nogrowth_tf2_onmt240/model.py
–auto_config
train --with_eval
–num_gpus 2
&>>2src_sp32k_200_6L_fulltrain_nogrowth_tf2_onmt240/log.out
yml:
model_dir: 2src_sp32k_200_6L_fulltrain_nogrowth_tf2_onmt240
data:
train_features_file: [
data/train-nocrawl.sp32k.ru,
data/train-nocrawl.wv200b.tf2.ru.gz
]
train_labels_file: data/train-nocrawl.sp32k.en
eval_features_file: [
data/valid.sp32k.ru,
data/valid.wv200b.tf2.ru.gz,
]
eval_labels_file: data/valid.sp32k.en
source_1_vocabulary: data/wmt19-ruen-ru-32k.onmt.vocab
target_vocabulary: data/wmt19-ruen-en-32k.onmt.vocab
train:
save_checkpoints_steps: 10000
batch_size: 256
eval:
external_evaluators: BLEU
model_description.py:
import opennmt as onmt
def model():
return onmt.models.Transformer(
source_inputter=onmt.inputters.ParallelInputter(inputters=[
onmt.inputters.WordEmbedder(embedding_size=512),
onmt.inputters.SequenceRecordInputter(input_depth=390)
]
,reducer=None
,share_parameters=False
,combine_features=True
),
target_inputter=onmt.inputters.WordEmbedder(embedding_size=512),
num_layers=6,
num_units=512,
num_heads=8,
ffn_inner_dim=1024,
dropout=0.1,
attention_dropout=0.1)
log:
OpenNMT-tf 2.4.0
INFO:tensorflow:Using parameters:
data:
eval_features_file:
- data/valid.sp32k.ru
- data/valid.wv200b.tf2.ru.gz
eval_labels_file: data/valid.sp32k.en
source_1_vocabulary: data/wmt19-ruen-ru-32k.onmt.vocab
target_vocabulary: data/wmt19-ruen-en-32k.onmt.vocab
train_features_file:- data/train-nocrawl.sp32k.ru
- data/train-nocrawl.wv200b.tf2.ru.gz
train_labels_file: data/train-nocrawl.sp32k.en
eval:
batch_size: 32
external_evaluators: BLEU
infer:
batch_size: 32
length_bucket_width: 5
model_dir: 2src_sp32k_200_6L_fulltrain_nogrowth_tf2_onmt240
params:
average_loss_in_time: true
beam_width: 4
decay_params:
model_dim: 512
warmup_steps: 8000
decay_type: NoamDecay
label_smoothing: 0.1
learning_rate: 2.0
num_hypotheses: 1
optimizer: LazyAdam
optimizer_params:
beta_1: 0.9
beta_2: 0.998
score:
batch_size: 64
train:
average_last_checkpoints: 8
batch_size: 256
batch_type: tokens
effective_batch_size: 25000
keep_checkpoint_max: 8
length_bucket_width: 1
max_step: 500000
maximum_features_length: 100
maximum_labels_length: 100
sample_buffer_size: -1
save_checkpoints_steps: 10000
save_summary_steps: 1002020-02-21 15:12:33.281049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-02-21 15:12:33.290734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla M40 major: 5 minor: 2 memoryClockRate(GHz): 1.112
pciBusID: 0000:02:00.0
2020-02-21 15:12:33.291831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: Tesla M40 major: 5 minor: 2 memoryClockRate(GHz): 1.112
pciBusID: 0000:81:00.0
2020-02-21 15:12:33.294807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-02-21 15:12:33.299452: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-02-21 15:12:33.303918: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-02-21 15:12:33.307457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-02-21 15:12:33.312305: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-02-21 15:12:33.316914: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-02-21 15:12:33.324542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-21 15:12:33.328831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1
2020-02-21 15:12:33.329327: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-02-21 15:12:33.341950: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599895000 Hz
2020-02-21 15:12:33.343915: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x43b52a0 executing computations on platform Host. Devices:
2020-02-21 15:12:33.343939: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2020-02-21 15:12:33.514518: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x44182b0 executing computations on platform CUDA. Devices:
2020-02-21 15:12:33.514573: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Tesla M40, Compute Capability 5.2
2020-02-21 15:12:33.514583: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (1): Tesla M40, Compute Capability 5.2
2020-02-21 15:12:33.516786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla M40 major: 5 minor: 2 memoryClockRate(GHz): 1.112
pciBusID: 0000:02:00.0
2020-02-21 15:12:33.518515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: Tesla M40 major: 5 minor: 2 memoryClockRate(GHz): 1.112
pciBusID: 0000:81:00.0
2020-02-21 15:12:33.518593: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-02-21 15:12:33.518626: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-02-21 15:12:33.518651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-02-21 15:12:33.518676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-02-21 15:12:33.518700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-02-21 15:12:33.518724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-02-21 15:12:33.518753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-21 15:12:33.525199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1
2020-02-21 15:12:33.525279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-02-21 15:12:33.529175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-21 15:12:33.529209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 1
2020-02-21 15:12:33.529224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N N
2020-02-21 15:12:33.529235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1: N N
2020-02-21 15:12:33.534037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10772 MB memory) -> physical GPU (device: 0, name: Tesla M40, pci bus id: 0000:02:00.0, compute capability: 5.2)
2020-02-21 15:12:33.536312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10772 MB memory) -> physical GPU (device: 1, name: Tesla M40, pci bus id: 0000:81:00.0, compute capability: 5.2)
WARNING:tensorflow:No checkpoint to restore in 2src_sp32k_200_6L_fulltrain_nogrowth_tf2_onmt240
INFO:tensorflow:Training on 17819 examples
WARNING:tensorflow:From /home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/summary/summary_iterator.py:68: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
tf.data.TFRecordDataset(path)
INFO:tensorflow:Accumulate gradients of 49 iterations to reach effective batch size of 25000
2020-02-21 15:12:42.078785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla M40 major: 5 minor: 2 memoryClockRate(GHz): 1.112
pciBusID: 0000:02:00.0
2020-02-21 15:12:42.079455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: Tesla M40 major: 5 minor: 2 memoryClockRate(GHz): 1.112
pciBusID: 0000:81:00.0
2020-02-21 15:12:42.079500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-02-21 15:12:42.079512: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-02-21 15:12:42.079538: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-02-21 15:12:42.079556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-02-21 15:12:42.079567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-02-21 15:12:42.079587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-02-21 15:12:42.079608: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-21 15:12:42.081475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1
2020-02-21 15:12:42.081535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-21 15:12:42.081544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 1
2020-02-21 15:12:42.081550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N N
2020-02-21 15:12:42.081554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1: N N
2020-02-21 15:12:42.082881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 10772 MB memory) -> physical GPU (device: 0, name: Tesla M40, pci bus id: 0000:02:00.0, compute capability: 5.2)
2020-02-21 15:12:42.083547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:1 with 10772 MB memory) -> physical GPU (device: 1, name: Tesla M40, pci bus id: 0000:81:00.0, compute capability: 5.2)
2020-02-21 15:12:43.152207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Traceback (most recent call last):
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py”, line 8961, in shape
name, _ctx._post_execution_callbacks, input, “out_type”, out_type)
tensorflow.python.eager.core._FallbackException: This function does not handle the case of the path where all inputs are not already EagerTensors.During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/gerd/bin/onmt240/bin/onmt-main”, line 13, in
load_entry_point(‘OpenNMT-tf’, ‘console_scripts’, ‘onmt-main’)()
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/bin/main.py”, line 189, in main
checkpoint_path=args.checkpoint_path)
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/runner.py”, line 198, in train
mixed_precision=self._mixed_precision)
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/training.py”, line 44, in init
self._model.create_variables()
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/models/model.py”, line 286, in create_variables
_ = self(features, labels=labels, training=True, step=0)
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/models/model.py”, line 95, in call
return super(Model, self).call(*args, **kwargs)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 891, in call
outputs = self.call(cast_inputs, *args, **kwargs)
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/models/sequence_to_sequence.py”, line 163, in call
training=training)
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/models/sequence_to_sequence.py”, line 197, in _decode_target
initial_state=encoder_state)
File “/home/gerd/OpenNMT-tf_v2.4.0/opennmt/decoders/decoder.py”, line 178, in initial_state
batch_size = tf.shape(sentinel)[0]
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py”, line 432, in shape_v2
return shape(input, name, out_type)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py”, line 458, in shape
return shape_internal(input, name, optimize=True, out_type=out_type)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py”, line 486, in shape_internal
return gen_array_ops.shape(input, name=name, out_type=out_type)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py”, line 8966, in shape
input, out_type=out_type, name=name, ctx=_ctx)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py”, line 9005, in shape_eager_fallback
_attr_T, (input,) = _execute.args_to_matching_eager([input], _ctx)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/eager/execute.py”, line 257, in args_to_matching_eager
t, dtype, preferred_dtype=default_dtype, ctx=ctx))
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/framework/ops.py”, line 1296, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py”, line 286, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py”, line 227, in constant
allow_broadcast=True)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py”, line 235, in _constant_impl
t = convert_to_eager_tensor(value, ctx, dtype)
File “/home/gerd/bin/onmt240/lib64/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py”, line 96, in convert_to_eager_tensor
return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (None) with an unsupported type (<class ‘NoneType’>) to a Tensor.